2019-10-13T00:30:40.930 INFO:root:teuthology version: 1.0.0-471e0e3 2019-10-13T00:30:40.931 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.PrgtVl.tmp -- --verbose --lock --description fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml omap_limit/10.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{rhel_7.yaml} tasks/cfuse_workunit_suites_ffsb.yaml} --name pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407525 2019-10-13T00:30:40.974 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T00:30:41.029 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407525 branch: wip-pdonnell-testing-20191012.025926 description: fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml omap_limit/10.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{rhel_7.yaml} tasks/cfuse_workunit_suites_ffsb.yaml} email: pdonnell@redhat.com first_in_suite: false job_id: '4407525' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '7.7' overrides: admin_socket: branch: wip-pdonnell-testing-20191012.025926 ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd_max_omap_entries_per_request: 10 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20191012.025926 sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd owner: scheduled_pdonnell@teuthology priority: 95 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd suite: fs suite_branch: wip-pdonnell-testing-20191012.025926 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - check-counter: counters: mds: - mds.dir_split - workunit: clients: all: - suites/ffsb.sh teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.15246 2019-10-13T00:30:41.029 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa; will attempt to use it 2019-10-13T00:30:41.030 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks 2019-10-13T00:30:41.030 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-13T00:30:41.032 INFO:teuthology.task.internal:Checking packages... 2019-10-13T00:30:41.045 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd' 2019-10-13T00:30:41.045 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-13T00:30:41.045 INFO:teuthology.packaging:ref: None 2019-10-13T00:30:41.046 INFO:teuthology.packaging:tag: None 2019-10-13T00:30:41.046 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20191012.025926 2019-10-13T00:30:41.046 INFO:teuthology.packaging:sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T00:30:41.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-pdonnell-testing-20191012.025926 2019-10-13T00:30:41.202 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6132.ga8ee8ac 2019-10-13T00:30:41.202 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-13T00:30:41.214 INFO:teuthology.task.internal:no buildpackages task found 2019-10-13T00:30:41.214 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-13T00:30:41.229 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-13T00:30:41.230 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T00:30:41.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:30:51.281 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:31:01.296 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:31:11.312 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:31:21.329 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:31:31.339 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:31:41.356 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:31:51.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:32:01.388 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:32:11.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:32:21.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:32:31.435 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:32:41.452 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:32:51.461 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:33:01.470 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:33:11.485 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:33:21.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:33:31.517 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T00:33:41.532 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_pdonnell@teuthology', 'description': '/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407525', 'machine_type': 'smithi'} 2019-10-13T00:33:41.549 DEBUG:teuthology.lock.ops:locked ubuntu@smithi155.front.sepia.ceph.com, ubuntu@smithi142.front.sepia.ceph.com 2019-10-13T00:33:41.674 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-13T00:33:41.865 INFO:teuthology.provision.fog.smithi155:Scheduling deploy of rhel 7.7 2019-10-13T00:33:41.873 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of rhel 7.7 2019-10-13T00:33:42.175 INFO:teuthology.orchestra.console:Power off smithi142 2019-10-13T00:33:42.176 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-13T00:33:42.227 INFO:teuthology.orchestra.console:Power off smithi155 2019-10-13T00:33:42.227 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-13T00:33:42.252 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:42.280 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:46.307 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:46.334 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:50.436 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:50.464 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:50.666 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2019-10-13T00:33:50.695 INFO:teuthology.orchestra.console:Power off for smithi155 completed 2019-10-13T00:33:50.768 INFO:teuthology.orchestra.console:Power on smithi142 2019-10-13T00:33:50.769 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-13T00:33:50.797 INFO:teuthology.orchestra.console:Power on smithi155 2019-10-13T00:33:50.797 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-13T00:33:50.824 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:50.852 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:54.892 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:54.919 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:59.021 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:59.048 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T00:33:59.250 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2019-10-13T00:33:59.275 INFO:teuthology.orchestra.console:Power on for smithi155 completed 2019-10-13T00:33:59.353 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2019-10-13T00:33:59.376 INFO:teuthology.provision.fog.smithi155:Waiting for deploy to finish 2019-10-13T00:36:45.450 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:36:54.510 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:00.519 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:03.561 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:09.578 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:12.624 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:18.638 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:21.705 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:27.690 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:30.761 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:36.746 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:39.817 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:45.804 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:37:54.857 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:38:03.918 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:38:12.974 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:38:22.034 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:38:45.836 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:38:46.037 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:38:46.037 INFO:teuthology.orchestra.run.smithi142:> hostname 2019-10-13T00:38:46.537 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2019-10-13T00:38:46.538 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:38:46.538 INFO:teuthology.orchestra.run.smithi142:> grep smithi142.front.sepia.ceph.com /etc/hosts 2019-10-13T00:38:46.579 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2019-10-13T00:38:46.579 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:38:46.579 INFO:teuthology.orchestra.run.smithi142:> sudo hostname smithi142 2019-10-13T00:38:46.724 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:38:46.724 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/smithi142.front.sepia.ceph.com/smithi142/g' /etc/hosts 2019-10-13T00:38:46.779 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:38:46.779 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/172.21.15.142/172.21.15.142/g' /etc/hosts 2019-10-13T00:38:46.832 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:38:46.832 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/smithi142.front.sepia.ceph.com/smithi142/g' /etc/hostname 2019-10-13T00:38:46.884 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:38:46.884 INFO:teuthology.orchestra.run.smithi142:> sudo hostname smithi142 2019-10-13T00:38:46.935 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2019-10-13T00:38:46.937 DEBUG:teuthology.parallel:result is None 2019-10-13T00:39:28.069 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:39:28.238 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:28.238 INFO:teuthology.orchestra.run.smithi155:> hostname 2019-10-13T00:39:28.742 INFO:teuthology.orchestra.run.smithi155.stdout:smithi155.front.sepia.ceph.com 2019-10-13T00:39:28.743 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:28.743 INFO:teuthology.orchestra.run.smithi155:> grep smithi155.front.sepia.ceph.com /etc/hosts 2019-10-13T00:39:28.784 INFO:teuthology.orchestra.run.smithi155.stdout:172.21.15.155 smithi155.front.sepia.ceph.com smithi155 2019-10-13T00:39:28.785 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:28.785 INFO:teuthology.orchestra.run.smithi155:> sudo hostname smithi155 2019-10-13T00:39:28.921 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:28.922 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/smithi155.front.sepia.ceph.com/smithi155/g' /etc/hosts 2019-10-13T00:39:28.977 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:28.977 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/172.21.15.155/172.21.15.155/g' /etc/hosts 2019-10-13T00:39:29.030 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:29.031 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/smithi155.front.sepia.ceph.com/smithi155/g' /etc/hostname 2019-10-13T00:39:29.083 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:29.083 INFO:teuthology.orchestra.run.smithi155:> sudo hostname smithi155 2019-10-13T00:39:29.136 INFO:teuthology.provision.fog.smithi155:Deploy complete! 2019-10-13T00:39:29.138 DEBUG:teuthology.parallel:result is None 2019-10-13T00:39:29.319 INFO:teuthology.lock.keys:Checking smithi155.front.sepia.ceph.com 2019-10-13T00:39:29.319 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-13T00:39:29.337 INFO:teuthology.lock.keys:Checking smithi142.front.sepia.ceph.com 2019-10-13T00:39:29.337 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-13T00:39:29.351 INFO:teuthology.lock.ops:Updating [smithi155.front.sepia.ceph.com]: set os type and version on server 2019-10-13T00:39:29.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T00:39:29.352 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:39:29.435 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:29.435 INFO:teuthology.orchestra.run.smithi155:> true 2019-10-13T00:39:29.700 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:29.700 INFO:teuthology.orchestra.run.smithi155:> uname -m 2019-10-13T00:39:29.742 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2019-10-13T00:39:29.742 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:29.742 INFO:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2019-10-13T00:39:29.845 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Red Hat Enterprise Linux Server" 2019-10-13T00:39:29.845 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="7.7 (Maipo)" 2019-10-13T00:39:29.845 INFO:teuthology.orchestra.run.smithi155.stdout:ID="rhel" 2019-10-13T00:39:29.846 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="fedora" 2019-10-13T00:39:29.846 INFO:teuthology.orchestra.run.smithi155.stdout:VARIANT="Server" 2019-10-13T00:39:29.846 INFO:teuthology.orchestra.run.smithi155.stdout:VARIANT_ID="server" 2019-10-13T00:39:29.846 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="7.7" 2019-10-13T00:39:29.846 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Employee SKU" 2019-10-13T00:39:29.846 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2019-10-13T00:39:29.846 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server" 2019-10-13T00:39:29.847 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.redhat.com/" 2019-10-13T00:39:29.847 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2019-10-13T00:39:29.847 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:39:29.847 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2019-10-13T00:39:29.847 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.7 2019-10-13T00:39:29.847 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2019-10-13T00:39:29.848 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.7" 2019-10-13T00:39:29.850 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2019-10-13T00:39:29.868 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2019-10-13T00:39:29.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T00:39:29.869 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:39:29.952 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:29.953 INFO:teuthology.orchestra.run.smithi142:> true 2019-10-13T00:39:30.234 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:30.234 INFO:teuthology.orchestra.run.smithi142:> uname -m 2019-10-13T00:39:30.275 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2019-10-13T00:39:30.276 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:30.276 INFO:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2019-10-13T00:39:30.378 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Red Hat Enterprise Linux Server" 2019-10-13T00:39:30.379 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="7.7 (Maipo)" 2019-10-13T00:39:30.379 INFO:teuthology.orchestra.run.smithi142.stdout:ID="rhel" 2019-10-13T00:39:30.379 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="fedora" 2019-10-13T00:39:30.379 INFO:teuthology.orchestra.run.smithi142.stdout:VARIANT="Server" 2019-10-13T00:39:30.380 INFO:teuthology.orchestra.run.smithi142.stdout:VARIANT_ID="server" 2019-10-13T00:39:30.380 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="7.7" 2019-10-13T00:39:30.380 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Employee SKU" 2019-10-13T00:39:30.380 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2019-10-13T00:39:30.380 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server" 2019-10-13T00:39:30.380 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.redhat.com/" 2019-10-13T00:39:30.380 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2019-10-13T00:39:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:39:30.381 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2019-10-13T00:39:30.381 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.7 2019-10-13T00:39:30.381 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2019-10-13T00:39:30.381 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.7" 2019-10-13T00:39:30.382 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2019-10-13T00:39:30.396 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-10-13T00:39:30.423 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4b0St8zDSR0B4D85NtVDdeb+fY0k53IcgxQK2TbuOxZW7phtL+y2+Tpdj/vQsbpO0D8v3hNQ0BNmUT7dFmyJaGcXJEPMtTNY4BseMWc7K2b44JrQWrAoZ9wm0EqRPFTcApOeIzfbUmMAhT4aGtr4/sOVthRtofIEZVMonQAEDroa3mHBf2kFgUvhPHLB6vZPWvRBjVwb6fdGoyV61xw5Bhnt4N5eOptQPPFfcFvsjIT6Fb7mK6vP/P18RJVJOxMpJQewPXZ6JwDkFDTALoXRckijE4pjB6+QLA1obNuPzOS2JahGuVi6Vr/8vLFLpl27GxiUwsBEWI7QQBnsziqLb smithi155.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDj86cZDNkWAQHRr0k23eESV3p4ff7NEOn98aDo43SUQjWuFJJj77k92Q87d+PRT+c6kdVST5Za1q/WT2IYGc6c6SPTd3VhyHLoxpKkv79uDKS4EKA6X0s9YRLeO0o+yBRXO+FNKM1kNe1ej0EvZ/M0LzK1ALGLhmSIqKG49odtZpayMlY+CT593k+46XVA/ETB0IhM0ZQv1d+4FoA5CcdOHxtyaQleBa2qi39EJbhiqA8eZYtdSDNaCiMHlv/OWkTa5ffKZI9xb3PT/N8TM5/8Hfl6W9tQSJT2SU5HnyQAjx/FYCC5VgyoyhfLXuN7oxncmKtTSbyzjdMvKyo2SJH/ 2019-10-13T00:39:30.423 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T00:39:30.464 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-10-13T00:39:30.471 INFO:teuthology.task.internal:Saving configuration 2019-10-13T00:39:30.488 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-10-13T00:39:30.497 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-10-13T00:39:30.511 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-13 00:33:41.538860', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi155.front.sepia.ceph.com', u'os_version': u'7.7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'rhel', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRhXgD/FENSNCE2a0DzYe0eXEjca8RcR43y3tALfunbd+Bq+bVFqM75LwzJCrDgj8zZx0gthpJLTO+rYXPMdGc=', u'description': u'/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407525'} 2019-10-13T00:39:30.524 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-13 00:33:41.538841', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi142.front.sepia.ceph.com', u'os_version': u'7.7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'rhel', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLnOySFx3x3T9bnGANZc3EeXEywlam1MLQz1Bi8xp8sJWEP/bz3tZhrLN+IkhbiHkJoCbep+sUmJxs6EfR0Kzp8=', u'description': u'/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407525'} 2019-10-13T00:39:30.524 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-10-13T00:39:30.534 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-10-13T00:39:30.535 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2019-10-13T00:39:30.535 INFO:teuthology.run_tasks:Running task console_log... 2019-10-13T00:39:30.700 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-13T00:39:30.700 INFO:teuthology.run_tasks:Running task internal.connect... 2019-10-13T00:39:30.712 INFO:teuthology.task.internal:Opening connections... 2019-10-13T00:39:30.712 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2019-10-13T00:39:30.714 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:39:30.799 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2019-10-13T00:39:30.800 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:39:30.883 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-10-13T00:39:30.901 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:30.901 INFO:teuthology.orchestra.run.smithi155:> uname -m 2019-10-13T00:39:31.033 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2019-10-13T00:39:31.034 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:31.034 INFO:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2019-10-13T00:39:31.074 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Red Hat Enterprise Linux Server" 2019-10-13T00:39:31.074 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="7.7 (Maipo)" 2019-10-13T00:39:31.074 INFO:teuthology.orchestra.run.smithi155.stdout:ID="rhel" 2019-10-13T00:39:31.075 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="fedora" 2019-10-13T00:39:31.075 INFO:teuthology.orchestra.run.smithi155.stdout:VARIANT="Server" 2019-10-13T00:39:31.075 INFO:teuthology.orchestra.run.smithi155.stdout:VARIANT_ID="server" 2019-10-13T00:39:31.075 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="7.7" 2019-10-13T00:39:31.075 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Employee SKU" 2019-10-13T00:39:31.075 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2019-10-13T00:39:31.076 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server" 2019-10-13T00:39:31.076 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.redhat.com/" 2019-10-13T00:39:31.077 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2019-10-13T00:39:31.077 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:39:31.077 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2019-10-13T00:39:31.077 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.7 2019-10-13T00:39:31.077 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2019-10-13T00:39:31.078 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.7" 2019-10-13T00:39:31.078 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2019-10-13T00:39:31.090 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:31.090 INFO:teuthology.orchestra.run.smithi142:> uname -m 2019-10-13T00:39:31.154 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2019-10-13T00:39:31.154 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:31.154 INFO:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2019-10-13T00:39:31.196 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Red Hat Enterprise Linux Server" 2019-10-13T00:39:31.196 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="7.7 (Maipo)" 2019-10-13T00:39:31.196 INFO:teuthology.orchestra.run.smithi142.stdout:ID="rhel" 2019-10-13T00:39:31.196 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="fedora" 2019-10-13T00:39:31.196 INFO:teuthology.orchestra.run.smithi142.stdout:VARIANT="Server" 2019-10-13T00:39:31.196 INFO:teuthology.orchestra.run.smithi142.stdout:VARIANT_ID="server" 2019-10-13T00:39:31.197 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="7.7" 2019-10-13T00:39:31.197 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Employee SKU" 2019-10-13T00:39:31.197 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2019-10-13T00:39:31.197 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server" 2019-10-13T00:39:31.197 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.redhat.com/" 2019-10-13T00:39:31.197 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2019-10-13T00:39:31.198 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:39:31.198 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2019-10-13T00:39:31.198 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.7 2019-10-13T00:39:31.198 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2019-10-13T00:39:31.198 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.7" 2019-10-13T00:39:31.199 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2019-10-13T00:39:31.210 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-10-13T00:39:31.258 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-10-13T00:39:31.272 INFO:teuthology.task.internal:Checking for old test directory... 2019-10-13T00:39:31.272 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:31.273 INFO:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2019-10-13T00:39:31.277 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:31.277 INFO:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2019-10-13T00:39:31.317 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-10-13T00:39:31.328 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-10-13T00:39:31.328 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:31.328 INFO:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2019-10-13T00:39:31.366 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:31.366 INFO:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2019-10-13T00:39:31.413 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-10-13T00:39:31.452 INFO:teuthology.run_tasks:Running task kernel... 2019-10-13T00:39:31.471 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mds.a': {'sha1': 'distro', 'kdb': True}, 'mds.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-10-13T00:39:31.471 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-10-13T00:39:31.472 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi155.front.sepia.ceph.com 2019-10-13T00:39:31.472 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:31.472 INFO:teuthology.orchestra.run.smithi155:> sudo package-cleanup -y --oldkernels 2019-10-13T00:39:32.469 INFO:teuthology.orchestra.run.smithi155.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2019-10-13T00:39:32.469 INFO:teuthology.orchestra.run.smithi155.stdout: : manager 2019-10-13T00:39:32.711 INFO:teuthology.orchestra.run.smithi155.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2019-10-13T00:39:34.172 INFO:teuthology.orchestra.run.smithi155.stdout:No old kernels to remove 2019-10-13T00:39:34.181 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi142.front.sepia.ceph.com 2019-10-13T00:39:34.181 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:34.181 INFO:teuthology.orchestra.run.smithi142:> sudo package-cleanup -y --oldkernels 2019-10-13T00:39:35.215 INFO:teuthology.orchestra.run.smithi142.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2019-10-13T00:39:35.215 INFO:teuthology.orchestra.run.smithi142.stdout: : manager 2019-10-13T00:39:35.468 INFO:teuthology.orchestra.run.smithi142.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2019-10-13T00:39:37.034 INFO:teuthology.orchestra.run.smithi142.stdout:No old kernels to remove 2019-10-13T00:39:37.049 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:37.049 INFO:teuthology.orchestra.run.smithi155:> uname -r 2019-10-13T00:39:37.099 INFO:teuthology.orchestra.run.smithi155.stdout:3.10.0-1062.el7.x86_64 2019-10-13T00:39:37.099 INFO:teuthology.task.kernel:Running kernel on smithi155: 3.10.0-1062.el7.x86_64 2019-10-13T00:39:37.099 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:37.100 INFO:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2019-10-13T00:39:37.320 INFO:teuthology.orchestra.run.smithi155.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:39:37.321 INFO:teuthology.orchestra.run.smithi155.stdout: : disabled-repos, subscription-manager 2019-10-13T00:39:37.403 INFO:teuthology.orchestra.run.smithi155.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2019-10-13T00:39:37.409 INFO:teuthology.orchestra.run.smithi155.stdout:Loading mirror speeds from cached hostfile 2019-10-13T00:39:37.415 INFO:teuthology.orchestra.run.smithi155.stdout: * epel: ftp.linux.ncsu.edu 2019-10-13T00:39:38.220 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do 2019-10-13T00:39:38.237 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:38.237 INFO:teuthology.orchestra.run.smithi155:> echo no | sudo yum reinstall kernel || true 2019-10-13T00:39:38.429 INFO:teuthology.orchestra.run.smithi155.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:39:38.430 INFO:teuthology.orchestra.run.smithi155.stdout: : disabled-repos, subscription-manager 2019-10-13T00:39:38.511 INFO:teuthology.orchestra.run.smithi155.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2019-10-13T00:39:38.547 INFO:teuthology.orchestra.run.smithi155.stderr:Skipping the running kernel: kernel-3.10.0-1062.el7.x86_64 2019-10-13T00:39:38.547 INFO:teuthology.orchestra.run.smithi155.stdout:Loading mirror speeds from cached hostfile 2019-10-13T00:39:38.548 INFO:teuthology.orchestra.run.smithi155.stdout: * epel: ftp.linux.ncsu.edu 2019-10-13T00:39:38.552 INFO:teuthology.orchestra.run.smithi155.stderr:Error: Nothing to do 2019-10-13T00:39:38.568 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:38.568 INFO:teuthology.orchestra.run.smithi155:> rpm -q kernel --last | head -n 1 2019-10-13T00:39:38.716 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-3.10.0-1062.el7.x86_64 Thu 10 Oct 2019 03:57:28 PM UTC 2019-10-13T00:39:38.717 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.el7.x86_64 2019-10-13T00:39:38.718 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:38.718 INFO:teuthology.orchestra.run.smithi142:> uname -r 2019-10-13T00:39:38.772 INFO:teuthology.orchestra.run.smithi142.stdout:3.10.0-1062.el7.x86_64 2019-10-13T00:39:38.772 INFO:teuthology.task.kernel:Running kernel on smithi142: 3.10.0-1062.el7.x86_64 2019-10-13T00:39:38.772 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:38.772 INFO:teuthology.orchestra.run.smithi142:> sudo yum install -y kernel 2019-10-13T00:39:38.982 INFO:teuthology.orchestra.run.smithi142.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:39:38.982 INFO:teuthology.orchestra.run.smithi142.stdout: : disabled-repos, subscription-manager 2019-10-13T00:39:39.056 INFO:teuthology.orchestra.run.smithi142.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2019-10-13T00:39:39.062 INFO:teuthology.orchestra.run.smithi142.stdout:Loading mirror speeds from cached hostfile 2019-10-13T00:39:39.063 INFO:teuthology.orchestra.run.smithi142.stdout: * epel: ftp.linux.ncsu.edu 2019-10-13T00:39:40.017 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do 2019-10-13T00:39:40.033 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.033 INFO:teuthology.orchestra.run.smithi142:> echo no | sudo yum reinstall kernel || true 2019-10-13T00:39:40.222 INFO:teuthology.orchestra.run.smithi142.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:39:40.222 INFO:teuthology.orchestra.run.smithi142.stdout: : disabled-repos, subscription-manager 2019-10-13T00:39:40.312 INFO:teuthology.orchestra.run.smithi142.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2019-10-13T00:39:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:Skipping the running kernel: kernel-3.10.0-1062.el7.x86_64 2019-10-13T00:39:40.347 INFO:teuthology.orchestra.run.smithi142.stdout:Loading mirror speeds from cached hostfile 2019-10-13T00:39:40.348 INFO:teuthology.orchestra.run.smithi142.stdout: * epel: ftp.linux.ncsu.edu 2019-10-13T00:39:40.352 INFO:teuthology.orchestra.run.smithi142.stderr:Error: Nothing to do 2019-10-13T00:39:40.368 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.368 INFO:teuthology.orchestra.run.smithi142:> rpm -q kernel --last | head -n 1 2019-10-13T00:39:40.498 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-3.10.0-1062.el7.x86_64 Thu 10 Oct 2019 03:57:28 PM UTC 2019-10-13T00:39:40.499 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.el7.x86_64 2019-10-13T00:39:40.499 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-10-13T00:39:40.499 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:40.499 INFO:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-13T00:39:40.576 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2019-10-13T00:39:40.587 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-10-13T00:39:40.587 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.587 INFO:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-13T00:39:40.650 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2019-10-13T00:39:40.665 INFO:teuthology.run_tasks:Running task internal.base... 2019-10-13T00:39:40.676 INFO:teuthology.task.internal:Creating test directory... 2019-10-13T00:39:40.676 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.676 INFO:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-13T00:39:40.680 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:40.680 INFO:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-13T00:39:40.727 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-10-13T00:39:40.738 INFO:teuthology.run_tasks:Running task internal.archive... 2019-10-13T00:39:40.752 INFO:teuthology.task.internal:Creating archive directory... 2019-10-13T00:39:40.753 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.753 INFO:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-13T00:39:40.759 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:40.759 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-13T00:39:40.812 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-10-13T00:39:40.844 INFO:teuthology.task.internal:Enabling coredump saving... 2019-10-13T00:39:40.845 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.845 INFO:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T00:39:40.849 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:40.850 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T00:39:40.897 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T00:39:40.904 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T00:39:40.906 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-10-13T00:39:40.917 INFO:teuthology.task.internal:Configuring sudo... 2019-10-13T00:39:40.917 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.918 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-13T00:39:40.921 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:40.922 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-13T00:39:40.976 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-10-13T00:39:40.988 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-10-13T00:39:40.988 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:40.989 INFO:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-13T00:39:40.995 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:40.995 INFO:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-13T00:39:41.035 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:41.035 INFO:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T00:39:41.156 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:41.156 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T00:39:41.245 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:41.245 INFO:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T00:39:41.301 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:41.301 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T00:39:41.384 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:41.384 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-13T00:39:41.437 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:41.437 INFO:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T00:39:41.492 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:41.492 INFO:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T00:39:41.595 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:41.595 INFO:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T00:39:41.651 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:41.651 INFO:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T00:39:41.734 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:41.734 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-13T00:39:41.787 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:41.787 INFO:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2019-10-13T00:39:41.854 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:41.854 INFO:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2019-10-13T00:39:41.922 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-10-13T00:39:41.932 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-10-13T00:39:41.980 INFO:teuthology.run_tasks:Running task internal.timer... 2019-10-13T00:39:42.002 INFO:teuthology.task.internal:Starting timer... 2019-10-13T00:39:42.002 INFO:teuthology.run_tasks:Running task pcp... 2019-10-13T00:39:42.035 INFO:teuthology.run_tasks:Running task selinux... 2019-10-13T00:39:42.048 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:42.048 INFO:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2019-10-13T00:39:42.128 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: [ OK ] 2019-10-13T00:39:42.130 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:42.130 INFO:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2019-10-13T00:39:42.223 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: [ OK ] 2019-10-13T00:39:42.225 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-10-13T00:39:42.225 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:42.226 INFO:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2019-10-13T00:39:42.271 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2019-10-13T00:39:42.272 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:42.272 INFO:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2019-10-13T00:39:42.321 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2019-10-13T00:39:42.322 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi142.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2019-10-13T00:39:42.322 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:42.322 INFO:teuthology.orchestra.run.smithi155:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-10-13T00:39:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-13T00:39:42.375 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:42.375 INFO:teuthology.orchestra.run.smithi142:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-10-13T00:39:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-13T00:39:42.428 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-10-13T00:39:42.428 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:39:42.428 INFO:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2019-10-13T00:39:42.488 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:39:42.488 INFO:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2019-10-13T00:39:42.555 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-10-13T00:39:42.567 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2019-10-13T00:39:42.590 INFO:teuthology.repo_utils:Fetching master from upstream 2019-10-13T00:39:42.664 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch master 2019-10-13T00:39:42.695 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'set_python_path.yml'}, {'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-10-13T00:39:42.699 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-10-13T00:39:43.070 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-10-13T00:39:46.784 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-13T00:39:46.807 INFO:teuthology.task.ansible.out: TASK [Check for /usr/bin/python] *********************************************** 2019-10-13T00:39:46.809 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:46 +0000 (0:00:00.066) 0:00:00.066 ******** 2019-10-13T00:39:48.594 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.002211", "end": "2019-10-13 00:39:48.570986", "rc": 0, "start": "2019-10-13 00:39:48.568775", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-13T00:39:48.651 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.002228", "end": "2019-10-13 00:39:48.608273", "rc": 0, "start": "2019-10-13 00:39:48.606045", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} TASK [Set ansible_python_interpreter=/usr/bin/python3] ************************* Sunday 13 October 2019 00:39:48 +0000 (0:00:01.836) 0:00:01.902 ******** 2019-10-13T00:39:48.669 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:48.682 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:48.691 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-13T00:39:48.701 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:48 +0000 (0:00:00.057) 0:00:01.960 ******** 2019-10-13T00:39:48.715 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:48 +0000 (0:00:00.011) 0:00:01.972 ******** 2019-10-13T00:39:49.799 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-13T00:39:49.807 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.142"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:6e0d"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": "ttyS1,115200", "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570927189", "hour": "00", "iso8601": "2019-10-13T00:39:49Z", "iso8601_basic": "20191013T003949730492", "iso8601_basic_short": "20191013T003949", "iso8601_micro": "2019-10-13T00:39:49.730552Z", "minute": "39", "month": "10", "second": "49", "time": "00:39:49", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.142", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6e:0d", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:6e0d", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6e:0d", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rJiMgt0AT4ZxLqO 2019-10-13T00:39:49.808 INFO:teuthology.task.ansible.out:Ew0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD"], "nvme0n1": ["lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["98c5ae01-6063-4cb3-bd64-7f2983a5b25c"], "sda1": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd"], "labels": [], "masters": [], "uuids": ["98c5ae01-6063-4cb3-bd64-7f2983a5b25c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "8 2019-10-13T00:39:49.809 INFO:teuthology.task.ansible.out:9.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "la 2019-10-13T00:39:49.811 INFO:teuthology.task.ansible.out:bels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, "ansible_distribution": "RedHat", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_search_string": "Red Hat", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Maipo", "ansible_distribution_version": "7.7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_ 2019-10-13T00:39:49.812 INFO:teuthology.task.ansible.out:offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:3c", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksu 2019-10-13T00:39:49.814 INFO:teuthology.task.ansible.out:m_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:3d", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic" 2019-10-13T00:39:49.815 INFO:teuthology.task.ansible.out:: "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:6e:0c", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checks 2019-10-13T00:39:49.817 INFO:teuthology.task.ansible.out:um_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.142", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88: 2019-10-13T00:39:49.818 INFO:teuthology.task.ansible.out:6e0d", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:6e:0d", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 3, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-gertzshcfvycodndpwsebusteelsfeog ; /usr/bin/python", "SUDO_GID": "1215", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "4", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi142", "ansible_hostname": "smithi142", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "lo", "enp3s0f0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.el7.x86_64", "ansible_lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_ 2019-10-13T00:39:49.820 INFO:teuthology.task.ansible.out:generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "Maipo", "description": "Red Hat Enterprise Linux Server release 7.7 (Maipo)", "id": "RedHatEnterpriseServer", "major_release": "7", "release": "7.7"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "2dffd4b24d5c4fa487904568daaf3290", "ansible_memfree_mb": 31235, "ansible_m 2019-10-13T00:39:49.821 INFO:teuthology.task.ansible.out:emory_mb": {"nocache": {"free": 31502, "used": 396}, "real": {"free": 31235, "total": 31898, "used": 663}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 31898, "ansible_mounts": [{"block_available": 226806550, "block_size": 4096, "block_total": 240324484, "block_used": 13517934, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999628800, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}], "ansible_nodename": "smithi142", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": ["ttyS1,115200", "tty0", "ttyS1,115200"], "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6928413", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9363C", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final 2019-10-13T00:39:49.823 INFO:teuthology.task.ansible.out:", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLnOySFx3x3T9bnGANZc3EeXEywlam1MLQz1Bi8xp8sJWEP/bz3tZhrLN+IkhbiHkJoCbep+sUmJxs6EfR0Kzp8=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIKWmkFQuqfJfYPP1a+iEhYzy7eyThCCXF6mJZnYRyMK7", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQC4b0St8zDSR0B4D85NtVDdeb+fY0k53IcgxQK2TbuOxZW7phtL+y2+Tpdj/vQsbpO0D8v3hNQ0BNmUT7dFmyJaGcXJEPMtTNY4BseMWc7K2b44JrQWrAoZ9wm0EqRPFTcApOeIzfbUmMAhT4aGtr4/sOVthRtofIEZVMonQAEDroa3mHBf2kFgUvhPHLB6vZPWvRBjVwb6fdGoyV61xw5Bhnt4N5eOptQPPFfcFvsjIT6Fb7mK6vP/P18RJVJOxMpJQewPXZ6JwDkFDTALoXRckijE4pjB6+QLA1obNuPzOS2JahGuVi6Vr/8vLFLpl27GxiUwsBEWI7QQBnsziqLb", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 114, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansi 2019-10-13T00:39:49.824 INFO:teuthology.task.ansible.out:ble_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T00:39:49.825 INFO:teuthology.task.ansible.out: 2019-10-13T00:39:49.882 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.155"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:6ded"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "08/02/2016", "ansible_bios_version": "2.0a", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": "ttyS1,115200", "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570927189", "hour": "00", "iso8601": "2019-10-13T00:39:49Z", "iso8601_basic": "20191013T003949780185", "iso8601_basic_short": "20191013T003949", "iso8601_micro": "2019-10-13T00:39:49.780245Z", "minute": "39", "month": "10", "second": "49", "time": "00:39:49", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.155", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6d:ed", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:6ded", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6d:ed", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RUAGxY0ByR5t5J 2019-10-13T00:39:49.883 INFO:teuthology.task.ansible.out:a6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr"], "nvme0n1": ["lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["4ff5e40b-9654-459d-90d0-7e2a1501e847"], "sda1": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc"], "labels": [], "masters": [], "uuids": ["4ff5e40b-9654-459d-90d0-7e2a1501e847"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": " 2019-10-13T00:39:49.885 INFO:teuthology.task.ansible.out:89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "l 2019-10-13T00:39:49.886 INFO:teuthology.task.ansible.out:abels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "ansible_distribution": "RedHat", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_search_string": "Red Hat", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Maipo", "ansible_distribution_version": "7.7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation 2019-10-13T00:39:49.888 INFO:teuthology.task.ansible.out:_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timesta 2019-10-13T00:39:49.890 INFO:teuthology.task.ansible.out:mp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:a0:fa", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all" 2019-10-13T00:39:49.891 INFO:teuthology.task.ansible.out:], "macaddress": "0c:c4:7a:d9:a0:fb", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], 2019-10-13T00:39:49.893 INFO:teuthology.task.ansible.out:"macaddress": "0c:c4:7a:88:6d:ec", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address" 2019-10-13T00:39:49.894 INFO:teuthology.task.ansible.out:: "172.21.15.155", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:6ded", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:6d:ed", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 3, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-qzxwretzhapiejbcjlxayhkfgouxlpja ; /usr/bin/python", "SUDO_GID": "1215", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "4", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi155", "ansible_hostname": "smithi155", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "lo", "enp3s0f0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.el7.x86_64", "ansible_lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan 2019-10-13T00:39:49.895 INFO:teuthology.task.ansible.out:_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "Maipo", "description": "Red Hat Enterprise Linux Server release 7.7 (Maipo)", "id": "RedHatEnterpriseServer", "major_release": "7", "release": "7.7"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "siz 2019-10-13T00:39:49.897 INFO:teuthology.task.ansible.out:e_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "2dffd4b24d5c4fa487904568daaf3290", "ansible_memfree_mb": 31238, "ansible_memory_mb": {"nocache": {"free": 31505, "used": 393}, "real": {"free": 31238, "total": 31898, "used": 660}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 31898, "ansible_mounts": [{"block_available": 226806543, "block_size": 4096, "block_total": 240324484, "block_used": 13517941, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999600128, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}], "ansible_nodename": "smithi155", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": ["ttyS1,115200", "tty0", "ttyS1,115200"], "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6A00619", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9A0FA", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": 2019-10-13T00:39:49.898 INFO:teuthology.task.ansible.out: true, "type": "CPython", "version": {"major": 2, "micro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRhXg 2019-10-13T00:39:49.900 INFO:teuthology.task.ansible.out:D/FENSNCE2a0DzYe0eXEjca8RcR43y3tALfunbd+Bq+bVFqM75LwzJCrDgj8zZx0gthpJLTO+rYXPMdGc=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAILDR3lvP9qJD3054KExvFqx0CbDhT7lx+fv+HpuzvEsK", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDj86cZDNkWAQHRr0k23eESV3p4ff7NEOn98aDo43SUQjWuFJJj77k92Q87d+PRT+c6kdVST5Za1q/WT2IYGc6c6SPTd3VhyHLoxpKkv79uDKS4EKA6X0s9YRLeO0o+yBRXO+FNKM1kNe1ej0EvZ/M0LzK1ALGLhmSIqKG49odtZpayMlY+CT593k+46XVA/ETB0IhM0ZQv1d+4FoA5CcdOHxtyaQleBa2qi39EJbhiqA8eZYtdSDNaCiMHlv/OWkTa5ffKZI9xb3PT/N8TM5/8Hfl6W9tQSJT2SU5HnyQAjx/FYCC5VgyoyhfLXuN7oxncmKtTSbyzjdMvKyo2SJH/", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 74, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T00:39:49.940 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:49 +0000 (0:00:01.225) 0:00:03.198 ******** 2019-10-13T00:39:50.005 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:50 +0000 (0:00:00.066) 0:00:03.264 ******** 2019-10-13T00:39:50.248 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2019-10-13T00:39:50.250 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-10-13T00:39:50.286 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:50 +0000 (0:00:00.280) 0:00:03.545 ******** 2019-10-13T00:39:50.322 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:50 +0000 (0:00:00.035) 0:00:03.580 ******** 2019-10-13T00:39:50.711 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-10-13T00:39:50.713 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-13T00:39:50.714 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-13T00:39:50.750 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:50 +0000 (0:00:00.427) 0:00:04.008 ******** 2019-10-13T00:39:50.767 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:39:50.786 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:50 +0000 (0:00:00.036) 0:00:04.045 ******** 2019-10-13T00:39:50.791 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-10-13T00:39:50.793 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:50.803 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:39:50.828 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:50 +0000 (0:00:00.041) 0:00:04.087 ******** 2019-10-13T00:39:50.833 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:50.870 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:50 +0000 (0:00:00.041) 0:00:04.128 ******** 2019-10-13T00:39:51.132 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-10-13T00:39:51.134 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:39:51.169 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:51 +0000 (0:00:00.299) 0:00:04.428 ******** 2019-10-13T00:39:51.201 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:51 +0000 (0:00:00.031) 0:00:04.460 ******** 2019-10-13T00:39:51.960 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} 2019-10-13T00:39:52.009 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:52 +0000 (0:00:00.808) 0:00:05.268 ******** 2019-10-13T00:39:52.044 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:52 +0000 (0:00:00.034) 0:00:05.303 ******** 2019-10-13T00:39:52.385 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2019-10-13T00:39:52.385 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:52.393 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:52.569 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:52.580 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:52.747 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:52.748 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:52.917 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:52.929 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:53.093 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:53.094 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:53.272 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:53.273 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:53.439 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:53.440 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T00:39:53.583 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-10-13T00:39:53.590 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-10-13T00:39:53.763 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:53 +0000 (0:00:01.718) 0:00:07.022 ******** 2019-10-13T00:39:53.795 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:53 +0000 (0:00:00.031) 0:00:07.054 ******** 2019-10-13T00:39:54.328 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-13T00:39:54.338 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.142"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:6e0d"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": "ttyS1,115200", "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570927194", "hour": "00", "iso8601": "2019-10-13T00:39:54Z", "iso8601_basic": "20191013T003954254110", "iso8601_basic_short": "20191013T003954", "iso8601_micro": "2019-10-13T00:39:54.254169Z", "minute": "39", "month": "10", "second": "54", "time": "00:39:54", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.142", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6e:0d", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:6e0d", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6e:0d", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rJiMgt0AT4ZxLqO 2019-10-13T00:39:54.339 INFO:teuthology.task.ansible.out:Ew0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD"], "nvme0n1": ["lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["98c5ae01-6063-4cb3-bd64-7f2983a5b25c"], "sda1": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd"], "labels": [], "masters": [], "uuids": ["98c5ae01-6063-4cb3-bd64-7f2983a5b25c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "8 2019-10-13T00:39:54.341 INFO:teuthology.task.ansible.out:9.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "la 2019-10-13T00:39:54.342 INFO:teuthology.task.ansible.out:bels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, "ansible_distribution": "RedHat", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_search_string": "Red Hat", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Maipo", "ansible_distribution_version": "7.7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_ 2019-10-13T00:39:54.343 INFO:teuthology.task.ansible.out:offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:3c", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksu 2019-10-13T00:39:54.345 INFO:teuthology.task.ansible.out:m_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:3d", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic" 2019-10-13T00:39:54.346 INFO:teuthology.task.ansible.out:: "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:6e:0c", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checks 2019-10-13T00:39:54.347 INFO:teuthology.task.ansible.out:um_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.142", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88: 2019-10-13T00:39:54.349 INFO:teuthology.task.ansible.out:6e0d", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:6e:0d", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 3, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-ebigkrbixnfmljhktegttdwetwblmuqf ; /usr/bin/python", "SUDO_GID": "1215", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "4", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi142", "ansible_hostname": "smithi142", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "lo", "enp3s0f0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.el7.x86_64", "ansible_lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_ 2019-10-13T00:39:54.350 INFO:teuthology.task.ansible.out:generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "Maipo", "description": "Red Hat Enterprise Linux Server release 7.7 (Maipo)", "id": "RedHatEnterpriseServer", "major_release": "7", "release": "7.7"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "2dffd4b24d5c4fa487904568daaf3290", "ansible_memfree_mb": 31231, "ansible_m 2019-10-13T00:39:54.352 INFO:teuthology.task.ansible.out:emory_mb": {"nocache": {"free": 31499, "used": 399}, "real": {"free": 31231, "total": 31898, "used": 667}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 31898, "ansible_mounts": [{"block_available": 226806535, "block_size": 4096, "block_total": 240324484, "block_used": 13517949, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999567360, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}], "ansible_nodename": "smithi142", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": ["ttyS1,115200", "tty0", "ttyS1,115200"], "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6928413", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9363C", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final 2019-10-13T00:39:54.353 INFO:teuthology.task.ansible.out:", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLnOySFx3x3T9bnGANZc3EeXEywlam1MLQz1Bi8xp8sJWEP/bz3tZhrLN+IkhbiHkJoCbep+sUmJxs6EfR0Kzp8=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIKWmkFQuqfJfYPP1a+iEhYzy7eyThCCXF6mJZnYRyMK7", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQC4b0St8zDSR0B4D85NtVDdeb+fY0k53IcgxQK2TbuOxZW7phtL+y2+Tpdj/vQsbpO0D8v3hNQ0BNmUT7dFmyJaGcXJEPMtTNY4BseMWc7K2b44JrQWrAoZ9wm0EqRPFTcApOeIzfbUmMAhT4aGtr4/sOVthRtofIEZVMonQAEDroa3mHBf2kFgUvhPHLB6vZPWvRBjVwb6fdGoyV61xw5Bhnt4N5eOptQPPFfcFvsjIT6Fb7mK6vP/P18RJVJOxMpJQewPXZ6JwDkFDTALoXRckijE4pjB6+QLA1obNuPzOS2JahGuVi6Vr/8vLFLpl27GxiUwsBEWI7QQBnsziqLb", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 118, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansi 2019-10-13T00:39:54.355 INFO:teuthology.task.ansible.out:ble_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T00:39:54.365 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.155"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:6ded"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "08/02/2016", "ansible_bios_version": "2.0a", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": "ttyS1,115200", "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570927194", "hour": "00", "iso8601": "2019-10-13T00:39:54Z", "iso8601_basic": "20191013T003954271877", "iso8601_basic_short": "20191013T003954", "iso8601_micro": "2019-10-13T00:39:54.271938Z", "minute": "39", "month": "10", "second": "54", "time": "00:39:54", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.155", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6d:ed", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:6ded", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:6d:ed", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RUAGxY0ByR5t5J 2019-10-13T00:39:54.367 INFO:teuthology.task.ansible.out:a6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr"], "nvme0n1": ["lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["4ff5e40b-9654-459d-90d0-7e2a1501e847"], "sda1": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc"], "labels": [], "masters": [], "uuids": ["4ff5e40b-9654-459d-90d0-7e2a1501e847"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": " 2019-10-13T00:39:54.368 INFO:teuthology.task.ansible.out:89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "l 2019-10-13T00:39:54.370 INFO:teuthology.task.ansible.out:abels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "ansible_distribution": "RedHat", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_search_string": "Red Hat", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Maipo", "ansible_distribution_version": "7.7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation 2019-10-13T00:39:54.371 INFO:teuthology.task.ansible.out:_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timesta 2019-10-13T00:39:54.372 INFO:teuthology.task.ansible.out:mp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:a0:fa", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "on", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all" 2019-10-13T00:39:54.374 INFO:teuthology.task.ansible.out:], "macaddress": "0c:c4:7a:d9:a0:fb", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], 2019-10-13T00:39:54.375 INFO:teuthology.task.ansible.out:"macaddress": "0c:c4:7a:88:6d:ec", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address" 2019-10-13T00:39:54.376 INFO:teuthology.task.ansible.out:: "172.21.15.155", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:6ded", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:6d:ed", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 3, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-havufhajvhqwgqhcmeybovuwdtuhahoq ; /usr/bin/python", "SUDO_GID": "1215", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "4", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi155", "ansible_hostname": "smithi155", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "lo", "enp3s0f0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.el7.x86_64", "ansible_lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan 2019-10-13T00:39:54.378 INFO:teuthology.task.ansible.out:_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed] 2019-10-13T00:39:54.379 INFO:teuthology.task.ansible.out:", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "Maipo", "description": "Red Hat Enterprise Linux Server release 7.7 (Maipo)", "id": "RedHatEnterpriseServer", "major_release": "7", "release": "7.7"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "2dffd4b24d5c4fa487904568daaf3290", "ansible_memfree_mb": 31235, "ansible_memory_mb": {"nocache": {"free": 31503, "used": 395}, "real": {"free": 31235, "total": 31898, "used": 663}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 31898, "ansible_mounts": [{"block_available": 226806528, "block_size": 4096, "block_total": 240324484, "block_used" 2019-10-13T00:39:54.381 INFO:teuthology.task.ansible.out:: 13517956, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999538688, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}], "ansible_nodename": "smithi155", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.el7.x86_64", "console": ["ttyS1,115200", "tty0", "ttyS1,115200"], "crashkernel": "auto", "ro": true, "root": "UUID=0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6A00619", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9A0FA", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", 2019-10-13T00:39:54.382 INFO:teuthology.task.ansible.out:"ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRhXgD/FENSNCE2a0DzYe0eXEjca8RcR43y3tALfunbd+Bq+bVFqM75LwzJCrDgj8zZx0gthpJLTO+rYXPMdGc=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAILDR3lvP9qJD3054KExvFqx0CbDhT7lx+fv+HpuzvEsK", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDj86cZDNkWAQHRr0k23eESV3p4ff7NEOn98aDo43SUQjWuFJJj77k92Q87d+PRT+c6kdVST5Za1q/WT2IYGc6c6SPTd3VhyHLoxpKkv79uDKS4EKA6X0s9YRLeO0o+yBRXO+FNKM1kNe1ej0EvZ/M0LzK1ALGLhmSIqKG49odtZpayMlY+CT593k+46XVA/ETB0IhM0ZQv1d+4FoA5CcdOHxtyaQleBa2qi39EJbhiqA8eZYtdSDNaCiMHlv/OWkTa5ffKZI9xb3PT/N8TM5/8Hfl6W9tQSJT2SU5HnyQAjx/FYCC5VgyoyhfLXuN7oxncmKtTSbyzjdMvKyo2SJH/", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 78, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T00:39:54.476 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.681) 0:00:07.735 ******** 2019-10-13T00:39:54.509 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.032) 0:00:07.768 ******** 2019-10-13T00:39:54.513 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-13T00:39:54.513 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.551 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.041) 0:00:07.810 ******** 2019-10-13T00:39:54.557 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.594 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.042) 0:00:07.852 ******** 2019-10-13T00:39:54.598 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-13T00:39:54.599 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.643 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.049) 0:00:07.902 ******** 2019-10-13T00:39:54.648 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.694 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.051) 0:00:07.953 ******** 2019-10-13T00:39:54.700 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-13T00:39:54.700 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.738 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.044) 0:00:07.997 ******** 2019-10-13T00:39:54.743 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.781 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.042) 0:00:08.040 ******** 2019-10-13T00:39:54.786 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-10-13T00:39:54.789 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.828 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.046) 0:00:08.087 ******** 2019-10-13T00:39:54.833 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:39:54.872 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:54 +0000 (0:00:00.044) 0:00:08.131 ******** 2019-10-13T00:39:54.992 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-10-13T00:39:54.993 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:39:55.032 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:55 +0000 (0:00:00.158) 0:00:08.289 ******** 2019-10-13T00:39:55.047 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:39:55.087 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:39:55 +0000 (0:00:00.056) 0:00:08.346 ******** 2019-10-13T00:39:55.247 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T00:39:55.248 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-13T00:39:55.301 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-13T00:39:55.418 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-13T00:39:55.472 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-13T00:39:55.590 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-13T00:39:55.643 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-13T00:39:55.760 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-13T00:39:55.814 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-13T00:39:55.928 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-13T00:39:55.982 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-13T00:39:56.097 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-13T00:39:56.161 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-13T00:39:56.276 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-13T00:39:56.329 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-13T00:39:56.445 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-13T00:39:56.498 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-13T00:39:56.615 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-13T00:39:56.668 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-13T00:39:56.784 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-13T00:39:56.839 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-13T00:39:56.959 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-13T00:39:57.010 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-13T00:39:57.127 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-13T00:39:57.190 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-13T00:39:57.295 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-13T00:39:57.369 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-13T00:39:57.473 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-13T00:39:57.537 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-13T00:39:57.642 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-13T00:39:57.705 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-13T00:39:57.810 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-13T00:39:57.884 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-13T00:39:57.981 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-13T00:39:58.054 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-13T00:39:58.153 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-13T00:39:58.227 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-13T00:39:58.324 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-13T00:39:58.395 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-13T00:39:58.490 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-13T00:39:58.574 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-13T00:39:58.658 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-13T00:39:58.753 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-13T00:39:58.827 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-13T00:39:58.921 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-13T00:39:58.995 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-13T00:39:59.099 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-13T00:39:59.169 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-13T00:39:59.281 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-13T00:39:59.334 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-13T00:39:59.453 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-13T00:39:59.496 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-13T00:39:59.631 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-13T00:39:59.675 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-13T00:39:59.800 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-13T00:39:59.842 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-13T00:39:59.978 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-13T00:40:00.005 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-13T00:40:00.148 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-13T00:40:00.197 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-13T00:40:00.331 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-13T00:40:00.353 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-13T00:40:00.501 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-13T00:40:00.523 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-13T00:40:00.677 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-13T00:40:00.689 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-13T00:40:00.839 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-13T00:40:00.864 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-13T00:40:01.031 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-13T00:40:01.043 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-13T00:40:01.212 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-13T00:40:01.213 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-13T00:40:01.392 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-13T00:40:01.393 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-13T00:40:01.562 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-13T00:40:01.575 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-13T00:40:01.726 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-13T00:40:01.748 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-13T00:40:01.909 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-13T00:40:01.920 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-13T00:40:02.088 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-13T00:40:02.089 INFO:teuthology.task.ansible.out: 2019-10-13T00:40:02.110 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-13T00:40:02.258 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-13T00:40:02.270 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T00:40:02.447 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T00:40:02.448 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-13T00:40:02.615 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-13T00:40:02.626 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-13T00:40:02.812 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-13T00:40:02.813 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-13T00:40:02.980 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-13T00:40:02.991 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-13T00:40:03.168 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-13T00:40:03.169 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-13T00:40:03.350 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-13T00:40:03.351 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-13T00:40:03.517 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-13T00:40:03.519 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-13T00:40:03.686 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-13T00:40:03.688 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-13T00:40:03.853 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-13T00:40:03.855 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-13T00:40:04.025 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-13T00:40:04.028 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-13T00:40:04.204 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-13T00:40:04.370 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-13T00:40:04.371 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-13T00:40:04.538 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-13T00:40:04.540 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-13T00:40:04.706 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-13T00:40:04.707 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-13T00:40:04.874 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-13T00:40:04.876 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-13T00:40:05.048 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-13T00:40:05.050 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-13T00:40:05.220 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-13T00:40:05.401 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-13T00:40:05.403 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-13T00:40:05.563 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-13T00:40:05.564 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-13T00:40:05.731 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-13T00:40:05.742 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-13T00:40:05.899 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-13T00:40:05.900 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-13T00:40:06.079 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-13T00:40:06.081 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-13T00:40:06.250 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-13T00:40:06.251 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-13T00:40:06.428 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-13T00:40:06.429 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-13T00:40:06.593 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-13T00:40:06.594 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-13T00:40:06.768 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-13T00:40:06.770 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-13T00:40:06.942 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-13T00:40:06.944 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-13T00:40:06.946 INFO:teuthology.task.ansible.out: 2019-10-13T00:40:07.111 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-13T00:40:07.112 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-13T00:40:07.279 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-13T00:40:07.291 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-13T00:40:07.458 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-13T00:40:07.460 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-13T00:40:07.628 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-13T00:40:07.629 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-13T00:40:07.800 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-13T00:40:07.801 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-13T00:40:07.978 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-13T00:40:07.980 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-13T00:40:08.147 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-13T00:40:08.148 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T00:40:08.315 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T00:40:08.316 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-13T00:40:08.486 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-13T00:40:08.498 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-13T00:40:08.654 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-13T00:40:08.666 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-13T00:40:08.842 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-13T00:40:08.854 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-13T00:40:09.011 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-13T00:40:09.022 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-13T00:40:09.189 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-13T00:40:09.212 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-13T00:40:09.349 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-13T00:40:09.382 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-13T00:40:09.518 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-13T00:40:09.550 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-13T00:40:09.687 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-13T00:40:09.729 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-13T00:40:09.871 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-13T00:40:09.897 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-13T00:40:10.028 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-13T00:40:10.081 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-13T00:40:10.204 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-13T00:40:10.252 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-13T00:40:10.368 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-13T00:40:10.431 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-13T00:40:10.546 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-13T00:40:10.600 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-13T00:40:10.716 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-13T00:40:10.780 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-13T00:40:10.886 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-13T00:40:10.956 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-13T00:40:11.065 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-13T00:40:11.129 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-13T00:40:11.236 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-13T00:40:11.312 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-13T00:40:11.412 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-13T00:40:11.487 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-13T00:40:11.584 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-13T00:40:11.669 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-13T00:40:11.753 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-13T00:40:11.840 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-13T00:40:11.934 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-13T00:40:12.018 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-13T00:40:12.104 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-13T00:40:12.191 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-13T00:40:12.277 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-13T00:40:12.359 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-13T00:40:12.443 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-13T00:40:12.528 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-13T00:40:12.622 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-13T00:40:12.706 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-13T00:40:12.790 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-13T00:40:12.874 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-13T00:40:12.969 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-13T00:40:13.048 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-13T00:40:13.144 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-13T00:40:13.228 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-13T00:40:13.312 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-13T00:40:13.406 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-13T00:40:13.480 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-13T00:40:13.564 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-13T00:40:13.648 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-13T00:40:13.754 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-13T00:40:13.823 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-13T00:40:13.927 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-13T00:40:14.000 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-10-13T00:40:14.109 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-13T00:40:14.164 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-10-13T00:40:14.280 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-10-13T00:40:14.345 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-10-13T00:40:14.453 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-10-13T00:40:14.512 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-10-13T00:40:14.630 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-10-13T00:40:14.713 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-10-13T00:40:14.801 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-10-13T00:40:14.896 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-10-13T00:40:15.006 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-10-13T00:40:15.042 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:15 +0000 (0:00:19.954) 0:00:28.300 ******** 2019-10-13T00:40:15.074 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-13T00:40:15.113 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:15 +0000 (0:00:00.071) 0:00:28.372 ******** 2019-10-13T00:40:15.151 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T00:40:15.152 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-10-13T00:40:15.165 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-13T00:40:15.171 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-13T00:40:15.172 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-13T00:40:15.174 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-13T00:40:15.175 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-13T00:40:15.177 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-13T00:40:15.178 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h 2019-10-13T00:40:15.180 INFO:teuthology.task.ansible.out:7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0 2019-10-13T00:40:15.181 INFO:teuthology.task.ansible.out:bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux 2019-10-13T00:40:15.187 INFO:teuthology.task.ansible.out:", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "o 2019-10-13T00:40:15.189 INFO:teuthology.task.ansible.out:vpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e 2019-10-13T00:40:15.190 INFO:teuthology.task.ansible.out:5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, 2019-10-13T00:40:15.192 INFO:teuthology.task.ansible.out: {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-13T00:40:15.217 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T00:40:15.308 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-10-13T00:40:15.310 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:15 +0000 (0:00:00.195) 0:00:28.567 ******** 2019-10-13T00:40:15.345 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:15 +0000 (0:00:00.036) 0:00:28.604 ******** 2019-10-13T00:40:15.372 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-13T00:40:15.409 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:15 +0000 (0:00:00.063) 0:00:28.667 ******** 2019-10-13T00:40:15.459 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-13T00:40:15.463 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-13T00:40:15.465 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-13T00:40:15.466 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-13T00:40:15.468 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-13T00:40:15.469 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-13T00:40:15.470 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h 2019-10-13T00:40:15.472 INFO:teuthology.task.ansible.out:7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6 2019-10-13T00:40:15.473 INFO:teuthology.task.ansible.out:c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa 2019-10-13T00:40:15.475 INFO:teuthology.task.ansible.out:2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db2 2019-10-13T00:40:15.476 INFO:teuthology.task.ansible.out:93dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.red 2019-10-13T00:40:15.478 INFO:teuthology.task.ansible.out:hat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f 2019-10-13T00:40:15.479 INFO:teuthology.task.ansible.out:57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-13T00:40:15.520 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:15 +0000 (0:00:00.111) 0:00:28.779 ******** 2019-10-13T00:40:16.229 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:16.237 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-10-13T00:40:16.237 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-13T00:40:16.280 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:16.281 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:16 +0000 (0:00:00.760) 0:00:29.540 ******** 2019-10-13T00:40:16.298 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-13T00:40:16.343 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:16 +0000 (0:00:00.062) 0:00:29.602 ******** 2019-10-13T00:40:16.520 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T00:40:16.522 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-13T00:40:16.594 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-13T00:40:16.699 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-13T00:40:16.773 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-13T00:40:16.898 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-13T00:40:16.971 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-13T00:40:17.075 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-13T00:40:17.149 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-13T00:40:17.256 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-13T00:40:17.258 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-13T00:40:17.331 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-13T00:40:17.333 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-13T00:40:17.425 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-13T00:40:17.510 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-13T00:40:17.615 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-10-13T00:40:17.616 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-13T00:40:17.688 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-10-13T00:40:17.690 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-13T00:40:17.803 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-13T00:40:17.876 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-13T00:40:17.971 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-13T00:40:17.983 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.047 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-13T00:40:18.057 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.162 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-13T00:40:18.163 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.174 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.175 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.185 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.229 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-13T00:40:18.231 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-10-13T00:40:18.240 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.251 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.252 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.263 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:18.357 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-13T00:40:18.441 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-13T00:40:18.535 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-13T00:40:18.537 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-13T00:40:18.620 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-13T00:40:18.621 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-13T00:40:18.705 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-13T00:40:18.789 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-13T00:40:18.884 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-13T00:40:18.967 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-13T00:40:19.072 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-13T00:40:19.146 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-13T00:40:19.250 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-13T00:40:19.323 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-13T00:40:19.428 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-13T00:40:19.511 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-13T00:40:19.605 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-13T00:40:19.710 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-13T00:40:19.784 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-13T00:40:19.787 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-13T00:40:19.889 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-13T00:40:19.890 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-13T00:40:19.973 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-13T00:40:19.975 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-13T00:40:20.072 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-13T00:40:20.074 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-13T00:40:20.146 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-13T00:40:20.241 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-13T00:40:20.337 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-13T00:40:20.421 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-13T00:40:20.506 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-13T00:40:20.507 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-13T00:40:20.601 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-13T00:40:20.603 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-13T00:40:20.685 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-13T00:40:20.790 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-13T00:40:20.864 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-13T00:40:20.968 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-13T00:40:21.063 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-10-13T00:40:21.147 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-10-13T00:40:21.245 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-13T00:40:21.319 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-13T00:40:21.424 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-13T00:40:21.497 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-13T00:40:21.612 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-13T00:40:21.614 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-13T00:40:21.677 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-13T00:40:21.678 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-13T00:40:21.802 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+ 2019-10-13T00:40:21.803 INFO:teuthology.task.ansible.out:oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-13T00:40:21.855 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+ 2019-10-13T00:40:21.856 INFO:teuthology.task.ansible.out:oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-13T00:40:21.970 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-13T00:40:22.023 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-13T00:40:22.148 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-13T00:40:22.201 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-13T00:40:22.327 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-13T00:40:22.380 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-13T00:40:22.505 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-13T00:40:22.558 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-13T00:40:22.674 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-13T00:40:22.750 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-13T00:40:22.854 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-13T00:40:22.917 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-13T00:40:23.032 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T00:40:23.095 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T00:40:23.211 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-13T00:40:23.274 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-13T00:40:23.399 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-13T00:40:23.442 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-13T00:40:23.563 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-13T00:40:23.616 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-13T00:40:23.744 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-13T00:40:23.745 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-13T00:40:23.797 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-13T00:40:23.799 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-13T00:40:23.922 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-13T00:40:23.924 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-13T00:40:23.986 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-13T00:40:23.987 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-13T00:40:24.103 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-13T00:40:24.166 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-13T00:40:24.282 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-13T00:40:24.367 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-13T00:40:24.461 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-13T00:40:24.555 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-13T00:40:24.639 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-13T00:40:24.736 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-13T00:40:24.806 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-13T00:40:24.901 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-13T00:40:24.985 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-13T00:40:25.080 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-13T00:40:25.164 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-13T00:40:25.258 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-13T00:40:25.342 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-13T00:40:25.438 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-13T00:40:25.534 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-13T00:40:25.535 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-13T00:40:25.537 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-13T00:40:25.628 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-13T00:40:25.629 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-13T00:40:25.631 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-13T00:40:25.716 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-13T00:40:25.806 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-13T00:40:25.900 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-13T00:40:25.973 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-13T00:40:26.078 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-13T00:40:26.151 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-13T00:40:26.256 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-13T00:40:26.258 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-13T00:40:26.329 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-13T00:40:26.331 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-13T00:40:26.434 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-13T00:40:26.508 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-13T00:40:26.602 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-13T00:40:26.696 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-13T00:40:26.780 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-13T00:40:26.874 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1r 2019-10-13T00:40:26.875 INFO:teuthology.task.ansible.out:J7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-13T00:40:26.958 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-13T00:40:27.055 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-13T00:40:27.128 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-13T00:40:27.213 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-13T00:40:27.307 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-13T00:40:27.391 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-13T00:40:27.486 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-13T00:40:27.488 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-13T00:40:27.570 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-13T00:40:27.572 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-13T00:40:27.669 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-13T00:40:27.764 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-13T00:40:27.830 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-13T00:40:27.924 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-13T00:40:28.013 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-13T00:40:28.015 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-13T00:40:28.108 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-13T00:40:28.110 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-13T00:40:28.181 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-13T00:40:28.275 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-13T00:40:28.359 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-13T00:40:28.454 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-13T00:40:28.558 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-13T00:40:28.623 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-13T00:40:28.737 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-13T00:40:28.790 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-13T00:40:28.915 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-13T00:40:28.968 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-13T00:40:29.083 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T00:40:29.136 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T00:40:29.256 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-13T00:40:29.329 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-13T00:40:29.437 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-13T00:40:29.501 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-13T00:40:29.612 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-13T00:40:29.675 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-13T00:40:29.789 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-13T00:40:29.853 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-13T00:40:29.957 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-13T00:40:30.032 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-13T00:40:30.136 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-13T00:40:30.212 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-13T00:40:30.306 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-13T00:40:30.392 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-13T00:40:30.495 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-13T00:40:30.570 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-13T00:40:30.664 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-13T00:40:30.738 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-13T00:40:30.844 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-13T00:40:30.908 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-13T00:40:31.012 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-13T00:40:31.086 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-13T00:40:31.190 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-13T00:40:31.264 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-13T00:40:31.372 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-13T00:40:31.435 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-13T00:40:31.550 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/Z 2019-10-13T00:40:31.552 INFO:teuthology.task.ansible.out:yG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-13T00:40:31.614 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/Z 2019-10-13T00:40:31.615 INFO:teuthology.task.ansible.out:yG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-13T00:40:31.730 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9V 2019-10-13T00:40:31.732 INFO:teuthology.task.ansible.out:ENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-13T00:40:31.785 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9V 2019-10-13T00:40:31.787 INFO:teuthology.task.ansible.out:ENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-13T00:40:31.912 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-13T00:40:31.965 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-13T00:40:32.080 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-13T00:40:32.144 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-13T00:40:32.258 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-13T00:40:32.321 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-13T00:40:32.446 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-13T00:40:32.511 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-13T00:40:32.626 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-13T00:40:32.679 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-13T00:40:32.796 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-13T00:40:32.861 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-13T00:40:32.965 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-13T00:40:33.028 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-13T00:40:33.142 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-13T00:40:33.205 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-13T00:40:33.320 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-10-13T00:40:33.322 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-13T00:40:33.383 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-10-13T00:40:33.385 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-13T00:40:33.493 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-13T00:40:33.495 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-13T00:40:33.566 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-13T00:40:33.568 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-13T00:40:33.681 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-13T00:40:33.734 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-13T00:40:33.858 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-13T00:40:33.910 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-13T00:40:34.024 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-13T00:40:34.077 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-13T00:40:34.201 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-13T00:40:34.253 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-13T00:40:34.373 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-13T00:40:34.425 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-13T00:40:34.559 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-13T00:40:34.602 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-13T00:40:34.736 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-13T00:40:34.769 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-13T00:40:34.909 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-13T00:40:34.941 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-13T00:40:35.088 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-13T00:40:35.122 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-13T00:40:35.258 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-13T00:40:35.310 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-13T00:40:35.435 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-13T00:40:35.490 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-13T00:40:35.604 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-13T00:40:35.668 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-13T00:40:35.782 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-13T00:40:35.835 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-13T00:40:35.972 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-13T00:40:36.044 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-13T00:40:36.128 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:36 +0000 (0:00:19.784) 0:00:49.387 ******** 2019-10-13T00:40:36.162 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:36 +0000 (0:00:00.033) 0:00:49.421 ******** 2019-10-13T00:40:36.172 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.189 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.190 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.192 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.202 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.203 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.213 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.214 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.215 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.225 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.226 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.237 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.238 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.249 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.249 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.261 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.261 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.390 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-13T00:40:36.392 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.455 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-13T00:40:36.466 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:36.572 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-13T00:40:36.634 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-13T00:40:36.749 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-13T00:40:36.812 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-13T00:40:36.934 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-13T00:40:37.014 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-13T00:40:37.098 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-13T00:40:37.099 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.110 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.121 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.121 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.132 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.133 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.144 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.145 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.156 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.167 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.168 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.179 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.180 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-13T00:40:37.182 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.192 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.197 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.199 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.209 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.210 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.221 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.222 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.223 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.233 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.245 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.246 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.257 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.258 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.259 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.261 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.270 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.271 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.272 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.274 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.285 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.285 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.287 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.298 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.299 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.300 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.311 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.311 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.313 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.323 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.323 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.325 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.326 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.336 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.336 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.338 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.349 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.349 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.351 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.352 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.364 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.366 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.378 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.389 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.390 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.391 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.402 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.402 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.404 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.414 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.415 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.416 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.427 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.428 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.429 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.440 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.441 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.442 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.454 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.465 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.465 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.467 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.477 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.478 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.479 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.490 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.491 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.492 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.503 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.503 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.505 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.515 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.516 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.517 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.528 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.529 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.530 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.541 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.541 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.543 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.553 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.554 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.555 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.557 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.569 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.570 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.581 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.581 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.583 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.593 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.594 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.596 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.606 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.607 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.608 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.619 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.619 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.621 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.622 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.633 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.634 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.635 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.646 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.646 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.648 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.658 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.659 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.660 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.671 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.672 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.673 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.683 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.684 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.685 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.696 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.697 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.698 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.709 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.710 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.712 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.722 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.723 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.725 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.726 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.757 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.758 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.799 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:37 +0000 (0:00:01.637) 0:00:51.058 ******** 2019-10-13T00:40:37.807 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.808 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.816 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a3 2019-10-13T00:40:37.818 INFO:teuthology.task.ansible.out:4f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ET 2019-10-13T00:40:37.819 INFO:teuthology.task.ansible.out:OAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.831 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.831 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.882 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:37 +0000 (0:00:00.082) 0:00:51.140 ******** 2019-10-13T00:40:37.887 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T00:40:37.887 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.889 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.890 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.892 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.893 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:37.953 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:37 +0000 (0:00:00.071) 0:00:51.212 ******** 2019-10-13T00:40:37.979 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-10-13T00:40:37.980 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:38.016 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:38 +0000 (0:00:00.062) 0:00:51.275 ******** 2019-10-13T00:40:38.083 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T00:40:38.085 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-13T00:40:38.221 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-13T00:40:38.255 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-13T00:40:38.380 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-13T00:40:38.422 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-13T00:40:38.554 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-13T00:40:38.586 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-13T00:40:38.711 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-13T00:40:38.754 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-13T00:40:38.890 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-13T00:40:38.913 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-13T00:40:39.049 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-13T00:40:39.081 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-13T00:40:39.216 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-13T00:40:39.251 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-13T00:40:39.376 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-13T00:40:39.419 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-13T00:40:39.544 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-13T00:40:39.576 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-13T00:40:39.712 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-13T00:40:39.764 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-13T00:40:39.879 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-13T00:40:39.943 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-13T00:40:40.037 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-13T00:40:40.124 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-13T00:40:40.210 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-13T00:40:40.283 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-13T00:40:40.381 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-13T00:40:40.455 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-13T00:40:40.539 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-13T00:40:40.623 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-13T00:40:40.717 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-13T00:40:40.791 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-13T00:40:40.885 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-13T00:40:40.959 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-13T00:40:41.051 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-13T00:40:41.121 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-13T00:40:41.216 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-13T00:40:41.289 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-13T00:40:41.384 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-13T00:40:41.457 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-13T00:40:41.553 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-13T00:40:41.627 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-13T00:40:41.721 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-13T00:40:41.786 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-13T00:40:41.890 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-13T00:40:41.965 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-13T00:40:42.070 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-13T00:40:42.123 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-13T00:40:42.237 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-13T00:40:42.290 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-13T00:40:42.417 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-13T00:40:42.450 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-13T00:40:42.587 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-13T00:40:42.619 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-13T00:40:42.681 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:42 +0000 (0:00:04.664) 0:00:55.940 ******** 2019-10-13T00:40:42.708 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-13T00:40:42.709 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => { "msg": "Host smithi142.front.sepia.ceph.com is running RedHat 7.7 (Maipo)" } 2019-10-13T00:40:42.770 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:42 +0000 (0:00:00.089) 0:00:56.029 ******** 2019-10-13T00:40:42.776 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T00:40:42.777 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-13T00:40:42.809 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-13T00:40:42.810 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2019-10-13T00:40:42.848 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:42 +0000 (0:00:00.077) 0:00:56.107 ******** 2019-10-13T00:40:42.882 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:42 +0000 (0:00:00.033) 0:00:56.140 ******** 2019-10-13T00:40:42.887 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-13T00:40:42.888 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:42.926 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:42 +0000 (0:00:00.044) 0:00:56.185 ******** 2019-10-13T00:40:42.932 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-13T00:40:42.933 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => { "msg": "Host smithi155.front.sepia.ceph.com is running RedHat 7.7 (Maipo)" } 2019-10-13T00:40:42.974 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:42 +0000 (0:00:00.048) 0:00:56.233 ******** 2019-10-13T00:40:43.010 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-13T00:40:43.011 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2019-10-13T00:40:43.046 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.071) 0:00:56.305 ******** 2019-10-13T00:40:43.072 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-13T00:40:43.074 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.108 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.061) 0:00:56.366 ******** 2019-10-13T00:40:43.167 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-13T00:40:43.168 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.064153", "end": "2019-10-13 00:40:43.141007", "rc": 0, "start": "2019-10-13 00:40:43.076854", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-13T00:40:43.205 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.097) 0:00:56.464 ******** 2019-10-13T00:40:43.232 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-13T00:40:43.234 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.271 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.066) 0:00:56.530 ******** 2019-10-13T00:40:43.299 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-13T00:40:43.300 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.337 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.065) 0:00:56.596 ******** 2019-10-13T00:40:43.354 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-13T00:40:43.355 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.069900", "end": "2019-10-13 00:40:43.327787", "rc": 0, "start": "2019-10-13 00:40:43.257887", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-13T00:40:43.392 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.055) 0:00:56.651 ******** 2019-10-13T00:40:43.397 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-13T00:40:43.398 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.436 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.043) 0:00:56.695 ******** 2019-10-13T00:40:43.440 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-13T00:40:43.441 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.479 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.042) 0:00:56.737 ******** 2019-10-13T00:40:43.483 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-10-13T00:40:43.483 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.520 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.041) 0:00:56.778 ******** 2019-10-13T00:40:43.525 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-13T00:40:43.528 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.565 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.045) 0:00:56.824 ******** 2019-10-13T00:40:43.570 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-13T00:40:43.571 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.607 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.041) 0:00:56.866 ******** 2019-10-13T00:40:43.612 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-13T00:40:43.613 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.650 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.042) 0:00:56.909 ******** 2019-10-13T00:40:43.655 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-13T00:40:43.656 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2019-10-13T00:40:43.693 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.042) 0:00:56.951 ******** 2019-10-13T00:40:43.698 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-10-13T00:40:43.699 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.740 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.047) 0:00:56.999 ******** 2019-10-13T00:40:43.744 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-13T00:40:43.745 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T00:40:43.746 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T00:40:43.782 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.041) 0:00:57.041 ******** 2019-10-13T00:40:43.787 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-13T00:40:43.788 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:43.825 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.042) 0:00:57.084 ******** 2019-10-13T00:40:43.829 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-13T00:40:43.831 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2019-10-13T00:40:43.879 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.054) 0:00:57.138 ******** 2019-10-13T00:40:43.885 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-13T00:40:43.886 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2019-10-13T00:40:43.923 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:43 +0000 (0:00:00.044) 0:00:57.182 ******** 2019-10-13T00:40:43.970 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-13T00:40:43.972 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T00:40:44.008 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:44 +0000 (0:00:00.084) 0:00:57.267 ******** 2019-10-13T00:40:44.044 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-13T00:40:44.045 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2019-10-13T00:40:44.081 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:44 +0000 (0:00:00.073) 0:00:57.340 ******** 2019-10-13T00:40:45.761 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1570927244.71-aChVkX/katello-ca-consumer-latest.noarchLYWMFA.rpm is already installed"]} 2019-10-13T00:40:45.804 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:45 +0000 (0:00:01.723) 0:00:59.063 ******** 2019-10-13T00:40:45.813 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1570927244.71-oEygsd/katello-ca-consumer-latest.noarch9_SLLX.rpm is already installed"]} 2019-10-13T00:40:45.857 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:45 +0000 (0:00:00.052) 0:00:59.116 ******** 2019-10-13T00:40:46.437 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-13T00:40:46.438 INFO:teuthology.task.ansible.out:fatal: [smithi142.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2019-10-13T00:40:46.473 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:46 +0000 (0:00:00.616) 0:00:59.732 ******** 2019-10-13T00:40:46.512 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-13T00:40:46.517 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2019-10-13T00:40:46.559 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:46 +0000 (0:00:00.085) 0:00:59.818 ******** 2019-10-13T00:40:46.567 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-13T00:40:46.568 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2019-10-13T00:40:46.607 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:46 +0000 (0:00:00.045) 0:00:59.864 ******** 2019-10-13T00:40:46.645 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-13T00:40:46.646 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2019-10-13T00:40:46.682 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:46 +0000 (0:00:00.077) 0:00:59.941 ******** 2019-10-13T00:40:53.341 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:53.357 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-10-13T00:40:53.359 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-13T00:40:53.393 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:53 +0000 (0:00:06.710) 0:01:06.652 ******** 2019-10-13T00:40:53.417 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:53.430 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-10-13T00:40:53.431 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2019-10-13T00:40:53.469 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:53 +0000 (0:00:00.075) 0:01:06.728 ******** 2019-10-13T00:40:53.502 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:53.517 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-10-13T00:40:53.517 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-13T00:40:53.554 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:53 +0000 (0:00:00.085) 0:01:06.813 ******** 2019-10-13T00:40:53.579 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:53.591 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-10-13T00:40:53.593 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2019-10-13T00:40:53.626 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:53 +0000 (0:00:00.072) 0:01:06.885 ******** 2019-10-13T00:40:54.153 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-10-13T00:40:54.154 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.482719", "end": "2019-10-13 00:40:54.119514", "failed_when_result": false, "rc": 0, "start": "2019-10-13 00:40:53.636795", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7Server\n8\n8.0", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7Server", "8", "8.0"]} 2019-10-13T00:40:54.192 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.565) 0:01:07.451 ******** 2019-10-13T00:40:54.217 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-10-13T00:40:54.219 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:54.254 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.062) 0:01:07.513 ******** 2019-10-13T00:40:54.278 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|skipped` use `result is skipped`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:54.290 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-13T00:40:54.291 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : List CDN releases available to system] ************************** 2019-10-13T00:40:54.293 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.470157", "end": "2019-10-13 00:40:54.265484", "failed_when_result": false, "rc": 0, "start": "2019-10-13 00:40:53.795327", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7Server\n8\n8.0", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7Server", "8", "8.0"]} 2019-10-13T00:40:54.329 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.073) 0:01:07.586 ******** 2019-10-13T00:40:54.375 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.047) 0:01:07.634 ******** 2019-10-13T00:40:54.381 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-13T00:40:54.381 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T00:40:54.418 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.042) 0:01:07.676 ******** 2019-10-13T00:40:54.424 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-10-13T00:40:54.425 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:54.459 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.041) 0:01:07.718 ******** 2019-10-13T00:40:54.466 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:54.481 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|skipped` use `result is skipped`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:54.501 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.041) 0:01:07.760 ******** 2019-10-13T00:40:54.506 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-13T00:40:54.506 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:54.543 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.042) 0:01:07.802 ******** 2019-10-13T00:40:54.569 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-13T00:40:54.571 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T00:40:54.605 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.061) 0:01:07.864 ******** 2019-10-13T00:40:54.640 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T00:40:54.641 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:54.676 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:54 +0000 (0:00:00.071) 0:01:07.935 ******** 2019-10-13T00:40:58.090 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:03.402128", "end": "2019-10-13 00:40:58.060657", "rc": 0, "start": "2019-10-13 00:40:54.658529", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2019-10-13T00:40:58.126 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:03.449) 0:01:11.385 ******** 2019-10-13T00:40:58.151 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|skipped` use `result is skipped`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:58.173 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-13T00:40:58.173 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2019-10-13T00:40:58.209 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.083) 0:01:11.468 ******** 2019-10-13T00:40:58.233 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:58.246 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-10-13T00:40:58.247 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:58.282 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.072) 0:01:11.541 ******** 2019-10-13T00:40:58.318 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-13T00:40:58.320 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2019-10-13T00:40:58.355 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.072) 0:01:11.614 ******** 2019-10-13T00:40:58.380 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:58.391 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-13T00:40:58.393 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2019-10-13T00:40:58.427 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.071) 0:01:11.686 ******** 2019-10-13T00:40:58.432 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-13T00:40:58.433 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:03.546039", "end": "2019-10-13 00:40:58.382757", "rc": 0, "start": "2019-10-13 00:40:54.836718", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2019-10-13T00:40:58.472 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.045) 0:01:11.731 ******** 2019-10-13T00:40:58.496 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|skipped` use `result is skipped`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:58.509 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-13T00:40:58.509 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2019-10-13T00:40:58.545 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.072) 0:01:11.804 ******** 2019-10-13T00:40:58.570 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:58.581 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-10-13T00:40:58.583 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:40:58.618 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.073) 0:01:11.877 ******** 2019-10-13T00:40:58.659 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-13T00:40:58.661 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2019-10-13T00:40:58.703 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.085) 0:01:11.962 ******** 2019-10-13T00:40:58.734 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:40:58.751 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-13T00:40:58.751 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2019-10-13T00:40:58.794 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:40:58 +0000 (0:00:00.090) 0:01:12.053 ******** 2019-10-13T00:41:01.747 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-10-13T00:41:01.748 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-13T00:41:01.784 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:01 +0000 (0:00:02.989) 0:01:15.043 ******** 2019-10-13T00:41:02.042 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-13T00:41:02.090 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:02 +0000 (0:00:00.306) 0:01:15.349 ******** 2019-10-13T00:41:05.757 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-13T00:41:05.758 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=rhel-7-server-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-7-server-rpms"], "delta": "0:00:03.771600", "end": "2019-10-13 00:41:05.728557", "item": "rhel-7-server-rpms", "rc": 0, "start": "2019-10-13 00:41:01.956957", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system."]} 2019-10-13T00:41:06.059 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=rhel-7-server-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-7-server-rpms"], "delta": "0:00:03.771482", "end": "2019-10-13 00:41:06.028416", "item": "rhel-7-server-rpms", "rc": 0, "start": "2019-10-13 00:41:02.256934", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system."]} 2019-10-13T00:41:10.040 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=rhel-7-server-optional-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-7-server-optional-rpms"], "delta": "0:00:04.120561", "end": "2019-10-13 00:41:10.019611", "item": "rhel-7-server-optional-rpms", "rc": 0, "start": "2019-10-13 00:41:05.899050", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-optional-rpms' is enabled for this system."]} 2019-10-13T00:41:10.466 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=rhel-7-server-optional-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-7-server-optional-rpms"], "delta": "0:00:04.243670", "end": "2019-10-13 00:41:10.439363", "item": "rhel-7-server-optional-rpms", "rc": 0, "start": "2019-10-13 00:41:06.195693", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-optional-rpms' is enabled for this system."]} 2019-10-13T00:41:14.709 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=rhel-7-server-extras-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-7-server-extras-rpms"], "delta": "0:00:04.491391", "end": "2019-10-13 00:41:14.682032", "item": "rhel-7-server-extras-rpms", "rc": 0, "start": "2019-10-13 00:41:10.190641", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2019-10-13T00:41:15.303 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=rhel-7-server-extras-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-7-server-extras-rpms"], "delta": "0:00:04.665535", "end": "2019-10-13 00:41:15.273805", "item": "rhel-7-server-extras-rpms", "rc": 0, "start": "2019-10-13 00:41:10.608270", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2019-10-13T00:41:19.931 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=rhel-ha-for-rhel-7-server-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:05.058940", "end": "2019-10-13 00:41:19.910291", "item": "rhel-ha-for-rhel-7-server-rpms", "rc": 0, "start": "2019-10-13 00:41:14.851351", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system."]} 2019-10-13T00:41:19.972 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:19 +0000 (0:00:17.882) 0:01:33.231 ******** 2019-10-13T00:41:20.000 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:41:20.159 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-13T00:41:20.161 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2019-10-13T00:41:20.213 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:20 +0000 (0:00:00.240) 0:01:33.472 ******** 2019-10-13T00:41:20.399 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-13T00:41:20.400 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:41:20.437 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:20 +0000 (0:00:00.223) 0:01:33.696 ******** 2019-10-13T00:41:20.537 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-13T00:41:20.539 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=rhel-ha-for-rhel-7-server-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:05.058350", "end": "2019-10-13 00:41:20.510195", "item": "rhel-ha-for-rhel-7-server-rpms", "rc": 0, "start": "2019-10-13 00:41:15.451845", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system."]} 2019-10-13T00:41:20.578 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:20 +0000 (0:00:00.141) 0:01:33.837 ******** 2019-10-13T00:41:20.602 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:41:20.768 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-13T00:41:20.769 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2019-10-13T00:41:20.807 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:20 +0000 (0:00:00.228) 0:01:34.066 ******** 2019-10-13T00:41:20.968 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-13T00:41:20.969 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-10-13T00:41:20.981 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-13T00:41:20.982 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:41:21.017 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:21 +0000 (0:00:00.210) 0:01:34.276 ******** 2019-10-13T00:41:21.450 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-13T00:41:21.451 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-10-13T00:41:21.495 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:21 +0000 (0:00:00.477) 0:01:34.754 ******** 2019-10-13T00:41:21.533 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-10-13T00:41:21.997 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-13T00:41:21.998 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4db37afba2545ac8871903964bcd8b701978565a", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 294, "state": "file", "uid": 0} 2019-10-13T00:41:22.019 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-13T00:41:22.021 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-10-13T00:41:22.065 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.570) 0:01:35.324 ******** 2019-10-13T00:41:22.496 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-13T00:41:22.497 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "f1e1ffd80fcbd46e5168d3829bdcb34dffb3743b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 334, "state": "file", "uid": 0} 2019-10-13T00:41:22.542 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.476) 0:01:35.801 ******** 2019-10-13T00:41:22.575 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:41:22.585 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4db37afba2545ac8871903964bcd8b701978565a", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 294, "state": "file", "uid": 0} 2019-10-13T00:41:22.586 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:22.628 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.086) 0:01:35.887 ******** 2019-10-13T00:41:22.654 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T00:41:22.656 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:22.696 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.067) 0:01:35.954 ******** 2019-10-13T00:41:22.724 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:22.783 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.087) 0:01:36.042 ******** 2019-10-13T00:41:22.815 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:22.859 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.075) 0:01:36.118 ******** 2019-10-13T00:41:22.886 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T00:41:22.888 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:22.924 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.064) 0:01:36.183 ******** 2019-10-13T00:41:22.950 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-13T00:41:22.952 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:22.990 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:22 +0000 (0:00:00.066) 0:01:36.249 ******** 2019-10-13T00:41:23.017 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-13T00:41:23.018 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.055 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.064) 0:01:36.314 ******** 2019-10-13T00:41:23.082 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-13T00:41:23.083 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "f1e1ffd80fcbd46e5168d3829bdcb34dffb3743b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 334, "state": "file", "uid": 0} 2019-10-13T00:41:23.085 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-13T00:41:23.087 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.125 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.069) 0:01:36.383 ******** 2019-10-13T00:41:23.171 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.046) 0:01:36.430 ******** 2019-10-13T00:41:23.176 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-13T00:41:23.177 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.194 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:41:23.230 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.055) 0:01:36.486 ******** 2019-10-13T00:41:23.233 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T00:41:23.234 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.274 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.045) 0:01:36.531 ******** 2019-10-13T00:41:23.300 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T00:41:23.301 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.339 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.066) 0:01:36.598 ******** 2019-10-13T00:41:23.365 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-10-13T00:41:23.367 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.403 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.064) 0:01:36.662 ******** 2019-10-13T00:41:23.430 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-10-13T00:41:23.431 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.491 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.087) 0:01:36.750 ******** 2019-10-13T00:41:23.531 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T00:41:23.533 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.591 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.100) 0:01:36.850 ******** 2019-10-13T00:41:23.651 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.686 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.095) 0:01:36.945 ******** 2019-10-13T00:41:23.712 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-13T00:41:23.714 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.750 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.063) 0:01:37.009 ******** 2019-10-13T00:41:23.776 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-13T00:41:23.778 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.811 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.061) 0:01:37.070 ******** 2019-10-13T00:41:23.847 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-13T00:41:23.849 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:23.884 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:23 +0000 (0:00:00.072) 0:01:37.143 ******** 2019-10-13T00:41:24.526 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-10-13T00:41:24.527 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.125057", "end": "2019-10-13 00:41:24.500292", "rc": 0, "start": "2019-10-13 00:41:23.375235", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:41:24.578 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:24 +0000 (0:00:00.693) 0:01:37.836 ******** 2019-10-13T00:41:25.521 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.460076", "end": "2019-10-13 00:41:25.500297", "rc": 0, "start": "2019-10-13 00:41:24.040221", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:41:25.559 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:25 +0000 (0:00:00.981) 0:01:38.818 ******** 2019-10-13T00:41:26.344 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-10-13T00:41:26.344 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-10-13T00:41:26.382 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:26 +0000 (0:00:00.823) 0:01:39.641 ******** 2019-10-13T00:41:26.409 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-13T00:41:26.411 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:26.446 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:26 +0000 (0:00:00.063) 0:01:39.704 ******** 2019-10-13T00:41:26.474 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-10-13T00:41:26.476 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:26.517 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:26 +0000 (0:00:00.071) 0:01:39.776 ******** 2019-10-13T00:41:26.546 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-10-13T00:41:26.548 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:26.584 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:26 +0000 (0:00:00.066) 0:01:39.843 ******** 2019-10-13T00:41:26.610 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-10-13T00:41:26.612 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:26.652 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:26 +0000 (0:00:00.067) 0:01:39.911 ******** 2019-10-13T00:41:27.142 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-10-13T00:41:27.144 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-10-13T00:41:27.178 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:27 +0000 (0:00:00.525) 0:01:40.437 ******** 2019-10-13T00:41:27.228 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-10-13T00:41:27.269 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:27 +0000 (0:00:00.090) 0:01:40.528 ******** 2019-10-13T00:41:27.296 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-13T00:41:27.297 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:27.337 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:27 +0000 (0:00:00.068) 0:01:40.596 ******** 2019-10-13T00:41:27.358 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-10-13T00:41:27.360 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:27.401 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:27 +0000 (0:00:00.064) 0:01:40.660 ******** 2019-10-13T00:41:27.429 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-10-13T00:41:27.429 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:27.485 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:27 +0000 (0:00:00.083) 0:01:40.744 ******** 2019-10-13T00:41:27.515 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-10-13T00:41:27.517 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:41:27.555 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:27 +0000 (0:00:00.069) 0:01:40.814 ******** 2019-10-13T00:41:28.053 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-10-13T00:41:28.089 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:28 +0000 (0:00:00.534) 0:01:41.348 ******** 2019-10-13T00:41:57.724 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-10-13T00:41:57.725 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-10-13T00:41:57.761 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:57 +0000 (0:00:29.671) 0:02:11.020 ******** 2019-10-13T00:41:58.275 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T00:41:58.275 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-10-13T00:41:58.740 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-10-13T00:41:58.844 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-10-13T00:41:58.885 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:58 +0000 (0:00:01.123) 0:02:12.144 ******** 2019-10-13T00:41:59.191 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T00:41:59.191 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-10-13T00:41:59.231 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:59 +0000 (0:00:00.346) 0:02:12.490 ******** 2019-10-13T00:41:59.406 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-13T00:41:59.407 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T00:41:59.447 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:41:59 +0000 (0:00:00.215) 0:02:12.706 ******** 2019-10-13T00:41:59.453 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T00:41:59.455 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-10-13T00:41:59.885 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-10-13T00:41:59.886 INFO:teuthology.task.ansible.out: 2019-10-13T00:41:59.928 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T00:41:59.929 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-10-13T00:42:00.344 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T00:42:00.346 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-10-13T00:42:00.388 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:00 +0000 (0:00:00.941) 0:02:13.647 ******** 2019-10-13T00:42:00.396 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T00:42:00.397 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-10-13T00:42:00.556 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T00:42:00.602 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:00 +0000 (0:00:00.213) 0:02:13.860 ******** 2019-10-13T00:42:00.861 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi142.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-10-13T00:42:00.899 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:00 +0000 (0:00:00.296) 0:02:14.158 ******** 2019-10-13T00:42:00.935 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-13T00:42:00.937 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:42:00.978 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:00 +0000 (0:00:00.078) 0:02:14.236 ******** 2019-10-13T00:42:01.014 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-13T00:42:01.016 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-10-13T00:42:01.057 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:01 +0000 (0:00:00.080) 0:02:14.316 ******** 2019-10-13T00:42:01.094 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T00:42:01.095 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-10-13T00:42:01.231 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-13T00:42:01.233 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-13T00:42:01.269 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:01 +0000 (0:00:00.210) 0:02:14.527 ******** 2019-10-13T00:42:01.305 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-13T00:42:01.307 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:01.316 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:01.338 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:01.349 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:01.388 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:01 +0000 (0:00:00.119) 0:02:14.646 ******** 2019-10-13T00:42:01.542 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T00:42:01.543 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-10-13T00:42:01.990 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-10-13T00:42:02.032 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:02 +0000 (0:00:00.644) 0:02:15.291 ******** 2019-10-13T00:42:02.058 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-13T00:42:02.060 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:42:02.097 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:02 +0000 (0:00:00.064) 0:02:15.355 ******** 2019-10-13T00:42:02.134 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-13T00:42:02.135 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-10-13T00:42:02.172 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:02 +0000 (0:00:00.075) 0:02:15.431 ******** 2019-10-13T00:42:02.347 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-13T00:42:02.349 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-13T00:42:02.386 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:02 +0000 (0:00:00.214) 0:02:15.645 ******** 2019-10-13T00:42:02.433 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-13T00:42:02.435 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:02.444 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:02.455 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:02.466 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:02.506 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:02 +0000 (0:00:00.119) 0:02:15.765 ******** 2019-10-13T00:42:08.626 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-13T00:42:08.626 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-common", "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", ""]} 2019-10-13T00:42:10.867 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-common", "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", ""]} 2019-10-13T00:42:16.860 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nrpe", "msg": "", "rc": 0, "results": ["All packages providing nrpe are up to date", ""]} 2019-10-13T00:42:18.151 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nrpe", "msg": "", "rc": 0, "results": ["All packages providing nrpe are up to date", ""]} 2019-10-13T00:42:24.251 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-plugins", "msg": "", "rc": 0, "results": ["All packages providing nagios-plugins are up to date", ""]} 2019-10-13T00:42:25.451 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-plugins", "msg": "", "rc": 0, "results": ["All packages providing nagios-plugins are up to date", ""]} 2019-10-13T00:42:31.730 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-plugins-load", "msg": "", "rc": 0, "results": ["All packages providing nagios-plugins-load are up to date", ""]} 2019-10-13T00:42:31.772 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:31 +0000 (0:00:29.265) 0:02:45.030 ******** 2019-10-13T00:42:31.809 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-13T00:42:31.811 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-common", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "item": "nrpe", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:31.821 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:31.832 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-load", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:31.870 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:31 +0000 (0:00:00.097) 0:02:45.128 ******** 2019-10-13T00:42:32.362 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-10-13T00:42:32.363 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-10-13T00:42:32.400 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:32 +0000 (0:00:00.530) 0:02:45.659 ******** 2019-10-13T00:42:32.885 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-13T00:42:32.886 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-10-13T00:42:32.927 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:32 +0000 (0:00:00.527) 0:02:46.186 ******** 2019-10-13T00:42:32.955 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-13T00:42:32.956 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:42:32.995 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:32 +0000 (0:00:00.068) 0:02:46.254 ******** 2019-10-13T00:42:33.147 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-13T00:42:33.148 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-plugins-load", "msg": "", "rc": 0, "results": ["All packages providing nagios-plugins-load are up to date", ""]} 2019-10-13T00:42:33.212 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:33 +0000 (0:00:00.217) 0:02:46.471 ******** 2019-10-13T00:42:33.221 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-10-13T00:42:33.223 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:42:33.291 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:33 +0000 (0:00:00.078) 0:02:46.550 ******** 2019-10-13T00:42:33.295 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-13T00:42:33.296 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-common", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:33.297 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "item": "nrpe", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:33.299 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-load", "skip_reason": "Conditional result was False"} 2019-10-13T00:42:33.335 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:33 +0000 (0:00:00.044) 0:02:46.594 ******** 2019-10-13T00:42:33.509 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi142.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-10-13T00:42:33.545 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:33 +0000 (0:00:00.209) 0:02:46.803 ******** 2019-10-13T00:42:33.570 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-13T00:42:33.572 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:42:33.607 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:33 +0000 (0:00:00.062) 0:02:46.866 ******** 2019-10-13T00:42:33.840 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-10-13T00:42:33.876 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:33 +0000 (0:00:00.268) 0:02:47.135 ******** 2019-10-13T00:42:34.113 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-10-13T00:42:34.114 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-10-13T00:42:34.154 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:34 +0000 (0:00:00.277) 0:02:47.412 ******** 2019-10-13T00:42:34.376 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-13T00:42:34.377 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-10-13T00:42:34.417 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:34 +0000 (0:00:00.262) 0:02:47.675 ******** 2019-10-13T00:42:34.442 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-13T00:42:34.444 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:42:34.480 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:34 +0000 (0:00:00.063) 0:02:47.739 ******** 2019-10-13T00:42:34.662 INFO:teuthology.task.ansible.out: 2019-10-13T00:42:34.663 INFO:teuthology.task.ansible.out:TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:42:34.700 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:34 +0000 (0:00:00.220) 0:02:47.959 ******** 2019-10-13T00:42:34.811 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-10-13T00:42:34.813 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 00:38:14 UTC", "ActiveEnterTimestampMonotonic": "19890615", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target network.target system.slice var-run.mount systemd-journald.socket basic.target nss-lookup.target local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 00:38:14 UTC", "AssertTimestampMonotonic": "19877229", "Before": "getty@tty1.service xdm.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 00:38:14 UTC", "ConditionTimestampMonotonic": "19877229", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2082", "ExecMainStartTimestamp": "Sun 2019-10-13 00:38:14 UTC", "ExecMainStartTimestampMonotonic": "19889985", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2019-10-13T00:42:34.815 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Sun 2019-10-13 00:38:14 UTC] ; stop_time=[n/a] ; pid=2082 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 00:38:14 UTC", "InactiveExitTimestampMonotonic": "19890615", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "2082", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target network-online.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2019-10-13T00:42:34.817 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Sun 2019-10-13 00:38:14 UTC", "WatchdogTimestampMonotonic": "19890591", "WatchdogUSec": "0"}} 2019-10-13T00:42:34.851 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:34 +0000 (0:00:00.150) 0:02:48.110 ******** 2019-10-13T00:42:34.911 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-13T00:42:34.913 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-10-13T00:42:34.948 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:34 +0000 (0:00:00.095) 0:02:48.206 ******** 2019-10-13T00:42:34.974 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-13T00:42:34.976 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:42:35.012 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:35 +0000 (0:00:00.065) 0:02:48.271 ******** 2019-10-13T00:42:35.030 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-10-13T00:42:35.036 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001915", "end": "2019-10-13 00:42:35.009820", "rc": 0, "start": "2019-10-13 00:42:35.007905", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-10-13T00:42:35.073 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:35 +0000 (0:00:00.060) 0:02:48.331 ******** 2019-10-13T00:42:35.516 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-10-13T00:42:35.517 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-10-13T00:42:35.561 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:35 +0000 (0:00:00.488) 0:02:48.820 ******** 2019-10-13T00:42:35.932 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-10-13T00:42:35.934 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 00:38:55 UTC", "ActiveEnterTimestampMonotonic": "20449604", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target system.slice systemd-journald.socket nss-lookup.target basic.target local-fs.target network.target var-run.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 00:38:55 UTC", "AssertTimestampMonotonic": "20432291", "Before": "shutdown.target multi-user.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 00:38:55 UTC", "ConditionTimestampMonotonic": "20432290", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2082", "ExecMainStartTimestamp": "Sun 2019-10-13 00:38:55 UTC", "ExecMainStartTimestampMonotonic": "20449506", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2019-10-13T00:42:35.936 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Sun 2019-10-13 00:38:55 UTC] ; stop_time=[n/a] ; pid=2082 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 00:38:55 UTC", "InactiveExitTimestampMonotonic": "20449604", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "2082", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2019-10-13T00:42:35.937 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Sun 2019-10-13 00:38:55 UTC", "WatchdogTimestampMonotonic": "20449574", "WatchdogUSec": "0"}} 2019-10-13T00:42:35.975 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:35 +0000 (0:00:00.414) 0:02:49.234 ******** 2019-10-13T00:42:36.260 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-10-13T00:42:36.262 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001940", "end": "2019-10-13 00:42:36.244521", "rc": 0, "start": "2019-10-13 00:42:36.242581", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-10-13T00:42:36.301 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:36 +0000 (0:00:00.325) 0:02:49.560 ******** 2019-10-13T00:42:36.997 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-10-13T00:42:36.999 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed"]} 2019-10-13T00:42:38.381 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed"]} 2019-10-13T00:42:38.817 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "msg": "", "rc": 0, "results": ["policycoreutils-python-2.5-33.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-10-13T00:42:38.853 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:38 +0000 (0:00:02.551) 0:02:52.112 ******** 2019-10-13T00:42:40.001 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "msg": "", "rc": 0, "results": ["policycoreutils-python-2.5-33.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-10-13T00:42:40.040 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:40 +0000 (0:00:01.187) 0:02:53.299 ******** 2019-10-13T00:42:46.080 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2019-10-13T00:42:46.110 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:46 +0000 (0:00:06.069) 0:02:59.369 ******** 2019-10-13T00:42:47.185 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-10-13T00:42:47.222 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:47 +0000 (0:00:01.112) 0:03:00.481 ******** 2019-10-13T00:42:47.240 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-10-13T00:42:47.242 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2019-10-13T00:42:47.282 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:47 +0000 (0:00:00.059) 0:03:00.541 ******** 2019-10-13T00:42:48.286 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-10-13T00:42:48.288 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-10-13T00:42:48.323 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:48 +0000 (0:00:01.040) 0:03:01.581 ******** 2019-10-13T00:42:55.151 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-10-13T00:42:55.153 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:07.752031", "end": "2019-10-13 00:42:55.128016", "failed_when_result": false, "rc": 0, "start": "2019-10-13 00:42:47.375985", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-10-13T00:42:55.192 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:55 +0000 (0:00:06.869) 0:03:08.451 ******** 2019-10-13T00:42:55.991 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-13T00:42:55.991 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "md5sum": "286c682fc73965c17876891a4581da3e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 934, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1570927375.22-115020482035964/source", "state": "file", "uid": 0} 2019-10-13T00:42:56.032 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:56 +0000 (0:00:00.840) 0:03:09.291 ******** 2019-10-13T00:42:56.170 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-10-13T00:42:56.171 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:07.669134", "end": "2019-10-13 00:42:56.147501", "failed_when_result": false, "rc": 0, "start": "2019-10-13 00:42:48.478367", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-10-13T00:42:56.214 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:56 +0000 (0:00:00.181) 0:03:09.473 ******** 2019-10-13T00:42:56.230 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-10-13T00:42:56.231 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.021799", "end": "2019-10-13 00:42:56.211632", "rc": 0, "start": "2019-10-13 00:42:56.189833", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-10-13T00:42:56.272 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:56 +0000 (0:00:00.057) 0:03:09.530 ******** 2019-10-13T00:42:56.453 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-13T00:42:56.455 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.008527", "end": "2019-10-13 00:42:56.432560", "rc": 0, "start": "2019-10-13 00:42:56.424033", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:42:56.497 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:56 +0000 (0:00:00.225) 0:03:09.756 ******** 2019-10-13T00:42:56.856 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-13T00:42:56.858 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "md5sum": "286c682fc73965c17876891a4581da3e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 934, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1570927376.24-203612692308733/source", "state": "file", "uid": 0} 2019-10-13T00:42:56.892 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:56 +0000 (0:00:00.394) 0:03:10.151 ******** 2019-10-13T00:42:57.074 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-10-13T00:42:57.074 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.012085", "end": "2019-10-13 00:42:57.055644", "rc": 0, "start": "2019-10-13 00:42:57.043559", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-10-13T00:42:57.108 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:57 +0000 (0:00:00.216) 0:03:10.367 ******** 2019-10-13T00:42:57.290 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-13T00:42:57.291 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009689", "end": "2019-10-13 00:42:57.270460", "rc": 0, "start": "2019-10-13 00:42:57.260771", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:42:57.328 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:42:57 +0000 (0:00:00.219) 0:03:10.586 ******** 2019-10-13T00:43:04.449 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-13T00:43:04.450 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:07.778829", "end": "2019-10-13 00:43:04.430021", "rc": 0, "start": "2019-10-13 00:42:56.651192", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:43:04.490 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:04 +0000 (0:00:07.162) 0:03:17.749 ******** 2019-10-13T00:43:04.708 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-13T00:43:04.710 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-10-13T00:43:04.747 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:04 +0000 (0:00:00.256) 0:03:18.006 ******** 2019-10-13T00:43:04.785 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-10-13T00:43:04.798 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-10-13T00:43:04.827 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:04 +0000 (0:00:00.080) 0:03:18.086 ******** 2019-10-13T00:43:04.864 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-13T00:43:04.865 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T00:43:04.908 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:04 +0000 (0:00:00.080) 0:03:18.167 ******** 2019-10-13T00:43:04.945 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-13T00:43:04.946 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2019-10-13T00:43:04.991 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:04 +0000 (0:00:00.080) 0:03:18.247 ******** 2019-10-13T00:43:04.994 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-13T00:43:04.995 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:07.481351", "end": "2019-10-13 00:43:04.958628", "rc": 0, "start": "2019-10-13 00:42:57.477277", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:43:05.032 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.044) 0:03:18.291 ******** 2019-10-13T00:43:05.037 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-10-13T00:43:05.038 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T00:43:05.085 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.052) 0:03:18.344 ******** 2019-10-13T00:43:05.229 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-13T00:43:05.231 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-10-13T00:43:05.271 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.185) 0:03:18.530 ******** 2019-10-13T00:43:05.277 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-13T00:43:05.278 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-10-13T00:43:05.328 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.055) 0:03:18.586 ******** 2019-10-13T00:43:05.333 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-10-13T00:43:05.334 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-10-13T00:43:05.372 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.045) 0:03:18.631 ******** 2019-10-13T00:43:05.425 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-13T00:43:05.427 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T00:43:05.463 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.090) 0:03:18.721 ******** 2019-10-13T00:43:05.512 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-13T00:43:05.513 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-10-13T00:43:05.514 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-13T00:43:05.516 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2019-10-13T00:43:05.553 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.090) 0:03:18.812 ******** 2019-10-13T00:43:05.588 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.033) 0:03:18.845 ******** 2019-10-13T00:43:05.591 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-13T00:43:05.592 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:05.636 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.049) 0:03:18.895 ******** 2019-10-13T00:43:05.642 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-10-13T00:43:05.643 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T00:43:05.654 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:43:05.686 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.049) 0:03:18.945 ******** 2019-10-13T00:43:05.691 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-10-13T00:43:05.692 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:05.731 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.044) 0:03:18.989 ******** 2019-10-13T00:43:05.861 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-13T00:43:05.863 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-10-13T00:43:05.908 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.177) 0:03:19.167 ******** 2019-10-13T00:43:05.925 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-10-13T00:43:05.926 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-10-13T00:43:05.964 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:05 +0000 (0:00:00.055) 0:03:19.222 ******** 2019-10-13T00:43:06.094 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-13T00:43:06.096 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-10-13T00:43:06.136 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:06 +0000 (0:00:00.171) 0:03:19.394 ******** 2019-10-13T00:43:06.170 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-13T00:43:06.172 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:06.212 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:06 +0000 (0:00:00.076) 0:03:19.471 ******** 2019-10-13T00:43:06.231 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:43:06.238 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-10-13T00:43:06.240 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:06.274 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:06 +0000 (0:00:00.062) 0:03:19.533 ******** 2019-10-13T00:43:06.470 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-10-13T00:43:06.472 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-10-13T00:43:06.512 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:06 +0000 (0:00:00.236) 0:03:19.770 ******** 2019-10-13T00:43:06.558 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-10-13T00:43:06.559 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-10-13T00:43:06.601 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:06 +0000 (0:00:00.089) 0:03:19.860 ******** 2019-10-13T00:43:06.647 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-10-13T00:43:06.649 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi142"}, "changed": false} 2019-10-13T00:43:06.686 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:06 +0000 (0:00:00.084) 0:03:19.945 ******** 2019-10-13T00:43:07.094 INFO:teuthology.task.ansible.out: 2019-10-13T00:43:07.096 INFO:teuthology.task.ansible.out:TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-10-13T00:43:07.135 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:07 +0000 (0:00:00.449) 0:03:20.394 ******** 2019-10-13T00:43:07.137 INFO:teuthology.task.ansible.out: 2019-10-13T00:43:07.182 INFO:teuthology.task.ansible.out: 2019-10-13T00:43:07.183 INFO:teuthology.task.ansible.out:TASK [testnode : Set hostname var] ********************************************* ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi155"}, "changed": false} 2019-10-13T00:43:07.224 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:07 +0000 (0:00:00.088) 0:03:20.483 ******** 2019-10-13T00:43:07.226 INFO:teuthology.task.ansible.out: 2019-10-13T00:43:07.451 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-10-13T00:43:07.453 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi142", "ansible_hostname": "smithi142", "ansible_nodename": "smithi142"}, "changed": false, "name": "smithi142"} 2019-10-13T00:43:07.489 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:07 +0000 (0:00:00.264) 0:03:20.748 ******** 2019-10-13T00:43:07.520 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi155", "ansible_hostname": "smithi155", "ansible_nodename": "smithi155"}, "changed": false, "name": "smithi155"} 2019-10-13T00:43:07.557 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:07 +0000 (0:00:00.068) 0:03:20.816 ******** 2019-10-13T00:43:07.991 INFO:teuthology.task.ansible.out: 2019-10-13T00:43:07.993 INFO:teuthology.task.ansible.out:TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2019-10-13T00:43:08.043 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:08 +0000 (0:00:00.485) 0:03:21.302 ******** 2019-10-13T00:43:08.056 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2019-10-13T00:43:08.097 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:08 +0000 (0:00:00.053) 0:03:21.355 ******** 2019-10-13T00:43:08.538 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-10-13T00:43:08.539 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-10-13T00:43:08.580 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:08 +0000 (0:00:00.483) 0:03:21.839 ******** 2019-10-13T00:43:08.608 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-10-13T00:43:08.666 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:08 +0000 (0:00:00.085) 0:03:21.925 ******** 2019-10-13T00:43:09.117 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:43:09.122 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-13T00:43:09.124 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-13T00:43:09.160 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:09 +0000 (0:00:00.493) 0:03:22.418 ******** 2019-10-13T00:43:09.226 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:43:09.242 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-13T00:43:09.280 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:09 +0000 (0:00:00.120) 0:03:22.539 ******** 2019-10-13T00:43:09.663 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-10-13T00:43:09.664 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-10-13T00:43:09.711 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:09 +0000 (0:00:00.430) 0:03:22.970 ******** 2019-10-13T00:43:09.793 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-10-13T00:43:09.830 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:09 +0000 (0:00:00.119) 0:03:23.089 ******** 2019-10-13T00:43:09.891 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-13T00:43:09.930 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:09 +0000 (0:00:00.100) 0:03:23.189 ******** 2019-10-13T00:43:10.012 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-13T00:43:10.049 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:10 +0000 (0:00:00.118) 0:03:23.308 ******** 2019-10-13T00:43:10.109 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T00:43:10.111 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2019-10-13T00:43:10.191 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:10 +0000 (0:00:00.141) 0:03:23.450 ******** 2019-10-13T00:43:10.229 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2019-10-13T00:43:10.274 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:10 +0000 (0:00:00.082) 0:03:23.533 ******** 2019-10-13T00:43:10.385 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T00:43:10.387 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-13T00:43:10.449 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:10 +0000 (0:00:00.175) 0:03:23.708 ******** 2019-10-13T00:43:10.479 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-13T00:43:10.525 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:10 +0000 (0:00:00.075) 0:03:23.784 ******** 2019-10-13T00:43:10.973 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T00:43:10.976 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-10-13T00:43:11.024 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:11 +0000 (0:00:00.498) 0:03:24.283 ******** 2019-10-13T00:43:11.079 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-10-13T00:43:11.123 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:11 +0000 (0:00:00.099) 0:03:24.382 ******** 2019-10-13T00:43:11.538 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T00:43:11.539 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-10-13T00:43:11.577 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:11 +0000 (0:00:00.454) 0:03:24.836 ******** 2019-10-13T00:43:11.640 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-10-13T00:43:11.679 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:11 +0000 (0:00:00.100) 0:03:24.937 ******** 2019-10-13T00:43:13.390 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T00:43:13.390 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-10-13T00:43:13.392 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.624571", "end": "2019-10-13 00:43:13.366601", "rc": 0, "start": "2019-10-13 00:43:11.742030", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:43:13.429 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:13 +0000 (0:00:01.750) 0:03:26.688 ******** 2019-10-13T00:43:13.559 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.701000", "end": "2019-10-13 00:43:13.538625", "rc": 0, "start": "2019-10-13 00:43:11.837625", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:43:13.604 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:13 +0000 (0:00:00.174) 0:03:26.863 ******** 2019-10-13T00:43:13.958 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-13T00:43:13.959 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'RHEL 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/'}, 'key': u'rhel-7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2019-10-13T00:43:14.130 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'RHEL 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/'}, 'key': u'rhel-7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2019-10-13T00:43:14.448 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2019-10-13T00:43:14.496 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.892) 0:03:27.755 ******** 2019-10-13T00:43:14.525 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-13T00:43:14.569 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.072) 0:03:27.828 ******** 2019-10-13T00:43:14.595 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-13T00:43:14.597 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.635 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.065) 0:03:27.893 ******** 2019-10-13T00:43:14.656 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2019-10-13T00:43:14.694 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.059) 0:03:27.953 ******** 2019-10-13T00:43:14.699 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-10-13T00:43:14.700 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.701 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.744 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.049) 0:03:28.003 ******** 2019-10-13T00:43:14.750 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-13T00:43:14.776 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:43:14.792 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.047) 0:03:28.051 ******** 2019-10-13T00:43:14.797 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.833 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.041) 0:03:28.092 ******** 2019-10-13T00:43:14.838 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-13T00:43:14.840 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.876 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.042) 0:03:28.135 ******** 2019-10-13T00:43:14.881 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:43:14.882 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.924 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.043) 0:03:28.178 ******** TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.925 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-13T00:43:14.963 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:14 +0000 (0:00:00.043) 0:03:28.222 ******** 2019-10-13T00:43:14.988 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:43:14.999 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-10-13T00:43:15.000 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:15.035 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:15 +0000 (0:00:00.071) 0:03:28.293 ******** 2019-10-13T00:43:15.061 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:43:15.062 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:15.098 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:15 +0000 (0:00:00.062) 0:03:28.356 ******** 2019-10-13T00:43:16.530 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-10-13T00:43:16.532 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-52.el7.noarch providing yum-utils is already installed"]} 2019-10-13T00:43:16.576 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:16 +0000 (0:00:01.479) 0:03:29.835 ******** 2019-10-13T00:43:16.837 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-52.el7.noarch providing yum-utils is already installed"]} 2019-10-13T00:43:16.873 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:16 +0000 (0:00:00.296) 0:03:30.132 ******** 2019-10-13T00:43:17.645 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-13T00:43:17.646 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.878692", "end": "2019-10-13 00:43:17.618590", "rc": 0, "start": "2019-10-13 00:43:16.739898", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: ftp.linux.ncsu.edu", "No unfinished transactions left."]} 2019-10-13T00:43:17.688 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:17 +0000 (0:00:00.814) 0:03:30.947 ******** 2019-10-13T00:43:17.896 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-13T00:43:17.897 INFO:teuthology.task.ansible.out:fatal: [smithi142.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.023287", "end": "2019-10-13 00:43:17.872515", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 00:43:17.849228", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-10-13T00:43:17.934 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:17 +0000 (0:00:00.245) 0:03:31.193 ******** 2019-10-13T00:43:17.938 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-13T00:43:17.939 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.874208", "end": "2019-10-13 00:43:17.912290", "rc": 0, "start": "2019-10-13 00:43:17.038082", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: ftp.linux.ncsu.edu", "No unfinished transactions left."]} 2019-10-13T00:43:17.977 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:17 +0000 (0:00:00.043) 0:03:31.236 ******** 2019-10-13T00:43:17.983 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-13T00:43:17.984 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:18.022 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:18 +0000 (0:00:00.044) 0:03:31.281 ******** 2019-10-13T00:43:18.184 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-13T00:43:18.185 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.023696", "end": "2019-10-13 00:43:18.164526", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 00:43:18.140830", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-10-13T00:43:18.223 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:18 +0000 (0:00:00.200) 0:03:31.482 ******** 2019-10-13T00:43:18.263 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-13T00:43:18.265 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:43:18.303 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:18 +0000 (0:00:00.080) 0:03:31.562 ******** 2019-10-13T00:43:19.318 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-13T00:43:19.320 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "msg": "", "rc": 0, "results": ["ceph is not installed"]} 2019-10-13T00:43:19.533 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "msg": "", "rc": 0, "results": ["ceph is not installed"]} 2019-10-13T00:43:20.534 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ceph-base) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-base", "msg": "", "rc": 0, "results": ["ceph-base is not installed"]} 2019-10-13T00:43:20.776 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ceph-base) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-base", "msg": "", "rc": 0, "results": ["ceph-base is not installed"]} 2019-10-13T00:43:21.825 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ceph-selinux) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-selinux", "msg": "", "rc": 0, "results": ["ceph-selinux is not installed"]} 2019-10-13T00:43:22.097 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ceph-selinux) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-selinux", "msg": "", "rc": 0, "results": ["ceph-selinux is not installed"]} 2019-10-13T00:43:23.058 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "msg": "", "rc": 0, "results": ["ceph-common is not installed"]} 2019-10-13T00:43:23.302 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "msg": "", "rc": 0, "results": ["ceph-common is not installed"]} 2019-10-13T00:43:24.317 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ceph-debuginfo) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-debuginfo", "msg": "", "rc": 0, "results": ["ceph-debuginfo is not installed"]} 2019-10-13T00:43:24.526 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ceph-debuginfo) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-debuginfo", "msg": "", "rc": 0, "results": ["ceph-debuginfo is not installed"]} 2019-10-13T00:43:25.571 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ceph-release) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-release", "msg": "", "rc": 0, "results": ["ceph-release is not installed"]} 2019-10-13T00:43:25.866 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ceph-release) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-release", "msg": "", "rc": 0, "results": ["ceph-release is not installed"]} 2019-10-13T00:43:26.746 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "msg": "", "rc": 0, "results": ["libcephfs1 is not installed"]} 2019-10-13T00:43:27.095 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "msg": "", "rc": 0, "results": ["libcephfs1 is not installed"]} 2019-10-13T00:43:28.011 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ceph-radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-radosgw", "msg": "", "rc": 0, "results": ["ceph-radosgw is not installed"]} 2019-10-13T00:43:28.328 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ceph-radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-radosgw", "msg": "", "rc": 0, "results": ["ceph-radosgw is not installed"]} 2019-10-13T00:43:29.584 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-13T00:43:29.830 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-13T00:43:30.866 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "msg": "", "rc": 0, "results": ["python-rados is not installed"]} 2019-10-13T00:43:31.077 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "msg": "", "rc": 0, "results": ["python-rados is not installed"]} 2019-10-13T00:43:32.233 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "msg": "", "rc": 0, "results": ["python-rbd is not installed"]} 2019-10-13T00:43:32.381 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "msg": "", "rc": 0, "results": ["python-rbd is not installed"]} 2019-10-13T00:43:33.542 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "msg": "", "rc": 0, "results": ["python-cephfs is not installed"]} 2019-10-13T00:43:33.709 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "msg": "", "rc": 0, "results": ["python-cephfs is not installed"]} 2019-10-13T00:43:42.851 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librbd1"]}, "item": "librbd1", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package (+3 Dependent packages)\n\nInstalled size: 16 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 2019-10-13T00:43:42.853 INFO:teuthology.task.ansible.out: 1/4 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/4 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/4 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/4 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/4 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/4 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 3/4 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 4/4 \n\nRemoved:\n librbd1.x86_64 1:10.2.5-4.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-10-13T00:43:43.355 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librbd1"]}, "item": "librbd1", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package (+3 Dependent packages)\n\nInstalled size: 16 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 2019-10-13T00:43:43.357 INFO:teuthology.task.ansible.out: 1/4 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/4 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/4 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/4 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/4 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/4 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 3/4 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 4/4 \n\nRemoved:\n librbd1.x86_64 1:10.2.5-4.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-10-13T00:43:46.543 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librados2"]}, "item": "librados2", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 5.5 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 \n\nComplete!\n"]} 2019-10-13T00:43:47.125 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librados2"]}, "item": "librados2", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 5.5 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 \n\nComplete!\n"]} 2019-10-13T00:43:50.392 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=mod_fastcgi) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["mod_fastcgi"]}, "item": "mod_fastcgi", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7-fcgi-ceph 213 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \n\nRemoved:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:43:50.891 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=mod_fastcgi) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["mod_fastcgi"]}, "item": "mod_fastcgi", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7-fcgi-ceph 213 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \n\nRemoved:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:43:53.970 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7 @lab-extras 1.3 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 1.3 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : iozone-3.424-2_ceph.el7.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.x86_64 1/1 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:43:54.038 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:54 +0000 (0:00:35.734) 0:04:07.297 ******** 2019-10-13T00:43:54.517 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7 @lab-extras 1.3 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 1.3 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : iozone-3.424-2_ceph.el7.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.x86_64 1/1 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:43:54.566 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:43:54 +0000 (0:00:00.527) 0:04:07.824 ******** 2019-10-13T00:43:57.813 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T00:43:57.814 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["boost-random"]}, "item": "boost-random", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @rhel-7-server-rpms 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-10-13T00:43:58.390 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["boost-random"]}, "item": "boost-random", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @rhel-7-server-rpms 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-10-13T00:43:59.088 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=boost-program-options) => {"ansible_loop_var": "item", "changed": false, "item": "boost-program-options", "msg": "", "rc": 0, "results": ["boost-program-options is not installed"]} 2019-10-13T00:43:59.651 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=boost-program-options) => {"ansible_loop_var": "item", "changed": false, "item": "boost-program-options", "msg": "", "rc": 0, "results": ["boost-program-options is not installed"]} 2019-10-13T00:44:02.808 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=leveldb) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["leveldb"]}, "item": "leveldb", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 439 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : leveldb-1.12.0-11.el7.x86_64 1/1 \n Verifying : leveldb-1.12.0-11.el7.x86_64 1/1 \n\nRemoved:\n leveldb.x86_64 0:1.12.0-11.el7 \n\nComplete!\n"]} 2019-10-13T00:44:03.444 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=leveldb) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["leveldb"]}, "item": "leveldb", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 439 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : leveldb-1.12.0-11.el7.x86_64 1/1 \n Verifying : leveldb-1.12.0-11.el7.x86_64 1/1 \n\nRemoved:\n leveldb.x86_64 0:1.12.0-11.el7 \n\nComplete!\n"]} 2019-10-13T00:44:06.604 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=xmlstarlet) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["xmlstarlet"]}, "item": "xmlstarlet", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 192 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n\nRemoved:\n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2019-10-13T00:44:07.210 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=xmlstarlet) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["xmlstarlet"]}, "item": "xmlstarlet", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 192 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n\nRemoved:\n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2019-10-13T00:44:07.859 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-jinja2) => {"ansible_loop_var": "item", "changed": false, "item": "python-jinja2", "msg": "", "rc": 0, "results": ["python-jinja2 is not installed"]} 2019-10-13T00:44:08.433 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-jinja2) => {"ansible_loop_var": "item", "changed": false, "item": "python-jinja2", "msg": "", "rc": 0, "results": ["python-jinja2 is not installed"]} 2019-10-13T00:44:09.060 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-13T00:44:09.675 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-13T00:44:10.287 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-flask) => {"ansible_loop_var": "item", "changed": false, "item": "python-flask", "msg": "", "rc": 0, "results": ["python-flask is not installed"]} 2019-10-13T00:44:10.930 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-flask) => {"ansible_loop_var": "item", "changed": false, "item": "python-flask", "msg": "", "rc": 0, "results": ["python-flask is not installed"]} 2019-10-13T00:44:11.476 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-requests) => {"ansible_loop_var": "item", "changed": false, "item": "python-requests", "msg": "", "rc": 0, "results": ["python-requests is not installed"]} 2019-10-13T00:44:12.162 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-requests) => {"ansible_loop_var": "item", "changed": false, "item": "python-requests", "msg": "", "rc": 0, "results": ["python-requests is not installed"]} 2019-10-13T00:44:12.660 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": false, "item": "boost-random", "msg": "", "rc": 0, "results": ["boost-random is not installed"]} 2019-10-13T00:44:13.439 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": false, "item": "boost-random", "msg": "", "rc": 0, "results": ["boost-random is not installed"]} 2019-10-13T00:44:13.872 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-urllib3) => {"ansible_loop_var": "item", "changed": false, "item": "python-urllib3", "msg": "", "rc": 0, "results": ["python-urllib3 is not installed"]} 2019-10-13T00:44:14.672 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-urllib3) => {"ansible_loop_var": "item", "changed": false, "item": "python-urllib3", "msg": "", "rc": 0, "results": ["python-urllib3 is not installed"]} 2019-10-13T00:44:15.124 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-babel) => {"ansible_loop_var": "item", "changed": false, "item": "python-babel", "msg": "", "rc": 0, "results": ["python-babel is not installed"]} 2019-10-13T00:44:15.932 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-babel) => {"ansible_loop_var": "item", "changed": false, "item": "python-babel", "msg": "", "rc": 0, "results": ["python-babel is not installed"]} 2019-10-13T00:44:16.318 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=hdparm) => {"ansible_loop_var": "item", "changed": false, "item": "hdparm", "msg": "", "rc": 0, "results": ["hdparm is not installed"]} 2019-10-13T00:44:17.122 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=hdparm) => {"ansible_loop_var": "item", "changed": false, "item": "hdparm", "msg": "", "rc": 0, "results": ["hdparm is not installed"]} 2019-10-13T00:44:17.519 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-markupsafe) => {"ansible_loop_var": "item", "changed": false, "item": "python-markupsafe", "msg": "", "rc": 0, "results": ["python-markupsafe is not installed"]} 2019-10-13T00:44:18.384 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-markupsafe) => {"ansible_loop_var": "item", "changed": false, "item": "python-markupsafe", "msg": "", "rc": 0, "results": ["python-markupsafe is not installed"]} 2019-10-13T00:44:18.719 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-werkzeug) => {"ansible_loop_var": "item", "changed": false, "item": "python-werkzeug", "msg": "", "rc": 0, "results": ["python-werkzeug is not installed"]} 2019-10-13T00:44:19.647 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-werkzeug) => {"ansible_loop_var": "item", "changed": false, "item": "python-werkzeug", "msg": "", "rc": 0, "results": ["python-werkzeug is not installed"]} 2019-10-13T00:44:19.980 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-itsdangerous) => {"ansible_loop_var": "item", "changed": false, "item": "python-itsdangerous", "msg": "", "rc": 0, "results": ["python-itsdangerous is not installed"]} 2019-10-13T00:44:20.038 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:44:20 +0000 (0:00:25.472) 0:04:33.297 ******** 2019-10-13T00:44:20.890 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-itsdangerous) => {"ansible_loop_var": "item", "changed": false, "item": "python-itsdangerous", "msg": "", "rc": 0, "results": ["python-itsdangerous is not installed"]} 2019-10-13T00:44:20.936 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:44:20 +0000 (0:00:00.897) 0:04:34.195 ******** 2019-10-13T00:44:22.018 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T00:44:22.020 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=@core) => {"ansible_loop_var": "item", "changed": false, "item": "@core", "msg": "", "rc": 0, "results": []} 2019-10-13T00:44:22.905 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=@core) => {"ansible_loop_var": "item", "changed": false, "item": "@core", "msg": "", "rc": 0, "results": []} 2019-10-13T00:44:24.936 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=@base) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": ["@base"]}, "item": "@base", "msg": "Warning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nNothing to do\n"]} 2019-10-13T00:44:25.803 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=@base) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": ["@base"]}, "item": "@base", "msg": "Warning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nNothing to do\n"]} 2019-10-13T00:44:26.525 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=yum-plugin-priorities) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-priorities", "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed"]} 2019-10-13T00:44:27.427 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=yum-plugin-priorities) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-priorities", "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed"]} 2019-10-13T00:44:28.161 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=yum-plugin-fastestmirror) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-fastestmirror", "msg": "", "rc": 0, "results": ["yum-plugin-fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is already installed"]} 2019-10-13T00:44:29.042 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=yum-plugin-fastestmirror) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-fastestmirror", "msg": "", "rc": 0, "results": ["yum-plugin-fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is already installed"]} 2019-10-13T00:44:29.765 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=redhat-lsb) => {"ansible_loop_var": "item", "changed": false, "item": "redhat-lsb", "msg": "", "rc": 0, "results": ["redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed"]} 2019-10-13T00:44:30.692 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=redhat-lsb) => {"ansible_loop_var": "item", "changed": false, "item": "redhat-lsb", "msg": "", "rc": 0, "results": ["redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed"]} 2019-10-13T00:44:31.390 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=sysstat) => {"ansible_loop_var": "item", "changed": false, "item": "sysstat", "msg": "", "rc": 0, "results": ["sysstat-10.1.5-18.el7.x86_64 providing sysstat is already installed"]} 2019-10-13T00:44:32.251 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=sysstat) => {"ansible_loop_var": "item", "changed": false, "item": "sysstat", "msg": "", "rc": 0, "results": ["sysstat-10.1.5-18.el7.x86_64 providing sysstat is already installed"]} 2019-10-13T00:44:32.970 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=gdb) => {"ansible_loop_var": "item", "changed": false, "item": "gdb", "msg": "", "rc": 0, "results": ["gdb-7.6.1-115.el7.x86_64 providing gdb is already installed"]} 2019-10-13T00:44:33.878 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=gdb) => {"ansible_loop_var": "item", "changed": false, "item": "gdb", "msg": "", "rc": 0, "results": ["gdb-7.6.1-115.el7.x86_64 providing gdb is already installed"]} 2019-10-13T00:44:34.602 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=git-all) => {"ansible_loop_var": "item", "changed": false, "item": "git-all", "msg": "", "rc": 0, "results": ["git-all-1.8.3.1-20.el7.noarch providing git-all is already installed"]} 2019-10-13T00:44:35.566 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=git-all) => {"ansible_loop_var": "item", "changed": false, "item": "git-all", "msg": "", "rc": 0, "results": ["git-all-1.8.3.1-20.el7.noarch providing git-all is already installed"]} 2019-10-13T00:44:36.236 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-configobj) => {"ansible_loop_var": "item", "changed": false, "item": "python-configobj", "msg": "", "rc": 0, "results": ["python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed"]} 2019-10-13T00:44:37.196 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-configobj) => {"ansible_loop_var": "item", "changed": false, "item": "python-configobj", "msg": "", "rc": 0, "results": ["python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed"]} 2019-10-13T00:44:37.791 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=gcc-c++) => {"ansible_loop_var": "item", "changed": false, "item": "gcc-c++", "msg": "", "rc": 0, "results": ["gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed"]} 2019-10-13T00:44:38.852 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=gcc-c++) => {"ansible_loop_var": "item", "changed": false, "item": "gcc-c++", "msg": "", "rc": 0, "results": ["gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed"]} 2019-10-13T00:44:39.450 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libedit) => {"ansible_loop_var": "item", "changed": false, "item": "libedit", "msg": "", "rc": 0, "results": ["libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed"]} 2019-10-13T00:44:40.535 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libedit) => {"ansible_loop_var": "item", "changed": false, "item": "libedit", "msg": "", "rc": 0, "results": ["libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed"]} 2019-10-13T00:44:41.303 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=openssl098e) => {"ansible_loop_var": "item", "changed": false, "item": "openssl098e", "msg": "", "rc": 0, "results": ["openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed"]} 2019-10-13T00:44:42.197 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=openssl098e) => {"ansible_loop_var": "item", "changed": false, "item": "openssl098e", "msg": "", "rc": 0, "results": ["openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed"]} 2019-10-13T00:44:42.915 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=boost-thread) => {"ansible_loop_var": "item", "changed": false, "item": "boost-thread", "msg": "", "rc": 0, "results": ["boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed"]} 2019-10-13T00:44:43.829 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=boost-thread) => {"ansible_loop_var": "item", "changed": false, "item": "boost-thread", "msg": "", "rc": 0, "results": ["boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed"]} 2019-10-13T00:44:44.555 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=xfsprogs) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs", "msg": "", "rc": 0, "results": ["xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed"]} 2019-10-13T00:44:45.524 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=xfsprogs) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs", "msg": "", "rc": 0, "results": ["xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed"]} 2019-10-13T00:44:46.306 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=xfsprogs-devel) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs-devel", "msg": "", "rc": 0, "results": ["xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed"]} 2019-10-13T00:44:47.185 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=xfsprogs-devel) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs-devel", "msg": "", "rc": 0, "results": ["xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed"]} 2019-10-13T00:44:47.877 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed"]} 2019-10-13T00:44:48.786 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed"]} 2019-10-13T00:44:49.520 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=parted) => {"ansible_loop_var": "item", "changed": false, "item": "parted", "msg": "", "rc": 0, "results": ["parted-3.1-31.el7.x86_64 providing parted is already installed"]} 2019-10-13T00:44:50.408 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=parted) => {"ansible_loop_var": "item", "changed": false, "item": "parted", "msg": "", "rc": 0, "results": ["parted-3.1-31.el7.x86_64 providing parted is already installed"]} 2019-10-13T00:44:51.143 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libgcrypt) => {"ansible_loop_var": "item", "changed": false, "item": "libgcrypt", "msg": "", "rc": 0, "results": ["libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed"]} 2019-10-13T00:44:52.069 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libgcrypt) => {"ansible_loop_var": "item", "changed": false, "item": "libgcrypt", "msg": "", "rc": 0, "results": ["libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed"]} 2019-10-13T00:44:52.722 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=fuse) => {"ansible_loop_var": "item", "changed": false, "item": "fuse", "msg": "", "rc": 0, "results": ["fuse-2.9.2-11.el7.x86_64 providing fuse is already installed"]} 2019-10-13T00:44:53.673 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=fuse) => {"ansible_loop_var": "item", "changed": false, "item": "fuse", "msg": "", "rc": 0, "results": ["fuse-2.9.2-11.el7.x86_64 providing fuse is already installed"]} 2019-10-13T00:44:54.287 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=fuse-libs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-libs", "msg": "", "rc": 0, "results": ["fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed"]} 2019-10-13T00:44:55.200 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=fuse-libs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-libs", "msg": "", "rc": 0, "results": ["fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed"]} 2019-10-13T00:44:55.897 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=lvm2) => {"ansible_loop_var": "item", "changed": false, "item": "lvm2", "msg": "", "rc": 0, "results": ["7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed"]} 2019-10-13T00:44:56.861 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=lvm2) => {"ansible_loop_var": "item", "changed": false, "item": "lvm2", "msg": "", "rc": 0, "results": ["7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed"]} 2019-10-13T00:44:57.423 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=openssl) => {"ansible_loop_var": "item", "changed": false, "item": "openssl", "msg": "", "rc": 0, "results": ["1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed"]} 2019-10-13T00:44:58.403 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=openssl) => {"ansible_loop_var": "item", "changed": false, "item": "openssl", "msg": "", "rc": 0, "results": ["1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed"]} 2019-10-13T00:44:59.025 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libuuid) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid", "msg": "", "rc": 0, "results": ["libuuid-2.23.2-61.el7.x86_64 providing libuuid is already installed"]} 2019-10-13T00:45:00.053 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libuuid) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid", "msg": "", "rc": 0, "results": ["libuuid-2.23.2-61.el7.x86_64 providing libuuid is already installed"]} 2019-10-13T00:45:00.691 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=btrfs-progs) => {"ansible_loop_var": "item", "changed": false, "item": "btrfs-progs", "msg": "", "rc": 0, "results": ["btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed"]} 2019-10-13T00:45:01.678 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=btrfs-progs) => {"ansible_loop_var": "item", "changed": false, "item": "btrfs-progs", "msg": "", "rc": 0, "results": ["btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed"]} 2019-10-13T00:45:02.334 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=attr) => {"ansible_loop_var": "item", "changed": false, "item": "attr", "msg": "", "rc": 0, "results": ["attr-2.4.46-13.el7.x86_64 providing attr is already installed"]} 2019-10-13T00:45:03.272 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=attr) => {"ansible_loop_var": "item", "changed": false, "item": "attr", "msg": "", "rc": 0, "results": ["attr-2.4.46-13.el7.x86_64 providing attr is already installed"]} 2019-10-13T00:45:03.975 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=valgrind) => {"ansible_loop_var": "item", "changed": false, "item": "valgrind", "msg": "", "rc": 0, "results": ["1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed"]} 2019-10-13T00:45:04.913 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=valgrind) => {"ansible_loop_var": "item", "changed": false, "item": "valgrind", "msg": "", "rc": 0, "results": ["1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed"]} 2019-10-13T00:45:05.594 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-nose) => {"ansible_loop_var": "item", "changed": false, "item": "python-nose", "msg": "", "rc": 0, "results": ["python-nose-1.3.7-1.el7.noarch providing python-nose is already installed"]} 2019-10-13T00:45:06.542 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-nose) => {"ansible_loop_var": "item", "changed": false, "item": "python-nose", "msg": "", "rc": 0, "results": ["python-nose-1.3.7-1.el7.noarch providing python-nose is already installed"]} 2019-10-13T00:45:10.074 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "msg": "", "rc": 0, "results": ["mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed"]} 2019-10-13T00:45:10.935 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "msg": "", "rc": 0, "results": ["mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed"]} 2019-10-13T00:45:11.692 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ant) => {"ansible_loop_var": "item", "changed": false, "item": "ant", "msg": "", "rc": 0, "results": ["ant-1.9.4-2.el7.noarch providing ant is already installed"]} 2019-10-13T00:45:12.576 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ant) => {"ansible_loop_var": "item", "changed": false, "item": "ant", "msg": "", "rc": 0, "results": ["ant-1.9.4-2.el7.noarch providing ant is already installed"]} 2019-10-13T00:45:13.271 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=lsof) => {"ansible_loop_var": "item", "changed": false, "item": "lsof", "msg": "", "rc": 0, "results": ["lsof-4.87-6.el7.x86_64 providing lsof is already installed"]} 2019-10-13T00:45:14.165 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=lsof) => {"ansible_loop_var": "item", "changed": false, "item": "lsof", "msg": "", "rc": 0, "results": ["lsof-4.87-6.el7.x86_64 providing lsof is already installed"]} 2019-10-13T00:45:19.637 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7 lab-extras 800 k\n\nTransaction Summary\n================================================================================ 2019-10-13T00:45:19.639 INFO:teuthology.task.ansible.out:\nInstall 1 Package\n\nTotal download size: 800 k\nInstalled size: 800 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : iozone-3.424-2_ceph.el7.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.x86_64 1/1 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:45:20.638 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7 lab-extras 800 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 800 k\nInstalled size: 800 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : iozone-3.424-2_ceph.el7.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.x86_64 1/1 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:45:21.169 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libtool) => {"ansible_loop_var": "item", "changed": false, "item": "libtool", "msg": "", "rc": 0, "results": ["libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed"]} 2019-10-13T00:45:22.227 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libtool) => {"ansible_loop_var": "item", "changed": false, "item": "libtool", "msg": "", "rc": 0, "results": ["libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed"]} 2019-10-13T00:45:22.713 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=automake) => {"ansible_loop_var": "item", "changed": false, "item": "automake", "msg": "", "rc": 0, "results": ["automake-1.13.4-3.el7.noarch providing automake is already installed"]} 2019-10-13T00:45:23.816 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=automake) => {"ansible_loop_var": "item", "changed": false, "item": "automake", "msg": "", "rc": 0, "results": ["automake-1.13.4-3.el7.noarch providing automake is already installed"]} 2019-10-13T00:45:24.298 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=gettext) => {"ansible_loop_var": "item", "changed": false, "item": "gettext", "msg": "", "rc": 0, "results": ["gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed"]} 2019-10-13T00:45:25.391 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=gettext) => {"ansible_loop_var": "item", "changed": false, "item": "gettext", "msg": "", "rc": 0, "results": ["gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed"]} 2019-10-13T00:45:25.957 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libuuid-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid-devel", "msg": "", "rc": 0, "results": ["libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed"]} 2019-10-13T00:45:27.043 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libuuid-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid-devel", "msg": "", "rc": 0, "results": ["libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed"]} 2019-10-13T00:45:27.578 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libacl-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libacl-devel", "msg": "", "rc": 0, "results": ["libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed"]} 2019-10-13T00:45:28.604 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libacl-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libacl-devel", "msg": "", "rc": 0, "results": ["libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed"]} 2019-10-13T00:45:29.207 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=bc) => {"ansible_loop_var": "item", "changed": false, "item": "bc", "msg": "", "rc": 0, "results": ["bc-1.06.95-13.el7.x86_64 providing bc is already installed"]} 2019-10-13T00:45:30.281 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=bc) => {"ansible_loop_var": "item", "changed": false, "item": "bc", "msg": "", "rc": 0, "results": ["bc-1.06.95-13.el7.x86_64 providing bc is already installed"]} 2019-10-13T00:45:30.861 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=xfsdump) => {"ansible_loop_var": "item", "changed": false, "item": "xfsdump", "msg": "", "rc": 0, "results": ["xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed"]} 2019-10-13T00:45:32.064 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=xfsdump) => {"ansible_loop_var": "item", "changed": false, "item": "xfsdump", "msg": "", "rc": 0, "results": ["xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed"]} 2019-10-13T00:45:32.553 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=blktrace) => {"ansible_loop_var": "item", "changed": false, "item": "blktrace", "msg": "", "rc": 0, "results": ["blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed"]} 2019-10-13T00:45:33.907 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=blktrace) => {"ansible_loop_var": "item", "changed": false, "item": "blktrace", "msg": "", "rc": 0, "results": ["blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed"]} 2019-10-13T00:45:34.418 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=numpy) => {"ansible_loop_var": "item", "changed": false, "item": "numpy", "msg": "", "rc": 0, "results": ["1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed"]} 2019-10-13T00:45:35.740 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=numpy) => {"ansible_loop_var": "item", "changed": false, "item": "numpy", "msg": "", "rc": 0, "results": ["1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed"]} 2019-10-13T00:45:36.264 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-matplotlib) => {"ansible_loop_var": "item", "changed": false, "item": "python-matplotlib", "msg": "", "rc": 0, "results": ["python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed"]} 2019-10-13T00:45:37.479 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-matplotlib) => {"ansible_loop_var": "item", "changed": false, "item": "python-matplotlib", "msg": "", "rc": 0, "results": ["python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed"]} 2019-10-13T00:46:12.978 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=qemu-kvm) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["qemu-kvm"]}, "item": "qemu-kvm", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 rhel-7-server-rpms 39 k\n librados2 2019-10-13T00:46:12.980 INFO:teuthology.task.ansible.out:x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 rhel-7-server-rpms 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 rhel-7-server-rpms 435 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 25 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/6 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/6 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/6 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/6 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/6 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/6 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 3/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/6 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Verifying : boost-random-1.53.0-27.el7.x86_64 6/6 \n\nInstalled:\n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 2019-10-13T00:46:12.981 INFO:teuthology.task.ansible.out: 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-10-13T00:46:14.059 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=qemu-kvm) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["qemu-kvm"]}, "item": "qemu-kvm", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 rhel-7-server-rpms 39 k\n librados2 2019-10-13T00:46:14.060 INFO:teuthology.task.ansible.out:x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 rhel-7-server-rpms 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 rhel-7-server-rpms 435 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 21 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/6 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/6 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/6 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/6 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/6 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/6 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 3/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/6 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Verifying : boost-random-1.53.0-27.el7.x86_64 6/6 \n\nInstalled:\n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 2019-10-13T00:46:14.062 INFO:teuthology.task.ansible.out: 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-10-13T00:46:15.024 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=usbredir) => {"ansible_loop_var": "item", "changed": false, "item": "usbredir", "msg": "", "rc": 0, "results": ["usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed"]} 2019-10-13T00:46:15.846 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=usbredir) => {"ansible_loop_var": "item", "changed": false, "item": "usbredir", "msg": "", "rc": 0, "results": ["usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed"]} 2019-10-13T00:46:17.013 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=genisoimage) => {"ansible_loop_var": "item", "changed": false, "item": "genisoimage", "msg": "", "rc": 0, "results": ["genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed"]} 2019-10-13T00:46:17.771 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=genisoimage) => {"ansible_loop_var": "item", "changed": false, "item": "genisoimage", "msg": "", "rc": 0, "results": ["genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed"]} 2019-10-13T00:46:19.149 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=httpd) => {"ansible_loop_var": "item", "changed": false, "item": "httpd", "msg": "", "rc": 0, "results": ["httpd-2.4.6-90.el7.x86_64 providing httpd is already installed"]} 2019-10-13T00:46:19.928 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=httpd) => {"ansible_loop_var": "item", "changed": false, "item": "httpd", "msg": "", "rc": 0, "results": ["httpd-2.4.6-90.el7.x86_64 providing httpd is already installed"]} 2019-10-13T00:46:21.106 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=httpd-devel) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-devel", "msg": "", "rc": 0, "results": ["httpd-devel-2.4.6-90.el7.x86_64 providing httpd-devel is already installed"]} 2019-10-13T00:46:21.718 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=httpd-devel) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-devel", "msg": "", "rc": 0, "results": ["httpd-devel-2.4.6-90.el7.x86_64 providing httpd-devel is already installed"]} 2019-10-13T00:46:22.863 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=httpd-tools) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-tools", "msg": "", "rc": 0, "results": ["httpd-tools-2.4.6-90.el7.x86_64 providing httpd-tools is already installed"]} 2019-10-13T00:46:23.441 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=httpd-tools) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-tools", "msg": "", "rc": 0, "results": ["httpd-tools-2.4.6-90.el7.x86_64 providing httpd-tools is already installed"]} 2019-10-13T00:46:24.507 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=mod_ssl) => {"ansible_loop_var": "item", "changed": false, "item": "mod_ssl", "msg": "", "rc": 0, "results": ["1:mod_ssl-2.4.6-90.el7.x86_64 providing mod_ssl is already installed"]} 2019-10-13T00:46:25.295 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=mod_ssl) => {"ansible_loop_var": "item", "changed": false, "item": "mod_ssl", "msg": "", "rc": 0, "results": ["1:mod_ssl-2.4.6-90.el7.x86_64 providing mod_ssl is already installed"]} 2019-10-13T00:46:38.713 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=mod_fastcgi-2.4.7-1.ceph.el7) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["mod_fastcgi-2.4.7-1.ceph.el7"]}, "item": "mod_fastcgi-2.4.7-1.ceph.el7", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 70 k\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:46:39.496 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=mod_fastcgi-2.4.7-1.ceph.el7) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["mod_fastcgi-2.4.7-1.ceph.el7"]}, "item": "mod_fastcgi-2.4.7-1.ceph.el7", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 70 k\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 1/1 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nComplete!\n"]} 2019-10-13T00:46:40.382 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=libev-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libev-devel", "msg": "", "rc": 0, "results": ["libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed"]} 2019-10-13T00:46:41.234 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=libev-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libev-devel", "msg": "", "rc": 0, "results": ["libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed"]} 2019-10-13T00:46:42.129 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=perl-XML-Twig) => {"ansible_loop_var": "item", "changed": false, "item": "perl-XML-Twig", "msg": "", "rc": 0, "results": ["perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed"]} 2019-10-13T00:46:43.018 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=perl-XML-Twig) => {"ansible_loop_var": "item", "changed": false, "item": "perl-XML-Twig", "msg": "", "rc": 0, "results": ["perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed"]} 2019-10-13T00:46:43.718 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=java-1.6.0-openjdk-devel) => {"ansible_loop_var": "item", "changed": false, "item": "java-1.6.0-openjdk-devel", "msg": "", "rc": 0, "results": ["1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed"]} 2019-10-13T00:46:44.675 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=java-1.6.0-openjdk-devel) => {"ansible_loop_var": "item", "changed": false, "item": "java-1.6.0-openjdk-devel", "msg": "", "rc": 0, "results": ["1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed"]} 2019-10-13T00:46:46.765 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=junit4) => {"ansible_loop_var": "item", "changed": false, "item": "junit4", "msg": "", "rc": 0, "results": ["junit-4.11-8.el7.noarch providing junit4 is already installed"]} 2019-10-13T00:46:47.658 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=junit4) => {"ansible_loop_var": "item", "changed": false, "item": "junit4", "msg": "", "rc": 0, "results": ["junit-4.11-8.el7.noarch providing junit4 is already installed"]} 2019-10-13T00:46:48.346 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nfs-utils) => {"ansible_loop_var": "item", "changed": false, "item": "nfs-utils", "msg": "", "rc": 0, "results": ["1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed"]} 2019-10-13T00:46:49.291 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nfs-utils) => {"ansible_loop_var": "item", "changed": false, "item": "nfs-utils", "msg": "", "rc": 0, "results": ["1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed"]} 2019-10-13T00:46:49.885 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ncurses-devel) => {"ansible_loop_var": "item", "changed": false, "item": "ncurses-devel", "msg": "", "rc": 0, "results": ["ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed"]} 2019-10-13T00:46:50.985 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=ncurses-devel) => {"ansible_loop_var": "item", "changed": false, "item": "ncurses-devel", "msg": "", "rc": 0, "results": ["ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed"]} 2019-10-13T00:46:51.504 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-devel) => {"ansible_loop_var": "item", "changed": false, "item": "python-devel", "msg": "", "rc": 0, "results": ["python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed"]} 2019-10-13T00:46:52.529 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-devel) => {"ansible_loop_var": "item", "changed": false, "item": "python-devel", "msg": "", "rc": 0, "results": ["python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed"]} 2019-10-13T00:46:53.112 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=python-virtualenv) => {"ansible_loop_var": "item", "changed": false, "item": "python-virtualenv", "msg": "", "rc": 0, "results": ["python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed"]} 2019-10-13T00:46:54.064 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=python-virtualenv) => {"ansible_loop_var": "item", "changed": false, "item": "python-virtualenv", "msg": "", "rc": 0, "results": ["python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed"]} 2019-10-13T00:46:54.703 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=perl-CPAN) => {"ansible_loop_var": "item", "changed": false, "item": "perl-CPAN", "msg": "", "rc": 0, "results": ["perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed"]} 2019-10-13T00:46:54.766 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:46:54 +0000 (0:02:33.829) 0:07:08.024 ******** 2019-10-13T00:46:55.647 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=perl-CPAN) => {"ansible_loop_var": "item", "changed": false, "item": "perl-CPAN", "msg": "", "rc": 0, "results": ["perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed"]} 2019-10-13T00:46:55.723 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:46:55 +0000 (0:00:00.957) 0:07:08.982 ******** 2019-10-13T00:46:56.456 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-13T00:46:56.459 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=cryptopp-devel) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp-devel", "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed"]} 2019-10-13T00:46:57.431 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=cryptopp-devel) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp-devel", "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed"]} 2019-10-13T00:46:58.090 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=cryptopp) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp", "msg": "", "rc": 0, "results": ["cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed"]} 2019-10-13T00:46:59.055 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=cryptopp) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp", "msg": "", "rc": 0, "results": ["cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed"]} 2019-10-13T00:46:59.781 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=dbench) => {"ansible_loop_var": "item", "changed": false, "item": "dbench", "msg": "", "rc": 0, "results": ["dbench-4.0-10.el7.x86_64 providing dbench is already installed"]} 2019-10-13T00:47:00.795 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=dbench) => {"ansible_loop_var": "item", "changed": false, "item": "dbench", "msg": "", "rc": 0, "results": ["dbench-4.0-10.el7.x86_64 providing dbench is already installed"]} 2019-10-13T00:47:01.517 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=fcgi) => {"ansible_loop_var": "item", "changed": false, "item": "fcgi", "msg": "", "rc": 0, "results": ["fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed"]} 2019-10-13T00:47:02.404 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=fcgi) => {"ansible_loop_var": "item", "changed": false, "item": "fcgi", "msg": "", "rc": 0, "results": ["fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed"]} 2019-10-13T00:47:03.126 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=fuse-sshfs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-sshfs", "msg": "", "rc": 0, "results": ["fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed"]} 2019-10-13T00:47:03.919 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=fuse-sshfs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-sshfs", "msg": "", "rc": 0, "results": ["fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed"]} 2019-10-13T00:47:04.646 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=perl-JSON-XS) => {"ansible_loop_var": "item", "changed": false, "item": "perl-JSON-XS", "msg": "", "rc": 0, "results": ["1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-10-13T00:47:05.519 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=perl-JSON-XS) => {"ansible_loop_var": "item", "changed": false, "item": "perl-JSON-XS", "msg": "", "rc": 0, "results": ["1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-10-13T00:47:11.123 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=leveldb) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["leveldb"]}, "item": "leveldb", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 161 k\nInstalled size: 439 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : leveldb-1.12.0-11.el7.x86_64 1/1 \n Verifying : leveldb-1.12.0-11.el7.x86_64 1/1 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 \n\nComplete!\n"]} 2019-10-13T00:47:11.932 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=leveldb) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["leveldb"]}, "item": "leveldb", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 161 k\nInstalled size: 439 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : leveldb-1.12.0-11.el7.x86_64 1/1 \n Verifying : leveldb-1.12.0-11.el7.x86_64 1/1 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 \n\nComplete!\n"]} 2019-10-13T00:47:17.432 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=xmlstarlet) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["xmlstarlet"]}, "item": "xmlstarlet", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 62 k\nInstalled size: 192 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n\nInstalled:\n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2019-10-13T00:47:17.490 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:17 +0000 (0:00:21.766) 0:07:30.749 ******** 2019-10-13T00:47:17.520 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T00:47:17.575 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:17 +0000 (0:00:00.084) 0:07:30.834 ******** 2019-10-13T00:47:17.602 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T00:47:17.643 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:17 +0000 (0:00:00.068) 0:07:30.902 ******** 2019-10-13T00:47:17.885 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-13T00:47:17.886 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice dbus.service polkit.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ebtables.service iptables.service ipset.service shutdown.target ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2019-10-13T00:47:17.888 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2019-10-13T00:47:17.889 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-13T00:47:17.928 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:17 +0000 (0:00:00.284) 0:07:31.187 ******** 2019-10-13T00:47:18.174 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-10-13T00:47:18.176 INFO:teuthology.task.ansible.out:fatal: [smithi142.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-10-13T00:47:18.211 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:18 +0000 (0:00:00.283) 0:07:31.470 ******** 2019-10-13T00:47:18.384 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-13T00:47:18.385 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=xmlstarlet) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["xmlstarlet"]}, "item": "xmlstarlet", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: ftp.linux.ncsu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 62 k\nInstalled size: 192 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 1/1 \n\nInstalled:\n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2019-10-13T00:47:18.430 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:18 +0000 (0:00:00.218) 0:07:31.689 ******** 2019-10-13T00:47:18.457 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T00:47:18.494 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:18 +0000 (0:00:00.063) 0:07:31.753 ******** 2019-10-13T00:47:18.524 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T00:47:18.561 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:18 +0000 (0:00:00.066) 0:07:31.820 ******** 2019-10-13T00:47:18.803 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-13T00:47:18.805 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice dbus.service polkit.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ebtables.service ipset.service shutdown.target ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2019-10-13T00:47:18.806 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2019-10-13T00:47:18.808 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-13T00:47:18.852 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:18 +0000 (0:00:00.291) 0:07:32.111 ******** 2019-10-13T00:47:18.858 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-10-13T00:47:18.859 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-10-13T00:47:18.903 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:18 +0000 (0:00:00.050) 0:07:32.162 ******** 2019-10-13T00:47:18.932 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:18.968 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:18 +0000 (0:00:00.064) 0:07:32.227 ******** 2019-10-13T00:47:18.994 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-13T00:47:19.032 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.064) 0:07:32.291 ******** 2019-10-13T00:47:19.059 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-13T00:47:19.061 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.105 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.072) 0:07:32.364 ******** 2019-10-13T00:47:19.111 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-10-13T00:47:19.111 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-10-13T00:47:19.150 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.044) 0:07:32.409 ******** 2019-10-13T00:47:19.158 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-10-13T00:47:19.158 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.199 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.048) 0:07:32.458 ******** 2019-10-13T00:47:19.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi142.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.285 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.086) 0:07:32.544 ******** 2019-10-13T00:47:19.312 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.354 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.069) 0:07:32.613 ******** 2019-10-13T00:47:19.373 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-10-13T00:47:19.374 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-10-13T00:47:19.411 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.056) 0:07:32.670 ******** 2019-10-13T00:47:19.417 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-13T00:47:19.460 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.048) 0:07:32.719 ******** 2019-10-13T00:47:19.465 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-13T00:47:19.467 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.506 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.045) 0:07:32.764 ******** 2019-10-13T00:47:19.510 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-10-13T00:47:19.512 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.548 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.042) 0:07:32.807 ******** 2019-10-13T00:47:19.553 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-13T00:47:19.591 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.043) 0:07:32.850 ******** 2019-10-13T00:47:19.596 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-10-13T00:47:19.599 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.635 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.043) 0:07:32.894 ******** 2019-10-13T00:47:19.641 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-13T00:47:19.643 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.656 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:19.679 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.044) 0:07:32.938 ******** 2019-10-13T00:47:19.705 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-10-13T00:47:19.707 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.745 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.065) 0:07:33.004 ******** 2019-10-13T00:47:19.750 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-10-13T00:47:19.751 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.797 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.051) 0:07:33.056 ******** 2019-10-13T00:47:19.802 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.803 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.841 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.044) 0:07:33.100 ******** 2019-10-13T00:47:19.847 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T00:47:19.885 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.043) 0:07:33.144 ******** 2019-10-13T00:47:19.890 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-10-13T00:47:19.892 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:19.927 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.042) 0:07:33.186 ******** 2019-10-13T00:47:19.932 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-13T00:47:19.971 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:19 +0000 (0:00:00.044) 0:07:33.230 ******** 2019-10-13T00:47:19.978 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-13T00:47:19.997 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:20.016 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.043) 0:07:33.274 ******** 2019-10-13T00:47:20.043 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-10-13T00:47:20.045 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.080 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.064) 0:07:33.338 ******** 2019-10-13T00:47:20.106 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-10-13T00:47:20.107 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.153 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.072) 0:07:33.411 ******** 2019-10-13T00:47:20.174 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:20.176 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T00:47:20.179 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.219 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.066) 0:07:33.477 ******** 2019-10-13T00:47:20.235 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-10-13T00:47:20.237 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.275 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.056) 0:07:33.533 ******** 2019-10-13T00:47:20.280 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-13T00:47:20.319 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.044) 0:07:33.578 ******** 2019-10-13T00:47:20.326 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T00:47:20.365 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.045) 0:07:33.623 ******** 2019-10-13T00:47:20.369 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-13T00:47:20.370 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.409 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.044) 0:07:33.667 ******** 2019-10-13T00:47:20.414 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-13T00:47:20.451 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.041) 0:07:33.709 ******** 2019-10-13T00:47:20.454 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-13T00:47:20.456 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.470 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:20.494 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.043) 0:07:33.753 ******** 2019-10-13T00:47:20.520 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-13T00:47:20.523 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.557 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.063) 0:07:33.816 ******** 2019-10-13T00:47:20.583 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-13T00:47:20.585 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.622 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.064) 0:07:33.881 ******** 2019-10-13T00:47:20.627 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T00:47:20.627 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.665 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.043) 0:07:33.924 ******** 2019-10-13T00:47:20.672 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-10-13T00:47:20.672 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.708 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.042) 0:07:33.967 ******** 2019-10-13T00:47:20.713 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-13T00:47:20.751 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.042) 0:07:34.010 ******** 2019-10-13T00:47:20.756 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-13T00:47:20.758 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.796 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.045) 0:07:34.055 ******** 2019-10-13T00:47:20.802 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-13T00:47:20.804 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.842 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.045) 0:07:34.100 ******** 2019-10-13T00:47:20.851 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-13T00:47:20.860 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.891 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.048) 0:07:34.149 ******** 2019-10-13T00:47:20.896 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-13T00:47:20.897 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.936 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.045) 0:07:34.195 ******** 2019-10-13T00:47:20.940 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-13T00:47:20.941 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:20.980 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:20 +0000 (0:00:00.043) 0:07:34.238 ******** 2019-10-13T00:47:20.986 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-13T00:47:20.986 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.026 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.046) 0:07:34.285 ******** 2019-10-13T00:47:21.031 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T00:47:21.032 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.070 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.043) 0:07:34.328 ******** 2019-10-13T00:47:21.079 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-13T00:47:21.080 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.117 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.046) 0:07:34.375 ******** 2019-10-13T00:47:21.122 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T00:47:21.124 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.161 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.044) 0:07:34.419 ******** 2019-10-13T00:47:21.167 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-10-13T00:47:21.168 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.208 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.046) 0:07:34.466 ******** 2019-10-13T00:47:21.213 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T00:47:21.213 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.251 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.043) 0:07:34.510 ******** 2019-10-13T00:47:21.257 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-13T00:47:21.258 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.295 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.043) 0:07:34.554 ******** 2019-10-13T00:47:21.300 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T00:47:21.303 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.340 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.045) 0:07:34.599 ******** 2019-10-13T00:47:21.346 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-13T00:47:21.348 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.385 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.044) 0:07:34.643 ******** 2019-10-13T00:47:21.390 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T00:47:21.390 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.430 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.045) 0:07:34.689 ******** 2019-10-13T00:47:21.438 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.450 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:21.475 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.045) 0:07:34.734 ******** 2019-10-13T00:47:21.502 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T00:47:21.503 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi142.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.541 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.065) 0:07:34.800 ******** 2019-10-13T00:47:21.559 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_dependency_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:21.577 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.035) 0:07:34.836 ******** 2019-10-13T00:47:21.604 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T00:47:21.605 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.606 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi142.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.641 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.064) 0:07:34.900 ******** 2019-10-13T00:47:21.679 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.037) 0:07:34.938 ******** 2019-10-13T00:47:21.684 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T00:47:21.721 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.042) 0:07:34.980 ******** 2019-10-13T00:47:21.727 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T00:47:21.729 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.741 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:21.764 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.043) 0:07:35.023 ******** 2019-10-13T00:47:21.791 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T00:47:21.793 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.829 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.064) 0:07:35.087 ******** 2019-10-13T00:47:21.854 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T00:47:21.856 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.900 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.071) 0:07:35.159 ******** 2019-10-13T00:47:21.906 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T00:47:21.908 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:21.919 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:21.945 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:21 +0000 (0:00:00.045) 0:07:35.204 ******** 2019-10-13T00:47:21.973 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T00:47:21.973 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi155.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.015 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.069) 0:07:35.273 ******** 2019-10-13T00:47:22.049 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.034) 0:07:35.308 ******** 2019-10-13T00:47:22.054 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T00:47:22.056 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.068 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_dependency_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:22.092 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.042) 0:07:35.351 ******** 2019-10-13T00:47:22.119 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T00:47:22.120 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi155.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.157 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.065) 0:07:35.416 ******** 2019-10-13T00:47:22.191 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.033) 0:07:35.449 ******** 2019-10-13T00:47:22.196 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T00:47:22.197 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.234 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.043) 0:07:35.493 ******** 2019-10-13T00:47:22.239 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T00:47:22.283 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.048) 0:07:35.542 ******** 2019-10-13T00:47:22.288 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T00:47:22.289 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.305 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:22.327 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.044) 0:07:35.586 ******** 2019-10-13T00:47:22.457 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T00:47:22.458 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.498 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.167) 0:07:35.754 ******** 2019-10-13T00:47:22.520 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T00:47:22.557 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.062) 0:07:35.816 ******** 2019-10-13T00:47:22.583 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T00:47:22.585 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.619 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.062) 0:07:35.878 ******** 2019-10-13T00:47:22.647 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.682 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.062) 0:07:35.941 ******** 2019-10-13T00:47:22.708 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T00:47:22.710 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.744 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.062) 0:07:36.003 ******** 2019-10-13T00:47:22.770 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T00:47:22.772 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:22.807 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.063) 0:07:36.066 ******** 2019-10-13T00:47:22.834 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T00:47:22.835 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2019-10-13T00:47:22.871 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:22 +0000 (0:00:00.063) 0:07:36.130 ******** 2019-10-13T00:47:23.118 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T00:47:23.119 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 00:38:15 UTC", "ActiveEnterTimestampMonotonic": "20157699", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "rpc-gssd.service local-fs.target nfs-mountd.service proc-fs-nfsd.mount system.slice rpcbind.socket network-online.target -.mount systemd-journald.socket nfs-idmapd.service rpc-statd.service tmp.mount nfs-config.service gssproxy.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 00:38:15 UTC", "AssertTimestampMonotonic": "19980266", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-mountd.service nfs-idmapd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 00:38:15 UTC", "ConditionTimestampMonotonic": "19980266", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Sun 2019-10-13 00:38:15 UTC", "ExecMainExitTimestampMonotonic": "20128085", "ExecMainPID": "2129", "ExecMainStartTimestamp": "Sun 2019-10-13 00:38:15 UTC", "ExecMainStartTimestampMonotonic": "20002177", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop 2019-10-13T00:47:23.121 INFO:teuthology.task.ansible.out:_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 00:38:15 UTC", "InactiveExitTimestampMonotonic": "19981087", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "Me 2019-10-13T00:47:23.122 INFO:teuthology.task.ansible.out:moryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "-.mount network.target nfs-mountd.service proc-fs-nfsd.mount tmp.mount", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "rpcbind.socket nfs-idmapd.service rpc-statd-notify.service rpc-statd.service auth-rpcgss-module.service system.slice nfs-config.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-13T00:47:23.156 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.283) 0:07:36.414 ******** 2019-10-13T00:47:23.192 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T00:47:23.194 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.227 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.072) 0:07:36.486 ******** 2019-10-13T00:47:23.274 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T00:47:23.276 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.314 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.086) 0:07:36.573 ******** 2019-10-13T00:47:23.319 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T00:47:23.320 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2019-10-13T00:47:23.364 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.049) 0:07:36.623 ******** 2019-10-13T00:47:23.369 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T00:47:23.370 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.409 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.045) 0:07:36.668 ******** 2019-10-13T00:47:23.436 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T00:47:23.438 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.475 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.065) 0:07:36.733 ******** 2019-10-13T00:47:23.501 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T00:47:23.503 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.536 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.061) 0:07:36.795 ******** 2019-10-13T00:47:23.573 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T00:47:23.574 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.619 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.082) 0:07:36.878 ******** 2019-10-13T00:47:23.626 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T00:47:23.628 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 00:38:55 UTC", "ActiveEnterTimestampMonotonic": "20792436", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "rpc-statd.service nfs-config.service local-fs.target gssproxy.service systemd-journald.socket -.mount proc-fs-nfsd.mount system.slice nfs-idmapd.service rpcbind.socket nfs-mountd.service rpc-gssd.service network-online.target tmp.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 00:38:55 UTC", "AssertTimestampMonotonic": "20587408", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-idmapd.service nfs-mountd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 00:38:55 UTC", "ConditionTimestampMonotonic": "20587408", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Sun 2019-10-13 00:38:55 UTC", "ExecMainExitTimestampMonotonic": "20749033", "ExecMainPID": "2138", "ExecMainStartTimestamp": "Sun 2019-10-13 00:38:55 UTC", "ExecMainStartTimestampMonotonic": "20635328", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop 2019-10-13T00:47:23.630 INFO:teuthology.task.ansible.out:_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 00:38:55 UTC", "InactiveExitTimestampMonotonic": "20588196", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "Me 2019-10-13T00:47:23.632 INFO:teuthology.task.ansible.out:moryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "nfs-mountd.service network.target tmp.mount -.mount proc-fs-nfsd.mount", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "rpcbind.socket network-online.target rpc-statd-notify.service rpc-statd.service nfs-config.service system.slice auth-rpcgss-module.service nfs-idmapd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-13T00:47:23.668 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.049) 0:07:36.927 ******** 2019-10-13T00:47:23.674 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.711 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.042) 0:07:36.970 ******** 2019-10-13T00:47:23.717 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T00:47:23.719 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.755 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.043) 0:07:37.014 ******** 2019-10-13T00:47:23.760 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-13T00:47:23.761 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.799 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.044) 0:07:37.058 ******** 2019-10-13T00:47:23.805 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T00:47:23.805 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.842 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.042) 0:07:37.101 ******** 2019-10-13T00:47:23.846 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-13T00:47:23.849 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.887 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.045) 0:07:37.146 ******** 2019-10-13T00:47:23.893 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T00:47:23.894 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.931 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.043) 0:07:37.190 ******** 2019-10-13T00:47:23.936 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T00:47:23.936 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:23.974 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:23 +0000 (0:00:00.042) 0:07:37.233 ******** 2019-10-13T00:47:23.979 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T00:47:23.980 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.017 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.042) 0:07:37.276 ******** 2019-10-13T00:47:24.021 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T00:47:24.021 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.058 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.041) 0:07:37.317 ******** 2019-10-13T00:47:24.063 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T00:47:24.067 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.101 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.042) 0:07:37.360 ******** 2019-10-13T00:47:24.106 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-13T00:47:24.106 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.144 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.042) 0:07:37.403 ******** 2019-10-13T00:47:24.149 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T00:47:24.149 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.202 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.058) 0:07:37.461 ******** 2019-10-13T00:47:24.207 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-10-13T00:47:24.208 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.248 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.045) 0:07:37.507 ******** 2019-10-13T00:47:24.253 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T00:47:24.255 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.293 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.045) 0:07:37.552 ******** 2019-10-13T00:47:24.297 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-13T00:47:24.300 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.336 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:37.595 ******** 2019-10-13T00:47:24.342 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-13T00:47:24.343 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.380 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:37.639 ******** 2019-10-13T00:47:24.385 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-13T00:47:24.385 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.423 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:37.682 ******** 2019-10-13T00:47:24.428 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-13T00:47:24.430 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.466 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.042) 0:07:37.725 ******** 2019-10-13T00:47:24.470 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-10-13T00:47:24.470 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.510 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:37.769 ******** 2019-10-13T00:47:24.518 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T00:47:24.519 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.554 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.044) 0:07:37.813 ******** 2019-10-13T00:47:24.559 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-13T00:47:24.560 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.598 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:37.857 ******** 2019-10-13T00:47:24.603 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T00:47:24.603 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.642 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.044) 0:07:37.901 ******** 2019-10-13T00:47:24.647 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-13T00:47:24.647 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.686 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:37.944 ******** 2019-10-13T00:47:24.691 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-13T00:47:24.691 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.728 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.042) 0:07:37.987 ******** 2019-10-13T00:47:24.732 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-13T00:47:24.735 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.771 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:38.030 ******** 2019-10-13T00:47:24.777 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-10-13T00:47:24.777 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.815 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:38.074 ******** 2019-10-13T00:47:24.820 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-13T00:47:24.821 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.859 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.043) 0:07:38.118 ******** 2019-10-13T00:47:24.865 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-13T00:47:24.865 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.907 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.047) 0:07:38.165 ******** 2019-10-13T00:47:24.911 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:47:24.912 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-13T00:47:24.949 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.042) 0:07:38.207 ******** 2019-10-13T00:47:24.953 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-13T00:47:24.956 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:24.993 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:24 +0000 (0:00:00.044) 0:07:38.252 ******** 2019-10-13T00:47:24.999 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-13T00:47:25.000 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.002 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-10-13T00:47:25.013 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.013 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.015 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi142.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.051 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.057) 0:07:38.310 ******** 2019-10-13T00:47:25.055 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.056 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.058 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.099 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.048) 0:07:38.358 ******** 2019-10-13T00:47:25.105 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-13T00:47:25.105 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.143 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.044) 0:07:38.402 ******** 2019-10-13T00:47:25.169 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-13T00:47:25.170 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.205 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.062) 0:07:38.464 ******** 2019-10-13T00:47:25.231 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-13T00:47:25.233 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.269 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.063) 0:07:38.528 ******** 2019-10-13T00:47:25.274 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-13T00:47:25.275 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003680", "end": "2019-10-13 00:47:25.218492", "rc": 0, "start": "2019-10-13 00:47:25.214812", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-10-13T00:47:25.318 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.048) 0:07:38.577 ******** 2019-10-13T00:47:25.324 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-13T00:47:25.326 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.363 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.044) 0:07:38.622 ******** 2019-10-13T00:47:25.368 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-13T00:47:25.368 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.409 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.046) 0:07:38.668 ******** 2019-10-13T00:47:25.414 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:47:25.415 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-13T00:47:25.452 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.042) 0:07:38.711 ******** 2019-10-13T00:47:25.457 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-13T00:47:25.458 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.495 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.043) 0:07:38.754 ******** 2019-10-13T00:47:25.502 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-13T00:47:25.502 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.504 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-10-13T00:47:25.514 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.525 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.526 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.527 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi155.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.567 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.071) 0:07:38.826 ******** 2019-10-13T00:47:25.573 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.574 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.583 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:25.617 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.049) 0:07:38.876 ******** 2019-10-13T00:47:25.621 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-13T00:47:25.622 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.660 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.042) 0:07:38.919 ******** 2019-10-13T00:47:25.759 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-13T00:47:25.760 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003827", "end": "2019-10-13 00:47:25.734513", "rc": 0, "start": "2019-10-13 00:47:25.730686", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-10-13T00:47:25.801 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.140) 0:07:39.060 ******** 2019-10-13T00:47:25.838 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-13T00:47:25.840 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.878 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.077) 0:07:39.137 ******** 2019-10-13T00:47:25.915 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-13T00:47:25.916 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:25.957 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:25 +0000 (0:00:00.078) 0:07:39.216 ******** 2019-10-13T00:47:25.983 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-10-13T00:47:25.984 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:26.021 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:26 +0000 (0:00:00.063) 0:07:39.279 ******** 2019-10-13T00:47:26.039 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:26.056 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-13T00:47:26.058 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T00:47:26.095 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:26 +0000 (0:00:00.074) 0:07:39.354 ******** 2019-10-13T00:47:27.278 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-13T00:47:27.280 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed"]} 2019-10-13T00:47:27.693 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed"]} 2019-10-13T00:47:28.830 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "msg": "", "rc": 0, "results": ["7:device-mapper-1.02.158-2.el7.x86_64 providing device-mapper is already installed"]} 2019-10-13T00:47:28.868 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:28 +0000 (0:00:02.772) 0:07:42.127 ******** 2019-10-13T00:47:28.904 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T00:47:28.905 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'block_used': 13517949, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 226806535, u'size_available': 928999567360, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60972642, u'device': u'/dev/sda1', u'inode_used': 82334, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226806535, "block_size": 4096, "block_total": 240324484, "block_used": 13517949, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999567360, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}} 2019-10-13T00:47:28.943 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:28 +0000 (0:00:00.075) 0:07:42.202 ******** 2019-10-13T00:47:29.125 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-10-13T00:47:29.126 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005660", "end": "2019-10-13 00:47:29.098942", "rc": 0, "start": "2019-10-13 00:47:29.093282", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:29.161 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.217) 0:07:42.420 ******** 2019-10-13T00:47:29.199 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-13T00:47:29.199 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'block_used': 13517949, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 226806535, u'size_available': 928999567360, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60972642, u'device': u'/dev/sda1', u'inode_used': 82334, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226806535, "block_size": 4096, "block_total": 240324484, "block_used": 13517949, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999567360, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:29.238 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.076) 0:07:42.497 ******** 2019-10-13T00:47:29.264 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-13T00:47:29.301 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.062) 0:07:42.560 ******** 2019-10-13T00:47:29.306 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-13T00:47:29.307 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "msg": "", "rc": 0, "results": ["7:device-mapper-1.02.158-2.el7.x86_64 providing device-mapper is already installed"]} 2019-10-13T00:47:29.347 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.046) 0:07:42.606 ******** 2019-10-13T00:47:29.353 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T00:47:29.355 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ', u'wwn-0x5000c5009292426c'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFVQ', u'wwn': u'0x5000c5009292426c', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1', u'wwn-0x5000c5009292426c-part1'], u'uuids': [u'0f4a2fb5-54ef-4627-832a-0ac320d3e510']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-10-13T00:47:29.356 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:29.388 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T00:47:29.389 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'block_used': 13517956, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 226806528, u'size_available': 928999538688, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60972642, u'device': u'/dev/sda1', u'inode_used': 82334, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226806528, "block_size": 4096, "block_total": 240324484, "block_used": 13517956, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999538688, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}} 2019-10-13T00:47:29.430 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.082) 0:07:42.688 ******** 2019-10-13T00:47:29.603 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-10-13T00:47:29.603 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005615", "end": "2019-10-13 00:47:29.587673", "rc": 0, "start": "2019-10-13 00:47:29.582058", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:29.642 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.212) 0:07:42.901 ******** 2019-10-13T00:47:29.679 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-13T00:47:29.680 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'block_used': 13517956, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 226806528, u'size_available': 928999538688, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60972642, u'device': u'/dev/sda1', u'inode_used': 82334, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226806528, "block_size": 4096, "block_total": 240324484, "block_used": 13517956, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999538688, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:29.719 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.077) 0:07:42.978 ******** 2019-10-13T00:47:29.747 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-13T00:47:29.785 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:29 +0000 (0:00:00.065) 0:07:43.043 ******** 2019-10-13T00:47:29.823 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T00:47:29.824 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW', u'wwn-0x5000c5009294102d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XNRW', u'wwn': u'0x5000c5009294102d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1', u'wwn-0x5000c5009294102d-part1'], u'uuids': [u'0f4a2fb5-54ef-4627-832a-0ac320d3e510']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-10-13T00:47:29.826 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:30.553 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ', u'nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN', u'nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.049919", "end": "2019-10-13 00:47:30.524420", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:29.474501", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-13T00:47:30.555 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:31.121 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN', u'nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.138394", "end": "2019-10-13 00:47:31.100264", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:29.961870", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-13T00:47:31.123 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:31.784 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.041453", "end": "2019-10-13 00:47:31.757939", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:30.716486", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:32.377 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.074732", "end": "2019-10-13 00:47:32.350481", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:31.275749", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:32.995 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.043192", "end": "2019-10-13 00:47:32.973989", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:31.930797", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:33.597 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.046951", "end": "2019-10-13 00:47:33.566941", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:32.519990", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:34.225 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.059180", "end": "2019-10-13 00:47:34.200334", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:33.141154", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:34.814 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.044582", "end": "2019-10-13 00:47:34.790997", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:33.746415", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:35.435 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd'], u'uuids': [u'98c5ae01-6063-4cb3-bd64-7f2983a5b25c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.040369", "end": "2019-10-13 00:47:35.416012", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd"], "labels": [], "masters": [], "uuids": ["98c5ae01-6063-4cb3-bd64-7f2983a5b25c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:34.375643", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:36.044 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc'], u'uuids': [u'4ff5e40b-9654-459d-90d0-7e2a1501e847']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.041164", "end": "2019-10-13 00:47:36.007952", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc"], "labels": [], "masters": [], "uuids": ["4ff5e40b-9654-459d-90d0-7e2a1501e847"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:34.966788", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:36.661 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.043693", "end": "2019-10-13 00:47:36.632355", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:35.588662", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:36.708 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:36 +0000 (0:00:06.923) 0:07:49.967 ******** 2019-10-13T00:47:36.757 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T00:47:36.759 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ', u'wwn-0x5000c5009292426c'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFVQ', u'wwn': u'0x5000c5009292426c', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1', u'wwn-0x5000c5009292426c-part1'], u'uuids': [u'0f4a2fb5-54ef-4627-832a-0ac320d3e510']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-10-13T00:47:36.760 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:36.982 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ', u'nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN', u'nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039447", "end": "2019-10-13 00:47:36.951646", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TM93le-YJ4r-go26-Ltv9-bj53-7naE-bUqQTJ", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:36.912199", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) co 2019-10-13T00:47:36.983 INFO:teuthology.task.ansible.out:pied, 0.0365648 s, 3.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0365648 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:37.294 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.153789", "end": "2019-10-13 00:47:37.268868", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA2gXAsvtES7P6ZLFiKvYdP1KsNqfknsD"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:37.115079", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.151031 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.151031 s, 764 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:37.296 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.094967", "end": "2019-10-13 00:47:37.275163", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:36.180196", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T00:47:37.349 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:37 +0000 (0:00:00.640) 0:07:50.608 ******** 2019-10-13T00:47:37.397 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T00:47:37.398 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW', u'wwn-0x5000c5009294102d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XNRW', u'wwn': u'0x5000c5009294102d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1', u'wwn-0x5000c5009294102d-part1'], u'uuids': [u'0f4a2fb5-54ef-4627-832a-0ac320d3e510']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["0f4a2fb5-54ef-4627-832a-0ac320d3e510"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-10-13T00:47:37.399 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:37.597 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.144840", "end": "2019-10-13 00:47:37.574602", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1r86HmRanrEKuyepfPTzcU6zQ4VoJAKMT9"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:37.429762", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.142106 s, 812 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.142106 s, 812 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:37.611 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN', u'nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.038619", "end": "2019-10-13 00:47:37.583291", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-z6Cf3n-BS9d-e4Pb-Prqr-Jivt-eeVf-yD7JdK", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:37.544672", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) co 2019-10-13T00:47:37.614 INFO:teuthology.task.ansible.out:pied, 0.0357374 s, 3.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0357374 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:37.903 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.142928", "end": "2019-10-13 00:47:37.878324", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rA25bZE0zOIcUhxEjZZwae9PhS21Hyfsg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:37.735396", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.140199 s, 823 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.140199 s, 823 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:37.926 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.142734", "end": "2019-10-13 00:47:37.905418", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPsfTsMR02JAe3HxJDG3BZjtQhskfHemNr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:37.762684", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.139956 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.139956 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:38.203 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd'], u'uuids': [u'98c5ae01-6063-4cb3-bd64-7f2983a5b25c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144602", "end": "2019-10-13 00:47:38.182891", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1ronOP3Av2S1A0ytJLwDENSSe5y3Rn6BHd"], "labels": [], "masters": [], "uuids": ["98c5ae01-6063-4cb3-bd64-7f2983a5b25c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:38.038289", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.141901 s, 813 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.141901 s, 813 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:38.233 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.143527", "end": "2019-10-13 00:47:38.207468", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPJHJlf5WOWHOc1WYNxEKEXUAS4erwk0iY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:38.063941", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.140718 s, 820 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.140718 s, 820 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:38.505 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.142420", "end": "2019-10-13 00:47:38.482928", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rJiMgt0AT4ZxLqOEw0hCAkWol03RLI1rDd2N0pIXUNiopM8ehEeymIYvJLIS4NjL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:38.340508", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.139704 s, 826 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.139704 s, 826 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:38.526 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.142037", "end": "2019-10-13 00:47:38.507142", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPxZXN2sTMBMYKLTdozeioddzlTRCmf4vG"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:38.365105", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.139302 s, 828 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.139302 s, 828 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:38.565 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:38 +0000 (0:00:01.215) 0:07:51.824 ******** 2019-10-13T00:47:38.835 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc'], u'uuids': [u'4ff5e40b-9654-459d-90d0-7e2a1501e847']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.142673", "end": "2019-10-13 00:47:38.806524", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXP3MTYs2SB2KVquk8tndjeKMWpPmlo5Lvc"], "labels": [], "masters": [], "uuids": ["4ff5e40b-9654-459d-90d0-7e2a15 2019-10-13T00:47:38.836 INFO:teuthology.task.ansible.out:01e847"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:38.663851", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.139852 s, 825 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.139852 s, 825 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:38.848 INFO:teuthology.task.ansible.out: 2019-10-13T00:47:38.850 INFO:teuthology.task.ansible.out:TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.092147", "end": "2019-10-13 00:47:38.821494", "rc": 0, "start": "2019-10-13 00:47:38.729347", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:38.887 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:38 +0000 (0:00:00.321) 0:07:52.145 ******** 2019-10-13T00:47:39.121 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-10-13T00:47:39.123 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049995", "end": "2019-10-13 00:47:39.097110", "rc": 0, "start": "2019-10-13 00:47:39.047115", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:39.162 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.275) 0:07:52.421 ******** 2019-10-13T00:47:39.178 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T00:47:39.179 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.141668", "end": "2019-10-13 00:47:39.108264", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RUAGxY0ByR5t5Ja6SQIv8uN42Ls20yXPg0IspbZYOwM6rhuPILYl34Ob6OjWHPMu"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 00:47:38.966596", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.138963 s, 830 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.138963 s, 830 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:39.190 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-13T00:47:39.230 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.067) 0:07:52.489 ******** 2019-10-13T00:47:39.265 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.035) 0:07:52.524 ******** 2019-10-13T00:47:39.271 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:39.313 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.047) 0:07:52.572 ******** 2019-10-13T00:47:39.339 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-13T00:47:39.341 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:39.377 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.063) 0:07:52.635 ******** 2019-10-13T00:47:39.403 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T00:47:39.405 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:39.447 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.070) 0:07:52.706 ******** 2019-10-13T00:47:39.473 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-13T00:47:39.475 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:39.518 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.066) 0:07:52.772 ******** 2019-10-13T00:47:39.531 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-10-13T00:47:39.532 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.085184", "end": "2019-10-13 00:47:39.499760", "rc": 0, "start": "2019-10-13 00:47:39.414576", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:39.574 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.060) 0:07:52.833 ******** 2019-10-13T00:47:39.691 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-13T00:47:39.692 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:47:39.733 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.158) 0:07:52.991 ******** 2019-10-13T00:47:39.770 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T00:47:39.771 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'block_used': 13517949, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 226806535, u'size_available': 928999567360, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60972642, u'device': u'/dev/sda1', u'inode_used': 82334, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226806535, "block_size": 4096, "block_total": 240324484, "block_used": 13517949, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999567360, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:39.824 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.090) 0:07:53.082 ******** 2019-10-13T00:47:39.830 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-10-13T00:47:39.831 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.039770", "end": "2019-10-13 00:47:39.770661", "rc": 0, "start": "2019-10-13 00:47:39.730891", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:39.869 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.045) 0:07:53.128 ******** 2019-10-13T00:47:39.877 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-10-13T00:47:39.878 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:39.914 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.045) 0:07:53.173 ******** 2019-10-13T00:47:39.920 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-13T00:47:39.966 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:39 +0000 (0:00:00.051) 0:07:53.225 ******** 2019-10-13T00:47:39.971 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:47:39.971 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.009 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.043) 0:07:53.268 ******** 2019-10-13T00:47:40.014 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-13T00:47:40.015 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.055 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.045) 0:07:53.314 ******** 2019-10-13T00:47:40.060 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:47:40.061 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.098 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.043) 0:07:53.357 ******** 2019-10-13T00:47:40.103 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-13T00:47:40.104 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.141 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.042) 0:07:53.400 ******** 2019-10-13T00:47:40.146 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-13T00:47:40.146 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.184 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.042) 0:07:53.443 ******** 2019-10-13T00:47:40.191 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T00:47:40.192 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.243 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.052) 0:07:53.495 ******** TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.279 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.042) 0:07:53.538 ******** 2019-10-13T00:47:40.283 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-13T00:47:40.286 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.321 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.042) 0:07:53.580 ******** 2019-10-13T00:47:40.326 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-13T00:47:40.326 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.364 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.042) 0:07:53.623 ******** 2019-10-13T00:47:40.493 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-13T00:47:40.495 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T00:47:40.530 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.165) 0:07:53.789 ******** 2019-10-13T00:47:40.578 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi155.front.sepia.ceph.com] => (item={u'block_used': 13517956, u'uuid': u'0f4a2fb5-54ef-4627-832a-0ac320d3e510', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 226806528, u'size_available': 928999538688, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60972642, u'device': u'/dev/sda1', u'inode_used': 82334, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226806528, "block_size": 4096, "block_total": 240324484, "block_used": 13517956, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60972642, "inode_total": 61054976, "inode_used": 82334, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928999538688, "size_total": 984369086464, "uuid": "0f4a2fb5-54ef-4627-832a-0ac320d3e510"}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.616 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.086) 0:07:53.875 ******** 2019-10-13T00:47:40.663 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-10-13T00:47:40.664 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.699 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.083) 0:07:53.958 ******** 2019-10-13T00:47:40.736 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:47:40.744 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.779 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.079) 0:07:54.038 ******** 2019-10-13T00:47:40.805 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:47:40.807 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.845 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.065) 0:07:54.104 ******** 2019-10-13T00:47:40.872 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-13T00:47:40.873 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:40.909 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.063) 0:07:54.168 ******** 2019-10-13T00:47:40.947 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Create volume group(s)] *************************************** 2019-10-13T00:47:40.949 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} [WARNING]: The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change. 2019-10-13T00:47:40.986 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:40 +0000 (0:00:00.076) 0:07:54.245 ******** 2019-10-13T00:47:41.021 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:41 +0000 (0:00:00.033) 0:07:54.278 ******** 2019-10-13T00:47:41.047 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-13T00:47:41.048 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:41.084 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:41 +0000 (0:00:00.064) 0:07:54.342 ******** 2019-10-13T00:47:41.448 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-13T00:47:41.450 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} 2019-10-13T00:47:41.487 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:41 +0000 (0:00:00.403) 0:07:54.746 ******** 2019-10-13T00:47:41.556 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:41.791 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:41.823 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.057 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.103 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.333 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.375 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.608 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.656 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.695 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:42 +0000 (0:00:01.207) 0:07:55.954 ******** 2019-10-13T00:47:42.869 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T00:47:42.871 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-13T00:47:42.907 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:42 +0000 (0:00:00.210) 0:07:56.165 ******** 2019-10-13T00:47:42.911 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-13T00:47:42.912 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T00:47:42.955 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:42 +0000 (0:00:00.048) 0:07:56.213 ******** 2019-10-13T00:47:42.961 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** skipping: [smithi142.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:43.120 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.122 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-13T00:47:43.160 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:43 +0000 (0:00:00.205) 0:07:56.419 ******** 2019-10-13T00:47:43.197 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T00:47:43.199 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:43.279 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.368 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.453 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.527 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.632 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.691 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:43 +0000 (0:00:00.530) 0:07:56.950 ******** 2019-10-13T00:47:43.699 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.847 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T00:47:43.893 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:43 +0000 (0:00:00.202) 0:07:57.152 ******** 2019-10-13T00:47:43.902 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T00:47:43.941 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:43 +0000 (0:00:00.047) 0:07:57.199 ******** 2019-10-13T00:47:43.966 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-10-13T00:47:43.968 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:44.010 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:44 +0000 (0:00:00.068) 0:07:57.268 ******** 2019-10-13T00:47:44.069 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-10-13T00:47:44.071 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T00:47:44.134 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:44 +0000 (0:00:00.124) 0:07:57.392 ******** 2019-10-13T00:47:44.172 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:44.213 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:44 +0000 (0:00:00.079) 0:07:57.472 ******** 2019-10-13T00:47:44.553 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-10-13T00:47:44.555 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true} changed: [smithi142.front.sepia.ceph.com] => {"changed": true} 2019-10-13T00:47:44.595 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:44 +0000 (0:00:00.382) 0:07:57.854 ******** 2019-10-13T00:47:44.633 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:44 +0000 (0:00:00.038) 0:07:57.892 ******** 2019-10-13T00:47:45.111 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-10-13T00:47:45.113 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-10-13T00:47:45.152 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.518) 0:07:58.411 ******** 2019-10-13T00:47:45.189 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.037) 0:07:58.448 ******** 2019-10-13T00:47:45.194 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-10-13T00:47:45.196 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.232 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.043) 0:07:58.491 ******** 2019-10-13T00:47:45.237 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.276 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.044) 0:07:58.535 ******** 2019-10-13T00:47:45.281 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-13T00:47:45.282 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.319 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.043) 0:07:58.578 ******** 2019-10-13T00:47:45.325 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.363 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.044) 0:07:58.622 ******** 2019-10-13T00:47:45.377 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-10-13T00:47:45.379 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.412 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.048) 0:07:58.671 ******** 2019-10-13T00:47:45.418 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.456 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.044) 0:07:58.715 ******** 2019-10-13T00:47:45.461 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-13T00:47:45.462 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.500 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.043) 0:07:58.759 ******** 2019-10-13T00:47:45.506 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.547 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.046) 0:07:58.806 ******** 2019-10-13T00:47:45.552 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-13T00:47:45.553 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.589 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.041) 0:07:58.847 ******** 2019-10-13T00:47:45.596 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.631 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.041) 0:07:58.889 ******** 2019-10-13T00:47:45.635 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-13T00:47:45.636 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.672 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.041) 0:07:58.931 ******** 2019-10-13T00:47:45.677 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.715 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.042) 0:07:58.973 ******** 2019-10-13T00:47:45.719 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-13T00:47:45.720 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.756 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.041) 0:07:59.015 ******** 2019-10-13T00:47:45.760 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:45.797 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:45 +0000 (0:00:00.041) 0:07:59.056 ******** 2019-10-13T00:47:47.316 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-13T00:47:47.317 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7.x86_64 providing ntp is already installed"]} 2019-10-13T00:47:47.356 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:47 +0000 (0:00:01.559) 0:08:00.615 ******** 2019-10-13T00:47:47.363 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7.x86_64 providing ntp is already installed"]} 2019-10-13T00:47:47.403 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:47 +0000 (0:00:00.046) 0:08:00.662 ******** 2019-10-13T00:47:47.410 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:47.449 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:47 +0000 (0:00:00.045) 0:08:00.707 ******** 2019-10-13T00:47:47.454 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:47.493 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:47 +0000 (0:00:00.044) 0:08:00.752 ******** 2019-10-13T00:47:47.497 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-10-13T00:47:47.498 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:47.536 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:47 +0000 (0:00:00.043) 0:08:00.795 ******** 2019-10-13T00:47:47.549 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:47.582 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:47 +0000 (0:00:00.046) 0:08:00.841 ******** 2019-10-13T00:47:48.047 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-10-13T00:47:48.050 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-10-13T00:47:48.089 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:48 +0000 (0:00:00.506) 0:08:01.347 ******** 2019-10-13T00:47:48.100 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-10-13T00:47:48.139 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:48 +0000 (0:00:00.050) 0:08:01.398 ******** 2019-10-13T00:47:48.144 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-10-13T00:47:48.146 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:48.183 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:48 +0000 (0:00:00.044) 0:08:01.442 ******** 2019-10-13T00:47:48.188 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:48.228 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:48 +0000 (0:00:00.045) 0:08:01.487 ******** 2019-10-13T00:47:48.472 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-13T00:47:48.474 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice tmp.mount ntpdate.service -.mount systemd-journald.socket basic.target sntp.service syslog.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "multi-user.target shutdown.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-10-13T00:47:48.475 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-10-13T00:47:48.477 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-13T00:47:48.513 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:48 +0000 (0:00:00.284) 0:08:01.772 ******** 2019-10-13T00:47:48.522 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "tmp.mount syslog.target system.slice ntpdate.service systemd-journald.socket sntp.service basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "chronyd.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-10-13T00:47:48.524 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-10-13T00:47:48.525 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-13T00:47:48.563 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:48 +0000 (0:00:00.049) 0:08:01.822 ******** 2019-10-13T00:47:51.348 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-13T00:47:51.350 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-10-13T00:47:51.384 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:51 +0000 (0:00:02.821) 0:08:04.643 ******** 2019-10-13T00:47:51.400 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-10-13T00:47:51.440 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:51 +0000 (0:00:00.054) 0:08:04.697 ******** 2019-10-13T00:47:51.444 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-13T00:47:51.445 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:51.485 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:51 +0000 (0:00:00.046) 0:08:04.744 ******** 2019-10-13T00:47:51.491 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:51.530 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:51 +0000 (0:00:00.045) 0:08:04.789 ******** 2019-10-13T00:47:51.665 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-13T00:47:51.667 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T00:47:51.703 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:51 +0000 (0:00:00.172) 0:08:04.962 ******** 2019-10-13T00:47:51.720 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T00:47:51.761 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:51 +0000 (0:00:00.058) 0:08:05.020 ******** 2019-10-13T00:47:52.233 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-10-13T00:47:52.234 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-10-13T00:47:52.279 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:52 +0000 (0:00:00.518) 0:08:05.538 ******** 2019-10-13T00:47:52.287 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-10-13T00:47:52.300 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:52.327 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:52 +0000 (0:00:00.048) 0:08:05.586 ******** 2019-10-13T00:47:52.349 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:53.468 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-10-13T00:47:53.469 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.2.3 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.2.3 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-13T00:47:53.517 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:01.189) 0:08:06.776 ******** 2019-10-13T00:47:53.525 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.2.3 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.2.3 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-13T00:47:53.566 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.049) 0:08:06.825 ******** 2019-10-13T00:47:53.571 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-10-13T00:47:53.572 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:53.610 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.043) 0:08:06.869 ******** 2019-10-13T00:47:53.616 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:53.659 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.048) 0:08:06.917 ******** 2019-10-13T00:47:53.665 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-13T00:47:53.667 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-13T00:47:53.702 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.043) 0:08:06.961 ******** 2019-10-13T00:47:53.709 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-13T00:47:53.748 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.044) 0:08:07.005 ******** 2019-10-13T00:47:53.877 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-10-13T00:47:53.878 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-10-13T00:47:53.912 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.166) 0:08:07.171 ******** 2019-10-13T00:47:53.920 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-10-13T00:47:53.957 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.044) 0:08:07.216 ******** 2019-10-13T00:47:53.962 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-13T00:47:53.963 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.000 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:53 +0000 (0:00:00.042) 0:08:07.258 ******** 2019-10-13T00:47:54.006 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.015 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:54.043 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.043) 0:08:07.302 ******** 2019-10-13T00:47:54.048 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-13T00:47:54.049 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-13T00:47:54.059 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T00:47:54.086 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.043) 0:08:07.345 ******** 2019-10-13T00:47:54.095 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-13T00:47:54.132 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.045) 0:08:07.391 ******** 2019-10-13T00:47:54.141 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.174 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.041) 0:08:07.433 ******** 2019-10-13T00:47:54.180 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.217 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.042) 0:08:07.476 ******** 2019-10-13T00:47:54.347 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-10-13T00:47:54.347 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-13T00:47:54.384 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.166) 0:08:07.642 ******** 2019-10-13T00:47:54.400 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-13T00:47:54.437 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.053) 0:08:07.696 ******** 2019-10-13T00:47:54.567 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-10-13T00:47:54.569 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-13T00:47:54.605 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.167) 0:08:07.863 ******** 2019-10-13T00:47:54.611 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-13T00:47:54.649 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.044) 0:08:07.908 ******** 2019-10-13T00:47:54.654 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-10-13T00:47:54.656 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.691 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.042) 0:08:07.950 ******** 2019-10-13T00:47:54.697 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.753 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.061) 0:08:08.012 ******** 2019-10-13T00:47:54.759 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-10-13T00:47:54.760 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.761 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.763 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.805 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.052) 0:08:08.064 ******** 2019-10-13T00:47:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.813 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.815 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.854 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.048) 0:08:08.113 ******** 2019-10-13T00:47:54.864 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.869 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.875 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.923 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.068) 0:08:08.181 ******** 2019-10-13T00:47:54.929 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.930 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.931 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.933 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:54.972 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:54 +0000 (0:00:00.047) 0:08:08.229 ******** 2019-10-13T00:47:54.976 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T00:47:54.977 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.015 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.044) 0:08:08.274 ******** 2019-10-13T00:47:55.021 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.066 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.051) 0:08:08.325 ******** 2019-10-13T00:47:55.071 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-10-13T00:47:55.071 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.108 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.042) 0:08:08.367 ******** 2019-10-13T00:47:55.114 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.152 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.043) 0:08:08.411 ******** 2019-10-13T00:47:55.164 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.199 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.046) 0:08:08.458 ******** 2019-10-13T00:47:55.205 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.244 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.045) 0:08:08.503 ******** 2019-10-13T00:47:55.249 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-13T00:47:55.250 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.287 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.042) 0:08:08.546 ******** 2019-10-13T00:47:55.293 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.330 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.043) 0:08:08.589 ******** 2019-10-13T00:47:55.335 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-13T00:47:55.336 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.382 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.045) 0:08:08.635 ******** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.420 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.044) 0:08:08.679 ******** 2019-10-13T00:47:55.426 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T00:47:55.426 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.466 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.045) 0:08:08.724 ******** 2019-10-13T00:47:55.472 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.512 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.045) 0:08:08.770 ******** 2019-10-13T00:47:55.517 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.560 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.044) 0:08:08.815 ******** 2019-10-13T00:47:55.563 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.603 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.046) 0:08:08.862 ******** 2019-10-13T00:47:55.608 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T00:47:55.609 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.646 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.043) 0:08:08.905 ******** 2019-10-13T00:47:55.652 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T00:47:55.690 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:55 +0000 (0:00:00.043) 0:08:08.948 ******** 2019-10-13T00:47:55.849 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T00:47:55.849 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.006652", "end": "2019-10-13 00:47:55.816054", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-13 00:47:55.809402", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:55.892 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.006590", "end": "2019-10-13 00:47:55.860836", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-13 00:47:55.854246", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:56.007 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003267", "end": "2019-10-13 00:47:55.983207", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-13 00:47:55.979940", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:56.060 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003193", "end": "2019-10-13 00:47:56.030226", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-13 00:47:56.027033", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:56.165 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003002", "end": "2019-10-13 00:47:56.141451", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 00:47:56.138449", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:56.238 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003062", "end": "2019-10-13 00:47:56.189641", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 00:47:56.186579", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:56.334 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015773", "end": "2019-10-13 00:47:56.312301", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-13 00:47:56.296528", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:56.381 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:56 +0000 (0:00:00.691) 0:08:09.640 ******** 2019-10-13T00:47:56.408 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-13T00:47:56.449 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:56 +0000 (0:00:00.068) 0:08:09.708 ******** 2019-10-13T00:47:56.455 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T00:47:56.456 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.019551", "end": "2019-10-13 00:47:56.388281", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-13 00:47:56.368730", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T00:47:56.502 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:56 +0000 (0:00:00.052) 0:08:09.761 ******** 2019-10-13T00:47:56.531 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-13T00:47:56.568 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 00:47:56 +0000 (0:00:00.065) 0:08:09.827 ******** 2019-10-13T00:47:56.619 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-13T00:47:56.621 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-13T00:47:56.736 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-13T00:47:56.829 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2019-10-13T00:47:56.831 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2019-10-13T00:47:56.833 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2019-10-13T00:47:56.835 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi142.front.sepia.ceph.com : ok=128 changed=34 unreachable=0 failed=0 skipped=148 rescued=0 ignored=4 smithi155.front.sepia.ceph.com : ok=128 changed=34 unreachable=0 failed=0 skipped=148 rescued=0 ignored=4 Sunday 13 October 2019 00:47:56 +0000 (0:00:00.265) 0:08:10.093 ******** =============================================================================== 2019-10-13T00:47:56.836 INFO:teuthology.task.ansible.out:testnode : Install packages ------------------------------------------- 153.83s testnode : Ensure ceph packages are not present. ----------------------- 35.73s common : Make sure smartmontools is installed -------------------------- 29.67s common : Install nrpe package and dependencies (RHEL/CentOS) ----------- 29.27s testnode : Ensure ceph dependency packages are not present. ------------ 25.47s testnode : Install epel packages --------------------------------------- 21.77s users : Create all admin users with sudo access. ----------------------- 19.95s users : Update authorized_keys using the keys repo --------------------- 19.78s common : Enable necessary rhsm repos ----------------------------------- 17.88s common : nrpe - Load SELinux policy package ----------------------------- 7.16s testnode : Zap all non-root disks --------------------------------------- 6.92s common : nrpe - Remove SELinux policy package --------------------------- 6.87s common : Register with subscription-manager. ---------------------------- 6.71s common : nrpe - Ensure SELinux policy is up to date --------------------- 6.07s users : Remove revoked users -------------------------------------------- 4.66s common : Get list of enabled RHSM repos --------------------------------- 3.45s common : Disable all rhsm repos ----------------------------------------- 2.99s testnode : Install python-pip on rpm based systems. --------------------- 2.82s testnode : Make sure rpm dependencies are installed --------------------- 2.77s common : nrpe - Install SELinux dependencies ---------------------------- 2.55s 2019-10-13T00:47:57.109 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2019-10-13T00:47:57.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T00:47:57.113 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:47:57.199 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:47:57.199 INFO:teuthology.orchestra.run.smithi155:> true 2019-10-13T00:47:57.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T00:47:57.466 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T00:47:57.551 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:47:57.551 INFO:teuthology.orchestra.run.smithi142:> true 2019-10-13T00:47:57.829 INFO:teuthology.run_tasks:Running task clock... 2019-10-13T00:47:57.842 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-10-13T00:47:57.843 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:47:57.843 INFO:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-13T00:47:57.905 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-10-13T00:48:07.953 INFO:teuthology.orchestra.run.smithi155.stdout:ntpd: time slew -0.002284s 2019-10-13T00:48:07.971 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit not found. 2019-10-13T00:48:08.016 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2019-10-13T00:48:08.016 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2019-10-13T00:48:08.016 INFO:teuthology.orchestra.run.smithi155.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:08.016 INFO:teuthology.orchestra.run.smithi155.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:08.016 INFO:teuthology.orchestra.run.smithi155.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:08.017 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:08.017 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:08.017 INFO:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-13T00:48:08.092 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-10-13T00:48:18.147 INFO:teuthology.orchestra.run.smithi142.stdout:ntpd: time slew -0.001354s 2019-10-13T00:48:18.164 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit not found. 2019-10-13T00:48:18.221 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2019-10-13T00:48:18.221 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2019-10-13T00:48:18.222 INFO:teuthology.orchestra.run.smithi142.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:18.222 INFO:teuthology.orchestra.run.smithi142.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:18.222 INFO:teuthology.orchestra.run.smithi142.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:18.222 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T00:48:18.222 INFO:teuthology.run_tasks:Running task install... 2019-10-13T00:48:18.268 DEBUG:teuthology.task.install:project ceph 2019-10-13T00:48:18.269 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}} 2019-10-13T00:48:18.269 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'} 2019-10-13T00:48:18.269 INFO:teuthology.task.install:Using flavor: basic 2019-10-13T00:48:18.274 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-10-13T00:48:18.274 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2019-10-13T00:48:18.274 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:18.274 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2019-10-13T00:48:18.362 INFO:teuthology.orchestra.run.smithi155.stdout:RedHatEnterpriseServer 2019-10-13T00:48:18.363 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2019-10-13T00:48:18.363 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:18.364 INFO:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2019-10-13T00:48:18.364 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'flavor': 'basic', 'extras': None} 2019-10-13T00:48:18.364 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:18.365 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2019-10-13T00:48:18.433 INFO:teuthology.orchestra.run.smithi155.stdout:RedHatEnterpriseServer 2019-10-13T00:48:18.434 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2019-10-13T00:48:18.435 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T00:48:18.446 INFO:teuthology.orchestra.run.smithi142.stdout:RedHatEnterpriseServer 2019-10-13T00:48:18.447 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2019-10-13T00:48:18.447 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'flavor': 'basic', 'extras': None} 2019-10-13T00:48:18.447 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:18.448 INFO:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2019-10-13T00:48:18.513 INFO:teuthology.orchestra.run.smithi142.stdout:RedHatEnterpriseServer 2019-10-13T00:48:18.515 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2019-10-13T00:48:18.515 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T00:48:18.611 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/centos/7/flavors/default/ 2019-10-13T00:48:18.612 INFO:teuthology.task.install.rpm:Package version is 15.0.0-6132.ga8ee8ac 2019-10-13T00:48:18.684 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/centos/7/flavors/default/ 2019-10-13T00:48:18.685 INFO:teuthology.task.install.rpm:Package version is 15.0.0-6132.ga8ee8ac 2019-10-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:18.738 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-10-13T00:48:18.793 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2019-10-13T00:48:18.793 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:18.793 INFO:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-10-13T00:48:18.804 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:18.804 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-10-13T00:48:18.858 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2019-10-13T00:48:18.859 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:18.859 INFO:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-10-13T00:48:18.914 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:18.914 INFO:teuthology.orchestra.run.smithi155:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-10-13T00:48:18.968 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:18.968 INFO:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-10-13T00:48:18.976 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:18.976 INFO:teuthology.orchestra.run.smithi142:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-10-13T00:48:19.024 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2019-10-13T00:48:19.025 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:19.025 INFO:teuthology.orchestra.run.smithi155:> sudo yum clean all 2019-10-13T00:48:19.029 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:19.029 INFO:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-10-13T00:48:19.081 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2019-10-13T00:48:19.082 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:19.083 INFO:teuthology.orchestra.run.smithi142:> sudo yum clean all 2019-10-13T00:48:19.217 INFO:teuthology.orchestra.run.smithi155.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:48:19.217 INFO:teuthology.orchestra.run.smithi155.stdout: : disabled-repos, subscription-manager 2019-10-13T00:48:19.275 INFO:teuthology.orchestra.run.smithi142.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:48:19.276 INFO:teuthology.orchestra.run.smithi142.stdout: : disabled-repos, subscription-manager 2019-10-13T00:48:19.681 INFO:teuthology.orchestra.run.smithi155.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2019-10-13T00:48:19.681 INFO:teuthology.orchestra.run.smithi155.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2019-10-13T00:48:19.681 INFO:teuthology.orchestra.run.smithi155.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2019-10-13T00:48:19.700 INFO:teuthology.orchestra.run.smithi142.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2019-10-13T00:48:19.700 INFO:teuthology.orchestra.run.smithi142.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2019-10-13T00:48:19.700 INFO:teuthology.orchestra.run.smithi142.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2019-10-13T00:48:19.834 INFO:teuthology.orchestra.run.smithi142.stdout:Cleaning up list of fastest mirrors 2019-10-13T00:48:19.840 INFO:teuthology.orchestra.run.smithi155.stdout:Cleaning up list of fastest mirrors 2019-10-13T00:48:19.858 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:48:19.859 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-daemon ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2019-10-13T00:48:19.860 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:48:19.860 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-daemon ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2019-10-13T00:48:20.057 INFO:teuthology.orchestra.run.smithi155.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:48:20.057 INFO:teuthology.orchestra.run.smithi155.stdout: : disabled-repos, subscription-manager 2019-10-13T00:48:20.060 INFO:teuthology.orchestra.run.smithi142.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2019-10-13T00:48:20.060 INFO:teuthology.orchestra.run.smithi142.stdout: : disabled-repos, subscription-manager 2019-10-13T00:48:20.490 INFO:teuthology.orchestra.run.smithi155.stdout:Determining fastest mirrors 2019-10-13T00:48:20.510 INFO:teuthology.orchestra.run.smithi142.stdout:Determining fastest mirrors 2019-10-13T00:48:20.567 INFO:teuthology.orchestra.run.smithi155.stdout: * epel: ftp.linux.ncsu.edu 2019-10-13T00:48:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: * epel: ftp.linux.ncsu.edu 2019-10-13T00:48:48.498 INFO:teuthology.orchestra.run.smithi142.stdout:29 packages excluded due to repository priority protections 2019-10-13T00:48:48.890 INFO:teuthology.orchestra.run.smithi155.stdout:29 packages excluded due to repository priority protections 2019-10-13T00:49:11.868 INFO:teuthology.orchestra.run.smithi142.stdout:Resolving Dependencies 2019-10-13T00:49:11.868 INFO:teuthology.orchestra.run.smithi142.stdout:--> Running transaction check 2019-10-13T00:49:11.869 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2019-10-13T00:49:11.877 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:11.878 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:11.879 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:11.880 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:11.881 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:11.882 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: podman for package: 2:ceph-daemon-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:11.990 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:11.998 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python36 for package: 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.001 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.002 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.004 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.006 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.007 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.008 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.009 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.010 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.011 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.015 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.015 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.016 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.016 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.016 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.017 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.017 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.018 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.018 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.019 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.022 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.024 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.024 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.026 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.027 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-test.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.031 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.032 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.033 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.035 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.036 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-2.el7 will be installed 2019-10-13T00:49:12.037 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.el7.x86_64 2019-10-13T00:49:12.042 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2019-10-13T00:49:12.043 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.043 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:libcephfs-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.044 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libcephfs2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.045 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-10-13T00:49:12.046 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package librados2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be an update 2019-10-13T00:49:12.047 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.048 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-10-13T00:49:12.048 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package librbd1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be an update 2019-10-13T00:49:12.049 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed 2019-10-13T00:49:12.049 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64 2019-10-13T00:49:12.056 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.057 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.057 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.058 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-rados = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.059 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.060 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python36-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.061 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python36-ceph-argparse = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.062 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python36-rados = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.063 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.065 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.066 INFO:teuthology.orchestra.run.smithi142.stdout:--> Running transaction check 2019-10-13T00:49:12.066 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-base.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.072 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.074 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.075 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.076 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.077 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.078 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.078 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.078 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mds.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.079 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-mon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.079 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-osd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.082 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.085 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-10-13T00:49:12.085 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-10-13T00:49:12.086 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package krb5-devel.x86_64 0:1.15.1-37.el7_7.2 will be installed 2019-10-13T00:49:12.089 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.091 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.098 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.107 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.110 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-10-13T00:49:12.110 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-10-13T00:49:12.110 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package librados-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.111 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.111 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package librgw2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.112 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-10-13T00:49:12.113 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-13T00:49:12.114 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-13T00:49:12.115 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package podman.x86_64 0:1.4.4-4.el7 will be installed 2019-10-13T00:49:12.118 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: skopeo-containers >= 0.1.29-3 for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.127 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: containernetworking-plugins >= 0.8.1-1 for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.129 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: runc for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.136 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: container-selinux for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.150 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.150 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-10-13T00:49:12.164 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-10-13T00:49:12.164 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.165 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.166 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.167 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.168 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.169 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.170 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-rbd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.170 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-10-13T00:49:12.170 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-10-13T00:49:12.171 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-requests.noarch 0:2.6.0-5.el7 will be installed 2019-10-13T00:49:12.172 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-5.el7.noarch 2019-10-13T00:49:12.174 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-rgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.174 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-10-13T00:49:12.175 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-10-13T00:49:12.175 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-10-13T00:49:12.176 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-10-13T00:49:12.176 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-10-13T00:49:12.177 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-kubernetes.noarch 0:8.0.1-1.el7 will be installed 2019-10-13T00:49:12.178 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.178 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.179 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.180 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.185 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-10-13T00:49:12.185 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-10-13T00:49:12.185 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-10-13T00:49:12.186 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-10-13T00:49:12.191 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python36-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.191 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python36-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.191 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-10-13T00:49:12.194 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-10-13T00:49:12.196 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-13T00:49:12.198 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-13T00:49:12.198 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-10-13T00:49:12.199 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed 2019-10-13T00:49:12.199 INFO:teuthology.orchestra.run.smithi142.stdout:--> Running transaction check 2019-10-13T00:49:12.200 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package container-selinux.noarch 2:2.107-3.el7 will be installed 2019-10-13T00:49:12.202 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package containernetworking-plugins.x86_64 0:0.8.1-2.el7 will be installed 2019-10-13T00:49:12.202 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package containers-common.x86_64 1:0.1.37-3.el7 will be installed 2019-10-13T00:49:12.202 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2019-10-13T00:49:12.202 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-10-13T00:49:12.203 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-16.el7 will be installed 2019-10-13T00:49:12.203 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libselinux-devel.x86_64 0:2.5-14.1.el7 will be installed 2019-10-13T00:49:12.204 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-10-13T00:49:12.206 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-10-13T00:49:12.206 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-10-13T00:49:12.212 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2019-10-13T00:49:12.212 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-10-13T00:49:12.214 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-10-13T00:49:12.214 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.214 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.214 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-10-13T00:49:12.215 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-10-13T00:49:12.215 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-10-13T00:49:12.215 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-10-13T00:49:12.216 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-10-13T00:49:12.216 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-10-13T00:49:12.217 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-10-13T00:49:12.217 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-10-13T00:49:12.217 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-10-13T00:49:12.218 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-10-13T00:49:12.218 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-10-13T00:49:12.218 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-10-13T00:49:12.218 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-10-13T00:49:12.218 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-10-13T00:49:12.219 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-13T00:49:12.219 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-13T00:49:12.220 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-13T00:49:12.221 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-10-13T00:49:12.222 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-10-13T00:49:12.223 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-10-13T00:49:12.223 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-10-13T00:49:12.223 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-10-13T00:49:12.223 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package runc.x86_64 0:1.0.0-65.rc8.el7 will be installed 2019-10-13T00:49:12.224 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: criu for package: runc-1.0.0-65.rc8.el7.x86_64 2019-10-13T00:49:12.226 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-10-13T00:49:12.227 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-13T00:49:12.229 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-13T00:49:12.229 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-10-13T00:49:12.229 INFO:teuthology.orchestra.run.smithi142.stdout:--> Running transaction check 2019-10-13T00:49:12.230 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package criu.x86_64 0:3.12-2.el7 will be installed 2019-10-13T00:49:12.230 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.12-2.el7.x86_64 2019-10-13T00:49:12.234 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.12-2.el7.x86_64 2019-10-13T00:49:12.234 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.12-2.el7.x86_64 2019-10-13T00:49:12.235 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2019-10-13T00:49:12.235 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2019-10-13T00:49:12.236 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-10-13T00:49:12.237 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-10-13T00:49:12.237 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-10-13T00:49:12.238 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-10-13T00:49:12.238 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-10-13T00:49:12.238 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-10-13T00:49:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-10-13T00:49:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-10-13T00:49:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-10-13T00:49:12.241 INFO:teuthology.orchestra.run.smithi142.stdout:--> Running transaction check 2019-10-13T00:49:12.241 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed 2019-10-13T00:49:12.241 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed 2019-10-13T00:49:12.242 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-10-13T00:49:12.242 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-10-13T00:49:12.242 INFO:teuthology.orchestra.run.smithi142.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-10-13T00:49:12.242 INFO:teuthology.orchestra.run.smithi142.stdout:--> Running transaction check 2019-10-13T00:49:12.243 INFO:teuthology.orchestra.run.smithi142.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-10-13T00:49:12.514 INFO:teuthology.orchestra.run.smithi155.stdout:Resolving Dependencies 2019-10-13T00:49:12.515 INFO:teuthology.orchestra.run.smithi155.stdout:--> Running transaction check 2019-10-13T00:49:12.515 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2019-10-13T00:49:12.523 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.525 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.526 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.527 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.528 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.529 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: podman for package: 2:ceph-daemon-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.637 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.644 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python36 for package: 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.647 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.648 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.651 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.652 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.654 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.655 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.656 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.657 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.658 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.659 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.660 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.661 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.661 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.662 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.663 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.664 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.665 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.665 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-6132.ga8ee8ac.el7.noarch 2019-10-13T00:49:12.666 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.670 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.671 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.672 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.674 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.674 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-test.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.678 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.680 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.681 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.682 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.685 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-2.el7 will be installed 2019-10-13T00:49:12.685 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.el7.x86_64 2019-10-13T00:49:12.691 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2019-10-13T00:49:12.691 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.691 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:libcephfs-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.692 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libcephfs2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.692 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-10-13T00:49:12.693 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package librados2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be an update 2019-10-13T00:49:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.695 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-10-13T00:49:12.696 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package librbd1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be an update 2019-10-13T00:49:12.696 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed 2019-10-13T00:49:12.697 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64 2019-10-13T00:49:12.703 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.704 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.705 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.706 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-rados = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.707 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.708 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python36-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.709 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python36-ceph-argparse = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.710 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python36-rados = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.711 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.713 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.713 INFO:teuthology.orchestra.run.smithi155.stdout:--> Running transaction check 2019-10-13T00:49:12.714 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-base.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.720 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.723 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.723 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-6132.ga8ee8ac.el7 for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.724 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.725 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.727 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 2019-10-13T00:49:12.727 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.727 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mds.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.728 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-mon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.728 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-osd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.730 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.733 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-10-13T00:49:12.733 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-10-13T00:49:12.734 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package krb5-devel.x86_64 0:1.15.1-37.el7_7.2 will be installed 2019-10-13T00:49:12.737 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.740 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.746 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.756 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-10-13T00:49:12.758 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-10-13T00:49:12.758 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-10-13T00:49:12.759 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package librados-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.759 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.760 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package librgw2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.763 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-10-13T00:49:12.764 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-13T00:49:12.764 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-13T00:49:12.764 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package podman.x86_64 0:1.4.4-4.el7 will be installed 2019-10-13T00:49:12.767 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: skopeo-containers >= 0.1.29-3 for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.776 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: containernetworking-plugins >= 0.8.1-1 for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.777 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: runc for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.785 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: container-selinux for package: podman-1.4.4-4.el7.x86_64 2019-10-13T00:49:12.797 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.798 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-10-13T00:49:12.809 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-10-13T00:49:12.810 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.812 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.812 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.814 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.815 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-10-13T00:49:12.815 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.816 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-rbd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.816 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-10-13T00:49:12.817 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-10-13T00:49:12.817 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-requests.noarch 0:2.6.0-5.el7 will be installed 2019-10-13T00:49:12.818 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-5.el7.noarch 2019-10-13T00:49:12.820 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-rgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.821 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-10-13T00:49:12.821 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-10-13T00:49:12.822 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-10-13T00:49:12.822 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-10-13T00:49:12.822 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-10-13T00:49:12.823 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-kubernetes.noarch 0:8.0.1-1.el7 will be installed 2019-10-13T00:49:12.824 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.825 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.826 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.827 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-13T00:49:12.831 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-10-13T00:49:12.831 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-10-13T00:49:12.832 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-10-13T00:49:12.833 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-10-13T00:49:12.838 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python36-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.838 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python36-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.838 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-10-13T00:49:12.841 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-10-13T00:49:12.843 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-13T00:49:12.845 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-13T00:49:12.845 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-10-13T00:49:12.846 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed 2019-10-13T00:49:12.846 INFO:teuthology.orchestra.run.smithi155.stdout:--> Running transaction check 2019-10-13T00:49:12.847 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package container-selinux.noarch 2:2.107-3.el7 will be installed 2019-10-13T00:49:12.848 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package containernetworking-plugins.x86_64 0:0.8.1-2.el7 will be installed 2019-10-13T00:49:12.849 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package containers-common.x86_64 1:0.1.37-3.el7 will be installed 2019-10-13T00:49:12.850 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2019-10-13T00:49:12.850 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-10-13T00:49:12.851 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-16.el7 will be installed 2019-10-13T00:49:12.851 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libselinux-devel.x86_64 0:2.5-14.1.el7 will be installed 2019-10-13T00:49:12.851 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-10-13T00:49:12.853 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-10-13T00:49:12.853 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-10-13T00:49:12.859 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2019-10-13T00:49:12.859 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-10-13T00:49:12.861 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-10-13T00:49:12.862 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.862 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 will be installed 2019-10-13T00:49:12.862 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-10-13T00:49:12.862 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-10-13T00:49:12.862 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-10-13T00:49:12.863 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-10-13T00:49:12.864 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-10-13T00:49:12.864 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-10-13T00:49:12.864 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-10-13T00:49:12.864 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-10-13T00:49:12.864 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-10-13T00:49:12.865 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-10-13T00:49:12.865 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-10-13T00:49:12.866 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-10-13T00:49:12.866 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-10-13T00:49:12.866 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-10-13T00:49:12.866 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-13T00:49:12.866 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-13T00:49:12.868 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-13T00:49:12.869 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-10-13T00:49:12.869 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-10-13T00:49:12.870 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-10-13T00:49:12.871 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-10-13T00:49:12.871 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-10-13T00:49:12.871 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package runc.x86_64 0:1.0.0-65.rc8.el7 will be installed 2019-10-13T00:49:12.871 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: criu for package: runc-1.0.0-65.rc8.el7.x86_64 2019-10-13T00:49:12.874 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-10-13T00:49:12.874 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-13T00:49:12.877 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-13T00:49:12.877 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-10-13T00:49:12.877 INFO:teuthology.orchestra.run.smithi155.stdout:--> Running transaction check 2019-10-13T00:49:12.877 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package criu.x86_64 0:3.12-2.el7 will be installed 2019-10-13T00:49:12.878 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.12-2.el7.x86_64 2019-10-13T00:49:12.880 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.12-2.el7.x86_64 2019-10-13T00:49:12.880 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.12-2.el7.x86_64 2019-10-13T00:49:12.883 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2019-10-13T00:49:12.883 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2019-10-13T00:49:12.884 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-10-13T00:49:12.884 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-10-13T00:49:12.885 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-10-13T00:49:12.885 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-10-13T00:49:12.886 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-10-13T00:49:12.886 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-10-13T00:49:12.887 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-10-13T00:49:12.889 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-10-13T00:49:12.889 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-10-13T00:49:12.889 INFO:teuthology.orchestra.run.smithi155.stdout:--> Running transaction check 2019-10-13T00:49:12.889 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed 2019-10-13T00:49:12.889 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed 2019-10-13T00:49:12.890 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-10-13T00:49:12.890 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-10-13T00:49:12.890 INFO:teuthology.orchestra.run.smithi155.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-10-13T00:49:12.891 INFO:teuthology.orchestra.run.smithi155.stdout:--> Running transaction check 2019-10-13T00:49:12.891 INFO:teuthology.orchestra.run.smithi155.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-10-13T00:49:12.903 INFO:teuthology.orchestra.run.smithi142.stdout:--> Finished Dependency Resolution 2019-10-13T00:49:12.985 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:49:12.985 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies Resolved 2019-10-13T00:49:12.998 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:49:12.998 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2019-10-13T00:49:12.998 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2019-10-13T00:49:12.998 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2019-10-13T00:49:12.998 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2019-10-13T00:49:12.998 INFO:teuthology.orchestra.run.smithi142.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2019-10-13T00:49:12.998 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 2.0 k 2019-10-13T00:49:12.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-daemon x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 11 k 2019-10-13T00:49:12.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 545 k 2019-10-13T00:49:12.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 1.6 M 2019-10-13T00:49:12.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 3.9 M 2019-10-13T00:49:12.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-cloud 2019-10-13T00:49:12.999 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 83 k 2019-10-13T00:49:12.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local 2019-10-13T00:49:13.000 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 1.1 M 2019-10-13T00:49:13.000 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 24 k 2019-10-13T00:49:13.000 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-ssh noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 21 k 2019-10-13T00:49:13.000 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 6.0 M 2019-10-13T00:49:13.000 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 37 M 2019-10-13T00:49:13.000 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel 2019-10-13T00:49:13.000 INFO:teuthology.orchestra.run.smithi142.stdout: x86_64 0.176-2.el7 rhel-7-server-rpms 39 k 2019-10-13T00:49:13.001 INFO:teuthology.orchestra.run.smithi142.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2019-10-13T00:49:13.001 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 17 k 2019-10-13T00:49:13.001 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 482 k 2019-10-13T00:49:13.001 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel x86_64 1:1.0.2k-19.el7 rhel-7-server-rpms 1.5 M 2019-10-13T00:49:13.001 INFO:teuthology.orchestra.run.smithi142.stdout: python-ceph-compat x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 2.2 k 2019-10-13T00:49:13.002 INFO:teuthology.orchestra.run.smithi142.stdout: python36-cephfs x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 139 k 2019-10-13T00:49:13.002 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 67 k 2019-10-13T00:49:13.002 INFO:teuthology.orchestra.run.smithi142.stdout:Updating: 2019-10-13T00:49:13.002 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 3.5 M 2019-10-13T00:49:13.002 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 1.7 M 2019-10-13T00:49:13.002 INFO:teuthology.orchestra.run.smithi142.stdout:Installing for dependencies: 2019-10-13T00:49:13.002 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 5.4 M 2019-10-13T00:49:13.003 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 19 M 2019-10-13T00:49:13.003 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards 2019-10-13T00:49:13.003 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 18 k 2019-10-13T00:49:13.003 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 1.9 M 2019-10-13T00:49:13.003 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 4.3 M 2019-10-13T00:49:13.003 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 16 M 2019-10-13T00:49:13.003 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 20 k 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: container-selinux noarch 2:2.107-3.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: 39 k 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: containernetworking-plugins 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: x86_64 0.8.1-2.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: 16 M 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: containers-common x86_64 1:0.1.37-3.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: 21 k 2019-10-13T00:49:13.004 INFO:teuthology.orchestra.run.smithi142.stdout: criu x86_64 3.12-2.el7 rhel-7-server-rpms 453 k 2019-10-13T00:49:13.005 INFO:teuthology.orchestra.run.smithi142.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-10-13T00:49:13.005 INFO:teuthology.orchestra.run.smithi142.stdout: keyutils-libs-devel x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2019-10-13T00:49:13.006 INFO:teuthology.orchestra.run.smithi142.stdout: krb5-devel x86_64 1.15.1-37.el7_7.2 rhel-7-server-rpms 272 k 2019-10-13T00:49:13.006 INFO:teuthology.orchestra.run.smithi142.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-10-13T00:49:13.006 INFO:teuthology.orchestra.run.smithi142.stdout: libcom_err-devel x86_64 1.42.9-16.el7 rhel-7-server-rpms 32 k 2019-10-13T00:49:13.006 INFO:teuthology.orchestra.run.smithi142.stdout: libnet x86_64 1.1.6-7.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.006 INFO:teuthology.orchestra.run.smithi142.stdout: 59 k 2019-10-13T00:49:13.007 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-10-13T00:49:13.007 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.8.0-2.el7 rhel-7-server-rpms 37 k 2019-10-13T00:49:13.007 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 85 k 2019-10-13T00:49:13.007 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 361 k 2019-10-13T00:49:13.008 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 5.3 M 2019-10-13T00:49:13.008 INFO:teuthology.orchestra.run.smithi142.stdout: libselinux-devel x86_64 2.5-14.1.el7 rhel-7-server-rpms 187 k 2019-10-13T00:49:13.008 INFO:teuthology.orchestra.run.smithi142.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2019-10-13T00:49:13.008 INFO:teuthology.orchestra.run.smithi142.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2019-10-13T00:49:13.008 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-10-13T00:49:13.008 INFO:teuthology.orchestra.run.smithi142.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2019-10-13T00:49:13.008 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-10-13T00:49:13.009 INFO:teuthology.orchestra.run.smithi142.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2019-10-13T00:49:13.009 INFO:teuthology.orchestra.run.smithi142.stdout: podman x86_64 1.4.4-4.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.009 INFO:teuthology.orchestra.run.smithi142.stdout: 9.3 M 2019-10-13T00:49:13.009 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-c x86_64 1.0.2-3.el7 rhel-7-server-rpms 28 k 2019-10-13T00:49:13.009 INFO:teuthology.orchestra.run.smithi142.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2019-10-13T00:49:13.009 INFO:teuthology.orchestra.run.smithi142.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-10-13T00:49:13.009 INFO:teuthology.orchestra.run.smithi142.stdout: python-ceph-argparse 2019-10-13T00:49:13.010 INFO:teuthology.orchestra.run.smithi142.stdout: x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 35 k 2019-10-13T00:49:13.010 INFO:teuthology.orchestra.run.smithi142.stdout: python-ceph-common x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 13 k 2019-10-13T00:49:13.010 INFO:teuthology.orchestra.run.smithi142.stdout: python-cephfs x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 100 k 2019-10-13T00:49:13.010 INFO:teuthology.orchestra.run.smithi142.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2019-10-13T00:49:13.010 INFO:teuthology.orchestra.run.smithi142.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-10-13T00:49:13.010 INFO:teuthology.orchestra.run.smithi142.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2019-10-13T00:49:13.010 INFO:teuthology.orchestra.run.smithi142.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2019-10-13T00:49:13.011 INFO:teuthology.orchestra.run.smithi142.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2019-10-13T00:49:13.011 INFO:teuthology.orchestra.run.smithi142.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2019-10-13T00:49:13.011 INFO:teuthology.orchestra.run.smithi142.stdout: rhel-7-server-rpms 866 k 2019-10-13T00:49:13.011 INFO:teuthology.orchestra.run.smithi142.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-10-13T00:49:13.011 INFO:teuthology.orchestra.run.smithi142.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2019-10-13T00:49:13.012 INFO:teuthology.orchestra.run.smithi142.stdout: python-rados x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 208 k 2019-10-13T00:49:13.012 INFO:teuthology.orchestra.run.smithi142.stdout: python-rbd x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 189 k 2019-10-13T00:49:13.012 INFO:teuthology.orchestra.run.smithi142.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-10-13T00:49:13.012 INFO:teuthology.orchestra.run.smithi142.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-10-13T00:49:13.012 INFO:teuthology.orchestra.run.smithi142.stdout: python-requests noarch 2.6.0-5.el7 rhel-7-server-rpms 94 k 2019-10-13T00:49:13.012 INFO:teuthology.orchestra.run.smithi142.stdout: python-rgw x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 77 k 2019-10-13T00:49:13.012 INFO:teuthology.orchestra.run.smithi142.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-10-13T00:49:13.013 INFO:teuthology.orchestra.run.smithi142.stdout: python-simplegeneric 2019-10-13T00:49:13.013 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 0.8-7.el7 epel 12 k 2019-10-13T00:49:13.013 INFO:teuthology.orchestra.run.smithi142.stdout: python-singledispatch 2019-10-13T00:49:13.013 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2019-10-13T00:49:13.015 INFO:teuthology.orchestra.run.smithi142.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2019-10-13T00:49:13.015 INFO:teuthology.orchestra.run.smithi142.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2019-10-13T00:49:13.015 INFO:teuthology.orchestra.run.smithi142.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.015 INFO:teuthology.orchestra.run.smithi142.stdout: 202 k 2019-10-13T00:49:13.015 INFO:teuthology.orchestra.run.smithi142.stdout: python-websocket-client 2019-10-13T00:49:13.015 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.015 INFO:teuthology.orchestra.run.smithi142.stdout: 66 k 2019-10-13T00:49:13.016 INFO:teuthology.orchestra.run.smithi142.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.016 INFO:teuthology.orchestra.run.smithi142.stdout: 102 k 2019-10-13T00:49:13.016 INFO:teuthology.orchestra.run.smithi142.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.016 INFO:teuthology.orchestra.run.smithi142.stdout: 562 k 2019-10-13T00:49:13.016 INFO:teuthology.orchestra.run.smithi142.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-10-13T00:49:13.016 INFO:teuthology.orchestra.run.smithi142.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-10-13T00:49:13.016 INFO:teuthology.orchestra.run.smithi142.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-10-13T00:49:13.017 INFO:teuthology.orchestra.run.smithi142.stdout: python2-kubernetes noarch 8.0.1-1.el7 epel 1.0 M 2019-10-13T00:49:13.017 INFO:teuthology.orchestra.run.smithi142.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2019-10-13T00:49:13.017 INFO:teuthology.orchestra.run.smithi142.stdout: python2-pyasn1-modules 2019-10-13T00:49:13.017 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.017 INFO:teuthology.orchestra.run.smithi142.stdout: 59 k 2019-10-13T00:49:13.017 INFO:teuthology.orchestra.run.smithi142.stdout: python2-requests-oauthlib 2019-10-13T00:49:13.017 INFO:teuthology.orchestra.run.smithi142.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2019-10-13T00:49:13.018 INFO:teuthology.orchestra.run.smithi142.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-10-13T00:49:13.018 INFO:teuthology.orchestra.run.smithi142.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-10-13T00:49:13.018 INFO:teuthology.orchestra.run.smithi142.stdout: python3 x86_64 3.6.8-10.el7 rhel-7-server-rpms 69 k 2019-10-13T00:49:13.018 INFO:teuthology.orchestra.run.smithi142.stdout: python3-libs x86_64 3.6.8-10.el7 rhel-7-server-rpms 7.0 M 2019-10-13T00:49:13.020 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pip noarch 9.0.3-5.el7 rhel-7-server-rpms 1.8 M 2019-10-13T00:49:13.020 INFO:teuthology.orchestra.run.smithi142.stdout: python3-setuptools noarch 39.2.0-10.el7 rhel-7-server-rpms 629 k 2019-10-13T00:49:13.020 INFO:teuthology.orchestra.run.smithi142.stdout: python36-ceph-argparse 2019-10-13T00:49:13.020 INFO:teuthology.orchestra.run.smithi142.stdout: x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 36 k 2019-10-13T00:49:13.020 INFO:teuthology.orchestra.run.smithi142.stdout: python36-rados x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 333 k 2019-10-13T00:49:13.021 INFO:teuthology.orchestra.run.smithi142.stdout: runc x86_64 1.0.0-65.rc8.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.021 INFO:teuthology.orchestra.run.smithi142.stdout: 2.0 M 2019-10-13T00:49:13.021 INFO:teuthology.orchestra.run.smithi142.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2019-10-13T00:49:13.021 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2019-10-13T00:49:13.021 INFO:teuthology.orchestra.run.smithi142.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2019-10-13T00:49:13.021 INFO:teuthology.orchestra.run.smithi142.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2019-10-13T00:49:13.021 INFO:teuthology.orchestra.run.smithi142.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-10-13T00:49:13.022 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel x86_64 1.2.7-18.el7 rhel-7-server-rpms 50 k 2019-10-13T00:49:13.022 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:49:13.022 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2019-10-13T00:49:13.022 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2019-10-13T00:49:13.023 INFO:teuthology.orchestra.run.smithi142.stdout:Install 20 Packages (+81 Dependent packages) 2019-10-13T00:49:13.023 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 2 Packages 2019-10-13T00:49:13.023 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:49:13.024 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 168 M 2019-10-13T00:49:13.024 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading packages: 2019-10-13T00:49:13.024 INFO:teuthology.orchestra.run.smithi142.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-10-13T00:49:13.575 INFO:teuthology.orchestra.run.smithi155.stdout:--> Finished Dependency Resolution 2019-10-13T00:49:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:49:13.656 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies Resolved 2019-10-13T00:49:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:49:13.669 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2019-10-13T00:49:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2019-10-13T00:49:13.669 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 2.0 k 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-daemon x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 11 k 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 545 k 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 1.6 M 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 3.9 M 2019-10-13T00:49:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-cloud 2019-10-13T00:49:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 83 k 2019-10-13T00:49:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local 2019-10-13T00:49:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 1.1 M 2019-10-13T00:49:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 24 k 2019-10-13T00:49:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-ssh noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 21 k 2019-10-13T00:49:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 6.0 M 2019-10-13T00:49:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 37 M 2019-10-13T00:49:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel 2019-10-13T00:49:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 0.176-2.el7 rhel-7-server-rpms 39 k 2019-10-13T00:49:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2019-10-13T00:49:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 17 k 2019-10-13T00:49:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 482 k 2019-10-13T00:49:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:1.0.2k-19.el7 rhel-7-server-rpms 1.5 M 2019-10-13T00:49:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: python-ceph-compat x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 2.2 k 2019-10-13T00:49:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: python36-cephfs x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 139 k 2019-10-13T00:49:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 67 k 2019-10-13T00:49:13.673 INFO:teuthology.orchestra.run.smithi155.stdout:Updating: 2019-10-13T00:49:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 3.5 M 2019-10-13T00:49:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 1.7 M 2019-10-13T00:49:13.675 INFO:teuthology.orchestra.run.smithi155.stdout:Installing for dependencies: 2019-10-13T00:49:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 5.4 M 2019-10-13T00:49:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 19 M 2019-10-13T00:49:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards 2019-10-13T00:49:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 2:15.0.0-6132.ga8ee8ac.el7 ceph-noarch 18 k 2019-10-13T00:49:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 1.9 M 2019-10-13T00:49:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 4.3 M 2019-10-13T00:49:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 16 M 2019-10-13T00:49:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 20 k 2019-10-13T00:49:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: container-selinux noarch 2:2.107-3.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: 39 k 2019-10-13T00:49:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: containernetworking-plugins 2019-10-13T00:49:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 0.8.1-2.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: 16 M 2019-10-13T00:49:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: containers-common x86_64 1:0.1.37-3.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: 21 k 2019-10-13T00:49:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: criu x86_64 3.12-2.el7 rhel-7-server-rpms 453 k 2019-10-13T00:49:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-10-13T00:49:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: keyutils-libs-devel x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2019-10-13T00:49:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: krb5-devel x86_64 1.15.1-37.el7_7.2 rhel-7-server-rpms 272 k 2019-10-13T00:49:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-10-13T00:49:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: libcom_err-devel x86_64 1.42.9-16.el7 rhel-7-server-rpms 32 k 2019-10-13T00:49:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: libnet x86_64 1.1.6-7.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: 59 k 2019-10-13T00:49:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-10-13T00:49:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.8.0-2.el7 rhel-7-server-rpms 37 k 2019-10-13T00:49:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 85 k 2019-10-13T00:49:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 361 k 2019-10-13T00:49:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 5.3 M 2019-10-13T00:49:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: libselinux-devel x86_64 2.5-14.1.el7 rhel-7-server-rpms 187 k 2019-10-13T00:49:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2019-10-13T00:49:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2019-10-13T00:49:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-10-13T00:49:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2019-10-13T00:49:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-10-13T00:49:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2019-10-13T00:49:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: podman x86_64 1.4.4-4.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: 9.3 M 2019-10-13T00:49:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-c x86_64 1.0.2-3.el7 rhel-7-server-rpms 28 k 2019-10-13T00:49:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2019-10-13T00:49:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-10-13T00:49:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: python-ceph-argparse 2019-10-13T00:49:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 35 k 2019-10-13T00:49:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: python-ceph-common x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 13 k 2019-10-13T00:49:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: python-cephfs x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 100 k 2019-10-13T00:49:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2019-10-13T00:49:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-10-13T00:49:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2019-10-13T00:49:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2019-10-13T00:49:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2019-10-13T00:49:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2019-10-13T00:49:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: rhel-7-server-rpms 866 k 2019-10-13T00:49:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-10-13T00:49:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2019-10-13T00:49:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: python-rados x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 208 k 2019-10-13T00:49:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: python-rbd x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 189 k 2019-10-13T00:49:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-10-13T00:49:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-10-13T00:49:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: python-requests noarch 2.6.0-5.el7 rhel-7-server-rpms 94 k 2019-10-13T00:49:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: python-rgw x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 77 k 2019-10-13T00:49:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-10-13T00:49:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: python-simplegeneric 2019-10-13T00:49:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 0.8-7.el7 epel 12 k 2019-10-13T00:49:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: python-singledispatch 2019-10-13T00:49:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2019-10-13T00:49:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2019-10-13T00:49:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: 202 k 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: python-websocket-client 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: 66 k 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: 102 k 2019-10-13T00:49:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: 562 k 2019-10-13T00:49:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-10-13T00:49:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-10-13T00:49:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-10-13T00:49:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: python2-kubernetes noarch 8.0.1-1.el7 epel 1.0 M 2019-10-13T00:49:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2019-10-13T00:49:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: python2-pyasn1-modules 2019-10-13T00:49:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2019-10-13T00:49:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: 59 k 2019-10-13T00:49:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: python2-requests-oauthlib 2019-10-13T00:49:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2019-10-13T00:49:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-10-13T00:49:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-10-13T00:49:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: python3 x86_64 3.6.8-10.el7 rhel-7-server-rpms 69 k 2019-10-13T00:49:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: python3-libs x86_64 3.6.8-10.el7 rhel-7-server-rpms 7.0 M 2019-10-13T00:49:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pip noarch 9.0.3-5.el7 rhel-7-server-rpms 1.8 M 2019-10-13T00:49:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: python3-setuptools noarch 39.2.0-10.el7 rhel-7-server-rpms 629 k 2019-10-13T00:49:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: python36-ceph-argparse 2019-10-13T00:49:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 36 k 2019-10-13T00:49:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: python36-rados x86_64 2:15.0.0-6132.ga8ee8ac.el7 ceph 333 k 2019-10-13T00:49:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: runc x86_64 1.0.0-65.rc8.el7 rhel-7-server-extras-rpms 2019-10-13T00:49:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: 2.0 M 2019-10-13T00:49:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2019-10-13T00:49:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2019-10-13T00:49:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2019-10-13T00:49:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2019-10-13T00:49:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-10-13T00:49:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.7-18.el7 rhel-7-server-rpms 50 k 2019-10-13T00:49:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:49:13.697 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2019-10-13T00:49:13.697 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2019-10-13T00:49:13.698 INFO:teuthology.orchestra.run.smithi155.stdout:Install 20 Packages (+81 Dependent packages) 2019-10-13T00:49:13.698 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2019-10-13T00:49:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:49:13.699 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 168 M 2019-10-13T00:49:13.699 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading packages: 2019-10-13T00:49:13.699 INFO:teuthology.orchestra.run.smithi155.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-10-13T00:49:19.434 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2019-10-13T00:49:19.434 INFO:teuthology.orchestra.run.smithi142.stdout:Total 26 MB/s | 168 MB 00:06 2019-10-13T00:49:19.488 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2019-10-13T00:49:19.555 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2019-10-13T00:49:19.978 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded 2019-10-13T00:49:19.978 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2019-10-13T00:49:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/105 2019-10-13T00:49:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/105 2019-10-13T00:49:22.403 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2019-10-13T00:49:22.403 INFO:teuthology.orchestra.run.smithi155.stdout:Total 19 MB/s | 168 MB 00:08 2019-10-13T00:49:22.455 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2019-10-13T00:49:22.522 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2019-10-13T00:49:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 3/105 2019-10-13T00:49:22.962 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded 2019-10-13T00:49:22.962 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2019-10-13T00:49:23.107 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-3.6.8-10.el7.x86_64 4/105 2019-10-13T00:49:23.565 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/105 2019-10-13T00:49:23.853 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 5/105 2019-10-13T00:49:24.214 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 6/105 2019-10-13T00:49:24.425 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_64 7/105 2019-10-13T00:49:24.730 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-six-1.9.0-0.el7.noarch 8/105 2019-10-13T00:49:25.104 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 9/105 2019-10-13T00:49:25.223 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/105 2019-10-13T00:49:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 10/105 2019-10-13T00:49:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 3/105 2019-10-13T00:49:25.765 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-webob-1.2.3-7.el7.noarch 11/105 2019-10-13T00:49:26.044 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-3.6.8-10.el7.x86_64 4/105 2019-10-13T00:49:26.792 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 5/105 2019-10-13T00:49:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 6/105 2019-10-13T00:49:27.368 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_64 7/105 2019-10-13T00:49:27.696 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-six-1.9.0-0.el7.noarch 8/105 2019-10-13T00:49:28.103 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 9/105 2019-10-13T00:49:28.448 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 10/105 2019-10-13T00:49:28.771 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-webob-1.2.3-7.el7.noarch 11/105 2019-10-13T00:49:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:container-selinux-2.107-3.el7.noarch 12/105 2019-10-13T00:49:34.857 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : zlib-devel-1.2.7-18.el7.x86_64 13/105 2019-10-13T00:49:35.157 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 14/105 2019-10-13T00:49:35.445 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-requests-2.6.0-5.el7.noarch 15/105 2019-10-13T00:49:35.744 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 16/105 2019-10-13T00:49:36.035 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 17/105 2019-10-13T00:49:36.234 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python36-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_ 18/105 2019-10-13T00:49:36.635 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/105 2019-10-13T00:49:37.034 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/105 2019-10-13T00:49:37.420 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:container-selinux-2.107-3.el7.noarch 12/105 2019-10-13T00:49:37.443 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/105 2019-10-13T00:49:37.733 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.7-18.el7.x86_64 13/105 2019-10-13T00:49:37.752 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/105 2019-10-13T00:49:38.024 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/105 2019-10-13T00:49:38.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 14/105 2019-10-13T00:49:38.255 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/105 2019-10-13T00:49:38.346 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-requests-2.6.0-5.el7.noarch 15/105 2019-10-13T00:49:38.538 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 25/105 2019-10-13T00:49:38.669 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 16/105 2019-10-13T00:49:38.830 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-6132.ga8ee8ac.el7.noa 26/105 2019-10-13T00:49:38.985 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 17/105 2019-10-13T00:49:39.119 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/105 2019-10-13T00:49:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python36-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_ 18/105 2019-10-13T00:49:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 28/105 2019-10-13T00:49:39.627 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/105 2019-10-13T00:49:39.726 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 1:containers-common-0.1.37-3.el7.x86_64 29/105 2019-10-13T00:49:40.018 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/105 2019-10-13T00:49:40.018 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libnet-1.1.6-7.el7.x86_64 30/105 2019-10-13T00:49:40.396 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 31/105 2019-10-13T00:49:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/105 2019-10-13T00:49:40.779 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/105 2019-10-13T00:49:41.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/105 2019-10-13T00:49:41.306 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/105 2019-10-13T00:49:41.589 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 25/105 2019-10-13T00:49:41.908 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-6132.ga8ee8ac.el7.noa 26/105 2019-10-13T00:49:42.170 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/105 2019-10-13T00:49:42.249 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 32/105 2019-10-13T00:49:42.484 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 28/105 2019-10-13T00:49:42.537 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : containernetworking-plugins-0.8.1-2.el7.x86_64 33/105 2019-10-13T00:49:42.788 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 1:containers-common-0.1.37-3.el7.x86_64 29/105 2019-10-13T00:49:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcom_err-devel-1.42.9-16.el7.x86_64 34/105 2019-10-13T00:49:43.055 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libnet-1.1.6-7.el7.x86_64 30/105 2019-10-13T00:49:43.100 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 35/105 2019-10-13T00:49:43.359 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 36/105 2019-10-13T00:49:43.463 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 31/105 2019-10-13T00:49:43.629 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 37/105 2019-10-13T00:49:43.918 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/105 2019-10-13T00:49:44.231 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 39/105 2019-10-13T00:49:44.541 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 40/105 2019-10-13T00:49:45.344 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 32/105 2019-10-13T00:49:45.557 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 41/105 2019-10-13T00:49:45.665 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : containernetworking-plugins-0.8.1-2.el7.x86_64 33/105 2019-10-13T00:49:45.900 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python2-kubernetes-8.0.1-1.el7.noarch 42/105 2019-10-13T00:49:46.200 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcom_err-devel-1.42.9-16.el7.x86_64 34/105 2019-10-13T00:49:46.236 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 43/105 2019-10-13T00:49:46.463 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 35/105 2019-10-13T00:49:46.706 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 36/105 2019-10-13T00:49:47.006 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 37/105 2019-10-13T00:49:47.337 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 44/105 2019-10-13T00:49:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/105 2019-10-13T00:49:47.634 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 45/105 2019-10-13T00:49:47.701 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 39/105 2019-10-13T00:49:47.904 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 46/105 2019-10-13T00:49:48.044 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 40/105 2019-10-13T00:49:48.167 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 47/105 2019-10-13T00:49:48.507 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 48/105 2019-10-13T00:49:48.856 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : jq-1.5-1.el7.x86_64 49/105 2019-10-13T00:49:49.116 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 41/105 2019-10-13T00:49:49.135 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 50/105 2019-10-13T00:49:49.429 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python2-kubernetes-8.0.1-1.el7.noarch 42/105 2019-10-13T00:49:49.760 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 43/105 2019-10-13T00:49:49.765 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 51/105 2019-10-13T00:49:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 44/105 2019-10-13T00:49:50.123 INFO:teuthology.orchestra.run.smithi142.stdout: Updating : 2:librados2-15.0.0-6132.ga8ee8ac.el7.x86_64 52/105 2019-10-13T00:49:50.395 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 45/105 2019-10-13T00:49:50.559 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:libcephfs2-15.0.0-6132.ga8ee8ac.el7.x86_64 53/105 2019-10-13T00:49:50.681 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 46/105 2019-10-13T00:49:50.906 INFO:teuthology.orchestra.run.smithi142.stdout: Updating : 2:librbd1-15.0.0-6132.ga8ee8ac.el7.x86_64 54/105 2019-10-13T00:49:51.657 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 55/105 2019-10-13T00:49:51.912 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 47/105 2019-10-13T00:49:52.185 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 48/105 2019-10-13T00:49:52.443 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.5-1.el7.x86_64 49/105 2019-10-13T00:49:52.456 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:librgw2-15.0.0-6132.ga8ee8ac.el7.x86_64 56/105 2019-10-13T00:49:52.739 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 50/105 2019-10-13T00:49:52.768 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python-rgw-15.0.0-6132.ga8ee8ac.el7.x86_64 57/105 2019-10-13T00:49:53.061 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python-rbd-15.0.0-6132.ga8ee8ac.el7.x86_64 58/105 2019-10-13T00:49:53.393 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 51/105 2019-10-13T00:49:53.399 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 59/105 2019-10-13T00:49:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:libradosstriper1-15.0.0-6132.ga8ee8ac.el7.x86_64 60/105 2019-10-13T00:49:53.743 INFO:teuthology.orchestra.run.smithi155.stdout: Updating : 2:librados2-15.0.0-6132.ga8ee8ac.el7.x86_64 52/105 2019-10-13T00:49:53.978 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python36-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 61/105 2019-10-13T00:49:54.211 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:libcephfs2-15.0.0-6132.ga8ee8ac.el7.x86_64 53/105 2019-10-13T00:49:54.239 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:librados-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 62/105 2019-10-13T00:49:54.567 INFO:teuthology.orchestra.run.smithi155.stdout: Updating : 2:librbd1-15.0.0-6132.ga8ee8ac.el7.x86_64 54/105 2019-10-13T00:49:54.578 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 63/105 2019-10-13T00:49:54.862 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libselinux-devel-2.5-14.1.el7.x86_64 64/105 2019-10-13T00:49:55.223 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 65/105 2019-10-13T00:49:55.335 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 55/105 2019-10-13T00:49:55.528 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-routes-1.13-2.el7.noarch 66/105 2019-10-13T00:49:55.647 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:librgw2-15.0.0-6132.ga8ee8ac.el7.x86_64 56/105 2019-10-13T00:49:55.875 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-c-1.0.2-3.el7.x86_64 67/105 2019-10-13T00:49:55.926 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python-rgw-15.0.0-6132.ga8ee8ac.el7.x86_64 57/105 2019-10-13T00:49:56.326 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : criu-3.12-2.el7.x86_64 68/105 2019-10-13T00:49:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python-rbd-15.0.0-6132.ga8ee8ac.el7.x86_64 58/105 2019-10-13T00:49:57.268 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 59/105 2019-10-13T00:49:57.530 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:libradosstriper1-15.0.0-6132.ga8ee8ac.el7.x86_64 60/105 2019-10-13T00:49:57.593 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : runc-1.0.0-65.rc8.el7.x86_64 69/105 2019-10-13T00:49:57.824 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python36-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 61/105 2019-10-13T00:49:57.949 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : podman-1.4.4-4.el7.x86_64 70/105 2019-10-13T00:49:58.118 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:librados-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 62/105 2019-10-13T00:49:58.256 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-daemon-15.0.0-6132.ga8ee8ac.el7.x86_64 71/105 2019-10-13T00:49:58.477 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 63/105 2019-10-13T00:49:58.575 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 72/105 2019-10-13T00:49:58.783 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libselinux-devel-2.5-14.1.el7.x86_64 64/105 2019-10-13T00:49:58.964 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-mako-0.8.1-2.el7.noarch 73/105 2019-10-13T00:49:59.126 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 65/105 2019-10-13T00:49:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 74/105 2019-10-13T00:49:59.424 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-routes-1.13-2.el7.noarch 66/105 2019-10-13T00:49:59.738 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-c-1.0.2-3.el7.x86_64 67/105 2019-10-13T00:50:00.240 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : criu-3.12-2.el7.x86_64 68/105 2019-10-13T00:50:01.558 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : runc-1.0.0-65.rc8.el7.x86_64 69/105 2019-10-13T00:50:01.622 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python-ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 75/105 2019-10-13T00:50:01.908 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : podman-1.4.4-4.el7.x86_64 70/105 2019-10-13T00:50:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-daemon-15.0.0-6132.ga8ee8ac.el7.x86_64 71/105 2019-10-13T00:50:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 72/105 2019-10-13T00:50:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 76/105 2019-10-13T00:50:02.904 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-mako-0.8.1-2.el7.noarch 73/105 2019-10-13T00:50:03.076 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-base-15.0.0-6132.ga8ee8ac.el7.x86_64 77/105 2019-10-13T00:50:03.253 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 74/105 2019-10-13T00:50:05.589 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python-ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 75/105 2019-10-13T00:50:06.583 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 76/105 2019-10-13T00:50:07.130 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-base-15.0.0-6132.ga8ee8ac.el7.x86_64 77/105 2019-10-13T00:50:11.641 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-selinux-15.0.0-6132.ga8ee8ac.el7.x86_64 78/105 2019-10-13T00:50:12.885 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 79/105 2019-10-13T00:50:13.388 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mon-15.0.0-6132.ga8ee8ac.el7.x86_64 80/105 2019-10-13T00:50:15.162 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mds-15.0.0-6132.ga8ee8ac.el7.x86_64 81/105 2019-10-13T00:50:15.517 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-osd-15.0.0-6132.ga8ee8ac.el7.x86_64 82/105 2019-10-13T00:50:15.806 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-selinux-15.0.0-6132.ga8ee8ac.el7.x86_64 78/105 2019-10-13T00:50:15.830 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 83/105 2019-10-13T00:50:16.140 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : krb5-devel-1.15.1-37.el7_7.2.x86_64 84/105 2019-10-13T00:50:16.624 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 79/105 2019-10-13T00:50:17.311 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mon-15.0.0-6132.ga8ee8ac.el7.x86_64 80/105 2019-10-13T00:50:17.753 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 85/105 2019-10-13T00:50:18.473 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : scipy-0.12.1-6.el7.x86_64 86/105 2019-10-13T00:50:19.071 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mds-15.0.0-6132.ga8ee8ac.el7.x86_64 81/105 2019-10-13T00:50:19.079 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-6132.ga8ee8ac.e 87/105 2019-10-13T00:50:19.390 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-osd-15.0.0-6132.ga8ee8ac.el7.x86_64 82/105 2019-10-13T00:50:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 1:openssl-devel-1.0.2k-19.el7.x86_64 88/105 2019-10-13T00:50:19.665 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 89/105 2019-10-13T00:50:19.689 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 83/105 2019-10-13T00:50:19.937 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mgr-rook-15.0.0-6132.ga8ee8ac.el7.noarch 90/105 2019-10-13T00:50:20.030 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : krb5-devel-1.15.1-37.el7_7.2.x86_64 84/105 2019-10-13T00:50:20.949 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-6132.ga8ee8ac.e 91/105 2019-10-13T00:50:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 92/105 2019-10-13T00:50:21.628 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 85/105 2019-10-13T00:50:22.175 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-6132.ga8ee8ac.el7.noarch 93/105 2019-10-13T00:50:22.363 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : scipy-0.12.1-6.el7.x86_64 86/105 2019-10-13T00:50:23.032 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-6132.ga8ee8ac.e 87/105 2019-10-13T00:50:23.387 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 1:openssl-devel-1.0.2k-19.el7.x86_64 88/105 2019-10-13T00:50:23.693 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 89/105 2019-10-13T00:50:24.025 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mgr-rook-15.0.0-6132.ga8ee8ac.el7.noarch 90/105 2019-10-13T00:50:25.098 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-6132.ga8ee8ac.e 91/105 2019-10-13T00:50:25.494 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 92/105 2019-10-13T00:50:25.965 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 94/105 2019-10-13T00:50:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 95/105 2019-10-13T00:50:26.336 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-6132.ga8ee8ac.el7.noarch 93/105 2019-10-13T00:50:26.602 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:libcephfs-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 96/105 2019-10-13T00:50:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 97/105 2019-10-13T00:50:27.513 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 98/105 2019-10-13T00:50:27.806 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:rbd-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 99/105 2019-10-13T00:50:28.078 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 100/105 2019-10-13T00:50:28.358 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : elfutils-libelf-devel-0.176-2.el7.x86_64 101/105 2019-10-13T00:50:28.785 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flex-2.5.37-6.el7.x86_64 102/105 2019-10-13T00:50:29.105 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : bison-3.0.4-2.el7.x86_64 103/105 2019-10-13T00:50:29.344 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 104/105 2019-10-13T00:50:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 105/105 2019-10-13T00:50:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python36-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 1/105 2019-10-13T00:50:29.872 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : podman-1.4.4-4.el7.x86_64 2/105 2019-10-13T00:50:29.989 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 3/105 2019-10-13T00:50:30.106 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/105 2019-10-13T00:50:30.215 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-routes-1.13-2.el7.noarch 5/105 2019-10-13T00:50:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 6/105 2019-10-13T00:50:30.449 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/105 2019-10-13T00:50:30.499 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 94/105 2019-10-13T00:50:30.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 8/105 2019-10-13T00:50:30.684 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 9/105 2019-10-13T00:50:30.786 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 95/105 2019-10-13T00:50:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : zlib-devel-1.2.7-18.el7.x86_64 10/105 2019-10-13T00:50:30.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mon-15.0.0-6132.ga8ee8ac.el7.x86_64 11/105 2019-10-13T00:50:31.019 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-base-15.0.0-6132.ga8ee8ac.el7.x86_64 12/105 2019-10-13T00:50:31.066 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:libcephfs-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 96/105 2019-10-13T00:50:31.102 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python-ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 13/105 2019-10-13T00:50:31.203 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 14/105 2019-10-13T00:50:31.300 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-c-1.0.2-3.el7.x86_64 15/105 2019-10-13T00:50:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : runc-1.0.0-65.rc8.el7.x86_64 16/105 2019-10-13T00:50:31.479 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mds-15.0.0-6132.ga8ee8ac.el7.x86_64 17/105 2019-10-13T00:50:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:container-selinux-2.107-3.el7.noarch 18/105 2019-10-13T00:50:31.722 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 19/105 2019-10-13T00:50:31.803 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 97/105 2019-10-13T00:50:31.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 20/105 2019-10-13T00:50:32.077 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 98/105 2019-10-13T00:50:32.435 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:rbd-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 99/105 2019-10-13T00:50:32.557 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:librados2-15.0.0-6132.ga8ee8ac.el7.x86_64 21/105 2019-10-13T00:50:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 22/105 2019-10-13T00:50:32.708 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 100/105 2019-10-13T00:50:32.808 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 23/105 2019-10-13T00:50:32.921 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 24/105 2019-10-13T00:50:33.021 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.176-2.el7.x86_64 101/105 2019-10-13T00:50:33.026 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-3.6.8-10.el7.x86_64 25/105 2019-10-13T00:50:33.127 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 26/105 2019-10-13T00:50:33.219 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 27/105 2019-10-13T00:50:33.311 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-requests-2.6.0-5.el7.noarch 28/105 2019-10-13T00:50:33.410 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.5.37-6.el7.x86_64 102/105 2019-10-13T00:50:33.437 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-6132.ga8ee8ac.e 29/105 2019-10-13T00:50:33.529 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 30/105 2019-10-13T00:50:33.646 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 31/105 2019-10-13T00:50:33.728 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.0.4-2.el7.x86_64 103/105 2019-10-13T00:50:33.763 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libselinux-devel-2.5-14.1.el7.x86_64 32/105 2019-10-13T00:50:33.881 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 33/105 2019-10-13T00:50:33.996 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 104/105 2019-10-13T00:50:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 34/105 2019-10-13T00:50:34.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 35/105 2019-10-13T00:50:34.232 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 36/105 2019-10-13T00:50:34.274 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 105/105 2019-10-13T00:50:34.357 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-6132.ga8ee8ac.el7.noarch 37/105 2019-10-13T00:50:34.444 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python36-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 1/105 2019-10-13T00:50:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-osd-15.0.0-6132.ga8ee8ac.el7.x86_64 38/105 2019-10-13T00:50:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : podman-1.4.4-4.el7.x86_64 2/105 2019-10-13T00:50:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-daemon-15.0.0-6132.ga8ee8ac.el7.x86_64 39/105 2019-10-13T00:50:34.693 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-radosgw-15.0.0-6132.ga8ee8ac.el7.x86_64 3/105 2019-10-13T00:50:34.709 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 40/105 2019-10-13T00:50:34.804 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python36-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_ 41/105 2019-10-13T00:50:34.812 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/105 2019-10-13T00:50:34.894 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 42/105 2019-10-13T00:50:34.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-routes-1.13-2.el7.noarch 5/105 2019-10-13T00:50:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 43/105 2019-10-13T00:50:35.030 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 6/105 2019-10-13T00:50:35.086 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python-rbd-15.0.0-6132.ga8ee8ac.el7.x86_64 44/105 2019-10-13T00:50:35.172 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/105 2019-10-13T00:50:35.178 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 45/105 2019-10-13T00:50:35.272 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 8/105 2019-10-13T00:50:35.279 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:librgw2-15.0.0-6132.ga8ee8ac.el7.x86_64 46/105 2019-10-13T00:50:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 9/105 2019-10-13T00:50:35.396 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 47/105 2019-10-13T00:50:35.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.7-18.el7.x86_64 10/105 2019-10-13T00:50:35.522 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : criu-3.12-2.el7.x86_64 48/105 2019-10-13T00:50:35.590 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mon-15.0.0-6132.ga8ee8ac.el7.x86_64 11/105 2019-10-13T00:50:35.639 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 49/105 2019-10-13T00:50:35.699 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-base-15.0.0-6132.ga8ee8ac.el7.x86_64 12/105 2019-10-13T00:50:35.781 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 50/105 2019-10-13T00:50:35.816 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python-ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 13/105 2019-10-13T00:50:35.898 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 51/105 2019-10-13T00:50:35.975 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 14/105 2019-10-13T00:50:36.007 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-6132.ga8ee8ac.e 52/105 2019-10-13T00:50:36.092 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-c-1.0.2-3.el7.x86_64 15/105 2019-10-13T00:50:36.124 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 53/105 2019-10-13T00:50:36.233 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : runc-1.0.0-65.rc8.el7.x86_64 16/105 2019-10-13T00:50:36.256 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 54/105 2019-10-13T00:50:36.359 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 55/105 2019-10-13T00:50:36.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mds-15.0.0-6132.ga8ee8ac.el7.x86_64 17/105 2019-10-13T00:50:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 56/105 2019-10-13T00:50:36.543 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 57/105 2019-10-13T00:50:36.644 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python-rgw-15.0.0-6132.ga8ee8ac.el7.x86_64 58/105 2019-10-13T00:50:36.744 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 59/105 2019-10-13T00:50:36.845 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcom_err-devel-1.42.9-16.el7.x86_64 60/105 2019-10-13T00:50:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : containernetworking-plugins-0.8.1-2.el7.x86_64 61/105 2019-10-13T00:50:37.137 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:container-selinux-2.107-3.el7.noarch 18/105 2019-10-13T00:50:37.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 19/105 2019-10-13T00:50:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : jq-1.5-1.el7.x86_64 62/105 2019-10-13T00:50:37.364 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 20/105 2019-10-13T00:50:37.416 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 63/105 2019-10-13T00:50:37.483 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:librados2-15.0.0-6132.ga8ee8ac.el7.x86_64 21/105 2019-10-13T00:50:37.541 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 64/105 2019-10-13T00:50:37.585 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 22/105 2019-10-13T00:50:37.659 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 65/105 2019-10-13T00:50:37.680 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python-ceph-compat-15.0.0-6132.ga8ee8ac.el7.x86_64 23/105 2019-10-13T00:50:37.773 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 24/105 2019-10-13T00:50:37.776 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 66/105 2019-10-13T00:50:37.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 1:openssl-devel-1.0.2k-19.el7.x86_64 67/105 2019-10-13T00:50:37.906 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-3.6.8-10.el7.x86_64 25/105 2019-10-13T00:50:37.935 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 68/105 2019-10-13T00:50:38.016 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 26/105 2019-10-13T00:50:38.027 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:libcephfs-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 69/105 2019-10-13T00:50:38.134 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 27/105 2019-10-13T00:50:38.161 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libnet-1.1.6-7.el7.x86_64 70/105 2019-10-13T00:50:38.245 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:librados-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 71/105 2019-10-13T00:50:38.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-requests-2.6.0-5.el7.noarch 28/105 2019-10-13T00:50:38.345 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:libcephfs2-15.0.0-6132.ga8ee8ac.el7.x86_64 72/105 2019-10-13T00:50:38.409 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-6132.ga8ee8ac.e 29/105 2019-10-13T00:50:38.463 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 73/105 2019-10-13T00:50:38.526 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 30/105 2019-10-13T00:50:38.571 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 74/105 2019-10-13T00:50:38.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-test-15.0.0-6132.ga8ee8ac.el7.x86_64 31/105 2019-10-13T00:50:38.689 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 1:containers-common-0.1.37-3.el7.x86_64 75/105 2019-10-13T00:50:38.785 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libselinux-devel-2.5-14.1.el7.x86_64 32/105 2019-10-13T00:50:38.797 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-selinux-15.0.0-6132.ga8ee8ac.el7.x86_64 76/105 2019-10-13T00:50:38.906 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 33/105 2019-10-13T00:50:38.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 77/105 2019-10-13T00:50:39.025 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:librbd1-15.0.0-6132.ga8ee8ac.el7.x86_64 78/105 2019-10-13T00:50:39.044 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 34/105 2019-10-13T00:50:39.133 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : krb5-devel-1.15.1-37.el7_7.2.x86_64 79/105 2019-10-13T00:50:39.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 35/105 2019-10-13T00:50:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 80/105 2019-10-13T00:50:39.279 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 36/105 2019-10-13T00:50:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 81/105 2019-10-13T00:50:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-6132.ga8ee8ac.el7.noarch 37/105 2019-10-13T00:50:39.468 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 82/105 2019-10-13T00:50:39.521 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-osd-15.0.0-6132.ga8ee8ac.el7.x86_64 38/105 2019-10-13T00:50:39.560 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 83/105 2019-10-13T00:50:39.613 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-daemon-15.0.0-6132.ga8ee8ac.el7.x86_64 39/105 2019-10-13T00:50:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-6132.ga8ee8ac.el7.noa 84/105 2019-10-13T00:50:39.706 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 40/105 2019-10-13T00:50:39.728 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 85/105 2019-10-13T00:50:39.819 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_64 86/105 2019-10-13T00:50:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python36-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_ 41/105 2019-10-13T00:50:39.929 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:rbd-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 87/105 2019-10-13T00:50:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 42/105 2019-10-13T00:50:40.046 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 88/105 2019-10-13T00:50:40.057 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 43/105 2019-10-13T00:50:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 89/105 2019-10-13T00:50:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python-rbd-15.0.0-6132.ga8ee8ac.el7.x86_64 44/105 2019-10-13T00:50:40.255 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python2-kubernetes-8.0.1-1.el7.noarch 90/105 2019-10-13T00:50:40.333 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 45/105 2019-10-13T00:50:40.364 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 91/105 2019-10-13T00:50:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:librgw2-15.0.0-6132.ga8ee8ac.el7.x86_64 46/105 2019-10-13T00:50:40.489 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-6132.ga8ee8ac.el7.noarch 92/105 2019-10-13T00:50:40.584 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 47/105 2019-10-13T00:50:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libelf-devel-0.176-2.el7.x86_64 93/105 2019-10-13T00:50:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : criu-3.12-2.el7.x86_64 48/105 2019-10-13T00:50:40.724 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:libradosstriper1-15.0.0-6132.ga8ee8ac.el7.x86_64 94/105 2019-10-13T00:50:40.827 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 49/105 2019-10-13T00:50:40.833 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 95/105 2019-10-13T00:50:40.933 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 96/105 2019-10-13T00:50:40.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 50/105 2019-10-13T00:50:41.026 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 97/105 2019-10-13T00:50:41.078 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 51/105 2019-10-13T00:50:41.109 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : bison-3.0.4-2.el7.x86_64 98/105 2019-10-13T00:50:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-6132.ga8ee8ac.e 52/105 2019-10-13T00:50:41.194 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 99/105 2019-10-13T00:50:41.303 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 100/105 2019-10-13T00:50:41.346 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-6132.ga8ee8ac.el7.noarch 53/105 2019-10-13T00:50:41.386 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flex-2.5.37-6.el7.x86_64 101/105 2019-10-13T00:50:41.495 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 102/105 2019-10-13T00:50:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 103/105 2019-10-13T00:50:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 104/105 2019-10-13T00:50:41.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 54/105 2019-10-13T00:50:41.975 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 105/105 2019-10-13T00:50:41.975 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:50:41.975 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2019-10-13T00:50:41.975 INFO:teuthology.orchestra.run.smithi142.stdout: bison.x86_64 0:3.0.4-2.el7 2019-10-13T00:50:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: ceph.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-daemon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-ssh.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel.x86_64 0:0.176-2.el7 2019-10-13T00:50:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: flex.x86_64 0:2.5.37-6.el7 2019-10-13T00:50:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel.x86_64 1:1.0.2k-19.el7 2019-10-13T00:50:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: python-ceph-compat.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: python36-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:50:41.979 INFO:teuthology.orchestra.run.smithi142.stdout:Dependency Installed: 2019-10-13T00:50:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.980 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.980 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: container-selinux.noarch 2:2.107-3.el7 2019-10-13T00:50:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: containernetworking-plugins.x86_64 0:0.8.1-2.el7 2019-10-13T00:50:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: containers-common.x86_64 1:0.1.37-3.el7 2019-10-13T00:50:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: criu.x86_64 0:3.12-2.el7 2019-10-13T00:50:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: jq.x86_64 0:1.5-1.el7 2019-10-13T00:50:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2019-10-13T00:50:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: krb5-devel.x86_64 0:1.15.1-37.el7_7.2 2019-10-13T00:50:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-10-13T00:50:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: libcom_err-devel.x86_64 0:1.42.9-16.el7 2019-10-13T00:50:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: libnet.x86_64 0:1.1.6-7.el7 2019-10-13T00:50:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-10-13T00:50:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-10-13T00:50:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: libselinux-devel.x86_64 0:2.5-14.1.el7 2019-10-13T00:50:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2019-10-13T00:50:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2019-10-13T00:50:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-10-13T00:50:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2019-10-13T00:50:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma.x86_64 0:5.9.5-3.el7 2019-10-13T00:50:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: pcre-devel.x86_64 0:8.32-17.el7 2019-10-13T00:50:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: podman.x86_64 0:1.4.4-4.el7 2019-10-13T00:50:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-c.x86_64 0:1.0.2-3.el7 2019-10-13T00:50:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-10-13T00:50:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-10-13T00:50:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: python-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: python-ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: python-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-10-13T00:50:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: python-execnet.noarch 0:1.2.0-7.el7 2019-10-13T00:50:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-10-13T00:50:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-10-13T00:50:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-10-13T00:50:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-10-13T00:50:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-10-13T00:50:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-10-13T00:50:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: python-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: python-rbd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: python-remoto.noarch 0:0.0.25-1.el7 2019-10-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-10-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: python-requests.noarch 0:2.6.0-5.el7 2019-10-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: python-rgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: python-routes.noarch 0:1.13-2.el7 2019-10-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-10-13T00:50:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-10-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-10-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-10-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-10-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-10-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-10-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-10-13T00:50:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-kubernetes.noarch 0:8.0.1-1.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-10-13T00:50:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: python3.x86_64 0:3.6.8-10.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: python3-libs.x86_64 0:3.6.8-10.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pip.noarch 0:9.0.3-5.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: python3-setuptools.noarch 0:39.2.0-10.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: python36-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: python36-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: runc.x86_64 0:1.0.0-65.rc8.el7 2019-10-13T00:50:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: scipy.x86_64 0:0.12.1-6.el7 2019-10-13T00:50:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: socat.x86_64 0:1.7.3.2-2.el7 2019-10-13T00:50:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2019-10-13T00:50:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2019-10-13T00:50:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-10-13T00:50:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel.x86_64 0:1.2.7-18.el7 2019-10-13T00:50:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:50:41.992 INFO:teuthology.orchestra.run.smithi142.stdout:Updated: 2019-10-13T00:50:41.993 INFO:teuthology.orchestra.run.smithi142.stdout: librados2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.993 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:41.993 INFO:teuthology.orchestra.run.smithi142.stdout: 2019-10-13T00:50:41.993 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2019-10-13T00:50:42.091 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 55/105 2019-10-13T00:50:42.156 DEBUG:teuthology.parallel:result is None 2019-10-13T00:50:42.216 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 56/105 2019-10-13T00:50:42.358 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 57/105 2019-10-13T00:50:42.484 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python-rgw-15.0.0-6132.ga8ee8ac.el7.x86_64 58/105 2019-10-13T00:50:42.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 59/105 2019-10-13T00:50:42.768 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcom_err-devel-1.42.9-16.el7.x86_64 60/105 2019-10-13T00:50:42.885 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : containernetworking-plugins-0.8.1-2.el7.x86_64 61/105 2019-10-13T00:50:43.044 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.5-1.el7.x86_64 62/105 2019-10-13T00:50:43.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 63/105 2019-10-13T00:50:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 64/105 2019-10-13T00:50:43.355 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 65/105 2019-10-13T00:50:43.463 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 66/105 2019-10-13T00:50:43.547 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 1:openssl-devel-1.0.2k-19.el7.x86_64 67/105 2019-10-13T00:50:43.630 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 68/105 2019-10-13T00:50:43.723 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:libcephfs-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 69/105 2019-10-13T00:50:43.848 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libnet-1.1.6-7.el7.x86_64 70/105 2019-10-13T00:50:43.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:librados-devel-15.0.0-6132.ga8ee8ac.el7.x86_64 71/105 2019-10-13T00:50:44.083 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:libcephfs2-15.0.0-6132.ga8ee8ac.el7.x86_64 72/105 2019-10-13T00:50:44.233 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 73/105 2019-10-13T00:50:44.342 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-common-15.0.0-6132.ga8ee8ac.el7.x86_64 74/105 2019-10-13T00:50:44.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 1:containers-common-0.1.37-3.el7.x86_64 75/105 2019-10-13T00:50:44.593 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-selinux-15.0.0-6132.ga8ee8ac.el7.x86_64 76/105 2019-10-13T00:50:44.702 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 77/105 2019-10-13T00:50:44.811 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:librbd1-15.0.0-6132.ga8ee8ac.el7.x86_64 78/105 2019-10-13T00:50:44.945 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : krb5-devel-1.15.1-37.el7_7.2.x86_64 79/105 2019-10-13T00:50:45.054 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 80/105 2019-10-13T00:50:45.146 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 81/105 2019-10-13T00:50:45.240 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 82/105 2019-10-13T00:50:45.363 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 83/105 2019-10-13T00:50:45.447 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-6132.ga8ee8ac.el7.noa 84/105 2019-10-13T00:50:45.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 85/105 2019-10-13T00:50:45.673 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python-ceph-argparse-15.0.0-6132.ga8ee8ac.el7.x86_64 86/105 2019-10-13T00:50:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:rbd-fuse-15.0.0-6132.ga8ee8ac.el7.x86_64 87/105 2019-10-13T00:50:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python-rados-15.0.0-6132.ga8ee8ac.el7.x86_64 88/105 2019-10-13T00:50:46.008 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 89/105 2019-10-13T00:50:46.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python2-kubernetes-8.0.1-1.el7.noarch 90/105 2019-10-13T00:50:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-15.0.0-6132.ga8ee8ac.el7.x86_64 91/105 2019-10-13T00:50:46.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-6132.ga8ee8ac.el7.noarch 92/105 2019-10-13T00:50:46.510 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.176-2.el7.x86_64 93/105 2019-10-13T00:50:46.818 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:libradosstriper1-15.0.0-6132.ga8ee8ac.el7.x86_64 94/105 2019-10-13T00:50:46.911 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:python36-cephfs-15.0.0-6132.ga8ee8ac.el7.x86_64 95/105 2019-10-13T00:50:46.996 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 2:ceph-mgr-15.0.0-6132.ga8ee8ac.el7.x86_64 96/105 2019-10-13T00:50:47.094 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 97/105 2019-10-13T00:50:47.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.0.4-2.el7.x86_64 98/105 2019-10-13T00:50:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 99/105 2019-10-13T00:50:47.414 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 100/105 2019-10-13T00:50:47.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.5.37-6.el7.x86_64 101/105 2019-10-13T00:50:47.682 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 102/105 2019-10-13T00:50:47.791 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 103/105 2019-10-13T00:50:47.791 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 104/105 2019-10-13T00:50:48.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 105/105 2019-10-13T00:50:48.153 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:50:48.153 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2019-10-13T00:50:48.154 INFO:teuthology.orchestra.run.smithi155.stdout: bison.x86_64 0:3.0.4-2.el7 2019-10-13T00:50:48.154 INFO:teuthology.orchestra.run.smithi155.stdout: ceph.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.154 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-daemon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.154 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.154 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.154 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.154 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.155 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.155 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.155 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-ssh.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.155 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.155 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.155 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel.x86_64 0:0.176-2.el7 2019-10-13T00:50:48.155 INFO:teuthology.orchestra.run.smithi155.stdout: flex.x86_64 0:2.5.37-6.el7 2019-10-13T00:50:48.156 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.156 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.156 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel.x86_64 1:1.0.2k-19.el7 2019-10-13T00:50:48.156 INFO:teuthology.orchestra.run.smithi155.stdout: python-ceph-compat.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.156 INFO:teuthology.orchestra.run.smithi155.stdout: python36-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.156 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.156 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout:Dependency Installed: 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.158 INFO:teuthology.orchestra.run.smithi155.stdout: container-selinux.noarch 2:2.107-3.el7 2019-10-13T00:50:48.158 INFO:teuthology.orchestra.run.smithi155.stdout: containernetworking-plugins.x86_64 0:0.8.1-2.el7 2019-10-13T00:50:48.158 INFO:teuthology.orchestra.run.smithi155.stdout: containers-common.x86_64 1:0.1.37-3.el7 2019-10-13T00:50:48.158 INFO:teuthology.orchestra.run.smithi155.stdout: criu.x86_64 0:3.12-2.el7 2019-10-13T00:50:48.158 INFO:teuthology.orchestra.run.smithi155.stdout: jq.x86_64 0:1.5-1.el7 2019-10-13T00:50:48.158 INFO:teuthology.orchestra.run.smithi155.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2019-10-13T00:50:48.158 INFO:teuthology.orchestra.run.smithi155.stdout: krb5-devel.x86_64 0:1.15.1-37.el7_7.2 2019-10-13T00:50:48.159 INFO:teuthology.orchestra.run.smithi155.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-10-13T00:50:48.159 INFO:teuthology.orchestra.run.smithi155.stdout: libcom_err-devel.x86_64 0:1.42.9-16.el7 2019-10-13T00:50:48.159 INFO:teuthology.orchestra.run.smithi155.stdout: libnet.x86_64 0:1.1.6-7.el7 2019-10-13T00:50:48.159 INFO:teuthology.orchestra.run.smithi155.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-10-13T00:50:48.159 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-10-13T00:50:48.159 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.159 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.160 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.160 INFO:teuthology.orchestra.run.smithi155.stdout: libselinux-devel.x86_64 0:2.5-14.1.el7 2019-10-13T00:50:48.160 INFO:teuthology.orchestra.run.smithi155.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2019-10-13T00:50:48.160 INFO:teuthology.orchestra.run.smithi155.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2019-10-13T00:50:48.160 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-10-13T00:50:48.160 INFO:teuthology.orchestra.run.smithi155.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2019-10-13T00:50:48.160 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma.x86_64 0:5.9.5-3.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: pcre-devel.x86_64 0:8.32-17.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: podman.x86_64 0:1.4.4-4.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-c.x86_64 0:1.0.2-3.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: python-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: python-ceph-common.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.161 INFO:teuthology.orchestra.run.smithi155.stdout: python-cephfs.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-10-13T00:50:48.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-execnet.noarch 0:1.2.0-7.el7 2019-10-13T00:50:48.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-10-13T00:50:48.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-10-13T00:50:48.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-10-13T00:50:48.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-10-13T00:50:48.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-10-13T00:50:48.163 INFO:teuthology.orchestra.run.smithi155.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-10-13T00:50:48.163 INFO:teuthology.orchestra.run.smithi155.stdout: python-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.163 INFO:teuthology.orchestra.run.smithi155.stdout: python-rbd.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.163 INFO:teuthology.orchestra.run.smithi155.stdout: python-remoto.noarch 0:0.0.25-1.el7 2019-10-13T00:50:48.163 INFO:teuthology.orchestra.run.smithi155.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-10-13T00:50:48.163 INFO:teuthology.orchestra.run.smithi155.stdout: python-requests.noarch 0:2.6.0-5.el7 2019-10-13T00:50:48.163 INFO:teuthology.orchestra.run.smithi155.stdout: python-rgw.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: python-routes.noarch 0:1.13-2.el7 2019-10-13T00:50:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-10-13T00:50:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-10-13T00:50:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-10-13T00:50:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-10-13T00:50:48.165 INFO:teuthology.orchestra.run.smithi155.stdout: python2-kubernetes.noarch 0:8.0.1-1.el7 2019-10-13T00:50:48.166 INFO:teuthology.orchestra.run.smithi155.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-10-13T00:50:48.166 INFO:teuthology.orchestra.run.smithi155.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-10-13T00:50:48.166 INFO:teuthology.orchestra.run.smithi155.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-10-13T00:50:48.166 INFO:teuthology.orchestra.run.smithi155.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-10-13T00:50:48.166 INFO:teuthology.orchestra.run.smithi155.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-10-13T00:50:48.166 INFO:teuthology.orchestra.run.smithi155.stdout: python3.x86_64 0:3.6.8-10.el7 2019-10-13T00:50:48.166 INFO:teuthology.orchestra.run.smithi155.stdout: python3-libs.x86_64 0:3.6.8-10.el7 2019-10-13T00:50:48.167 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pip.noarch 0:9.0.3-5.el7 2019-10-13T00:50:48.167 INFO:teuthology.orchestra.run.smithi155.stdout: python3-setuptools.noarch 0:39.2.0-10.el7 2019-10-13T00:50:48.167 INFO:teuthology.orchestra.run.smithi155.stdout: python36-ceph-argparse.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.167 INFO:teuthology.orchestra.run.smithi155.stdout: python36-rados.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.167 INFO:teuthology.orchestra.run.smithi155.stdout: runc.x86_64 0:1.0.0-65.rc8.el7 2019-10-13T00:50:48.167 INFO:teuthology.orchestra.run.smithi155.stdout: scipy.x86_64 0:0.12.1-6.el7 2019-10-13T00:50:48.167 INFO:teuthology.orchestra.run.smithi155.stdout: socat.x86_64 0:1.7.3.2-2.el7 2019-10-13T00:50:48.168 INFO:teuthology.orchestra.run.smithi155.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2019-10-13T00:50:48.168 INFO:teuthology.orchestra.run.smithi155.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2019-10-13T00:50:48.168 INFO:teuthology.orchestra.run.smithi155.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-10-13T00:50:48.168 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel.x86_64 0:1.2.7-18.el7 2019-10-13T00:50:48.168 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:50:48.168 INFO:teuthology.orchestra.run.smithi155.stdout:Updated: 2019-10-13T00:50:48.168 INFO:teuthology.orchestra.run.smithi155.stdout: librados2.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.169 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1.x86_64 2:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.169 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:50:48.169 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2019-10-13T00:50:48.331 DEBUG:teuthology.parallel:result is None 2019-10-13T00:50:48.331 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:48.331 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2019-10-13T00:50:48.424 INFO:teuthology.orchestra.run.smithi155.stdout:RedHatEnterpriseServer 2019-10-13T00:50:48.425 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2019-10-13T00:50:48.425 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T00:50:48.614 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:48.615 INFO:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-10-13T00:50:48.680 INFO:teuthology.orchestra.run.smithi155.stdout:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.681 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:48.681 INFO:teuthology.task.install:The correct ceph version 15.0.0-6132.ga8ee8ac is installed. 2019-10-13T00:50:48.681 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:48.681 INFO:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2019-10-13T00:50:48.791 INFO:teuthology.orchestra.run.smithi142.stdout:RedHatEnterpriseServer 2019-10-13T00:50:48.792 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2019-10-13T00:50:48.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T00:50:48.956 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:48.957 INFO:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-10-13T00:50:49.024 INFO:teuthology.orchestra.run.smithi142.stdout:15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:49.024 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6132.ga8ee8ac.el7 2019-10-13T00:50:49.024 INFO:teuthology.task.install:The correct ceph version 15.0.0-6132.ga8ee8ac is installed. 2019-10-13T00:50:49.025 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-10-13T00:50:49.025 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.025 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-13T00:50:49.080 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.080 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-13T00:50:49.135 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-10-13T00:50:49.135 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.135 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-13T00:50:49.190 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.190 INFO:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-13T00:50:49.242 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.242 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-13T00:50:49.295 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.295 INFO:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-13T00:50:49.348 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-10-13T00:50:49.348 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.348 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-13T00:50:49.402 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.402 INFO:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-13T00:50:49.517 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.517 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-13T00:50:49.571 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.571 INFO:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-13T00:50:49.687 INFO:teuthology.run_tasks:Running task ceph... 2019-10-13T00:50:49.760 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-10-13T00:50:49.760 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.760 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 777 /var/log/ceph 2019-10-13T00:50:49.765 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.765 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2019-10-13T00:50:49.817 INFO:tasks.ceph:Disabling ceph logrotate... 2019-10-13T00:50:49.818 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.818 INFO:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-13T00:50:49.822 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.822 INFO:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-13T00:50:49.876 INFO:tasks.ceph:Creating extra log directories... 2019-10-13T00:50:49.876 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:49.876 INFO:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-13T00:50:49.881 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.882 INFO:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-13T00:50:49.934 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2019-10-13T00:50:49.934 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2019-10-13T00:50:49.934 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2019-10-13T00:50:49.935 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.935 INFO:teuthology.orchestra.run.smithi155:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-10-13T00:50:49.979 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:49.979 INFO:teuthology.orchestra.run.smithi155:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-10-13T00:50:50.144 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:50.144 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2019-10-13T00:50:50.196 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:50.196 INFO:teuthology.orchestra.run.smithi142:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-10-13T00:50:50.242 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:50.242 INFO:teuthology.orchestra.run.smithi142:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-10-13T00:50:50.418 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:50.419 INFO:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2019-10-13T00:50:50.471 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-10-13T00:50:50.471 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'filestore flush min': 0, 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'osd_max_omap_entries_per_request': 10, 'debug journal': 20, 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'osd op complaint time': 180}}} 2019-10-13T00:50:50.472 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}}, {'ceph': None}, {'ceph-fuse': None}, {'check-counter': {'counters': {'mds': ['mds.dir_split']}}}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.15246', 'owner': 'scheduled_pdonnell@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'targets': {'smithi155.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDj86cZDNkWAQHRr0k23eESV3p4ff7NEOn98aDo43SUQjWuFJJj77k92Q87d+PRT+c6kdVST5Za1q/WT2IYGc6c6SPTd3VhyHLoxpKkv79uDKS4EKA6X0s9YRLeO0o+yBRXO+FNKM1kNe1ej0EvZ/M0LzK1ALGLhmSIqKG49odtZpayMlY+CT593k+46XVA/ETB0IhM0ZQv1d+4FoA5CcdOHxtyaQleBa2qi39EJbhiqA8eZYtdSDNaCiMHlv/OWkTa5ffKZI9xb3PT/N8TM5/8Hfl6W9tQSJT2SU5HnyQAjx/FYCC5VgyoyhfLXuN7oxncmKtTSbyzjdMvKyo2SJH/', 'smithi142.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4b0St8zDSR0B4D85NtVDdeb+fY0k53IcgxQK2TbuOxZW7phtL+y2+Tpdj/vQsbpO0D8v3hNQ0BNmUT7dFmyJaGcXJEPMtTNY4BseMWc7K2b44JrQWrAoZ9wm0EqRPFTcApOeIzfbUmMAhT4aGtr4/sOVthRtofIEZVMonQAEDroa3mHBf2kFgUvhPHLB6vZPWvRBjVwb6fdGoyV61xw5Bhnt4N5eOptQPPFfcFvsjIT6Fb7mK6vP/P18RJVJOxMpJQewPXZ6JwDkFDTALoXRckijE4pjB6+QLA1obNuPzOS2JahGuVi6Vr/8vLFLpl27GxiUwsBEWI7QQBnsziqLb'}, 'job_id': '4407525', 'tube': 'smithi', 'suite_branch': 'wip-pdonnell-testing-20191012.025926', 'priority': 95, 'os_version': '7.7', 'first_in_suite': False, 'branch': 'wip-pdonnell-testing-20191012.025926', 'suite': 'fs', 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'email': 'pdonnell@redhat.com', 'archive_path': '/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407525', 'description': 'fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml omap_limit/10.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{rhel_7.yaml} tasks/cfuse_workunit_suites_ffsb.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'name': 'pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'overrides': {'ceph': {'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'filestore flush min': 0, 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'osd_max_omap_entries_per_request': 10, 'debug journal': 20, 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'osd op complaint time': 180}}, 'fs': 'xfs', 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}}, 'admin_socket': {'branch': 'wip-pdonnell-testing-20191012.025926'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'branch': 'wip-pdonnell-testing-20191012.025926'}}, 'os_type': 'rhel', 'suite_sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'} 2019-10-13T00:50:50.472 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:50.472 INFO:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-13T00:50:50.476 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:50.476 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-13T00:50:50.517 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:50.518 INFO:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/run/ceph 2019-10-13T00:50:50.588 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:50.588 INFO:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2019-10-13T00:50:50.683 DEBUG:teuthology.orchestra.remote:smithi155:/scratch_devs is 72B 2019-10-13T00:50:50.733 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T00:50:50.733 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:50.734 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-13T00:50:50.773 INFO:teuthology.orchestra.run.smithi155.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-10-13T00:50:50.774 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:50.774 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 111760 Links: 1 2019-10-13T00:50:50.774 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:50.774 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:50.774 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2019-10-13 00:50:26.946379684 +0000 2019-10-13T00:50:50.774 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2019-10-13 00:47:42.039758805 +0000 2019-10-13T00:50:50.774 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2019-10-13 00:47:42.039758805 +0000 2019-10-13T00:50:50.775 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2019-10-13T00:50:50.785 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2019-10-13T00:50:50.786 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2019-10-13T00:50:50.786 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes (512 B) copied, 0.000379384 s, 1.3 MB/s 2019-10-13T00:50:50.788 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:50.788 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-13T00:50:50.892 INFO:teuthology.orchestra.run.smithi155.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-10-13T00:50:50.892 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:50.893 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 110796 Links: 1 2019-10-13T00:50:50.893 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:50.893 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:50.893 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2019-10-13 00:50:26.946379684 +0000 2019-10-13T00:50:50.894 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2019-10-13 00:47:42.315755170 +0000 2019-10-13T00:50:50.894 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2019-10-13 00:47:42.315755170 +0000 2019-10-13T00:50:50.894 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2019-10-13T00:50:50.904 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2019-10-13T00:50:50.904 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2019-10-13T00:50:50.904 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes (512 B) copied, 0.000236831 s, 2.2 MB/s 2019-10-13T00:50:50.912 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:50.912 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-13T00:50:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-10-13T00:50:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:51.019 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 111819 Links: 1 2019-10-13T00:50:51.019 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:51.020 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:51.020 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2019-10-13 00:50:26.946379684 +0000 2019-10-13T00:50:51.020 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2019-10-13 00:47:42.582751654 +0000 2019-10-13T00:50:51.020 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2019-10-13 00:47:42.582751654 +0000 2019-10-13T00:50:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2019-10-13T00:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2019-10-13T00:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2019-10-13T00:50:51.035 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes (512 B) copied, 0.000310026 s, 1.7 MB/s 2019-10-13T00:50:51.035 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:51.035 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-13T00:50:51.115 INFO:teuthology.orchestra.run.smithi155.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-10-13T00:50:51.115 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:51.116 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 110924 Links: 1 2019-10-13T00:50:51.116 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:51.116 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:51.116 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2019-10-13 00:50:26.946379684 +0000 2019-10-13T00:50:51.116 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2019-10-13 00:47:42.861747979 +0000 2019-10-13T00:50:51.116 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2019-10-13 00:47:42.861747979 +0000 2019-10-13T00:50:51.117 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2019-10-13T00:50:51.127 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2019-10-13T00:50:51.127 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2019-10-13T00:50:51.127 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes (512 B) copied, 0.00025243 s, 2.0 MB/s 2019-10-13T00:50:51.129 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-13T00:50:51.130 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T00:50:51.130 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:51.130 INFO:teuthology.orchestra.run.smithi155:> ls -l /dev/disk/by-id/wwn-* 2019-10-13T00:50:51.236 INFO:teuthology.orchestra.run.smithi155.stdout:lrwxrwxrwx. 1 root root 9 Oct 13 00:38 /dev/disk/by-id/wwn-0x5000c5009294102d -> ../../sda 2019-10-13T00:50:51.237 INFO:teuthology.orchestra.run.smithi155.stdout:lrwxrwxrwx. 1 root root 10 Oct 13 00:38 /dev/disk/by-id/wwn-0x5000c5009294102d-part1 -> ../../sda1 2019-10-13T00:50:51.237 INFO:tasks.ceph:dev map: {} 2019-10-13T00:50:51.284 DEBUG:teuthology.orchestra.remote:smithi142:/scratch_devs is 72B 2019-10-13T00:50:51.332 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T00:50:51.333 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:51.333 INFO:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-13T00:50:51.374 INFO:teuthology.orchestra.run.smithi142.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-10-13T00:50:51.375 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:51.375 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 105766 Links: 1 2019-10-13T00:50:51.375 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:51.375 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:51.375 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2019-10-13 00:50:22.417196007 +0000 2019-10-13T00:50:51.375 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2019-10-13 00:47:41.798688964 +0000 2019-10-13T00:50:51.376 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2019-10-13 00:47:41.798688964 +0000 2019-10-13T00:50:51.376 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2019-10-13T00:50:51.387 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2019-10-13T00:50:51.387 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2019-10-13T00:50:51.387 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes (512 B) copied, 0.000268517 s, 1.9 MB/s 2019-10-13T00:50:51.389 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:51.389 INFO:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-13T00:50:51.498 INFO:teuthology.orchestra.run.smithi142.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-10-13T00:50:51.498 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:51.498 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 109279 Links: 1 2019-10-13T00:50:51.498 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:51.498 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:51.499 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2019-10-13 00:50:22.417196007 +0000 2019-10-13T00:50:51.499 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2019-10-13 00:47:42.076682831 +0000 2019-10-13T00:50:51.499 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2019-10-13 00:47:42.076682831 +0000 2019-10-13T00:50:51.499 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2019-10-13T00:50:51.509 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2019-10-13T00:50:51.510 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2019-10-13T00:50:51.510 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes (512 B) copied, 0.000208896 s, 2.5 MB/s 2019-10-13T00:50:51.512 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:51.512 INFO:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-13T00:50:51.619 INFO:teuthology.orchestra.run.smithi142.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-10-13T00:50:51.619 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:51.619 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 105829 Links: 1 2019-10-13T00:50:51.619 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:51.619 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:51.619 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2019-10-13 00:50:22.417196007 +0000 2019-10-13T00:50:51.620 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2019-10-13 00:47:42.359676586 +0000 2019-10-13T00:50:51.620 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2019-10-13 00:47:42.359676586 +0000 2019-10-13T00:50:51.620 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2019-10-13T00:50:51.631 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2019-10-13T00:50:51.631 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2019-10-13T00:50:51.631 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes (512 B) copied, 0.000299685 s, 1.7 MB/s 2019-10-13T00:50:51.633 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:51.633 INFO:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-13T00:50:51.737 INFO:teuthology.orchestra.run.smithi142.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-10-13T00:50:51.738 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T00:50:51.738 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 111964 Links: 1 2019-10-13T00:50:51.738 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:50:51.738 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2019-10-13T00:50:51.738 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2019-10-13 00:50:22.417196007 +0000 2019-10-13T00:50:51.739 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2019-10-13 00:47:42.626670695 +0000 2019-10-13T00:50:51.739 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2019-10-13 00:47:42.626670695 +0000 2019-10-13T00:50:51.739 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2019-10-13T00:50:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2019-10-13T00:50:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2019-10-13T00:50:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes (512 B) copied, 0.000287907 s, 1.8 MB/s 2019-10-13T00:50:51.752 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-13T00:50:51.752 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T00:50:51.752 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:51.752 INFO:teuthology.orchestra.run.smithi142:> ls -l /dev/disk/by-id/wwn-* 2019-10-13T00:50:51.856 INFO:teuthology.orchestra.run.smithi142.stdout:lrwxrwxrwx. 1 root root 9 Oct 13 00:38 /dev/disk/by-id/wwn-0x5000c5009292426c -> ../../sda 2019-10-13T00:50:51.856 INFO:teuthology.orchestra.run.smithi142.stdout:lrwxrwxrwx. 1 root root 10 Oct 13 00:38 /dev/disk/by-id/wwn-0x5000c5009292426c-part1 -> ../../sda1 2019-10-13T00:50:51.856 INFO:tasks.ceph:dev map: {} 2019-10-13T00:50:51.856 INFO:tasks.ceph:Generating config... 2019-10-13T00:50:51.857 INFO:tasks.ceph:[mds] mds debug frag = True 2019-10-13T00:50:51.858 INFO:tasks.ceph:[mds] mds bal split bits = 3 2019-10-13T00:50:51.858 INFO:tasks.ceph:[mds] mds bal split size = 100 2019-10-13T00:50:51.858 INFO:tasks.ceph:[mds] debug mds = 20 2019-10-13T00:50:51.858 INFO:tasks.ceph:[mds] mds bal merge size = 5 2019-10-13T00:50:51.858 INFO:tasks.ceph:[mds] debug ms = 1 2019-10-13T00:50:51.858 INFO:tasks.ceph:[mds] mds bal frag = True 2019-10-13T00:50:51.858 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2019-10-13T00:50:51.859 INFO:tasks.ceph:[mds] mds op complaint time = 180 2019-10-13T00:50:51.859 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2019-10-13T00:50:51.859 INFO:tasks.ceph:[mds] osd op complaint time = 180 2019-10-13T00:50:51.859 INFO:tasks.ceph:[mds] mds verify scatter = True 2019-10-13T00:50:51.859 INFO:tasks.ceph:[client] debug ms = 1 2019-10-13T00:50:51.859 INFO:tasks.ceph:[client] debug client = 20 2019-10-13T00:50:51.859 INFO:tasks.ceph:[client] client mount timeout = 600 2019-10-13T00:50:51.860 INFO:tasks.ceph:[mon] debug mon = 20 2019-10-13T00:50:51.860 INFO:tasks.ceph:[mon] debug paxos = 20 2019-10-13T00:50:51.860 INFO:tasks.ceph:[mon] mon op complaint time = 120 2019-10-13T00:50:51.860 INFO:tasks.ceph:[mon] debug ms = 1 2019-10-13T00:50:51.860 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-10-13T00:50:51.860 INFO:tasks.ceph:[osd] debug ms = 1 2019-10-13T00:50:51.860 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] filestore flush min = 0 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] debug osd = 25 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] osd_max_omap_entries_per_request = 10 2019-10-13T00:50:51.861 INFO:tasks.ceph:[osd] debug journal = 20 2019-10-13T00:50:51.862 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-10-13T00:50:51.862 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-10-13T00:50:51.862 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-10-13T00:50:51.862 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-10-13T00:50:51.862 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-10-13T00:50:51.862 INFO:tasks.ceph:[osd] debug filestore = 20 2019-10-13T00:50:51.862 INFO:tasks.ceph:[osd] osd op complaint time = 180 2019-10-13T00:50:51.863 INFO:tasks.ceph:Setting up mon.a... 2019-10-13T00:50:51.863 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:51.863 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-10-13T00:50:52.088 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.keyring 2019-10-13T00:50:52.091 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:52.092 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-10-13T00:50:52.166 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:52.166 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T00:50:52.219 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.155'), ('mon.c', '[v2:172.21.15.142:3301,v1:172.21.15.142:6790]'), ('mon.b', '172.21.15.142')] 2019-10-13T00:50:52.220 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:52.220 INFO:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.155 --addv c '[v2:172.21.15.142:3301,v1:172.21.15.142:6790]' --add b 172.21.15.142 --print /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:50:52.288 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:50:52.288 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: generated fsid 9c6ba46f-16c8-4ed6-8640-ff20acbbecb0 2019-10-13T00:50:52.288 INFO:teuthology.orchestra.run.smithi155.stdout:epoch 0 2019-10-13T00:50:52.288 INFO:teuthology.orchestra.run.smithi155.stdout:fsid 9c6ba46f-16c8-4ed6-8640-ff20acbbecb0 2019-10-13T00:50:52.288 INFO:teuthology.orchestra.run.smithi155.stdout:last_changed 2019-10-13T00:50:52.289530+0000 2019-10-13T00:50:52.289 INFO:teuthology.orchestra.run.smithi155.stdout:created 2019-10-13T00:50:52.289530+0000 2019-10-13T00:50:52.289 INFO:teuthology.orchestra.run.smithi155.stdout:min_mon_release 0 (unknown) 2019-10-13T00:50:52.289 INFO:teuthology.orchestra.run.smithi155.stdout:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2019-10-13T00:50:52.289 INFO:teuthology.orchestra.run.smithi155.stdout:1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2019-10-13T00:50:52.289 INFO:teuthology.orchestra.run.smithi155.stdout:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2019-10-13T00:50:52.289 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-10-13T00:50:52.290 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9c6ba46f-16c8-4ed6-8640-ff20acbbecb0... 2019-10-13T00:50:52.291 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:52.291 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-13T00:50:52.296 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:52.297 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-13T00:50:52.396 INFO:tasks.ceph:Creating admin key on mon.a... 2019-10-13T00:50:52.398 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:52.398 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-10-13T00:50:52.478 INFO:tasks.ceph:Copying monmap to all nodes... 2019-10-13T00:50:52.521 DEBUG:teuthology.orchestra.remote:smithi155:/etc/ceph/ceph.keyring is 206B 2019-10-13T00:50:52.613 DEBUG:teuthology.orchestra.remote:smithi155:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-10-13T00:50:52.662 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2019-10-13T00:50:52.663 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:52.663 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T00:50:52.728 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:52.728 INFO:teuthology.orchestra.run.smithi155:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:50:52.836 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi142.front.sepia.ceph.com 2019-10-13T00:50:52.836 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:52.836 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T00:50:52.912 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:52.912 INFO:teuthology.orchestra.run.smithi142:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:50:53.021 INFO:tasks.ceph:Setting up mon nodes... 2019-10-13T00:50:53.021 INFO:tasks.ceph:Setting up mgr nodes... 2019-10-13T00:50:53.022 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.022 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-10-13T00:50:53.101 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-10-13T00:50:53.103 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:53.104 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-10-13T00:50:53.203 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-10-13T00:50:53.206 INFO:tasks.ceph:Setting up mds nodes... 2019-10-13T00:50:53.206 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.206 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2019-10-13T00:50:53.285 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2019-10-13T00:50:53.287 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.288 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2019-10-13T00:50:53.339 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:53.339 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2019-10-13T00:50:53.419 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2019-10-13T00:50:53.426 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:50:53.426 INFO:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2019-10-13T00:50:53.478 INFO:tasks.ceph_client:Setting up client nodes... 2019-10-13T00:50:53.479 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.479 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-10-13T00:50:53.545 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-10-13T00:50:53.560 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-10-13T00:50:53.560 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {}} 2019-10-13T00:50:53.560 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.560 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-10-13T00:50:53.612 INFO:tasks.ceph:{} 2019-10-13T00:50:53.612 INFO:tasks.ceph:{} 2019-10-13T00:50:53.612 INFO:tasks.ceph:osd.0 2019-10-13T00:50:53.613 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.613 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-10-13T00:50:53.664 INFO:tasks.ceph:{} 2019-10-13T00:50:53.664 INFO:tasks.ceph:{} 2019-10-13T00:50:53.665 INFO:tasks.ceph:osd.1 2019-10-13T00:50:53.665 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.665 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-10-13T00:50:53.721 INFO:tasks.ceph:{} 2019-10-13T00:50:53.722 INFO:tasks.ceph:{} 2019-10-13T00:50:53.722 INFO:tasks.ceph:osd.2 2019-10-13T00:50:53.724 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.724 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-10-13T00:50:53.776 INFO:tasks.ceph:{} 2019-10-13T00:50:53.776 INFO:tasks.ceph:{} 2019-10-13T00:50:53.776 INFO:tasks.ceph:osd.3 2019-10-13T00:50:53.776 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:53.777 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:50:53.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:53.848+0000 7efcd9af0bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-10-13T00:50:53.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:53.848+0000 7efcd9af0bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-10-13T00:50:53.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:53.848+0000 7efcd9af0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T00:50:53.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:53.848+0000 7efcd9af0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T00:50:53.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:53.848+0000 7efcd9af0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T00:50:53.848 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:53.849+0000 7efcd9af0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-10-13T00:50:56.133 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:56.133 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-10-13T00:50:56.188 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:56.188 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:50:56.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:56.261+0000 7f86be7d7bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-10-13T00:50:56.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:56.261+0000 7f86be7d7bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-10-13T00:50:56.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:56.261+0000 7f86be7d7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T00:50:56.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:56.261+0000 7f86be7d7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T00:50:56.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:56.261+0000 7f86be7d7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T00:50:56.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:56.261+0000 7f86be7d7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-10-13T00:50:58.543 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:58.544 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-10-13T00:50:58.600 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:50:58.600 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:50:58.671 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:58.672+0000 7f1b0fa33bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-10-13T00:50:58.671 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:58.673+0000 7f1b0fa33bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-10-13T00:50:58.671 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:58.673+0000 7f1b0fa33bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T00:50:58.671 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:58.673+0000 7f1b0fa33bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T00:50:58.671 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:58.673+0000 7f1b0fa33bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T00:50:58.672 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:50:58.673+0000 7f1b0fa33bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-10-13T00:51:00.960 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:00.960 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-10-13T00:51:01.015 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:01.015 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:51:01.086 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:01.087+0000 7f0b8a7b0bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-10-13T00:51:01.086 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:01.088+0000 7f0b8a7b0bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-10-13T00:51:01.086 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:01.088+0000 7f0b8a7b0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T00:51:01.086 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:01.088+0000 7f0b8a7b0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T00:51:01.087 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:01.088+0000 7f0b8a7b0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T00:51:01.087 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:01.088+0000 7f0b8a7b0bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-10-13T00:51:03.373 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:03.373 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-10-13T00:51:03.428 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:03.428 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-10-13T00:51:03.487 INFO:tasks.ceph:{} 2019-10-13T00:51:03.488 INFO:tasks.ceph:{} 2019-10-13T00:51:03.488 INFO:tasks.ceph:osd.4 2019-10-13T00:51:03.488 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:03.488 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-10-13T00:51:03.543 INFO:tasks.ceph:{} 2019-10-13T00:51:03.543 INFO:tasks.ceph:{} 2019-10-13T00:51:03.543 INFO:tasks.ceph:osd.5 2019-10-13T00:51:03.543 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:03.544 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-10-13T00:51:03.596 INFO:tasks.ceph:{} 2019-10-13T00:51:03.596 INFO:tasks.ceph:{} 2019-10-13T00:51:03.596 INFO:tasks.ceph:osd.6 2019-10-13T00:51:03.597 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:03.597 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-10-13T00:51:03.648 INFO:tasks.ceph:{} 2019-10-13T00:51:03.649 INFO:tasks.ceph:{} 2019-10-13T00:51:03.649 INFO:tasks.ceph:osd.7 2019-10-13T00:51:03.649 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:03.649 INFO:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:51:03.719 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:03.720+0000 7f98dbe6fbc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-10-13T00:51:03.719 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:03.720+0000 7f98dbe6fbc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-10-13T00:51:03.720 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:03.720+0000 7f98dbe6fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T00:51:03.720 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:03.720+0000 7f98dbe6fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T00:51:03.720 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:03.720+0000 7f98dbe6fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T00:51:03.720 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:03.721+0000 7f98dbe6fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-10-13T00:51:06.013 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:06.013 INFO:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-10-13T00:51:06.067 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:06.067 INFO:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:51:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:06.139+0000 7f2d1bd68bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-10-13T00:51:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:06.139+0000 7f2d1bd68bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-10-13T00:51:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:06.139+0000 7f2d1bd68bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T00:51:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:06.139+0000 7f2d1bd68bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T00:51:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:06.139+0000 7f2d1bd68bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T00:51:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:06.139+0000 7f2d1bd68bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-10-13T00:51:08.422 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:08.423 INFO:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-10-13T00:51:08.476 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:08.476 INFO:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:51:08.547 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:08.548+0000 7fa12f258bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-10-13T00:51:08.547 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:08.548+0000 7fa12f258bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-10-13T00:51:08.547 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:08.548+0000 7fa12f258bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T00:51:08.547 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:08.548+0000 7fa12f258bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T00:51:08.547 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:08.548+0000 7fa12f258bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T00:51:08.547 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:08.548+0000 7fa12f258bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2019-10-13T00:51:10.835 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:10.836 INFO:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-10-13T00:51:10.894 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:10.894 INFO:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:51:10.967 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:10.968+0000 7f87df579bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-10-13T00:51:10.967 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:10.968+0000 7f87df579bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-10-13T00:51:10.967 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:10.968+0000 7f87df579bc0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T00:51:10.967 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:10.968+0000 7f87df579bc0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T00:51:10.968 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:10.968+0000 7f87df579bc0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T00:51:10.968 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:10.968+0000 7f87df579bc0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2019-10-13T00:51:13.256 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:13.256 INFO:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-10-13T00:51:13.320 INFO:tasks.ceph:Reading keys from all nodes... 2019-10-13T00:51:13.320 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.320 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:13.367 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.qc0V6jap9t 2019-10-13T00:51:13.367 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.368 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.qc0V6jap9t 2019-10-13T00:51:13.422 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.423 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.qc0V6jap9t 2019-10-13T00:51:13.518 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.qc0V6jap9t is 56B 2019-10-13T00:51:13.565 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.565 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.qc0V6jap9t 2019-10-13T00:51:13.605 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.605 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:13.709 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.B43az3mSJq 2019-10-13T00:51:13.709 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.710 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.B43az3mSJq 2019-10-13T00:51:13.824 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.824 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.B43az3mSJq 2019-10-13T00:51:13.918 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.B43az3mSJq is 56B 2019-10-13T00:51:13.966 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:13.966 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.B43az3mSJq 2019-10-13T00:51:14.006 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.006 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:14.109 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.yGOLeKwWjQ 2019-10-13T00:51:14.110 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.110 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.yGOLeKwWjQ 2019-10-13T00:51:14.225 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.225 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.yGOLeKwWjQ 2019-10-13T00:51:14.321 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.yGOLeKwWjQ is 56B 2019-10-13T00:51:14.369 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.370 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.yGOLeKwWjQ 2019-10-13T00:51:14.409 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.409 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:14.513 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.0WqxO6B5Nq 2019-10-13T00:51:14.514 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.514 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.0WqxO6B5Nq 2019-10-13T00:51:14.630 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.630 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.0WqxO6B5Nq 2019-10-13T00:51:14.725 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.0WqxO6B5Nq is 56B 2019-10-13T00:51:14.773 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.774 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.0WqxO6B5Nq 2019-10-13T00:51:14.813 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.813 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:14.917 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.PFAFqmG17c 2019-10-13T00:51:14.918 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:14.918 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.PFAFqmG17c 2019-10-13T00:51:15.034 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:15.034 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.PFAFqmG17c 2019-10-13T00:51:15.128 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.PFAFqmG17c is 56B 2019-10-13T00:51:15.177 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:15.177 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.PFAFqmG17c 2019-10-13T00:51:15.216 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:15.216 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:15.320 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.hp06O5MJsE 2019-10-13T00:51:15.320 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:15.321 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.hp06O5MJsE 2019-10-13T00:51:15.438 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:15.439 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.hp06O5MJsE 2019-10-13T00:51:15.532 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.hp06O5MJsE is 56B 2019-10-13T00:51:15.582 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:15.582 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.hp06O5MJsE 2019-10-13T00:51:15.622 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:15.622 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:15.667 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.InRUHMcG4E 2019-10-13T00:51:15.667 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:15.667 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.InRUHMcG4E 2019-10-13T00:51:15.786 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:15.786 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.InRUHMcG4E 2019-10-13T00:51:15.881 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.InRUHMcG4E is 56B 2019-10-13T00:51:15.932 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:15.932 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.InRUHMcG4E 2019-10-13T00:51:15.972 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:15.972 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:16.075 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.XMRTbuQTHI 2019-10-13T00:51:16.076 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.076 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.XMRTbuQTHI 2019-10-13T00:51:16.191 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.191 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.XMRTbuQTHI 2019-10-13T00:51:16.285 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.XMRTbuQTHI is 56B 2019-10-13T00:51:16.333 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.333 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.XMRTbuQTHI 2019-10-13T00:51:16.373 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.373 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:16.477 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.5ibsnbXsdv 2019-10-13T00:51:16.477 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.477 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.5ibsnbXsdv 2019-10-13T00:51:16.592 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.592 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.5ibsnbXsdv 2019-10-13T00:51:16.687 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.5ibsnbXsdv is 56B 2019-10-13T00:51:16.736 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.737 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.5ibsnbXsdv 2019-10-13T00:51:16.776 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.776 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:16.880 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.MJUuxMEwn4 2019-10-13T00:51:16.880 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.880 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.MJUuxMEwn4 2019-10-13T00:51:16.997 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:16.997 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.MJUuxMEwn4 2019-10-13T00:51:17.091 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.MJUuxMEwn4 is 56B 2019-10-13T00:51:17.140 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.140 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.MJUuxMEwn4 2019-10-13T00:51:17.180 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.180 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:17.283 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.tfy7OiCmgv 2019-10-13T00:51:17.283 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.283 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.tfy7OiCmgv 2019-10-13T00:51:17.399 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.399 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.tfy7OiCmgv 2019-10-13T00:51:17.493 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.tfy7OiCmgv is 56B 2019-10-13T00:51:17.541 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.541 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.tfy7OiCmgv 2019-10-13T00:51:17.581 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.581 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:17.684 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.DZl8JrbPKb 2019-10-13T00:51:17.684 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.685 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.DZl8JrbPKb 2019-10-13T00:51:17.800 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.801 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.DZl8JrbPKb 2019-10-13T00:51:17.895 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.DZl8JrbPKb is 56B 2019-10-13T00:51:17.945 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:17.945 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.DZl8JrbPKb 2019-10-13T00:51:18.034 DEBUG:teuthology.orchestra.remote:smithi155:/etc/ceph/ceph.client.0.keyring is 59B 2019-10-13T00:51:18.082 INFO:tasks.ceph:Adding keys to all mons... 2019-10-13T00:51:18.083 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.083 INFO:teuthology.orchestra.run.smithi142:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-13T00:51:18.087 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.087 INFO:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-13T00:51:18.133 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.y] 2019-10-13T00:51:18.133 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDtdKJdVw8xBhAAaYE/d1621HpTw+ZCBhSKHQ== 2019-10-13T00:51:18.133 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.a] 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDtdKJd0/EnERAAxAs8ZZ2DfIZD8NZJ9qWrEw== 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.0] 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDtdKJdRZudMhAA65/XpP9B9fA3oknfTBNrvA== 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.1] 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDwdKJdN3GWDxAALcWhAOsxVehGn9JU8RPmNw== 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.2] 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDydKJdnk8jKBAAmd2YYSGqNiW4gKLmzBSZfg== 2019-10-13T00:51:18.134 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.3] 2019-10-13T00:51:18.135 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQD1dKJdD4ZDBRAAnyD1ZUUYV0kx9qznXEGzeg== 2019-10-13T00:51:18.135 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.x] 2019-10-13T00:51:18.135 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDtdKJd5BlADBAAyWbQaPfaeO137hQMLOVKVA== 2019-10-13T00:51:18.135 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.b] 2019-10-13T00:51:18.135 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDtdKJdAWIdGRAAkIhPh6HTUGjEMOd4bQyKng== 2019-10-13T00:51:18.135 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.4] 2019-10-13T00:51:18.135 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQD3dKJd5Rj9KhAAkobh4pIH5d/oW//T9kRTfw== 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.5] 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQD6dKJdw0VVCBAASyk/x6ARp3V/N+nowjxxOg== 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.6] 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQD8dKJdDAu4IBAAPr4xAlwPDBcBCEgiD6oSFw== 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.7] 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQD+dKJdF+jCORAAJTsTUAzcCVBIRPw4AgLI+w== 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout:[client.0] 2019-10-13T00:51:18.136 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDtdKJdYNeqIBAAQycC2LsL7cE7ZY6DLeKMdQ== 2019-10-13T00:51:18.139 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2019-10-13T00:51:18.139 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDtdKJdVw8xBhAAaYE/d1621HpTw+ZCBhSKHQ== 2019-10-13T00:51:18.139 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.a] 2019-10-13T00:51:18.140 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDtdKJd0/EnERAAxAs8ZZ2DfIZD8NZJ9qWrEw== 2019-10-13T00:51:18.140 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2019-10-13T00:51:18.140 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDtdKJdRZudMhAA65/XpP9B9fA3oknfTBNrvA== 2019-10-13T00:51:18.140 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2019-10-13T00:51:18.140 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDwdKJdN3GWDxAALcWhAOsxVehGn9JU8RPmNw== 2019-10-13T00:51:18.140 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2019-10-13T00:51:18.140 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDydKJdnk8jKBAAmd2YYSGqNiW4gKLmzBSZfg== 2019-10-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2019-10-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD1dKJdD4ZDBRAAnyD1ZUUYV0kx9qznXEGzeg== 2019-10-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2019-10-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDtdKJd5BlADBAAyWbQaPfaeO137hQMLOVKVA== 2019-10-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.b] 2019-10-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDtdKJdAWIdGRAAkIhPh6HTUGjEMOd4bQyKng== 2019-10-13T00:51:18.141 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2019-10-13T00:51:18.142 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD3dKJd5Rj9KhAAkobh4pIH5d/oW//T9kRTfw== 2019-10-13T00:51:18.142 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2019-10-13T00:51:18.142 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD6dKJdw0VVCBAASyk/x6ARp3V/N+nowjxxOg== 2019-10-13T00:51:18.142 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2019-10-13T00:51:18.142 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD8dKJdDAu4IBAAPr4xAlwPDBcBCEgiD6oSFw== 2019-10-13T00:51:18.142 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2019-10-13T00:51:18.142 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD+dKJdF+jCORAAJTsTUAzcCVBIRPw4AgLI+w== 2019-10-13T00:51:18.143 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2019-10-13T00:51:18.143 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDtdKJdYNeqIBAAQycC2LsL7cE7ZY6DLeKMdQ== 2019-10-13T00:51:18.143 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.144 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T00:51:18.211 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.211 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T00:51:18.288 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.289 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.293 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.293 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.373 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.374 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.378 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.378 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.454 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.454 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.458 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.458 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.534 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.534 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.538 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.538 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.613 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.614 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.618 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.618 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.693 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.693 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T00:51:18.697 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.698 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T00:51:18.773 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.773 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.777 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.777 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.856 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.856 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.860 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.860 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.935 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:18.935 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:18.940 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:18.940 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:19.015 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.015 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:19.019 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.019 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:19.094 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.094 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:19.098 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.098 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T00:51:19.174 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.174 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-13T00:51:19.178 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.178 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-13T00:51:19.255 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-10-13T00:51:19.256 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.256 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-10-13T00:51:19.307 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.308 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T00:51:19.387 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.387 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-10-13T00:51:19.439 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.439 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-10-13T00:51:19.491 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.492 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T00:51:19.572 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.572 INFO:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-10-13T00:51:19.624 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.624 INFO:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-10-13T00:51:19.675 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.676 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T00:51:19.757 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.757 INFO:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-10-13T00:51:19.809 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.809 INFO:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:51:19.814 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.814 INFO:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-13T00:51:19.859 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-10-13T00:51:19.860 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.860 INFO:teuthology.orchestra.run.smithi155:> which systemctl 2019-10-13T00:51:19.903 INFO:teuthology.orchestra.run.smithi155.stdout:/usr/bin/systemctl 2019-10-13T00:51:19.903 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-13T00:51:19.903 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:19.903 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-13T00:51:19.970 INFO:tasks.ceph.mon.a:Started 2019-10-13T00:51:19.971 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:19.971 INFO:teuthology.orchestra.run.smithi142:> which systemctl 2019-10-13T00:51:20.011 INFO:teuthology.orchestra.run.smithi142.stdout:/usr/bin/systemctl 2019-10-13T00:51:20.011 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-13T00:51:20.011 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:20.012 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-13T00:51:20.078 INFO:tasks.ceph.mon.b:Started 2019-10-13T00:51:20.078 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-13T00:51:20.079 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:20.079 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-13T00:51:20.082 INFO:tasks.ceph.mon.c:Started 2019-10-13T00:51:20.082 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-10-13T00:51:20.083 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-10-13T00:51:20.083 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:20.083 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2019-10-13T00:51:20.087 INFO:tasks.ceph.mgr.y:Started 2019-10-13T00:51:20.087 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-10-13T00:51:20.087 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:20.087 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2019-10-13T00:51:20.090 INFO:tasks.ceph.mgr.x:Started 2019-10-13T00:51:20.091 INFO:tasks.ceph:Setting crush tunables to default 2019-10-13T00:51:20.091 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:20.091 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd crush tunables default 2019-10-13T00:51:20.474 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:20.474 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:51:20.478 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:20.478 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:51:30.422 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.214+0000 7f7ea9bc1700 1 Processor -- start 2019-10-13T00:51:30.423 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.214+0000 7f7ea9bc1700 1 -- start start 2019-10-13T00:51:30.423 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7ea9bc1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:30.423 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7ea9bc1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea4060a10 con 0x7f7ea4060400 2019-10-13T00:51:30.423 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7ea9bc1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea4062890 con 0x7f7ea4098120 2019-10-13T00:51:30.423 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7ea9bc1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea406dac0 con 0x7f7ea40990f0 2019-10-13T00:51:30.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7ea895f700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7ea4098120 0x7f7ea40985c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48428/0 (socket says 172.21.15.155:48428) 2019-10-13T00:51:30.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7ea895f700 1 -- 172.21.15.155:0/3841772349 learned_addr learned my addr 172.21.15.155:0/3841772349 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:30.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T00:51:30.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.215+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T00:51:30.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.416+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.416+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T00:51:30.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.416+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.416+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.416+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.400000 2019-10-13T00:51:30.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.817+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.817+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T00:51:30.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.817+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.817+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:20.817+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.800000 2019-10-13T00:51:30.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:21.618+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:21.618+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T00:51:30.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:21.618+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:21.618+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:21.618+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 1.600000 2019-10-13T00:51:30.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:23.220+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:23.221+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T00:51:30.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:23.221+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:23.221+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:23.221+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 3.200000 2019-10-13T00:51:30.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:26.424+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:26.424+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T00:51:30.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:26.424+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:26.424+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:26.424+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 6.400000 2019-10-13T00:51:30.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 msgr2=0x7f7ea4086b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-10-13T00:51:30.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea40990f0 0x7f7ea4086b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:30.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 -- 172.21.15.155:0/3841772349 >> v1:172.21.15.155:6789/0 conn(0x7f7ea4098120 legacy=0x7f7ea40985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:30.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 -- 172.21.15.155:0/3841772349 >> v1:172.21.15.142:6789/0 conn(0x7f7ea4060400 legacy=0x7f7ea4086320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:30.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 0x7f7e90005430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:30.431 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 0x7f7e900080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:30.431 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 -- 172.21.15.155:0/3841772349 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e90008840 con 0x7f7e90005bd0 2019-10-13T00:51:30.431 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea27fc700 1 -- 172.21.15.155:0/3841772349 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e90008a70 con 0x7f7e90002fc0 2019-10-13T00:51:30.431 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 0x7f7e900080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.431 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 0x7f7e90005430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.431 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea3fff700 1 -- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 msgr2=0x7f7e90005430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T00:51:30.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea3fff700 1 -- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 msgr2=0x7f7e90005430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 0x7f7e90005430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.215+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 msgr2=0x7f7e900080a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-10-13T00:51:30.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.216+0000 7f7e9bfff700 1 -- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 msgr2=0x7f7e900080a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T00:51:30.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.216+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 0x7f7e900080a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T00:51:30.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.216+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 0x7f7e90005430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T00:51:30.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.216+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 0x7f7e900080a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T00:51:30.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 0x7f7e90005430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 0x7f7e900080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 -- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 msgr2=0x7f7e900080a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:30.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.142:3300/0 conn(0x7f7e90005bd0 0x7f7e900080a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:30.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 -- 172.21.15.155:0/3841772349 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e90008d20 con 0x7f7e90002fc0 2019-10-13T00:51:30.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 0x7f7e90005430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7e9400b430 tx=0x7f7e9400b7d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> v2:172.21.15.155:3300/0 conn(0x7f7e90002fc0 0x7f7e90005430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7e9400b430 tx=0x7f7e9400b7d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e90002fc0 0x7f7e90005430 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f7e9400b430 tx=0x7f7e9400b7d0).ready entity=mon.1 client_cookie=a82680644ed702a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:30.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e90002fc0 0x7f7e90005430 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f7e9400b430 tx=0x7f7e9400b7d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e90002fc0 0x7f7e90005430 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f7e9400b430 tx=0x7f7e9400b7d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e90002fc0 0x7f7e90005430 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f7e9400b430 tx=0x7f7e9400b7d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea37fe700 1 -- 172.21.15.155:0/3841772349 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e9400be00 con 0x7f7e90002fc0 2019-10-13T00:51:30.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea37fe700 1 -- 172.21.15.155:0/3841772349 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9400a780 con 0x7f7e90002fc0 2019-10-13T00:51:30.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea37fe700 1 -- 172.21.15.155:0/3841772349 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e9400c3e0 con 0x7f7e90002fc0 2019-10-13T00:51:30.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/3841772349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e90002fc0 msgr2=0x7f7e90005430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:30.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.417+0000 7f7ea9bc1700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e90002fc0 0x7f7e90005430 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f7e9400b430 tx=0x7f7e9400b7d0).stop 2019-10-13T00:51:30.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.418+0000 7f7ea895f700 1 -- 172.21.15.155:0/3841772349 reap_dead start 2019-10-13T00:51:30.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.418+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/3841772349 shutdown_connections 2019-10-13T00:51:30.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.418+0000 7f7ea9bc1700 1 --2- 172.21.15.155:0/3841772349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e90002fc0 0x7f7e90005430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:30.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.418+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/3841772349 shutdown_connections 2019-10-13T00:51:30.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.418+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/3841772349 wait complete. 2019-10-13T00:51:30.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.418+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/3841772349 >> 172.21.15.155:0/3841772349 conn(0x7f7ea4062020 msgr2=0x7f7ea405f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:30.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.421+0000 7f7ea9bc1700 1 Processor -- start 2019-10-13T00:51:30.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.421+0000 7f7ea9bc1700 1 -- start start 2019-10-13T00:51:30.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea9bc1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea414c8b0 0x7f7ea414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:30.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea9bc1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ea40916c0 0x7f7ea40906f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:30.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea9bc1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7ea4090c30 0x7f7ea4082040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:30.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea9bc1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea412d6c0 con 0x7f7ea4090c30 2019-10-13T00:51:30.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea9bc1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea408fd00 con 0x7f7ea40916c0 2019-10-13T00:51:30.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea9bc1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea412e4b0 con 0x7f7ea414c8b0 2019-10-13T00:51:30.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ea40916c0 0x7f7ea40906f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ea40916c0 0x7f7ea40906f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58402/0 (socket says 172.21.15.155:58402) 2019-10-13T00:51:30.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 -- 172.21.15.155:0/4180262913 learned_addr learned my addr 172.21.15.155:0/4180262913 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:30.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea895f700 1 --2- 172.21.15.155:0/4180262913 >> v2:172.21.15.142:3300/0 conn(0x7f7ea4090c30 0x7f7ea4082040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7ea3fff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea414c8b0 0x7f7ea414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:30.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 -- 172.21.15.155:0/4180262913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea414c8b0 msgr2=0x7f7ea414cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:30.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea414c8b0 0x7f7ea414cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:30.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 -- 172.21.15.155:0/4180262913 >> v2:172.21.15.142:3300/0 conn(0x7f7ea4090c30 msgr2=0x7f7ea4082040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:30.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> v2:172.21.15.142:3300/0 conn(0x7f7ea4090c30 0x7f7ea4082040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:30.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 -- 172.21.15.155:0/4180262913 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea408e320 con 0x7f7ea40916c0 2019-10-13T00:51:30.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> v2:172.21.15.155:3300/0 conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> v2:172.21.15.155:3300/0 conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.422+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:30.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7ea17fa700 1 -- 172.21.15.155:0/4180262913 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e8c005250 con 0x7f7ea40916c0 2019-10-13T00:51:30.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7ea17fa700 1 -- 172.21.15.155:0/4180262913 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c005aa0 con 0x7f7ea40916c0 2019-10-13T00:51:30.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7ea17fa700 1 -- 172.21.15.155:0/4180262913 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e8c0061f0 con 0x7f7ea40916c0 2019-10-13T00:51:30.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea408e320 con 0x7f7ea40916c0 2019-10-13T00:51:30.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea408e320 con 0x7f7ea40916c0 2019-10-13T00:51:30.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7ea17fa700 1 -- 172.21.15.155:0/4180262913 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 341+0+0 (secure 0 0 0) 0x7f7e8c005730 con 0x7f7ea40916c0 2019-10-13T00:51:30.441 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.441 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.423+0000 7f7ea17fa700 1 -- 172.21.15.155:0/4180262913 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7e8c001020 con 0x7f7ea40916c0 2019-10-13T00:51:30.450 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.453+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea4091380 con 0x7f7ea40916c0 2019-10-13T00:51:30.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.456+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.456+0000 7f7ea17fa700 1 -- 172.21.15.155:0/4180262913 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67206 (secure 0 0 0) 0x7f7e8c005aa0 con 0x7f7ea40916c0 2019-10-13T00:51:30.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.552+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f7ea4091380 con 0x7f7ea40916c0 2019-10-13T00:51:30.558 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.560+0000 7f7e9bfff700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:30.558 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.561+0000 7f7ea17fa700 1 -- 172.21.15.155:0/4180262913 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7e8c005aa0 con 0x7f7ea40916c0 2019-10-13T00:51:30.564 INFO:teuthology.orchestra.run.smithi155.stderr:adjusted tunables profile to default 2019-10-13T00:51:30.567 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 msgr2=0x7f7ea40906f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:30.567 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f7e8c054470 tx=0x7f7e8c054810).stop 2019-10-13T00:51:30.567 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 shutdown_connections 2019-10-13T00:51:30.567 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 --2- 172.21.15.155:0/4180262913 >> v2:172.21.15.142:3300/0 conn(0x7f7ea4090c30 0x7f7ea4082040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:30.567 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ea40916c0 0x7f7ea40906f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:30.567 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 --2- 172.21.15.155:0/4180262913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea414c8b0 0x7f7ea414cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:30.567 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 shutdown_connections 2019-10-13T00:51:30.568 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 wait complete. 2019-10-13T00:51:30.568 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:30.569+0000 7f7ea9bc1700 1 -- 172.21.15.155:0/4180262913 >> 172.21.15.155:0/4180262913 conn(0x7f7ea40813a0 msgr2=0x7f7ea4083150 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:30.576 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-10-13T00:51:30.576 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:30.576 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:30.618 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.cyZ1SZT7Wc 2019-10-13T00:51:30.619 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:30.619 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.cyZ1SZT7Wc 2019-10-13T00:51:30.740 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:30.740 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.cyZ1SZT7Wc 2019-10-13T00:51:30.837 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.cyZ1SZT7Wc is 37B 2019-10-13T00:51:30.892 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:30.893 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.cyZ1SZT7Wc 2019-10-13T00:51:30.934 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:30.934 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:31.040 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.zhY9l4gi23 2019-10-13T00:51:31.040 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.040 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.zhY9l4gi23 2019-10-13T00:51:31.159 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.159 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.zhY9l4gi23 2019-10-13T00:51:31.256 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.zhY9l4gi23 is 37B 2019-10-13T00:51:31.307 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.307 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.zhY9l4gi23 2019-10-13T00:51:31.348 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.348 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:31.454 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.5IrWpHP57r 2019-10-13T00:51:31.454 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.454 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.5IrWpHP57r 2019-10-13T00:51:31.570 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.570 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.5IrWpHP57r 2019-10-13T00:51:31.667 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.5IrWpHP57r is 37B 2019-10-13T00:51:31.717 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.717 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.5IrWpHP57r 2019-10-13T00:51:31.758 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.758 INFO:teuthology.orchestra.run.smithi155:> mktemp 2019-10-13T00:51:31.863 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.ZsYCMCOfUx 2019-10-13T00:51:31.864 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.864 INFO:teuthology.orchestra.run.smithi155:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.ZsYCMCOfUx 2019-10-13T00:51:31.980 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:31.980 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.ZsYCMCOfUx 2019-10-13T00:51:32.073 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.ZsYCMCOfUx is 37B 2019-10-13T00:51:32.121 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:32.121 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.ZsYCMCOfUx 2019-10-13T00:51:32.161 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.161 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:32.203 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.melTjVBhW8 2019-10-13T00:51:32.203 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.203 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.melTjVBhW8 2019-10-13T00:51:32.321 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.321 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.melTjVBhW8 2019-10-13T00:51:32.415 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.melTjVBhW8 is 37B 2019-10-13T00:51:32.463 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.464 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.melTjVBhW8 2019-10-13T00:51:32.503 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.504 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:32.607 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.75AI3NiKFW 2019-10-13T00:51:32.607 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.608 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.75AI3NiKFW 2019-10-13T00:51:32.725 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.726 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.75AI3NiKFW 2019-10-13T00:51:32.821 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.75AI3NiKFW is 37B 2019-10-13T00:51:32.868 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.868 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.75AI3NiKFW 2019-10-13T00:51:32.908 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:32.908 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:33.011 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.Jqf5LLp5PZ 2019-10-13T00:51:33.011 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.011 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.Jqf5LLp5PZ 2019-10-13T00:51:33.127 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.127 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.Jqf5LLp5PZ 2019-10-13T00:51:33.222 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.Jqf5LLp5PZ is 37B 2019-10-13T00:51:33.272 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.272 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.Jqf5LLp5PZ 2019-10-13T00:51:33.314 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.314 INFO:teuthology.orchestra.run.smithi142:> mktemp 2019-10-13T00:51:33.418 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.9ALAo5oVvK 2019-10-13T00:51:33.419 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.419 INFO:teuthology.orchestra.run.smithi142:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.9ALAo5oVvK 2019-10-13T00:51:33.510 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.511 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.9ALAo5oVvK 2019-10-13T00:51:33.605 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.9ALAo5oVvK is 37B 2019-10-13T00:51:33.654 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.655 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.9ALAo5oVvK 2019-10-13T00:51:33.694 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:33.694 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 90b46d14-0a9a-4f76-b5d7-c1e4716ea22d 0 2019-10-13T00:51:33.880 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.877+0000 7f7e88029700 1 Processor -- start 2019-10-13T00:51:33.880 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.877+0000 7f7e88029700 1 -- start start 2019-10-13T00:51:33.880 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e88029700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7e80060400 0x7f7e80086320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:33.880 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e88029700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7e80098120 0x7f7e80098540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:33.880 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e88029700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e80060a10 con 0x7f7e800990f0 2019-10-13T00:51:33.880 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e88029700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8006dac0 con 0x7f7e80060400 2019-10-13T00:51:33.881 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e88029700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8006dd90 con 0x7f7e80098120 2019-10-13T00:51:33.881 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e865c6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7e80060400 0x7f7e80086320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:33.881 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e865c6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7e80060400 0x7f7e80086320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58078/0 (socket says 172.21.15.142:58078) 2019-10-13T00:51:33.881 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.878+0000 7f7e865c6700 1 -- 172.21.15.142:0/4024404286 learned_addr learned my addr 172.21.15.142:0/4024404286 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:33.881 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e86dc7700 1 --2- 172.21.15.142:0/4024404286 >> v2:172.21.15.155:3300/0 conn(0x7f7e80098120 0x7f7e80098540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:33.881 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 -- 172.21.15.142:0/4024404286 >> v2:172.21.15.155:3300/0 conn(0x7f7e80098120 msgr2=0x7f7e80098540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:33.881 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 --2- 172.21.15.142:0/4024404286 >> v2:172.21.15.155:3300/0 conn(0x7f7e80098120 0x7f7e80098540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:33.882 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 -- 172.21.15.142:0/4024404286 >> v1:172.21.15.155:6789/0 conn(0x7f7e800990f0 legacy=0x7f7e80086860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:33.882 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 -- 172.21.15.142:0/4024404286 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80097820 con 0x7f7e80060400 2019-10-13T00:51:33.882 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 --2- 172.21.15.142:0/4024404286 >> v2:172.21.15.142:3300/0 conn(0x7f7e80060400 0x7f7e80086320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7e7c00f730 tx=0x7f7e7c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.882 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 --2- 172.21.15.142:0/4024404286 >> v2:172.21.15.142:3300/0 conn(0x7f7e80060400 0x7f7e80086320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7e7c00f730 tx=0x7f7e7c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.882 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 --2- 172.21.15.142:0/4024404286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e80060400 0x7f7e80086320 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7f7e7c00f730 tx=0x7f7e7c00fad0).ready entity=mon.0 client_cookie=3081556c3419aa45 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:33.882 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 --2- 172.21.15.142:0/4024404286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e80060400 0x7f7e80086320 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f7e7c00f730 tx=0x7f7e7c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.882 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 --2- 172.21.15.142:0/4024404286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e80060400 0x7f7e80086320 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f7e7c00f730 tx=0x7f7e7c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.883 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e865c6700 1 --2- 172.21.15.142:0/4024404286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e80060400 0x7f7e80086320 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f7e7c00f730 tx=0x7f7e7c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.883 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e855c4700 1 -- 172.21.15.142:0/4024404286 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c000960 con 0x7f7e80060400 2019-10-13T00:51:33.883 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e855c4700 1 -- 172.21.15.142:0/4024404286 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c000c90 con 0x7f7e80060400 2019-10-13T00:51:33.883 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e855c4700 1 -- 172.21.15.142:0/4024404286 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c001360 con 0x7f7e80060400 2019-10-13T00:51:33.883 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e88029700 1 -- 172.21.15.142:0/4024404286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e80060400 msgr2=0x7f7e80086320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:33.883 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.879+0000 7f7e88029700 1 --2- 172.21.15.142:0/4024404286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e80060400 0x7f7e80086320 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7f7e7c00f730 tx=0x7f7e7c00fad0).stop 2019-10-13T00:51:33.883 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.880+0000 7f7e88029700 1 -- 172.21.15.142:0/4024404286 shutdown_connections 2019-10-13T00:51:33.884 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.880+0000 7f7e88029700 1 --2- 172.21.15.142:0/4024404286 >> v2:172.21.15.155:3300/0 conn(0x7f7e80098120 0x7f7e80098540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:33.884 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.880+0000 7f7e88029700 1 --2- 172.21.15.142:0/4024404286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e80060400 0x7f7e80086320 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:33.884 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.880+0000 7f7e88029700 1 -- 172.21.15.142:0/4024404286 shutdown_connections 2019-10-13T00:51:33.884 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.880+0000 7f7e88029700 1 -- 172.21.15.142:0/4024404286 wait complete. 2019-10-13T00:51:33.884 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.880+0000 7f7e88029700 1 -- 172.21.15.142:0/4024404286 >> 172.21.15.142:0/4024404286 conn(0x7f7e80062020 msgr2=0x7f7e8005f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:33.887 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.882+0000 7f7e88029700 1 Processor -- start 2019-10-13T00:51:33.887 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.882+0000 7f7e88029700 1 -- start start 2019-10-13T00:51:33.887 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e88029700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:33.887 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:33.887 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55548/0 (socket says 172.21.15.142:55548) 2019-10-13T00:51:33.888 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e88029700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7e80076480 0x7f7e80152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:33.888 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 -- 172.21.15.142:0/2327038303 learned_addr learned my addr 172.21.15.142:0/2327038303 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:33.888 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e88029700 1 --2- 172.21.15.142:0/2327038303 >> v2:172.21.15.142:3300/0 conn(0x7f7e800754b0 0x7f7e80075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:33.888 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e80149c70 con 0x7f7e800754b0 2019-10-13T00:51:33.888 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e80137650 con 0x7f7e80076480 2019-10-13T00:51:33.888 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e80137920 con 0x7f7e800822b0 2019-10-13T00:51:33.888 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e86dc7700 1 --2- 172.21.15.142:0/2327038303 >> v2:172.21.15.142:3300/0 conn(0x7f7e800754b0 0x7f7e80075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:33.889 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e85dc5700 1 --2- 172.21.15.142:0/2327038303 >> v2:172.21.15.155:3300/0 conn(0x7f7e80076480 0x7f7e80152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:33.889 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 -- 172.21.15.142:0/2327038303 >> v2:172.21.15.155:3300/0 conn(0x7f7e80076480 msgr2=0x7f7e80152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:33.889 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> v2:172.21.15.155:3300/0 conn(0x7f7e80076480 0x7f7e80152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:33.889 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 -- 172.21.15.142:0/2327038303 >> v2:172.21.15.142:3300/0 conn(0x7f7e800754b0 msgr2=0x7f7e80075930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:33.889 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> v2:172.21.15.142:3300/0 conn(0x7f7e800754b0 0x7f7e80075930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:33.889 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 -- 172.21.15.142:0/2327038303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80074bb0 con 0x7f7e800822b0 2019-10-13T00:51:33.890 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.883+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.890 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.890 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:33.890 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.890 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e76ffd700 1 -- 172.21.15.142:0/2327038303 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c008170 con 0x7f7e800822b0 2019-10-13T00:51:33.890 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e80074bb0 con 0x7f7e800822b0 2019-10-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e76ffd700 1 -- 172.21.15.142:0/2327038303 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c008170 con 0x7f7e800822b0 2019-10-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e80082960 con 0x7f7e800822b0 2019-10-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e76ffd700 1 -- 172.21.15.142:0/2327038303 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c005550 con 0x7f7e800822b0 2019-10-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.891 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.892 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e76ffd700 1 -- 172.21.15.142:0/2327038303 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f7e7c005170 con 0x7f7e800822b0 2019-10-13T00:51:33.892 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e76ffd700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7e60017c60 0x7f7e6001a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:33.892 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.884+0000 7f7e76ffd700 1 -- 172.21.15.142:0/2327038303 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7e7c030310 con 0x7f7e800822b0 2019-10-13T00:51:33.892 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.885+0000 7f7e85dc5700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7e60017c60 0x7f7e6001a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:33.892 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.885+0000 7f7e85dc5700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7e60017c60 0x7f7e6001a110 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:33.893 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.893+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e80074bb0 con 0x7f7e800822b0 2019-10-13T00:51:33.894 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.896+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:33.894 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:33.896+0000 7f7e76ffd700 1 -- 172.21.15.142:0/2327038303 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f7e7c016a10 con 0x7f7e800822b0 2019-10-13T00:51:34.052 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.054+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd new", "uuid": "90b46d14-0a9a-4f76-b5d7-c1e4716ea22d", "id": 0} v 0) v1 -- 0x7f7e80074bb0 con 0x7f7e800822b0 2019-10-13T00:51:34.060 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.062+0000 7f7e865c6700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.061 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.062+0000 7f7e76ffd700 1 -- 172.21.15.142:0/2327038303 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90b46d14-0a9a-4f76-b5d7-c1e4716ea22d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7e7c005170 con 0x7f7e800822b0 2019-10-13T00:51:34.068 INFO:teuthology.orchestra.run.smithi142.stdout:0 2019-10-13T00:51:34.070 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7e60017c60 msgr2=0x7f7e6001a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.071 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7e60017c60 0x7f7e6001a110 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.071 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 msgr2=0x7f7e801520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.071 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7f7e7c010430 tx=0x7f7e7c0107d0).stop 2019-10-13T00:51:34.071 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 shutdown_connections 2019-10-13T00:51:34.071 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 --2- 172.21.15.142:0/2327038303 >> v2:172.21.15.142:3300/0 conn(0x7f7e800754b0 0x7f7e80075930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.071 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 --2- 172.21.15.142:0/2327038303 >> v2:172.21.15.155:3300/0 conn(0x7f7e80076480 0x7f7e80152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.071 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7e60017c60 0x7f7e6001a110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.072 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 --2- 172.21.15.142:0/2327038303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e800822b0 0x7f7e801520d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.072 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 shutdown_connections 2019-10-13T00:51:34.072 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 wait complete. 2019-10-13T00:51:34.072 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.072+0000 7f7e88029700 1 -- 172.21.15.142:0/2327038303 >> 172.21.15.142:0/2327038303 conn(0x7f7e8008c9c0 msgr2=0x7f7e800a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:34.079 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:34.080 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 9fbcb085-3256-410b-85fc-dfd7fc572b50 1 2019-10-13T00:51:34.201 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.199+0000 7f7ca0061700 1 Processor -- start 2019-10-13T00:51:34.201 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.199+0000 7f7ca0061700 1 -- start start 2019-10-13T00:51:34.201 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.199+0000 7f7ca0061700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c98060400 0x7f7c98086320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:34.201 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7ca0061700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7c98098120 0x7f7c98098540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:34.201 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7ca0061700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c98060a10 con 0x7f7c980990f0 2019-10-13T00:51:34.201 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7ca0061700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9806dac0 con 0x7f7c98060400 2019-10-13T00:51:34.202 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7ca0061700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9806dd90 con 0x7f7c98098120 2019-10-13T00:51:34.202 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c98060400 0x7f7c98086320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.202 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c98060400 0x7f7c98086320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58092/0 (socket says 172.21.15.142:58092) 2019-10-13T00:51:34.202 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/1318937354 learned_addr learned my addr 172.21.15.142:0/1318937354 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:34.202 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9ddfd700 1 --1- 172.21.15.142:0/1318937354 >> v1:172.21.15.142:6789/0 conn(0x7f7c980990f0 0x7f7c98086860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51262/0 (socket says 172.21.15.142:51262) 2019-10-13T00:51:34.202 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9edff700 1 --2- 172.21.15.142:0/1318937354 >> v2:172.21.15.155:3300/0 conn(0x7f7c98098120 0x7f7c98098540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.202 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/1318937354 >> v2:172.21.15.155:3300/0 conn(0x7f7c98098120 msgr2=0x7f7c98098540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:34.203 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/1318937354 >> v2:172.21.15.155:3300/0 conn(0x7f7c98098120 0x7f7c98098540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:34.203 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/1318937354 >> v1:172.21.15.142:6789/0 conn(0x7f7c980990f0 legacy=0x7f7c98086860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.203 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9d5fc700 1 -- 172.21.15.142:0/1318937354 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520179422 0 0) 0x7f7c80001070 con 0x7f7c980990f0 2019-10-13T00:51:34.203 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/1318937354 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98097820 con 0x7f7c98060400 2019-10-13T00:51:34.203 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/1318937354 >> v2:172.21.15.142:3300/0 conn(0x7f7c98060400 0x7f7c98086320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c8800fab0 tx=0x7f7c8800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.203 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/1318937354 >> v2:172.21.15.142:3300/0 conn(0x7f7c98060400 0x7f7c98086320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c8800fab0 tx=0x7f7c8800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.204 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/1318937354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c98060400 0x7f7c98086320 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f7c8800fab0 tx=0x7f7c8800fe50).ready entity=mon.0 client_cookie=c96aff9bf9bd92f server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.204 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/1318937354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c98060400 0x7f7c98086320 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f7c8800fab0 tx=0x7f7c8800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.204 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9d5fc700 1 -- 172.21.15.142:0/1318937354 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c88000ce0 con 0x7f7c98060400 2019-10-13T00:51:34.204 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/1318937354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c98060400 0x7f7c98086320 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f7c8800fab0 tx=0x7f7c8800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.204 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/1318937354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c98060400 0x7f7c98086320 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f7c8800fab0 tx=0x7f7c8800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.204 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9d5fc700 1 -- 172.21.15.142:0/1318937354 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c88001010 con 0x7f7c98060400 2019-10-13T00:51:34.204 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7c9d5fc700 1 -- 172.21.15.142:0/1318937354 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c880016e0 con 0x7f7c98060400 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7ca0061700 1 -- 172.21.15.142:0/1318937354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c98060400 msgr2=0x7f7c98086320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.200+0000 7f7ca0061700 1 --2- 172.21.15.142:0/1318937354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c98060400 0x7f7c98086320 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f7c8800fab0 tx=0x7f7c8800fe50).stop 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.201+0000 7f7ca0061700 1 -- 172.21.15.142:0/1318937354 shutdown_connections 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.201+0000 7f7ca0061700 1 --2- 172.21.15.142:0/1318937354 >> v2:172.21.15.155:3300/0 conn(0x7f7c98098120 0x7f7c98098540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.201+0000 7f7ca0061700 1 --2- 172.21.15.142:0/1318937354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c98060400 0x7f7c98086320 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.201+0000 7f7ca0061700 1 -- 172.21.15.142:0/1318937354 shutdown_connections 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.201+0000 7f7ca0061700 1 -- 172.21.15.142:0/1318937354 wait complete. 2019-10-13T00:51:34.205 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.201+0000 7f7ca0061700 1 -- 172.21.15.142:0/1318937354 >> 172.21.15.142:0/1318937354 conn(0x7f7c98062020 msgr2=0x7f7c9805f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:34.209 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.203+0000 7f7ca0061700 1 Processor -- start 2019-10-13T00:51:34.209 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.203+0000 7f7ca0061700 1 -- start start 2019-10-13T00:51:34.209 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7ca0061700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c9814b730 0x7f7c9814bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.209 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7ca0061700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7c9809d780 0x7f7c98155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.209 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7ca0061700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9809dbc0 con 0x7f7c9809df50 2019-10-13T00:51:34.210 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7ca0061700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9809bff0 con 0x7f7c9814b730 2019-10-13T00:51:34.210 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7ca0061700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c9809b5c0 con 0x7f7c9809d780 2019-10-13T00:51:34.210 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7c9e5fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c9814b730 0x7f7c9814bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.210 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7c9e5fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c9814b730 0x7f7c9814bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58102/0 (socket says 172.21.15.142:58102) 2019-10-13T00:51:34.210 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/3748991247 learned_addr learned my addr 172.21.15.142:0/3748991247 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:34.210 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.204+0000 7f7c9edff700 1 --2- 172.21.15.142:0/3748991247 >> v2:172.21.15.155:3300/0 conn(0x7f7c9809d780 0x7f7c98155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.210 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/3748991247 >> v2:172.21.15.155:3300/0 conn(0x7f7c9809d780 msgr2=0x7f7c98155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.211 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> v2:172.21.15.155:3300/0 conn(0x7f7c9809d780 0x7f7c98155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.211 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/3748991247 >> v1:172.21.15.155:6789/0 conn(0x7f7c9809df50 legacy=0x7f7c9809cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.211 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 -- 172.21.15.142:0/3748991247 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9809c680 con 0x7f7c9814b730 2019-10-13T00:51:34.211 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> v2:172.21.15.142:3300/0 conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.211 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> v2:172.21.15.142:3300/0 conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.211 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.212 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.212 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.212 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.212 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c96ffd700 1 -- 172.21.15.142:0/3748991247 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c880084f0 con 0x7f7c9814b730 2019-10-13T00:51:34.212 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c96ffd700 1 -- 172.21.15.142:0/3748991247 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c880056e0 con 0x7f7c9814b730 2019-10-13T00:51:34.212 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c96ffd700 1 -- 172.21.15.142:0/3748991247 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c88005cf0 con 0x7f7c9814b730 2019-10-13T00:51:34.212 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9809c680 con 0x7f7c9814b730 2019-10-13T00:51:34.213 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9809c680 con 0x7f7c9814b730 2019-10-13T00:51:34.213 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.213 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.213 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c96ffd700 1 -- 172.21.15.142:0/3748991247 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f7c880058d0 con 0x7f7c9814b730 2019-10-13T00:51:34.213 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.205+0000 7f7c96ffd700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7c7c017c60 0x7f7c7c01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.213 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.206+0000 7f7c96ffd700 1 -- 172.21.15.142:0/3748991247 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f7c88030470 con 0x7f7c9814b730 2019-10-13T00:51:34.213 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.206+0000 7f7c9ddfd700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7c7c017c60 0x7f7c7c01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.214 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.206+0000 7f7c9ddfd700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7c7c017c60 0x7f7c7c01a0f0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.214 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.214+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9809c680 con 0x7f7c9814b730 2019-10-13T00:51:34.215 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.217+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.215 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.217+0000 7f7c96ffd700 1 -- 172.21.15.142:0/3748991247 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f7c88007630 con 0x7f7c9814b730 2019-10-13T00:51:34.373 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.375+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9fbcb085-3256-410b-85fc-dfd7fc572b50", "id": 1} v 0) v1 -- 0x7f7c9809c680 con 0x7f7c9814b730 2019-10-13T00:51:34.376 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.378+0000 7f7c9e5fe700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.376 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.378+0000 7f7c96ffd700 1 -- 172.21.15.142:0/3748991247 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9fbcb085-3256-410b-85fc-dfd7fc572b50", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7c880058d0 con 0x7f7c9814b730 2019-10-13T00:51:34.380 INFO:teuthology.orchestra.run.smithi142.stdout:1 2019-10-13T00:51:34.383 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7c7c017c60 msgr2=0x7f7c7c01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.383 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7c7c017c60 0x7f7c7c01a0f0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.383 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 msgr2=0x7f7c9814bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.383 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f7c880107b0 tx=0x7f7c88010b50).stop 2019-10-13T00:51:34.383 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 shutdown_connections 2019-10-13T00:51:34.384 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 --2- 172.21.15.142:0/3748991247 >> v2:172.21.15.155:3300/0 conn(0x7f7c9809d780 0x7f7c98155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.384 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7c7c017c60 0x7f7c7c01a0f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.384 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 --2- 172.21.15.142:0/3748991247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c9814b730 0x7f7c9814bb20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.384 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 shutdown_connections 2019-10-13T00:51:34.384 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 wait complete. 2019-10-13T00:51:34.384 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.384+0000 7f7ca0061700 1 -- 172.21.15.142:0/3748991247 >> 172.21.15.142:0/3748991247 conn(0x7f7c980814b0 msgr2=0x7f7c980a30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:34.392 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:34.392 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new df99f110-da25-4a6d-93d8-20d8ab22b345 2 2019-10-13T00:51:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.511+0000 7f83e7c05700 1 Processor -- start 2019-10-13T00:51:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.512+0000 7f83e7c05700 1 -- start start 2019-10-13T00:51:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.512+0000 7f83e7c05700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83e0060400 0x7f83e0086320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.512+0000 7f83e7c05700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f83e0098120 0x7f83e0098540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.512+0000 7f83e7c05700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e0060a10 con 0x7f83e00990f0 2019-10-13T00:51:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.512+0000 7f83e7c05700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e006dac0 con 0x7f83e0060400 2019-10-13T00:51:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.512+0000 7f83e7c05700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e006dd90 con 0x7f83e0098120 2019-10-13T00:51:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e61a2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83e0060400 0x7f83e0086320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e61a2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83e0060400 0x7f83e0086320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58106/0 (socket says 172.21.15.142:58106) 2019-10-13T00:51:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f83e0098120 0x7f83e0098540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e61a2700 1 -- 172.21.15.142:0/2631958238 learned_addr learned my addr 172.21.15.142:0/2631958238 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e59a1700 1 --1- 172.21.15.142:0/2631958238 >> v1:172.21.15.142:6789/0 conn(0x7f83e00990f0 0x7f83e0086860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51274/0 (socket says 172.21.15.142:51274) 2019-10-13T00:51:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 -- 172.21.15.142:0/2631958238 >> v2:172.21.15.142:3300/0 conn(0x7f83e0060400 msgr2=0x7f83e0086320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- 172.21.15.142:0/2631958238 >> v2:172.21.15.142:3300/0 conn(0x7f83e0060400 0x7f83e0086320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 -- 172.21.15.142:0/2631958238 >> v1:172.21.15.142:6789/0 conn(0x7f83e00990f0 legacy=0x7f83e0086860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 -- 172.21.15.142:0/2631958238 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0097820 con 0x7f83e0098120 2019-10-13T00:51:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e61a2700 1 --2- 172.21.15.142:0/2631958238 >> v2:172.21.15.142:3300/0 conn(0x7f83e0060400 0x7f83e0086320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- 172.21.15.142:0/2631958238 >> v2:172.21.15.155:3300/0 conn(0x7f83e0098120 0x7f83e0098540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f83d800af70 tx=0x7f83d800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.517 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- 172.21.15.142:0/2631958238 >> v2:172.21.15.155:3300/0 conn(0x7f83e0098120 0x7f83e0098540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f83d800af70 tx=0x7f83d800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.517 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- 172.21.15.142:0/2631958238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83e0098120 0x7f83e0098540 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7f83d800af70 tx=0x7f83d800b780).ready entity=mon.1 client_cookie=96efa732c3d7d41c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.517 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- 172.21.15.142:0/2631958238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83e0098120 0x7f83e0098540 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f83d800af70 tx=0x7f83d800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.517 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- 172.21.15.142:0/2631958238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83e0098120 0x7f83e0098540 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f83d800af70 tx=0x7f83d800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.517 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e51a0700 1 -- 172.21.15.142:0/2631958238 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83d8007230 con 0x7f83e0098120 2019-10-13T00:51:34.517 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e69a3700 1 --2- 172.21.15.142:0/2631958238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83e0098120 0x7f83e0098540 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f83d800af70 tx=0x7f83d800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.517 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e51a0700 1 -- 172.21.15.142:0/2631958238 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8000bb0 con 0x7f83e0098120 2019-10-13T00:51:34.518 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e51a0700 1 -- 172.21.15.142:0/2631958238 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83d8009e50 con 0x7f83e0098120 2019-10-13T00:51:34.518 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e7c05700 1 -- 172.21.15.142:0/2631958238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83e0098120 msgr2=0x7f83e0098540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.518 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.513+0000 7f83e7c05700 1 --2- 172.21.15.142:0/2631958238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83e0098120 0x7f83e0098540 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7f83d800af70 tx=0x7f83d800b780).stop 2019-10-13T00:51:34.518 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.514+0000 7f83e7c05700 1 -- 172.21.15.142:0/2631958238 shutdown_connections 2019-10-13T00:51:34.518 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.514+0000 7f83e7c05700 1 --2- 172.21.15.142:0/2631958238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83e0098120 0x7f83e0098540 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.518 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.514+0000 7f83e7c05700 1 --2- 172.21.15.142:0/2631958238 >> v2:172.21.15.142:3300/0 conn(0x7f83e0060400 0x7f83e0086320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:34.518 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.514+0000 7f83e7c05700 1 -- 172.21.15.142:0/2631958238 shutdown_connections 2019-10-13T00:51:34.519 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.514+0000 7f83e7c05700 1 -- 172.21.15.142:0/2631958238 wait complete. 2019-10-13T00:51:34.519 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.514+0000 7f83e7c05700 1 -- 172.21.15.142:0/2631958238 >> 172.21.15.142:0/2631958238 conn(0x7f83e0062020 msgr2=0x7f83e005f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:34.522 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.516+0000 7f83e7c05700 1 Processor -- start 2019-10-13T00:51:34.522 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.516+0000 7f83e7c05700 1 -- start start 2019-10-13T00:51:34.522 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e7c05700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83e0082000 0x7f83e00823f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.522 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e7c05700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f83e008e0f0 0x7f83e0082930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.522 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e7c05700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e00a0770 con 0x7f83e008e8c0 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e7c05700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e008c960 con 0x7f83e0082000 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e7c05700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e009f6c0 con 0x7f83e008e0f0 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e61a2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83e0082000 0x7f83e00823f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e59a1700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f83e008e8c0 0x7f83e008d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51278/0 (socket says 172.21.15.142:51278) 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e61a2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83e0082000 0x7f83e00823f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58110/0 (socket says 172.21.15.142:58110) 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e59a1700 1 -- 172.21.15.142:0/249544905 learned_addr learned my addr 172.21.15.142:0/249544905 (peer_addr_for_me v1:172.21.15.142:0/0) 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.517+0000 7f83e69a3700 1 --2- 172.21.15.142:0/249544905 >> v2:172.21.15.155:3300/0 conn(0x7f83e008e0f0 0x7f83e0082930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.523 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 -- 172.21.15.142:0/249544905 >> v2:172.21.15.155:3300/0 conn(0x7f83e008e0f0 msgr2=0x7f83e0082930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.524 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> v2:172.21.15.155:3300/0 conn(0x7f83e008e0f0 0x7f83e0082930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.524 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 -- 172.21.15.142:0/249544905 >> v1:172.21.15.142:6789/0 conn(0x7f83e008e8c0 legacy=0x7f83e008d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.524 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986006337 0 0) 0x7f83d0001070 con 0x7f83e008e8c0 2019-10-13T00:51:34.524 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 -- 172.21.15.142:0/249544905 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e008cff0 con 0x7f83e0082000 2019-10-13T00:51:34.524 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> v2:172.21.15.142:3300/0 conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.524 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> v2:172.21.15.142:3300/0 conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.524 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.525 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.525 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83dc000cc0 con 0x7f83e0082000 2019-10-13T00:51:34.525 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.525 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83dc000cc0 con 0x7f83e0082000 2019-10-13T00:51:34.525 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.525 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e008cff0 con 0x7f83e0082000 2019-10-13T00:51:34.525 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83dc00c2c0 con 0x7f83e0082000 2019-10-13T00:51:34.526 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.518+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e008cff0 con 0x7f83e0082000 2019-10-13T00:51:34.526 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.519+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.526 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.519+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.526 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.519+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f83dc00bee0 con 0x7f83e0082000 2019-10-13T00:51:34.526 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.519+0000 7f83d6ffd700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f83c8017c60 0x7f83c801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.527 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.519+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f83dc0303c0 con 0x7f83e0082000 2019-10-13T00:51:34.527 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.519+0000 7f83e59a1700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f83c8017c60 0x7f83c801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.527 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.519+0000 7f83e59a1700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f83c8017c60 0x7f83c801a0f0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.527 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.527+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e008cff0 con 0x7f83e0082000 2019-10-13T00:51:34.528 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.530+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.528 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.530+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f83dc00bcd0 con 0x7f83e0082000 2019-10-13T00:51:34.686 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.689+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "df99f110-da25-4a6d-93d8-20d8ab22b345", "id": 2} v 0) v1 -- 0x7f83e008cff0 con 0x7f83e0082000 2019-10-13T00:51:34.690 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.692+0000 7f83e61a2700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.690 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.692+0000 7f83d6ffd700 1 -- 172.21.15.142:0/249544905 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "df99f110-da25-4a6d-93d8-20d8ab22b345", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f83dc00bee0 con 0x7f83e0082000 2019-10-13T00:51:34.693 INFO:teuthology.orchestra.run.smithi142.stdout:2 2019-10-13T00:51:34.696 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f83c8017c60 msgr2=0x7f83c801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.696 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f83c8017c60 0x7f83c801a0f0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.697 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 msgr2=0x7f83e00823f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.697 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f83dc010730 tx=0x7f83dc010ad0).stop 2019-10-13T00:51:34.697 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 shutdown_connections 2019-10-13T00:51:34.697 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f83c8017c60 0x7f83c801a0f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.697 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 --2- 172.21.15.142:0/249544905 >> v2:172.21.15.155:3300/0 conn(0x7f83e008e0f0 0x7f83e0082930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.697 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 --2- 172.21.15.142:0/249544905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83e0082000 0x7f83e00823f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.697 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 shutdown_connections 2019-10-13T00:51:34.698 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 wait complete. 2019-10-13T00:51:34.698 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.698+0000 7f83e7c05700 1 -- 172.21.15.142:0/249544905 >> 172.21.15.142:0/249544905 conn(0x7f83e00a0e10 msgr2=0x7f83e012a340 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:34.706 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:34.706 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 7cbcdaa5-ac79-4661-aa09-faf34ceb2662 3 2019-10-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.828+0000 7f29b80bf700 1 Processor -- start 2019-10-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.828+0000 7f29b80bf700 1 -- start start 2019-10-13T00:51:34.831 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b80bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b80bf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f29b00990f0 0x7f29b0086860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b80bf700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b0060a10 con 0x7f29b0098120 2019-10-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b80bf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b006dac0 con 0x7f29b00990f0 2019-10-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b80bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b006dd90 con 0x7f29b0060400 2019-10-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b665c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b5e5b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f29b00990f0 0x7f29b0086860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b665c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55582/0 (socket says 172.21.15.142:55582) 2019-10-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b5e5b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f29b00990f0 0x7f29b0086860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58122/0 (socket says 172.21.15.142:58122) 2019-10-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.829+0000 7f29b665c700 1 -- 172.21.15.142:0/1878332578 learned_addr learned my addr 172.21.15.142:0/1878332578 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b565a700 1 -- 172.21.15.142:0/1878332578 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981501036 0 0) 0x7f29a80009b0 con 0x7f29b0098120 2019-10-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 -- 172.21.15.142:0/1878332578 >> v2:172.21.15.142:3300/0 conn(0x7f29b00990f0 msgr2=0x7f29b0086860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 --2- 172.21.15.142:0/1878332578 >> v2:172.21.15.142:3300/0 conn(0x7f29b00990f0 0x7f29b0086860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 -- 172.21.15.142:0/1878332578 >> v1:172.21.15.155:6789/0 conn(0x7f29b0098120 legacy=0x7f29b00985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 -- 172.21.15.142:0/1878332578 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0097820 con 0x7f29b0060400 2019-10-13T00:51:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/1878332578 >> v2:172.21.15.142:3300/0 conn(0x7f29b00990f0 0x7f29b0086860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f299800fab0 tx=0x7f299800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f299800fab0 tx=0x7f299800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f299800fab0 tx=0x7f299800fe50).ready entity=mon.2 client_cookie=8072340311a2e12a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f299800fab0 tx=0x7f299800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f299800fab0 tx=0x7f299800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b565a700 1 -- 172.21.15.142:0/1878332578 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2998000ce0 con 0x7f29b0060400 2019-10-13T00:51:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b665c700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f299800fab0 tx=0x7f299800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b565a700 1 -- 172.21.15.142:0/1878332578 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2998001010 con 0x7f29b0060400 2019-10-13T00:51:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b565a700 1 -- 172.21.15.142:0/1878332578 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29980016e0 con 0x7f29b0060400 2019-10-13T00:51:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b80bf700 1 -- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 msgr2=0x7f29b0086320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.830+0000 7f29b80bf700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f299800fab0 tx=0x7f299800fe50).stop 2019-10-13T00:51:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.831+0000 7f29b80bf700 1 -- 172.21.15.142:0/1878332578 shutdown_connections 2019-10-13T00:51:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.831+0000 7f29b80bf700 1 --2- 172.21.15.142:0/1878332578 >> v2:172.21.15.142:3300/0 conn(0x7f29b00990f0 0x7f29b0086860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.831+0000 7f29b80bf700 1 --2- 172.21.15.142:0/1878332578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b0060400 0x7f29b0086320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.831+0000 7f29b80bf700 1 -- 172.21.15.142:0/1878332578 shutdown_connections 2019-10-13T00:51:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.831+0000 7f29b80bf700 1 -- 172.21.15.142:0/1878332578 wait complete. 2019-10-13T00:51:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.831+0000 7f29b80bf700 1 -- 172.21.15.142:0/1878332578 >> 172.21.15.142:0/1878332578 conn(0x7f29b0062020 msgr2=0x7f29b005f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.833+0000 7f29b80bf700 1 Processor -- start 2019-10-13T00:51:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.833+0000 7f29b80bf700 1 -- start start 2019-10-13T00:51:34.839 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b80bf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f29b009eed0 0x7f29b014cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.840 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b80bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b009e700 0x7f29b0156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.840 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b80bf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b009eb60 con 0x7f29b014c880 2019-10-13T00:51:34.840 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b80bf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b009cf70 con 0x7f29b009eed0 2019-10-13T00:51:34.840 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b80bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b009c540 con 0x7f29b009e700 2019-10-13T00:51:34.840 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b665c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f29b014c880 0x7f29b009df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51290/0 (socket says 172.21.15.142:51290) 2019-10-13T00:51:34.840 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b665c700 1 -- 172.21.15.142:0/2647985119 learned_addr learned my addr 172.21.15.142:0/2647985119 (peer_addr_for_me v1:172.21.15.142:0/0) 2019-10-13T00:51:34.841 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f29b009eed0 0x7f29b014cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.841 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b6e5d700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b009e700 0x7f29b0156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.841 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 -- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b009e700 msgr2=0x7f29b0156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.841 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b009e700 0x7f29b0156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:34.841 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695457048 0 0) 0x7f2998007be0 con 0x7f29b014c880 2019-10-13T00:51:34.841 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 -- 172.21.15.142:0/2647985119 >> v1:172.21.15.142:6789/0 conn(0x7f29b014c880 legacy=0x7f29b009df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:34.842 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 -- 172.21.15.142:0/2647985119 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b009d600 con 0x7f29b009eed0 2019-10-13T00:51:34.842 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b6e5d700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b009e700 0x7f29b0156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:34.842 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> v2:172.21.15.142:3300/0 conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.842 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> v2:172.21.15.142:3300/0 conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.842 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.842 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.842 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.834+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29a0000940 con 0x7f29b009eed0 2019-10-13T00:51:34.843 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.843 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.843 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0000e80 con 0x7f29b009eed0 2019-10-13T00:51:34.843 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29a0001550 con 0x7f29b009eed0 2019-10-13T00:51:34.843 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b009d600 con 0x7f29b009eed0 2019-10-13T00:51:34.843 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b009d600 con 0x7f29b009eed0 2019-10-13T00:51:34.843 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.844 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f29a0001070 con 0x7f29b009eed0 2019-10-13T00:51:34.844 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.844 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29aeffd700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2994017c60 0x7f299401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:34.844 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f29a002ffd0 con 0x7f29b009eed0 2019-10-13T00:51:34.844 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.835+0000 7f29b665c700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2994017c60 0x7f299401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:34.844 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.836+0000 7f29b665c700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2994017c60 0x7f299401a0f0 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:34.845 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.844+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b009d600 con 0x7f29b009eed0 2019-10-13T00:51:34.845 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.847+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:34.845 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:34.847+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f29a0009270 con 0x7f29b009eed0 2019-10-13T00:51:35.007 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.009+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7cbcdaa5-ac79-4661-aa09-faf34ceb2662", "id": 3} v 0) v1 -- 0x7f29b009c140 con 0x7f29b009eed0 2019-10-13T00:51:35.010 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.012+0000 7f29b5e5b700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.011 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.013+0000 7f29aeffd700 1 -- 172.21.15.142:0/2647985119 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7cbcdaa5-ac79-4661-aa09-faf34ceb2662", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f29a0001050 con 0x7f29b009eed0 2019-10-13T00:51:35.014 INFO:teuthology.orchestra.run.smithi142.stdout:3 2019-10-13T00:51:35.017 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2994017c60 msgr2=0x7f299401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.017 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2994017c60 0x7f299401a0f0 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.017 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 msgr2=0x7f29b014cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.017 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f29a00103b0 tx=0x7f29a0010750).stop 2019-10-13T00:51:35.017 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 shutdown_connections 2019-10-13T00:51:35.017 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2994017c60 0x7f299401a0f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.018 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29b009e700 0x7f29b0156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.018 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 --2- 172.21.15.142:0/2647985119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29b009eed0 0x7f29b014cc70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.018 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.018+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 shutdown_connections 2019-10-13T00:51:35.018 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.019+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 wait complete. 2019-10-13T00:51:35.018 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.019+0000 7f29b80bf700 1 -- 172.21.15.142:0/2647985119 >> 172.21.15.142:0/2647985119 conn(0x7f29b0091340 msgr2=0x7f29b0081340 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:35.026 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:35.026 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 83c00323-8853-4f30-84a0-711565f5a804 4 2019-10-13T00:51:35.148 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.145+0000 7f344b8e1700 1 Processor -- start 2019-10-13T00:51:35.148 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.145+0000 7f344b8e1700 1 -- start start 2019-10-13T00:51:35.148 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f344b8e1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:35.148 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f344b8e1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34440990f0 0x7f3444086860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:35.148 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f344b8e1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444060a10 con 0x7f3444098120 2019-10-13T00:51:35.149 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f344b8e1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344406dac0 con 0x7f34440990f0 2019-10-13T00:51:35.149 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f344b8e1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344406dd90 con 0x7f3444060400 2019-10-13T00:51:35.149 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f3449e7e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.149 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f344967d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34440990f0 0x7f3444086860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.149 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f3449e7e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55596/0 (socket says 172.21.15.142:55596) 2019-10-13T00:51:35.149 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f344967d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34440990f0 0x7f3444086860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58136/0 (socket says 172.21.15.142:58136) 2019-10-13T00:51:35.149 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f3449e7e700 1 -- 172.21.15.142:0/1801750411 learned_addr learned my addr 172.21.15.142:0/1801750411 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:35.150 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f3449e7e700 1 -- 172.21.15.142:0/1801750411 >> v2:172.21.15.142:3300/0 conn(0x7f34440990f0 msgr2=0x7f3444086860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:35.150 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f3449e7e700 1 --2- 172.21.15.142:0/1801750411 >> v2:172.21.15.142:3300/0 conn(0x7f34440990f0 0x7f3444086860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:35.150 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f3448e7c700 1 -- 172.21.15.142:0/1801750411 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979995104 0 0) 0x7f343c0009b0 con 0x7f3444098120 2019-10-13T00:51:35.150 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.146+0000 7f3449e7e700 1 -- 172.21.15.142:0/1801750411 >> v1:172.21.15.155:6789/0 conn(0x7f3444098120 legacy=0x7f34440985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.150 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3449e7e700 1 -- 172.21.15.142:0/1801750411 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444097820 con 0x7f3444060400 2019-10-13T00:51:35.150 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f344967d700 1 --2- 172.21.15.142:0/1801750411 >> v2:172.21.15.142:3300/0 conn(0x7f34440990f0 0x7f3444086860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:35.150 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3449e7e700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f344000fab0 tx=0x7f344000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.151 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3449e7e700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f344000fab0 tx=0x7f344000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.151 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3449e7e700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f344000fab0 tx=0x7f344000fe50).ready entity=mon.2 client_cookie=11ff9a02ffb261fb server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.151 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3449e7e700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f344000fab0 tx=0x7f344000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.151 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3448e7c700 1 -- 172.21.15.142:0/1801750411 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3440000ce0 con 0x7f3444060400 2019-10-13T00:51:35.151 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3449e7e700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f344000fab0 tx=0x7f344000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.151 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3449e7e700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f344000fab0 tx=0x7f344000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.151 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3448e7c700 1 -- 172.21.15.142:0/1801750411 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3440001010 con 0x7f3444060400 2019-10-13T00:51:35.152 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f3448e7c700 1 -- 172.21.15.142:0/1801750411 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34400016e0 con 0x7f3444060400 2019-10-13T00:51:35.152 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f344b8e1700 1 -- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 msgr2=0x7f3444086320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.152 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.147+0000 7f344b8e1700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f344000fab0 tx=0x7f344000fe50).stop 2019-10-13T00:51:35.152 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.148+0000 7f344b8e1700 1 -- 172.21.15.142:0/1801750411 shutdown_connections 2019-10-13T00:51:35.152 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.148+0000 7f344b8e1700 1 --2- 172.21.15.142:0/1801750411 >> v2:172.21.15.142:3300/0 conn(0x7f34440990f0 0x7f3444086860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:35.152 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.148+0000 7f344b8e1700 1 --2- 172.21.15.142:0/1801750411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3444060400 0x7f3444086320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.152 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.148+0000 7f344b8e1700 1 -- 172.21.15.142:0/1801750411 shutdown_connections 2019-10-13T00:51:35.153 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.148+0000 7f344b8e1700 1 -- 172.21.15.142:0/1801750411 wait complete. 2019-10-13T00:51:35.153 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.148+0000 7f344b8e1700 1 -- 172.21.15.142:0/1801750411 >> 172.21.15.142:0/1801750411 conn(0x7f3444062020 msgr2=0x7f344405f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:35.153 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.150+0000 7f344b8e1700 1 Processor -- start 2019-10-13T00:51:35.153 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.150+0000 7f344b8e1700 1 -- start start 2019-10-13T00:51:35.156 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344b8e1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.156 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344b8e1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f344409cd70 0x7f344408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.156 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344b8e1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444099950 con 0x7f344409d540 2019-10-13T00:51:35.156 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344b8e1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344409abb0 con 0x7f344409cd70 2019-10-13T00:51:35.156 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344b8e1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344409ae80 con 0x7f344408f670 2019-10-13T00:51:35.156 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.157 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344967d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f344409d540 0x7f344409c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51306/0 (socket says 172.21.15.142:51306) 2019-10-13T00:51:35.157 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344967d700 1 -- 172.21.15.142:0/660303753 learned_addr learned my addr 172.21.15.142:0/660303753 (peer_addr_for_me v1:172.21.15.142:0/0) 2019-10-13T00:51:35.157 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344a67f700 1 --2- 172.21.15.142:0/660303753 >> v2:172.21.15.155:3300/0 conn(0x7f344409cd70 0x7f344408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.157 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920206288 0 0) 0x7f34340010b0 con 0x7f344409d540 2019-10-13T00:51:35.157 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3424002e70 con 0x7f344409d540 2019-10-13T00:51:35.157 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 -- 172.21.15.142:0/660303753 >> v2:172.21.15.155:3300/0 conn(0x7f344409cd70 msgr2=0x7f344408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.157 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> v2:172.21.15.155:3300/0 conn(0x7f344409cd70 0x7f344408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.158 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 -- 172.21.15.142:0/660303753 >> v1:172.21.15.142:6789/0 conn(0x7f344409d540 legacy=0x7f344409c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.158 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 -- 172.21.15.142:0/660303753 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344409d210 con 0x7f344408f670 2019-10-13T00:51:35.158 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.158 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.158 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.158 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.158 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.159 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3440008560 con 0x7f344408f670 2019-10-13T00:51:35.159 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.159 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34400056e0 con 0x7f344408f670 2019-10-13T00:51:35.159 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3440005db0 con 0x7f344408f670 2019-10-13T00:51:35.159 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.151+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344409d210 con 0x7f344408f670 2019-10-13T00:51:35.159 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344409d210 con 0x7f344408f670 2019-10-13T00:51:35.159 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.160 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f34400058d0 con 0x7f344408f670 2019-10-13T00:51:35.160 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.160 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f343a7fc700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3424017ca0 0x7f342401a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.160 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f34400745e0 con 0x7f344408f670 2019-10-13T00:51:35.160 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f344967d700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3424017ca0 0x7f342401a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.160 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.152+0000 7f344967d700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3424017ca0 0x7f342401a110 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.161 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.161+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344409bc70 con 0x7f344408f670 2019-10-13T00:51:35.162 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.164+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.162 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.164+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f34400081d0 con 0x7f344408f670 2019-10-13T00:51:35.322 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.325+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd new", "uuid": "83c00323-8853-4f30-84a0-711565f5a804", "id": 4} v 0) v1 -- 0x7f344409bc70 con 0x7f344408f670 2019-10-13T00:51:35.327 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.329+0000 7f3449e7e700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.327 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.330+0000 7f343a7fc700 1 -- 172.21.15.142:0/660303753 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83c00323-8853-4f30-84a0-711565f5a804", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f34400056e0 con 0x7f344408f670 2019-10-13T00:51:35.330 INFO:teuthology.orchestra.run.smithi142.stdout:4 2019-10-13T00:51:35.332 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3424017ca0 msgr2=0x7f342401a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.333 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3424017ca0 0x7f342401a110 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.333 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 msgr2=0x7f344408fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.333 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f3440054730 tx=0x7f3440054ad0).stop 2019-10-13T00:51:35.333 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 shutdown_connections 2019-10-13T00:51:35.333 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 --2- 172.21.15.142:0/660303753 >> v2:172.21.15.155:3300/0 conn(0x7f344409cd70 0x7f344408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.333 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3424017ca0 0x7f342401a110 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.333 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 --2- 172.21.15.142:0/660303753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f344408f670 0x7f344408fa60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.334 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 shutdown_connections 2019-10-13T00:51:35.334 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 wait complete. 2019-10-13T00:51:35.334 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.334+0000 7f344b8e1700 1 -- 172.21.15.142:0/660303753 >> 172.21.15.142:0/660303753 conn(0x7f3444091340 msgr2=0x7f3444081340 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:35.341 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:35.342 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new d1142116-2074-437f-bb80-2cdef5bcd668 5 2019-10-13T00:51:35.463 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.461+0000 7fb27c87c700 1 Processor -- start 2019-10-13T00:51:35.463 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.461+0000 7fb27c87c700 1 -- start start 2019-10-13T00:51:35.463 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27c87c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb278086b20 0x7fb278098920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:35.463 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27c87c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb278060a10 con 0x7fb278060400 2019-10-13T00:51:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27c87c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb278062890 con 0x7fb2780990f0 2019-10-13T00:51:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27c87c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27806dac0 con 0x7fb278086b20 2019-10-13T00:51:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb278086b20 0x7fb278098920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb278086b20 0x7fb278098920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58148/0 (socket says 172.21.15.142:58148) 2019-10-13T00:51:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 -- 172.21.15.142:0/176125934 learned_addr learned my addr 172.21.15.142:0/176125934 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb276d9d700 1 --1- 172.21.15.142:0/176125934 >> v1:172.21.15.142:6789/0 conn(0x7fb278060400 0x7fb278086320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51312/0 (socket says 172.21.15.142:51312) 2019-10-13T00:51:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb275d9b700 1 -- 172.21.15.142:0/176125934 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786840493 0 0) 0x7fb26c001080 con 0x7fb2780990f0 2019-10-13T00:51:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb275d9b700 1 -- 172.21.15.142:0/176125934 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb260002e70 con 0x7fb2780990f0 2019-10-13T00:51:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb275d9b700 1 -- 172.21.15.142:0/176125934 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385547827 0 0) 0x7fb268001480 con 0x7fb278060400 2019-10-13T00:51:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 -- 172.21.15.142:0/176125934 >> v1:172.21.15.155:6789/0 conn(0x7fb2780990f0 legacy=0x7fb278098120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 -- 172.21.15.142:0/176125934 >> v1:172.21.15.142:6789/0 conn(0x7fb278060400 legacy=0x7fb278086320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 -- 172.21.15.142:0/176125934 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb278097820 con 0x7fb278086b20 2019-10-13T00:51:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- 172.21.15.142:0/176125934 >> v2:172.21.15.142:3300/0 conn(0x7fb278086b20 0x7fb278098920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb27000f720 tx=0x7fb27000fac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- 172.21.15.142:0/176125934 >> v2:172.21.15.142:3300/0 conn(0x7fb278086b20 0x7fb278098920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb27000f720 tx=0x7fb27000fac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- 172.21.15.142:0/176125934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278086b20 0x7fb278098920 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7fb27000f720 tx=0x7fb27000fac0).ready entity=mon.0 client_cookie=c9c761eb5ac1300a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- 172.21.15.142:0/176125934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278086b20 0x7fb278098920 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fb27000f720 tx=0x7fb27000fac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb275d9b700 1 -- 172.21.15.142:0/176125934 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2700010e0 con 0x7fb278086b20 2019-10-13T00:51:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- 172.21.15.142:0/176125934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278086b20 0x7fb278098920 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fb27000f720 tx=0x7fb27000fac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27759e700 1 --2- 172.21.15.142:0/176125934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278086b20 0x7fb278098920 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fb27000f720 tx=0x7fb27000fac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb275d9b700 1 -- 172.21.15.142:0/176125934 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb270000a70 con 0x7fb278086b20 2019-10-13T00:51:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb275d9b700 1 -- 172.21.15.142:0/176125934 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2700016c0 con 0x7fb278086b20 2019-10-13T00:51:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27c87c700 1 -- 172.21.15.142:0/176125934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278086b20 msgr2=0x7fb278098920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.462+0000 7fb27c87c700 1 --2- 172.21.15.142:0/176125934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278086b20 0x7fb278098920 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7fb27000f720 tx=0x7fb27000fac0).stop 2019-10-13T00:51:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.463+0000 7fb27c87c700 1 -- 172.21.15.142:0/176125934 shutdown_connections 2019-10-13T00:51:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.463+0000 7fb27c87c700 1 --2- 172.21.15.142:0/176125934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278086b20 0x7fb278098920 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.463+0000 7fb27c87c700 1 -- 172.21.15.142:0/176125934 shutdown_connections 2019-10-13T00:51:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.463+0000 7fb27c87c700 1 -- 172.21.15.142:0/176125934 wait complete. 2019-10-13T00:51:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.463+0000 7fb27c87c700 1 -- 172.21.15.142:0/176125934 >> 172.21.15.142:0/176125934 conn(0x7fb278062020 msgr2=0x7fb27805f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.465+0000 7fb27c87c700 1 Processor -- start 2019-10-13T00:51:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb27c87c700 1 -- start start 2019-10-13T00:51:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb27c87c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb278081fc0 0x7fb278072780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb27c87c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2780823b0 0x7fb278082850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb27c87c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb278082d90 con 0x7fb278072f30 2019-10-13T00:51:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb27c87c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27809a380 con 0x7fb278081fc0 2019-10-13T00:51:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb27c87c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb278093cb0 con 0x7fb2780823b0 2019-10-13T00:51:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb276d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb278081fc0 0x7fb278072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb276d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb278081fc0 0x7fb278072780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58152/0 (socket says 172.21.15.142:58152) 2019-10-13T00:51:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb276d9d700 1 -- 172.21.15.142:0/2805737336 learned_addr learned my addr 172.21.15.142:0/2805737336 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.466+0000 7fb27759e700 1 --2- 172.21.15.142:0/2805737336 >> v2:172.21.15.155:3300/0 conn(0x7fb2780823b0 0x7fb278082850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 -- 172.21.15.142:0/2805737336 >> v2:172.21.15.155:3300/0 conn(0x7fb2780823b0 msgr2=0x7fb278082850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> v2:172.21.15.155:3300/0 conn(0x7fb2780823b0 0x7fb278082850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 -- 172.21.15.142:0/2805737336 >> v1:172.21.15.155:6789/0 conn(0x7fb278072f30 legacy=0x7fb27809b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 -- 172.21.15.142:0/2805737336 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27809aa10 con 0x7fb278081fc0 2019-10-13T00:51:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> v2:172.21.15.142:3300/0 conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> v2:172.21.15.142:3300/0 conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb2677fe700 1 -- 172.21.15.142:0/2805737336 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb268001010 con 0x7fb278081fc0 2019-10-13T00:51:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb2677fe700 1 -- 172.21.15.142:0/2805737336 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2680045b0 con 0x7fb278081fc0 2019-10-13T00:51:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb2677fe700 1 -- 172.21.15.142:0/2805737336 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb268004bc0 con 0x7fb278081fc0 2019-10-13T00:51:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27809aa10 con 0x7fb278081fc0 2019-10-13T00:51:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.467+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27809aa10 con 0x7fb278081fc0 2019-10-13T00:51:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.468+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.468+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.468+0000 7fb2677fe700 1 -- 172.21.15.142:0/2805737336 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb2680045b0 con 0x7fb278081fc0 2019-10-13T00:51:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.468+0000 7fb2677fe700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb25c017c60 0x7fb25c01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.468+0000 7fb2677fe700 1 -- 172.21.15.142:0/2805737336 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7fb268030420 con 0x7fb278081fc0 2019-10-13T00:51:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.468+0000 7fb27659c700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb25c017c60 0x7fb25c01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.468+0000 7fb27659c700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb25c017c60 0x7fb25c01a0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.476+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27809aa10 con 0x7fb278081fc0 2019-10-13T00:51:35.477 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.479+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.477 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.479+0000 7fb2677fe700 1 -- 172.21.15.142:0/2805737336 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb268001270 con 0x7fb278081fc0 2019-10-13T00:51:35.635 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.637+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d1142116-2074-437f-bb80-2cdef5bcd668", "id": 5} v 0) v1 -- 0x7fb27809aa10 con 0x7fb278081fc0 2019-10-13T00:51:35.638 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.641+0000 7fb276d9d700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.639 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.641+0000 7fb2677fe700 1 -- 172.21.15.142:0/2805737336 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d1142116-2074-437f-bb80-2cdef5bcd668", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb2680045b0 con 0x7fb278081fc0 2019-10-13T00:51:35.642 INFO:teuthology.orchestra.run.smithi142.stdout:5 2019-10-13T00:51:35.645 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb25c017c60 msgr2=0x7fb25c01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.645 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb25c017c60 0x7fb25c01a0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.645 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 msgr2=0x7fb278072780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.645 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7fb2680107a0 tx=0x7fb268010b40).stop 2019-10-13T00:51:35.645 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 shutdown_connections 2019-10-13T00:51:35.645 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 --2- 172.21.15.142:0/2805737336 >> v2:172.21.15.155:3300/0 conn(0x7fb2780823b0 0x7fb278082850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.646 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb25c017c60 0x7fb25c01a0f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.646 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 --2- 172.21.15.142:0/2805737336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb278081fc0 0x7fb278072780 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.646 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 shutdown_connections 2019-10-13T00:51:35.646 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 wait complete. 2019-10-13T00:51:35.646 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.646+0000 7fb27c87c700 1 -- 172.21.15.142:0/2805737336 >> 172.21.15.142:0/2805737336 conn(0x7fb2780814b0 msgr2=0x7fb2780a30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:35.654 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:35.654 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 410ababe-9879-4664-941a-5f5764c1eb0e 6 2019-10-13T00:51:35.776 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.774+0000 7fcca6d4d700 1 Processor -- start 2019-10-13T00:51:35.777 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.774+0000 7fcca6d4d700 1 -- start start 2019-10-13T00:51:35.777 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca6d4d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcca0060400 0x7fcca0086320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:35.777 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca6d4d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcca00990f0 0x7fcca0086860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:35.777 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca6d4d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcca0098120 0x7fcca00985c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:35.777 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca6d4d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca006dac0 con 0x7fcca0060400 2019-10-13T00:51:35.778 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca6d4d700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca006dd90 con 0x7fcca0098120 2019-10-13T00:51:35.778 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca6d4d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca0097820 con 0x7fcca00990f0 2019-10-13T00:51:35.778 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcca0060400 0x7fcca0086320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.778 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca4ae9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcca00990f0 0x7fcca0086860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.778 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca4ae9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcca00990f0 0x7fcca0086860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55624/0 (socket says 172.21.15.142:55624) 2019-10-13T00:51:35.778 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcca0060400 0x7fcca0086320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58160/0 (socket says 172.21.15.142:58160) 2019-10-13T00:51:35.778 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca4ae9700 1 -- 172.21.15.142:0/964819631 learned_addr learned my addr 172.21.15.142:0/964819631 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca5aeb700 1 --2- 172.21.15.142:0/964819631 >> v2:172.21.15.155:3300/0 conn(0x7fcca0098120 0x7fcca00985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 -- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcca00990f0 msgr2=0x7fcca0086860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcca00990f0 0x7fcca0086860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 -- 172.21.15.142:0/964819631 >> v2:172.21.15.155:3300/0 conn(0x7fcca0098120 msgr2=0x7fcca00985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> v2:172.21.15.155:3300/0 conn(0x7fcca0098120 0x7fcca00985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 -- 172.21.15.142:0/964819631 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca0062440 con 0x7fcca0060400 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcca00990f0 0x7fcca0086860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> v2:172.21.15.142:3300/0 conn(0x7fcca0060400 0x7fcca0086320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc9800b2c0 tx=0x7fcc9800bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.780 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> v2:172.21.15.142:3300/0 conn(0x7fcca0060400 0x7fcca0086320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc9800b2c0 tx=0x7fcc9800bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.780 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.775+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca0060400 0x7fcca0086320 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fcc9800b2c0 tx=0x7fcc9800bad0).ready entity=mon.0 client_cookie=6866d13ecacd2ae6 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.780 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca0060400 0x7fcca0086320 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fcc9800b2c0 tx=0x7fcc9800bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.780 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca0060400 0x7fcca0086320 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fcc9800b2c0 tx=0x7fcc9800bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.780 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca52ea700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca0060400 0x7fcca0086320 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fcc9800b2c0 tx=0x7fcc9800bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.780 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcc9ffff700 1 -- 172.21.15.142:0/964819631 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc98000ce0 con 0x7fcca0060400 2019-10-13T00:51:35.780 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcc9ffff700 1 -- 172.21.15.142:0/964819631 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc98009bd0 con 0x7fcca0060400 2019-10-13T00:51:35.781 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcc9ffff700 1 -- 172.21.15.142:0/964819631 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc9800a2a0 con 0x7fcca0060400 2019-10-13T00:51:35.781 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 -- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca0060400 msgr2=0x7fcca0086320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.781 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca0060400 0x7fcca0086320 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fcc9800b2c0 tx=0x7fcc9800bad0).stop 2019-10-13T00:51:35.781 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 -- 172.21.15.142:0/964819631 shutdown_connections 2019-10-13T00:51:35.781 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/964819631 >> v2:172.21.15.155:3300/0 conn(0x7fcca0098120 0x7fcca00985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:35.781 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcca00990f0 0x7fcca0086860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:35.781 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/964819631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca0060400 0x7fcca0086320 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.782 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 -- 172.21.15.142:0/964819631 shutdown_connections 2019-10-13T00:51:35.782 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 -- 172.21.15.142:0/964819631 wait complete. 2019-10-13T00:51:35.782 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.776+0000 7fcca6d4d700 1 -- 172.21.15.142:0/964819631 >> 172.21.15.142:0/964819631 conn(0x7fcca0062020 msgr2=0x7fcca005f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:35.782 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.779+0000 7fcca6d4d700 1 Processor -- start 2019-10-13T00:51:35.782 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.779+0000 7fcca6d4d700 1 -- start start 2019-10-13T00:51:35.785 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca6d4d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcca009eea0 0x7fcca009ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.785 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca6d4d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcca0090280 0x7fcca009e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.785 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca6d4d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca0090800 con 0x7fcca008f670 2019-10-13T00:51:35.786 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca6d4d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca009cf40 con 0x7fcca009eea0 2019-10-13T00:51:35.786 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca6d4d700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca009c510 con 0x7fcca0090280 2019-10-13T00:51:35.786 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcca009eea0 0x7fcca009ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.786 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca52ea700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcca008f670 0x7fcca008fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51330/0 (socket says 172.21.15.142:51330) 2019-10-13T00:51:35.786 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca52ea700 1 -- 172.21.15.142:0/685001930 learned_addr learned my addr 172.21.15.142:0/685001930 (peer_addr_for_me v1:172.21.15.142:0/0) 2019-10-13T00:51:35.786 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca5aeb700 1 --2- 172.21.15.142:0/685001930 >> v2:172.21.15.155:3300/0 conn(0x7fcca0090280 0x7fcca009e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.786 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 -- 172.21.15.142:0/685001930 >> v2:172.21.15.155:3300/0 conn(0x7fcca0090280 msgr2=0x7fcca009e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> v2:172.21.15.155:3300/0 conn(0x7fcca0090280 0x7fcca009e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 -- 172.21.15.142:0/685001930 >> v1:172.21.15.142:6789/0 conn(0x7fcca008f670 legacy=0x7fcca008fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 -- 172.21.15.142:0/685001930 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca009d5d0 con 0x7fcca009eea0 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893123673 0 0) 0x7fcc98000cc0 con 0x7fcca008f670 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> v2:172.21.15.142:3300/0 conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> v2:172.21.15.142:3300/0 conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.787 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.788 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc9000bd50 con 0x7fcca009eea0 2019-10-13T00:51:35.788 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.788 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.788 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc90000940 con 0x7fcca009eea0 2019-10-13T00:51:35.788 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc9000c330 con 0x7fcca009eea0 2019-10-13T00:51:35.788 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca009d5d0 con 0x7fcca009eea0 2019-10-13T00:51:35.788 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.780+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca009d5d0 con 0x7fcca009eea0 2019-10-13T00:51:35.789 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.781+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.789 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.781+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fcc9000bb60 con 0x7fcca009eea0 2019-10-13T00:51:35.789 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.781+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.789 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.781+0000 7fcc9dffb700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fcc80017c60 0x7fcc8001a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:35.789 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.781+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7fcc90030070 con 0x7fcca009eea0 2019-10-13T00:51:35.789 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.781+0000 7fcca52ea700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fcc80017c60 0x7fcc8001a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:35.789 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.781+0000 7fcca52ea700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fcc80017c60 0x7fcc8001a0f0 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:35.790 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.789+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca009d5d0 con 0x7fcca009eea0 2019-10-13T00:51:35.790 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.792+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.791 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.792+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fcc9000b950 con 0x7fcca009eea0 2019-10-13T00:51:35.950 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.952+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "410ababe-9879-4664-941a-5f5764c1eb0e", "id": 6} v 0) v1 -- 0x7fcca009d5d0 con 0x7fcca009eea0 2019-10-13T00:51:35.953 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.956+0000 7fcca4ae9700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:35.953 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.956+0000 7fcc9dffb700 1 -- 172.21.15.142:0/685001930 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "410ababe-9879-4664-941a-5f5764c1eb0e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcc9000bb60 con 0x7fcca009eea0 2019-10-13T00:51:35.957 INFO:teuthology.orchestra.run.smithi142.stdout:6 2019-10-13T00:51:35.960 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.961+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fcc80017c60 msgr2=0x7fcc8001a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.960 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.961+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fcc80017c60 0x7fcc8001a0f0 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.960 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.961+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 msgr2=0x7fcca009ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:35.960 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.961+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fcc900103b0 tx=0x7fcc90010750).stop 2019-10-13T00:51:35.960 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.962+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 shutdown_connections 2019-10-13T00:51:35.960 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.962+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fcc80017c60 0x7fcc8001a0f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.961 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.962+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/685001930 >> v2:172.21.15.155:3300/0 conn(0x7fcca0090280 0x7fcca009e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.961 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.962+0000 7fcca6d4d700 1 --2- 172.21.15.142:0/685001930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca009eea0 0x7fcca009ded0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:35.961 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.962+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 shutdown_connections 2019-10-13T00:51:35.961 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.962+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 wait complete. 2019-10-13T00:51:35.961 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:35.962+0000 7fcca6d4d700 1 -- 172.21.15.142:0/685001930 >> 172.21.15.142:0/685001930 conn(0x7fcca0091340 msgr2=0x7fcca0081340 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:35.969 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:35.969 INFO:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 1bf99cb1-c304-47f3-9fde-1ca95c7a8266 7 2019-10-13T00:51:36.091 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.089+0000 7f732a80f700 1 Processor -- start 2019-10-13T00:51:36.091 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.089+0000 7f732a80f700 1 -- start start 2019-10-13T00:51:36.091 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f732a80f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7324060400 0x7f7324086320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:36.091 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f732a80f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:36.092 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f732a80f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7324060a10 con 0x7f7324098120 2019-10-13T00:51:36.092 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f732a80f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732406dac0 con 0x7f7324060400 2019-10-13T00:51:36.092 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f732a80f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732406dd90 con 0x7f73240990f0 2019-10-13T00:51:36.092 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:36.092 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7328dac700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7324060400 0x7f7324086320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:36.092 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f73295ad700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7324098120 0x7f73240985c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51338/0 (socket says 172.21.15.142:51338) 2019-10-13T00:51:36.092 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7328dac700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7324060400 0x7f7324086320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58172/0 (socket says 172.21.15.142:58172) 2019-10-13T00:51:36.093 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55636/0 (socket says 172.21.15.142:55636) 2019-10-13T00:51:36.093 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f73295ad700 1 -- 172.21.15.142:0/2872455100 learned_addr learned my addr 172.21.15.142:0/2872455100 (peer_addr_for_me v1:172.21.15.142:0/0) 2019-10-13T00:51:36.093 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 -- 172.21.15.142:0/2872455100 >> v2:172.21.15.142:3300/0 conn(0x7f7324060400 msgr2=0x7f7324086320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:36.093 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- 172.21.15.142:0/2872455100 >> v2:172.21.15.142:3300/0 conn(0x7f7324060400 0x7f7324086320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:36.093 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 -- 172.21.15.142:0/2872455100 >> v1:172.21.15.142:6789/0 conn(0x7f7324098120 legacy=0x7f73240985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:36.093 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 -- 172.21.15.142:0/2872455100 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7324098dc0 con 0x7f73240990f0 2019-10-13T00:51:36.093 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f73237fe700 1 -- 172.21.15.142:0/2872455100 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228202163 0 0) 0x7f731c001150 con 0x7f7324098120 2019-10-13T00:51:36.094 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7328dac700 1 --2- 172.21.15.142:0/2872455100 >> v2:172.21.15.142:3300/0 conn(0x7f7324060400 0x7f7324086320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:36.094 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f730c00af40 tx=0x7f730c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.094 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f730c00af40 tx=0x7f730c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.094 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f730c00af40 tx=0x7f730c00b750).ready entity=mon.2 client_cookie=d0691c8687fe78a5 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:36.094 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f730c00af40 tx=0x7f730c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.094 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f73237fe700 1 -- 172.21.15.142:0/2872455100 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f730c000960 con 0x7f73240990f0 2019-10-13T00:51:36.094 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.090+0000 7f7323fff700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f730c00af40 tx=0x7f730c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.095 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f7323fff700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f730c00af40 tx=0x7f730c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.095 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f73237fe700 1 -- 172.21.15.142:0/2872455100 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c009850 con 0x7f73240990f0 2019-10-13T00:51:36.095 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f73237fe700 1 -- 172.21.15.142:0/2872455100 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f730c009f20 con 0x7f73240990f0 2019-10-13T00:51:36.095 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 -- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 msgr2=0x7f7324086860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:36.095 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f730c00af40 tx=0x7f730c00b750).stop 2019-10-13T00:51:36.095 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 -- 172.21.15.142:0/2872455100 shutdown_connections 2019-10-13T00:51:36.095 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 --2- 172.21.15.142:0/2872455100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73240990f0 0x7f7324086860 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:36.096 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 --2- 172.21.15.142:0/2872455100 >> v2:172.21.15.142:3300/0 conn(0x7f7324060400 0x7f7324086320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:36.096 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 -- 172.21.15.142:0/2872455100 shutdown_connections 2019-10-13T00:51:36.096 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 -- 172.21.15.142:0/2872455100 wait complete. 2019-10-13T00:51:36.096 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.091+0000 7f732a80f700 1 -- 172.21.15.142:0/2872455100 >> 172.21.15.142:0/2872455100 conn(0x7f7324062020 msgr2=0x7f732405f910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:36.099 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.093+0000 7f732a80f700 1 Processor -- start 2019-10-13T00:51:36.099 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.093+0000 7f732a80f700 1 -- start start 2019-10-13T00:51:36.099 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f732a80f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:36.100 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f732a80f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7324072d80 0x7f7324082840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:36.100 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f732a80f700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73240a06b0 con 0x7f732409b6b0 2019-10-13T00:51:36.100 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f732a80f700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7324099a80 con 0x7f7324072d80 2019-10-13T00:51:36.100 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f732a80f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732409a870 con 0x7f732408f9d0 2019-10-13T00:51:36.100 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f7328dac700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:36.100 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f7328dac700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55638/0 (socket says 172.21.15.142:55638) 2019-10-13T00:51:36.101 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.094+0000 7f7328dac700 1 -- 172.21.15.142:0/79519459 learned_addr learned my addr 172.21.15.142:0/79519459 (peer_addr_for_me v2:172.21.15.142:0/0) 2019-10-13T00:51:36.101 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f73295ad700 1 --2- 172.21.15.142:0/79519459 >> v2:172.21.15.155:3300/0 conn(0x7f7324072d80 0x7f7324082840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:36.101 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 -- 172.21.15.142:0/79519459 >> v2:172.21.15.155:3300/0 conn(0x7f7324072d80 msgr2=0x7f7324082840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:36.101 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> v2:172.21.15.155:3300/0 conn(0x7f7324072d80 0x7f7324082840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:36.101 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 -- 172.21.15.142:0/79519459 >> v1:172.21.15.155:6789/0 conn(0x7f732409b6b0 legacy=0x7f7324082040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:36.101 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 -- 172.21.15.142:0/79519459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73240a01b0 con 0x7f732408f9d0 2019-10-13T00:51:36.101 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f73295ad700 1 --2- 172.21.15.142:0/79519459 >> v2:172.21.15.155:3300/0 conn(0x7f7324072d80 0x7f7324082840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-10-13T00:51:36.102 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.102 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.102 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:36.102 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.102 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.102 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.102 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f73217fa700 1 -- 172.21.15.142:0/79519459 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7314000cc0 con 0x7f732408f9d0 2019-10-13T00:51:36.103 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f73217fa700 1 -- 172.21.15.142:0/79519459 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7314001490 con 0x7f732408f9d0 2019-10-13T00:51:36.103 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f73217fa700 1 -- 172.21.15.142:0/79519459 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7314001b60 con 0x7f732408f9d0 2019-10-13T00:51:36.103 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73240a01b0 con 0x7f732408f9d0 2019-10-13T00:51:36.103 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73240a01b0 con 0x7f732408f9d0 2019-10-13T00:51:36.103 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.095+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.103 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.096+0000 7f73217fa700 1 -- 172.21.15.142:0/79519459 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f7314001010 con 0x7f732408f9d0 2019-10-13T00:51:36.103 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.096+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.104 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.096+0000 7f73217fa700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7308017c60 0x7f730801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:36.104 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.096+0000 7f73217fa700 1 -- 172.21.15.142:0/79519459 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f7314030440 con 0x7f732408f9d0 2019-10-13T00:51:36.104 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.096+0000 7f7323fff700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7308017c60 0x7f730801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:36.104 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.096+0000 7f7323fff700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7308017c60 0x7f730801a0f0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:36.104 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.104+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732409a870 con 0x7f732408f9d0 2019-10-13T00:51:36.105 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.108+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.106 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.108+0000 7f73217fa700 1 -- 172.21.15.142:0/79519459 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f7314002030 con 0x7f732408f9d0 2019-10-13T00:51:36.264 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.266+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1bf99cb1-c304-47f3-9fde-1ca95c7a8266", "id": 7} v 0) v1 -- 0x7f732409a870 con 0x7f732408f9d0 2019-10-13T00:51:36.268 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.270+0000 7f7328dac700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:36.268 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.270+0000 7f73217fa700 1 -- 172.21.15.142:0/79519459 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1bf99cb1-c304-47f3-9fde-1ca95c7a8266", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7314001010 con 0x7f732408f9d0 2019-10-13T00:51:36.271 INFO:teuthology.orchestra.run.smithi142.stdout:7 2019-10-13T00:51:36.274 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7308017c60 msgr2=0x7f730801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:36.274 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7308017c60 0x7f730801a0f0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:36.274 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 msgr2=0x7f732409b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:36.274 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7f73140107a0 tx=0x7f7314010b40).stop 2019-10-13T00:51:36.274 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 shutdown_connections 2019-10-13T00:51:36.274 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 --2- 172.21.15.142:0/79519459 >> v2:172.21.15.155:3300/0 conn(0x7f7324072d80 0x7f7324082840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:36.275 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f7308017c60 0x7f730801a0f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:36.275 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.275+0000 7f732a80f700 1 --2- 172.21.15.142:0/79519459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f732408f9d0 0x7f732409b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:36.275 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.276+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 shutdown_connections 2019-10-13T00:51:36.275 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.276+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 wait complete. 2019-10-13T00:51:36.275 INFO:teuthology.orchestra.run.smithi142.stderr:2019-10-13T00:51:36.276+0000 7f732a80f700 1 -- 172.21.15.142:0/79519459 >> 172.21.15.142:0/79519459 conn(0x7f7324091340 msgr2=0x7f7324081340 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:36.283 INFO:tasks.ceph.osd.0:Restarting daemon 2019-10-13T00:51:36.283 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:36.283 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2019-10-13T00:51:36.287 INFO:tasks.ceph.osd.0:Started 2019-10-13T00:51:36.287 INFO:tasks.ceph.osd.1:Restarting daemon 2019-10-13T00:51:36.287 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:36.287 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2019-10-13T00:51:36.290 INFO:tasks.ceph.osd.1:Started 2019-10-13T00:51:36.290 INFO:tasks.ceph.osd.2:Restarting daemon 2019-10-13T00:51:36.290 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:36.291 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2019-10-13T00:51:36.294 INFO:tasks.ceph.osd.2:Started 2019-10-13T00:51:36.294 INFO:tasks.ceph.osd.3:Restarting daemon 2019-10-13T00:51:36.294 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:36.294 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2019-10-13T00:51:36.297 INFO:tasks.ceph.osd.3:Started 2019-10-13T00:51:36.297 INFO:tasks.ceph.osd.4:Restarting daemon 2019-10-13T00:51:36.298 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:36.298 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2019-10-13T00:51:36.301 INFO:tasks.ceph.osd.4:Started 2019-10-13T00:51:36.302 INFO:tasks.ceph.osd.5:Restarting daemon 2019-10-13T00:51:36.302 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:36.302 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2019-10-13T00:51:36.305 INFO:tasks.ceph.osd.5:Started 2019-10-13T00:51:36.305 INFO:tasks.ceph.osd.6:Restarting daemon 2019-10-13T00:51:36.305 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:36.306 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2019-10-13T00:51:36.309 INFO:tasks.ceph.osd.6:Started 2019-10-13T00:51:36.309 INFO:tasks.ceph.osd.7:Restarting daemon 2019-10-13T00:51:36.309 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:36.309 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2019-10-13T00:51:36.313 INFO:tasks.ceph.osd.7:Started 2019-10-13T00:51:36.313 INFO:tasks.ceph:Waiting for OSDs to come up 2019-10-13T00:51:36.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T00:51:36.886+0000 7fddecf50bc0 -1 Falling back to public interface 2019-10-13T00:51:36.892 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T00:51:36.895+0000 7f74c916fbc0 -1 Falling back to public interface 2019-10-13T00:51:36.896 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T00:51:36.899+0000 7efe4a311bc0 -1 Falling back to public interface 2019-10-13T00:51:36.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T00:51:36.899+0000 7f5ea2053bc0 -1 Falling back to public interface 2019-10-13T00:51:36.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T00:51:36.902+0000 7f5e9c915700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6828/0: (98) Address already in use 2019-10-13T00:51:36.900 INFO:tasks.ceph.osd.4.smithi142.stderr:2019-10-13T00:51:36.902+0000 7fc0cfeacbc0 -1 Falling back to public interface 2019-10-13T00:51:36.908 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T00:51:36.910+0000 7efcdeb70bc0 -1 Falling back to public interface 2019-10-13T00:51:36.916 INFO:tasks.ceph.osd.7.smithi142.stderr:2019-10-13T00:51:36.918+0000 7ff227154bc0 -1 Falling back to public interface 2019-10-13T00:51:36.916 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T00:51:36.918+0000 7f51a2520bc0 -1 Falling back to public interface 2019-10-13T00:51:37.120 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:37.120 INFO:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:37.261 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.260+0000 7f98746cc700 1 Processor -- start 2019-10-13T00:51:37.261 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.260+0000 7f98746cc700 1 -- start start 2019-10-13T00:51:37.261 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f98746cc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f986c09a700 0x7f986c09c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:37.261 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f98746cc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f986c09d620 0x7f986c09cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:37.262 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f98746cc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c064f00 con 0x7f986c099b10 2019-10-13T00:51:37.262 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f98746cc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c09b680 con 0x7f986c09d620 2019-10-13T00:51:37.262 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f98746cc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c09ac50 con 0x7f986c09a700 2019-10-13T00:51:37.262 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f986c09a700 0x7f986c09c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:37.262 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f986c09a700 0x7f986c09c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58432/0 (socket says 172.21.15.155:58432) 2019-10-13T00:51:37.262 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 -- 172.21.15.155:0/2768228387 learned_addr learned my addr 172.21.15.155:0/2768228387 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:37.262 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f987346a700 1 --2- 172.21.15.155:0/2768228387 >> v2:172.21.15.142:3300/0 conn(0x7f986c09d620 0x7f986c09cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:37.263 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 -- 172.21.15.155:0/2768228387 >> v2:172.21.15.142:3300/0 conn(0x7f986c09d620 msgr2=0x7f986c09cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:37.263 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- 172.21.15.155:0/2768228387 >> v2:172.21.15.142:3300/0 conn(0x7f986c09d620 0x7f986c09cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:37.263 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 -- 172.21.15.155:0/2768228387 >> v1:172.21.15.142:6789/0 conn(0x7f986c099b10 legacy=0x7f986c099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:37.263 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 -- 172.21.15.155:0/2768228387 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c09bd10 con 0x7f986c09a700 2019-10-13T00:51:37.263 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- 172.21.15.155:0/2768228387 >> v2:172.21.15.155:3300/0 conn(0x7f986c09a700 0x7f986c09c7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f986800f730 tx=0x7f986800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.263 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- 172.21.15.155:0/2768228387 >> v2:172.21.15.155:3300/0 conn(0x7f986c09a700 0x7f986c09c7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f986800f730 tx=0x7f986800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.263 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- 172.21.15.155:0/2768228387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f986c09a700 0x7f986c09c7f0 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7f986800f730 tx=0x7f986800fad0).ready entity=mon.1 client_cookie=8c43ca81bc3b295a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:37.264 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- 172.21.15.155:0/2768228387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f986c09a700 0x7f986c09c7f0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f986800f730 tx=0x7f986800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.264 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- 172.21.15.155:0/2768228387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f986c09a700 0x7f986c09c7f0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f986800f730 tx=0x7f986800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.264 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9871c67700 1 -- 172.21.15.155:0/2768228387 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9868000960 con 0x7f986c09a700 2019-10-13T00:51:37.264 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9872468700 1 --2- 172.21.15.155:0/2768228387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f986c09a700 0x7f986c09c7f0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f986800f730 tx=0x7f986800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.264 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9871c67700 1 -- 172.21.15.155:0/2768228387 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9868000c90 con 0x7f986c09a700 2019-10-13T00:51:37.264 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f9871c67700 1 -- 172.21.15.155:0/2768228387 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9868001360 con 0x7f986c09a700 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f98746cc700 1 -- 172.21.15.155:0/2768228387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f986c09a700 msgr2=0x7f986c09c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.261+0000 7f98746cc700 1 --2- 172.21.15.155:0/2768228387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f986c09a700 0x7f986c09c7f0 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7f986800f730 tx=0x7f986800fad0).stop 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.262+0000 7f98746cc700 1 -- 172.21.15.155:0/2768228387 shutdown_connections 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.262+0000 7f98746cc700 1 --2- 172.21.15.155:0/2768228387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f986c09a700 0x7f986c09c7f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.262+0000 7f98746cc700 1 --2- 172.21.15.155:0/2768228387 >> v2:172.21.15.142:3300/0 conn(0x7f986c09d620 0x7f986c09cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.262+0000 7f98746cc700 1 -- 172.21.15.155:0/2768228387 shutdown_connections 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.262+0000 7f98746cc700 1 -- 172.21.15.155:0/2768228387 wait complete. 2019-10-13T00:51:37.265 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.262+0000 7f98746cc700 1 -- 172.21.15.155:0/2768228387 >> 172.21.15.155:0/2768228387 conn(0x7f986c099550 msgr2=0x7f986c061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:37.268 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.264+0000 7f98746cc700 1 Processor -- start 2019-10-13T00:51:37.269 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.264+0000 7f98746cc700 1 -- start start 2019-10-13T00:51:37.269 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f98746cc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c073990 0x7f986c0906c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:37.269 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f98746cc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f986c091690 0x7f986c090c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:37.269 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f98746cc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f986c091140 0x7f986c08bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:37.269 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f98746cc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c08ed00 con 0x7f986c091690 2019-10-13T00:51:37.271 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f98746cc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c08efd0 con 0x7f986c091140 2019-10-13T00:51:37.271 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f98746cc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c08d9b0 con 0x7f986c073990 2019-10-13T00:51:37.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f987346a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f986c091140 0x7f986c08bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:37.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f9872468700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f986c091690 0x7f986c090c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:37.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f9872468700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f986c091690 0x7f986c090c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.155:58174/0 (socket says 172.21.15.155:58174) 2019-10-13T00:51:37.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f9872468700 1 -- 172.21.15.155:0/2579789855 learned_addr learned my addr 172.21.15.155:0/2579789855 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:37.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.265+0000 7f9872c69700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c073990 0x7f986c0906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:37.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 -- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c073990 msgr2=0x7f986c0906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:37.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c073990 0x7f986c0906c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 -- 172.21.15.155:0/2579789855 >> v2:172.21.15.155:3300/0 conn(0x7f986c091140 msgr2=0x7f986c08bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> v2:172.21.15.155:3300/0 conn(0x7f986c091140 0x7f986c08bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 -- 172.21.15.155:0/2579789855 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c08fdc0 con 0x7f986c091690 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> v2:172.21.15.142:3300/0 conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f987346a700 1 --2- 172.21.15.155:0/2579789855 >> v2:172.21.15.155:3300/0 conn(0x7f986c091140 0x7f986c08bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> v2:172.21.15.142:3300/0 conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:37.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f98637fe700 1 -- 172.21.15.155:0/2579789855 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9868008170 con 0x7f986c091690 2019-10-13T00:51:37.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f98637fe700 1 -- 172.21.15.155:0/2579789855 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9868005360 con 0x7f986c091690 2019-10-13T00:51:37.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f98637fe700 1 -- 172.21.15.155:0/2579789855 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9868008170 con 0x7f986c091690 2019-10-13T00:51:37.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c08fdc0 con 0x7f986c091690 2019-10-13T00:51:37.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c08fdc0 con 0x7f986c091690 2019-10-13T00:51:37.275 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.266+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.275 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.267+0000 7f98637fe700 1 -- 172.21.15.155:0/2579789855 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f9868005550 con 0x7f986c091690 2019-10-13T00:51:37.275 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.267+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.275 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.267+0000 7f98637fe700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9854017c60 0x7f985401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:37.275 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.267+0000 7f98637fe700 1 -- 172.21.15.155:0/2579789855 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7f98680300d0 con 0x7f986c091690 2019-10-13T00:51:37.275 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.267+0000 7f9872c69700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9854017c60 0x7f985401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:37.275 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.267+0000 7f9872c69700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9854017c60 0x7f985401a0f0 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:37.276 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.275+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c08fdc0 con 0x7f986c091690 2019-10-13T00:51:37.276 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.278+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.276 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.278+0000 7f98637fe700 1 -- 172.21.15.155:0/2579789855 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f986802df50 con 0x7f986c091690 2019-10-13T00:51:37.442 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.445+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f986c08fdc0 con 0x7f986c091690 2019-10-13T00:51:37.443 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.446+0000 7f9872468700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:37.443 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.446+0000 7f98637fe700 1 -- 172.21.15.155:0/2579789855 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6849 (secure 0 0 0) 0x7f986802df50 con 0x7f986c091690 2019-10-13T00:51:37.444 INFO:teuthology.misc.health.smithi155.stdout: 2019-10-13T00:51:37.444 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:36.267867+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:37.452 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9854017c60 msgr2=0x7f985401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:37.452 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9854017c60 0x7f985401a0f0 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.452 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 msgr2=0x7f986c090c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:37.453 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f9868010430 tx=0x7f98680107d0).stop 2019-10-13T00:51:37.453 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 shutdown_connections 2019-10-13T00:51:37.453 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 --2- 172.21.15.155:0/2579789855 >> v2:172.21.15.155:3300/0 conn(0x7f986c091140 0x7f986c08bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.453 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c091690 0x7f986c090c00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.453 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9854017c60 0x7f985401a0f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.453 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 --2- 172.21.15.155:0/2579789855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c073990 0x7f986c0906c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:37.453 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 shutdown_connections 2019-10-13T00:51:37.454 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 wait complete. 2019-10-13T00:51:37.454 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:37.448+0000 7f98746cc700 1 -- 172.21.15.155:0/2579789855 >> 172.21.15.155:0/2579789855 conn(0x7f986c06d080 msgr2=0x7f986c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:37.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-10-13T00:51:37.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T00:51:37.918+0000 7fddecf50bc0 -1 osd.0 0 log_to_monitors {default=true} 2019-10-13T00:51:37.934 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T00:51:37.936+0000 7f74c916fbc0 -1 osd.1 0 log_to_monitors {default=true} 2019-10-13T00:51:37.936 INFO:tasks.ceph.osd.4.smithi142.stderr:2019-10-13T00:51:37.938+0000 7fc0cfeacbc0 -1 osd.4 0 log_to_monitors {default=true} 2019-10-13T00:51:37.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T00:51:37.942+0000 7efe4a311bc0 -1 osd.3 0 log_to_monitors {default=true} 2019-10-13T00:51:37.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T00:51:37.946+0000 7f5ea2053bc0 -1 osd.2 0 log_to_monitors {default=true} 2019-10-13T00:51:37.952 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T00:51:37.954+0000 7efcdeb70bc0 -1 osd.5 0 log_to_monitors {default=true} 2019-10-13T00:51:37.954 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T00:51:37.956+0000 7f51a2520bc0 -1 osd.6 0 log_to_monitors {default=true} 2019-10-13T00:51:37.954 INFO:tasks.ceph.osd.7.smithi142.stderr:2019-10-13T00:51:37.957+0000 7ff227154bc0 -1 osd.7 0 log_to_monitors {default=true} 2019-10-13T00:51:39.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T00:51:39.800+0000 7f74b1778700 -1 osd.1 0 waiting for initial osdmap 2019-10-13T00:51:39.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T00:51:39.800+0000 7f5e8a65c700 -1 osd.2 0 waiting for initial osdmap 2019-10-13T00:51:39.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T00:51:39.801+0000 7efe3291a700 -1 osd.3 0 waiting for initial osdmap 2019-10-13T00:51:39.799 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T00:51:39.802+0000 7efcc7179700 -1 osd.5 0 waiting for initial osdmap 2019-10-13T00:51:39.800 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T00:51:39.802+0000 7f518ab29700 -1 osd.6 0 waiting for initial osdmap 2019-10-13T00:51:39.800 INFO:tasks.ceph.osd.4.smithi142.stderr:2019-10-13T00:51:39.802+0000 7fc0b84b5700 -1 osd.4 0 waiting for initial osdmap 2019-10-13T00:51:39.801 INFO:tasks.ceph.osd.7.smithi142.stderr:2019-10-13T00:51:39.803+0000 7ff20f75d700 -1 osd.7 0 waiting for initial osdmap 2019-10-13T00:51:39.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T00:51:39.806+0000 7fddd5559700 -1 osd.0 0 waiting for initial osdmap 2019-10-13T00:51:44.261 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:44.261 INFO:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:44.381 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.380+0000 7fd1dfc94700 1 Processor -- start 2019-10-13T00:51:44.381 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.380+0000 7fd1dfc94700 1 -- start start 2019-10-13T00:51:44.381 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dfc94700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd1d809d620 0x7fd1d8153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:44.381 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dfc94700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d8064f00 con 0x7fd1d8099b10 2019-10-13T00:51:44.382 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dfc94700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d80987e0 con 0x7fd1d809a700 2019-10-13T00:51:44.382 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dfc94700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d809b680 con 0x7fd1d809d620 2019-10-13T00:51:44.382 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dda30700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd1d809a700 0x7fd1d809c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48676/0 (socket says 172.21.15.155:48676) 2019-10-13T00:51:44.382 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd1d809d620 0x7fd1d8153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.382 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dda30700 1 -- 172.21.15.155:0/1687163783 learned_addr learned my addr 172.21.15.155:0/1687163783 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:44.383 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 -- 172.21.15.155:0/1687163783 >> v1:172.21.15.155:6789/0 conn(0x7fd1d809a700 legacy=0x7fd1d809c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.383 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 -- 172.21.15.155:0/1687163783 >> v1:172.21.15.142:6789/0 conn(0x7fd1d8099b10 legacy=0x7fd1d8099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.383 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dd22f700 1 -- 172.21.15.155:0/1687163783 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026717234 0 0) 0x7fd1c0001150 con 0x7fd1d809a700 2019-10-13T00:51:44.383 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 -- 172.21.15.155:0/1687163783 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d809bd10 con 0x7fd1d809d620 2019-10-13T00:51:44.383 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 --2- 172.21.15.155:0/1687163783 >> v2:172.21.15.155:3300/0 conn(0x7fd1d809d620 0x7fd1d8153c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd1c800af40 tx=0x7fd1c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.383 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 --2- 172.21.15.155:0/1687163783 >> v2:172.21.15.155:3300/0 conn(0x7fd1d809d620 0x7fd1d8153c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd1c800af40 tx=0x7fd1c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.384 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 --2- 172.21.15.155:0/1687163783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d809d620 0x7fd1d8153c20 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7fd1c800af40 tx=0x7fd1c800b750).ready entity=mon.1 client_cookie=a53bc68f59e54903 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:44.384 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 --2- 172.21.15.155:0/1687163783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d809d620 0x7fd1d8153c20 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7fd1c800af40 tx=0x7fd1c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.385 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dd22f700 1 -- 172.21.15.155:0/1687163783 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1c8009b90 con 0x7fd1d809d620 2019-10-13T00:51:44.385 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 --2- 172.21.15.155:0/1687163783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d809d620 0x7fd1d8153c20 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7fd1c800af40 tx=0x7fd1c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.385 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dd22f700 1 -- 172.21.15.155:0/1687163783 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c8000b10 con 0x7fd1d809d620 2019-10-13T00:51:44.385 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dea32700 1 --2- 172.21.15.155:0/1687163783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d809d620 0x7fd1d8153c20 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7fd1c800af40 tx=0x7fd1c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.385 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dd22f700 1 -- 172.21.15.155:0/1687163783 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1c8009b90 con 0x7fd1d809d620 2019-10-13T00:51:44.385 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.381+0000 7fd1dfc94700 1 -- 172.21.15.155:0/1687163783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d809d620 msgr2=0x7fd1d8153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.385 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.382+0000 7fd1dfc94700 1 --2- 172.21.15.155:0/1687163783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d809d620 0x7fd1d8153c20 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7fd1c800af40 tx=0x7fd1c800b750).stop 2019-10-13T00:51:44.386 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.382+0000 7fd1dfc94700 1 -- 172.21.15.155:0/1687163783 shutdown_connections 2019-10-13T00:51:44.386 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.382+0000 7fd1dfc94700 1 --2- 172.21.15.155:0/1687163783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d809d620 0x7fd1d8153c20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.386 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.382+0000 7fd1dfc94700 1 -- 172.21.15.155:0/1687163783 shutdown_connections 2019-10-13T00:51:44.386 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.382+0000 7fd1dfc94700 1 -- 172.21.15.155:0/1687163783 wait complete. 2019-10-13T00:51:44.386 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.382+0000 7fd1dfc94700 1 -- 172.21.15.155:0/1687163783 >> 172.21.15.155:0/1687163783 conn(0x7fd1d8099550 msgr2=0x7fd1d8061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:44.389 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1dfc94700 1 Processor -- start 2019-10-13T00:51:44.389 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1dfc94700 1 -- start start 2019-10-13T00:51:44.389 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1dfc94700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd1d8073990 0x7fd1d808ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.390 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1dfc94700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd1d8153ef0 0x7fd1d81562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.390 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1dfc94700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d80890d0 con 0x7fd1d808fe30 2019-10-13T00:51:44.392 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1dfc94700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d8093d40 con 0x7fd1d8153ef0 2019-10-13T00:51:44.392 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1dfc94700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d8093310 con 0x7fd1d8073990 2019-10-13T00:51:44.392 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1de231700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd1d8073990 0x7fd1d808ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.392 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1de231700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd1d8073990 0x7fd1d808ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58606/0 (socket says 172.21.15.155:58606) 2019-10-13T00:51:44.392 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.385+0000 7fd1de231700 1 -- 172.21.15.155:0/3254353192 learned_addr learned my addr 172.21.15.155:0/3254353192 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:44.392 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1dea32700 1 --2- 172.21.15.155:0/3254353192 >> v2:172.21.15.142:3300/0 conn(0x7fd1d8153ef0 0x7fd1d81562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.393 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 -- 172.21.15.155:0/3254353192 >> v2:172.21.15.142:3300/0 conn(0x7fd1d8153ef0 msgr2=0x7fd1d81562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.393 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> v2:172.21.15.142:3300/0 conn(0x7fd1d8153ef0 0x7fd1d81562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.393 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 -- 172.21.15.155:0/3254353192 >> v1:172.21.15.142:6789/0 conn(0x7fd1d808fe30 legacy=0x7fd1d808f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.393 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 -- 172.21.15.155:0/3254353192 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d808e560 con 0x7fd1d8073990 2019-10-13T00:51:44.393 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1dea32700 1 --2- 172.21.15.155:0/3254353192 >> v2:172.21.15.142:3300/0 conn(0x7fd1d8153ef0 0x7fd1d81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:51:44.394 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> v2:172.21.15.155:3300/0 conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.394 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> v2:172.21.15.155:3300/0 conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.395 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:44.395 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.395 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.395 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.395 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1d6ffd700 1 -- 172.21.15.155:0/3254353192 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1d0000ce0 con 0x7fd1d8073990 2019-10-13T00:51:44.395 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1d6ffd700 1 -- 172.21.15.155:0/3254353192 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d0001420 con 0x7fd1d8073990 2019-10-13T00:51:44.396 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1d6ffd700 1 -- 172.21.15.155:0/3254353192 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1d0001b70 con 0x7fd1d8073990 2019-10-13T00:51:44.396 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d808e560 con 0x7fd1d8073990 2019-10-13T00:51:44.396 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d808e560 con 0x7fd1d8073990 2019-10-13T00:51:44.396 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.386+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.396 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.387+0000 7fd1d6ffd700 1 -- 172.21.15.155:0/3254353192 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd1d0001610 con 0x7fd1d8073990 2019-10-13T00:51:44.396 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.387+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.396 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.387+0000 7fd1d6ffd700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd1bc017c60 0x7fd1bc01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.397 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.387+0000 7fd1d6ffd700 1 -- 172.21.15.155:0/3254353192 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7fd1d0030690 con 0x7fd1d8073990 2019-10-13T00:51:44.397 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.387+0000 7fd1dda30700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd1bc017c60 0x7fd1bc01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.397 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.387+0000 7fd1dda30700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd1bc017c60 0x7fd1bc01a0f0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:44.397 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.395+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d808e560 con 0x7fd1d8073990 2019-10-13T00:51:44.398 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.398+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.398 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.398+0000 7fd1d6ffd700 1 -- 172.21.15.155:0/3254353192 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd1d0007920 con 0x7fd1d8073990 2019-10-13T00:51:44.565 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.568+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1d808e560 con 0x7fd1d8073990 2019-10-13T00:51:44.565 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.568+0000 7fd1de231700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.566 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.568+0000 7fd1d6ffd700 1 -- 172.21.15.155:0/3254353192 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10391 (secure 0 0 0) 0x7fd1d0016ec0 con 0x7fd1d8073990 2019-10-13T00:51:44.566 INFO:teuthology.misc.health.smithi155.stdout: 2019-10-13T00:51:44.566 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":14,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:41.807644+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:44.569 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.570+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd1bc017c60 msgr2=0x7fd1bc01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.569 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.570+0000 7fd1dfc94700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd1bc017c60 0x7fd1bc01a0f0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.569 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.570+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 msgr2=0x7fd1d808ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.569 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.570+0000 7fd1dfc94700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7fd1d0010780 tx=0x7fd1d0010b20).stop 2019-10-13T00:51:44.569 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.570+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 shutdown_connections 2019-10-13T00:51:44.569 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.570+0000 7fd1dfc94700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd1bc017c60 0x7fd1bc01a0f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.569 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.570+0000 7fd1dfc94700 1 --2- 172.21.15.155:0/3254353192 >> v2:172.21.15.142:3300/0 conn(0x7fd1d8153ef0 0x7fd1d81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.570 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.571+0000 7fd1dfc94700 1 --2- 172.21.15.155:0/3254353192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1d8073990 0x7fd1d808ee60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.570 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.571+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 shutdown_connections 2019-10-13T00:51:44.570 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.571+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 wait complete. 2019-10-13T00:51:44.570 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:44.571+0000 7fd1dfc94700 1 -- 172.21.15.155:0/3254353192 >> 172.21.15.155:0/3254353192 conn(0x7fd1d806d080 msgr2=0x7fd1d806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:44.577 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-13T00:51:44.577 INFO:tasks.ceph:Creating RBD pool 2019-10-13T00:51:44.577 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:44.577 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd pool create rbd 8 2019-10-13T00:51:44.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.698+0000 7f4ac108c700 1 Processor -- start 2019-10-13T00:51:44.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.698+0000 7f4ac108c700 1 -- start start 2019-10-13T00:51:44.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4ac108c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4abc09e9c0 0x7f4abc14c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:44.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4ac108c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4abc09e1f0 0x7f4abc0a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:44.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4ac108c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc09e630 con 0x7f4abc14be40 2019-10-13T00:51:44.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4ac108c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc09ca60 con 0x7f4abc09e9c0 2019-10-13T00:51:44.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4ac108c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc09c030 con 0x7f4abc09e1f0 2019-10-13T00:51:44.701 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4abad9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4abc09e9c0 0x7f4abc14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.701 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4abb59e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4abc14be40 0x7f4abc09d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48690/0 (socket says 172.21.15.155:48690) 2019-10-13T00:51:44.701 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4abb59e700 1 -- 172.21.15.155:0/2357894982 learned_addr learned my addr 172.21.15.155:0/2357894982 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:44.701 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.699+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4abc09e1f0 0x7f4abc0a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.701 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 -- 172.21.15.155:0/2357894982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4abc09e1f0 msgr2=0x7f4abc0a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:44.701 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4abc09e1f0 0x7f4abc0a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:44.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 -- 172.21.15.155:0/2357894982 >> v1:172.21.15.155:6789/0 conn(0x7f4abc14be40 legacy=0x7f4abc09d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 -- 172.21.15.155:0/2357894982 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc09d0f0 con 0x7f4abc09e9c0 2019-10-13T00:51:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4aba59c700 1 -- 172.21.15.155:0/2357894982 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349496397 0 0) 0x7f4aa40014d0 con 0x7f4abc14be40 2019-10-13T00:51:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2357894982 >> v2:172.21.15.155:3300/0 conn(0x7f4abc09e9c0 0x7f4abc14c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4aac00af40 tx=0x7f4aac00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2357894982 >> v2:172.21.15.155:3300/0 conn(0x7f4abc09e9c0 0x7f4abc14c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4aac00af40 tx=0x7f4aac00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4abc09e9c0 0x7f4abc14c230 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f4aac00af40 tx=0x7f4aac00b750).ready entity=mon.1 client_cookie=e6daea6d41414971 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4abc09e9c0 0x7f4abc14c230 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f4aac00af40 tx=0x7f4aac00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4aba59c700 1 -- 172.21.15.155:0/2357894982 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aac007200 con 0x7f4abc09e9c0 2019-10-13T00:51:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4abc09e9c0 0x7f4abc14c230 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f4aac00af40 tx=0x7f4aac00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4abc09e9c0 0x7f4abc14c230 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f4aac00af40 tx=0x7f4aac00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4aba59c700 1 -- 172.21.15.155:0/2357894982 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac0099a0 con 0x7f4abc09e9c0 2019-10-13T00:51:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4aba59c700 1 -- 172.21.15.155:0/2357894982 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aac00a070 con 0x7f4abc09e9c0 2019-10-13T00:51:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4ac108c700 1 -- 172.21.15.155:0/2357894982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4abc09e9c0 msgr2=0x7f4abc14c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.700+0000 7f4ac108c700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4abc09e9c0 0x7f4abc14c230 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f4aac00af40 tx=0x7f4aac00b750).stop 2019-10-13T00:51:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.701+0000 7f4ac108c700 1 -- 172.21.15.155:0/2357894982 shutdown_connections 2019-10-13T00:51:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.701+0000 7f4ac108c700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4abc09e9c0 0x7f4abc14c230 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.701+0000 7f4ac108c700 1 --2- 172.21.15.155:0/2357894982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4abc09e1f0 0x7f4abc0a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.701+0000 7f4ac108c700 1 -- 172.21.15.155:0/2357894982 shutdown_connections 2019-10-13T00:51:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.701+0000 7f4ac108c700 1 -- 172.21.15.155:0/2357894982 wait complete. 2019-10-13T00:51:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.701+0000 7f4ac108c700 1 -- 172.21.15.155:0/2357894982 >> 172.21.15.155:0/2357894982 conn(0x7f4abc059b50 msgr2=0x7f4abc061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:44.729 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.703+0000 7f4ac108c700 1 Processor -- start 2019-10-13T00:51:44.729 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.703+0000 7f4ac108c700 1 -- start start 2019-10-13T00:51:44.729 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ac108c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4abc08aa40 0x7f4abc081880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.729 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ac108c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc14b5f0 con 0x7f4abc152e90 2019-10-13T00:51:44.729 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ac108c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc07ebd0 con 0x7f4abc082850 2019-10-13T00:51:44.729 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ac108c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc0808f0 con 0x7f4abc08aa40 2019-10-13T00:51:44.729 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4abad9d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4abc082850 0x7f4abc081dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48698/0 (socket says 172.21.15.155:48698) 2019-10-13T00:51:44.730 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4abad9d700 1 -- 172.21.15.155:0/2715500662 learned_addr learned my addr 172.21.15.155:0/2715500662 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:44.730 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4abb59e700 1 --2- 172.21.15.155:0/2715500662 >> v2:172.21.15.142:3300/0 conn(0x7f4abc08aa40 0x7f4abc081880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.730 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639141964 0 0) 0x7f4aac00ac10 con 0x7f4abc082850 2019-10-13T00:51:44.730 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94002e70 con 0x7f4abc082850 2019-10-13T00:51:44.730 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4086992400 0 0) 0x7f4aac00ac10 con 0x7f4abc082850 2019-10-13T00:51:44.730 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a940032d0 con 0x7f4abc082850 2019-10-13T00:51:44.730 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f4aac009570 con 0x7f4abc082850 2019-10-13T00:51:44.731 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 >> v2:172.21.15.142:3300/0 conn(0x7f4abc08aa40 msgr2=0x7f4abc081880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.731 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 --2- 172.21.15.155:0/2715500662 >> v2:172.21.15.142:3300/0 conn(0x7f4abc08aa40 0x7f4abc081880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.731 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 >> v1:172.21.15.155:6789/0 conn(0x7f4abc082850 legacy=0x7f4abc081dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.731 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 >> v1:172.21.15.142:6789/0 conn(0x7f4abc152e90 legacy=0x7f4abc155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.731 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4a94003320 0x7f4a94003750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.731 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a94003da0 0x7f4a94004220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.731 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.732 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400bc70 con 0x7f4a94003da0 2019-10-13T00:51:44.732 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400bea0 con 0x7f4a94004a20 2019-10-13T00:51:44.732 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400c0d0 con 0x7f4a94003320 2019-10-13T00:51:44.732 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.704+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.732 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4abb59e700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4a94003320 0x7f4a94003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.732 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4abad9d700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a94003da0 0x7f4a94004220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.733 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 -- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4a94003320 msgr2=0x7f4a94003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.733 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4a94003320 0x7f4a94003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.733 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 -- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a94003da0 msgr2=0x7f4a94004220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:44.733 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a94003da0 0x7f4a94004220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:44.733 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9400c380 con 0x7f4a94004a20 2019-10-13T00:51:44.733 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc080f80 con 0x7f4a94004a20 2019-10-13T00:51:44.733 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.734 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc07ebd0 con 0x7f4a94004a20 2019-10-13T00:51:44.734 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 -- 172.21.15.155:0/2715500662 reap_dead start 2019-10-13T00:51:44.734 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.734 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:44.734 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.734 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab4000a70 con 0x7f4a94004a20 2019-10-13T00:51:44.734 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.735 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.735 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab4001270 con 0x7f4a94004a20 2019-10-13T00:51:44.735 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab4001940 con 0x7f4a94004a20 2019-10-13T00:51:44.735 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.705+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.735 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.706+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4ab4000a70 con 0x7f4a94004a20 2019-10-13T00:51:44.735 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.706+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.735 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.706+0000 7f4ab3fff700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4a9401a970 0x7f4a9401ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:44.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.706+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f4ab402fe70 con 0x7f4a94004a20 2019-10-13T00:51:44.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.706+0000 7f4abb59e700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4a9401a970 0x7f4a9401ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:44.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.706+0000 7f4abb59e700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4a9401a970 0x7f4a9401ce00 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:44.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.714+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a98002e50 con 0x7f4a94004a20 2019-10-13T00:51:44.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.717+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:44.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.718+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f4ab4000e70 con 0x7f4a94004a20 2019-10-13T00:51:44.879 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:44.882+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f4a98001d10 con 0x7f4a94004a20 2019-10-13T00:51:45.821 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.824+0000 7f4ac088b700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:45.822 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.824+0000 7f4ab3fff700 1 -- 172.21.15.155:0/2715500662 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f4ab4000a70 con 0x7f4a94004a20 2019-10-13T00:51:45.844 INFO:teuthology.orchestra.run.smithi155.stderr:pool 'rbd' created 2019-10-13T00:51:45.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4a9401a970 msgr2=0x7f4a9401ce00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:45.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4a9401a970 0x7f4a9401ce00 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:45.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 msgr2=0x7f4a94004ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:45.847 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f4ab4010380 tx=0x7f4ab4010720).stop 2019-10-13T00:51:45.848 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 shutdown_connections 2019-10-13T00:51:45.848 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4a9401a970 0x7f4a9401ce00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:45.848 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 --2- 172.21.15.155:0/2715500662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a94004a20 0x7f4a94004ee0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:45.848 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 shutdown_connections 2019-10-13T00:51:45.848 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 wait complete. 2019-10-13T00:51:45.848 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.850+0000 7f4ac108c700 1 -- 172.21.15.155:0/2715500662 >> 172.21.15.155:0/2715500662 conn(0x7f4abc07bef0 msgr2=0x7f4abc11a160 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:45.857 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:45.857 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-10-13T00:51:45.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.988+0000 7f3da894a700 1 Processor -- start 2019-10-13T00:51:45.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.988+0000 7f3da894a700 1 -- start start 2019-10-13T00:51:45.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da894a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3da4098870 0x7f3da4098c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:45.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da894a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3da409d380 0x7f3da40991a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:45.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da894a700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da40651f0 con 0x7f3da409c3b0 2019-10-13T00:51:45.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da894a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da409b420 con 0x7f3da4098870 2019-10-13T00:51:45.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da894a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da409a9f0 con 0x7f3da409d380 2019-10-13T00:51:45.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3da409d380 0x7f3da40991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:45.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da359e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3da409c3b0 0x7f3da409c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48766/0 (socket says 172.21.15.155:48766) 2019-10-13T00:51:45.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da359e700 1 -- 172.21.15.155:0/1477334869 learned_addr learned my addr 172.21.15.155:0/1477334869 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:45.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da2d9d700 1 --2- 172.21.15.155:0/1477334869 >> v2:172.21.15.142:3300/0 conn(0x7f3da4098870 0x7f3da4098c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:45.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da1d9b700 1 -- 172.21.15.155:0/1477334869 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183211789 0 0) 0x7f3d9c001150 con 0x7f3da409c3b0 2019-10-13T00:51:45.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da1d9b700 1 -- 172.21.15.155:0/1477334869 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d84002e70 con 0x7f3da409c3b0 2019-10-13T00:51:45.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da259c700 1 -- 172.21.15.155:0/1477334869 >> v2:172.21.15.142:3300/0 conn(0x7f3da4098870 msgr2=0x7f3da4098c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:45.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da259c700 1 --2- 172.21.15.155:0/1477334869 >> v2:172.21.15.142:3300/0 conn(0x7f3da4098870 0x7f3da4098c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:45.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da259c700 1 -- 172.21.15.155:0/1477334869 >> v1:172.21.15.155:6789/0 conn(0x7f3da409c3b0 legacy=0x7f3da409c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:45.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da259c700 1 -- 172.21.15.155:0/1477334869 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da409d050 con 0x7f3da409d380 2019-10-13T00:51:45.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da259c700 1 --2- 172.21.15.155:0/1477334869 >> v2:172.21.15.155:3300/0 conn(0x7f3da409d380 0x7f3da40991a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d8c00b2c0 tx=0x7f3d8c00bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:45.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.989+0000 7f3da259c700 1 --2- 172.21.15.155:0/1477334869 >> v2:172.21.15.155:3300/0 conn(0x7f3da409d380 0x7f3da40991a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d8c00b2c0 tx=0x7f3d8c00bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:45.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da259c700 1 --2- 172.21.15.155:0/1477334869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da409d380 0x7f3da40991a0 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f3d8c00b2c0 tx=0x7f3d8c00bad0).ready entity=mon.1 client_cookie=b528f181516ddaf3 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:45.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da259c700 1 --2- 172.21.15.155:0/1477334869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da409d380 0x7f3da40991a0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f3d8c00b2c0 tx=0x7f3d8c00bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:45.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da1d9b700 1 -- 172.21.15.155:0/1477334869 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d8c000ce0 con 0x7f3da409d380 2019-10-13T00:51:45.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da259c700 1 --2- 172.21.15.155:0/1477334869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da409d380 0x7f3da40991a0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f3d8c00b2c0 tx=0x7f3d8c00bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:45.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da259c700 1 --2- 172.21.15.155:0/1477334869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da409d380 0x7f3da40991a0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f3d8c00b2c0 tx=0x7f3d8c00bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:45.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da1d9b700 1 -- 172.21.15.155:0/1477334869 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c009bd0 con 0x7f3da409d380 2019-10-13T00:51:45.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da1d9b700 1 -- 172.21.15.155:0/1477334869 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d8c00a2a0 con 0x7f3da409d380 2019-10-13T00:51:45.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 -- 172.21.15.155:0/1477334869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da409d380 msgr2=0x7f3da40991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:45.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 --2- 172.21.15.155:0/1477334869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da409d380 0x7f3da40991a0 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f3d8c00b2c0 tx=0x7f3d8c00bad0).stop 2019-10-13T00:51:45.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 -- 172.21.15.155:0/1477334869 shutdown_connections 2019-10-13T00:51:45.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 --2- 172.21.15.155:0/1477334869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da409d380 0x7f3da40991a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:45.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 --2- 172.21.15.155:0/1477334869 >> v2:172.21.15.142:3300/0 conn(0x7f3da4098870 0x7f3da4098c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:45.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 -- 172.21.15.155:0/1477334869 shutdown_connections 2019-10-13T00:51:45.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 -- 172.21.15.155:0/1477334869 wait complete. 2019-10-13T00:51:45.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.990+0000 7f3da894a700 1 -- 172.21.15.155:0/1477334869 >> 172.21.15.155:0/1477334869 conn(0x7f3da40a38d0 msgr2=0x7f3da4061dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:45.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da894a700 1 Processor -- start 2019-10-13T00:51:45.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da894a700 1 -- start start 2019-10-13T00:51:45.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da894a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3da414c8e0 0x7f3da414ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:45.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da894a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3da41381e0 0x7f3da41565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:45.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da894a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da412a150 con 0x7f3da4138810 2019-10-13T00:51:45.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da894a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da412b3e0 con 0x7f3da41381e0 2019-10-13T00:51:45.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da894a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da412b6b0 con 0x7f3da414c8e0 2019-10-13T00:51:45.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.993+0000 7f3da359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3da41381e0 0x7f3da41565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:45.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3da41381e0 0x7f3da41565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58698/0 (socket says 172.21.15.155:58698) 2019-10-13T00:51:45.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 -- 172.21.15.155:0/1488625982 learned_addr learned my addr 172.21.15.155:0/1488625982 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:45.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da2d9d700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3da414c8e0 0x7f3da414ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:45.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 -- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3da414c8e0 msgr2=0x7f3da414ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:45.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3da414c8e0 0x7f3da414ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:45.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 -- 172.21.15.155:0/1488625982 >> v1:172.21.15.142:6789/0 conn(0x7f3da4138810 legacy=0x7f3da41379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:45.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 -- 172.21.15.155:0/1488625982 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da412c4a0 con 0x7f3da41381e0 2019-10-13T00:51:45.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da2d9d700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3da414c8e0 0x7f3da414ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:51:46.000 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> v2:172.21.15.155:3300/0 conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.000 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> v2:172.21.15.155:3300/0 conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.000 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:46.000 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.000 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.000 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.000 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3d9b7fe700 1 -- 172.21.15.155:0/1488625982 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d9c000ef0 con 0x7f3da41381e0 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3d9b7fe700 1 -- 172.21.15.155:0/1488625982 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c004230 con 0x7f3da41381e0 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3d9b7fe700 1 -- 172.21.15.155:0/1488625982 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d9c004940 con 0x7f3da41381e0 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da412c4a0 con 0x7f3da41381e0 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da412c4a0 con 0x7f3da41381e0 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.994+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.995+0000 7f3d9b7fe700 1 -- 172.21.15.155:0/1488625982 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f3d9c004420 con 0x7f3da41381e0 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.995+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.001 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.995+0000 7f3d9b7fe700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3d7c017c60 0x7f3d7c01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:46.002 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.995+0000 7f3d9b7fe700 1 -- 172.21.15.155:0/1488625982 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4842+0+0 (secure 0 0 0) 0x7f3d9c030300 con 0x7f3da41381e0 2019-10-13T00:51:46.002 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.995+0000 7f3da2d9d700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3d7c017c60 0x7f3d7c01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.002 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:45.995+0000 7f3da2d9d700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3d7c017c60 0x7f3d7c01a0f0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:46.002 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.003+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da412c4a0 con 0x7f3da41381e0 2019-10-13T00:51:46.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.007+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.007+0000 7f3d9b7fe700 1 -- 172.21.15.155:0/1488625982 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f3d9c000940 con 0x7f3da41381e0 2019-10-13T00:51:46.165 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.168+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3da412c4a0 con 0x7f3da41381e0 2019-10-13T00:51:46.825 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.827+0000 7f3da359e700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.825 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.827+0000 7f3d9b7fe700 1 -- 172.21.15.155:0/1488625982 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3d9c004350 con 0x7f3da41381e0 2019-10-13T00:51:46.830 INFO:teuthology.orchestra.run.smithi155.stderr:enabled application 'rbd' on pool 'rbd' 2019-10-13T00:51:46.833 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3d7c017c60 msgr2=0x7f3d7c01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.834 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3d7c017c60 0x7f3d7c01a0f0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.834 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 msgr2=0x7f3da41565d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.834 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7f3d9c010420 tx=0x7f3d9c0107c0).stop 2019-10-13T00:51:46.834 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 shutdown_connections 2019-10-13T00:51:46.834 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da41381e0 0x7f3da41565d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.835 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3d7c017c60 0x7f3d7c01a0f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.835 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 --2- 172.21.15.155:0/1488625982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3da414c8e0 0x7f3da414ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.835 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 shutdown_connections 2019-10-13T00:51:46.835 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 wait complete. 2019-10-13T00:51:46.835 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.836+0000 7f3da894a700 1 -- 172.21.15.155:0/1488625982 >> 172.21.15.155:0/1488625982 conn(0x7f3da4077890 msgr2=0x7f3da4089d70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:46.843 INFO:tasks.ceph:Setting up CephFS filesystem... 2019-10-13T00:51:46.843 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:46.843 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:46.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.964+0000 7f25948df700 1 Processor -- start 2019-10-13T00:51:46.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.965+0000 7f25948df700 1 -- start start 2019-10-13T00:51:46.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.965+0000 7f25948df700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2590099b10 0x7f2590099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:46.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.965+0000 7f25948df700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2590064f00 con 0x7f259009d620 2019-10-13T00:51:46.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.965+0000 7f25948df700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900987e0 con 0x7f259009cff0 2019-10-13T00:51:46.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.965+0000 7f25948df700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259009b680 con 0x7f2590099b10 2019-10-13T00:51:46.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258f59e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f259009cff0 0x7f2590152c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48778/0 (socket says 172.21.15.155:48778) 2019-10-13T00:51:46.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258f59e700 1 -- 172.21.15.155:0/3982895341 learned_addr learned my addr 172.21.15.155:0/3982895341 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:46.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258ed9d700 1 --2- 172.21.15.155:0/3982895341 >> v2:172.21.15.142:3300/0 conn(0x7f2590099b10 0x7f2590099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377997327 0 0) 0x7f2588001150 con 0x7f259009cff0 2019-10-13T00:51:46.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2570002e70 con 0x7f259009cff0 2019-10-13T00:51:46.975 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2808663105 0 0) 0x7f2588001130 con 0x7f259009cff0 2019-10-13T00:51:46.975 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 >> v2:172.21.15.142:3300/0 conn(0x7f2590099b10 msgr2=0x7f2590099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:46.975 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 --2- 172.21.15.155:0/3982895341 >> v2:172.21.15.142:3300/0 conn(0x7f2590099b10 0x7f2590099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:46.975 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 >> v1:172.21.15.142:6789/0 conn(0x7f259009d620 legacy=0x7f259009c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.975 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259009bd10 con 0x7f259009cff0 2019-10-13T00:51:46.975 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f2588001660 con 0x7f259009cff0 2019-10-13T00:51:46.976 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 >> v1:172.21.15.155:6789/0 conn(0x7f259009cff0 legacy=0x7f2590152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.976 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2570003b10 0x7f2570005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:46.976 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:46.976 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2570009430 0x7f257000b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:46.976 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257000c0b0 con 0x7f2570009430 2019-10-13T00:51:46.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25700502b0 con 0x7f2570006720 2019-10-13T00:51:46.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25700504e0 con 0x7f2570003b10 2019-10-13T00:51:46.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258ed9d700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2570003b10 0x7f2570005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258f59e700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2570009430 0x7f257000b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.966+0000 7f258e59c700 1 -- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2570003b10 msgr2=0x7f2570005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:46.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2570003b10 0x7f2570005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:46.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 -- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2570009430 msgr2=0x7f257000b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:46.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2570009430 0x7f257000b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:46.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258f59e700 1 -- 172.21.15.155:0/3982895341 reap_dead start 2019-10-13T00:51:46.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 -- 172.21.15.155:0/3982895341 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570050790 con 0x7f2570006720 2019-10-13T00:51:46.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258ed9d700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2570003b10 0x7f2570005f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:51:46.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f258000af50 tx=0x7f258000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f258000af50 tx=0x7f258000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7f258000af50 tx=0x7f258000b760).ready entity=mon.1 client_cookie=dabaeee6e24cb4a0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:46.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f258000af50 tx=0x7f258000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2580000a70 con 0x7f2570006720 2019-10-13T00:51:46.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f258000af50 tx=0x7f258000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258e59c700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f258000af50 tx=0x7f258000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580009980 con 0x7f2570006720 2019-10-13T00:51:46.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f258dd9b700 1 -- 172.21.15.155:0/3982895341 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f258000a050 con 0x7f2570006720 2019-10-13T00:51:46.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f25948df700 1 -- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 msgr2=0x7f2570008bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.967+0000 7f25948df700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7f258000af50 tx=0x7f258000b760).stop 2019-10-13T00:51:46.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.968+0000 7f25948df700 1 -- 172.21.15.155:0/3982895341 shutdown_connections 2019-10-13T00:51:46.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.968+0000 7f25948df700 1 --2- 172.21.15.155:0/3982895341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2570006720 0x7f2570008bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.968+0000 7f25948df700 1 -- 172.21.15.155:0/3982895341 shutdown_connections 2019-10-13T00:51:46.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.968+0000 7f25948df700 1 -- 172.21.15.155:0/3982895341 wait complete. 2019-10-13T00:51:46.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.968+0000 7f25948df700 1 -- 172.21.15.155:0/3982895341 >> 172.21.15.155:0/3982895341 conn(0x7f2590099550 msgr2=0x7f2590061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:46.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.970+0000 7f25948df700 1 Processor -- start 2019-10-13T00:51:46.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.970+0000 7f25948df700 1 -- start start 2019-10-13T00:51:46.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25948df700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2590089090 0x7f25900924e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:46.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25948df700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259008fc80 con 0x7f259008f4b0 2019-10-13T00:51:46.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25948df700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900896c0 con 0x7f259008e8c0 2019-10-13T00:51:46.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25948df700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2590091550 con 0x7f2590089090 2019-10-13T00:51:46.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f258ed9d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f259008e8c0 0x7f259008ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48790/0 (socket says 172.21.15.155:48790) 2019-10-13T00:51:46.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f258ed9d700 1 -- 172.21.15.155:0/726145555 learned_addr learned my addr 172.21.15.155:0/726145555 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:46.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> v2:172.21.15.142:3300/0 conn(0x7f2590089090 0x7f25900924e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270950518 0 0) 0x7f2578000ce0 con 0x7f259008e8c0 2019-10-13T00:51:46.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2568002e70 con 0x7f259008e8c0 2019-10-13T00:51:46.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3792198089 0 0) 0x7f2578000cc0 con 0x7f259008e8c0 2019-10-13T00:51:46.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25680032d0 con 0x7f259008e8c0 2019-10-13T00:51:46.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f2578001930 con 0x7f259008e8c0 2019-10-13T00:51:46.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 >> v2:172.21.15.142:3300/0 conn(0x7f2590089090 msgr2=0x7f25900924e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 --2- 172.21.15.155:0/726145555 >> v2:172.21.15.142:3300/0 conn(0x7f2590089090 0x7f25900924e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 >> v1:172.21.15.155:6789/0 conn(0x7f259008e8c0 legacy=0x7f259008ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 >> v1:172.21.15.142:6789/0 conn(0x7f259008f4b0 legacy=0x7f2590092a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2568003490 0x7f25680038a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:46.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:46.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2568004bb0 0x7f256800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:46.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25680052e0 con 0x7f2568004bb0 2019-10-13T00:51:46.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256800bf70 con 0x7f2568003ef0 2019-10-13T00:51:46.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256800c130 con 0x7f2568003490 2019-10-13T00:51:46.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> v2:172.21.15.142:3300/0 conn(0x7f2590089090 0x7f25900924e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:46.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f258f59e700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2568004bb0 0x7f256800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.971+0000 7f258ed9d700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2568003490 0x7f25680038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 -- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2568003490 msgr2=0x7f25680038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2568003490 0x7f25680038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 -- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2568004bb0 msgr2=0x7f256800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:46.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2568004bb0 0x7f256800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:46.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258f59e700 1 -- 172.21.15.155:0/726145555 reap_dead start 2019-10-13T00:51:46.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256800c3a0 con 0x7f2568003ef0 2019-10-13T00:51:46.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2590091be0 con 0x7f2568003ef0 2019-10-13T00:51:46.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25900896c0 con 0x7f2568003ef0 2019-10-13T00:51:46.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:46.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2580009790 con 0x7f2568003ef0 2019-10-13T00:51:46.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580000940 con 0x7f2568003ef0 2019-10-13T00:51:46.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2580009980 con 0x7f2568003ef0 2019-10-13T00:51:46.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.972+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.973+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f2580009790 con 0x7f2568003ef0 2019-10-13T00:51:46.989 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.973+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.973+0000 7f25877fe700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f256801aa00 0x7f256801ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:46.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.973+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f258002ff90 con 0x7f2568003ef0 2019-10-13T00:51:46.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.973+0000 7f258ed9d700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f256801aa00 0x7f256801ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:46.990 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.973+0000 7f258ed9d700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f256801aa00 0x7f256801ce30 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:46.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.980+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c002e50 con 0x7f2568003ef0 2019-10-13T00:51:46.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.984+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:46.991 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:46.984+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f2580009580 con 0x7f2568003ef0 2019-10-13T00:51:47.148 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.151+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f256c001d00 con 0x7f2568003ef0 2019-10-13T00:51:47.149 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.152+0000 7f258e59c700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.149 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.152+0000 7f25877fe700 1 -- 172.21.15.155:0/726145555 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11778 (secure 0 0 0) 0x7f2580009790 con 0x7f2568003ef0 2019-10-13T00:51:47.150 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:47.150 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":16,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:46.822205+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:47.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f256801aa00 msgr2=0x7f256801ce30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f256801aa00 0x7f256801ce30 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 msgr2=0x7f25680043b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7f258000b2a0 tx=0x7f25800106f0).stop 2019-10-13T00:51:47.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 shutdown_connections 2019-10-13T00:51:47.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f256801aa00 0x7f256801ce30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 --2- 172.21.15.155:0/726145555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2568003ef0 0x7f25680043b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 shutdown_connections 2019-10-13T00:51:47.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 wait complete. 2019-10-13T00:51:47.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.154+0000 7f25948df700 1 -- 172.21.15.155:0/726145555 >> 172.21.15.155:0/726145555 conn(0x7f259006bfe0 msgr2=0x7f259006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:47.161 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:44.894568+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T00:51:47.162 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:47.162 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T00:51:47.288 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.287+0000 7f52cd6e5700 1 Processor -- start 2019-10-13T00:51:47.288 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.287+0000 7f52cd6e5700 1 -- start start 2019-10-13T00:51:47.288 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52cd6e5700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f52c809e9c0 0x7f52c814c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:47.288 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52cd6e5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c809e650 con 0x7f52c809e1f0 2019-10-13T00:51:47.289 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52cd6e5700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c809ada0 con 0x7f52c814be40 2019-10-13T00:51:47.289 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52cd6e5700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c809ca60 con 0x7f52c809e9c0 2019-10-13T00:51:47.289 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f52c809e9c0 0x7f52c814c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c77fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f52c814be40 0x7f52c809d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48804/0 (socket says 172.21.15.155:48804) 2019-10-13T00:51:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c77fe700 1 -- 172.21.15.155:0/1923803706 learned_addr learned my addr 172.21.15.155:0/1923803706 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 -- 172.21.15.155:0/1923803706 >> v1:172.21.15.155:6789/0 conn(0x7f52c814be40 legacy=0x7f52c809d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 -- 172.21.15.155:0/1923803706 >> v1:172.21.15.142:6789/0 conn(0x7f52c809e1f0 legacy=0x7f52c8152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c67fc700 1 -- 172.21.15.155:0/1923803706 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647598134 0 0) 0x7f52b80014d0 con 0x7f52c814be40 2019-10-13T00:51:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 -- 172.21.15.155:0/1923803706 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c809d0f0 con 0x7f52c809e9c0 2019-10-13T00:51:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/1923803706 >> v2:172.21.15.155:3300/0 conn(0x7f52c809e9c0 0x7f52c814c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f52b000af40 tx=0x7f52b000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/1923803706 >> v2:172.21.15.155:3300/0 conn(0x7f52c809e9c0 0x7f52c814c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f52b000af40 tx=0x7f52b000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/1923803706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c809e9c0 0x7f52c814c230 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f52b000af40 tx=0x7f52b000b750).ready entity=mon.1 client_cookie=e3f4b0891b14de90 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/1923803706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c809e9c0 0x7f52c814c230 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f52b000af40 tx=0x7f52b000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c67fc700 1 -- 172.21.15.155:0/1923803706 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52b0000960 con 0x7f52c809e9c0 2019-10-13T00:51:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/1923803706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c809e9c0 0x7f52c814c230 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f52b000af40 tx=0x7f52b000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/1923803706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c809e9c0 0x7f52c814c230 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f52b000af40 tx=0x7f52b000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c67fc700 1 -- 172.21.15.155:0/1923803706 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0009850 con 0x7f52c809e9c0 2019-10-13T00:51:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52c67fc700 1 -- 172.21.15.155:0/1923803706 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52b0009f20 con 0x7f52c809e9c0 2019-10-13T00:51:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52cd6e5700 1 -- 172.21.15.155:0/1923803706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c809e9c0 msgr2=0x7f52c814c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.288+0000 7f52cd6e5700 1 --2- 172.21.15.155:0/1923803706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c809e9c0 0x7f52c814c230 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f52b000af40 tx=0x7f52b000b750).stop 2019-10-13T00:51:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.289+0000 7f52cd6e5700 1 -- 172.21.15.155:0/1923803706 shutdown_connections 2019-10-13T00:51:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.289+0000 7f52cd6e5700 1 --2- 172.21.15.155:0/1923803706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c809e9c0 0x7f52c814c230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.289+0000 7f52cd6e5700 1 -- 172.21.15.155:0/1923803706 shutdown_connections 2019-10-13T00:51:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.289+0000 7f52cd6e5700 1 -- 172.21.15.155:0/1923803706 wait complete. 2019-10-13T00:51:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.289+0000 7f52cd6e5700 1 -- 172.21.15.155:0/1923803706 >> 172.21.15.155:0/1923803706 conn(0x7f52c8059b50 msgr2=0x7f52c8061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:47.296 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.291+0000 7f52cd6e5700 1 Processor -- start 2019-10-13T00:51:47.296 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52cd6e5700 1 -- start start 2019-10-13T00:51:47.296 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52cd6e5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52c8075f60 0x7f52c814b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:47.296 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52cd6e5700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f52c807a760 0x7f52c8079630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:47.296 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52cd6e5700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c80769e0 con 0x7f52c8075370 2019-10-13T00:51:47.296 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52cd6e5700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c80786a0 con 0x7f52c807a760 2019-10-13T00:51:47.296 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52cd6e5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8077c70 con 0x7f52c8075f60 2019-10-13T00:51:47.297 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52c77fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f52c8075370 0x7f52c8075760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48810/0 (socket says 172.21.15.155:48810) 2019-10-13T00:51:47.297 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52c77fe700 1 -- 172.21.15.155:0/3341647241 learned_addr learned my addr 172.21.15.155:0/3341647241 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:47.297 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> v2:172.21.15.155:3300/0 conn(0x7f52c807a760 0x7f52c8079630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.297 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52c8075f60 0x7f52c814b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.297 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.292+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747285899 0 0) 0x7f52b80014d0 con 0x7f52c8075370 2019-10-13T00:51:47.297 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ac002e70 con 0x7f52c8075370 2019-10-13T00:51:47.297 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 -- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52c8075f60 msgr2=0x7f52c814b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.298 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52c8075f60 0x7f52c814b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.298 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 -- 172.21.15.155:0/3341647241 >> v1:172.21.15.155:6789/0 conn(0x7f52c8075370 legacy=0x7f52c8075760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.298 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 -- 172.21.15.155:0/3341647241 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c8078d30 con 0x7f52c807a760 2019-10-13T00:51:47.298 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c6ffd700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52c8075f60 0x7f52c814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:51:47.298 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> v2:172.21.15.155:3300/0 conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.298 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> v2:172.21.15.155:3300/0 conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.298 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c0000960 con 0x7f52c807a760 2019-10-13T00:51:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c000edd0 con 0x7f52c807a760 2019-10-13T00:51:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c000f4e0 con 0x7f52c807a760 2019-10-13T00:51:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c8078d30 con 0x7f52c807a760 2019-10-13T00:51:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c8078d30 con 0x7f52c807a760 2019-10-13T00:51:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.293+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.294+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f52c0000940 con 0x7f52c807a760 2019-10-13T00:51:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.294+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.294+0000 7f52bffff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f52ac017ca0 0x7f52ac01a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.294+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f52c002fff0 con 0x7f52c807a760 2019-10-13T00:51:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.294+0000 7f52c77fe700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f52ac017ca0 0x7f52ac01a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.294+0000 7f52c77fe700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f52ac017ca0 0x7f52ac01a110 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.302+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c8078d30 con 0x7f52c807a760 2019-10-13T00:51:47.303 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.306+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.304 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.306+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f52c000e8a0 con 0x7f52c807a760 2019-10-13T00:51:47.461 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.464+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f52c8078d30 con 0x7f52c807a760 2019-10-13T00:51:47.462 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.465+0000 7f52c7fff700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.462 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.465+0000 7f52bffff700 1 -- 172.21.15.155:0/3341647241 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f52c0000940 con 0x7f52c807a760 2019-10-13T00:51:47.463 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:51:47.465 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f52ac017ca0 msgr2=0x7f52ac01a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.465 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f52ac017ca0 0x7f52ac01a110 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.465 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 msgr2=0x7f52c8079630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.465 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f52c00103a0 tx=0x7f52c0010740).stop 2019-10-13T00:51:47.465 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 shutdown_connections 2019-10-13T00:51:47.466 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52c807a760 0x7f52c8079630 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.466 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f52ac017ca0 0x7f52ac01a110 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.466 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 --2- 172.21.15.155:0/3341647241 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52c8075f60 0x7f52c814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.466 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 shutdown_connections 2019-10-13T00:51:47.466 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 wait complete. 2019-10-13T00:51:47.466 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.467+0000 7f52cd6e5700 1 -- 172.21.15.155:0/3341647241 >> 172.21.15.155:0/3341647241 conn(0x7f52c806f990 msgr2=0x7f52c80880e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:47.474 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:47.474 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2019-10-13T00:51:47.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.595+0000 7f20a9ddb700 1 Processor -- start 2019-10-13T00:51:47.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.595+0000 7f20a9ddb700 1 -- start start 2019-10-13T00:51:47.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a9ddb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a409a700 0x7f20a409c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:47.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a9ddb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20a409d620 0x7f20a409cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:47.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a9ddb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a4064f00 con 0x7f20a4099b10 2019-10-13T00:51:47.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a9ddb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a409b680 con 0x7f20a409d620 2019-10-13T00:51:47.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a9ddb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a409ac50 con 0x7f20a409a700 2019-10-13T00:51:47.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a8b79700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20a409d620 0x7f20a409cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a8b79700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20a409d620 0x7f20a409cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58744/0 (socket says 172.21.15.155:58744) 2019-10-13T00:51:47.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f20a8b79700 1 -- 172.21.15.155:0/3420249423 learned_addr learned my addr 172.21.15.155:0/3420249423 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:47.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.596+0000 7f209bfff700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a409a700 0x7f20a409c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 -- 172.21.15.155:0/3420249423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a409a700 msgr2=0x7f20a409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:47.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a409a700 0x7f20a409c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:47.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 -- 172.21.15.155:0/3420249423 >> v1:172.21.15.142:6789/0 conn(0x7f20a4099b10 legacy=0x7f20a4099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 -- 172.21.15.155:0/3420249423 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a409bd10 con 0x7f20a409d620 2019-10-13T00:51:47.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 --2- 172.21.15.155:0/3420249423 >> v2:172.21.15.155:3300/0 conn(0x7f20a409d620 0x7f20a409cd30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f209c00faa0 tx=0x7f209c00fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 --2- 172.21.15.155:0/3420249423 >> v2:172.21.15.155:3300/0 conn(0x7f20a409d620 0x7f20a409cd30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f209c00faa0 tx=0x7f209c00fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20a409d620 0x7f20a409cd30 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f209c00faa0 tx=0x7f209c00fe40).ready entity=mon.1 client_cookie=b97de6fd8b9b493c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20a409d620 0x7f20a409cd30 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f209c00faa0 tx=0x7f209c00fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20a409d620 0x7f20a409cd30 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f209c00faa0 tx=0x7f209c00fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a37fe700 1 -- 172.21.15.155:0/3420249423 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f209c001460 con 0x7f20a409d620 2019-10-13T00:51:47.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a8b79700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20a409d620 0x7f20a409cd30 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f209c00faa0 tx=0x7f209c00fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a37fe700 1 -- 172.21.15.155:0/3420249423 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c000df0 con 0x7f20a409d620 2019-10-13T00:51:47.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a37fe700 1 -- 172.21.15.155:0/3420249423 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f209c001a40 con 0x7f20a409d620 2019-10-13T00:51:47.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a9ddb700 1 -- 172.21.15.155:0/3420249423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20a409d620 msgr2=0x7f20a409cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.597+0000 7f20a9ddb700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20a409d620 0x7f20a409cd30 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f209c00faa0 tx=0x7f209c00fe40).stop 2019-10-13T00:51:47.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.598+0000 7f20a9ddb700 1 -- 172.21.15.155:0/3420249423 shutdown_connections 2019-10-13T00:51:47.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.598+0000 7f20a9ddb700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20a409d620 0x7f20a409cd30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.601 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.598+0000 7f20a9ddb700 1 --2- 172.21.15.155:0/3420249423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a409a700 0x7f20a409c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:47.601 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.598+0000 7f20a9ddb700 1 -- 172.21.15.155:0/3420249423 shutdown_connections 2019-10-13T00:51:47.601 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.598+0000 7f20a9ddb700 1 -- 172.21.15.155:0/3420249423 wait complete. 2019-10-13T00:51:47.601 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.598+0000 7f20a9ddb700 1 -- 172.21.15.155:0/3420249423 >> 172.21.15.155:0/3420249423 conn(0x7f20a4099550 msgr2=0x7f20a4061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:47.604 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.600+0000 7f20a9ddb700 1 Processor -- start 2019-10-13T00:51:47.604 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.600+0000 7f20a9ddb700 1 -- start start 2019-10-13T00:51:47.605 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.600+0000 7f20a9ddb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:47.605 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.600+0000 7f20a9ddb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40912f0 con 0x7f20a4091690 2019-10-13T00:51:47.605 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.600+0000 7f20a9ddb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a408d4b0 con 0x7f20a4073990 2019-10-13T00:51:47.605 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.600+0000 7f20a9ddb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a408f730 con 0x7f20a4090ec0 2019-10-13T00:51:47.605 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a3fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f20a4073990 0x7f20a40906c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48824/0 (socket says 172.21.15.155:48824) 2019-10-13T00:51:47.605 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a3fff700 1 -- 172.21.15.155:0/615772294 learned_addr learned my addr 172.21.15.155:0/615772294 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:47.606 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.606 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755439204 0 0) 0x7f2094001110 con 0x7f20a4073990 2019-10-13T00:51:47.606 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2088002e70 con 0x7f20a4073990 2019-10-13T00:51:47.606 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336716317 0 0) 0x7f208c009570 con 0x7f20a4091690 2019-10-13T00:51:47.606 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20880030a0 con 0x7f20a4091690 2019-10-13T00:51:47.606 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 -- 172.21.15.155:0/615772294 >> v1:172.21.15.155:6789/0 conn(0x7f20a4073990 legacy=0x7f20a40906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.606 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3727024778 0 0) 0x7f20940012c0 con 0x7f20a4073990 2019-10-13T00:51:47.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 -- 172.21.15.155:0/615772294 >> v1:172.21.15.142:6789/0 conn(0x7f20a4091690 legacy=0x7f20a4152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 -- 172.21.15.155:0/615772294 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a408fdc0 con 0x7f20a4090ec0 2019-10-13T00:51:47.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f209c008180 con 0x7f20a4090ec0 2019-10-13T00:51:47.608 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.608 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c008180 con 0x7f20a4090ec0 2019-10-13T00:51:47.608 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.608 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a408fdc0 con 0x7f20a4090ec0 2019-10-13T00:51:47.608 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f209c001610 con 0x7f20a4090ec0 2019-10-13T00:51:47.609 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.601+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a408fdc0 con 0x7f20a4090ec0 2019-10-13T00:51:47.609 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.602+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.609 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.602+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f209c001230 con 0x7f20a4090ec0 2019-10-13T00:51:47.609 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.602+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.609 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.602+0000 7f20a17fa700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2088017c80 0x7f208801a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:47.609 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.602+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f209c030370 con 0x7f20a4090ec0 2019-10-13T00:51:47.610 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.602+0000 7f20a3fff700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2088017c80 0x7f208801a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.611 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.603+0000 7f20a3fff700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2088017c80 0x7f208801a110 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.612 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.610+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a408fdc0 con 0x7f20a4090ec0 2019-10-13T00:51:47.612 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.614+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.612 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.614+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f209c0079a0 con 0x7f20a4090ec0 2019-10-13T00:51:47.781 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.783+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f20a408fdc0 con 0x7f20a4090ec0 2019-10-13T00:51:47.781 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.784+0000 7f20a8b79700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.781 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.784+0000 7f20a17fa700 1 -- 172.21.15.155:0/615772294 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7f209c008260 con 0x7f20a4090ec0 2019-10-13T00:51:47.782 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:47.782 INFO:teuthology.orchestra.run.smithi155.stdout:[ 2019-10-13T00:51:47.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2019-10-13T00:51:47.782 INFO:teuthology.orchestra.run.smithi155.stdout: "poolnum": 1, 2019-10-13T00:51:47.782 INFO:teuthology.orchestra.run.smithi155.stdout: "poolname": "rbd" 2019-10-13T00:51:47.782 INFO:teuthology.orchestra.run.smithi155.stdout: } 2019-10-13T00:51:47.782 INFO:teuthology.orchestra.run.smithi155.stdout:] 2019-10-13T00:51:47.784 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2088017c80 msgr2=0x7f208801a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.784 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2088017c80 0x7f208801a110 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.785 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 msgr2=0x7f20a4156700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.785 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f209c0106d0 tx=0x7f209c010a70).stop 2019-10-13T00:51:47.785 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 shutdown_connections 2019-10-13T00:51:47.785 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2088017c80 0x7f208801a110 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.785 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 --2- 172.21.15.155:0/615772294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20a4090ec0 0x7f20a4156700 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.785 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 shutdown_connections 2019-10-13T00:51:47.785 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 wait complete. 2019-10-13T00:51:47.786 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.786+0000 7f20a9ddb700 1 -- 172.21.15.155:0/615772294 >> 172.21.15.155:0/615772294 conn(0x7f20a406d080 msgr2=0x7f20a406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:47.793 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2019-10-13T00:51:47.793 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:47.793 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2019-10-13T00:51:47.919 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.918+0000 7f5fce9f6700 1 Processor -- start 2019-10-13T00:51:47.919 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.918+0000 7f5fce9f6700 1 -- start start 2019-10-13T00:51:47.919 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.918+0000 7f5fce9f6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5fc814be40 0x7f5fc814c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:47.919 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.918+0000 7f5fce9f6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5fc809e1f0 0x7f5fc80a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:47.919 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.918+0000 7f5fce9f6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc809e650 con 0x7f5fc809e9c0 2019-10-13T00:51:47.920 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.918+0000 7f5fce9f6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc809ca60 con 0x7f5fc809e1f0 2019-10-13T00:51:47.920 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.918+0000 7f5fce9f6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc809c030 con 0x7f5fc814be40 2019-10-13T00:51:47.920 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5fc814be40 0x7f5fc814c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.920 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5fc814be40 0x7f5fc814c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58756/0 (socket says 172.21.15.155:58756) 2019-10-13T00:51:47.920 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 -- 172.21.15.155:0/2734822582 learned_addr learned my addr 172.21.15.155:0/2734822582 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:47.920 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fcd794700 1 --2- 172.21.15.155:0/2734822582 >> v2:172.21.15.142:3300/0 conn(0x7f5fc809e1f0 0x7f5fc80a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.920 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 -- 172.21.15.155:0/2734822582 >> v2:172.21.15.142:3300/0 conn(0x7f5fc809e1f0 msgr2=0x7f5fc80a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:47.921 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- 172.21.15.155:0/2734822582 >> v2:172.21.15.142:3300/0 conn(0x7f5fc809e1f0 0x7f5fc80a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:47.921 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 -- 172.21.15.155:0/2734822582 >> v1:172.21.15.142:6789/0 conn(0x7f5fc809e9c0 legacy=0x7f5fc809d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:47.921 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 -- 172.21.15.155:0/2734822582 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc809d0f0 con 0x7f5fc814be40 2019-10-13T00:51:47.921 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- 172.21.15.155:0/2734822582 >> v2:172.21.15.155:3300/0 conn(0x7f5fc814be40 0x7f5fc814c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5fb800fab0 tx=0x7f5fb800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.921 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- 172.21.15.155:0/2734822582 >> v2:172.21.15.155:3300/0 conn(0x7f5fc814be40 0x7f5fc814c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5fb800fab0 tx=0x7f5fb800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.921 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- 172.21.15.155:0/2734822582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc814be40 0x7f5fc814c230 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f5fb800fab0 tx=0x7f5fb800fe50).ready entity=mon.1 client_cookie=fd1c4683a245c3ec server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.921 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- 172.21.15.155:0/2734822582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc814be40 0x7f5fc814c230 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f5fb800fab0 tx=0x7f5fb800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.922 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- 172.21.15.155:0/2734822582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc814be40 0x7f5fc814c230 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f5fb800fab0 tx=0x7f5fb800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.922 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fc77fe700 1 -- 172.21.15.155:0/2734822582 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fb8000ce0 con 0x7f5fc814be40 2019-10-13T00:51:47.922 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fccf93700 1 --2- 172.21.15.155:0/2734822582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc814be40 0x7f5fc814c230 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f5fb800fab0 tx=0x7f5fb800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.922 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fc77fe700 1 -- 172.21.15.155:0/2734822582 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb8001010 con 0x7f5fc814be40 2019-10-13T00:51:47.922 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fc77fe700 1 -- 172.21.15.155:0/2734822582 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fb8001760 con 0x7f5fc814be40 2019-10-13T00:51:47.922 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fce9f6700 1 -- 172.21.15.155:0/2734822582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc814be40 msgr2=0x7f5fc814c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.919+0000 7f5fce9f6700 1 --2- 172.21.15.155:0/2734822582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc814be40 0x7f5fc814c230 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f5fb800fab0 tx=0x7f5fb800fe50).stop 2019-10-13T00:51:47.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.920+0000 7f5fce9f6700 1 -- 172.21.15.155:0/2734822582 shutdown_connections 2019-10-13T00:51:47.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.920+0000 7f5fce9f6700 1 --2- 172.21.15.155:0/2734822582 >> v2:172.21.15.142:3300/0 conn(0x7f5fc809e1f0 0x7f5fc80a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:47.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.920+0000 7f5fce9f6700 1 --2- 172.21.15.155:0/2734822582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc814be40 0x7f5fc814c230 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:47.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.920+0000 7f5fce9f6700 1 -- 172.21.15.155:0/2734822582 shutdown_connections 2019-10-13T00:51:47.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.920+0000 7f5fce9f6700 1 -- 172.21.15.155:0/2734822582 wait complete. 2019-10-13T00:51:47.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.920+0000 7f5fce9f6700 1 -- 172.21.15.155:0/2734822582 >> 172.21.15.155:0/2734822582 conn(0x7f5fc8059b50 msgr2=0x7f5fc8061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:47.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.922+0000 7f5fce9f6700 1 Processor -- start 2019-10-13T00:51:47.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.922+0000 7f5fce9f6700 1 -- start start 2019-10-13T00:51:47.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fce9f6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5fc81458a0 0x7f5fc8085c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:47.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fce9f6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc8072500 con 0x7f5fc8086400 2019-10-13T00:51:47.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fce9f6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc807b750 con 0x7f5fc81454b0 2019-10-13T00:51:47.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fce9f6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc80844a0 con 0x7f5fc81458a0 2019-10-13T00:51:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fccf93700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5fc81454b0 0x7f5fc8085430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48836/0 (socket says 172.21.15.155:48836) 2019-10-13T00:51:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fccf93700 1 -- 172.21.15.155:0/926314450 learned_addr learned my addr 172.21.15.155:0/926314450 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> v2:172.21.15.155:3300/0 conn(0x7f5fc81458a0 0x7f5fc8085c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289995331 0 0) 0x7f5fb8007e60 con 0x7f5fc81454b0 2019-10-13T00:51:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac002e70 con 0x7f5fc81454b0 2019-10-13T00:51:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704344166 0 0) 0x7f5fb0001040 con 0x7f5fc8086400 2019-10-13T00:51:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.923+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac0030a0 con 0x7f5fc8086400 2019-10-13T00:51:47.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 -- 172.21.15.155:0/926314450 >> v1:172.21.15.155:6789/0 conn(0x7f5fc81454b0 legacy=0x7f5fc8085430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 -- 172.21.15.155:0/926314450 >> v1:172.21.15.142:6789/0 conn(0x7f5fc8086400 legacy=0x7f5fc8198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:47.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 -- 172.21.15.155:0/926314450 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc8084b30 con 0x7f5fc81458a0 2019-10-13T00:51:47.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> v2:172.21.15.155:3300/0 conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> v2:172.21.15.155:3300/0 conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc00058a0 con 0x7f5fc81458a0 2019-10-13T00:51:47.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc00054c0 con 0x7f5fc81458a0 2019-10-13T00:51:47.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc00056b0 con 0x7f5fc81458a0 2019-10-13T00:51:47.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc8084b30 con 0x7f5fc81458a0 2019-10-13T00:51:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc8084b30 con 0x7f5fc81458a0 2019-10-13T00:51:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.924+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.925+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5fc00056b0 con 0x7f5fc81458a0 2019-10-13T00:51:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.925+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.925+0000 7f5fc57fa700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5fac017c80 0x7f5fac01a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.925+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5fc002ff80 con 0x7f5fc81458a0 2019-10-13T00:51:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.925+0000 7f5fccf93700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5fac017c80 0x7f5fac01a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:47.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.925+0000 7f5fccf93700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5fac017c80 0x7f5fac01a110 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:47.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.933+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc8084b30 con 0x7f5fc81458a0 2019-10-13T00:51:47.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.936+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:47.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:47.936+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f5fc0027f40 con 0x7f5fc81458a0 2019-10-13T00:51:48.095 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.098+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5fc8084b30 con 0x7f5fc81458a0 2019-10-13T00:51:48.843 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.846+0000 7f5fcd794700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:48.844 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.846+0000 7f5fc57fa700 1 -- 172.21.15.155:0/926314450 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5fc0027f40 con 0x7f5fc81458a0 2019-10-13T00:51:48.859 INFO:teuthology.orchestra.run.smithi155.stderr:pool 'cephfs_metadata' created 2019-10-13T00:51:48.862 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.864+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5fac017c80 msgr2=0x7f5fac01a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:48.862 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.864+0000 7f5fce9f6700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5fac017c80 0x7f5fac01a110 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:48.862 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.864+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 msgr2=0x7f5fc8085c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:48.862 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.864+0000 7f5fce9f6700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f5fc00103b0 tx=0x7f5fc0010750).stop 2019-10-13T00:51:48.863 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.864+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 shutdown_connections 2019-10-13T00:51:48.863 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.865+0000 7f5fce9f6700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fc81458a0 0x7f5fc8085c30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:48.863 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.865+0000 7f5fce9f6700 1 --2- 172.21.15.155:0/926314450 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5fac017c80 0x7f5fac01a110 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:48.863 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.865+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 shutdown_connections 2019-10-13T00:51:48.863 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.865+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 wait complete. 2019-10-13T00:51:48.863 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:48.865+0000 7f5fce9f6700 1 -- 172.21.15.155:0/926314450 >> 172.21.15.155:0/926314450 conn(0x7f5fc8071130 msgr2=0x7f5fc8074ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:48.871 INFO:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2019-10-13T00:51:48.872 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:48.872 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2019-10-13T00:51:49.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.007+0000 7f442fc71700 1 Processor -- start 2019-10-13T00:51:49.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.007+0000 7f442fc71700 1 -- start start 2019-10-13T00:51:49.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442fc71700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4428098870 0x7f4428098c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:49.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442fc71700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f442809c3b0 0x7f442809c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:49.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442fc71700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44280651f0 con 0x7f442809d380 2019-10-13T00:51:49.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442fc71700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442809b420 con 0x7f4428098870 2019-10-13T00:51:49.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442fc71700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442809a9f0 con 0x7f442809c3b0 2019-10-13T00:51:49.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4428098870 0x7f4428098c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:49.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4428098870 0x7f4428098c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58768/0 (socket says 172.21.15.155:58768) 2019-10-13T00:51:49.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 -- 172.21.15.155:0/1254653802 learned_addr learned my addr 172.21.15.155:0/1254653802 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:49.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442ea0f700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f442809c3b0 0x7f442809c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:49.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 -- 172.21.15.155:0/1254653802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f442809c3b0 msgr2=0x7f442809c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:49.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f442809c3b0 0x7f442809c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:49.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 -- 172.21.15.155:0/1254653802 >> v1:172.21.15.142:6789/0 conn(0x7f442809d380 legacy=0x7f44280991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 -- 172.21.15.155:0/1254653802 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442809bab0 con 0x7f4428098870 2019-10-13T00:51:49.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442ea0f700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f442809c3b0 0x7f442809c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:51:49.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- 172.21.15.155:0/1254653802 >> v2:172.21.15.155:3300/0 conn(0x7f4428098870 0x7f4428098c60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f441800f730 tx=0x7f441800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- 172.21.15.155:0/1254653802 >> v2:172.21.15.155:3300/0 conn(0x7f4428098870 0x7f4428098c60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f441800f730 tx=0x7f441800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4428098870 0x7f4428098c60 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7f441800f730 tx=0x7f441800fad0).ready entity=mon.1 client_cookie=1a96ee3aeb896afb server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:49.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4428098870 0x7f4428098c60 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f441800f730 tx=0x7f441800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4428098870 0x7f4428098c60 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f441800f730 tx=0x7f441800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442d20c700 1 -- 172.21.15.155:0/1254653802 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4418000960 con 0x7f4428098870 2019-10-13T00:51:49.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442e20e700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4428098870 0x7f4428098c60 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f441800f730 tx=0x7f441800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442d20c700 1 -- 172.21.15.155:0/1254653802 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418000c90 con 0x7f4428098870 2019-10-13T00:51:49.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442d20c700 1 -- 172.21.15.155:0/1254653802 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4418001360 con 0x7f4428098870 2019-10-13T00:51:49.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442fc71700 1 -- 172.21.15.155:0/1254653802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4428098870 msgr2=0x7f4428098c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.008+0000 7f442fc71700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4428098870 0x7f4428098c60 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7f441800f730 tx=0x7f441800fad0).stop 2019-10-13T00:51:49.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.009+0000 7f442fc71700 1 -- 172.21.15.155:0/1254653802 shutdown_connections 2019-10-13T00:51:49.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.009+0000 7f442fc71700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f442809c3b0 0x7f442809c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:49.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.009+0000 7f442fc71700 1 --2- 172.21.15.155:0/1254653802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4428098870 0x7f4428098c60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:49.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.009+0000 7f442fc71700 1 -- 172.21.15.155:0/1254653802 shutdown_connections 2019-10-13T00:51:49.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.009+0000 7f442fc71700 1 -- 172.21.15.155:0/1254653802 wait complete. 2019-10-13T00:51:49.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.009+0000 7f442fc71700 1 -- 172.21.15.155:0/1254653802 >> 172.21.15.155:0/1254653802 conn(0x7f44280a38d0 msgr2=0x7f4428061dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:49.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.011+0000 7f442fc71700 1 Processor -- start 2019-10-13T00:51:49.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.011+0000 7f442fc71700 1 -- start start 2019-10-13T00:51:49.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442fc71700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f442814c910 0x7f4428152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:49.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442fc71700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44280807c0 0x7f442814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:49.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442fc71700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442807dd60 con 0x7f4428080dc0 2019-10-13T00:51:49.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442fc71700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442812b7f0 con 0x7f442814c910 2019-10-13T00:51:49.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442fc71700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442812adc0 con 0x7f44280807c0 2019-10-13T00:51:49.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442e20e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f442814c910 0x7f4428152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:49.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442ea0f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44280807c0 0x7f442814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:49.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442e20e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f442814c910 0x7f4428152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.155:58510/0 (socket says 172.21.15.155:58510) 2019-10-13T00:51:49.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.012+0000 7f442e20e700 1 -- 172.21.15.155:0/3752470423 learned_addr learned my addr 172.21.15.155:0/3752470423 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:49.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 -- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44280807c0 msgr2=0x7f442814cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44280807c0 0x7f442814cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:49.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 -- 172.21.15.155:0/3752470423 >> v1:172.21.15.142:6789/0 conn(0x7f4428080dc0 legacy=0x7f442812c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 -- 172.21.15.155:0/3752470423 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442812be80 con 0x7f442814c910 2019-10-13T00:51:49.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442ea0f700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44280807c0 0x7f442814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:49.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> v2:172.21.15.142:3300/0 conn(0x7f442814c910 0x7f4428152660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> v2:172.21.15.142:3300/0 conn(0x7f442814c910 0x7f4428152660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:49.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f4426ffd700 1 -- 172.21.15.155:0/3752470423 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4418008150 con 0x7f442814c910 2019-10-13T00:51:49.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f4426ffd700 1 -- 172.21.15.155:0/3752470423 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418005360 con 0x7f442814c910 2019-10-13T00:51:49.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f4426ffd700 1 -- 172.21.15.155:0/3752470423 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4418005a30 con 0x7f442814c910 2019-10-13T00:51:49.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442812be80 con 0x7f442814c910 2019-10-13T00:51:49.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442812be80 con 0x7f442814c910 2019-10-13T00:51:49.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.013+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.014+0000 7f4426ffd700 1 -- 172.21.15.155:0/3752470423 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4418005550 con 0x7f442814c910 2019-10-13T00:51:49.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.014+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.014+0000 7f4426ffd700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4408017c60 0x7f440801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:49.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.014+0000 7f4426ffd700 1 -- 172.21.15.155:0/3752470423 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5207+0+0 (secure 0 0 0) 0x7f44180300b0 con 0x7f442814c910 2019-10-13T00:51:49.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.014+0000 7f442da0d700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4408017c60 0x7f440801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:49.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.014+0000 7f442da0d700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4408017c60 0x7f440801a0f0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:49.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.022+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442812be80 con 0x7f442814c910 2019-10-13T00:51:49.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.026+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.026+0000 7f4426ffd700 1 -- 172.21.15.155:0/3752470423 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f44180072b0 con 0x7f442814c910 2019-10-13T00:51:49.182 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.185+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"} v 0) v1 -- 0x7f442812be80 con 0x7f442814c910 2019-10-13T00:51:49.845 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.848+0000 7f442e20e700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.845 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.848+0000 7f4426ffd700 1 -- 172.21.15.155:0/3752470423 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"}]=0 v19) v1 ==== 154+0+0 (secure 0 0 0) 0x7f4418005660 con 0x7f442814c910 2019-10-13T00:51:49.850 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.852+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4408017c60 msgr2=0x7f440801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.850 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.852+0000 7f442fc71700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4408017c60 0x7f440801a0f0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:49.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.852+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 msgr2=0x7f4428152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.852+0000 7f442fc71700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f4418010450 tx=0x7f44180107f0).stop 2019-10-13T00:51:49.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.853+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 shutdown_connections 2019-10-13T00:51:49.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.853+0000 7f442fc71700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4408017c60 0x7f440801a0f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:49.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.853+0000 7f442fc71700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44280807c0 0x7f442814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:49.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.853+0000 7f442fc71700 1 --2- 172.21.15.155:0/3752470423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f442814c910 0x7f4428152660 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:49.852 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.853+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 shutdown_connections 2019-10-13T00:51:49.852 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.853+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 wait complete. 2019-10-13T00:51:49.852 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.853+0000 7f442fc71700 1 -- 172.21.15.155:0/3752470423 >> 172.21.15.155:0/3752470423 conn(0x7f44280963f0 msgr2=0x7f4428071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:49.860 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:49.860 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2019-10-13T00:51:49.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.993+0000 7f2deb9cb700 1 Processor -- start 2019-10-13T00:51:49.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.993+0000 7f2deb9cb700 1 -- start start 2019-10-13T00:51:49.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2deb9cb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2de4098870 0x7f2de4098c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:49.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2deb9cb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de40651f0 con 0x7f2de409c3b0 2019-10-13T00:51:49.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2deb9cb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de409a190 con 0x7f2de409d380 2019-10-13T00:51:49.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2deb9cb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de409b420 con 0x7f2de4098870 2019-10-13T00:51:49.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9f68700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2de4098870 0x7f2de4098c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:49.996 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9767700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2de409d380 0x7f2de40991a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48862/0 (socket says 172.21.15.155:48862) 2019-10-13T00:51:49.996 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9f68700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2de4098870 0x7f2de4098c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58782/0 (socket says 172.21.15.155:58782) 2019-10-13T00:51:49.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9767700 1 -- 172.21.15.155:0/4092505829 learned_addr learned my addr 172.21.15.155:0/4092505829 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:49.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9f68700 1 -- 172.21.15.155:0/4092505829 >> v1:172.21.15.155:6789/0 conn(0x7f2de409d380 legacy=0x7f2de40991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9f68700 1 -- 172.21.15.155:0/4092505829 >> v1:172.21.15.142:6789/0 conn(0x7f2de409c3b0 legacy=0x7f2de409c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de8f66700 1 -- 172.21.15.155:0/4092505829 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935540044 0 0) 0x7f2dd8001150 con 0x7f2de409d380 2019-10-13T00:51:49.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9f68700 1 -- 172.21.15.155:0/4092505829 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de409d050 con 0x7f2de4098870 2019-10-13T00:51:49.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.994+0000 7f2de9f68700 1 --2- 172.21.15.155:0/4092505829 >> v2:172.21.15.155:3300/0 conn(0x7f2de4098870 0x7f2de4098c60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2dd400b2c0 tx=0x7f2dd400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de9f68700 1 --2- 172.21.15.155:0/4092505829 >> v2:172.21.15.155:3300/0 conn(0x7f2de4098870 0x7f2de4098c60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2dd400b2c0 tx=0x7f2dd400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de9f68700 1 --2- 172.21.15.155:0/4092505829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4098870 0x7f2de4098c60 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f2dd400b2c0 tx=0x7f2dd400bad0).ready entity=mon.1 client_cookie=61a79630fc4e2994 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:49.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de9f68700 1 --2- 172.21.15.155:0/4092505829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4098870 0x7f2de4098c60 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f2dd400b2c0 tx=0x7f2dd400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de8f66700 1 -- 172.21.15.155:0/4092505829 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dd4007580 con 0x7f2de4098870 2019-10-13T00:51:49.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de9f68700 1 --2- 172.21.15.155:0/4092505829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4098870 0x7f2de4098c60 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f2dd400b2c0 tx=0x7f2dd400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de9f68700 1 --2- 172.21.15.155:0/4092505829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4098870 0x7f2de4098c60 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f2dd400b2c0 tx=0x7f2dd400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:49.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de8f66700 1 -- 172.21.15.155:0/4092505829 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd4009d20 con 0x7f2de4098870 2019-10-13T00:51:49.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2de8f66700 1 -- 172.21.15.155:0/4092505829 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dd400a3f0 con 0x7f2de4098870 2019-10-13T00:51:49.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2deb9cb700 1 -- 172.21.15.155:0/4092505829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4098870 msgr2=0x7f2de4098c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:49.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2deb9cb700 1 --2- 172.21.15.155:0/4092505829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4098870 0x7f2de4098c60 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f2dd400b2c0 tx=0x7f2dd400bad0).stop 2019-10-13T00:51:49.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2deb9cb700 1 -- 172.21.15.155:0/4092505829 shutdown_connections 2019-10-13T00:51:49.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.995+0000 7f2deb9cb700 1 --2- 172.21.15.155:0/4092505829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4098870 0x7f2de4098c60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:49.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.996+0000 7f2deb9cb700 1 -- 172.21.15.155:0/4092505829 shutdown_connections 2019-10-13T00:51:49.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.996+0000 7f2deb9cb700 1 -- 172.21.15.155:0/4092505829 wait complete. 2019-10-13T00:51:49.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.996+0000 7f2deb9cb700 1 -- 172.21.15.155:0/4092505829 >> 172.21.15.155:0/4092505829 conn(0x7f2de40a38d0 msgr2=0x7f2de4061dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:50.002 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.998+0000 7f2deb9cb700 1 Processor -- start 2019-10-13T00:51:50.002 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.998+0000 7f2deb9cb700 1 -- start start 2019-10-13T00:51:50.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2deb9cb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2de4127c10 0x7f2de41567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:50.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2deb9cb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de4134200 con 0x7f2de407d880 2019-10-13T00:51:50.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2deb9cb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de407dee0 con 0x7f2de4128310 2019-10-13T00:51:50.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2deb9cb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de4135ec0 con 0x7f2de4127c10 2019-10-13T00:51:50.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2de4127c10 0x7f2de41567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:50.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2de9767700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2de4128310 0x7f2de4152f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48864/0 (socket says 172.21.15.155:48864) 2019-10-13T00:51:50.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2de9767700 1 -- 172.21.15.155:0/466775556 learned_addr learned my addr 172.21.15.155:0/466775556 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:50.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 -- 172.21.15.155:0/466775556 >> v1:172.21.15.155:6789/0 conn(0x7f2de4128310 legacy=0x7f2de4152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:50.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 -- 172.21.15.155:0/466775556 >> v1:172.21.15.142:6789/0 conn(0x7f2de407d880 legacy=0x7f2de4136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:50.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 -- 172.21.15.155:0/466775556 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de4136550 con 0x7f2de4127c10 2019-10-13T00:51:50.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497463343 0 0) 0x7f2dd8001150 con 0x7f2de4128310 2019-10-13T00:51:50.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> v2:172.21.15.155:3300/0 conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> v2:172.21.15.155:3300/0 conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=READY pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:50.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ddc00f120 con 0x7f2de4127c10 2019-10-13T00:51:50.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc00ed40 con 0x7f2de4127c10 2019-10-13T00:51:50.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ddc00f310 con 0x7f2de4127c10 2019-10-13T00:51:50.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de4136550 con 0x7f2de4127c10 2019-10-13T00:51:50.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:49.999+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de4136550 con 0x7f2de4127c10 2019-10-13T00:51:50.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.000+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.000+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f2ddc00ef30 con 0x7f2de4127c10 2019-10-13T00:51:50.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.000+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.000+0000 7f2de27fc700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2dc4017c60 0x7f2dc401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:50.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.000+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5463+0+0 (secure 0 0 0) 0x7f2ddc02ffc0 con 0x7f2de4127c10 2019-10-13T00:51:50.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.000+0000 7f2de9f68700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2dc4017c60 0x7f2dc401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:50.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.001+0000 7f2de9f68700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2dc4017c60 0x7f2dc401a0f0 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:50.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.008+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de4136550 con 0x7f2de4127c10 2019-10-13T00:51:50.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.012+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:50.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.012+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f2ddc00eb30 con 0x7f2de4127c10 2019-10-13T00:51:50.168 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:50.171+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"} v 0) v1 -- 0x7f2de4136550 con 0x7f2de4127c10 2019-10-13T00:51:50.562 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:50.562 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:51:50.567 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:50.567 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:51:51.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.853+0000 7f2dea769700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:51.852 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.853+0000 7f2de27fc700 1 -- 172.21.15.155:0/466775556 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v21) v1 ==== 188+0+0 (secure 0 0 0) 0x7f2ddc00f060 con 0x7f2de4127c10 2019-10-13T00:51:51.885 INFO:teuthology.orchestra.run.smithi155.stderr:pool 'cephfs_data' created 2019-10-13T00:51:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2dc4017c60 msgr2=0x7f2dc401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2dc4017c60 0x7f2dc401a0f0 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 msgr2=0x7f2de41567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 secure :-1 s=READY pgs=61 cs=0 l=1 rx=0x7f2ddc0103b0 tx=0x7f2ddc010750).stop 2019-10-13T00:51:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 shutdown_connections 2019-10-13T00:51:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2de4127c10 0x7f2de41567b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 --2- 172.21.15.155:0/466775556 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2dc4017c60 0x7f2dc401a0f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:51.890 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 shutdown_connections 2019-10-13T00:51:51.890 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 wait complete. 2019-10-13T00:51:51.890 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:51.890+0000 7f2deb9cb700 1 -- 172.21.15.155:0/466775556 >> 172.21.15.155:0/466775556 conn(0x7f2de4077890 msgr2=0x7f2de4089d70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:51.898 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:51.898 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2019-10-13T00:51:52.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.022+0000 7fb48d300700 1 Processor -- start 2019-10-13T00:51:52.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.022+0000 7fb48d300700 1 -- start start 2019-10-13T00:51:52.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb48d300700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48809d380 0x7fb488099460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:52.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb48d300700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb48809c3b0 0x7fb48809c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:52.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb48d300700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4880651f0 con 0x7fb488098870 2019-10-13T00:51:52.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb48d300700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48809b420 con 0x7fb48809c3b0 2019-10-13T00:51:52.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb48d300700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48809a9f0 con 0x7fb48809d380 2019-10-13T00:51:52.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb487fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb48809c3b0 0x7fb48809c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:52.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb4877fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb488098870 0x7fb488098c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48872/0 (socket says 172.21.15.155:48872) 2019-10-13T00:51:52.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb4877fe700 1 -- 172.21.15.155:0/1843709554 learned_addr learned my addr 172.21.15.155:0/1843709554 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:52.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb487fff700 1 -- 172.21.15.155:0/1843709554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48809d380 msgr2=0x7fb488099460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb486ffd700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48809d380 0x7fb488099460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb4867fc700 1 -- 172.21.15.155:0/1843709554 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306939094 0 0) 0x7fb478001150 con 0x7fb488098870 2019-10-13T00:51:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb487fff700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48809d380 0x7fb488099460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb487fff700 1 -- 172.21.15.155:0/1843709554 >> v1:172.21.15.155:6789/0 conn(0x7fb488098870 legacy=0x7fb488098c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb487fff700 1 -- 172.21.15.155:0/1843709554 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48809bab0 con 0x7fb48809c3b0 2019-10-13T00:51:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.023+0000 7fb486ffd700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48809d380 0x7fb488099460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T00:51:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb487fff700 1 --2- 172.21.15.155:0/1843709554 >> v2:172.21.15.155:3300/0 conn(0x7fb48809c3b0 0x7fb48809c870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb48000b2c0 tx=0x7fb48000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb487fff700 1 --2- 172.21.15.155:0/1843709554 >> v2:172.21.15.155:3300/0 conn(0x7fb48809c3b0 0x7fb48809c870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb48000b2c0 tx=0x7fb48000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb487fff700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48809c3b0 0x7fb48809c870 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7fb48000b2c0 tx=0x7fb48000bad0).ready entity=mon.1 client_cookie=342879dfca11938d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb487fff700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48809c3b0 0x7fb48809c870 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7fb48000b2c0 tx=0x7fb48000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb4867fc700 1 -- 172.21.15.155:0/1843709554 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb480009f10 con 0x7fb48809c3b0 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb487fff700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48809c3b0 0x7fb48809c870 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7fb48000b2c0 tx=0x7fb48000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb487fff700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48809c3b0 0x7fb48809c870 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7fb48000b2c0 tx=0x7fb48000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb4867fc700 1 -- 172.21.15.155:0/1843709554 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480000df0 con 0x7fb48809c3b0 2019-10-13T00:51:52.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb4867fc700 1 -- 172.21.15.155:0/1843709554 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb48000a4f0 con 0x7fb48809c3b0 2019-10-13T00:51:52.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 -- 172.21.15.155:0/1843709554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48809c3b0 msgr2=0x7fb48809c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:52.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48809c3b0 0x7fb48809c870 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7fb48000b2c0 tx=0x7fb48000bad0).stop 2019-10-13T00:51:52.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 -- 172.21.15.155:0/1843709554 shutdown_connections 2019-10-13T00:51:52.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48809c3b0 0x7fb48809c870 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:52.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 --2- 172.21.15.155:0/1843709554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48809d380 0x7fb488099460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:52.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 -- 172.21.15.155:0/1843709554 shutdown_connections 2019-10-13T00:51:52.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 -- 172.21.15.155:0/1843709554 wait complete. 2019-10-13T00:51:52.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.024+0000 7fb48d300700 1 -- 172.21.15.155:0/1843709554 >> 172.21.15.155:0/1843709554 conn(0x7fb4880a38d0 msgr2=0x7fb488061dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:52.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.027+0000 7fb48d300700 1 Processor -- start 2019-10-13T00:51:52.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.027+0000 7fb48d300700 1 -- start start 2019-10-13T00:51:52.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb48d300700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb48814b7a0 0x7fb48814bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:52.035 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb48d300700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48807beb0 0x7fb488151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:52.035 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb48d300700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48807c2e0 con 0x7fb48807c680 2019-10-13T00:51:52.035 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb48d300700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48808a850 con 0x7fb48814b7a0 2019-10-13T00:51:52.035 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb48d300700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48808a2b0 con 0x7fb48807beb0 2019-10-13T00:51:52.036 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb48814b7a0 0x7fb48814bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:52.036 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb48814b7a0 0x7fb48814bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58802/0 (socket says 172.21.15.155:58802) 2019-10-13T00:51:52.036 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 -- 172.21.15.155:0/3615275981 learned_addr learned my addr 172.21.15.155:0/3615275981 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:52.036 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb487fff700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48807beb0 0x7fb488151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:52.036 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 -- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48807beb0 msgr2=0x7fb488151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:52.036 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48807beb0 0x7fb488151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:52.036 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 -- 172.21.15.155:0/3615275981 >> v1:172.21.15.142:6789/0 conn(0x7fb48807c680 legacy=0x7fb48807b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:52.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 -- 172.21.15.155:0/3615275981 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48807adb0 con 0x7fb48814b7a0 2019-10-13T00:51:52.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb487fff700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48807beb0 0x7fb488151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:51:52.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> v2:172.21.15.155:3300/0 conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> v2:172.21.15.155:3300/0 conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:52.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb47ffff700 1 -- 172.21.15.155:0/3615275981 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb478000ed0 con 0x7fb48814b7a0 2019-10-13T00:51:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb47ffff700 1 -- 172.21.15.155:0/3615275981 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb478004420 con 0x7fb48814b7a0 2019-10-13T00:51:52.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb47ffff700 1 -- 172.21.15.155:0/3615275981 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb478004a70 con 0x7fb48814b7a0 2019-10-13T00:51:52.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48807adb0 con 0x7fb48814b7a0 2019-10-13T00:51:52.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.028+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48807adb0 con 0x7fb48814b7a0 2019-10-13T00:51:52.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.029+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.029+0000 7fb47ffff700 1 -- 172.21.15.155:0/3615275981 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb478004610 con 0x7fb48814b7a0 2019-10-13T00:51:52.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.029+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.029+0000 7fb47ffff700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb460017c60 0x7fb46001a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:52.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.029+0000 7fb47ffff700 1 -- 172.21.15.155:0/3615275981 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5915+0+0 (secure 0 0 0) 0x7fb478030140 con 0x7fb48814b7a0 2019-10-13T00:51:52.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.029+0000 7fb486ffd700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb460017c60 0x7fb46001a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:52.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.029+0000 7fb486ffd700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb460017c60 0x7fb46001a0f0 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:52.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.037+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48807adb0 con 0x7fb48814b7a0 2019-10-13T00:51:52.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.041+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.041+0000 7fb47ffff700 1 -- 172.21.15.155:0/3615275981 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb47802bf90 con 0x7fb48814b7a0 2019-10-13T00:51:52.199 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.201+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"} v 0) v1 -- 0x7fb48807adb0 con 0x7fb48814b7a0 2019-10-13T00:51:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.853+0000 7fb4877fe700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.853+0000 7fb47ffff700 1 -- 172.21.15.155:0/3615275981 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"}]=0 set pool 3 allow_ec_overwrites to true v22) v1 ==== 166+0+0 (secure 0 0 0) 0x7fb47802bf90 con 0x7fb48814b7a0 2019-10-13T00:51:52.864 INFO:teuthology.orchestra.run.smithi155.stderr:set pool 3 allow_ec_overwrites to true 2019-10-13T00:51:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb460017c60 msgr2=0x7fb46001a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb460017c60 0x7fb46001a0f0 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 msgr2=0x7fb48814bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fb478010420 tx=0x7fb4780107c0).stop 2019-10-13T00:51:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 shutdown_connections 2019-10-13T00:51:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb460017c60 0x7fb46001a0f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48807beb0 0x7fb488151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.868+0000 7fb48d300700 1 --2- 172.21.15.155:0/3615275981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb48814b7a0 0x7fb48814bb90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.869+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 shutdown_connections 2019-10-13T00:51:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.869+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 wait complete. 2019-10-13T00:51:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:52.869+0000 7fb48d300700 1 -- 172.21.15.155:0/3615275981 >> 172.21.15.155:0/3615275981 conn(0x7fb48807f100 msgr2=0x7fb488071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:52.877 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:52.877 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2019-10-13T00:51:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.000+0000 7f9e18a1a700 1 Processor -- start 2019-10-13T00:51:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.000+0000 7f9e18a1a700 1 -- start start 2019-10-13T00:51:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.001+0000 7f9e18a1a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9e1414be40 0x7f9e1414c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.001+0000 7f9e18a1a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1409e650 con 0x7f9e1409e9c0 2019-10-13T00:51:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.001+0000 7f9e18a1a700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1409ada0 con 0x7f9e1409e1f0 2019-10-13T00:51:53.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.001+0000 7f9e18a1a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1409ca60 con 0x7f9e1414be40 2019-10-13T00:51:53.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.001+0000 7f9e1359e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f9e1409e1f0 0x7f9e14152b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48890/0 (socket says 172.21.15.155:48890) 2019-10-13T00:51:53.004 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.001+0000 7f9e1359e700 1 -- 172.21.15.155:0/3087049909 learned_addr learned my addr 172.21.15.155:0/3087049909 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:53.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.001+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/3087049909 >> v2:172.21.15.142:3300/0 conn(0x7f9e1414be40 0x7f9e1414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e11d9b700 1 -- 172.21.15.155:0/3087049909 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783495937 0 0) 0x7f9e0c001150 con 0x7f9e1409e1f0 2019-10-13T00:51:53.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e11d9b700 1 -- 172.21.15.155:0/3087049909 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc002e70 con 0x7f9e1409e1f0 2019-10-13T00:51:53.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 -- 172.21.15.155:0/3087049909 >> v1:172.21.15.155:6789/0 conn(0x7f9e1409e1f0 legacy=0x7f9e14152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e11d9b700 1 -- 172.21.15.155:0/3087049909 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1651421605 0 0) 0x7f9e0c001130 con 0x7f9e1409e1f0 2019-10-13T00:51:53.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 -- 172.21.15.155:0/3087049909 >> v1:172.21.15.142:6789/0 conn(0x7f9e1409e9c0 legacy=0x7f9e1409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 -- 172.21.15.155:0/3087049909 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1409d0f0 con 0x7f9e1414be40 2019-10-13T00:51:53.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/3087049909 >> v2:172.21.15.142:3300/0 conn(0x7f9e1414be40 0x7f9e1414c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e0400b2c0 tx=0x7f9e0400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/3087049909 >> v2:172.21.15.142:3300/0 conn(0x7f9e1414be40 0x7f9e1414c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e0400b2c0 tx=0x7f9e0400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/3087049909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e1414be40 0x7f9e1414c230 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f9e0400b2c0 tx=0x7f9e0400bad0).ready entity=mon.0 client_cookie=bd305cc4ef766f32 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/3087049909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e1414be40 0x7f9e1414c230 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f9e0400b2c0 tx=0x7f9e0400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e11d9b700 1 -- 172.21.15.155:0/3087049909 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e04000ce0 con 0x7f9e1414be40 2019-10-13T00:51:53.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/3087049909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e1414be40 0x7f9e1414c230 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f9e0400b2c0 tx=0x7f9e0400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e11d9b700 1 -- 172.21.15.155:0/3087049909 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04007580 con 0x7f9e1414be40 2019-10-13T00:51:53.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/3087049909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e1414be40 0x7f9e1414c230 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f9e0400b2c0 tx=0x7f9e0400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e11d9b700 1 -- 172.21.15.155:0/3087049909 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e04009f10 con 0x7f9e1414be40 2019-10-13T00:51:53.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e18a1a700 1 -- 172.21.15.155:0/3087049909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e1414be40 msgr2=0x7f9e1414c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.002+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/3087049909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e1414be40 0x7f9e1414c230 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f9e0400b2c0 tx=0x7f9e0400bad0).stop 2019-10-13T00:51:53.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.003+0000 7f9e18a1a700 1 -- 172.21.15.155:0/3087049909 shutdown_connections 2019-10-13T00:51:53.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.003+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/3087049909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e1414be40 0x7f9e1414c230 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.003+0000 7f9e18a1a700 1 -- 172.21.15.155:0/3087049909 shutdown_connections 2019-10-13T00:51:53.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.003+0000 7f9e18a1a700 1 -- 172.21.15.155:0/3087049909 wait complete. 2019-10-13T00:51:53.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.003+0000 7f9e18a1a700 1 -- 172.21.15.155:0/3087049909 >> 172.21.15.155:0/3087049909 conn(0x7f9e14059b50 msgr2=0x7f9e14061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:53.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.005+0000 7f9e18a1a700 1 Processor -- start 2019-10-13T00:51:53.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.005+0000 7f9e18a1a700 1 -- start start 2019-10-13T00:51:53.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e18a1a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e1412f930 0x7f9e1408a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e18a1a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9e1411a220 0x7f9e1408a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e18a1a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9e1408acf0 0x7f9e14085ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e18a1a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140892e0 con 0x7f9e1411a220 2019-10-13T00:51:53.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e18a1a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140888b0 con 0x7f9e1408acf0 2019-10-13T00:51:53.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e18a1a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14088b80 con 0x7f9e1412f930 2019-10-13T00:51:53.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9e1408acf0 0x7f9e14085ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9e1408acf0 0x7f9e14085ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58818/0 (socket says 172.21.15.155:58818) 2019-10-13T00:51:53.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1359e700 1 -- 172.21.15.155:0/1740394760 learned_addr learned my addr 172.21.15.155:0/1740394760 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:53.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1259c700 1 --2- 172.21.15.155:0/1740394760 >> v2:172.21.15.142:3300/0 conn(0x7f9e1411a220 0x7f9e1408a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e1412f930 0x7f9e1408a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1359e700 1 -- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e1412f930 msgr2=0x7f9e1408a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e1412f930 0x7f9e1408a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1359e700 1 -- 172.21.15.155:0/1740394760 >> v2:172.21.15.142:3300/0 conn(0x7f9e1411a220 msgr2=0x7f9e1408a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.006+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> v2:172.21.15.142:3300/0 conn(0x7f9e1411a220 0x7f9e1408a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 -- 172.21.15.155:0/1740394760 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14089970 con 0x7f9e1408acf0 2019-10-13T00:51:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e1412f930 0x7f9e1408a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:51:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> v2:172.21.15.155:3300/0 conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> v2:172.21.15.155:3300/0 conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e037fe700 1 -- 172.21.15.155:0/1740394760 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e0c000ef0 con 0x7f9e1408acf0 2019-10-13T00:51:53.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e037fe700 1 -- 172.21.15.155:0/1740394760 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c004230 con 0x7f9e1408acf0 2019-10-13T00:51:53.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e037fe700 1 -- 172.21.15.155:0/1740394760 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e0c004940 con 0x7f9e1408acf0 2019-10-13T00:51:53.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14089970 con 0x7f9e1408acf0 2019-10-13T00:51:53.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e14089970 con 0x7f9e1408acf0 2019-10-13T00:51:53.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e037fe700 1 -- 172.21.15.155:0/1740394760 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f9e0c004420 con 0x7f9e1408acf0 2019-10-13T00:51:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.007+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.008+0000 7f9e037fe700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9df8017c60 0x7f9df801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.008+0000 7f9e037fe700 1 -- 172.21.15.155:0/1740394760 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f9e0c02ff60 con 0x7f9e1408acf0 2019-10-13T00:51:53.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.008+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9df8017c60 0x7f9df801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.008+0000 7f9e12d9d700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9df8017c60 0x7f9df801a0f0 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.016+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e14089970 con 0x7f9e1408acf0 2019-10-13T00:51:53.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.019+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.019+0000 7f9e037fe700 1 -- 172.21.15.155:0/1740394760 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f9e0c0012f0 con 0x7f9e1408acf0 2019-10-13T00:51:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.187+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f9e14086ab0 con 0x7f9e1408acf0 2019-10-13T00:51:53.186 INFO:tasks.ceph.mon.b.smithi142.stderr:2019-10-13T00:51:53.187+0000 7f869ae39700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-10-13T00:51:53.189 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.192+0000 7f9e1359e700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.192+0000 7f9e037fe700 1 -- 172.21.15.155:0/1740394760 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9e0c01e940 con 0x7f9e1408acf0 2019-10-13T00:51:53.192 INFO:teuthology.orchestra.run.smithi155.stderr:new fs with metadata pool 2 and data pool 3 2019-10-13T00:51:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9df8017c60 msgr2=0x7f9df801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9df8017c60 0x7f9df801a0f0 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 msgr2=0x7f9e14085ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f9e0c010420 tx=0x7f9e0c0107c0).stop 2019-10-13T00:51:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 shutdown_connections 2019-10-13T00:51:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1408acf0 0x7f9e14085ca0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/1740394760 >> v2:172.21.15.142:3300/0 conn(0x7f9e1411a220 0x7f9e1408a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9df8017c60 0x7f9df801a0f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 --2- 172.21.15.155:0/1740394760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e1412f930 0x7f9e1408a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.196 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 shutdown_connections 2019-10-13T00:51:53.196 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 wait complete. 2019-10-13T00:51:53.196 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.196+0000 7f9e18a1a700 1 -- 172.21.15.155:0/1740394760 >> 172.21.15.155:0/1740394760 conn(0x7f9e1412b150 msgr2=0x7f9e1407bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:53.204 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:53.204 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:53.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.323+0000 7f620d8c7700 1 Processor -- start 2019-10-13T00:51:53.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.323+0000 7f620d8c7700 1 -- start start 2019-10-13T00:51:53.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f620d8c7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6208099b10 0x7f6208099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:53.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f620d8c7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f620809d620 0x7f620809a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:53.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f620d8c7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6208064f00 con 0x7f620809c7f0 2019-10-13T00:51:53.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f620d8c7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620809b680 con 0x7f620809d620 2019-10-13T00:51:53.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f620d8c7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620809ac50 con 0x7f6208099b10 2019-10-13T00:51:53.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6208099b10 0x7f6208099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6208099b10 0x7f6208099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58824/0 (socket says 172.21.15.155:58824) 2019-10-13T00:51:53.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 -- 172.21.15.155:0/3502418546 learned_addr learned my addr 172.21.15.155:0/3502418546 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:53.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f6207fff700 1 --1- 172.21.15.155:0/3502418546 >> v1:172.21.15.155:6789/0 conn(0x7f620809c7f0 0x7f620809cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48902/0 (socket says 172.21.15.155:48902) 2019-10-13T00:51:53.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f6206ffd700 1 --2- 172.21.15.155:0/3502418546 >> v2:172.21.15.142:3300/0 conn(0x7f620809d620 0x7f620809a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 -- 172.21.15.155:0/3502418546 >> v2:172.21.15.142:3300/0 conn(0x7f620809d620 msgr2=0x7f620809a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:53.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 --2- 172.21.15.155:0/3502418546 >> v2:172.21.15.142:3300/0 conn(0x7f620809d620 0x7f620809a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:53.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 -- 172.21.15.155:0/3502418546 >> v1:172.21.15.155:6789/0 conn(0x7f620809c7f0 legacy=0x7f620809cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 -- 172.21.15.155:0/3502418546 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620809bd10 con 0x7f6208099b10 2019-10-13T00:51:53.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62067fc700 1 -- 172.21.15.155:0/3502418546 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938585966 0 0) 0x7f62000009b0 con 0x7f620809c7f0 2019-10-13T00:51:53.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 --2- 172.21.15.155:0/3502418546 >> v2:172.21.15.155:3300/0 conn(0x7f6208099b10 0x7f6208099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f61f800fab0 tx=0x7f61f800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 --2- 172.21.15.155:0/3502418546 >> v2:172.21.15.155:3300/0 conn(0x7f6208099b10 0x7f6208099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f61f800fab0 tx=0x7f61f800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.324+0000 7f62077fe700 1 --2- 172.21.15.155:0/3502418546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208099b10 0x7f6208099f00 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f61f800fab0 tx=0x7f61f800fe50).ready entity=mon.1 client_cookie=ebd62e8b45be5939 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f62077fe700 1 --2- 172.21.15.155:0/3502418546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208099b10 0x7f6208099f00 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f61f800fab0 tx=0x7f61f800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f62067fc700 1 -- 172.21.15.155:0/3502418546 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f8000ce0 con 0x7f6208099b10 2019-10-13T00:51:53.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f62077fe700 1 --2- 172.21.15.155:0/3502418546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208099b10 0x7f6208099f00 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f61f800fab0 tx=0x7f61f800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f62077fe700 1 --2- 172.21.15.155:0/3502418546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208099b10 0x7f6208099f00 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f61f800fab0 tx=0x7f61f800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f62067fc700 1 -- 172.21.15.155:0/3502418546 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f8001010 con 0x7f6208099b10 2019-10-13T00:51:53.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f62067fc700 1 -- 172.21.15.155:0/3502418546 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f80016e0 con 0x7f6208099b10 2019-10-13T00:51:53.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 -- 172.21.15.155:0/3502418546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208099b10 msgr2=0x7f6208099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 --2- 172.21.15.155:0/3502418546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208099b10 0x7f6208099f00 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f61f800fab0 tx=0x7f61f800fe50).stop 2019-10-13T00:51:53.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 -- 172.21.15.155:0/3502418546 shutdown_connections 2019-10-13T00:51:53.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 --2- 172.21.15.155:0/3502418546 >> v2:172.21.15.142:3300/0 conn(0x7f620809d620 0x7f620809a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:53.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 --2- 172.21.15.155:0/3502418546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208099b10 0x7f6208099f00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 -- 172.21.15.155:0/3502418546 shutdown_connections 2019-10-13T00:51:53.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 -- 172.21.15.155:0/3502418546 wait complete. 2019-10-13T00:51:53.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.325+0000 7f620d8c7700 1 -- 172.21.15.155:0/3502418546 >> 172.21.15.155:0/3502418546 conn(0x7f6208099550 msgr2=0x7f6208061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:53.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f620d8c7700 1 Processor -- start 2019-10-13T00:51:53.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f620d8c7700 1 -- start start 2019-10-13T00:51:53.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f620d8c7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f62080925b0 0x7f620808db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f620d8c7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6208076df0 0x7f6208076450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f620d8c7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6208088fc0 con 0x7f62080929a0 2019-10-13T00:51:53.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f620d8c7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620808cbd0 con 0x7f62080925b0 2019-10-13T00:51:53.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f620d8c7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620808c1a0 con 0x7f6208076df0 2019-10-13T00:51:53.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.328+0000 7f62077fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f62080925b0 0x7f620808db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f62080925b0 0x7f620808db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58830/0 (socket says 172.21.15.155:58830) 2019-10-13T00:51:53.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 -- 172.21.15.155:0/1910763351 learned_addr learned my addr 172.21.15.155:0/1910763351 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:53.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f6206ffd700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6208076df0 0x7f6208076450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 -- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6208076df0 msgr2=0x7f6208076450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6208076df0 0x7f6208076450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 -- 172.21.15.155:0/1910763351 >> v1:172.21.15.155:6789/0 conn(0x7f62080929a0 legacy=0x7f6208092e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 -- 172.21.15.155:0/1910763351 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620808d260 con 0x7f62080925b0 2019-10-13T00:51:53.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> v2:172.21.15.155:3300/0 conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> v2:172.21.15.155:3300/0 conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f61effff700 1 -- 172.21.15.155:0/1910763351 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f80084f0 con 0x7f62080925b0 2019-10-13T00:51:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f61effff700 1 -- 172.21.15.155:0/1910763351 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f80056e0 con 0x7f62080925b0 2019-10-13T00:51:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f61effff700 1 -- 172.21.15.155:0/1910763351 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f8005cf0 con 0x7f62080925b0 2019-10-13T00:51:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620808d260 con 0x7f62080925b0 2019-10-13T00:51:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.329+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620808d260 con 0x7f62080925b0 2019-10-13T00:51:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.330+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.330+0000 7f61effff700 1 -- 172.21.15.155:0/1910763351 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f61f80054f0 con 0x7f62080925b0 2019-10-13T00:51:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.330+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.330+0000 7f61effff700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f61e8017c60 0x7f61e801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.330+0000 7f61effff700 1 -- 172.21.15.155:0/1910763351 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f61f80303e0 con 0x7f62080925b0 2019-10-13T00:51:53.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.330+0000 7f6206ffd700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f61e8017c60 0x7f61e801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.331+0000 7f6206ffd700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f61e8017c60 0x7f61e801a0f0 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.338+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620808d260 con 0x7f62080925b0 2019-10-13T00:51:53.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.342+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.342+0000 7f61effff700 1 -- 172.21.15.155:0/1910763351 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f61f8007630 con 0x7f62080925b0 2019-10-13T00:51:53.505 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.507+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f620808d260 con 0x7f62080925b0 2019-10-13T00:51:53.506 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.508+0000 7f62077fe700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.506 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.508+0000 7f61effff700 1 -- 172.21.15.155:0/1910763351 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f61f80054f0 con 0x7f62080925b0 2019-10-13T00:51:53.507 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:53.507 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:53.508 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f61e8017c60 msgr2=0x7f61e801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.508 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f61e8017c60 0x7f61e801a0f0 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.509 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 msgr2=0x7f620808db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.509 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7f61f80107b0 tx=0x7f61f8010b50).stop 2019-10-13T00:51:53.509 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 shutdown_connections 2019-10-13T00:51:53.509 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f61e8017c60 0x7f61e801a0f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.509 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6208076df0 0x7f6208076450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.509 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 --2- 172.21.15.155:0/1910763351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62080925b0 0x7f620808db60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 shutdown_connections 2019-10-13T00:51:53.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 wait complete. 2019-10-13T00:51:53.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.510+0000 7f620d8c7700 1 -- 172.21.15.155:0/1910763351 >> 172.21.15.155:0/1910763351 conn(0x7f620806d080 msgr2=0x7f620806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:53.519 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:53.519 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:53.640 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.638+0000 7f638ba19700 1 Processor -- start 2019-10-13T00:51:53.640 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.638+0000 7f638ba19700 1 -- start start 2019-10-13T00:51:53.641 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f638ba19700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f638409a700 0x7f638409c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:53.641 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f638ba19700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638409d620 0x7f638409cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:53.641 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f638ba19700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6384064f00 con 0x7f6384099b10 2019-10-13T00:51:53.641 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f638ba19700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638409b680 con 0x7f638409a700 2019-10-13T00:51:53.641 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f638ba19700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638409ac50 con 0x7f638409d620 2019-10-13T00:51:53.641 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f63897b5700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f638409a700 0x7f638409c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.641 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f638a7b7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638409d620 0x7f638409cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f63897b5700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f638409a700 0x7f638409c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58842/0 (socket says 172.21.15.155:58842) 2019-10-13T00:51:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f63897b5700 1 -- 172.21.15.155:0/3479424276 learned_addr learned my addr 172.21.15.155:0/3479424276 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f63897b5700 1 -- 172.21.15.155:0/3479424276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638409d620 msgr2=0x7f638409cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f63897b5700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638409d620 0x7f638409cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f63897b5700 1 -- 172.21.15.155:0/3479424276 >> v1:172.21.15.142:6789/0 conn(0x7f6384099b10 legacy=0x7f6384099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f6388fb4700 1 -- 172.21.15.155:0/3479424276 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935441330 0 0) 0x7f6380001080 con 0x7f6384099b10 2019-10-13T00:51:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.639+0000 7f63897b5700 1 -- 172.21.15.155:0/3479424276 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638409bd10 con 0x7f638409a700 2019-10-13T00:51:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f63897b5700 1 --2- 172.21.15.155:0/3479424276 >> v2:172.21.15.155:3300/0 conn(0x7f638409a700 0x7f638409c7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f637800fab0 tx=0x7f637800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f63897b5700 1 --2- 172.21.15.155:0/3479424276 >> v2:172.21.15.155:3300/0 conn(0x7f638409a700 0x7f638409c7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f637800fab0 tx=0x7f637800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f63897b5700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f638409a700 0x7f638409c7f0 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7f637800fab0 tx=0x7f637800fe50).ready entity=mon.1 client_cookie=a29682c2046dc60b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f63897b5700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f638409a700 0x7f638409c7f0 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7f637800fab0 tx=0x7f637800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f6388fb4700 1 -- 172.21.15.155:0/3479424276 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6378000ce0 con 0x7f638409a700 2019-10-13T00:51:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f63897b5700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f638409a700 0x7f638409c7f0 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7f637800fab0 tx=0x7f637800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f63897b5700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f638409a700 0x7f638409c7f0 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7f637800fab0 tx=0x7f637800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.644 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f6388fb4700 1 -- 172.21.15.155:0/3479424276 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6378001010 con 0x7f638409a700 2019-10-13T00:51:53.644 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f6388fb4700 1 -- 172.21.15.155:0/3479424276 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63780016e0 con 0x7f638409a700 2019-10-13T00:51:53.644 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f638ba19700 1 -- 172.21.15.155:0/3479424276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f638409a700 msgr2=0x7f638409c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.644 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f638ba19700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f638409a700 0x7f638409c7f0 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7f637800fab0 tx=0x7f637800fe50).stop 2019-10-13T00:51:53.644 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f638ba19700 1 -- 172.21.15.155:0/3479424276 shutdown_connections 2019-10-13T00:51:53.644 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f638ba19700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f638409d620 0x7f638409cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:53.644 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.640+0000 7f638ba19700 1 --2- 172.21.15.155:0/3479424276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f638409a700 0x7f638409c7f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.645 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.641+0000 7f638ba19700 1 -- 172.21.15.155:0/3479424276 shutdown_connections 2019-10-13T00:51:53.645 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.641+0000 7f638ba19700 1 -- 172.21.15.155:0/3479424276 wait complete. 2019-10-13T00:51:53.645 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.641+0000 7f638ba19700 1 -- 172.21.15.155:0/3479424276 >> 172.21.15.155:0/3479424276 conn(0x7f6384099550 msgr2=0x7f6384061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:53.649 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.643+0000 7f638ba19700 1 Processor -- start 2019-10-13T00:51:53.649 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.643+0000 7f638ba19700 1 -- start start 2019-10-13T00:51:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f638ba19700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6384073990 0x7f63840906c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f638ba19700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6384091690 0x7f6384090c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f638ba19700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638408d4b0 con 0x7f6384091140 2019-10-13T00:51:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f638ba19700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638408f730 con 0x7f6384073990 2019-10-13T00:51:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f638ba19700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638408ed00 con 0x7f6384091690 2019-10-13T00:51:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f638a7b7700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6384091140 0x7f638408bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48924/0 (socket says 172.21.15.155:48924) 2019-10-13T00:51:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f638a7b7700 1 -- 172.21.15.155:0/161829305 learned_addr learned my addr 172.21.15.155:0/161829305 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:53.651 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> v2:172.21.15.142:3300/0 conn(0x7f6384073990 0x7f63840906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.651 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f63897b5700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6384091690 0x7f6384090c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.651 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976493723 0 0) 0x7f637c001150 con 0x7f6384091140 2019-10-13T00:51:53.652 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636c002e70 con 0x7f6384091140 2019-10-13T00:51:53.652 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 -- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6384091690 msgr2=0x7f6384090c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.652 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6384091690 0x7f6384090c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.652 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 -- 172.21.15.155:0/161829305 >> v1:172.21.15.155:6789/0 conn(0x7f6384091140 legacy=0x7f638408bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.652 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 -- 172.21.15.155:0/161829305 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638408fdc0 con 0x7f6384073990 2019-10-13T00:51:53.652 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f63897b5700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6384091690 0x7f6384090c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:53.652 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> v2:172.21.15.142:3300/0 conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.653 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> v2:172.21.15.142:3300/0 conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.653 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.653 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.653 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6380000a70 con 0x7f6384073990 2019-10-13T00:51:53.653 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.653 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6380009be0 con 0x7f6384073990 2019-10-13T00:51:53.653 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.644+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6380000a70 con 0x7f6384073990 2019-10-13T00:51:53.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.645+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638408fdc0 con 0x7f6384073990 2019-10-13T00:51:53.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.645+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638408fdc0 con 0x7f6384073990 2019-10-13T00:51:53.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.645+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.645+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f63800096b0 con 0x7f6384073990 2019-10-13T00:51:53.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.645+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.645+0000 7f63767fc700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f636c017ca0 0x7f636c01a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.645+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f638002fe90 con 0x7f6384073990 2019-10-13T00:51:53.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.646+0000 7f63897b5700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f636c017ca0 0x7f636c01a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.646+0000 7f63897b5700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f636c017ca0 0x7f636c01a110 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.654+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638408fdc0 con 0x7f6384073990 2019-10-13T00:51:53.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.657+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.657+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f63800009f0 con 0x7f6384073990 2019-10-13T00:51:53.825 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.826+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f638408fdc0 con 0x7f6384073990 2019-10-13T00:51:53.825 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.827+0000 7f6389fb6700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.826 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.827+0000 7f63767fc700 1 -- 172.21.15.155:0/161829305 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f63800095f0 con 0x7f6384073990 2019-10-13T00:51:53.828 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:53.828 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:53.830 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.831+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f636c017ca0 msgr2=0x7f636c01a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.830 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.831+0000 7f638ba19700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f636c017ca0 0x7f636c01a110 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.830 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.831+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 msgr2=0x7f63840906c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.831 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.831+0000 7f638ba19700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f6380010390 tx=0x7f6380010730).stop 2019-10-13T00:51:53.831 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.832+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 shutdown_connections 2019-10-13T00:51:53.831 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.832+0000 7f638ba19700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f636c017ca0 0x7f636c01a110 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.831 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.832+0000 7f638ba19700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6384091690 0x7f6384090c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.831 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.832+0000 7f638ba19700 1 --2- 172.21.15.155:0/161829305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6384073990 0x7f63840906c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.832 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.832+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 shutdown_connections 2019-10-13T00:51:53.832 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.832+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 wait complete. 2019-10-13T00:51:53.833 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.832+0000 7f638ba19700 1 -- 172.21.15.155:0/161829305 >> 172.21.15.155:0/161829305 conn(0x7f638406d080 msgr2=0x7f638406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:53.840 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:53.841 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2019-10-13T00:51:53.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.965+0000 7f68db670700 1 Processor -- start 2019-10-13T00:51:53.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68db670700 1 -- start start 2019-10-13T00:51:53.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68db670700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68d409e9c0 0x7f68d414c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:53.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68db670700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f68d409e1f0 0x7f68d40a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:53.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68db670700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d409e650 con 0x7f68d414be40 2019-10-13T00:51:53.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68db670700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d409ca60 con 0x7f68d409e1f0 2019-10-13T00:51:53.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68db670700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d409c030 con 0x7f68d409e9c0 2019-10-13T00:51:53.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68d940c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68d409e9c0 0x7f68d414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68d9c0d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f68d414be40 0x7f68d409d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48928/0 (socket says 172.21.15.155:48928) 2019-10-13T00:51:53.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.966+0000 7f68d9c0d700 1 -- 172.21.15.155:0/3437993734 learned_addr learned my addr 172.21.15.155:0/3437993734 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:53.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68da40e700 1 --2- 172.21.15.155:0/3437993734 >> v2:172.21.15.142:3300/0 conn(0x7f68d409e1f0 0x7f68d40a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 -- 172.21.15.155:0/3437993734 >> v2:172.21.15.142:3300/0 conn(0x7f68d409e1f0 msgr2=0x7f68d40a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:53.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 --2- 172.21.15.155:0/3437993734 >> v2:172.21.15.142:3300/0 conn(0x7f68d409e1f0 0x7f68d40a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:53.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 -- 172.21.15.155:0/3437993734 >> v1:172.21.15.155:6789/0 conn(0x7f68d414be40 legacy=0x7f68d409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 -- 172.21.15.155:0/3437993734 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d409d0f0 con 0x7f68d409e9c0 2019-10-13T00:51:53.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d8c0b700 1 -- 172.21.15.155:0/3437993734 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283726575 0 0) 0x7f68c4001150 con 0x7f68d414be40 2019-10-13T00:51:53.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 --2- 172.21.15.155:0/3437993734 >> v2:172.21.15.155:3300/0 conn(0x7f68d409e9c0 0x7f68d414c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f68c800af40 tx=0x7f68c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 --2- 172.21.15.155:0/3437993734 >> v2:172.21.15.155:3300/0 conn(0x7f68d409e9c0 0x7f68d414c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f68c800af40 tx=0x7f68c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 --2- 172.21.15.155:0/3437993734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d409e9c0 0x7f68d414c230 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f68c800af40 tx=0x7f68c800b750).ready entity=mon.1 client_cookie=c5b0ebb3adc457a8 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 --2- 172.21.15.155:0/3437993734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d409e9c0 0x7f68d414c230 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f68c800af40 tx=0x7f68c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d8c0b700 1 -- 172.21.15.155:0/3437993734 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c8000960 con 0x7f68d409e9c0 2019-10-13T00:51:53.972 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 --2- 172.21.15.155:0/3437993734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d409e9c0 0x7f68d414c230 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f68c800af40 tx=0x7f68c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d940c700 1 --2- 172.21.15.155:0/3437993734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d409e9c0 0x7f68d414c230 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f68c800af40 tx=0x7f68c800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d8c0b700 1 -- 172.21.15.155:0/3437993734 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c8009850 con 0x7f68d409e9c0 2019-10-13T00:51:53.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68d8c0b700 1 -- 172.21.15.155:0/3437993734 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c8009f20 con 0x7f68d409e9c0 2019-10-13T00:51:53.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68db670700 1 -- 172.21.15.155:0/3437993734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d409e9c0 msgr2=0x7f68d414c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.967+0000 7f68db670700 1 --2- 172.21.15.155:0/3437993734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d409e9c0 0x7f68d414c230 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f68c800af40 tx=0x7f68c800b750).stop 2019-10-13T00:51:53.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.968+0000 7f68db670700 1 -- 172.21.15.155:0/3437993734 shutdown_connections 2019-10-13T00:51:53.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.968+0000 7f68db670700 1 --2- 172.21.15.155:0/3437993734 >> v2:172.21.15.142:3300/0 conn(0x7f68d409e1f0 0x7f68d40a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:53.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.968+0000 7f68db670700 1 --2- 172.21.15.155:0/3437993734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d409e9c0 0x7f68d414c230 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.968+0000 7f68db670700 1 -- 172.21.15.155:0/3437993734 shutdown_connections 2019-10-13T00:51:53.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.968+0000 7f68db670700 1 -- 172.21.15.155:0/3437993734 wait complete. 2019-10-13T00:51:53.974 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.968+0000 7f68db670700 1 -- 172.21.15.155:0/3437993734 >> 172.21.15.155:0/3437993734 conn(0x7f68d4059b50 msgr2=0x7f68d4061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:53.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.970+0000 7f68db670700 1 Processor -- start 2019-10-13T00:51:53.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.970+0000 7f68db670700 1 -- start start 2019-10-13T00:51:53.977 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68db670700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68d412f930 0x7f68d408a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68db670700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68d411a220 0x7f68d408a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68db670700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d4087440 con 0x7f68d408acf0 2019-10-13T00:51:53.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68db670700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d40892e0 con 0x7f68d411a220 2019-10-13T00:51:53.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68db670700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40888b0 con 0x7f68d412f930 2019-10-13T00:51:53.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68d411a220 0x7f68d408a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.978 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68d411a220 0x7f68d408a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58860/0 (socket says 172.21.15.155:58860) 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68da40e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f68d408acf0 0x7f68d4085ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48938/0 (socket says 172.21.15.155:48938) 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 -- 172.21.15.155:0/1753977445 learned_addr learned my addr 172.21.15.155:0/1753977445 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d9c0d700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68d412f930 0x7f68d408a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 -- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68d412f930 msgr2=0x7f68d408a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68d412f930 0x7f68d408a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 -- 172.21.15.155:0/1753977445 >> v1:172.21.15.155:6789/0 conn(0x7f68d408acf0 legacy=0x7f68d4085ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240627306 0 0) 0x7f68cc0055d0 con 0x7f68d408acf0 2019-10-13T00:51:53.979 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 -- 172.21.15.155:0/1753977445 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d4089970 con 0x7f68d411a220 2019-10-13T00:51:53.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> v2:172.21.15.155:3300/0 conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> v2:172.21.15.155:3300/0 conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.971+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.980 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c800af80 con 0x7f68d411a220 2019-10-13T00:51:53.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c800b180 con 0x7f68d411a220 2019-10-13T00:51:53.981 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c800af80 con 0x7f68d411a220 2019-10-13T00:51:53.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d4089970 con 0x7f68d411a220 2019-10-13T00:51:53.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d4086790 con 0x7f68d411a220 2019-10-13T00:51:53.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f68c8009570 con 0x7f68d411a220 2019-10-13T00:51:53.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.982 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d27fc700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f68b4017c60 0x7f68b401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:53.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.972+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f68c80300b0 con 0x7f68d411a220 2019-10-13T00:51:53.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.973+0000 7f68d9c0d700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f68b4017c60 0x7f68b401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:53.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.973+0000 7f68d9c0d700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f68b4017c60 0x7f68b401a0f0 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:53.983 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.981+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d4089970 con 0x7f68d411a220 2019-10-13T00:51:53.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.984+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:53.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:53.984+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f68c800abf0 con 0x7f68d411a220 2019-10-13T00:51:54.147 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.149+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f68d4089970 con 0x7f68d411a220 2019-10-13T00:51:54.874 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.876+0000 7f68d940c700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:54.874 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.876+0000 7f68d27fc700 1 -- 172.21.15.155:0/1753977445 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7f68c8009570 con 0x7f68d411a220 2019-10-13T00:51:54.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f68b4017c60 msgr2=0x7f68b401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:54.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f68b4017c60 0x7f68b401a0f0 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:54.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 msgr2=0x7f68d408a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:54.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7f68c8010420 tx=0x7f68c80107c0).stop 2019-10-13T00:51:54.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 shutdown_connections 2019-10-13T00:51:54.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d411a220 0x7f68d408a7b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:54.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f68b4017c60 0x7f68b401a0f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:54.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.916+0000 7f68db670700 1 --2- 172.21.15.155:0/1753977445 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68d412f930 0x7f68d408a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:54.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.917+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 shutdown_connections 2019-10-13T00:51:54.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.917+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 wait complete. 2019-10-13T00:51:54.917 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:54.917+0000 7f68db670700 1 -- 172.21.15.155:0/1753977445 >> 172.21.15.155:0/1753977445 conn(0x7f68d412b150 msgr2=0x7f68d407bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:54.924 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:54.925 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T00:51:55.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.044+0000 7fea10208700 1 Processor -- start 2019-10-13T00:51:55.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.044+0000 7fea10208700 1 -- start start 2019-10-13T00:51:55.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea10208700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fea08099b10 0x7fea08099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:55.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea10208700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:55.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea10208700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea08064f00 con 0x7fea0809c7f0 2019-10-13T00:51:55.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea10208700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0809b680 con 0x7fea08099b10 2019-10-13T00:51:55.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea10208700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0809ac50 con 0x7fea0809d620 2019-10-13T00:51:55.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea0e7a5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fea08099b10 0x7fea08099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea0dfa4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea0efa6700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fea0809c7f0 0x7fea0809ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:56960/0 (socket says 172.21.15.155:56960) 2019-10-13T00:51:55.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.045+0000 7fea0efa6700 1 -- 172.21.15.155:0/2592616917 learned_addr learned my addr 172.21.15.155:0/2592616917 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:55.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 -- 172.21.15.155:0/2592616917 >> v2:172.21.15.142:3300/0 conn(0x7fea08099b10 msgr2=0x7fea08099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:55.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/2592616917 >> v2:172.21.15.142:3300/0 conn(0x7fea08099b10 0x7fea08099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:55.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 -- 172.21.15.155:0/2592616917 >> v1:172.21.15.142:6789/0 conn(0x7fea0809c7f0 legacy=0x7fea0809ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0d7a3700 1 -- 172.21.15.155:0/2592616917 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177923277 0 0) 0x7fea00001190 con 0x7fea0809c7f0 2019-10-13T00:51:55.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 -- 172.21.15.155:0/2592616917 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0809bd10 con 0x7fea0809d620 2019-10-13T00:51:55.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/2592616917 >> v2:172.21.15.142:3300/0 conn(0x7fea08099b10 0x7fea08099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:55.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe9f000af40 tx=0x7fe9f000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe9f000af40 tx=0x7fe9f000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7fe9f000af40 tx=0x7fe9f000b750).ready entity=mon.2 client_cookie=916ae72ebd36414b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fe9f000af40 tx=0x7fe9f000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0d7a3700 1 -- 172.21.15.155:0/2592616917 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f0000960 con 0x7fea0809d620 2019-10-13T00:51:55.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fe9f000af40 tx=0x7fe9f000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7fe9f000af40 tx=0x7fe9f000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0d7a3700 1 -- 172.21.15.155:0/2592616917 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f0009850 con 0x7fea0809d620 2019-10-13T00:51:55.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea0d7a3700 1 -- 172.21.15.155:0/2592616917 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f0007200 con 0x7fea0809d620 2019-10-13T00:51:55.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea10208700 1 -- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 msgr2=0x7fea0809a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.046+0000 7fea10208700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7fe9f000af40 tx=0x7fe9f000b750).stop 2019-10-13T00:51:55.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.047+0000 7fea10208700 1 -- 172.21.15.155:0/2592616917 shutdown_connections 2019-10-13T00:51:55.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.047+0000 7fea10208700 1 --2- 172.21.15.155:0/2592616917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea0809d620 0x7fea0809a440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.047+0000 7fea10208700 1 --2- 172.21.15.155:0/2592616917 >> v2:172.21.15.142:3300/0 conn(0x7fea08099b10 0x7fea08099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:55.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.047+0000 7fea10208700 1 -- 172.21.15.155:0/2592616917 shutdown_connections 2019-10-13T00:51:55.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.047+0000 7fea10208700 1 -- 172.21.15.155:0/2592616917 wait complete. 2019-10-13T00:51:55.052 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.047+0000 7fea10208700 1 -- 172.21.15.155:0/2592616917 >> 172.21.15.155:0/2592616917 conn(0x7fea08099550 msgr2=0x7fea08061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:55.055 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.049+0000 7fea10208700 1 Processor -- start 2019-10-13T00:51:55.055 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.049+0000 7fea10208700 1 -- start start 2019-10-13T00:51:55.055 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea10208700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea08073990 0x7fea08091710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.055 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea10208700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea08154020 0x7fea08156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.055 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea10208700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea0808eac0 con 0x7fea080926e0 2019-10-13T00:51:55.056 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea10208700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea08090780 con 0x7fea08073990 2019-10-13T00:51:55.056 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea10208700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0808fd50 con 0x7fea08154020 2019-10-13T00:51:55.056 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea0e7a5700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea08073990 0x7fea08091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.056 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea0dfa4700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fea080926e0 0x7fea08091c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48950/0 (socket says 172.21.15.155:48950) 2019-10-13T00:51:55.056 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea0dfa4700 1 -- 172.21.15.155:0/1355626438 learned_addr learned my addr 172.21.15.155:0/1355626438 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:55.056 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea0efa6700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea08154020 0x7fea08156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.056 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea0e7a5700 1 -- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea08154020 msgr2=0x7fea08156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.057 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea08154020 0x7fea08156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.057 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622174194 0 0) 0x7fe9f000ac10 con 0x7fea080926e0 2019-10-13T00:51:55.057 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.050+0000 7fea0e7a5700 1 -- 172.21.15.155:0/1355626438 >> v1:172.21.15.155:6789/0 conn(0x7fea080926e0 legacy=0x7fea08091c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.057 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0e7a5700 1 -- 172.21.15.155:0/1355626438 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08090e10 con 0x7fea08073990 2019-10-13T00:51:55.057 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> v2:172.21.15.155:3300/0 conn(0x7fea08073990 0x7fea08091710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.057 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0efa6700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea08154020 0x7fea08156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:55.057 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> v2:172.21.15.155:3300/0 conn(0x7fea08073990 0x7fea08091710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.058 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.058 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.058 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f8000cc0 con 0x7fea08073990 2019-10-13T00:51:55.058 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.058 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f8001010 con 0x7fea08073990 2019-10-13T00:51:55.058 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.058 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f8000cc0 con 0x7fea08073990 2019-10-13T00:51:55.059 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea08090e10 con 0x7fea08073990 2019-10-13T00:51:55.059 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.051+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea08090e10 con 0x7fea08073990 2019-10-13T00:51:55.059 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.052+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.059 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.052+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fe9f8001010 con 0x7fea08073990 2019-10-13T00:51:55.059 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.052+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.059 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.052+0000 7fea06ffd700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fe9ec017c60 0x7fe9ec01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.059 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.052+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fe9f8030350 con 0x7fea08073990 2019-10-13T00:51:55.060 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.052+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fe9ec017c60 0x7fe9ec01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.060 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.052+0000 7fea0dfa4700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fe9ec017c60 0x7fe9ec01a0f0 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.060 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.060+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea08090e10 con 0x7fea08073990 2019-10-13T00:51:55.061 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.063+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.061 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.063+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fe9f80095f0 con 0x7fea08073990 2019-10-13T00:51:55.229 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.230+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea08090e10 con 0x7fea08073990 2019-10-13T00:51:55.229 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.231+0000 7fea0e7a5700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.229 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.231+0000 7fea06ffd700 1 -- 172.21.15.155:0/1355626438 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1420 (secure 0 0 0) 0x7fe9f8017150 con 0x7fea08073990 2019-10-13T00:51:55.231 INFO:teuthology.orchestra.run.smithi155.stderr:dumped fsmap epoch 3 2019-10-13T00:51:55.231 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:55.231 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T00:51:53.188987+0000","modified":"2019-10-13T00:51:54.870947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T00:51:55.232 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.233+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fe9ec017c60 msgr2=0x7fe9ec01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.232 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fe9ec017c60 0x7fe9ec01a0f0 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.232 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 msgr2=0x7fea08091710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.233 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7fe9f8010730 tx=0x7fe9f8010ad0).stop 2019-10-13T00:51:55.233 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 shutdown_connections 2019-10-13T00:51:55.233 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fe9ec017c60 0x7fe9ec01a0f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.233 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea08154020 0x7fea08156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.233 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 --2- 172.21.15.155:0/1355626438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea08073990 0x7fea08091710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.233 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 shutdown_connections 2019-10-13T00:51:55.233 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 wait complete. 2019-10-13T00:51:55.234 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.234+0000 7fea10208700 1 -- 172.21.15.155:0/1355626438 >> 172.21.15.155:0/1355626438 conn(0x7fea0806d080 msgr2=0x7fea0806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:55.242 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:55.242 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:55.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.361+0000 7fea4960e700 1 Processor -- start 2019-10-13T00:51:55.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.361+0000 7fea4960e700 1 -- start start 2019-10-13T00:51:55.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea4960e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea44099b10 0x7fea44099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:55.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea4960e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fea4409cff0 0x7fea4409a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:55.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea4960e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea44064f00 con 0x7fea4409d620 2019-10-13T00:51:55.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea4960e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4409ac50 con 0x7fea4409cff0 2019-10-13T00:51:55.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea4960e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4409af20 con 0x7fea44099b10 2019-10-13T00:51:55.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea437fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea44099b10 0x7fea44099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.365 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea43fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fea4409cff0 0x7fea4409a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.365 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea437fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea44099b10 0x7fea44099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.155:40320/0 (socket says 172.21.15.155:40320) 2019-10-13T00:51:55.366 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea437fe700 1 -- 172.21.15.155:0/3423555296 learned_addr learned my addr 172.21.15.155:0/3423555296 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:55.366 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea42ffd700 1 --1- 172.21.15.155:0/3423555296 >> v1:172.21.15.142:6789/0 conn(0x7fea4409d620 0x7fea4409c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:56976/0 (socket says 172.21.15.155:56976) 2019-10-13T00:51:55.366 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea43fff700 1 -- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea44099b10 msgr2=0x7fea44099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:55.366 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea43fff700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea44099b10 0x7fea44099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:55.366 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea43fff700 1 -- 172.21.15.155:0/3423555296 >> v1:172.21.15.142:6789/0 conn(0x7fea4409d620 legacy=0x7fea4409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.366 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea427fc700 1 -- 172.21.15.155:0/3423555296 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726260875 0 0) 0x7fea34001070 con 0x7fea4409d620 2019-10-13T00:51:55.366 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea43fff700 1 -- 172.21.15.155:0/3423555296 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4409bd10 con 0x7fea4409cff0 2019-10-13T00:51:55.367 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.362+0000 7fea437fe700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea44099b10 0x7fea44099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:55.367 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea43fff700 1 --2- 172.21.15.155:0/3423555296 >> v2:172.21.15.142:3300/0 conn(0x7fea4409cff0 0x7fea4409a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fea3c00af70 tx=0x7fea3c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.367 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea43fff700 1 --2- 172.21.15.155:0/3423555296 >> v2:172.21.15.142:3300/0 conn(0x7fea4409cff0 0x7fea4409a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fea3c00af70 tx=0x7fea3c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.367 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea43fff700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea4409cff0 0x7fea4409a440 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7fea3c00af70 tx=0x7fea3c00b780).ready entity=mon.0 client_cookie=b8e0adcb69e42a27 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.367 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea43fff700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea4409cff0 0x7fea4409a440 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7fea3c00af70 tx=0x7fea3c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.367 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea427fc700 1 -- 172.21.15.155:0/3423555296 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea3c007230 con 0x7fea4409cff0 2019-10-13T00:51:55.367 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea43fff700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea4409cff0 0x7fea4409a440 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7fea3c00af70 tx=0x7fea3c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.368 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea43fff700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea4409cff0 0x7fea4409a440 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7fea3c00af70 tx=0x7fea3c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.368 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea427fc700 1 -- 172.21.15.155:0/3423555296 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3c000bb0 con 0x7fea4409cff0 2019-10-13T00:51:55.368 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea427fc700 1 -- 172.21.15.155:0/3423555296 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea3c009e50 con 0x7fea4409cff0 2019-10-13T00:51:55.368 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 -- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea4409cff0 msgr2=0x7fea4409a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.368 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea4409cff0 0x7fea4409a440 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7fea3c00af70 tx=0x7fea3c00b780).stop 2019-10-13T00:51:55.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 -- 172.21.15.155:0/3423555296 shutdown_connections 2019-10-13T00:51:55.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea4409cff0 0x7fea4409a440 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 --2- 172.21.15.155:0/3423555296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea44099b10 0x7fea44099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:55.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 -- 172.21.15.155:0/3423555296 shutdown_connections 2019-10-13T00:51:55.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 -- 172.21.15.155:0/3423555296 wait complete. 2019-10-13T00:51:55.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.363+0000 7fea4960e700 1 -- 172.21.15.155:0/3423555296 >> 172.21.15.155:0/3423555296 conn(0x7fea44099550 msgr2=0x7fea44061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:55.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.366+0000 7fea4960e700 1 Processor -- start 2019-10-13T00:51:55.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.366+0000 7fea4960e700 1 -- start start 2019-10-13T00:51:55.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea4960e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fea44073990 0x7fea44091710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea4960e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea440926e0 0x7fea44091c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea4960e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4408eac0 con 0x7fea44092190 2019-10-13T00:51:55.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea4960e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea44090780 con 0x7fea44073990 2019-10-13T00:51:55.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea4960e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4408fd50 con 0x7fea440926e0 2019-10-13T00:51:55.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea440926e0 0x7fea44091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea440926e0 0x7fea44091c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58886/0 (socket says 172.21.15.155:58886) 2019-10-13T00:51:55.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 -- 172.21.15.155:0/996799764 learned_addr learned my addr 172.21.15.155:0/996799764 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:55.375 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea437fe700 1 --2- 172.21.15.155:0/996799764 >> v2:172.21.15.142:3300/0 conn(0x7fea44073990 0x7fea44091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.375 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 -- 172.21.15.155:0/996799764 >> v2:172.21.15.142:3300/0 conn(0x7fea44073990 msgr2=0x7fea44091710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.375 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> v2:172.21.15.142:3300/0 conn(0x7fea44073990 0x7fea44091710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.375 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 -- 172.21.15.155:0/996799764 >> v1:172.21.15.155:6789/0 conn(0x7fea44092190 legacy=0x7fea44088ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.376 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 -- 172.21.15.155:0/996799764 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44090e10 con 0x7fea440926e0 2019-10-13T00:51:55.376 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea437fe700 1 --2- 172.21.15.155:0/996799764 >> v2:172.21.15.142:3300/0 conn(0x7fea44073990 0x7fea44091710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:55.376 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579636945 0 0) 0x7fea3c00aee0 con 0x7fea44092190 2019-10-13T00:51:55.376 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> v2:172.21.15.155:3300/0 conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.376 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> v2:172.21.15.155:3300/0 conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.376 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.377 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.377 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea340012f0 con 0x7fea440926e0 2019-10-13T00:51:55.377 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.377 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.377 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea34000d50 con 0x7fea440926e0 2019-10-13T00:51:55.377 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea340012f0 con 0x7fea440926e0 2019-10-13T00:51:55.378 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea44090e10 con 0x7fea440926e0 2019-10-13T00:51:55.379 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.367+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea44090e10 con 0x7fea440926e0 2019-10-13T00:51:55.379 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.368+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.380 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.368+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fea34003fd0 con 0x7fea440926e0 2019-10-13T00:51:55.380 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.368+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.380 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.368+0000 7fea3bfff700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fea1c017c60 0x7fea1c01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.380 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.368+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fea340302d0 con 0x7fea440926e0 2019-10-13T00:51:55.380 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.368+0000 7fea437fe700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fea1c017c60 0x7fea1c01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.380 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.368+0000 7fea437fe700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fea1c017c60 0x7fea1c01a0f0 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.381 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.376+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea44090e10 con 0x7fea440926e0 2019-10-13T00:51:55.381 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.380+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.381 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.380+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fea34000940 con 0x7fea440926e0 2019-10-13T00:51:55.547 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.549+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea44090e10 con 0x7fea440926e0 2019-10-13T00:51:55.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.550+0000 7fea42ffd700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.550+0000 7fea3bfff700 1 -- 172.21.15.155:0/996799764 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7fea34000d30 con 0x7fea440926e0 2019-10-13T00:51:55.549 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:55.549 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:55.551 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fea1c017c60 msgr2=0x7fea1c01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.551 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fea1c017c60 0x7fea1c01a0f0 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.551 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 msgr2=0x7fea44091c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.551 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fea34010430 tx=0x7fea340107d0).stop 2019-10-13T00:51:55.552 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 shutdown_connections 2019-10-13T00:51:55.552 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fea1c017c60 0x7fea1c01a0f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.552 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 --2- 172.21.15.155:0/996799764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea440926e0 0x7fea44091c50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.552 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.552+0000 7fea4960e700 1 --2- 172.21.15.155:0/996799764 >> v2:172.21.15.142:3300/0 conn(0x7fea44073990 0x7fea44091710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.552 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.553+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 shutdown_connections 2019-10-13T00:51:55.552 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.553+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 wait complete. 2019-10-13T00:51:55.552 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.553+0000 7fea4960e700 1 -- 172.21.15.155:0/996799764 >> 172.21.15.155:0/996799764 conn(0x7fea4406d080 msgr2=0x7fea4406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:55.562 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-10-13T00:51:55.562 INFO:tasks.ceph.mds.a:Restarting daemon 2019-10-13T00:51:55.562 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:55.562 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-10-13T00:51:55.566 INFO:tasks.ceph.mds.a:Started 2019-10-13T00:51:55.566 INFO:tasks.ceph.mds.b:Restarting daemon 2019-10-13T00:51:55.566 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:51:55.566 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-10-13T00:51:55.570 INFO:tasks.ceph.mds.b:Started 2019-10-13T00:51:55.570 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:55.571 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:55.571 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-10-13T00:51:55.650 INFO:tasks.ceph.mds.a.smithi155.stdout:starting mds.a at 2019-10-13T00:51:55.652 INFO:tasks.ceph.mds.b.smithi142.stdout:starting mds.b at 2019-10-13T00:51:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.692+0000 7f56f2d8d700 1 Processor -- start 2019-10-13T00:51:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.692+0000 7f56f2d8d700 1 -- start start 2019-10-13T00:51:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f2d8d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f2d8d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec064f00 con 0x7f56ec099b10 2019-10-13T00:51:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f2d8d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0987e0 con 0x7f56ec09a700 2019-10-13T00:51:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f2d8d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec09b680 con 0x7f56ec09d620 2019-10-13T00:51:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f0b29700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f56ec09a700 0x7f56ec09c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48988/0 (socket says 172.21.15.155:48988) 2019-10-13T00:51:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f0b29700 1 -- 172.21.15.155:0/1498951646 learned_addr learned my addr 172.21.15.155:0/1498951646 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:55.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155185565 0 0) 0x7f56dc001150 con 0x7f56ec09a700 2019-10-13T00:51:55.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d4002e70 con 0x7f56ec09a700 2019-10-13T00:51:55.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890668220 0 0) 0x7f56e8001400 con 0x7f56ec099b10 2019-10-13T00:51:55.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d40030a0 con 0x7f56ec099b10 2019-10-13T00:51:55.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 691170965 0 0) 0x7f56dc001130 con 0x7f56ec09a700 2019-10-13T00:51:55.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f1b2b700 1 -- 172.21.15.155:0/1498951646 >> v1:172.21.15.155:6789/0 conn(0x7f56ec09a700 legacy=0x7f56ec09c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f1b2b700 1 -- 172.21.15.155:0/1498951646 >> v1:172.21.15.142:6789/0 conn(0x7f56ec099b10 legacy=0x7f56ec099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.693+0000 7f56f1b2b700 1 -- 172.21.15.155:0/1498951646 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec09bd10 con 0x7f56ec09d620 2019-10-13T00:51:55.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f56e400af70 tx=0x7f56e400b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f56e400af70 tx=0x7f56e400b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7f56e400af70 tx=0x7f56e400b780).ready entity=mon.2 client_cookie=6d8258a9db0e57b7 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f56e400af70 tx=0x7f56e400b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f56e400af70 tx=0x7f56e400b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56e4007230 con 0x7f56ec09d620 2019-10-13T00:51:55.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f56e400af70 tx=0x7f56e400b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e4000bb0 con 0x7f56ec09d620 2019-10-13T00:51:55.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56e3fff700 1 -- 172.21.15.155:0/1498951646 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56e4009e50 con 0x7f56ec09d620 2019-10-13T00:51:55.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f2d8d700 1 -- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 msgr2=0x7f56ec153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f2d8d700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7f56e400af70 tx=0x7f56e400b780).stop 2019-10-13T00:51:55.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f2d8d700 1 -- 172.21.15.155:0/1498951646 shutdown_connections 2019-10-13T00:51:55.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.694+0000 7f56f2d8d700 1 --2- 172.21.15.155:0/1498951646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec09d620 0x7f56ec153c20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.695+0000 7f56f2d8d700 1 -- 172.21.15.155:0/1498951646 shutdown_connections 2019-10-13T00:51:55.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.695+0000 7f56f2d8d700 1 -- 172.21.15.155:0/1498951646 wait complete. 2019-10-13T00:51:55.700 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.695+0000 7f56f2d8d700 1 -- 172.21.15.155:0/1498951646 >> 172.21.15.155:0/1498951646 conn(0x7f56ec099550 msgr2=0x7f56ec061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:55.705 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.697+0000 7f56f2d8d700 1 Processor -- start 2019-10-13T00:51:55.705 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.697+0000 7f56f2d8d700 1 -- start start 2019-10-13T00:51:55.705 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f2d8d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f56ec08fe30 0x7f56ec08f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.706 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f2d8d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0890d0 con 0x7f56ec153ef0 2019-10-13T00:51:55.706 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f2d8d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec088bd0 con 0x7f56ec073990 2019-10-13T00:51:55.706 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f2d8d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec093d40 con 0x7f56ec08fe30 2019-10-13T00:51:55.706 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f132a700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f56ec073990 0x7f56ec08ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48990/0 (socket says 172.21.15.155:48990) 2019-10-13T00:51:55.706 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f132a700 1 -- 172.21.15.155:0/2499218764 learned_addr learned my addr 172.21.15.155:0/2499218764 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:55.707 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> v2:172.21.15.142:3300/0 conn(0x7f56ec08fe30 0x7f56ec08f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.707 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848873044 0 0) 0x7f56e80010e0 con 0x7f56ec073990 2019-10-13T00:51:55.707 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc002e70 con 0x7f56ec073990 2019-10-13T00:51:55.707 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826703548 0 0) 0x7f56e4009900 con 0x7f56ec153ef0 2019-10-13T00:51:55.708 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc0030a0 con 0x7f56ec153ef0 2019-10-13T00:51:55.708 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3149614818 0 0) 0x7f56e8001710 con 0x7f56ec073990 2019-10-13T00:51:55.708 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56cc0033b0 con 0x7f56ec073990 2019-10-13T00:51:55.708 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f56e8001cc0 con 0x7f56ec073990 2019-10-13T00:51:55.708 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 >> v2:172.21.15.142:3300/0 conn(0x7f56ec08fe30 msgr2=0x7f56ec08f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.708 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 --2- 172.21.15.155:0/2499218764 >> v2:172.21.15.142:3300/0 conn(0x7f56ec08fe30 0x7f56ec08f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.708 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 >> v1:172.21.15.155:6789/0 conn(0x7f56ec073990 legacy=0x7f56ec08ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.709 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 >> v1:172.21.15.142:6789/0 conn(0x7f56ec153ef0 legacy=0x7f56ec1562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.709 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56cc005d20 0x7f56cc006110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.709 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.709 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56cc0051a0 0x7f56cc0032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.709 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc003a00 con 0x7f56cc0051a0 2019-10-13T00:51:55.709 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc003c30 con 0x7f56cc0046e0 2019-10-13T00:51:55.710 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc003e60 con 0x7f56cc005d20 2019-10-13T00:51:55.710 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.698+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> v2:172.21.15.142:3300/0 conn(0x7f56ec08fe30 0x7f56ec08f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:55.710 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f132a700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56cc005d20 0x7f56cc006110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.710 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.710 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f1b2b700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56cc0051a0 0x7f56cc0032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.710 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 -- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56cc005d20 msgr2=0x7f56cc006110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.710 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56cc005d20 0x7f56cc006110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.711 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 -- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56cc0051a0 msgr2=0x7f56cc0032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.711 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56cc0051a0 0x7f56cc0032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.711 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f1b2b700 1 -- 172.21.15.155:0/2499218764 reap_dead start 2019-10-13T00:51:55.711 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc004110 con 0x7f56cc0046e0 2019-10-13T00:51:55.711 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec08e560 con 0x7f56cc0046e0 2019-10-13T00:51:55.711 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.711 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec089580 con 0x7f56cc0046e0 2019-10-13T00:51:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56dc009760 con 0x7f56cc0046e0 2019-10-13T00:51:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc000dd0 con 0x7f56cc0046e0 2019-10-13T00:51:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56dc000fc0 con 0x7f56cc0046e0 2019-10-13T00:51:55.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.699+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.700+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f56dc000fc0 con 0x7f56cc0046e0 2019-10-13T00:51:55.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.700+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.700+0000 7f56e1ffb700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f56cc01a9d0 0x7f56cc01cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:55.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.700+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f56dc0300c0 con 0x7f56cc0046e0 2019-10-13T00:51:55.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.700+0000 7f56f132a700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f56cc01a9d0 0x7f56cc01cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:55.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.700+0000 7f56f132a700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f56cc01a9d0 0x7f56cc01cdf0 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:55.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.707+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c4002e50 con 0x7f56cc0046e0 2019-10-13T00:51:55.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.711+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.711+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f56dc00cdc0 con 0x7f56cc0046e0 2019-10-13T00:51:55.878 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.880+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56c4001d00 con 0x7f56cc0046e0 2019-10-13T00:51:55.878 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.880+0000 7f56f0b29700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:55.878 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.880+0000 7f56e1ffb700 1 -- 172.21.15.155:0/2499218764 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f56dc000db0 con 0x7f56cc0046e0 2019-10-13T00:51:55.879 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:55.879 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:55.881 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f56cc01a9d0 msgr2=0x7f56cc01cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.881 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f56cc01a9d0 0x7f56cc01cdf0 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.881 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 msgr2=0x7f56cc004b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:55.881 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7f56dc00a890 tx=0x7f56dc00d910).stop 2019-10-13T00:51:55.882 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 shutdown_connections 2019-10-13T00:51:55.882 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f56cc01a9d0 0x7f56cc01cdf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.882 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 --2- 172.21.15.155:0/2499218764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56cc0046e0 0x7f56cc004b80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:55.882 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 shutdown_connections 2019-10-13T00:51:55.882 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 wait complete. 2019-10-13T00:51:55.882 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:55.883+0000 7f56f2d8d700 1 -- 172.21.15.155:0/2499218764 >> 172.21.15.155:0/2499218764 conn(0x7f56ec06d080 msgr2=0x7f56ec06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:55.891 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:44.894568+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:48.109887+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:50.855681+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T00:51:55.892 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:55.892 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T00:51:56.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.016+0000 7f9219435700 1 Processor -- start 2019-10-13T00:51:56.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.016+0000 7f9219435700 1 -- start start 2019-10-13T00:51:56.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.017+0000 7f9219435700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:56.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.017+0000 7f9219435700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921409ada0 con 0x7f921414be40 2019-10-13T00:51:56.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.017+0000 7f9219435700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921409b250 con 0x7f921409e1f0 2019-10-13T00:51:56.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.017+0000 7f9219435700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921409ca60 con 0x7f921409e9c0 2019-10-13T00:51:56.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.017+0000 7f9213fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f921409e1f0 0x7f9214152b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49006/0 (socket says 172.21.15.155:49006) 2019-10-13T00:51:56.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.017+0000 7f9213fff700 1 -- 172.21.15.155:0/1382128741 learned_addr learned my addr 172.21.15.155:0/1382128741 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.017+0000 7f9212ffd700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f92127fc700 1 -- 172.21.15.155:0/1382128741 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566639712 0 0) 0x7f920c0014d0 con 0x7f921409e1f0 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f92127fc700 1 -- 172.21.15.155:0/1382128741 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91fc002e70 con 0x7f921409e1f0 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f92127fc700 1 -- 172.21.15.155:0/1382128741 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093886693 0 0) 0x7f9204001080 con 0x7f921414be40 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f92127fc700 1 -- 172.21.15.155:0/1382128741 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91fc004160 con 0x7f921414be40 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 -- 172.21.15.155:0/1382128741 >> v1:172.21.15.155:6789/0 conn(0x7f921409e1f0 legacy=0x7f9214152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 -- 172.21.15.155:0/1382128741 >> v1:172.21.15.142:6789/0 conn(0x7f921414be40 legacy=0x7f921409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 -- 172.21.15.155:0/1382128741 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921409d0f0 con 0x7f921409e9c0 2019-10-13T00:51:56.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f920800af40 tx=0x7f920800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f920800af40 tx=0x7f920800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f920800af40 tx=0x7f920800b750).ready entity=mon.2 client_cookie=6c235149d4d0148f server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f920800af40 tx=0x7f920800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f92127fc700 1 -- 172.21.15.155:0/1382128741 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9208000960 con 0x7f921409e9c0 2019-10-13T00:51:56.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f920800af40 tx=0x7f920800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.022 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9212ffd700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f920800af40 tx=0x7f920800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f92127fc700 1 -- 172.21.15.155:0/1382128741 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9208009850 con 0x7f921409e9c0 2019-10-13T00:51:56.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f92127fc700 1 -- 172.21.15.155:0/1382128741 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9208000960 con 0x7f921409e9c0 2019-10-13T00:51:56.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9219435700 1 -- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 msgr2=0x7f921414c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.018+0000 7f9219435700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f920800af40 tx=0x7f920800b750).stop 2019-10-13T00:51:56.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.019+0000 7f9219435700 1 -- 172.21.15.155:0/1382128741 shutdown_connections 2019-10-13T00:51:56.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.019+0000 7f9219435700 1 --2- 172.21.15.155:0/1382128741 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f921409e9c0 0x7f921414c230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.023 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.019+0000 7f9219435700 1 -- 172.21.15.155:0/1382128741 shutdown_connections 2019-10-13T00:51:56.024 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.019+0000 7f9219435700 1 -- 172.21.15.155:0/1382128741 wait complete. 2019-10-13T00:51:56.024 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.019+0000 7f9219435700 1 -- 172.21.15.155:0/1382128741 >> 172.21.15.155:0/1382128741 conn(0x7f9214059b50 msgr2=0x7f9214061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:56.024 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.021+0000 7f9219435700 1 Processor -- start 2019-10-13T00:51:56.024 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.021+0000 7f9219435700 1 -- start start 2019-10-13T00:51:56.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9219435700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9214087930 0x7f921407f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9219435700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9214080400 0x7f921407f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9219435700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9214086340 con 0x7f921407feb0 2019-10-13T00:51:56.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9219435700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9214086ba0 con 0x7f9214080400 2019-10-13T00:51:56.029 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9219435700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9214086e70 con 0x7f9214087930 2019-10-13T00:51:56.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9214080400 0x7f921407f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9214080400 0x7f921407f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58934/0 (socket says 172.21.15.155:58934) 2019-10-13T00:51:56.030 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 -- 172.21.15.155:0/2988233518 learned_addr learned my addr 172.21.15.155:0/2988233518 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:56.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f92137fe700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9214087930 0x7f921407f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 -- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9214087930 msgr2=0x7f921407f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9214087930 0x7f921407f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 -- 172.21.15.155:0/2988233518 >> v1:172.21.15.155:6789/0 conn(0x7f921407feb0 legacy=0x7f921407cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 -- 172.21.15.155:0/2988233518 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921407eb30 con 0x7f9214080400 2019-10-13T00:51:56.031 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f92137fe700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9214087930 0x7f921407f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-10-13T00:51:56.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> v2:172.21.15.155:3300/0 conn(0x7f9214080400 0x7f921407f970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> v2:172.21.15.155:3300/0 conn(0x7f9214080400 0x7f921407f970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.022+0000 7f91fbfff700 1 -- 172.21.15.155:0/2988233518 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f920800af80 con 0x7f9214080400 2019-10-13T00:51:56.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.032 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f91fbfff700 1 -- 172.21.15.155:0/2988233518 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920800b180 con 0x7f9214080400 2019-10-13T00:51:56.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f91fbfff700 1 -- 172.21.15.155:0/2988233518 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9208009be0 con 0x7f9214080400 2019-10-13T00:51:56.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921407eb30 con 0x7f9214080400 2019-10-13T00:51:56.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921407eb30 con 0x7f9214080400 2019-10-13T00:51:56.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f91fbfff700 1 -- 172.21.15.155:0/2988233518 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f9208009570 con 0x7f9214080400 2019-10-13T00:51:56.033 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f91fbfff700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f91f4017c60 0x7f91f401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.023+0000 7f91fbfff700 1 -- 172.21.15.155:0/2988233518 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9208030380 con 0x7f9214080400 2019-10-13T00:51:56.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.024+0000 7f92137fe700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f91f4017c60 0x7f91f401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.024+0000 7f92137fe700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f91f4017c60 0x7f91f401a0f0 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.034 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.032+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921407eb30 con 0x7f9214080400 2019-10-13T00:51:56.035 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.035+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.035 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.035+0000 7f91fbfff700 1 -- 172.21.15.155:0/2988233518 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f920800aba0 con 0x7f9214080400 2019-10-13T00:51:56.197 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.195+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f921407eb30 con 0x7f9214080400 2019-10-13T00:51:56.197 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.195+0000 7f9212ffd700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.197 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.195+0000 7f91fbfff700 1 -- 172.21.15.155:0/2988233518 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f920800b060 con 0x7f9214080400 2019-10-13T00:51:56.197 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f91f4017c60 msgr2=0x7f91f401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.198 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f91f4017c60 0x7f91f401a0f0 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.198 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 msgr2=0x7f921407f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.198 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f9208010420 tx=0x7f92080107c0).stop 2019-10-13T00:51:56.198 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 shutdown_connections 2019-10-13T00:51:56.198 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9214080400 0x7f921407f970 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.198 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f91f4017c60 0x7f91f401a0f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.198 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 --2- 172.21.15.155:0/2988233518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9214087930 0x7f921407f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.199 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 shutdown_connections 2019-10-13T00:51:56.199 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 wait complete. 2019-10-13T00:51:56.199 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.198+0000 7f9219435700 1 -- 172.21.15.155:0/2988233518 >> 172.21.15.155:0/2988233518 conn(0x7f921407c0b0 msgr2=0x7f921412af30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:56.199 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:51:56.206 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:56.206 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T00:51:56.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.329+0000 7ff1629ae700 1 Processor -- start 2019-10-13T00:51:56.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.329+0000 7ff1629ae700 1 -- start start 2019-10-13T00:51:56.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff1629ae700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff15c09e9c0 0x7ff15c14c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:56.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff1629ae700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c09e650 con 0x7ff15c14be40 2019-10-13T00:51:56.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff1629ae700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c09ada0 con 0x7ff15c09e1f0 2019-10-13T00:51:56.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff1629ae700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c09ca60 con 0x7ff15c09e9c0 2019-10-13T00:51:56.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff15bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff15c09e9c0 0x7ff15c14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.333 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff15bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff15c09e9c0 0x7ff15c14c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58940/0 (socket says 172.21.15.155:58940) 2019-10-13T00:51:56.333 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff16174c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff15c09e1f0 0x7ff15c152b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49016/0 (socket says 172.21.15.155:49016) 2019-10-13T00:51:56.333 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff15bfff700 1 -- 172.21.15.155:0/1578175176 learned_addr learned my addr 172.21.15.155:0/1578175176 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:56.333 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff15bfff700 1 -- 172.21.15.155:0/1578175176 >> v1:172.21.15.155:6789/0 conn(0x7ff15c09e1f0 legacy=0x7ff15c152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.333 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff15bfff700 1 -- 172.21.15.155:0/1578175176 >> v1:172.21.15.142:6789/0 conn(0x7ff15c14be40 legacy=0x7ff15c09d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.333 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.330+0000 7ff15b7fe700 1 -- 172.21.15.155:0/1578175176 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833086232 0 0) 0x7ff154000d30 con 0x7ff15c09e1f0 2019-10-13T00:51:56.334 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15bfff700 1 -- 172.21.15.155:0/1578175176 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c09d0f0 con 0x7ff15c09e9c0 2019-10-13T00:51:56.334 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15bfff700 1 --2- 172.21.15.155:0/1578175176 >> v2:172.21.15.155:3300/0 conn(0x7ff15c09e9c0 0x7ff15c14c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff14c00f730 tx=0x7ff14c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.334 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15bfff700 1 --2- 172.21.15.155:0/1578175176 >> v2:172.21.15.155:3300/0 conn(0x7ff15c09e9c0 0x7ff15c14c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff14c00f730 tx=0x7ff14c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.334 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15bfff700 1 --2- 172.21.15.155:0/1578175176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c09e9c0 0x7ff15c14c230 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7ff14c00f730 tx=0x7ff14c00fad0).ready entity=mon.1 client_cookie=a7c3467aeb8609d4 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.334 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15bfff700 1 --2- 172.21.15.155:0/1578175176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c09e9c0 0x7ff15c14c230 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7ff14c00f730 tx=0x7ff14c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.334 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15b7fe700 1 -- 172.21.15.155:0/1578175176 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c000960 con 0x7ff15c09e9c0 2019-10-13T00:51:56.335 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15bfff700 1 --2- 172.21.15.155:0/1578175176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c09e9c0 0x7ff15c14c230 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7ff14c00f730 tx=0x7ff14c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.335 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15bfff700 1 --2- 172.21.15.155:0/1578175176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c09e9c0 0x7ff15c14c230 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7ff14c00f730 tx=0x7ff14c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.335 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15b7fe700 1 -- 172.21.15.155:0/1578175176 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c000c90 con 0x7ff15c09e9c0 2019-10-13T00:51:56.335 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff15b7fe700 1 -- 172.21.15.155:0/1578175176 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c001360 con 0x7ff15c09e9c0 2019-10-13T00:51:56.335 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff1629ae700 1 -- 172.21.15.155:0/1578175176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c09e9c0 msgr2=0x7ff15c14c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.335 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff1629ae700 1 --2- 172.21.15.155:0/1578175176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c09e9c0 0x7ff15c14c230 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7ff14c00f730 tx=0x7ff14c00fad0).stop 2019-10-13T00:51:56.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff1629ae700 1 -- 172.21.15.155:0/1578175176 shutdown_connections 2019-10-13T00:51:56.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff1629ae700 1 --2- 172.21.15.155:0/1578175176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c09e9c0 0x7ff15c14c230 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.331+0000 7ff1629ae700 1 -- 172.21.15.155:0/1578175176 shutdown_connections 2019-10-13T00:51:56.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.332+0000 7ff1629ae700 1 -- 172.21.15.155:0/1578175176 wait complete. 2019-10-13T00:51:56.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.332+0000 7ff1629ae700 1 -- 172.21.15.155:0/1578175176 >> 172.21.15.155:0/1578175176 conn(0x7ff15c059b50 msgr2=0x7ff15c061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:56.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.334+0000 7ff1629ae700 1 Processor -- start 2019-10-13T00:51:56.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.334+0000 7ff1629ae700 1 -- start start 2019-10-13T00:51:56.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1629ae700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff15c11b2f0 0x7ff15c11ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1629ae700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c08b000 con 0x7ff15c12f930 2019-10-13T00:51:56.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1629ae700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c08ab50 con 0x7ff15c153ea0 2019-10-13T00:51:56.342 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1629ae700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c089470 con 0x7ff15c11b2f0 2019-10-13T00:51:56.342 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff15c11b2f0 0x7ff15c11ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.342 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff15c11b2f0 0x7ff15c11ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:58946/0 (socket says 172.21.15.155:58946) 2019-10-13T00:51:56.342 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff16174c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff15c153ea0 0x7ff15c1562a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49024/0 (socket says 172.21.15.155:49024) 2019-10-13T00:51:56.343 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 -- 172.21.15.155:0/160776314 learned_addr learned my addr 172.21.15.155:0/160776314 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:56.343 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 -- 172.21.15.155:0/160776314 >> v1:172.21.15.155:6789/0 conn(0x7ff15c153ea0 legacy=0x7ff15c1562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.343 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 -- 172.21.15.155:0/160776314 >> v1:172.21.15.142:6789/0 conn(0x7ff15c12f930 legacy=0x7ff15c11a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.343 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452186561 0 0) 0x7ff154001270 con 0x7ff15c153ea0 2019-10-13T00:51:56.344 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 -- 172.21.15.155:0/160776314 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c089b00 con 0x7ff15c11b2f0 2019-10-13T00:51:56.344 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> v2:172.21.15.155:3300/0 conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.344 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> v2:172.21.15.155:3300/0 conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.344 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.346 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.346 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.346 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c008170 con 0x7ff15c11b2f0 2019-10-13T00:51:56.346 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.346 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c005360 con 0x7ff15c11b2f0 2019-10-13T00:51:56.347 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff14c0059f0 con 0x7ff15c11b2f0 2019-10-13T00:51:56.347 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c089b00 con 0x7ff15c11b2f0 2019-10-13T00:51:56.347 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.335+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c089b00 con 0x7ff15c11b2f0 2019-10-13T00:51:56.347 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.336+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.347 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.336+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff14c005550 con 0x7ff15c11b2f0 2019-10-13T00:51:56.347 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.336+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.347 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.336+0000 7ff1597fa700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff140017c60 0x7ff14001a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.348 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.336+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff14c0300d0 con 0x7ff15c11b2f0 2019-10-13T00:51:56.348 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.336+0000 7ff160f4b700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff140017c60 0x7ff14001a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.348 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.337+0000 7ff160f4b700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff140017c60 0x7ff14001a0f0 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.349 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.345+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c089b00 con 0x7ff15c11b2f0 2019-10-13T00:51:56.349 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.348+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.349 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.348+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff14c0075c0 con 0x7ff15c11b2f0 2019-10-13T00:51:56.506 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.508+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff15c089b00 con 0x7ff15c11b2f0 2019-10-13T00:51:56.506 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.508+0000 7ff15bfff700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.506 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.508+0000 7ff1597fa700 1 -- 172.21.15.155:0/160776314 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff14c005550 con 0x7ff15c11b2f0 2019-10-13T00:51:56.507 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:51:56.509 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.510+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff140017c60 msgr2=0x7ff14001a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.510+0000 7ff1629ae700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff140017c60 0x7ff14001a0f0 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.510+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 msgr2=0x7ff15c11ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.510+0000 7ff1629ae700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7ff14c010440 tx=0x7ff14c0107e0).stop 2019-10-13T00:51:56.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.511+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 shutdown_connections 2019-10-13T00:51:56.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.511+0000 7ff1629ae700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff140017c60 0x7ff14001a0f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.511+0000 7ff1629ae700 1 --2- 172.21.15.155:0/160776314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff15c11b2f0 0x7ff15c11ab20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.511+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 shutdown_connections 2019-10-13T00:51:56.511 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.511+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 wait complete. 2019-10-13T00:51:56.511 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.511+0000 7ff1629ae700 1 -- 172.21.15.155:0/160776314 >> 172.21.15.155:0/160776314 conn(0x7ff15c12b150 msgr2=0x7ff15c07bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:56.519 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:56.519 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T00:51:56.645 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.643+0000 7f13d4a7d700 1 Processor -- start 2019-10-13T00:51:56.645 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.643+0000 7f13d4a7d700 1 -- start start 2019-10-13T00:51:56.645 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13d4a7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:56.646 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13d4a7d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d009ada0 con 0x7f13d014be40 2019-10-13T00:51:56.646 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13d4a7d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d009b250 con 0x7f13d009e9c0 2019-10-13T00:51:56.646 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13d4a7d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d009ca60 con 0x7f13d014c230 2019-10-13T00:51:56.646 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13ce59c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f13d009e9c0 0x7f13d0150b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49030/0 (socket says 172.21.15.155:49030) 2019-10-13T00:51:56.646 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13ce59c700 1 -- 172.21.15.155:0/1639802002 learned_addr learned my addr 172.21.15.155:0/1639802002 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:56.646 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.647 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cdd9b700 1 -- 172.21.15.155:0/1639802002 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981350933 0 0) 0x7f13b8001100 con 0x7f13d009e9c0 2019-10-13T00:51:56.647 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cdd9b700 1 -- 172.21.15.155:0/1639802002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b0002e70 con 0x7f13d009e9c0 2019-10-13T00:51:56.647 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cdd9b700 1 -- 172.21.15.155:0/1639802002 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903745635 0 0) 0x7f13c0001400 con 0x7f13d014be40 2019-10-13T00:51:56.647 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cdd9b700 1 -- 172.21.15.155:0/1639802002 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b00030a0 con 0x7f13d014be40 2019-10-13T00:51:56.648 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 -- 172.21.15.155:0/1639802002 >> v1:172.21.15.155:6789/0 conn(0x7f13d009e9c0 legacy=0x7f13d0150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.648 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 -- 172.21.15.155:0/1639802002 >> v1:172.21.15.142:6789/0 conn(0x7f13d014be40 legacy=0x7f13d009d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.648 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 -- 172.21.15.155:0/1639802002 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d009d0f0 con 0x7f13d014c230 2019-10-13T00:51:56.648 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f13c800af70 tx=0x7f13c800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.648 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f13c800af70 tx=0x7f13c800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.648 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f13c800af70 tx=0x7f13c800b780).ready entity=mon.2 client_cookie=414618f1bd0a62c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.648 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f13c800af70 tx=0x7f13c800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.649 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cdd9b700 1 -- 172.21.15.155:0/1639802002 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13c8007230 con 0x7f13d014c230 2019-10-13T00:51:56.649 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.644+0000 7f13cf59e700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f13c800af70 tx=0x7f13c800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.649 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13cf59e700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f13c800af70 tx=0x7f13c800b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.649 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13cdd9b700 1 -- 172.21.15.155:0/1639802002 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c8000bb0 con 0x7f13d014c230 2019-10-13T00:51:56.649 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13cdd9b700 1 -- 172.21.15.155:0/1639802002 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13c8009e50 con 0x7f13d014c230 2019-10-13T00:51:56.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 msgr2=0x7f13d009e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13d4a7d700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f13c800af70 tx=0x7f13c800b780).stop 2019-10-13T00:51:56.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1639802002 shutdown_connections 2019-10-13T00:51:56.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13d4a7d700 1 --2- 172.21.15.155:0/1639802002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d014c230 0x7f13d009e1f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1639802002 shutdown_connections 2019-10-13T00:51:56.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1639802002 wait complete. 2019-10-13T00:51:56.650 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.645+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1639802002 >> 172.21.15.155:0/1639802002 conn(0x7f13d0059b50 msgr2=0x7f13d0061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:56.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.647+0000 7f13d4a7d700 1 Processor -- start 2019-10-13T00:51:56.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13d4a7d700 1 -- start start 2019-10-13T00:51:56.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13d4a7d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f13d012f930 0x7f13d011a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.654 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13d4a7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13d4a7d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d008b000 con 0x7f13d011ada0 2019-10-13T00:51:56.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13d4a7d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d0089470 con 0x7f13d012f930 2019-10-13T00:51:56.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13d4a7d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d0088a40 con 0x7f13d011b2f0 2019-10-13T00:51:56.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13ce59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13ced9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f13d012f930 0x7f13d011a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.655 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13ce59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.155:40400/0 (socket says 172.21.15.155:40400) 2019-10-13T00:51:56.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.648+0000 7f13ce59c700 1 -- 172.21.15.155:0/1028206603 learned_addr learned my addr 172.21.15.155:0/1028206603 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:56.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 -- 172.21.15.155:0/1028206603 >> v2:172.21.15.142:3300/0 conn(0x7f13d012f930 msgr2=0x7f13d011a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> v2:172.21.15.142:3300/0 conn(0x7f13d012f930 0x7f13d011a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 -- 172.21.15.155:0/1028206603 >> v1:172.21.15.142:6789/0 conn(0x7f13d011ada0 legacy=0x7f13d0078870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 -- 172.21.15.155:0/1028206603 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d0089b00 con 0x7f13d011b2f0 2019-10-13T00:51:56.656 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.657 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.657 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.657 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.657 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.657 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.657 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13c77fe700 1 -- 172.21.15.155:0/1028206603 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b8000c90 con 0x7f13d011b2f0 2019-10-13T00:51:56.657 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13c77fe700 1 -- 172.21.15.155:0/1028206603 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b80012f0 con 0x7f13d011b2f0 2019-10-13T00:51:56.658 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13c77fe700 1 -- 172.21.15.155:0/1028206603 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b8001980 con 0x7f13d011b2f0 2019-10-13T00:51:56.658 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d0089b00 con 0x7f13d011b2f0 2019-10-13T00:51:56.658 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d0089b00 con 0x7f13d011b2f0 2019-10-13T00:51:56.658 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.649+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.658 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.650+0000 7f13c77fe700 1 -- 172.21.15.155:0/1028206603 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f13b80014e0 con 0x7f13d011b2f0 2019-10-13T00:51:56.658 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.650+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.659 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.650+0000 7f13c77fe700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f13a8017c60 0x7f13a801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.659 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.650+0000 7f13c77fe700 1 -- 172.21.15.155:0/1028206603 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f13b8030310 con 0x7f13d011b2f0 2019-10-13T00:51:56.659 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.650+0000 7f13ced9d700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f13a8017c60 0x7f13a801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.659 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.650+0000 7f13ced9d700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f13a8017c60 0x7f13a801a0f0 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.660 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.658+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d0089b00 con 0x7f13d011b2f0 2019-10-13T00:51:56.660 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.661+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.660 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.661+0000 7f13c77fe700 1 -- 172.21.15.155:0/1028206603 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f13b8000ea0 con 0x7f13d011b2f0 2019-10-13T00:51:56.818 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.820+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f13d0089b00 con 0x7f13d011b2f0 2019-10-13T00:51:56.819 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.820+0000 7f13ce59c700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.819 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.820+0000 7f13c77fe700 1 -- 172.21.15.155:0/1028206603 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f13b80012f0 con 0x7f13d011b2f0 2019-10-13T00:51:56.819 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:51:56.821 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f13a8017c60 msgr2=0x7f13a801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.822 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f13a8017c60 0x7f13a801a0f0 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.822 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 msgr2=0x7f13d011a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.822 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f13b80103d0 tx=0x7f13b8010770).stop 2019-10-13T00:51:56.822 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 shutdown_connections 2019-10-13T00:51:56.822 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f13a8017c60 0x7f13a801a0f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.822 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 --2- 172.21.15.155:0/1028206603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13d011b2f0 0x7f13d011a860 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.823 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 --2- 172.21.15.155:0/1028206603 >> v2:172.21.15.142:3300/0 conn(0x7f13d012f930 0x7f13d011a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.823 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 shutdown_connections 2019-10-13T00:51:56.823 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 wait complete. 2019-10-13T00:51:56.823 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.823+0000 7f13d4a7d700 1 -- 172.21.15.155:0/1028206603 >> 172.21.15.155:0/1028206603 conn(0x7f13d012b150 msgr2=0x7f13d007bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:56.831 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-10-13T00:51:56.831 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-10-13T00:51:56.831 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:56.831 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-10-13T00:51:56.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.951+0000 7f66ca738700 1 Processor -- start 2019-10-13T00:51:56.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.951+0000 7f66ca738700 1 -- start start 2019-10-13T00:51:56.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66ca738700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4099b10 0x7f66c4099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:56.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66ca738700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66c409d620 0x7f66c409a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:56.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66ca738700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c4064f00 con 0x7f66c409c7f0 2019-10-13T00:51:56.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66ca738700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c409ac50 con 0x7f66c409d620 2019-10-13T00:51:56.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66ca738700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c409af20 con 0x7f66c4099b10 2019-10-13T00:51:56.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c3fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66c409d620 0x7f66c409a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c8cd5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4099b10 0x7f66c4099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c94d6700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f66c409c7f0 0x7f66c409cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:57062/0 (socket says 172.21.15.155:57062) 2019-10-13T00:51:56.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c94d6700 1 -- 172.21.15.155:0/3089881802 learned_addr learned my addr 172.21.15.155:0/3089881802 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:56.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c3fff700 1 -- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4099b10 msgr2=0x7f66c4099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:56.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c3fff700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4099b10 0x7f66c4099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:56.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c3fff700 1 -- 172.21.15.155:0/3089881802 >> v1:172.21.15.142:6789/0 conn(0x7f66c409c7f0 legacy=0x7f66c409cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c3fff700 1 -- 172.21.15.155:0/3089881802 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c409bd10 con 0x7f66c409d620 2019-10-13T00:51:56.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c37fe700 1 -- 172.21.15.155:0/3089881802 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762552550 0 0) 0x7f66bc001480 con 0x7f66c409c7f0 2019-10-13T00:51:56.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.952+0000 7f66c8cd5700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4099b10 0x7f66c4099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:56.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c3fff700 1 --2- 172.21.15.155:0/3089881802 >> v2:172.21.15.142:3300/0 conn(0x7f66c409d620 0x7f66c409a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f66b400af40 tx=0x7f66b400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c3fff700 1 --2- 172.21.15.155:0/3089881802 >> v2:172.21.15.142:3300/0 conn(0x7f66c409d620 0x7f66c409a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f66b400af40 tx=0x7f66b400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c3fff700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c409d620 0x7f66c409a440 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f66b400af40 tx=0x7f66b400b750).ready entity=mon.0 client_cookie=43a0afd2c66cd650 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c3fff700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c409d620 0x7f66c409a440 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f66b400af40 tx=0x7f66b400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c37fe700 1 -- 172.21.15.155:0/3089881802 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66b4000960 con 0x7f66c409d620 2019-10-13T00:51:56.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c3fff700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c409d620 0x7f66c409a440 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f66b400af40 tx=0x7f66b400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c3fff700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c409d620 0x7f66c409a440 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f66b400af40 tx=0x7f66b400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c37fe700 1 -- 172.21.15.155:0/3089881802 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b4009850 con 0x7f66c409d620 2019-10-13T00:51:56.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66c37fe700 1 -- 172.21.15.155:0/3089881802 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66b4009f20 con 0x7f66c409d620 2019-10-13T00:51:56.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 -- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c409d620 msgr2=0x7f66c409a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c409d620 0x7f66c409a440 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f66b400af40 tx=0x7f66b400b750).stop 2019-10-13T00:51:56.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 -- 172.21.15.155:0/3089881802 shutdown_connections 2019-10-13T00:51:56.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c409d620 0x7f66c409a440 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 --2- 172.21.15.155:0/3089881802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4099b10 0x7f66c4099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:56.960 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 -- 172.21.15.155:0/3089881802 shutdown_connections 2019-10-13T00:51:56.960 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 -- 172.21.15.155:0/3089881802 wait complete. 2019-10-13T00:51:56.960 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.953+0000 7f66ca738700 1 -- 172.21.15.155:0/3089881802 >> 172.21.15.155:0/3089881802 conn(0x7f66c4099550 msgr2=0x7f66c4061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:56.963 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.956+0000 7f66ca738700 1 Processor -- start 2019-10-13T00:51:56.965 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.956+0000 7f66ca738700 1 -- start start 2019-10-13T00:51:56.965 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66ca738700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66c4091690 0x7f66c4090ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.966 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66ca738700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4152ee0 0x7f66c41552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.966 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66ca738700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c408d4b0 con 0x7f66c4073990 2019-10-13T00:51:56.966 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66ca738700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c408f730 con 0x7f66c4091690 2019-10-13T00:51:56.966 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66ca738700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c408ed00 con 0x7f66c4152ee0 2019-10-13T00:51:56.966 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c8cd5700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f66c4073990 0x7f66c40906c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49048/0 (socket says 172.21.15.155:49048) 2019-10-13T00:51:56.967 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c8cd5700 1 -- 172.21.15.155:0/2648107884 learned_addr learned my addr 172.21.15.155:0/2648107884 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:56.967 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c94d6700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4152ee0 0x7f66c41552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.967 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> v2:172.21.15.142:3300/0 conn(0x7f66c4091690 0x7f66c4090ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.967 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144850072 0 0) 0x7f66ac0010b0 con 0x7f66c4073990 2019-10-13T00:51:56.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a8002e70 con 0x7f66c4073990 2019-10-13T00:51:56.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 -- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4152ee0 msgr2=0x7f66c41552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4152ee0 0x7f66c41552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:56.968 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 -- 172.21.15.155:0/2648107884 >> v1:172.21.15.155:6789/0 conn(0x7f66c4073990 legacy=0x7f66c40906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:56.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 -- 172.21.15.155:0/2648107884 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c408fdc0 con 0x7f66c4091690 2019-10-13T00:51:56.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c94d6700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4152ee0 0x7f66c41552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:56.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> v2:172.21.15.142:3300/0 conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> v2:172.21.15.142:3300/0 conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.969 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=READY pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66b400b010 con 0x7f66c4091690 2019-10-13T00:51:56.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.970 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b4009970 con 0x7f66c4091690 2019-10-13T00:51:56.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66b400a040 con 0x7f66c4091690 2019-10-13T00:51:56.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c408fdc0 con 0x7f66c4091690 2019-10-13T00:51:56.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.957+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c408fdc0 con 0x7f66c4091690 2019-10-13T00:51:56.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.958+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.958+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f66b400b010 con 0x7f66c4091690 2019-10-13T00:51:56.971 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.958+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.972 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.958+0000 7f66c17fa700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f66a8017ca0 0x7f66a801a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:56.972 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.958+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f66b4076250 con 0x7f66c4091690 2019-10-13T00:51:56.972 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.958+0000 7f66c8cd5700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f66a8017ca0 0x7f66a801a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:56.972 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.959+0000 7f66c8cd5700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f66a8017ca0 0x7f66a801a110 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:56.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.966+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c408fdc0 con 0x7f66c4091690 2019-10-13T00:51:56.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.970+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:56.973 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:56.970+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f66b4009570 con 0x7f66c4091690 2019-10-13T00:51:57.142 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.144+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f66c408fdc0 con 0x7f66c4091690 2019-10-13T00:51:57.144 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.146+0000 7f66c3fff700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:57.144 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.146+0000 7f66c17fa700 1 -- 172.21.15.155:0/2648107884 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+68495 (secure 0 0 0) 0x7f66b400b010 con 0x7f66c4091690 2019-10-13T00:51:57.150 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:57.151 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f66a8017ca0 msgr2=0x7f66a801a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:57.151 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f66a8017ca0 0x7f66a801a110 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:57.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 msgr2=0x7f66c4090ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:57.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 secure :-1 s=READY pgs=82 cs=0 l=1 rx=0x7f66b4056580 tx=0x7f66b4056920).stop 2019-10-13T00:51:57.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 shutdown_connections 2019-10-13T00:51:57.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66c4091690 0x7f66c4090ec0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:57.152 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f66a8017ca0 0x7f66a801a110 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:57.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 --2- 172.21.15.155:0/2648107884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66c4152ee0 0x7f66c41552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:57.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 shutdown_connections 2019-10-13T00:51:57.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 wait complete. 2019-10-13T00:51:57.153 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:57.149+0000 7f66ca738700 1 -- 172.21.15.155:0/2648107884 >> 172.21.15.155:0/2648107884 conn(0x7f66c406d080 msgr2=0x7f66c406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":3,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":5662},{"type":"v1","addr":"172.21.15.142:6801","nonce":5662}]},"active_addr":"172.21.15.142:6801/5662","active_change":"2019-10-13T00:51:31.774150+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]}} 2019-10-13T00:51:57.162 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-10-13T00:51:57.971 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:57.972 INFO:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:58.084 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.080+0000 7fbb0f8d9700 1 Processor -- start 2019-10-13T00:51:58.084 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.081+0000 7fbb0f8d9700 1 -- start start 2019-10-13T00:51:58.084 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.081+0000 7fbb0f8d9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb0809a700 0x7fbb0809c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.084 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.081+0000 7fbb0f8d9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb08064f00 con 0x7fbb08099b10 2019-10-13T00:51:58.085 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.081+0000 7fbb0f8d9700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080987e0 con 0x7fbb0809d620 2019-10-13T00:51:58.085 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.081+0000 7fbb0f8d9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0809b680 con 0x7fbb0809a700 2019-10-13T00:51:58.085 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.081+0000 7fbb0e677700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbb0809d620 0x7fbb08152c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49060/0 (socket says 172.21.15.155:49060) 2019-10-13T00:51:58.085 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.081+0000 7fbb0e677700 1 -- 172.21.15.155:0/1041869426 learned_addr learned my addr 172.21.15.155:0/1041869426 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:58.085 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> v2:172.21.15.142:3300/0 conn(0x7fbb0809a700 0x7fbb0809c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.085 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123565571 0 0) 0x7fbb00001150 con 0x7fbb0809d620 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf0002e70 con 0x7fbb0809d620 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3808858921 0 0) 0x7fbb00001130 con 0x7fbb0809d620 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 >> v2:172.21.15.142:3300/0 conn(0x7fbb0809a700 msgr2=0x7fbb0809c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 --2- 172.21.15.155:0/1041869426 >> v2:172.21.15.142:3300/0 conn(0x7fbb0809a700 0x7fbb0809c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 >> v1:172.21.15.142:6789/0 conn(0x7fbb08099b10 legacy=0x7fbb08099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0809bd10 con 0x7fbb0809d620 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7fbb00001660 con 0x7fbb0809d620 2019-10-13T00:51:58.086 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 >> v1:172.21.15.155:6789/0 conn(0x7fbb0809d620 legacy=0x7fbb08152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.087 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbaf0003b30 0x7fbaf0005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.087 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.087 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbaf0009450 0x7fbaf000b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.087 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf000c0d0 con 0x7fbaf0009450 2019-10-13T00:51:58.087 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00502d0 con 0x7fbaf0006740 2019-10-13T00:51:58.087 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf0050500 con 0x7fbaf0003b30 2019-10-13T00:51:58.087 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> v2:172.21.15.142:3300/0 conn(0x7fbb0809a700 0x7fbb0809c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:58.088 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0e677700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbaf0009450 0x7fbaf000b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.088 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0de76700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbaf0003b30 0x7fbaf0005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.088 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.088 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 -- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbaf0003b30 msgr2=0x7fbaf0005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:58.089 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbaf0003b30 0x7fbaf0005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.089 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 -- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbaf0009450 msgr2=0x7fbaf000b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:58.089 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbaf0009450 0x7fbaf000b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.089 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0d675700 1 -- 172.21.15.155:0/1041869426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00507b0 con 0x7fbaf0006740 2019-10-13T00:51:58.089 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.082+0000 7fbb0e677700 1 -- 172.21.15.155:0/1041869426 reap_dead start 2019-10-13T00:51:58.089 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbaf8006b30 tx=0x7fbaf8007060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.089 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbaf8006b30 tx=0x7fbaf8007060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.090 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7fbaf8006b30 tx=0x7fbaf8007060).ready entity=mon.1 client_cookie=a3be7068c894acdb server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.090 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7fbaf8006b30 tx=0x7fbaf8007060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.090 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf8005170 con 0x7fbaf0006740 2019-10-13T00:51:58.090 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7fbaf8006b30 tx=0x7fbaf8007060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.090 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0d675700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7fbaf8006b30 tx=0x7fbaf8007060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.090 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8005370 con 0x7fbaf0006740 2019-10-13T00:51:58.090 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0ce74700 1 -- 172.21.15.155:0/1041869426 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf8005a40 con 0x7fbaf0006740 2019-10-13T00:51:58.091 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 msgr2=0x7fbaf0008bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.091 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0f8d9700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7fbaf8006b30 tx=0x7fbaf8007060).stop 2019-10-13T00:51:58.091 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/1041869426 shutdown_connections 2019-10-13T00:51:58.091 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.083+0000 7fbb0f8d9700 1 --2- 172.21.15.155:0/1041869426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbaf0006740 0x7fbaf0008bf0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.091 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.084+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/1041869426 shutdown_connections 2019-10-13T00:51:58.091 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.084+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/1041869426 wait complete. 2019-10-13T00:51:58.091 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.084+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/1041869426 >> 172.21.15.155:0/1041869426 conn(0x7fbb08099550 msgr2=0x7fbb08061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.092 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.086+0000 7fbb0f8d9700 1 Processor -- start 2019-10-13T00:51:58.092 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.086+0000 7fbb0f8d9700 1 -- start start 2019-10-13T00:51:58.095 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0f8d9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb08089090 0x7fbb080924e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.095 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0f8d9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0808fc80 con 0x7fbb0808e8c0 2019-10-13T00:51:58.095 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0f8d9700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080896c0 con 0x7fbb0808f4b0 2019-10-13T00:51:58.096 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0f8d9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb08091550 con 0x7fbb08089090 2019-10-13T00:51:58.096 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0e677700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbb0808f4b0 0x7fbb08092a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49074/0 (socket says 172.21.15.155:49074) 2019-10-13T00:51:58.096 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0e677700 1 -- 172.21.15.155:0/109668474 learned_addr learned my addr 172.21.15.155:0/109668474 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:58.096 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> v2:172.21.15.142:3300/0 conn(0x7fbb08089090 0x7fbb080924e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.096 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301915511 0 0) 0x7fbb00001150 con 0x7fbb0808f4b0 2019-10-13T00:51:58.096 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8002e70 con 0x7fbb0808f4b0 2019-10-13T00:51:58.096 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 -- 172.21.15.155:0/109668474 >> v1:172.21.15.155:6789/0 conn(0x7fbb0808f4b0 legacy=0x7fbb08092a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.097 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 -- 172.21.15.155:0/109668474 >> v1:172.21.15.142:6789/0 conn(0x7fbb0808e8c0 legacy=0x7fbb0808ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.097 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 -- 172.21.15.155:0/109668474 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08091be0 con 0x7fbb08089090 2019-10-13T00:51:58.097 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> v2:172.21.15.142:3300/0 conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.097 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> v2:172.21.15.142:3300/0 conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.097 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.097 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.097 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf8000940 con 0x7fbb08089090 2019-10-13T00:51:58.098 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.098 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8000940 con 0x7fbb08089090 2019-10-13T00:51:58.098 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.098 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb08091be0 con 0x7fbb08089090 2019-10-13T00:51:58.098 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.087+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaf8005550 con 0x7fbb08089090 2019-10-13T00:51:58.098 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.088+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb08091be0 con 0x7fbb08089090 2019-10-13T00:51:58.098 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.088+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.099 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.088+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fbaf8005210 con 0x7fbb08089090 2019-10-13T00:51:58.099 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.088+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.099 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.088+0000 7fbafe7fc700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fbae8017ca0 0x7fbae801a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.099 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.088+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fbaf802ffe0 con 0x7fbb08089090 2019-10-13T00:51:58.099 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.089+0000 7fbb0de76700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fbae8017ca0 0x7fbae801a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.099 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.089+0000 7fbb0de76700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fbae8017ca0 0x7fbae801a110 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.099 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.096+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb08091be0 con 0x7fbb08089090 2019-10-13T00:51:58.100 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.099+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.100 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.099+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fbaf80068c0 con 0x7fbb08089090 2019-10-13T00:51:58.267 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.269+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb08091be0 con 0x7fbb08089090 2019-10-13T00:51:58.268 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.270+0000 7fbb0d675700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.268 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.270+0000 7fbafe7fc700 1 -- 172.21.15.155:0/109668474 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7fbaf8000940 con 0x7fbb08089090 2019-10-13T00:51:58.270 INFO:teuthology.misc.health.smithi155.stdout: 2019-10-13T00:51:58.270 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:58.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fbae8017ca0 msgr2=0x7fbae801a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.272 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fbae8017ca0 0x7fbae801a110 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 msgr2=0x7fbb080924e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fbaf8010350 tx=0x7fbaf80106f0).stop 2019-10-13T00:51:58.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 shutdown_connections 2019-10-13T00:51:58.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fbae8017ca0 0x7fbae801a110 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 --2- 172.21.15.155:0/109668474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb08089090 0x7fbb080924e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 shutdown_connections 2019-10-13T00:51:58.273 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 wait complete. 2019-10-13T00:51:58.274 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:51:58.274+0000 7fbb0f8d9700 1 -- 172.21.15.155:0/109668474 >> 172.21.15.155:0/109668474 conn(0x7fbb0806bfe0 msgr2=0x7fbb0806d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.281 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-13T00:51:58.282 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:58.282 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:51:58.404 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.402+0000 7f1571467700 1 Processor -- start 2019-10-13T00:51:58.404 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.402+0000 7f1571467700 1 -- start start 2019-10-13T00:51:58.404 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f1571467700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f156c099b10 0x7f156c099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.404 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f1571467700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f156c09d620 0x7f156c09a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f1571467700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c09c7f0 0x7f156c09cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f1571467700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c09b680 con 0x7f156c09d620 2019-10-13T00:51:58.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f1571467700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c09ac50 con 0x7f156c099b10 2019-10-13T00:51:58.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f1571467700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c09af20 con 0x7f156c09c7f0 2019-10-13T00:51:58.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f156c099b10 0x7f156c099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f156c099b10 0x7f156c099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59002/0 (socket says 172.21.15.155:59002) 2019-10-13T00:51:58.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 -- 172.21.15.155:0/3249965604 learned_addr learned my addr 172.21.15.155:0/3249965604 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:58.406 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156affd700 1 --2- 172.21.15.155:0/3249965604 >> v2:172.21.15.142:3300/0 conn(0x7f156c09d620 0x7f156c09a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.406 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156bfff700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c09c7f0 0x7f156c09cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.406 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 -- 172.21.15.155:0/3249965604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c09c7f0 msgr2=0x7f156c09cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:58.406 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c09c7f0 0x7f156c09cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.406 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 -- 172.21.15.155:0/3249965604 >> v2:172.21.15.142:3300/0 conn(0x7f156c09d620 msgr2=0x7f156c09a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:58.406 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> v2:172.21.15.142:3300/0 conn(0x7f156c09d620 0x7f156c09a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.406 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 -- 172.21.15.155:0/3249965604 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c09bd10 con 0x7f156c099b10 2019-10-13T00:51:58.407 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> v2:172.21.15.155:3300/0 conn(0x7f156c099b10 0x7f156c099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f155c00fab0 tx=0x7f155c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.407 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> v2:172.21.15.155:3300/0 conn(0x7f156c099b10 0x7f156c099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f155c00fab0 tx=0x7f155c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.407 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.403+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c099b10 0x7f156c099f00 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f155c00fab0 tx=0x7f155c00fe50).ready entity=mon.1 client_cookie=8b59ba60b2519be0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.407 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c099b10 0x7f156c099f00 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f155c00fab0 tx=0x7f155c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.408 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f156a7fc700 1 -- 172.21.15.155:0/3249965604 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f155c000ce0 con 0x7f156c099b10 2019-10-13T00:51:58.408 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c099b10 0x7f156c099f00 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f155c00fab0 tx=0x7f155c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.409 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c099b10 0x7f156c099f00 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f155c00fab0 tx=0x7f155c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.409 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f156a7fc700 1 -- 172.21.15.155:0/3249965604 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c001010 con 0x7f156c099b10 2019-10-13T00:51:58.409 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f156a7fc700 1 -- 172.21.15.155:0/3249965604 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f155c001760 con 0x7f156c099b10 2019-10-13T00:51:58.409 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 -- 172.21.15.155:0/3249965604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c099b10 msgr2=0x7f156c099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.409 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c099b10 0x7f156c099f00 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f155c00fab0 tx=0x7f155c00fe50).stop 2019-10-13T00:51:58.409 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 -- 172.21.15.155:0/3249965604 shutdown_connections 2019-10-13T00:51:58.409 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c09c7f0 0x7f156c09cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.410 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 --2- 172.21.15.155:0/3249965604 >> v2:172.21.15.142:3300/0 conn(0x7f156c09d620 0x7f156c09a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.410 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 --2- 172.21.15.155:0/3249965604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c099b10 0x7f156c099f00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.410 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 -- 172.21.15.155:0/3249965604 shutdown_connections 2019-10-13T00:51:58.410 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 -- 172.21.15.155:0/3249965604 wait complete. 2019-10-13T00:51:58.410 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.404+0000 7f1571467700 1 -- 172.21.15.155:0/3249965604 >> 172.21.15.155:0/3249965604 conn(0x7f156c099550 msgr2=0x7f156c061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.407+0000 7f1571467700 1 Processor -- start 2019-10-13T00:51:58.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.407+0000 7f1571467700 1 -- start start 2019-10-13T00:51:58.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1571467700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c08fe30 0x7f156c08ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1571467700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f156c092320 0x7f156c08f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1571467700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c0890d0 con 0x7f156c091730 2019-10-13T00:51:58.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1571467700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c093d40 con 0x7f156c092320 2019-10-13T00:51:58.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1571467700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c093310 con 0x7f156c08fe30 2019-10-13T00:51:58.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f156c092320 0x7f156c08f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156b7fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f156c091730 0x7f156c091b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49084/0 (socket says 172.21.15.155:49084) 2019-10-13T00:51:58.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156b7fe700 1 -- 172.21.15.155:0/3139873357 learned_addr learned my addr 172.21.15.155:0/3139873357 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:58.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156affd700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c08fe30 0x7f156c08ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 -- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c08fe30 msgr2=0x7f156c08ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c08fe30 0x7f156c08ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 -- 172.21.15.155:0/3139873357 >> v1:172.21.15.155:6789/0 conn(0x7f156c091730 legacy=0x7f156c091b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 -- 172.21.15.155:0/3139873357 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c08e560 con 0x7f156c092320 2019-10-13T00:51:58.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430357813 0 0) 0x7f155c007b90 con 0x7f156c091730 2019-10-13T00:51:58.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> v2:172.21.15.155:3300/0 conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> v2:172.21.15.155:3300/0 conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15640053b0 con 0x7f156c092320 2019-10-13T00:51:58.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15640055b0 con 0x7f156c092320 2019-10-13T00:51:58.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1564005c80 con 0x7f156c092320 2019-10-13T00:51:58.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156c08e560 con 0x7f156c092320 2019-10-13T00:51:58.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.408+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156c08e560 con 0x7f156c092320 2019-10-13T00:51:58.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.409+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.409+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f15640051c0 con 0x7f156c092320 2019-10-13T00:51:58.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.409+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.409+0000 7f1563fff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1550017c60 0x7f155001a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.409+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f156402fff0 con 0x7f156c092320 2019-10-13T00:51:58.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.409+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1550017c60 0x7f155001a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.420 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.409+0000 7f156b7fe700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1550017c60 0x7f155001a0f0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.420 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.417+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156c08e560 con 0x7f156c092320 2019-10-13T00:51:58.420 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.421+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.421 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.421+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f15640092a0 con 0x7f156c092320 2019-10-13T00:51:58.585 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.587+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f156c08e560 con 0x7f156c092320 2019-10-13T00:51:58.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.588+0000 7f156bfff700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.588+0000 7f1563fff700 1 -- 172.21.15.155:0/3139873357 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f15640051c0 con 0x7f156c092320 2019-10-13T00:51:58.587 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:51:58.587 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:51:58.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1550017c60 msgr2=0x7f155001a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1550017c60 0x7f155001a0f0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 msgr2=0x7f156c08f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f1564010380 tx=0x7f1564010720).stop 2019-10-13T00:51:58.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 shutdown_connections 2019-10-13T00:51:58.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f156c092320 0x7f156c08f3a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1550017c60 0x7f155001a0f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 --2- 172.21.15.155:0/3139873357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f156c08fe30 0x7f156c08ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 shutdown_connections 2019-10-13T00:51:58.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 wait complete. 2019-10-13T00:51:58.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.590+0000 7f1571467700 1 -- 172.21.15.155:0/3139873357 >> 172.21.15.155:0/3139873357 conn(0x7f156c06d080 msgr2=0x7f156c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.598 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:58.599 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-10-13T00:51:58.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.734+0000 7f4e62fc3700 1 Processor -- start 2019-10-13T00:51:58.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.734+0000 7f4e62fc3700 1 -- start start 2019-10-13T00:51:58.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e62fc3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4e5c099b10 0x7f4e5c099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.736 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e62fc3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.737 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e62fc3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c064f00 con 0x7f4e5c09d620 2019-10-13T00:51:58.737 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e62fc3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5c09b680 con 0x7f4e5c099b10 2019-10-13T00:51:58.737 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e62fc3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c09ac50 con 0x7f4e5c09cff0 2019-10-13T00:51:58.737 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61560700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4e5c099b10 0x7f4e5c099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.737 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61d61700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.737 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61560700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4e5c099b10 0x7f4e5c099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.155:58754/0 (socket says 172.21.15.155:58754) 2019-10-13T00:51:58.737 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61d61700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.155:40460/0 (socket says 172.21.15.155:40460) 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61560700 1 -- 172.21.15.155:0/37069673 learned_addr learned my addr 172.21.15.155:0/37069673 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e60d5f700 1 --1- 172.21.15.155:0/37069673 >> v1:172.21.15.142:6789/0 conn(0x7f4e5c09d620 0x7f4e5c09c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:57114/0 (socket says 172.21.15.155:57114) 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61d61700 1 -- 172.21.15.155:0/37069673 >> v2:172.21.15.142:3300/0 conn(0x7f4e5c099b10 msgr2=0x7f4e5c099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61d61700 1 --2- 172.21.15.155:0/37069673 >> v2:172.21.15.142:3300/0 conn(0x7f4e5c099b10 0x7f4e5c099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61d61700 1 -- 172.21.15.155:0/37069673 >> v1:172.21.15.142:6789/0 conn(0x7f4e5c09d620 legacy=0x7f4e5c09c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61d61700 1 -- 172.21.15.155:0/37069673 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c09bd10 con 0x7f4e5c09cff0 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.735+0000 7f4e61560700 1 --2- 172.21.15.155:0/37069673 >> v2:172.21.15.142:3300/0 conn(0x7f4e5c099b10 0x7f4e5c099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:58.738 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e61d61700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e5400b2f0 tx=0x7f4e5400bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.739 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e61d61700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e5400b2f0 tx=0x7f4e5400bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.739 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e61d61700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f4e5400b2f0 tx=0x7f4e5400bb00).ready entity=mon.2 client_cookie=506aeea6f709ca5c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.739 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e61d61700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f4e5400b2f0 tx=0x7f4e5400bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.739 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e61d61700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f4e5400b2f0 tx=0x7f4e5400bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.739 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e5bfff700 1 -- 172.21.15.155:0/37069673 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e540075b0 con 0x7f4e5c09cff0 2019-10-13T00:51:58.739 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e61d61700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f4e5400b2f0 tx=0x7f4e5400bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.739 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e5bfff700 1 -- 172.21.15.155:0/37069673 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e54000f30 con 0x7f4e5c09cff0 2019-10-13T00:51:58.740 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e5bfff700 1 -- 172.21.15.155:0/37069673 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e5400a1d0 con 0x7f4e5c09cff0 2019-10-13T00:51:58.740 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e62fc3700 1 -- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 msgr2=0x7f4e5c09a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.740 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f4e5400b2f0 tx=0x7f4e5400bb00).stop 2019-10-13T00:51:58.740 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e62fc3700 1 -- 172.21.15.155:0/37069673 shutdown_connections 2019-10-13T00:51:58.740 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/37069673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c09cff0 0x7f4e5c09a440 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.740 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.736+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/37069673 >> v2:172.21.15.142:3300/0 conn(0x7f4e5c099b10 0x7f4e5c099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:58.740 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.737+0000 7f4e62fc3700 1 -- 172.21.15.155:0/37069673 shutdown_connections 2019-10-13T00:51:58.741 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.737+0000 7f4e62fc3700 1 -- 172.21.15.155:0/37069673 wait complete. 2019-10-13T00:51:58.741 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.737+0000 7f4e62fc3700 1 -- 172.21.15.155:0/37069673 >> 172.21.15.155:0/37069673 conn(0x7f4e5c099550 msgr2=0x7f4e5c061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.744 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.739+0000 7f4e62fc3700 1 Processor -- start 2019-10-13T00:51:58.744 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.739+0000 7f4e62fc3700 1 -- start start 2019-10-13T00:51:58.744 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e62fc3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4e5c0926e0 0x7f4e5c091f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.744 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e62fc3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c154020 0x7f4e5c156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.744 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e62fc3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c08eac0 con 0x7f4e5c073990 2019-10-13T00:51:58.745 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e62fc3700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5c090780 con 0x7f4e5c0926e0 2019-10-13T00:51:58.745 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e62fc3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c08fd50 con 0x7f4e5c154020 2019-10-13T00:51:58.745 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4e5c0926e0 0x7f4e5c091f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.745 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e61560700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4e5c073990 0x7f4e5c091710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49100/0 (socket says 172.21.15.155:49100) 2019-10-13T00:51:58.745 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e61560700 1 -- 172.21.15.155:0/1454270766 learned_addr learned my addr 172.21.15.155:0/1454270766 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:58.745 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e61d61700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c154020 0x7f4e5c156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 -- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c154020 msgr2=0x7f4e5c156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c154020 0x7f4e5c156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 -- 172.21.15.155:0/1454270766 >> v1:172.21.15.155:6789/0 conn(0x7f4e5c073990 legacy=0x7f4e5c091710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307487359 0 0) 0x7f4e4c000c90 con 0x7f4e5c073990 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 -- 172.21.15.155:0/1454270766 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c090e10 con 0x7f4e5c0926e0 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e61d61700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e5c154020 0x7f4e5c156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> v2:172.21.15.155:3300/0 conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.746 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> v2:172.21.15.155:3300/0 conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e50001260 con 0x7f4e5c0926e0 2019-10-13T00:51:58.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e500041b0 con 0x7f4e5c0926e0 2019-10-13T00:51:58.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e50004880 con 0x7f4e5c0926e0 2019-10-13T00:51:58.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c090e10 con 0x7f4e5c0926e0 2019-10-13T00:51:58.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.740+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c090e10 con 0x7f4e5c0926e0 2019-10-13T00:51:58.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.741+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.741+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4e50000e20 con 0x7f4e5c0926e0 2019-10-13T00:51:58.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.741+0000 7f4e60d5f700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.741+0000 7f4e59ffb700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4e3c017c60 0x7f4e3c01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.741+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4e50030240 con 0x7f4e5c0926e0 2019-10-13T00:51:58.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.741+0000 7f4e61560700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4e3c017c60 0x7f4e3c01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.741+0000 7f4e61560700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4e3c017c60 0x7f4e3c01a0f0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.749+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:6800/7109,v1:172.21.15.155:6801/7109] conn(0x7f4e5c0537f0 0x7f4e5c053be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.749+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 --> [v2:172.21.15.155:6800/7109,v1:172.21.15.155:6801/7109] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e5c064f00 con 0x7f4e5c0537f0 2019-10-13T00:51:58.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.749+0000 7f4e61d61700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:6800/7109,v1:172.21.15.155:6801/7109] conn(0x7f4e5c0537f0 0x7f4e5c053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.749+0000 7f4e61d61700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:6800/7109,v1:172.21.15.155:6801/7109] conn(0x7f4e5c0537f0 0x7f4e5c053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.750+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== osd.0 v2:172.21.15.155:6800/7109 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f4e5400b270 con 0x7f4e5c0537f0 2019-10-13T00:51:58.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.770+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 --> [v2:172.21.15.155:6800/7109,v1:172.21.15.155:6801/7109] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e5c064f00 con 0x7f4e5c0537f0 2019-10-13T00:51:58.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.770+0000 7f4e59ffb700 1 -- 172.21.15.155:0/1454270766 <== osd.0 v2:172.21.15.155:6800/7109 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4e54000df0 con 0x7f4e5c0537f0 2019-10-13T00:51:58.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:6800/7109,v1:172.21.15.155:6801/7109] conn(0x7f4e5c0537f0 msgr2=0x7f4e5c053be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:6800/7109,v1:172.21.15.155:6801/7109] conn(0x7f4e5c0537f0 0x7f4e5c053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4e3c017c60 msgr2=0x7f4e3c01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4e3c017c60 0x7f4e3c01a0f0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 msgr2=0x7f4e5c091f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f4e500103a0 tx=0x7f4e50010740).stop 2019-10-13T00:51:58.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e61d61700 1 -- 172.21.15.155:0/1454270766 reap_dead start 2019-10-13T00:51:58.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 shutdown_connections 2019-10-13T00:51:58.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 --2- 172.21.15.155:0/1454270766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e5c0926e0 0x7f4e5c091f10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 shutdown_connections 2019-10-13T00:51:58.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 wait complete. 2019-10-13T00:51:58.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.772+0000 7f4e62fc3700 1 -- 172.21.15.155:0/1454270766 >> 172.21.15.155:0/1454270766 conn(0x7f4e5c06d080 msgr2=0x7f4e5c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.779 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:58.779 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:58.780 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-10-13T00:51:58.905 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.903+0000 7f5b5bc12700 1 Processor -- start 2019-10-13T00:51:58.905 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.903+0000 7f5b5bc12700 1 -- start start 2019-10-13T00:51:58.905 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5bc12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:58.906 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5bc12700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b54064f00 con 0x7f5b54099b10 2019-10-13T00:51:58.906 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5bc12700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b540987e0 con 0x7f5b5409a700 2019-10-13T00:51:58.906 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5bc12700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5409b680 con 0x7f5b5409d620 2019-10-13T00:51:58.906 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b599ae700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5b5409a700 0x7f5b5409c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49114/0 (socket says 172.21.15.155:49114) 2019-10-13T00:51:58.906 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b599ae700 1 -- 172.21.15.155:0/2203451866 learned_addr learned my addr 172.21.15.155:0/2203451866 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:58.906 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.906 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b591ad700 1 -- 172.21.15.155:0/2203451866 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284412570 0 0) 0x7f5b48001150 con 0x7f5b5409a700 2019-10-13T00:51:58.907 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b591ad700 1 -- 172.21.15.155:0/2203451866 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b3c002e70 con 0x7f5b5409a700 2019-10-13T00:51:58.907 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b591ad700 1 -- 172.21.15.155:0/2203451866 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711849858 0 0) 0x7f5b44001400 con 0x7f5b54099b10 2019-10-13T00:51:58.907 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b591ad700 1 -- 172.21.15.155:0/2203451866 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b3c004160 con 0x7f5b54099b10 2019-10-13T00:51:58.907 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5a9b0700 1 -- 172.21.15.155:0/2203451866 >> v1:172.21.15.155:6789/0 conn(0x7f5b5409a700 legacy=0x7f5b5409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.907 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5a9b0700 1 -- 172.21.15.155:0/2203451866 >> v1:172.21.15.142:6789/0 conn(0x7f5b54099b10 legacy=0x7f5b54099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.907 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5a9b0700 1 -- 172.21.15.155:0/2203451866 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5409bd10 con 0x7f5b5409d620 2019-10-13T00:51:58.907 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5b4c00af70 tx=0x7f5b4c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.908 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5b4c00af70 tx=0x7f5b4c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.908 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.904+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f5b4c00af70 tx=0x7f5b4c00b780).ready entity=mon.2 client_cookie=fda6f24bc042c44a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.908 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f5b4c00af70 tx=0x7f5b4c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.908 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f5b4c00af70 tx=0x7f5b4c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.908 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b591ad700 1 -- 172.21.15.155:0/2203451866 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b4c007230 con 0x7f5b5409d620 2019-10-13T00:51:58.908 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f5b4c00af70 tx=0x7f5b4c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.908 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b591ad700 1 -- 172.21.15.155:0/2203451866 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c000bb0 con 0x7f5b5409d620 2019-10-13T00:51:58.909 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b591ad700 1 -- 172.21.15.155:0/2203451866 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b4c009e50 con 0x7f5b5409d620 2019-10-13T00:51:58.909 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5bc12700 1 -- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 msgr2=0x7f5b54153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.909 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5bc12700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f5b4c00af70 tx=0x7f5b4c00b780).stop 2019-10-13T00:51:58.909 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5bc12700 1 -- 172.21.15.155:0/2203451866 shutdown_connections 2019-10-13T00:51:58.909 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5bc12700 1 --2- 172.21.15.155:0/2203451866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b5409d620 0x7f5b54153c20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.909 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5bc12700 1 -- 172.21.15.155:0/2203451866 shutdown_connections 2019-10-13T00:51:58.909 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5bc12700 1 -- 172.21.15.155:0/2203451866 wait complete. 2019-10-13T00:51:58.910 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.905+0000 7f5b5bc12700 1 -- 172.21.15.155:0/2203451866 >> 172.21.15.155:0/2203451866 conn(0x7f5b54099550 msgr2=0x7f5b54061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b5bc12700 1 Processor -- start 2019-10-13T00:51:58.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b5bc12700 1 -- start start 2019-10-13T00:51:58.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b5bc12700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b5414bbe0 0x7f5b54073770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b5bc12700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b541367e0 con 0x7f5b5414b7f0 2019-10-13T00:51:58.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b5bc12700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b54070fd0 con 0x7f5b5408c8b0 2019-10-13T00:51:58.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b5bc12700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b540727e0 con 0x7f5b5414bbe0 2019-10-13T00:51:58.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b599ae700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5b5408c8b0 0x7f5b54070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49118/0 (socket says 172.21.15.155:49118) 2019-10-13T00:51:58.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b599ae700 1 -- 172.21.15.155:0/1800705061 learned_addr learned my addr 172.21.15.155:0/1800705061 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:58.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.908+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> v2:172.21.15.142:3300/0 conn(0x7f5b5414bbe0 0x7f5b54073770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549807830 0 0) 0x7f5b48000940 con 0x7f5b5408c8b0 2019-10-13T00:51:58.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b34002e70 con 0x7f5b5408c8b0 2019-10-13T00:51:58.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 -- 172.21.15.155:0/1800705061 >> v1:172.21.15.155:6789/0 conn(0x7f5b5408c8b0 legacy=0x7f5b54070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469904854 0 0) 0x7f5b440015c0 con 0x7f5b5414b7f0 2019-10-13T00:51:58.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 -- 172.21.15.155:0/1800705061 >> v1:172.21.15.142:6789/0 conn(0x7f5b5414b7f0 legacy=0x7f5b5406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 -- 172.21.15.155:0/1800705061 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54072e70 con 0x7f5b5414bbe0 2019-10-13T00:51:58.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> v2:172.21.15.142:3300/0 conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> v2:172.21.15.142:3300/0 conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b4c00ae80 con 0x7f5b5414bbe0 2019-10-13T00:51:58.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c0099a0 con 0x7f5b5414bbe0 2019-10-13T00:51:58.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b4c009b90 con 0x7f5b5414bbe0 2019-10-13T00:51:58.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5414bfd0 con 0x7f5b5414bbe0 2019-10-13T00:51:58.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.909+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b54072e70 con 0x7f5b5414bbe0 2019-10-13T00:51:58.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.910+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.917 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.910+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5b4c0095a0 con 0x7f5b5414bbe0 2019-10-13T00:51:58.917 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.910+0000 7f5b5a9b0700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:58.917 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.910+0000 7f5b52ffd700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5b34017cb0 0x7f5b3401a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.917 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.910+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5b4c02ff40 con 0x7f5b5414bbe0 2019-10-13T00:51:58.917 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.910+0000 7f5b5a1af700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5b34017cb0 0x7f5b3401a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.917 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.910+0000 7f5b5a1af700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5b34017cb0 0x7f5b3401a120 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.918 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.918+0000 7f5b5bc12700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.155:6808/7110,v1:172.21.15.155:6809/7110] conn(0x7f5b54153500 0x7f5b541538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:58.918 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.918+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 --> [v2:172.21.15.155:6808/7110,v1:172.21.15.155:6809/7110] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b541363c0 con 0x7f5b54153500 2019-10-13T00:51:58.918 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.918+0000 7f5b599ae700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.155:6808/7110,v1:172.21.15.155:6809/7110] conn(0x7f5b54153500 0x7f5b541538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:58.918 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.918+0000 7f5b599ae700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.155:6808/7110,v1:172.21.15.155:6809/7110] conn(0x7f5b54153500 0x7f5b541538f0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:58.919 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.919+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== osd.1 v2:172.21.15.155:6808/7110 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f5b48001bc0 con 0x7f5b54153500 2019-10-13T00:51:58.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.940+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 --> [v2:172.21.15.155:6808/7110,v1:172.21.15.155:6809/7110] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5b54137090 con 0x7f5b54153500 2019-10-13T00:51:58.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.940+0000 7f5b52ffd700 1 -- 172.21.15.155:0/1800705061 <== osd.1 v2:172.21.15.155:6808/7110 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5b48000f10 con 0x7f5b54153500 2019-10-13T00:51:58.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 >> [v2:172.21.15.155:6808/7110,v1:172.21.15.155:6809/7110] conn(0x7f5b54153500 msgr2=0x7f5b541538f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.155:6808/7110,v1:172.21.15.155:6809/7110] conn(0x7f5b54153500 0x7f5b541538f0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5b34017cb0 msgr2=0x7f5b3401a120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5b34017cb0 0x7f5b3401a120 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 msgr2=0x7f5b54073770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:58.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f5b4c0103a0 tx=0x7f5b4c010740).stop 2019-10-13T00:51:58.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5a9b0700 1 -- 172.21.15.155:0/1800705061 reap_dead start 2019-10-13T00:51:58.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 shutdown_connections 2019-10-13T00:51:58.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.941+0000 7f5b5bc12700 1 --2- 172.21.15.155:0/1800705061 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b5414bbe0 0x7f5b54073770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:58.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.942+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 shutdown_connections 2019-10-13T00:51:58.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.942+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 wait complete. 2019-10-13T00:51:58.942 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:58.942+0000 7f5b5bc12700 1 -- 172.21.15.155:0/1800705061 >> 172.21.15.155:0/1800705061 conn(0x7f5b5406c410 msgr2=0x7f5b54074840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:58.949 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:58.949 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:58.949 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-10-13T00:51:59.075 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.074+0000 7fb4c68b1700 1 Processor -- start 2019-10-13T00:51:59.076 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.074+0000 7fb4c68b1700 1 -- start start 2019-10-13T00:51:59.076 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.074+0000 7fb4c68b1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4c0099b10 0x7fb4c0099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.076 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c68b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4c009d620 0x7fb4c009a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.076 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c68b1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c0064f00 con 0x7fb4c009c7f0 2019-10-13T00:51:59.076 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c68b1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c009b680 con 0x7fb4c0099b10 2019-10-13T00:51:59.076 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c68b1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c009ac50 con 0x7fb4c009d620 2019-10-13T00:51:59.077 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4c0099b10 0x7fb4c0099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.077 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4c0099b10 0x7fb4c0099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59050/0 (socket says 172.21.15.155:59050) 2019-10-13T00:51:59.077 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 -- 172.21.15.155:0/2020105245 learned_addr learned my addr 172.21.15.155:0/2020105245 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:59.077 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c564f700 1 --1- 172.21.15.155:0/2020105245 >> v1:172.21.15.155:6789/0 conn(0x7fb4c009c7f0 0x7fb4c009ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49126/0 (socket says 172.21.15.155:49126) 2019-10-13T00:51:59.077 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4c009d620 0x7fb4c009a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.077 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 -- 172.21.15.155:0/2020105245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4c009d620 msgr2=0x7fb4c009a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.077 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4c009d620 0x7fb4c009a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 -- 172.21.15.155:0/2020105245 >> v1:172.21.15.155:6789/0 conn(0x7fb4c009c7f0 legacy=0x7fb4c009ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 -- 172.21.15.155:0/2020105245 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c009bd10 con 0x7fb4c0099b10 2019-10-13T00:51:59.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4bf7fe700 1 -- 172.21.15.155:0/2020105245 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006681025 0 0) 0x7fb4b8000d30 con 0x7fb4c009c7f0 2019-10-13T00:51:59.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2020105245 >> v2:172.21.15.155:3300/0 conn(0x7fb4c0099b10 0x7fb4c0099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb4a800f730 tx=0x7fb4a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2020105245 >> v2:172.21.15.155:3300/0 conn(0x7fb4c0099b10 0x7fb4c0099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb4a800f730 tx=0x7fb4a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0099b10 0x7fb4c0099f00 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7fb4a800f730 tx=0x7fb4a800fad0).ready entity=mon.1 client_cookie=7f1a87d1d12d8353 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0099b10 0x7fb4c0099f00 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7fb4a800f730 tx=0x7fb4a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4bf7fe700 1 -- 172.21.15.155:0/2020105245 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4a8000960 con 0x7fb4c0099b10 2019-10-13T00:51:59.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0099b10 0x7fb4c0099f00 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7fb4a800f730 tx=0x7fb4a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0099b10 0x7fb4c0099f00 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7fb4a800f730 tx=0x7fb4a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4bf7fe700 1 -- 172.21.15.155:0/2020105245 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a8000c90 con 0x7fb4c0099b10 2019-10-13T00:51:59.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4bf7fe700 1 -- 172.21.15.155:0/2020105245 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4a8007200 con 0x7fb4c0099b10 2019-10-13T00:51:59.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2020105245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0099b10 msgr2=0x7fb4c0099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.075+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0099b10 0x7fb4c0099f00 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7fb4a800f730 tx=0x7fb4a800fad0).stop 2019-10-13T00:51:59.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.076+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2020105245 shutdown_connections 2019-10-13T00:51:59.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.076+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4c009d620 0x7fb4c009a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.076+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2020105245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0099b10 0x7fb4c0099f00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.076+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2020105245 shutdown_connections 2019-10-13T00:51:59.088 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.076+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2020105245 wait complete. 2019-10-13T00:51:59.088 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.076+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2020105245 >> 172.21.15.155:0/2020105245 conn(0x7fb4c0099550 msgr2=0x7fb4c0061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.092 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.078+0000 7fb4c68b1700 1 Processor -- start 2019-10-13T00:51:59.092 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.078+0000 7fb4c68b1700 1 -- start start 2019-10-13T00:51:59.092 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4c68b1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4c0071f00 0x7fb4c0071730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.092 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4c68b1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb4c01540b0 0x7fb4c01564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.093 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4c68b1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c0129ff0 con 0x7fb4c012b450 2019-10-13T00:51:59.093 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4c68b1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c0149c60 con 0x7fb4c01540b0 2019-10-13T00:51:59.093 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4c68b1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c006ffa0 con 0x7fb4c0071f00 2019-10-13T00:51:59.093 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4bffff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4c0071f00 0x7fb4c0071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.093 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4bffff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4c0071f00 0x7fb4c0071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59054/0 (socket says 172.21.15.155:59054) 2019-10-13T00:51:59.093 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4bffff700 1 -- 172.21.15.155:0/2013538712 learned_addr learned my addr 172.21.15.155:0/2013538712 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:51:59.093 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4c4e4e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb4c012b450 0x7fb4c0070f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49134/0 (socket says 172.21.15.155:49134) 2019-10-13T00:51:59.094 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4c564f700 1 --2- 172.21.15.155:0/2013538712 >> v2:172.21.15.142:3300/0 conn(0x7fb4c01540b0 0x7fb4c01564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.094 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 -- 172.21.15.155:0/2013538712 >> v2:172.21.15.142:3300/0 conn(0x7fb4c01540b0 msgr2=0x7fb4c01564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.094 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> v2:172.21.15.142:3300/0 conn(0x7fb4c01540b0 0x7fb4c01564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.094 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.079+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054135711 0 0) 0x7fb4a80075c0 con 0x7fb4c012b450 2019-10-13T00:51:59.094 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 -- 172.21.15.155:0/2013538712 >> v1:172.21.15.155:6789/0 conn(0x7fb4c012b450 legacy=0x7fb4c0070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.094 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 -- 172.21.15.155:0/2013538712 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c0070630 con 0x7fb4c0071f00 2019-10-13T00:51:59.094 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> v2:172.21.15.155:3300/0 conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.095 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> v2:172.21.15.155:3300/0 conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.095 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.095 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.095 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4b0000940 con 0x7fb4c0071f00 2019-10-13T00:51:59.095 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.096 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.096 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c0070630 con 0x7fb4c0071f00 2019-10-13T00:51:59.096 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b0007200 con 0x7fb4c0071f00 2019-10-13T00:51:59.096 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4b00088d0 con 0x7fb4c0071f00 2019-10-13T00:51:59.096 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c0070630 con 0x7fb4c0071f00 2019-10-13T00:51:59.096 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.080+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.081+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb4b00084f0 con 0x7fb4c0071f00 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.081+0000 7fb4bffff700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.081+0000 7fb4bd7fa700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb4a4017c60 0x7fb4a401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.081+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb4b00300b0 con 0x7fb4c0071f00 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.081+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb4a4017c60 0x7fb4a401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.081+0000 7fb4c4e4e700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb4a4017c60 0x7fb4a401a0f0 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.089+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:6817/7112,v1:172.21.15.155:6819/7112] conn(0x7fb4c00537f0 0x7fb4c0053be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.097 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.089+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 --> [v2:172.21.15.155:6817/7112,v1:172.21.15.155:6819/7112] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4c0064f00 con 0x7fb4c00537f0 2019-10-13T00:51:59.098 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.089+0000 7fb4c564f700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:6817/7112,v1:172.21.15.155:6819/7112] conn(0x7fb4c00537f0 0x7fb4c0053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.098 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.089+0000 7fb4c564f700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:6817/7112,v1:172.21.15.155:6819/7112] conn(0x7fb4c00537f0 0x7fb4c0053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.098 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.090+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== osd.2 v2:172.21.15.155:6817/7112 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fb4b8000ce0 con 0x7fb4c00537f0 2019-10-13T00:51:59.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.110+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 --> [v2:172.21.15.155:6817/7112,v1:172.21.15.155:6819/7112] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb4c0064f00 con 0x7fb4c00537f0 2019-10-13T00:51:59.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.110+0000 7fb4bd7fa700 1 -- 172.21.15.155:0/2013538712 <== osd.2 v2:172.21.15.155:6817/7112 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb4b8007880 con 0x7fb4c00537f0 2019-10-13T00:51:59.110 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.111+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:6817/7112,v1:172.21.15.155:6819/7112] conn(0x7fb4c00537f0 msgr2=0x7fb4c0053be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.110 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.111+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:6817/7112,v1:172.21.15.155:6819/7112] conn(0x7fb4c00537f0 0x7fb4c0053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.110 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb4a4017c60 msgr2=0x7fb4a401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.110 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb4a4017c60 0x7fb4a401a0f0 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 msgr2=0x7fb4c0071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7fb4b0010440 tx=0x7fb4b00107e0).stop 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c564f700 1 -- 172.21.15.155:0/2013538712 reap_dead start 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 shutdown_connections 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 --2- 172.21.15.155:0/2013538712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4c0071f00 0x7fb4c0071730 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 shutdown_connections 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 wait complete. 2019-10-13T00:51:59.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.112+0000 7fb4c68b1700 1 -- 172.21.15.155:0/2013538712 >> 172.21.15.155:0/2013538712 conn(0x7fb4c008a800 msgr2=0x7fb4c006e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.119 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:59.119 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:59.119 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-10-13T00:51:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.243+0000 7fa6b7da2700 1 Processor -- start 2019-10-13T00:51:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.243+0000 7fa6b7da2700 1 -- start start 2019-10-13T00:51:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b7da2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa6b009a700 0x7fa6b009c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b7da2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b009d620 0x7fa6b009cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b7da2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b0064f00 con 0x7fa6b0099b10 2019-10-13T00:51:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b7da2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b009b680 con 0x7fa6b009a700 2019-10-13T00:51:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b7da2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b009ac50 con 0x7fa6b009d620 2019-10-13T00:51:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b633f700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa6b0099b10 0x7fa6b0099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49140/0 (socket says 172.21.15.155:49140) 2019-10-13T00:51:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b633f700 1 -- 172.21.15.155:0/1104688382 learned_addr learned my addr 172.21.15.155:0/1104688382 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b5b3e700 1 --2- 172.21.15.155:0/1104688382 >> v2:172.21.15.142:3300/0 conn(0x7fa6b009a700 0x7fa6b009c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b009d620 0x7fa6b009cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866905665 0 0) 0x7fa6980014d0 con 0x7fa6b0099b10 2019-10-13T00:51:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69c002e70 con 0x7fa6b0099b10 2019-10-13T00:51:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2293343401 0 0) 0x7fa6980014b0 con 0x7fa6b0099b10 2019-10-13T00:51:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b009d620 msgr2=0x7fa6b009cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b533d700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b009d620 0x7fa6b009cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 >> v2:172.21.15.142:3300/0 conn(0x7fa6b009a700 msgr2=0x7fa6b009c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.244+0000 7fa6b533d700 1 --2- 172.21.15.155:0/1104688382 >> v2:172.21.15.142:3300/0 conn(0x7fa6b009a700 0x7fa6b009c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b009bd10 con 0x7fa6b0099b10 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7fa6980019e0 con 0x7fa6b0099b10 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 >> v1:172.21.15.155:6789/0 conn(0x7fa6b0099b10 legacy=0x7fa6b0099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa69c047b20 0x7fa69c049f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa69c04a650 0x7fa69c04cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c04ffe0 con 0x7fa69c04a650 2019-10-13T00:51:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c0501d0 con 0x7fa69c04d300 2019-10-13T00:51:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c050400 con 0x7fa69c047b20 2019-10-13T00:51:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b633f700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa69c047b20 0x7fa69c049f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b5b3e700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa69c04a650 0x7fa69c04cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 -- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa69c047b20 msgr2=0x7fa69c049f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa69c047b20 0x7fa69c049f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 -- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa69c04a650 msgr2=0x7fa69c04cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa69c04a650 0x7fa69c04cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 -- 172.21.15.155:0/1104688382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0506b0 con 0x7fa69c04d300 2019-10-13T00:51:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa6a8002e20 tx=0x7fa6a800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa6a8002e20 tx=0x7fa6a800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7fa6a8002e20 tx=0x7fa6a800fa00).ready entity=mon.1 client_cookie=47dbf6ef54f2fe64 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fa6a8002e20 tx=0x7fa6a800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a8000b10 con 0x7fa69c04d300 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fa6a8002e20 tx=0x7fa6a800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fa6a8002e20 tx=0x7fa6a800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8001060 con 0x7fa69c04d300 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b533d700 1 -- 172.21.15.155:0/1104688382 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a8001730 con 0x7fa69c04d300 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b7da2700 1 -- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 msgr2=0x7fa69c04f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b7da2700 1 --2- 172.21.15.155:0/1104688382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa69c04d300 0x7fa69c04f7d0 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7fa6a8002e20 tx=0x7fa6a800fa00).stop 2019-10-13T00:51:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.245+0000 7fa6b6b40700 1 -- 172.21.15.155:0/1104688382 reap_dead start 2019-10-13T00:51:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.246+0000 7fa6b7da2700 1 -- 172.21.15.155:0/1104688382 shutdown_connections 2019-10-13T00:51:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.246+0000 7fa6b7da2700 1 -- 172.21.15.155:0/1104688382 shutdown_connections 2019-10-13T00:51:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.246+0000 7fa6b7da2700 1 -- 172.21.15.155:0/1104688382 wait complete. 2019-10-13T00:51:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.246+0000 7fa6b7da2700 1 -- 172.21.15.155:0/1104688382 >> 172.21.15.155:0/1104688382 conn(0x7fa6b0099550 msgr2=0x7fa6b0061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.248+0000 7fa6b7da2700 1 Processor -- start 2019-10-13T00:51:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.248+0000 7fa6b7da2700 1 -- start start 2019-10-13T00:51:59.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b7da2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa6b008e8c0 0x7fa6b008ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b7da2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa6b00934b0 0x7fa6b0092ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b7da2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b008fc80 con 0x7fa6b0089090 2019-10-13T00:51:59.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b7da2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b0091550 con 0x7fa6b008e8c0 2019-10-13T00:51:59.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b7da2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b0090b20 con 0x7fa6b00934b0 2019-10-13T00:51:59.258 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b5b3e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa6b0089090 0x7fa6b00924e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49154/0 (socket says 172.21.15.155:49154) 2019-10-13T00:51:59.258 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b5b3e700 1 -- 172.21.15.155:0/537228946 learned_addr learned my addr 172.21.15.155:0/537228946 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.258 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b633f700 1 --2- 172.21.15.155:0/537228946 >> v2:172.21.15.142:3300/0 conn(0x7fa6b008e8c0 0x7fa6b008ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.258 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> v2:172.21.15.155:3300/0 conn(0x7fa6b00934b0 0x7fa6b0092ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.258 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522435927 0 0) 0x7fa6a0009990 con 0x7fa6b0089090 2019-10-13T00:51:59.258 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.249+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa694002e70 con 0x7fa6b0089090 2019-10-13T00:51:59.258 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 -- 172.21.15.155:0/537228946 >> v2:172.21.15.142:3300/0 conn(0x7fa6b008e8c0 msgr2=0x7fa6b008ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> v2:172.21.15.142:3300/0 conn(0x7fa6b008e8c0 0x7fa6b008ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 -- 172.21.15.155:0/537228946 >> v1:172.21.15.155:6789/0 conn(0x7fa6b0089090 legacy=0x7fa6b00924e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 -- 172.21.15.155:0/537228946 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0091be0 con 0x7fa6b00934b0 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b633f700 1 --2- 172.21.15.155:0/537228946 >> v2:172.21.15.142:3300/0 conn(0x7fa6b008e8c0 0x7fa6b008ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> v2:172.21.15.155:3300/0 conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> v2:172.21.15.155:3300/0 conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a8002320 con 0x7fa6b00934b0 2019-10-13T00:51:59.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8000a70 con 0x7fa6b00934b0 2019-10-13T00:51:59.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a8001100 con 0x7fa6b00934b0 2019-10-13T00:51:59.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b0091be0 con 0x7fa6b00934b0 2019-10-13T00:51:59.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b0091be0 con 0x7fa6b00934b0 2019-10-13T00:51:59.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.250+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.251+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa6a8002680 con 0x7fa6b00934b0 2019-10-13T00:51:59.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.251+0000 7fa6b6b40700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.251+0000 7fa6aeffd700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fa694017ca0 0x7fa69401a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.251+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa6a802fe90 con 0x7fa6b00934b0 2019-10-13T00:51:59.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.251+0000 7fa6b633f700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fa694017ca0 0x7fa69401a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.251+0000 7fa6b633f700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fa694017ca0 0x7fa69401a110 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.258+0000 7fa6b7da2700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:6816/7111,v1:172.21.15.155:6818/7111] conn(0x7fa6b00537f0 0x7fa6b0053be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.258+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 --> [v2:172.21.15.155:6816/7111,v1:172.21.15.155:6818/7111] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b0064f00 con 0x7fa6b00537f0 2019-10-13T00:51:59.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.258+0000 7fa6b5b3e700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:6816/7111,v1:172.21.15.155:6818/7111] conn(0x7fa6b00537f0 0x7fa6b0053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.259+0000 7fa6b5b3e700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:6816/7111,v1:172.21.15.155:6818/7111] conn(0x7fa6b00537f0 0x7fa6b0053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.259+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== osd.3 v2:172.21.15.155:6816/7111 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fa6a0009570 con 0x7fa6b00537f0 2019-10-13T00:51:59.278 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.280+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 --> [v2:172.21.15.155:6816/7111,v1:172.21.15.155:6818/7111] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6b0064f00 con 0x7fa6b00537f0 2019-10-13T00:51:59.278 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.280+0000 7fa6aeffd700 1 -- 172.21.15.155:0/537228946 <== osd.3 v2:172.21.15.155:6816/7111 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa6a0009960 con 0x7fa6b00537f0 2019-10-13T00:51:59.280 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:6816/7111,v1:172.21.15.155:6818/7111] conn(0x7fa6b00537f0 msgr2=0x7fa6b0053be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.280 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:6816/7111,v1:172.21.15.155:6818/7111] conn(0x7fa6b00537f0 0x7fa6b0053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.281 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fa694017ca0 msgr2=0x7fa69401a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.281 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fa694017ca0 0x7fa69401a110 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.281 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 msgr2=0x7fa6b0092ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.281 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7fa6a8010390 tx=0x7fa6a8010730).stop 2019-10-13T00:51:59.281 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b6b40700 1 -- 172.21.15.155:0/537228946 reap_dead start 2019-10-13T00:51:59.281 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 shutdown_connections 2019-10-13T00:51:59.282 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 --2- 172.21.15.155:0/537228946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6b00934b0 0x7fa6b0092ce0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.282 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 shutdown_connections 2019-10-13T00:51:59.282 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 wait complete. 2019-10-13T00:51:59.282 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.282+0000 7fa6b7da2700 1 -- 172.21.15.155:0/537228946 >> 172.21.15.155:0/537228946 conn(0x7fa6b006bfe0 msgr2=0x7fa6b006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.289 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:59.289 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:59.290 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-10-13T00:51:59.415 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.413+0000 7f268cbac700 1 Processor -- start 2019-10-13T00:51:59.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.413+0000 7f268cbac700 1 -- start start 2019-10-13T00:51:59.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.414+0000 7f268cbac700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f268809a700 0x7f268809c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.414+0000 7f268cbac700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2688064f00 con 0x7f268809d620 2019-10-13T00:51:59.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.414+0000 7f268cbac700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26880987e0 con 0x7f2688099b10 2019-10-13T00:51:59.416 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.414+0000 7f268cbac700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268809b680 con 0x7f268809a700 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.414+0000 7f2686d9d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2688099b10 0x7f2688099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49162/0 (socket says 172.21.15.155:49162) 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.414+0000 7f2686d9d700 1 -- 172.21.15.155:0/1081044498 learned_addr learned my addr 172.21.15.155:0/1081044498 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.414+0000 7f268659c700 1 --2- 172.21.15.155:0/1081044498 >> v2:172.21.15.142:3300/0 conn(0x7f268809a700 0x7f268809c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f2685d9b700 1 -- 172.21.15.155:0/1081044498 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235143726 0 0) 0x7f26700014d0 con 0x7f2688099b10 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f2685d9b700 1 -- 172.21.15.155:0/1081044498 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2668002e70 con 0x7f2688099b10 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f2685d9b700 1 -- 172.21.15.155:0/1081044498 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326145636 0 0) 0x7f26800009b0 con 0x7f268809d620 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f2685d9b700 1 -- 172.21.15.155:0/1081044498 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26680030a0 con 0x7f268809d620 2019-10-13T00:51:59.417 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 -- 172.21.15.155:0/1081044498 >> v1:172.21.15.155:6789/0 conn(0x7f2688099b10 legacy=0x7f2688099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 -- 172.21.15.155:0/1081044498 >> v1:172.21.15.142:6789/0 conn(0x7f268809d620 legacy=0x7f2688152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 -- 172.21.15.155:0/1081044498 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268809bd10 con 0x7f268809a700 2019-10-13T00:51:59.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 --2- 172.21.15.155:0/1081044498 >> v2:172.21.15.142:3300/0 conn(0x7f268809a700 0x7f268809c7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f267800af40 tx=0x7f267800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 --2- 172.21.15.155:0/1081044498 >> v2:172.21.15.142:3300/0 conn(0x7f268809a700 0x7f268809c7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f267800af40 tx=0x7f267800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 --2- 172.21.15.155:0/1081044498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268809a700 0x7f268809c7f0 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f267800af40 tx=0x7f267800b750).ready entity=mon.0 client_cookie=6c55812b91d25071 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 --2- 172.21.15.155:0/1081044498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268809a700 0x7f268809c7f0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f267800af40 tx=0x7f267800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.418 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f2685d9b700 1 -- 172.21.15.155:0/1081044498 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2678000960 con 0x7f268809a700 2019-10-13T00:51:59.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 --2- 172.21.15.155:0/1081044498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268809a700 0x7f268809c7f0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f267800af40 tx=0x7f267800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268659c700 1 --2- 172.21.15.155:0/1081044498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268809a700 0x7f268809c7f0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f267800af40 tx=0x7f267800b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f2685d9b700 1 -- 172.21.15.155:0/1081044498 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2678009850 con 0x7f268809a700 2019-10-13T00:51:59.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f2685d9b700 1 -- 172.21.15.155:0/1081044498 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2678009f20 con 0x7f268809a700 2019-10-13T00:51:59.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268cbac700 1 -- 172.21.15.155:0/1081044498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268809a700 msgr2=0x7f268809c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.415+0000 7f268cbac700 1 --2- 172.21.15.155:0/1081044498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268809a700 0x7f268809c7f0 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f267800af40 tx=0x7f267800b750).stop 2019-10-13T00:51:59.419 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.416+0000 7f268cbac700 1 -- 172.21.15.155:0/1081044498 shutdown_connections 2019-10-13T00:51:59.420 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.416+0000 7f268cbac700 1 --2- 172.21.15.155:0/1081044498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268809a700 0x7f268809c7f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.420 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.416+0000 7f268cbac700 1 -- 172.21.15.155:0/1081044498 shutdown_connections 2019-10-13T00:51:59.420 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.416+0000 7f268cbac700 1 -- 172.21.15.155:0/1081044498 wait complete. 2019-10-13T00:51:59.420 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.416+0000 7f268cbac700 1 -- 172.21.15.155:0/1081044498 >> 172.21.15.155:0/1081044498 conn(0x7f2688099550 msgr2=0x7f2688061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.423 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.418+0000 7f268cbac700 1 Processor -- start 2019-10-13T00:51:59.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.418+0000 7f268cbac700 1 -- start start 2019-10-13T00:51:59.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268cbac700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f268808e800 0x7f268808e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268cbac700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2688198430 0x7f268819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268cbac700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26880894a0 con 0x7f2688088dd0 2019-10-13T00:51:59.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268cbac700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2688072e90 con 0x7f268808e800 2019-10-13T00:51:59.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268cbac700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2688072460 con 0x7f2688198430 2019-10-13T00:51:59.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f2686d9d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2688088dd0 0x7f268808d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49168/0 (socket says 172.21.15.155:49168) 2019-10-13T00:51:59.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f2686d9d700 1 -- 172.21.15.155:0/849520413 learned_addr learned my addr 172.21.15.155:0/849520413 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> v2:172.21.15.142:3300/0 conn(0x7f268808e800 0x7f268808e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268759e700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2688198430 0x7f268819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125703667 0 0) 0x7f26700012c0 con 0x7f2688088dd0 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2660002e70 con 0x7f2688088dd0 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 -- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2688198430 msgr2=0x7f268819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2688198430 0x7f268819a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 -- 172.21.15.155:0/849520413 >> v1:172.21.15.155:6789/0 conn(0x7f2688088dd0 legacy=0x7f268808d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 -- 172.21.15.155:0/849520413 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688073520 con 0x7f268808e800 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> v2:172.21.15.142:3300/0 conn(0x7f268808e800 0x7f268808e030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> v2:172.21.15.142:3300/0 conn(0x7f268808e800 0x7f268808e030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.419+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2678009780 con 0x7f268808e800 2019-10-13T00:51:59.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267800aea0 con 0x7f268808e800 2019-10-13T00:51:59.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f267800b090 con 0x7f268808e800 2019-10-13T00:51:59.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2688073520 con 0x7f268808e800 2019-10-13T00:51:59.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2688073520 con 0x7f268808e800 2019-10-13T00:51:59.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f267800b090 con 0x7f268808e800 2019-10-13T00:51:59.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f268659c700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f267f7fe700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2660017ca0 0x7f266001a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f267802ffe0 con 0x7f268808e800 2019-10-13T00:51:59.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.420+0000 7f2686d9d700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2660017ca0 0x7f266001a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.421+0000 7f2686d9d700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2660017ca0 0x7f266001a110 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.428+0000 7f268cbac700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6802/7037,v1:172.21.15.142:6803/7037] conn(0x7f26880537f0 0x7f2688053be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.428+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 --> [v2:172.21.15.142:6802/7037,v1:172.21.15.142:6803/7037] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2688149c00 con 0x7f26880537f0 2019-10-13T00:51:59.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.429+0000 7f268759e700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6802/7037,v1:172.21.15.142:6803/7037] conn(0x7f26880537f0 0x7f2688053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.429+0000 7f268759e700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6802/7037,v1:172.21.15.142:6803/7037] conn(0x7f26880537f0 0x7f2688053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.430+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== osd.4 v2:172.21.15.142:6802/7037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f268000de00 con 0x7f26880537f0 2019-10-13T00:51:59.447 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.449+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 --> [v2:172.21.15.142:6802/7037,v1:172.21.15.142:6803/7037] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f268808cfa0 con 0x7f26880537f0 2019-10-13T00:51:59.448 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.450+0000 7f267f7fe700 1 -- 172.21.15.155:0/849520413 <== osd.4 v2:172.21.15.142:6802/7037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2680009630 con 0x7f26880537f0 2019-10-13T00:51:59.451 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6802/7037,v1:172.21.15.142:6803/7037] conn(0x7f26880537f0 msgr2=0x7f2688053be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.451 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6802/7037,v1:172.21.15.142:6803/7037] conn(0x7f26880537f0 0x7f2688053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.452 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2660017ca0 msgr2=0x7f266001a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.452 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f2660017ca0 0x7f266001a110 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.452 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 msgr2=0x7f268808e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7f2678010380 tx=0x7f2678010720).stop 2019-10-13T00:51:59.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268759e700 1 -- 172.21.15.155:0/849520413 reap_dead start 2019-10-13T00:51:59.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 shutdown_connections 2019-10-13T00:51:59.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 --2- 172.21.15.155:0/849520413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f268808e800 0x7f268808e030 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 shutdown_connections 2019-10-13T00:51:59.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 wait complete. 2019-10-13T00:51:59.453 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.451+0000 7f268cbac700 1 -- 172.21.15.155:0/849520413 >> 172.21.15.155:0/849520413 conn(0x7f268806c410 msgr2=0x7f2688074840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.458 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:59.459 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:59.459 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-10-13T00:51:59.585 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.583+0000 7ff7a95ff700 1 Processor -- start 2019-10-13T00:51:59.585 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.583+0000 7ff7a95ff700 1 -- start start 2019-10-13T00:51:59.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a95ff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a95ff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a4064f00 con 0x7ff7a4099b10 2019-10-13T00:51:59.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a95ff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a40987e0 con 0x7ff7a409a700 2019-10-13T00:51:59.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a95ff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a409b680 con 0x7ff7a409d620 2019-10-13T00:51:59.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a2ffd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff7a409a700 0x7ff7a409c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49182/0 (socket says 172.21.15.155:49182) 2019-10-13T00:51:59.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a2ffd700 1 -- 172.21.15.155:0/2484411176 learned_addr learned my addr 172.21.15.155:0/2484411176 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a27fc700 1 -- 172.21.15.155:0/2484411176 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861782105 0 0) 0x7ff78c0014d0 con 0x7ff7a409a700 2019-10-13T00:51:59.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.584+0000 7ff7a27fc700 1 -- 172.21.15.155:0/2484411176 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff784002e70 con 0x7ff7a409a700 2019-10-13T00:51:59.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a27fc700 1 -- 172.21.15.155:0/2484411176 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553037377 0 0) 0x7ff794001080 con 0x7ff7a4099b10 2019-10-13T00:51:59.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a27fc700 1 -- 172.21.15.155:0/2484411176 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff784004160 con 0x7ff7a4099b10 2019-10-13T00:51:59.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 -- 172.21.15.155:0/2484411176 >> v1:172.21.15.155:6789/0 conn(0x7ff7a409a700 legacy=0x7ff7a409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 -- 172.21.15.155:0/2484411176 >> v1:172.21.15.142:6789/0 conn(0x7ff7a4099b10 legacy=0x7ff7a4099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 -- 172.21.15.155:0/2484411176 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a409bd10 con 0x7ff7a409d620 2019-10-13T00:51:59.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff79c00af70 tx=0x7ff79c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff79c00af70 tx=0x7ff79c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7ff79c00af70 tx=0x7ff79c00b780).ready entity=mon.2 client_cookie=3f4f27624e098e36 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7ff79c00af70 tx=0x7ff79c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7ff79c00af70 tx=0x7ff79c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a27fc700 1 -- 172.21.15.155:0/2484411176 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff79c007230 con 0x7ff7a409d620 2019-10-13T00:51:59.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7ff79c00af70 tx=0x7ff79c00b780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.589 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a27fc700 1 -- 172.21.15.155:0/2484411176 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c000bb0 con 0x7ff7a409d620 2019-10-13T00:51:59.589 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a27fc700 1 -- 172.21.15.155:0/2484411176 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff79c009e50 con 0x7ff7a409d620 2019-10-13T00:51:59.589 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a95ff700 1 -- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 msgr2=0x7ff7a4153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.589 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.585+0000 7ff7a95ff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7ff79c00af70 tx=0x7ff79c00b780).stop 2019-10-13T00:51:59.589 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.586+0000 7ff7a95ff700 1 -- 172.21.15.155:0/2484411176 shutdown_connections 2019-10-13T00:51:59.589 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.586+0000 7ff7a95ff700 1 --2- 172.21.15.155:0/2484411176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a409d620 0x7ff7a4153c20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.589 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.586+0000 7ff7a95ff700 1 -- 172.21.15.155:0/2484411176 shutdown_connections 2019-10-13T00:51:59.590 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.586+0000 7ff7a95ff700 1 -- 172.21.15.155:0/2484411176 wait complete. 2019-10-13T00:51:59.590 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.586+0000 7ff7a95ff700 1 -- 172.21.15.155:0/2484411176 >> 172.21.15.155:0/2484411176 conn(0x7ff7a4099550 msgr2=0x7ff7a4061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.588+0000 7ff7a95ff700 1 Processor -- start 2019-10-13T00:51:59.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.588+0000 7ff7a95ff700 1 -- start start 2019-10-13T00:51:59.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a95ff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a414b7f0 0x7ff7a414bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a95ff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff7a4153cf0 0x7ff7a4073770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a95ff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a41367e0 con 0x7ff7a408c8b0 2019-10-13T00:51:59.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a95ff700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a4071db0 con 0x7ff7a4153cf0 2019-10-13T00:51:59.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a95ff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a4072080 con 0x7ff7a414b7f0 2019-10-13T00:51:59.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a2ffd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff7a408c8b0 0x7ff7a41534f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49186/0 (socket says 172.21.15.155:49186) 2019-10-13T00:51:59.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a2ffd700 1 -- 172.21.15.155:0/3806074547 learned_addr learned my addr 172.21.15.155:0/3806074547 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> v2:172.21.15.155:3300/0 conn(0x7ff7a4153cf0 0x7ff7a4073770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a37fe700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a414b7f0 0x7ff7a414bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228220161 0 0) 0x7ff78c0014d0 con 0x7ff7a408c8b0 2019-10-13T00:51:59.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff77c002e70 con 0x7ff7a408c8b0 2019-10-13T00:51:59.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 -- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a414b7f0 msgr2=0x7ff7a414bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7a414b7f0 0x7ff7a414bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 -- 172.21.15.155:0/3806074547 >> v1:172.21.15.155:6789/0 conn(0x7ff7a408c8b0 legacy=0x7ff7a41534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 -- 172.21.15.155:0/3806074547 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a4072e70 con 0x7ff7a4153cf0 2019-10-13T00:51:59.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> v2:172.21.15.155:3300/0 conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> v2:172.21.15.155:3300/0 conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.589+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff79c00ad90 con 0x7ff7a4153cf0 2019-10-13T00:51:59.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c0097b0 con 0x7ff7a4153cf0 2019-10-13T00:51:59.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff79c009e80 con 0x7ff7a4153cf0 2019-10-13T00:51:59.596 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a4072e70 con 0x7ff7a4153cf0 2019-10-13T00:51:59.597 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a4072e70 con 0x7ff7a4153cf0 2019-10-13T00:51:59.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff79c0099a0 con 0x7ff7a4153cf0 2019-10-13T00:51:59.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a3fff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff79bfff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff77c017ca0 0x7ff77c01a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff79c02fff0 con 0x7ff7a4153cf0 2019-10-13T00:51:59.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.590+0000 7ff7a37fe700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff77c017ca0 0x7ff77c01a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.598 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.591+0000 7ff7a37fe700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff77c017ca0 0x7ff77c01a110 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.598+0000 7ff7a95ff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6810/7038,v1:172.21.15.142:6811/7038] conn(0x7ff7a419a180 0x7ff7a419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.599 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.598+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 --> [v2:172.21.15.142:6810/7038,v1:172.21.15.142:6811/7038] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7a41363c0 con 0x7ff7a419a180 2019-10-13T00:51:59.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.598+0000 7ff7a2ffd700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6810/7038,v1:172.21.15.142:6811/7038] conn(0x7ff7a419a180 0x7ff7a419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.599+0000 7ff7a2ffd700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6810/7038,v1:172.21.15.142:6811/7038] conn(0x7ff7a419a180 0x7ff7a419a570 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.600 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.600+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== osd.5 v2:172.21.15.142:6810/7038 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7ff78c000cc0 con 0x7ff7a419a180 2019-10-13T00:51:59.617 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.619+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 --> [v2:172.21.15.142:6810/7038,v1:172.21.15.142:6811/7038] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7a4137090 con 0x7ff7a419a180 2019-10-13T00:51:59.617 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.620+0000 7ff79bfff700 1 -- 172.21.15.155:0/3806074547 <== osd.5 v2:172.21.15.142:6810/7038 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff78c0045b0 con 0x7ff7a419a180 2019-10-13T00:51:59.619 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6810/7038,v1:172.21.15.142:6811/7038] conn(0x7ff7a419a180 msgr2=0x7ff7a419a570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.619 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6810/7038,v1:172.21.15.142:6811/7038] conn(0x7ff7a419a180 0x7ff7a419a570 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff77c017ca0 msgr2=0x7ff77c01a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff77c017ca0 0x7ff77c01a110 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 msgr2=0x7ff7a4073770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7ff79c0103a0 tx=0x7ff79c010740).stop 2019-10-13T00:51:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a3fff700 1 -- 172.21.15.155:0/3806074547 reap_dead start 2019-10-13T00:51:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 shutdown_connections 2019-10-13T00:51:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 --2- 172.21.15.155:0/3806074547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a4153cf0 0x7ff7a4073770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 shutdown_connections 2019-10-13T00:51:59.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 wait complete. 2019-10-13T00:51:59.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.621+0000 7ff7a95ff700 1 -- 172.21.15.155:0/3806074547 >> 172.21.15.155:0/3806074547 conn(0x7ff7a406c410 msgr2=0x7ff7a4074840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.628 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:59.628 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:59.629 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-10-13T00:51:59.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.752+0000 7f8a777d3700 1 Processor -- start 2019-10-13T00:51:59.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.752+0000 7f8a777d3700 1 -- start start 2019-10-13T00:51:59.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a777d3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a7009d620 0x7f8a70153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a777d3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a70064f00 con 0x7f8a7009a700 2019-10-13T00:51:59.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a777d3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700987e0 con 0x7f8a70099b10 2019-10-13T00:51:59.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a777d3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7009b680 con 0x7f8a7009d620 2019-10-13T00:51:59.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a75d70700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8a70099b10 0x7f8a70099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49196/0 (socket says 172.21.15.155:49196) 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a75d70700 1 -- 172.21.15.155:0/133240676 learned_addr learned my addr 172.21.15.155:0/133240676 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a7009d620 0x7f8a70153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592691351 0 0) 0x7f8a580014d0 con 0x7f8a70099b10 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.753+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c002e70 con 0x7f8a70099b10 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965954097 0 0) 0x7f8a60001080 con 0x7f8a7009a700 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c0030a0 con 0x7f8a7009a700 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1175828596 0 0) 0x7f8a580014b0 con 0x7f8a70099b10 2019-10-13T00:51:59.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a7009d620 msgr2=0x7f8a70153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.764 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a7009d620 0x7f8a70153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.764 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 >> v1:172.21.15.142:6789/0 conn(0x7f8a7009a700 legacy=0x7f8a7009c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.764 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7009bd10 con 0x7f8a70099b10 2019-10-13T00:51:59.764 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f8a580019e0 con 0x7f8a70099b10 2019-10-13T00:51:59.764 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 >> v1:172.21.15.155:6789/0 conn(0x7f8a70099b10 legacy=0x7f8a70099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.764 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a5c003c10 0x7f8a5c006070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.764 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a5c006760 0x7f8a5c008b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.765 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.765 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c04fff0 con 0x7f8a5c003c10 2019-10-13T00:51:59.765 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0501e0 con 0x7f8a5c04d310 2019-10-13T00:51:59.765 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c050410 con 0x7f8a5c006760 2019-10-13T00:51:59.765 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a7556f700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a5c006760 0x7f8a5c008b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.765 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a75d70700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a5c003c10 0x7f8a5c006070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.765 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.766 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 -- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a5c006760 msgr2=0x7f8a5c008b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.766 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a5c006760 0x7f8a5c008b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.766 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 -- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a5c003c10 msgr2=0x7f8a5c006070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.766 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a5c003c10 0x7f8a5c006070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.766 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 -- 172.21.15.155:0/133240676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c0506c0 con 0x7f8a5c04d310 2019-10-13T00:51:59.766 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8a68002e20 tx=0x7f8a6800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.766 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8a68002e20 tx=0x7f8a6800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f8a68002e20 tx=0x7f8a6800fa00).ready entity=mon.1 client_cookie=57fbc56944cd6dd8 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f8a68002e20 tx=0x7f8a6800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f8a68002e20 tx=0x7f8a6800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a68000b10 con 0x7f8a5c04d310 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f8a68002e20 tx=0x7f8a6800fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a68001060 con 0x7f8a5c04d310 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a74d6e700 1 -- 172.21.15.155:0/133240676 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a68001730 con 0x7f8a5c04d310 2019-10-13T00:51:59.767 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a777d3700 1 -- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 msgr2=0x7f8a5c04f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a777d3700 1 --2- 172.21.15.155:0/133240676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a5c04d310 0x7f8a5c04f7e0 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f8a68002e20 tx=0x7f8a6800fa00).stop 2019-10-13T00:51:59.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.754+0000 7f8a76571700 1 -- 172.21.15.155:0/133240676 reap_dead start 2019-10-13T00:51:59.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.755+0000 7f8a777d3700 1 -- 172.21.15.155:0/133240676 shutdown_connections 2019-10-13T00:51:59.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.755+0000 7f8a777d3700 1 -- 172.21.15.155:0/133240676 shutdown_connections 2019-10-13T00:51:59.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.755+0000 7f8a777d3700 1 -- 172.21.15.155:0/133240676 wait complete. 2019-10-13T00:51:59.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.755+0000 7f8a777d3700 1 -- 172.21.15.155:0/133240676 >> 172.21.15.155:0/133240676 conn(0x7f8a70099550 msgr2=0x7f8a70061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.768 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.757+0000 7f8a777d3700 1 Processor -- start 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.757+0000 7f8a777d3700 1 -- start start 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a777d3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8a70093420 0x7f8a70092c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a777d3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700901f0 con 0x7f8a70088da0 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a777d3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a70089660 con 0x7f8a70153e80 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a777d3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a70090a50 con 0x7f8a70093420 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a76571700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8a70153e80 0x7f8a70156280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49208/0 (socket says 172.21.15.155:49208) 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a76571700 1 -- 172.21.15.155:0/2423638159 learned_addr learned my addr 172.21.15.155:0/2423638159 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.769 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> v2:172.21.15.142:3300/0 conn(0x7f8a70093420 0x7f8a70092c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.758+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297507771 0 0) 0x7f8a68002aa0 con 0x7f8a70153e80 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a54002e70 con 0x7f8a70153e80 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2951173878 0 0) 0x7f8a68002380 con 0x7f8a70153e80 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a540032d0 con 0x7f8a70153e80 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f8a68000c80 con 0x7f8a70153e80 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 >> v2:172.21.15.142:3300/0 conn(0x7f8a70093420 msgr2=0x7f8a70092c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 --2- 172.21.15.155:0/2423638159 >> v2:172.21.15.142:3300/0 conn(0x7f8a70093420 0x7f8a70092c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.770 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 >> v1:172.21.15.155:6789/0 conn(0x7f8a70153e80 legacy=0x7f8a70156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 >> v1:172.21.15.142:6789/0 conn(0x7f8a70088da0 legacy=0x7f8a70092450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a54003320 0x7f8a54003750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a54004a60 0x7f8a54004f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5400bcc0 con 0x7f8a54003320 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5400bef0 con 0x7f8a54003da0 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5400c120 con 0x7f8a54004a60 2019-10-13T00:51:59.771 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> v2:172.21.15.142:3300/0 conn(0x7f8a70093420 0x7f8a70092c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:59.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a76571700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a54004a60 0x7f8a54004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a75d70700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a54003320 0x7f8a54003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 -- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a54004a60 msgr2=0x7f8a54004f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a54004a60 0x7f8a54004f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 -- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a54003320 msgr2=0x7f8a54003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.772 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a54003320 0x7f8a54003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5400c3d0 con 0x7f8a54003da0 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a76571700 1 -- 172.21.15.155:0/2423638159 reap_dead start 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a70091b50 con 0x7f8a54003da0 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a70091b50 con 0x7f8a54003da0 2019-10-13T00:51:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.759+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.774 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a600069d0 con 0x7f8a54003da0 2019-10-13T00:51:59.774 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.774 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.774 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a60009290 con 0x7f8a54003da0 2019-10-13T00:51:59.774 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a600069d0 con 0x7f8a54003da0 2019-10-13T00:51:59.774 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.774 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f8a600065f0 con 0x7f8a54003da0 2019-10-13T00:51:59.775 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a7556f700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.775 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a6e7fc700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8a5401a980 0x7f8a5401ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.775 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f8a60030120 con 0x7f8a54003da0 2019-10-13T00:51:59.775 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a75d70700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8a5401a980 0x7f8a5401ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.775 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.760+0000 7f8a75d70700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8a5401a980 0x7f8a5401ce10 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.776 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.767+0000 7f8a777d3700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] conn(0x7f8a48001330 0x7f8a48003800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.776 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.767+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a48006260 con 0x7f8a48001330 2019-10-13T00:51:59.776 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.767+0000 7f8a76571700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] conn(0x7f8a48001330 0x7f8a48003800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.776 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.768+0000 7f8a76571700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] conn(0x7f8a48001330 0x7f8a48003800 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.776 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.768+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== osd.6 v2:172.21.15.142:6819/7039 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f8a680095a0 con 0x7f8a48001330 2019-10-13T00:51:59.786 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.788+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 --> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a48005030 con 0x7f8a48001330 2019-10-13T00:51:59.787 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.789+0000 7f8a6e7fc700 1 -- 172.21.15.155:0/2423638159 <== osd.6 v2:172.21.15.142:6819/7039 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a68000c80 con 0x7f8a48001330 2019-10-13T00:51:59.788 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] conn(0x7f8a48001330 msgr2=0x7f8a48003800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.789 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] conn(0x7f8a48001330 0x7f8a48003800 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.789 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8a5401a980 msgr2=0x7f8a5401ce10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.789 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8a5401a980 0x7f8a5401ce10 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.789 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 msgr2=0x7f8a54004260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.789 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f8a600108e0 tx=0x7f8a60010c80).stop 2019-10-13T00:51:59.789 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 shutdown_connections 2019-10-13T00:51:59.789 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6819/7039,v1:172.21.15.142:6821/7039] conn(0x7f8a48001330 0x7f8a48003800 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.790 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8a5401a980 0x7f8a5401ce10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.790 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 --2- 172.21.15.155:0/2423638159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a54003da0 0x7f8a54004260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.790 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 shutdown_connections 2019-10-13T00:51:59.790 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 wait complete. 2019-10-13T00:51:59.790 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.790+0000 7f8a777d3700 1 -- 172.21.15.155:0/2423638159 >> 172.21.15.155:0/2423638159 conn(0x7f8a7006bfe0 msgr2=0x7f8a7006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.797 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:59.798 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:59.798 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-10-13T00:51:59.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.925+0000 7f1a28e77700 1 Processor -- start 2019-10-13T00:51:59.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.925+0000 7f1a28e77700 1 -- start start 2019-10-13T00:51:59.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.925+0000 7f1a28e77700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a28e77700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1a2409d620 0x7f1a2409a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:51:59.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a28e77700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a24064f00 con 0x7f1a2409c7f0 2019-10-13T00:51:59.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a28e77700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2409ac50 con 0x7f1a2409d620 2019-10-13T00:51:59.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a28e77700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2409af20 con 0x7f1a24099b10 2019-10-13T00:51:59.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a2259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1a2409d620 0x7f1a2409a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a2359e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f1a2409c7f0 0x7f1a2409cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:57240/0 (socket says 172.21.15.155:57240) 2019-10-13T00:51:59.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a2259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1a2409d620 0x7f1a2409a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59148/0 (socket says 172.21.15.155:59148) 2019-10-13T00:51:59.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a2359e700 1 -- 172.21.15.155:0/2095571018 learned_addr learned my addr 172.21.15.155:0/2095571018 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a21d9b700 1 -- 172.21.15.155:0/2095571018 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191066060 0 0) 0x7f1a1c001480 con 0x7f1a2409c7f0 2019-10-13T00:51:59.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 -- 172.21.15.155:0/2095571018 >> v2:172.21.15.155:3300/0 conn(0x7f1a2409d620 msgr2=0x7f1a2409a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:51:59.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/2095571018 >> v2:172.21.15.155:3300/0 conn(0x7f1a2409d620 0x7f1a2409a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 -- 172.21.15.155:0/2095571018 >> v1:172.21.15.142:6789/0 conn(0x7f1a2409c7f0 legacy=0x7f1a2409cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 -- 172.21.15.155:0/2095571018 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2409bd10 con 0x7f1a24099b10 2019-10-13T00:51:59.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a2259c700 1 --2- 172.21.15.155:0/2095571018 >> v2:172.21.15.155:3300/0 conn(0x7f1a2409d620 0x7f1a2409a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:51:59.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a1400af40 tx=0x7f1a1400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a1400af40 tx=0x7f1a1400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f1a1400af40 tx=0x7f1a1400b750).ready entity=mon.2 client_cookie=c425c0a804830240 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f1a1400af40 tx=0x7f1a1400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a21d9b700 1 -- 172.21.15.155:0/2095571018 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a14000960 con 0x7f1a24099b10 2019-10-13T00:51:59.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f1a1400af40 tx=0x7f1a1400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f1a1400af40 tx=0x7f1a1400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a21d9b700 1 -- 172.21.15.155:0/2095571018 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a14009850 con 0x7f1a24099b10 2019-10-13T00:51:59.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a21d9b700 1 -- 172.21.15.155:0/2095571018 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a14009f20 con 0x7f1a24099b10 2019-10-13T00:51:59.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a28e77700 1 -- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 msgr2=0x7f1a24099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.926+0000 7f1a28e77700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f1a1400af40 tx=0x7f1a1400b750).stop 2019-10-13T00:51:59.931 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.927+0000 7f1a28e77700 1 -- 172.21.15.155:0/2095571018 shutdown_connections 2019-10-13T00:51:59.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.927+0000 7f1a28e77700 1 --2- 172.21.15.155:0/2095571018 >> v2:172.21.15.155:3300/0 conn(0x7f1a2409d620 0x7f1a2409a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:51:59.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.927+0000 7f1a28e77700 1 --2- 172.21.15.155:0/2095571018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a24099b10 0x7f1a24099f00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.927+0000 7f1a28e77700 1 -- 172.21.15.155:0/2095571018 shutdown_connections 2019-10-13T00:51:59.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.927+0000 7f1a28e77700 1 -- 172.21.15.155:0/2095571018 wait complete. 2019-10-13T00:51:59.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.927+0000 7f1a28e77700 1 -- 172.21.15.155:0/2095571018 >> 172.21.15.155:0/2095571018 conn(0x7f1a24099550 msgr2=0x7f1a24061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.935 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.929+0000 7f1a28e77700 1 Processor -- start 2019-10-13T00:51:59.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a28e77700 1 -- start start 2019-10-13T00:51:59.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a28e77700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a240925b0 0x7f1a2408db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a28e77700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1a240929a0 0x7f1a24092df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a28e77700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a24088fc0 con 0x7f1a24076df0 2019-10-13T00:51:59.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a28e77700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2408c1a0 con 0x7f1a240929a0 2019-10-13T00:51:59.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a28e77700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2408c470 con 0x7f1a240925b0 2019-10-13T00:51:59.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a2359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1a240929a0 0x7f1a24092df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a2259c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1a24076df0 0x7f1a24076450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49232/0 (socket says 172.21.15.155:49232) 2019-10-13T00:51:59.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a2259c700 1 -- 172.21.15.155:0/1990299078 learned_addr learned my addr 172.21.15.155:0/1990299078 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:51:59.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.930+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a240925b0 0x7f1a2408db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 -- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a240925b0 msgr2=0x7f1a2408db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a240925b0 0x7f1a2408db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 -- 172.21.15.155:0/1990299078 >> v1:172.21.15.155:6789/0 conn(0x7f1a24076df0 legacy=0x7f1a24076450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442447619 0 0) 0x7f1a0c000ff0 con 0x7f1a24076df0 2019-10-13T00:51:59.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 -- 172.21.15.155:0/1990299078 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2408d260 con 0x7f1a240929a0 2019-10-13T00:51:59.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> v2:172.21.15.155:3300/0 conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> v2:172.21.15.155:3300/0 conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a1c001270 con 0x7f1a240929a0 2019-10-13T00:51:59.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c001270 con 0x7f1a240929a0 2019-10-13T00:51:59.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2408d260 con 0x7f1a240929a0 2019-10-13T00:51:59.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a1c004720 con 0x7f1a240929a0 2019-10-13T00:51:59.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2408d260 con 0x7f1a240929a0 2019-10-13T00:51:59.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.931+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.932+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f1a1c004530 con 0x7f1a240929a0 2019-10-13T00:51:59.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.932+0000 7f1a2359e700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:51:59.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.932+0000 7f1a1b7fe700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1a08017c60 0x7f1a0801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.932+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f1a1c0303c0 con 0x7f1a240929a0 2019-10-13T00:51:59.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.932+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1a08017c60 0x7f1a0801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.932+0000 7f1a22d9d700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1a08017c60 0x7f1a0801a0f0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.940+0000 7f1a28e77700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6818/7040,v1:172.21.15.142:6820/7040] conn(0x7f1a240537f0 0x7f1a24053be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:51:59.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.940+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 --> [v2:172.21.15.142:6818/7040,v1:172.21.15.142:6820/7040] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a240899f0 con 0x7f1a240537f0 2019-10-13T00:51:59.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.940+0000 7f1a2259c700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6818/7040,v1:172.21.15.142:6820/7040] conn(0x7f1a240537f0 0x7f1a24053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:51:59.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.940+0000 7f1a2259c700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6818/7040,v1:172.21.15.142:6820/7040] conn(0x7f1a240537f0 0x7f1a24053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:51:59.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.941+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== osd.7 v2:172.21.15.142:6818/7040 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f1a0c002300 con 0x7f1a240537f0 2019-10-13T00:51:59.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.961+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 --> [v2:172.21.15.142:6818/7040,v1:172.21.15.142:6820/7040] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a24073c80 con 0x7f1a240537f0 2019-10-13T00:51:59.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.961+0000 7f1a1b7fe700 1 -- 172.21.15.155:0/1990299078 <== osd.7 v2:172.21.15.142:6818/7040 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1a0c0092e0 con 0x7f1a240537f0 2019-10-13T00:51:59.961 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6818/7040,v1:172.21.15.142:6820/7040] conn(0x7f1a240537f0 msgr2=0x7f1a24053be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.961 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6818/7040,v1:172.21.15.142:6820/7040] conn(0x7f1a240537f0 0x7f1a24053be0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.961 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1a08017c60 msgr2=0x7f1a0801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f1a08017c60 0x7f1a0801a0f0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 msgr2=0x7f1a24092df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:51:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f1a1c010720 tx=0x7f1a1c010ac0).stop 2019-10-13T00:51:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a2359e700 1 -- 172.21.15.155:0/1990299078 reap_dead start 2019-10-13T00:51:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 shutdown_connections 2019-10-13T00:51:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 --2- 172.21.15.155:0/1990299078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a240929a0 0x7f1a24092df0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:51:59.962 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 shutdown_connections 2019-10-13T00:51:59.963 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 wait complete. 2019-10-13T00:51:59.963 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:51:59.963+0000 7f1a28e77700 1 -- 172.21.15.155:0/1990299078 >> 172.21.15.155:0/1990299078 conn(0x7f1a2406d080 msgr2=0x7f1a2406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:51:59.970 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:51:59.972 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:51:59.973 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-10-13T00:52:00.099 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.096+0000 7f9c56cc5700 1 Processor -- start 2019-10-13T00:52:00.099 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c56cc5700 1 -- start start 2019-10-13T00:52:00.099 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c56cc5700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9c5009a700 0x7f9c5009c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:00.099 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c56cc5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c50064f00 con 0x7f9c5009d620 2019-10-13T00:52:00.099 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c56cc5700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c500987e0 con 0x7f9c50099b10 2019-10-13T00:52:00.099 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c56cc5700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5009b680 con 0x7f9c5009a700 2019-10-13T00:52:00.100 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c54a61700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9c5009a700 0x7f9c5009c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:00.100 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c55262700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f9c50099b10 0x7f9c50099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49238/0 (socket says 172.21.15.155:49238) 2019-10-13T00:52:00.100 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c54a61700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9c5009a700 0x7f9c5009c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59166/0 (socket says 172.21.15.155:59166) 2019-10-13T00:52:00.100 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.097+0000 7f9c55262700 1 -- 172.21.15.155:0/2017098752 learned_addr learned my addr 172.21.15.155:0/2017098752 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:00.100 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 -- 172.21.15.155:0/2017098752 >> v1:172.21.15.155:6789/0 conn(0x7f9c50099b10 legacy=0x7f9c50099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:00.100 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 -- 172.21.15.155:0/2017098752 >> v1:172.21.15.142:6789/0 conn(0x7f9c5009d620 legacy=0x7f9c50152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:00.100 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c47fff700 1 -- 172.21.15.155:0/2017098752 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33548571 0 0) 0x7f9c4c001150 con 0x7f9c50099b10 2019-10-13T00:52:00.101 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 -- 172.21.15.155:0/2017098752 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5009bd10 con 0x7f9c5009a700 2019-10-13T00:52:00.101 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 --2- 172.21.15.155:0/2017098752 >> v2:172.21.15.155:3300/0 conn(0x7f9c5009a700 0x7f9c5009c7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9c4000b2c0 tx=0x7f9c4000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.101 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 --2- 172.21.15.155:0/2017098752 >> v2:172.21.15.155:3300/0 conn(0x7f9c5009a700 0x7f9c5009c7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9c4000b2c0 tx=0x7f9c4000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.101 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 --2- 172.21.15.155:0/2017098752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5009a700 0x7f9c5009c7f0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f9c4000b2c0 tx=0x7f9c4000bad0).ready entity=mon.1 client_cookie=8ea3e7ff1b1aea91 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:00.101 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 --2- 172.21.15.155:0/2017098752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5009a700 0x7f9c5009c7f0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f9c4000b2c0 tx=0x7f9c4000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.101 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c47fff700 1 -- 172.21.15.155:0/2017098752 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c40000ce0 con 0x7f9c5009a700 2019-10-13T00:52:00.101 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 --2- 172.21.15.155:0/2017098752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5009a700 0x7f9c5009c7f0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f9c4000b2c0 tx=0x7f9c4000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.102 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c54a61700 1 --2- 172.21.15.155:0/2017098752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5009a700 0x7f9c5009c7f0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f9c4000b2c0 tx=0x7f9c4000bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.102 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c47fff700 1 -- 172.21.15.155:0/2017098752 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40009bd0 con 0x7f9c5009a700 2019-10-13T00:52:00.102 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c47fff700 1 -- 172.21.15.155:0/2017098752 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c4000a2a0 con 0x7f9c5009a700 2019-10-13T00:52:00.102 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c56cc5700 1 -- 172.21.15.155:0/2017098752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5009a700 msgr2=0x7f9c5009c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:00.102 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.098+0000 7f9c56cc5700 1 --2- 172.21.15.155:0/2017098752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5009a700 0x7f9c5009c7f0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f9c4000b2c0 tx=0x7f9c4000bad0).stop 2019-10-13T00:52:00.102 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.099+0000 7f9c56cc5700 1 -- 172.21.15.155:0/2017098752 shutdown_connections 2019-10-13T00:52:00.103 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.099+0000 7f9c56cc5700 1 --2- 172.21.15.155:0/2017098752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5009a700 0x7f9c5009c7f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:00.103 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.099+0000 7f9c56cc5700 1 -- 172.21.15.155:0/2017098752 shutdown_connections 2019-10-13T00:52:00.103 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.099+0000 7f9c56cc5700 1 -- 172.21.15.155:0/2017098752 wait complete. 2019-10-13T00:52:00.103 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.099+0000 7f9c56cc5700 1 -- 172.21.15.155:0/2017098752 >> 172.21.15.155:0/2017098752 conn(0x7f9c50099550 msgr2=0x7f9c50061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:00.107 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.101+0000 7f9c56cc5700 1 Processor -- start 2019-10-13T00:52:00.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.101+0000 7f9c56cc5700 1 -- start start 2019-10-13T00:52:00.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c56cc5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:00.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c56cc5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9c500926e0 0x7f9c50091c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:00.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c56cc5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5008eac0 con 0x7f9c50092190 2019-10-13T00:52:00.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c56cc5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5008fd50 con 0x7f9c500926e0 2019-10-13T00:52:00.108 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c56cc5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c50090020 con 0x7f9c50073990 2019-10-13T00:52:00.109 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c55262700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:00.109 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c55262700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.155:40608/0 (socket says 172.21.15.155:40608) 2019-10-13T00:52:00.109 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c54a61700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9c500926e0 0x7f9c50091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:00.109 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.102+0000 7f9c55262700 1 -- 172.21.15.155:0/198644135 learned_addr learned my addr 172.21.15.155:0/198644135 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:00.109 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 -- 172.21.15.155:0/198644135 >> v2:172.21.15.142:3300/0 conn(0x7f9c500926e0 msgr2=0x7f9c50091c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:00.109 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> v2:172.21.15.142:3300/0 conn(0x7f9c500926e0 0x7f9c50091c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:00.110 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 -- 172.21.15.155:0/198644135 >> v1:172.21.15.142:6789/0 conn(0x7f9c50092190 legacy=0x7f9c50088ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:00.110 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 -- 172.21.15.155:0/198644135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c50090e10 con 0x7f9c50073990 2019-10-13T00:52:00.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:00.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.112 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c45ffb700 1 -- 172.21.15.155:0/198644135 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c4c000ed0 con 0x7f9c50073990 2019-10-13T00:52:00.112 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c45ffb700 1 -- 172.21.15.155:0/198644135 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c004420 con 0x7f9c50073990 2019-10-13T00:52:00.112 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c45ffb700 1 -- 172.21.15.155:0/198644135 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c4c004a70 con 0x7f9c50073990 2019-10-13T00:52:00.112 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c50090e10 con 0x7f9c50073990 2019-10-13T00:52:00.112 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c50090e10 con 0x7f9c50073990 2019-10-13T00:52:00.112 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.113 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c45ffb700 1 -- 172.21.15.155:0/198644135 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f9c4c004610 con 0x7f9c50073990 2019-10-13T00:52:00.113 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.103+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.113 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.104+0000 7f9c45ffb700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9c30017c60 0x7f9c3001a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:00.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.104+0000 7f9c45ffb700 1 -- 172.21.15.155:0/198644135 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9c4c030110 con 0x7f9c50073990 2019-10-13T00:52:00.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.104+0000 7f9c54a61700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9c30017c60 0x7f9c3001a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:00.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.104+0000 7f9c54a61700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9c30017c60 0x7f9c3001a0f0 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.112+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c50090e10 con 0x7f9c50073990 2019-10-13T00:52:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.115+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.115+0000 7f9c45ffb700 1 -- 172.21.15.155:0/198644135 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f9c4c0012b0 con 0x7f9c50073990 2019-10-13T00:52:00.282 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.285+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9c50090e10 con 0x7f9c50073990 2019-10-13T00:52:00.283 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.285+0000 7f9c55262700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:00.283 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.285+0000 7f9c45ffb700 1 -- 172.21.15.155:0/198644135 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9c4c004610 con 0x7f9c50073990 2019-10-13T00:52:00.284 INFO:teuthology.orchestra.run.smithi155.stdout:55834574854 2019-10-13T00:52:00.286 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.287+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9c30017c60 msgr2=0x7f9c3001a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:00.286 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.287+0000 7f9c56cc5700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9c30017c60 0x7f9c3001a0f0 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:00.286 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.287+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 msgr2=0x7f9c50091710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:00.286 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.287+0000 7f9c56cc5700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7f9c4c010420 tx=0x7f9c4c0107c0).stop 2019-10-13T00:52:00.286 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.288+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 shutdown_connections 2019-10-13T00:52:00.286 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.288+0000 7f9c56cc5700 1 --2- 172.21.15.155:0/198644135 >> v2:172.21.15.142:3300/0 conn(0x7f9c500926e0 0x7f9c50091c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:00.287 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.288+0000 7f9c56cc5700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9c30017c60 0x7f9c3001a0f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:00.287 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.288+0000 7f9c56cc5700 1 --2- 172.21.15.155:0/198644135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c50073990 0x7f9c50091710 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:00.287 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.288+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 shutdown_connections 2019-10-13T00:52:00.287 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.288+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 wait complete. 2019-10-13T00:52:00.287 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:00.288+0000 7f9c56cc5700 1 -- 172.21.15.155:0/198644135 >> 172.21.15.155:0/198644135 conn(0x7f9c5006d080 msgr2=0x7f9c5006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:00.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2019-10-13T00:52:01.296 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:01.296 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-10-13T00:52:01.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f4142456700 1 Processor -- start 2019-10-13T00:52:01.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f4142456700 1 -- start start 2019-10-13T00:52:01.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f4142456700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f413c09a700 0x7f413c09c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:01.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f4142456700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413c064f00 con 0x7f413c09d620 2019-10-13T00:52:01.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f4142456700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413c0987e0 con 0x7f413c099b10 2019-10-13T00:52:01.424 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f4142456700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c09b680 con 0x7f413c09a700 2019-10-13T00:52:01.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f41409f3700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f413c099b10 0x7f413c099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49254/0 (socket says 172.21.15.155:49254) 2019-10-13T00:52:01.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.422+0000 7f41409f3700 1 -- 172.21.15.155:0/981449788 learned_addr learned my addr 172.21.15.155:0/981449788 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:01.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 --2- 172.21.15.155:0/981449788 >> v2:172.21.15.142:3300/0 conn(0x7f413c09a700 0x7f413c09c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327218553 0 0) 0x7f4124001150 con 0x7f413c099b10 2019-10-13T00:52:01.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4128002e70 con 0x7f413c099b10 2019-10-13T00:52:01.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361047968 0 0) 0x7f4134000d30 con 0x7f413c09d620 2019-10-13T00:52:01.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41280030a0 con 0x7f413c09d620 2019-10-13T00:52:01.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 -- 172.21.15.155:0/981449788 >> v1:172.21.15.155:6789/0 conn(0x7f413c099b10 legacy=0x7f413c099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1338481615 0 0) 0x7f4124001130 con 0x7f413c099b10 2019-10-13T00:52:01.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 -- 172.21.15.155:0/981449788 >> v1:172.21.15.142:6789/0 conn(0x7f413c09d620 legacy=0x7f413c152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 -- 172.21.15.155:0/981449788 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c09bd10 con 0x7f413c09a700 2019-10-13T00:52:01.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 --2- 172.21.15.155:0/981449788 >> v2:172.21.15.142:3300/0 conn(0x7f413c09a700 0x7f413c09c7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f412c00af40 tx=0x7f412c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 --2- 172.21.15.155:0/981449788 >> v2:172.21.15.142:3300/0 conn(0x7f413c09a700 0x7f413c09c7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f412c00af40 tx=0x7f412c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 --2- 172.21.15.155:0/981449788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413c09a700 0x7f413c09c7f0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f412c00af40 tx=0x7f412c00b750).ready entity=mon.0 client_cookie=15ba008076063e95 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 --2- 172.21.15.155:0/981449788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413c09a700 0x7f413c09c7f0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f412c00af40 tx=0x7f412c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f412c000960 con 0x7f413c09a700 2019-10-13T00:52:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 --2- 172.21.15.155:0/981449788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413c09a700 0x7f413c09c7f0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f412c00af40 tx=0x7f412c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413bfff700 1 --2- 172.21.15.155:0/981449788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413c09a700 0x7f413c09c7f0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f412c00af40 tx=0x7f412c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c009850 con 0x7f413c09a700 2019-10-13T00:52:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f413b7fe700 1 -- 172.21.15.155:0/981449788 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f412c009f20 con 0x7f413c09a700 2019-10-13T00:52:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f4142456700 1 -- 172.21.15.155:0/981449788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413c09a700 msgr2=0x7f413c09c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.423+0000 7f4142456700 1 --2- 172.21.15.155:0/981449788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413c09a700 0x7f413c09c7f0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f412c00af40 tx=0x7f412c00b750).stop 2019-10-13T00:52:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.424+0000 7f4142456700 1 -- 172.21.15.155:0/981449788 shutdown_connections 2019-10-13T00:52:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.424+0000 7f4142456700 1 --2- 172.21.15.155:0/981449788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413c09a700 0x7f413c09c7f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.424+0000 7f4142456700 1 -- 172.21.15.155:0/981449788 shutdown_connections 2019-10-13T00:52:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.424+0000 7f4142456700 1 -- 172.21.15.155:0/981449788 wait complete. 2019-10-13T00:52:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.424+0000 7f4142456700 1 -- 172.21.15.155:0/981449788 >> 172.21.15.155:0/981449788 conn(0x7f413c099550 msgr2=0x7f413c061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:01.431 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.426+0000 7f4142456700 1 Processor -- start 2019-10-13T00:52:01.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.426+0000 7f4142456700 1 -- start start 2019-10-13T00:52:01.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f4142456700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f413c073990 0x7f413c0906c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:01.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f4142456700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f413c091690 0x7f413c090c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:01.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f4142456700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413c08d4b0 con 0x7f413c091140 2019-10-13T00:52:01.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f4142456700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c08f730 con 0x7f413c073990 2019-10-13T00:52:01.432 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f4142456700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c08ed00 con 0x7f413c091690 2019-10-13T00:52:01.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f413bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f413c091690 0x7f413c090c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f41411f4700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f413c091140 0x7f413c08bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49262/0 (socket says 172.21.15.155:49262) 2019-10-13T00:52:01.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f41411f4700 1 -- 172.21.15.155:0/1126101211 learned_addr learned my addr 172.21.15.155:0/1126101211 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:01.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.427+0000 7f41409f3700 1 --2- 172.21.15.155:0/1126101211 >> v2:172.21.15.142:3300/0 conn(0x7f413c073990 0x7f413c0906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 -- 172.21.15.155:0/1126101211 >> v2:172.21.15.142:3300/0 conn(0x7f413c073990 msgr2=0x7f413c0906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.433 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> v2:172.21.15.142:3300/0 conn(0x7f413c073990 0x7f413c0906c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 -- 172.21.15.155:0/1126101211 >> v1:172.21.15.155:6789/0 conn(0x7f413c091140 legacy=0x7f413c08bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 -- 172.21.15.155:0/1126101211 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c08fdc0 con 0x7f413c091690 2019-10-13T00:52:01.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063034252 0 0) 0x7f4134001470 con 0x7f413c091140 2019-10-13T00:52:01.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> v2:172.21.15.155:3300/0 conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> v2:172.21.15.155:3300/0 conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:01.434 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f412c009570 con 0x7f413c091690 2019-10-13T00:52:01.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c00aea0 con 0x7f413c091690 2019-10-13T00:52:01.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f412c009570 con 0x7f413c091690 2019-10-13T00:52:01.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c08fdc0 con 0x7f413c091690 2019-10-13T00:52:01.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c08fdc0 con 0x7f413c091690 2019-10-13T00:52:01.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.428+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.429+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f412c009570 con 0x7f413c091690 2019-10-13T00:52:01.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.429+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.429+0000 7f41397fa700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4120017c60 0x7f412001a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:01.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.429+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f412c02ffe0 con 0x7f413c091690 2019-10-13T00:52:01.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.429+0000 7f41409f3700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4120017c60 0x7f412001a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.429+0000 7f41409f3700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4120017c60 0x7f412001a0f0 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:01.437 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.437+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c08fdc0 con 0x7f413c091690 2019-10-13T00:52:01.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.440+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.438 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.440+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f412c02c340 con 0x7f413c091690 2019-10-13T00:52:01.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.608+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f413c08fdc0 con 0x7f413c091690 2019-10-13T00:52:01.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.609+0000 7f413bfff700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.607 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.609+0000 7f41397fa700 1 -- 172.21.15.155:0/1126101211 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f412c02c340 con 0x7f413c091690 2019-10-13T00:52:01.608 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:01.610 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.611+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4120017c60 msgr2=0x7f412001a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.610 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.611+0000 7f4142456700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4120017c60 0x7f412001a0f0 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.610 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.611+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 msgr2=0x7f413c090c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.610 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.611+0000 7f4142456700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f412c010380 tx=0x7f412c010720).stop 2019-10-13T00:52:01.610 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.612+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 shutdown_connections 2019-10-13T00:52:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.612+0000 7f4142456700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f4120017c60 0x7f412001a0f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.612+0000 7f4142456700 1 --2- 172.21.15.155:0/1126101211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f413c091690 0x7f413c090c00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.612+0000 7f4142456700 1 --2- 172.21.15.155:0/1126101211 >> v2:172.21.15.142:3300/0 conn(0x7f413c073990 0x7f413c0906c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.612+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 shutdown_connections 2019-10-13T00:52:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.612+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 wait complete. 2019-10-13T00:52:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.612+0000 7f4142456700 1 -- 172.21.15.155:0/1126101211 >> 172.21.15.155:0/1126101211 conn(0x7f413c06d080 msgr2=0x7f413c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:01.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2019-10-13T00:52:01.619 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:01.619 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-10-13T00:52:01.747 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.744+0000 7fc131e7d700 1 Processor -- start 2019-10-13T00:52:01.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.744+0000 7fc131e7d700 1 -- start start 2019-10-13T00:52:01.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc131e7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc12c099b10 0x7fc12c099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:01.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc131e7d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c064f00 con 0x7fc12c09d620 2019-10-13T00:52:01.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc131e7d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0987e0 con 0x7fc12c09cff0 2019-10-13T00:52:01.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc131e7d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c09ac50 con 0x7fc12c099b10 2019-10-13T00:52:01.748 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc130c1b700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc12c09cff0 0x7fc12c152c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49270/0 (socket says 172.21.15.155:49270) 2019-10-13T00:52:01.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc130c1b700 1 -- 172.21.15.155:0/1522619173 learned_addr learned my addr 172.21.15.155:0/1522619173 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:01.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12bfff700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc12c099b10 0x7fc12c099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562125047 0 0) 0x7fc124001480 con 0x7fc12c09cff0 2019-10-13T00:52:01.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc118002e70 con 0x7fc12c09cff0 2019-10-13T00:52:01.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2237032124 0 0) 0x7fc124001460 con 0x7fc12c09cff0 2019-10-13T00:52:01.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc12c099b10 msgr2=0x7fc12c099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:01.749 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc12c099b10 0x7fc12c099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:01.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 >> v1:172.21.15.142:6789/0 conn(0x7fc12c09d620 legacy=0x7fc12c09c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c09bd10 con 0x7fc12c09cff0 2019-10-13T00:52:01.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.745+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7fc124001990 con 0x7fc12c09cff0 2019-10-13T00:52:01.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 >> v1:172.21.15.155:6789/0 conn(0x7fc12c09cff0 legacy=0x7fc12c152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc118003b10 0x7fc118005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:01.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc118006640 0x7fc118008af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:01.750 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:01.751 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11800bfd0 con 0x7fc118006640 2019-10-13T00:52:01.751 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1180501d0 con 0x7fc1180092f0 2019-10-13T00:52:01.751 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc118050400 con 0x7fc118003b10 2019-10-13T00:52:01.751 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.751 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12bfff700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc118003b10 0x7fc118005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.751 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc123fff700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc118006640 0x7fc118008af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.751 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 -- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc118003b10 msgr2=0x7fc118005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:01.752 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc118003b10 0x7fc118005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:01.752 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 -- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc118006640 msgr2=0x7fc118008af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:01.752 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc118006640 0x7fc118008af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:01.752 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 -- 172.21.15.155:0/1522619173 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180506b0 con 0x7fc1180092f0 2019-10-13T00:52:01.752 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc123fff700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc118006640 0x7fc118008af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T00:52:01.752 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc12400fc50 tx=0x7fc12400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.752 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc12400fc50 tx=0x7fc12400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7fc12400fc50 tx=0x7fc12400fff0).ready entity=mon.1 client_cookie=5513f46ccdd9cc60 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:01.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7fc12400fc50 tx=0x7fc12400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc124001460 con 0x7fc1180092f0 2019-10-13T00:52:01.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7fc12400fc50 tx=0x7fc12400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7fc12400fc50 tx=0x7fc12400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1240047a0 con 0x7fc1180092f0 2019-10-13T00:52:01.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc12b7fe700 1 -- 172.21.15.155:0/1522619173 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc124004eb0 con 0x7fc1180092f0 2019-10-13T00:52:01.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc131e7d700 1 -- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 msgr2=0x7fc11800b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc131e7d700 1 --2- 172.21.15.155:0/1522619173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1180092f0 0x7fc11800b7c0 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7fc12400fc50 tx=0x7fc12400fff0).stop 2019-10-13T00:52:01.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.746+0000 7fc130c1b700 1 -- 172.21.15.155:0/1522619173 reap_dead start 2019-10-13T00:52:01.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.747+0000 7fc131e7d700 1 -- 172.21.15.155:0/1522619173 shutdown_connections 2019-10-13T00:52:01.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.747+0000 7fc131e7d700 1 -- 172.21.15.155:0/1522619173 shutdown_connections 2019-10-13T00:52:01.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.747+0000 7fc131e7d700 1 -- 172.21.15.155:0/1522619173 wait complete. 2019-10-13T00:52:01.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.747+0000 7fc131e7d700 1 -- 172.21.15.155:0/1522619173 >> 172.21.15.155:0/1522619173 conn(0x7fc12c099550 msgr2=0x7fc12c061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:01.755 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.749+0000 7fc131e7d700 1 Processor -- start 2019-10-13T00:52:01.755 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.749+0000 7fc131e7d700 1 -- start start 2019-10-13T00:52:01.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc131e7d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc12c092320 0x7fc12c091b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:01.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc131e7d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc12c152ec0 0x7fc12c1552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:01.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc131e7d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c08e6c0 con 0x7fc12c08c8c0 2019-10-13T00:52:01.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc131e7d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c148a70 con 0x7fc12c152ec0 2019-10-13T00:52:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc131e7d700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0903c0 con 0x7fc12c092320 2019-10-13T00:52:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc12c092320 0x7fc12c091b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc12c092320 0x7fc12c091b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59206/0 (socket says 172.21.15.155:59206) 2019-10-13T00:52:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 -- 172.21.15.155:0/513134384 learned_addr learned my addr 172.21.15.155:0/513134384 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc130c1b700 1 --2- 172.21.15.155:0/513134384 >> v2:172.21.15.142:3300/0 conn(0x7fc12c152ec0 0x7fc12c1552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 -- 172.21.15.155:0/513134384 >> v2:172.21.15.142:3300/0 conn(0x7fc12c152ec0 msgr2=0x7fc12c1552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> v2:172.21.15.142:3300/0 conn(0x7fc12c152ec0 0x7fc12c1552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.760 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 -- 172.21.15.155:0/513134384 >> v1:172.21.15.155:6789/0 conn(0x7fc12c08c8c0 legacy=0x7fc12c091350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.760 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990428017 0 0) 0x7fc11c0010b0 con 0x7fc12c08c8c0 2019-10-13T00:52:01.760 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 -- 172.21.15.155:0/513134384 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c090a50 con 0x7fc12c092320 2019-10-13T00:52:01.760 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> v2:172.21.15.155:3300/0 conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.760 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> v2:172.21.15.155:3300/0 conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.760 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.750+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:01.760 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc114000da0 con 0x7fc12c092320 2019-10-13T00:52:01.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc114001640 con 0x7fc12c092320 2019-10-13T00:52:01.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc114000da0 con 0x7fc12c092320 2019-10-13T00:52:01.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c090a50 con 0x7fc12c092320 2019-10-13T00:52:01.761 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c090a50 con 0x7fc12c092320 2019-10-13T00:52:01.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fc114000da0 con 0x7fc12c092320 2019-10-13T00:52:01.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc1297fa700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc110017c60 0x7fc11001a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:01.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc114030110 con 0x7fc12c092320 2019-10-13T00:52:01.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.751+0000 7fc12bfff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc110017c60 0x7fc11001a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:01.762 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.752+0000 7fc12bfff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc110017c60 0x7fc11001a0f0 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:01.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.759+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c090a50 con 0x7fc12c092320 2019-10-13T00:52:01.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.762+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.763 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.762+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fc114001240 con 0x7fc12c092320 2019-10-13T00:52:01.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T00:52:01.779+0000 7f459e559700 -1 mgr.server _prune_pending_service_map missing key mds.0 2019-10-13T00:52:01.929 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.931+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc12c090a50 con 0x7fc12c092320 2019-10-13T00:52:01.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.931+0000 7fc123fff700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:01.930 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.931+0000 7fc1297fa700 1 -- 172.21.15.155:0/513134384 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc114000da0 con 0x7fc12c092320 2019-10-13T00:52:01.930 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:01.932 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc110017c60 msgr2=0x7fc11001a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc110017c60 0x7fc11001a0f0 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 msgr2=0x7fc12c091b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:01.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7fc114010470 tx=0x7fc114010810).stop 2019-10-13T00:52:01.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 shutdown_connections 2019-10-13T00:52:01.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 --2- 172.21.15.155:0/513134384 >> v2:172.21.15.142:3300/0 conn(0x7fc12c152ec0 0x7fc12c1552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.933 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc12c092320 0x7fc12c091b50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 --2- 172.21.15.155:0/513134384 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc110017c60 0x7fc11001a0f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:01.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 shutdown_connections 2019-10-13T00:52:01.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 wait complete. 2019-10-13T00:52:01.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:01.934+0000 7fc131e7d700 1 -- 172.21.15.155:0/513134384 >> 172.21.15.155:0/513134384 conn(0x7fc12c06bfe0 msgr2=0x7fc12c06d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:01.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2019-10-13T00:52:01.942 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:01.943 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-10-13T00:52:02.069 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.067+0000 7f49e2a67700 1 Processor -- start 2019-10-13T00:52:02.070 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.067+0000 7f49e2a67700 1 -- start start 2019-10-13T00:52:02.070 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e2a67700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f49dc099b10 0x7f49dc099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:02.070 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e2a67700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc09d620 0x7f49dc09a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:02.070 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e2a67700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49dc064f00 con 0x7f49dc09c7f0 2019-10-13T00:52:02.070 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e2a67700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49dc09b680 con 0x7f49dc099b10 2019-10-13T00:52:02.070 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e2a67700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc09ac50 con 0x7f49dc09d620 2019-10-13T00:52:02.071 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e1805700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f49dc09c7f0 0x7f49dc09ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49292/0 (socket says 172.21.15.155:49292) 2019-10-13T00:52:02.071 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e1805700 1 -- 172.21.15.155:0/1285567115 learned_addr learned my addr 172.21.15.155:0/1285567115 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:02.071 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.068+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> v2:172.21.15.155:3300/0 conn(0x7f49dc099b10 0x7f49dc099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.071 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e0803700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc09d620 0x7f49dc09a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.071 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49dbfff700 1 -- 172.21.15.155:0/1285567115 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502696053 0 0) 0x7f49d40014f0 con 0x7f49dc09c7f0 2019-10-13T00:52:02.071 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49dbfff700 1 -- 172.21.15.155:0/1285567115 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c4002e70 con 0x7f49dc09c7f0 2019-10-13T00:52:02.071 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 -- 172.21.15.155:0/1285567115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc09d620 msgr2=0x7f49dc09a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:02.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc09d620 0x7f49dc09a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:02.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 -- 172.21.15.155:0/1285567115 >> v1:172.21.15.155:6789/0 conn(0x7f49dc09c7f0 legacy=0x7f49dc09ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 -- 172.21.15.155:0/1285567115 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc09bd10 con 0x7f49dc099b10 2019-10-13T00:52:02.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> v2:172.21.15.155:3300/0 conn(0x7f49dc099b10 0x7f49dc099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f49cc00af40 tx=0x7f49cc00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> v2:172.21.15.155:3300/0 conn(0x7f49dc099b10 0x7f49dc099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f49cc00af40 tx=0x7f49cc00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc099b10 0x7f49dc099f00 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f49cc00af40 tx=0x7f49cc00b750).ready entity=mon.1 client_cookie=3c9b9772a37a61b6 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc099b10 0x7f49dc099f00 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f49cc00af40 tx=0x7f49cc00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49dbfff700 1 -- 172.21.15.155:0/1285567115 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49cc000960 con 0x7f49dc099b10 2019-10-13T00:52:02.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc099b10 0x7f49dc099f00 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f49cc00af40 tx=0x7f49cc00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e1004700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc099b10 0x7f49dc099f00 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f49cc00af40 tx=0x7f49cc00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49dbfff700 1 -- 172.21.15.155:0/1285567115 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49cc009850 con 0x7f49dc099b10 2019-10-13T00:52:02.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49dbfff700 1 -- 172.21.15.155:0/1285567115 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49cc009f20 con 0x7f49dc099b10 2019-10-13T00:52:02.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e2a67700 1 -- 172.21.15.155:0/1285567115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc099b10 msgr2=0x7f49dc099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.069+0000 7f49e2a67700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc099b10 0x7f49dc099f00 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f49cc00af40 tx=0x7f49cc00b750).stop 2019-10-13T00:52:02.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.070+0000 7f49e2a67700 1 -- 172.21.15.155:0/1285567115 shutdown_connections 2019-10-13T00:52:02.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.070+0000 7f49e2a67700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc09d620 0x7f49dc09a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:02.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.070+0000 7f49e2a67700 1 --2- 172.21.15.155:0/1285567115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc099b10 0x7f49dc099f00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.070+0000 7f49e2a67700 1 -- 172.21.15.155:0/1285567115 shutdown_connections 2019-10-13T00:52:02.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.070+0000 7f49e2a67700 1 -- 172.21.15.155:0/1285567115 wait complete. 2019-10-13T00:52:02.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.070+0000 7f49e2a67700 1 -- 172.21.15.155:0/1285567115 >> 172.21.15.155:0/1285567115 conn(0x7f49dc099550 msgr2=0x7f49dc061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:02.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.072+0000 7f49e2a67700 1 Processor -- start 2019-10-13T00:52:02.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.072+0000 7f49e2a67700 1 -- start start 2019-10-13T00:52:02.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49e2a67700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.078 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49e2a67700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49dc08fa90 con 0x7f49dc08fe30 2019-10-13T00:52:02.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49e2a67700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49dc0890d0 con 0x7f49dc073990 2019-10-13T00:52:02.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49e2a67700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc093d40 con 0x7f49dc08f660 2019-10-13T00:52:02.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49e1004700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f49dc073990 0x7f49dc08ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49294/0 (socket says 172.21.15.155:49294) 2019-10-13T00:52:02.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49e1004700 1 -- 172.21.15.155:0/2957748589 learned_addr learned my addr 172.21.15.155:0/2957748589 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:02.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150775144 0 0) 0x7f49cc000940 con 0x7f49dc073990 2019-10-13T00:52:02.079 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.073+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc002e70 con 0x7f49dc073990 2019-10-13T00:52:02.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76894 0 0) 0x7f49d0009780 con 0x7f49dc08fe30 2019-10-13T00:52:02.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc0030a0 con 0x7f49dc08fe30 2019-10-13T00:52:02.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 -- 172.21.15.155:0/2957748589 >> v1:172.21.15.155:6789/0 conn(0x7f49dc073990 legacy=0x7f49dc08ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 -- 172.21.15.155:0/2957748589 >> v1:172.21.15.142:6789/0 conn(0x7f49dc08fe30 legacy=0x7f49dc153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 -- 172.21.15.155:0/2957748589 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc08e560 con 0x7f49dc08f660 2019-10-13T00:52:02.080 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.081 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.081 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.081 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.081 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49d4001270 con 0x7f49dc08f660 2019-10-13T00:52:02.081 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.081 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d40045b0 con 0x7f49dc08f660 2019-10-13T00:52:02.081 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.082 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49d4001270 con 0x7f49dc08f660 2019-10-13T00:52:02.082 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49dc08e560 con 0x7f49dc08f660 2019-10-13T00:52:02.082 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc08e560 con 0x7f49dc08f660 2019-10-13T00:52:02.082 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.074+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.082 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.075+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f49d4000e60 con 0x7f49dc08f660 2019-10-13T00:52:02.082 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.075+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.082 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.075+0000 7f49d9ffb700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f49bc017c80 0x7f49bc01a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.083 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.075+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f49d40305b0 con 0x7f49dc08f660 2019-10-13T00:52:02.083 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.075+0000 7f49e1004700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f49bc017c80 0x7f49bc01a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.083 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.075+0000 7f49e1004700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f49bc017c80 0x7f49bc01a110 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.083 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.083+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49dc08e560 con 0x7f49dc08f660 2019-10-13T00:52:02.084 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.086+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.084 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.086+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f49d4001580 con 0x7f49dc08f660 2019-10-13T00:52:02.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.253+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f49dc08e560 con 0x7f49dc08f660 2019-10-13T00:52:02.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.254+0000 7f49e1805700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.254+0000 7f49d9ffb700 1 -- 172.21.15.155:0/2957748589 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f49d4000da0 con 0x7f49dc08f660 2019-10-13T00:52:02.253 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:02.254 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f49bc017c80 msgr2=0x7f49bc01a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.254 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f49bc017c80 0x7f49bc01a110 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 msgr2=0x7f49dc157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f49d40106e0 tx=0x7f49d4010a80).stop 2019-10-13T00:52:02.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 shutdown_connections 2019-10-13T00:52:02.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f49bc017c80 0x7f49bc01a110 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 --2- 172.21.15.155:0/2957748589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49dc08f660 0x7f49dc157710 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 shutdown_connections 2019-10-13T00:52:02.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 wait complete. 2019-10-13T00:52:02.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.256+0000 7f49e2a67700 1 -- 172.21.15.155:0/2957748589 >> 172.21.15.155:0/2957748589 conn(0x7f49dc06d080 msgr2=0x7f49dc06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:02.264 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2019-10-13T00:52:02.264 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:02.264 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-10-13T00:52:02.390 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.388+0000 7f72e7468700 1 Processor -- start 2019-10-13T00:52:02.391 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.388+0000 7f72e7468700 1 -- start start 2019-10-13T00:52:02.391 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e7468700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0099b10 0x7f72e0099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:02.391 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e7468700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f72e009d620 0x7f72e009a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:02.391 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e7468700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e0064f00 con 0x7f72e009c7f0 2019-10-13T00:52:02.391 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e7468700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e009ac50 con 0x7f72e009d620 2019-10-13T00:52:02.391 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e7468700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e009af20 con 0x7f72e0099b10 2019-10-13T00:52:02.392 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e6206700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f72e009c7f0 0x7f72e009cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49302/0 (socket says 172.21.15.155:49302) 2019-10-13T00:52:02.392 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e6206700 1 -- 172.21.15.155:0/3884328909 learned_addr learned my addr 172.21.15.155:0/3884328909 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:02.392 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> v2:172.21.15.155:3300/0 conn(0x7f72e009d620 0x7f72e009a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.392 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.389+0000 7f72e5a05700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0099b10 0x7f72e0099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.392 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 -- 172.21.15.155:0/3884328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0099b10 msgr2=0x7f72e0099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:02.392 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0099b10 0x7f72e0099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:02.392 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 -- 172.21.15.155:0/3884328909 >> v1:172.21.15.155:6789/0 conn(0x7f72e009c7f0 legacy=0x7f72e009cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.393 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 -- 172.21.15.155:0/3884328909 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e009bd10 con 0x7f72e009d620 2019-10-13T00:52:02.393 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e4a03700 1 -- 172.21.15.155:0/3884328909 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196671726 0 0) 0x7f72d8001150 con 0x7f72e009c7f0 2019-10-13T00:52:02.393 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> v2:172.21.15.155:3300/0 conn(0x7f72e009d620 0x7f72e009a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f72d000af40 tx=0x7f72d000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.393 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> v2:172.21.15.155:3300/0 conn(0x7f72e009d620 0x7f72e009a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f72d000af40 tx=0x7f72d000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.393 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72e009d620 0x7f72e009a440 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7f72d000af40 tx=0x7f72d000b750).ready entity=mon.1 client_cookie=65b7c02dee76c7e1 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.393 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72e009d620 0x7f72e009a440 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f72d000af40 tx=0x7f72d000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.393 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e4a03700 1 -- 172.21.15.155:0/3884328909 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d0000960 con 0x7f72e009d620 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72e009d620 0x7f72e009a440 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f72d000af40 tx=0x7f72d000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e5204700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72e009d620 0x7f72e009a440 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f72d000af40 tx=0x7f72d000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e4a03700 1 -- 172.21.15.155:0/3884328909 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d0009850 con 0x7f72e009d620 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e4a03700 1 -- 172.21.15.155:0/3884328909 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d0009f20 con 0x7f72e009d620 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e7468700 1 -- 172.21.15.155:0/3884328909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72e009d620 msgr2=0x7f72e009a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.390+0000 7f72e7468700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72e009d620 0x7f72e009a440 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7f72d000af40 tx=0x7f72d000b750).stop 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.391+0000 7f72e7468700 1 -- 172.21.15.155:0/3884328909 shutdown_connections 2019-10-13T00:52:02.394 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.391+0000 7f72e7468700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72e009d620 0x7f72e009a440 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.395 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.391+0000 7f72e7468700 1 --2- 172.21.15.155:0/3884328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0099b10 0x7f72e0099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:02.395 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.391+0000 7f72e7468700 1 -- 172.21.15.155:0/3884328909 shutdown_connections 2019-10-13T00:52:02.395 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.391+0000 7f72e7468700 1 -- 172.21.15.155:0/3884328909 wait complete. 2019-10-13T00:52:02.395 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.391+0000 7f72e7468700 1 -- 172.21.15.155:0/3884328909 >> 172.21.15.155:0/3884328909 conn(0x7f72e0099550 msgr2=0x7f72e0061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:02.398 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.393+0000 7f72e7468700 1 Processor -- start 2019-10-13T00:52:02.398 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.393+0000 7f72e7468700 1 -- start start 2019-10-13T00:52:02.399 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e7468700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.399 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e7468700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f72e0091690 0x7f72e0090c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.399 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e7468700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e008d4b0 con 0x7f72e0091140 2019-10-13T00:52:02.399 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e7468700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e008ed00 con 0x7f72e0091690 2019-10-13T00:52:02.399 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e7468700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e008efd0 con 0x7f72e0073990 2019-10-13T00:52:02.399 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e6206700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f72e0091140 0x7f72e008bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49310/0 (socket says 172.21.15.155:49310) 2019-10-13T00:52:02.399 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e6206700 1 -- 172.21.15.155:0/1041032121 learned_addr learned my addr 172.21.15.155:0/1041032121 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:02.400 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e5204700 1 --2- 172.21.15.155:0/1041032121 >> v2:172.21.15.142:3300/0 conn(0x7f72e0091690 0x7f72e0090c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.400 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.400 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764793785 0 0) 0x7f72d8000f40 con 0x7f72e0091140 2019-10-13T00:52:02.400 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c0002e70 con 0x7f72e0091140 2019-10-13T00:52:02.400 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4097752199 0 0) 0x7f72d8000940 con 0x7f72e0091140 2019-10-13T00:52:02.400 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e5a05700 1 -- 172.21.15.155:0/1041032121 >> v2:172.21.15.142:3300/0 conn(0x7f72e0091690 msgr2=0x7f72e0090c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.400 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> v2:172.21.15.142:3300/0 conn(0x7f72e0091690 0x7f72e0090c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.401 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e5a05700 1 -- 172.21.15.155:0/1041032121 >> v1:172.21.15.155:6789/0 conn(0x7f72e0091140 legacy=0x7f72e008bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.401 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e5a05700 1 -- 172.21.15.155:0/1041032121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e008fdc0 con 0x7f72e0073990 2019-10-13T00:52:02.401 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.394+0000 7f72e5204700 1 --2- 172.21.15.155:0/1041032121 >> v2:172.21.15.142:3300/0 conn(0x7f72e0091690 0x7f72e0090c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:02.401 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.401 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.401 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.401 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d4000cc0 con 0x7f72e0073990 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d40058f0 con 0x7f72e0073990 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d4005fc0 con 0x7f72e0073990 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e008fdc0 con 0x7f72e0073990 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e008fdc0 con 0x7f72e0073990 2019-10-13T00:52:02.402 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.395+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.403 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.396+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f72d40054f0 con 0x7f72e0073990 2019-10-13T00:52:02.403 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.396+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.403 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.396+0000 7f72de7fc700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f72c0017cb0 0x7f72c001a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.403 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.396+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f72d4030330 con 0x7f72e0073990 2019-10-13T00:52:02.403 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.396+0000 7f72e5204700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f72c0017cb0 0x7f72c001a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.403 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.396+0000 7f72e5204700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f72c0017cb0 0x7f72c001a120 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.404 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.404+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e008fdc0 con 0x7f72e0073990 2019-10-13T00:52:02.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.407+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.405 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.407+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f72d4000cc0 con 0x7f72e0073990 2019-10-13T00:52:02.574 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.576+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f72e008fdc0 con 0x7f72e0073990 2019-10-13T00:52:02.574 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.577+0000 7f72e5a05700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.575 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.577+0000 7f72de7fc700 1 -- 172.21.15.155:0/1041032121 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f72d40054f0 con 0x7f72e0073990 2019-10-13T00:52:02.575 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:02.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f72c0017cb0 msgr2=0x7f72c001a120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f72c0017cb0 0x7f72c001a120 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 msgr2=0x7f72e00906c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f72d4010700 tx=0x7f72d4010aa0).stop 2019-10-13T00:52:02.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 shutdown_connections 2019-10-13T00:52:02.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 --2- 172.21.15.155:0/1041032121 >> v2:172.21.15.142:3300/0 conn(0x7f72e0091690 0x7f72e0090c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f72c0017cb0 0x7f72c001a120 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 --2- 172.21.15.155:0/1041032121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f72e0073990 0x7f72e00906c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 shutdown_connections 2019-10-13T00:52:02.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 wait complete. 2019-10-13T00:52:02.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.579+0000 7f72e7468700 1 -- 172.21.15.155:0/1041032121 >> 172.21.15.155:0/1041032121 conn(0x7f72e006d080 msgr2=0x7f72e006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:02.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2019-10-13T00:52:02.587 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:02.587 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-10-13T00:52:02.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.710+0000 7fa003d68700 1 Processor -- start 2019-10-13T00:52:02.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.710+0000 7fa003d68700 1 -- start start 2019-10-13T00:52:02.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa003d68700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc099b10 0x7f9ffc099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:02.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa003d68700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9ffc09d620 0x7f9ffc09a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:02.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa003d68700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc064f00 con 0x7f9ffc09c7f0 2019-10-13T00:52:02.713 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa003d68700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ffc09ac50 con 0x7f9ffc09d620 2019-10-13T00:52:02.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa003d68700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc09af20 con 0x7f9ffc099b10 2019-10-13T00:52:02.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa002b06700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f9ffc09c7f0 0x7f9ffc09cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49316/0 (socket says 172.21.15.155:49316) 2019-10-13T00:52:02.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa002b06700 1 -- 172.21.15.155:0/3556594038 learned_addr learned my addr 172.21.15.155:0/3556594038 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:02.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa002305700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc099b10 0x7f9ffc099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> v2:172.21.15.142:3300/0 conn(0x7f9ffc09d620 0x7f9ffc09a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.714 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa001303700 1 -- 172.21.15.155:0/3556594038 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506564800 0 0) 0x7f9ff4001150 con 0x7f9ffc09c7f0 2019-10-13T00:52:02.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.711+0000 7fa001303700 1 -- 172.21.15.155:0/3556594038 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe8002e70 con 0x7f9ffc09c7f0 2019-10-13T00:52:02.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 -- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc099b10 msgr2=0x7f9ffc099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:02.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc099b10 0x7f9ffc099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:02.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 -- 172.21.15.155:0/3556594038 >> v1:172.21.15.155:6789/0 conn(0x7f9ffc09c7f0 legacy=0x7f9ffc09cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 -- 172.21.15.155:0/3556594038 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc09bd10 con 0x7f9ffc09d620 2019-10-13T00:52:02.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa002305700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc099b10 0x7f9ffc099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:02.715 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> v2:172.21.15.142:3300/0 conn(0x7f9ffc09d620 0x7f9ffc09a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9fec00af40 tx=0x7f9fec00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.716 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> v2:172.21.15.142:3300/0 conn(0x7f9ffc09d620 0x7f9ffc09a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9fec00af40 tx=0x7f9fec00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.716 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc09d620 0x7f9ffc09a440 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f9fec00af40 tx=0x7f9fec00b750).ready entity=mon.0 client_cookie=6c30a3fc29fbd71 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.716 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc09d620 0x7f9ffc09a440 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9fec00af40 tx=0x7f9fec00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.716 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001303700 1 -- 172.21.15.155:0/3556594038 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fec000960 con 0x7f9ffc09d620 2019-10-13T00:52:02.716 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc09d620 0x7f9ffc09a440 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9fec00af40 tx=0x7f9fec00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.716 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001b04700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc09d620 0x7f9ffc09a440 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9fec00af40 tx=0x7f9fec00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.716 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001303700 1 -- 172.21.15.155:0/3556594038 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec009850 con 0x7f9ffc09d620 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa001303700 1 -- 172.21.15.155:0/3556594038 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fec009f20 con 0x7f9ffc09d620 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa003d68700 1 -- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc09d620 msgr2=0x7f9ffc09a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.712+0000 7fa003d68700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc09d620 0x7f9ffc09a440 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f9fec00af40 tx=0x7f9fec00b750).stop 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.713+0000 7fa003d68700 1 -- 172.21.15.155:0/3556594038 shutdown_connections 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.713+0000 7fa003d68700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc09d620 0x7f9ffc09a440 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.713+0000 7fa003d68700 1 --2- 172.21.15.155:0/3556594038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc099b10 0x7f9ffc099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.713+0000 7fa003d68700 1 -- 172.21.15.155:0/3556594038 shutdown_connections 2019-10-13T00:52:02.717 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.713+0000 7fa003d68700 1 -- 172.21.15.155:0/3556594038 wait complete. 2019-10-13T00:52:02.718 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.713+0000 7fa003d68700 1 -- 172.21.15.155:0/3556594038 >> 172.21.15.155:0/3556594038 conn(0x7f9ffc099550 msgr2=0x7f9ffc061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:02.721 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.715+0000 7fa003d68700 1 Processor -- start 2019-10-13T00:52:02.721 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.715+0000 7fa003d68700 1 -- start start 2019-10-13T00:52:02.721 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa003d68700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9ffc073990 0x7f9ffc091710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.721 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa003d68700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc0926e0 0x7f9ffc091c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.722 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa003d68700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc08eac0 con 0x7f9ffc092190 2019-10-13T00:52:02.722 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa003d68700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ffc090780 con 0x7f9ffc073990 2019-10-13T00:52:02.722 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa003d68700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc08fd50 con 0x7f9ffc0926e0 2019-10-13T00:52:02.722 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa002305700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9ffc073990 0x7f9ffc091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.722 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa002b06700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9ffc092190 0x7f9ffc088ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:57340/0 (socket says 172.21.15.155:57340) 2019-10-13T00:52:02.722 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa002b06700 1 -- 172.21.15.155:0/4067718920 learned_addr learned my addr 172.21.15.155:0/4067718920 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:02.723 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.716+0000 7fa001b04700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc0926e0 0x7f9ffc091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.723 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 -- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc0926e0 msgr2=0x7f9ffc091c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.723 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc0926e0 0x7f9ffc091c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.723 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101941452 0 0) 0x7f9ff4001150 con 0x7f9ffc092190 2019-10-13T00:52:02.723 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 -- 172.21.15.155:0/4067718920 >> v1:172.21.15.142:6789/0 conn(0x7f9ffc092190 legacy=0x7f9ffc088ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.723 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 -- 172.21.15.155:0/4067718920 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc090e10 con 0x7f9ffc073990 2019-10-13T00:52:02.724 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> v2:172.21.15.142:3300/0 conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.724 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> v2:172.21.15.142:3300/0 conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.724 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.724 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.724 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ff8000ce0 con 0x7f9ffc073990 2019-10-13T00:52:02.724 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff8005990 con 0x7f9ffc073990 2019-10-13T00:52:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ff8000ce0 con 0x7f9ffc073990 2019-10-13T00:52:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ffc090e10 con 0x7f9ffc073990 2019-10-13T00:52:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc090e10 con 0x7f9ffc073990 2019-10-13T00:52:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.717+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.718+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f9ff8005b80 con 0x7f9ffc073990 2019-10-13T00:52:02.726 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.718+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.726 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.718+0000 7f9ff2ffd700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9fe4017c60 0x7f9fe401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:02.726 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.718+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9ff80303b0 con 0x7f9ffc073990 2019-10-13T00:52:02.726 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.718+0000 7fa001b04700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9fe4017c60 0x7f9fe401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:02.726 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.718+0000 7fa001b04700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9fe4017c60 0x7f9fe401a0f0 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:02.727 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.725+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ffc090e10 con 0x7f9ffc073990 2019-10-13T00:52:02.727 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.729+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.727 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.729+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f9ff80054f0 con 0x7f9ffc073990 2019-10-13T00:52:02.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.900+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9ffc090e10 con 0x7f9ffc073990 2019-10-13T00:52:02.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.900+0000 7fa002305700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:02.899 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.900+0000 7f9ff2ffd700 1 -- 172.21.15.155:0/4067718920 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ff8005b80 con 0x7f9ffc073990 2019-10-13T00:52:02.899 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:02.901 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9fe4017c60 msgr2=0x7f9fe401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.902 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9fe4017c60 0x7f9fe401a0f0 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.902 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 msgr2=0x7f9ffc091710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:02.902 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7f9ff8010700 tx=0x7f9ff8010aa0).stop 2019-10-13T00:52:02.902 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 shutdown_connections 2019-10-13T00:52:02.902 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f9fe4017c60 0x7f9fe401a0f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.902 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ffc0926e0 0x7f9ffc091c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.903 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 --2- 172.21.15.155:0/4067718920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ffc073990 0x7f9ffc091710 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:02.903 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 shutdown_connections 2019-10-13T00:52:02.903 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 wait complete. 2019-10-13T00:52:02.903 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:02.903+0000 7fa003d68700 1 -- 172.21.15.155:0/4067718920 >> 172.21.15.155:0/4067718920 conn(0x7f9ffc06d080 msgr2=0x7f9ffc06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:02.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2019-10-13T00:52:02.911 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:02.911 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-10-13T00:52:03.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.035+0000 7f33be904700 1 Processor -- start 2019-10-13T00:52:03.037 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.035+0000 7f33be904700 1 -- start start 2019-10-13T00:52:03.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33be904700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f33b8099b10 0x7f33b8099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:03.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33be904700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f33b809d620 0x7f33b809a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:03.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33be904700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b8064f00 con 0x7f33b809c7f0 2019-10-13T00:52:03.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33be904700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b809b680 con 0x7f33b809d620 2019-10-13T00:52:03.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33be904700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b809ac50 con 0x7f33b8099b10 2019-10-13T00:52:03.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33bcea1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f33b8099b10 0x7f33b8099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.038 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33bcea1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f33b8099b10 0x7f33b8099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59252/0 (socket says 172.21.15.155:59252) 2019-10-13T00:52:03.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33bcea1700 1 -- 172.21.15.155:0/509485388 learned_addr learned my addr 172.21.15.155:0/509485388 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:03.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.036+0000 7f33b7fff700 1 --2- 172.21.15.155:0/509485388 >> v2:172.21.15.142:3300/0 conn(0x7f33b809d620 0x7f33b809a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 -- 172.21.15.155:0/509485388 >> v2:172.21.15.142:3300/0 conn(0x7f33b809d620 msgr2=0x7f33b809a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:03.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 --2- 172.21.15.155:0/509485388 >> v2:172.21.15.142:3300/0 conn(0x7f33b809d620 0x7f33b809a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:03.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 -- 172.21.15.155:0/509485388 >> v1:172.21.15.142:6789/0 conn(0x7f33b809c7f0 legacy=0x7f33b809cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 -- 172.21.15.155:0/509485388 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b809bd10 con 0x7f33b8099b10 2019-10-13T00:52:03.039 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 --2- 172.21.15.155:0/509485388 >> v2:172.21.15.155:3300/0 conn(0x7f33b8099b10 0x7f33b8099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f33a800f730 tx=0x7f33a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 --2- 172.21.15.155:0/509485388 >> v2:172.21.15.155:3300/0 conn(0x7f33b8099b10 0x7f33b8099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f33a800f730 tx=0x7f33a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 --2- 172.21.15.155:0/509485388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8099b10 0x7f33b8099f00 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7f33a800f730 tx=0x7f33a800fad0).ready entity=mon.1 client_cookie=15a66a22ea6aeefa server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 --2- 172.21.15.155:0/509485388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8099b10 0x7f33b8099f00 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f33a800f730 tx=0x7f33a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 --2- 172.21.15.155:0/509485388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8099b10 0x7f33b8099f00 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f33a800f730 tx=0x7f33a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.040 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33bcea1700 1 --2- 172.21.15.155:0/509485388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8099b10 0x7f33b8099f00 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f33a800f730 tx=0x7f33a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33b77fe700 1 -- 172.21.15.155:0/509485388 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33a8000960 con 0x7f33b8099b10 2019-10-13T00:52:03.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33b77fe700 1 -- 172.21.15.155:0/509485388 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a8000c90 con 0x7f33b8099b10 2019-10-13T00:52:03.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33b77fe700 1 -- 172.21.15.155:0/509485388 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33a8001360 con 0x7f33b8099b10 2019-10-13T00:52:03.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33be904700 1 -- 172.21.15.155:0/509485388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8099b10 msgr2=0x7f33b8099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.041 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.037+0000 7f33be904700 1 --2- 172.21.15.155:0/509485388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8099b10 0x7f33b8099f00 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7f33a800f730 tx=0x7f33a800fad0).stop 2019-10-13T00:52:03.042 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.038+0000 7f33be904700 1 -- 172.21.15.155:0/509485388 shutdown_connections 2019-10-13T00:52:03.042 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.038+0000 7f33be904700 1 --2- 172.21.15.155:0/509485388 >> v2:172.21.15.142:3300/0 conn(0x7f33b809d620 0x7f33b809a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:03.042 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.038+0000 7f33be904700 1 --2- 172.21.15.155:0/509485388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8099b10 0x7f33b8099f00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.042 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.038+0000 7f33be904700 1 -- 172.21.15.155:0/509485388 shutdown_connections 2019-10-13T00:52:03.042 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.038+0000 7f33be904700 1 -- 172.21.15.155:0/509485388 wait complete. 2019-10-13T00:52:03.042 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.038+0000 7f33be904700 1 -- 172.21.15.155:0/509485388 >> 172.21.15.155:0/509485388 conn(0x7f33b8099550 msgr2=0x7f33b8061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:03.045 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.040+0000 7f33be904700 1 Processor -- start 2019-10-13T00:52:03.046 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.040+0000 7f33be904700 1 -- start start 2019-10-13T00:52:03.046 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33be904700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33b8073990 0x7f33b8091710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:03.046 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33be904700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f33b8154020 0x7f33b8156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:03.046 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33be904700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b808eac0 con 0x7f33b80926e0 2019-10-13T00:52:03.046 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33be904700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b808fd50 con 0x7f33b8154020 2019-10-13T00:52:03.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33be904700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b8090020 con 0x7f33b8073990 2019-10-13T00:52:03.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33b7fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f33b80926e0 0x7f33b8091c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49334/0 (socket says 172.21.15.155:49334) 2019-10-13T00:52:03.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33b7fff700 1 -- 172.21.15.155:0/1998302957 learned_addr learned my addr 172.21.15.155:0/1998302957 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:03.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> v2:172.21.15.155:3300/0 conn(0x7f33b8154020 0x7f33b8156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bcea1700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33b8073990 0x7f33b8091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 -- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33b8073990 msgr2=0x7f33b8091710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.047 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33b8073990 0x7f33b8091710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 -- 172.21.15.155:0/1998302957 >> v1:172.21.15.155:6789/0 conn(0x7f33b80926e0 legacy=0x7f33b8091c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 -- 172.21.15.155:0/1998302957 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b8090e10 con 0x7f33b8154020 2019-10-13T00:52:03.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172419380 0 0) 0x7f33ac009990 con 0x7f33b80926e0 2019-10-13T00:52:03.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> v2:172.21.15.155:3300/0 conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> v2:172.21.15.155:3300/0 conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.048 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33b0000ce0 con 0x7f33b8154020 2019-10-13T00:52:03.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.041+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b000f150 con 0x7f33b8154020 2019-10-13T00:52:03.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33b000f860 con 0x7f33b8154020 2019-10-13T00:52:03.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b8090e10 con 0x7f33b8154020 2019-10-13T00:52:03.049 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b8090e10 con 0x7f33b8154020 2019-10-13T00:52:03.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f33b0000cc0 con 0x7f33b8154020 2019-10-13T00:52:03.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33b57fa700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3398017c60 0x7f339801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:03.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.042+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f33b00303e0 con 0x7f33b8154020 2019-10-13T00:52:03.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.043+0000 7f33bcea1700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3398017c60 0x7f339801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.050 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.043+0000 7f33bcea1700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3398017c60 0x7f339801a0f0 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.051 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.051+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b8090e10 con 0x7f33b8154020 2019-10-13T00:52:03.052 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.054+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.052 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.054+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f33b000ed30 con 0x7f33b8154020 2019-10-13T00:52:03.223 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.223+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f33b8090e10 con 0x7f33b8154020 2019-10-13T00:52:03.223 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.223+0000 7f33bd6a2700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.223 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.223+0000 7f33b57fa700 1 -- 172.21.15.155:0/1998302957 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f33b0000cc0 con 0x7f33b8154020 2019-10-13T00:52:03.223 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:03.224 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3398017c60 msgr2=0x7f339801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.224 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3398017c60 0x7f339801a0f0 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.225 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 msgr2=0x7f33b8156420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.225 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f33b0010720 tx=0x7f33b0010ac0).stop 2019-10-13T00:52:03.225 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 shutdown_connections 2019-10-13T00:52:03.225 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33b8154020 0x7f33b8156420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.225 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3398017c60 0x7f339801a0f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.225 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 --2- 172.21.15.155:0/1998302957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33b8073990 0x7f33b8091710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.225 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 shutdown_connections 2019-10-13T00:52:03.226 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 wait complete. 2019-10-13T00:52:03.226 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.226+0000 7f33be904700 1 -- 172.21.15.155:0/1998302957 >> 172.21.15.155:0/1998302957 conn(0x7f33b806d080 msgr2=0x7f33b806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:03.234 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2019-10-13T00:52:03.234 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:03.234 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-10-13T00:52:03.360 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.358+0000 7fc720a03700 1 Processor -- start 2019-10-13T00:52:03.360 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.358+0000 7fc720a03700 1 -- start start 2019-10-13T00:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc720a03700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc720a03700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c064f00 con 0x7fc71c09d620 2019-10-13T00:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc720a03700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c0987e0 con 0x7fc71c09cff0 2019-10-13T00:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc720a03700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c09ac50 con 0x7fc71c099b10 2019-10-13T00:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71b59e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc71c09cff0 0x7fc71c152c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49344/0 (socket says 172.21.15.155:49344) 2019-10-13T00:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71b59e700 1 -- 172.21.15.155:0/3958273197 learned_addr learned my addr 172.21.15.155:0/3958273197 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357313922 0 0) 0x7fc714001150 con 0x7fc71c09cff0 2019-10-13T00:52:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6fc002e70 con 0x7fc71c09cff0 2019-10-13T00:52:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083687453 0 0) 0x7fc70c0013b0 con 0x7fc71c09d620 2019-10-13T00:52:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6fc0030a0 con 0x7fc71c09d620 2019-10-13T00:52:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2838337086 0 0) 0x7fc714001130 con 0x7fc71c09cff0 2019-10-13T00:52:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71ad9d700 1 -- 172.21.15.155:0/3958273197 >> v1:172.21.15.155:6789/0 conn(0x7fc71c09cff0 legacy=0x7fc71c152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71ad9d700 1 -- 172.21.15.155:0/3958273197 >> v1:172.21.15.142:6789/0 conn(0x7fc71c09d620 legacy=0x7fc71c09c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71ad9d700 1 -- 172.21.15.155:0/3958273197 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c09bd10 con 0x7fc71c099b10 2019-10-13T00:52:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc70400af40 tx=0x7fc70400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc70400af40 tx=0x7fc70400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.359+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7fc70400af40 tx=0x7fc70400b750).ready entity=mon.2 client_cookie=eadcd477e49295bc server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fc70400af40 tx=0x7fc70400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fc70400af40 tx=0x7fc70400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc704000960 con 0x7fc71c099b10 2019-10-13T00:52:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fc70400af40 tx=0x7fc70400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704009850 con 0x7fc71c099b10 2019-10-13T00:52:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc719d9b700 1 -- 172.21.15.155:0/3958273197 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc704009f20 con 0x7fc71c099b10 2019-10-13T00:52:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc720a03700 1 -- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 msgr2=0x7fc71c099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc720a03700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7fc70400af40 tx=0x7fc70400b750).stop 2019-10-13T00:52:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc720a03700 1 -- 172.21.15.155:0/3958273197 shutdown_connections 2019-10-13T00:52:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc720a03700 1 --2- 172.21.15.155:0/3958273197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c099b10 0x7fc71c099f00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.365 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.360+0000 7fc720a03700 1 -- 172.21.15.155:0/3958273197 shutdown_connections 2019-10-13T00:52:03.365 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.361+0000 7fc720a03700 1 -- 172.21.15.155:0/3958273197 wait complete. 2019-10-13T00:52:03.365 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.361+0000 7fc720a03700 1 -- 172.21.15.155:0/3958273197 >> 172.21.15.155:0/3958273197 conn(0x7fc71c099550 msgr2=0x7fc71c061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:03.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc720a03700 1 Processor -- start 2019-10-13T00:52:03.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc720a03700 1 -- start start 2019-10-13T00:52:03.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc720a03700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:03.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc720a03700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c08d4b0 con 0x7fc71c152ee0 2019-10-13T00:52:03.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc720a03700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c08de90 con 0x7fc71c091690 2019-10-13T00:52:03.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc720a03700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c08ed00 con 0x7fc71c073990 2019-10-13T00:52:03.369 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc71a59c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc71c091690 0x7fc71c090c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49348/0 (socket says 172.21.15.155:49348) 2019-10-13T00:52:03.370 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.363+0000 7fc71a59c700 1 -- 172.21.15.155:0/714527822 learned_addr learned my addr 172.21.15.155:0/714527822 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:03.370 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.370 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171303214 0 0) 0x7fc70c0015c0 con 0x7fc71c091690 2019-10-13T00:52:03.371 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f4002e70 con 0x7fc71c091690 2019-10-13T00:52:03.371 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760386618 0 0) 0x7fc7140009b0 con 0x7fc71c152ee0 2019-10-13T00:52:03.371 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f40030a0 con 0x7fc71c152ee0 2019-10-13T00:52:03.371 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 -- 172.21.15.155:0/714527822 >> v1:172.21.15.155:6789/0 conn(0x7fc71c091690 legacy=0x7fc71c090c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.371 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 -- 172.21.15.155:0/714527822 >> v1:172.21.15.142:6789/0 conn(0x7fc71c152ee0 legacy=0x7fc71c1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.371 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 -- 172.21.15.155:0/714527822 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c08fdc0 con 0x7fc71c073990 2019-10-13T00:52:03.371 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.372 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.372 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.372 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.372 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc704009780 con 0x7fc71c073990 2019-10-13T00:52:03.372 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.372 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.372 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70400aea0 con 0x7fc71c073990 2019-10-13T00:52:03.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc704009d60 con 0x7fc71c073990 2019-10-13T00:52:03.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c08fdc0 con 0x7fc71c073990 2019-10-13T00:52:03.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.364+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c08fdc0 con 0x7fc71c073990 2019-10-13T00:52:03.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.365+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.365+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fc704009780 con 0x7fc71c073990 2019-10-13T00:52:03.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.365+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.373 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.365+0000 7fc7137fe700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc6f4017c80 0x7fc6f401a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:03.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.365+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc70402ffd0 con 0x7fc71c073990 2019-10-13T00:52:03.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.365+0000 7fc71a59c700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc6f4017c80 0x7fc6f401a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.365+0000 7fc71a59c700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc6f4017c80 0x7fc6f401a110 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.374 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.373+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c08fdc0 con 0x7fc71c073990 2019-10-13T00:52:03.375 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.376+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.375 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.377+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fc704027f90 con 0x7fc71c073990 2019-10-13T00:52:03.545 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.547+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc71c08fdc0 con 0x7fc71c073990 2019-10-13T00:52:03.546 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.548+0000 7fc71ad9d700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.546 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.548+0000 7fc7137fe700 1 -- 172.21.15.155:0/714527822 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc704027f90 con 0x7fc71c073990 2019-10-13T00:52:03.546 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:03.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc6f4017c80 msgr2=0x7fc6f401a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc6f4017c80 0x7fc6f401a110 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 msgr2=0x7fc71c0906c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7fc704010380 tx=0x7fc704010720).stop 2019-10-13T00:52:03.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 shutdown_connections 2019-10-13T00:52:03.549 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc6f4017c80 0x7fc6f401a110 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.550 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 --2- 172.21.15.155:0/714527822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc71c073990 0x7fc71c0906c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.550 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 shutdown_connections 2019-10-13T00:52:03.550 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 wait complete. 2019-10-13T00:52:03.550 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.550+0000 7fc720a03700 1 -- 172.21.15.155:0/714527822 >> 172.21.15.155:0/714527822 conn(0x7fc71c06d080 msgr2=0x7fc71c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:03.557 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2019-10-13T00:52:03.558 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:03.558 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-10-13T00:52:03.685 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.682+0000 7f3e9f142700 1 Processor -- start 2019-10-13T00:52:03.685 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.682+0000 7f3e9f142700 1 -- start start 2019-10-13T00:52:03.685 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9f142700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3e9809a700 0x7f3e9809c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:03.685 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9f142700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e98064f00 con 0x7f3e9809d620 2019-10-13T00:52:03.685 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9f142700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e980987e0 con 0x7f3e98099b10 2019-10-13T00:52:03.685 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9f142700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9809b680 con 0x7f3e9809a700 2019-10-13T00:52:03.686 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9cede700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3e9809a700 0x7f3e9809c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.686 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9d6df700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3e98099b10 0x7f3e98099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49356/0 (socket says 172.21.15.155:49356) 2019-10-13T00:52:03.686 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9d6df700 1 -- 172.21.15.155:0/1925575763 learned_addr learned my addr 172.21.15.155:0/1925575763 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:03.686 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9cede700 1 -- 172.21.15.155:0/1925575763 >> v1:172.21.15.155:6789/0 conn(0x7f3e98099b10 legacy=0x7f3e98099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.686 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9cede700 1 -- 172.21.15.155:0/1925575763 >> v1:172.21.15.142:6789/0 conn(0x7f3e9809d620 legacy=0x7f3e98152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.686 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e8ffff700 1 -- 172.21.15.155:0/1925575763 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903955065 0 0) 0x7f3e880014d0 con 0x7f3e98099b10 2019-10-13T00:52:03.686 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9cede700 1 -- 172.21.15.155:0/1925575763 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9809bd10 con 0x7f3e9809a700 2019-10-13T00:52:03.687 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.683+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1925575763 >> v2:172.21.15.155:3300/0 conn(0x7f3e9809a700 0x7f3e9809c7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3e9400af40 tx=0x7f3e9400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.687 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1925575763 >> v2:172.21.15.155:3300/0 conn(0x7f3e9809a700 0x7f3e9809c7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3e9400af40 tx=0x7f3e9400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.687 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1925575763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e9809a700 0x7f3e9809c7f0 secure :-1 s=READY pgs=148 cs=0 l=1 rx=0x7f3e9400af40 tx=0x7f3e9400b750).ready entity=mon.1 client_cookie=6432939c58292e1d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.687 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1925575763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e9809a700 0x7f3e9809c7f0 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f3e9400af40 tx=0x7f3e9400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.687 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1925575763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e9809a700 0x7f3e9809c7f0 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f3e9400af40 tx=0x7f3e9400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.687 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e8ffff700 1 -- 172.21.15.155:0/1925575763 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e94000960 con 0x7f3e9809a700 2019-10-13T00:52:03.687 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1925575763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e9809a700 0x7f3e9809c7f0 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f3e9400af40 tx=0x7f3e9400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.688 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e8ffff700 1 -- 172.21.15.155:0/1925575763 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e94009850 con 0x7f3e9809a700 2019-10-13T00:52:03.688 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e8ffff700 1 -- 172.21.15.155:0/1925575763 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e94009f20 con 0x7f3e9809a700 2019-10-13T00:52:03.688 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9f142700 1 -- 172.21.15.155:0/1925575763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e9809a700 msgr2=0x7f3e9809c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.688 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9f142700 1 --2- 172.21.15.155:0/1925575763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e9809a700 0x7f3e9809c7f0 secure :-1 s=READY pgs=148 cs=0 l=1 rx=0x7f3e9400af40 tx=0x7f3e9400b750).stop 2019-10-13T00:52:03.688 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9f142700 1 -- 172.21.15.155:0/1925575763 shutdown_connections 2019-10-13T00:52:03.688 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9f142700 1 --2- 172.21.15.155:0/1925575763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e9809a700 0x7f3e9809c7f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.688 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9f142700 1 -- 172.21.15.155:0/1925575763 shutdown_connections 2019-10-13T00:52:03.689 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9f142700 1 -- 172.21.15.155:0/1925575763 wait complete. 2019-10-13T00:52:03.689 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.684+0000 7f3e9f142700 1 -- 172.21.15.155:0/1925575763 >> 172.21.15.155:0/1925575763 conn(0x7f3e98099550 msgr2=0x7f3e98061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:03.689 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.687+0000 7f3e9f142700 1 Processor -- start 2019-10-13T00:52:03.689 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.687+0000 7f3e9f142700 1 -- start start 2019-10-13T00:52:03.693 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9f142700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3e98073990 0x7f3e9808ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:03.693 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9f142700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e980890d0 con 0x7f3e9808fe30 2019-10-13T00:52:03.693 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9f142700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e98088bd0 con 0x7f3e98153ef0 2019-10-13T00:52:03.693 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9f142700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e98093d40 con 0x7f3e98073990 2019-10-13T00:52:03.693 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9dee0700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3e98153ef0 0x7f3e981562f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49366/0 (socket says 172.21.15.155:49366) 2019-10-13T00:52:03.693 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9dee0700 1 -- 172.21.15.155:0/1075681864 learned_addr learned my addr 172.21.15.155:0/1075681864 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:03.693 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> v2:172.21.15.142:3300/0 conn(0x7f3e98073990 0x7f3e9808ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993858193 0 0) 0x7f3e90001140 con 0x7f3e98153ef0 2019-10-13T00:52:03.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e78002e70 con 0x7f3e98153ef0 2019-10-13T00:52:03.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746691635 0 0) 0x7f3e94000940 con 0x7f3e9808fe30 2019-10-13T00:52:03.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e780030a0 con 0x7f3e9808fe30 2019-10-13T00:52:03.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 -- 172.21.15.155:0/1075681864 >> v1:172.21.15.155:6789/0 conn(0x7f3e98153ef0 legacy=0x7f3e981562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 -- 172.21.15.155:0/1075681864 >> v1:172.21.15.142:6789/0 conn(0x7f3e9808fe30 legacy=0x7f3e9808f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.694 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 -- 172.21.15.155:0/1075681864 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9808e560 con 0x7f3e98073990 2019-10-13T00:52:03.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> v2:172.21.15.142:3300/0 conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> v2:172.21.15.142:3300/0 conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e88001270 con 0x7f3e98073990 2019-10-13T00:52:03.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.695 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e880045b0 con 0x7f3e98073990 2019-10-13T00:52:03.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e88004c80 con 0x7f3e98073990 2019-10-13T00:52:03.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.688+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9808e560 con 0x7f3e98073990 2019-10-13T00:52:03.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.689+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9808e560 con 0x7f3e98073990 2019-10-13T00:52:03.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.689+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.689+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f3e880047a0 con 0x7f3e98073990 2019-10-13T00:52:03.696 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.689+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.689+0000 7f3e8dffb700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3e78017c80 0x7f3e7801a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:03.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.689+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3e88030340 con 0x7f3e98073990 2019-10-13T00:52:03.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.689+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3e78017c80 0x7f3e7801a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:03.697 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.690+0000 7f3e9cede700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3e78017c80 0x7f3e7801a110 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:03.698 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.697+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9808e560 con 0x7f3e98073990 2019-10-13T00:52:03.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.701+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.699 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.701+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f3e88001010 con 0x7f3e98073990 2019-10-13T00:52:03.865 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.868+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3e9808e560 con 0x7f3e98073990 2019-10-13T00:52:03.866 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.868+0000 7f3e9d6df700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:03.866 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.868+0000 7f3e8dffb700 1 -- 172.21.15.155:0/1075681864 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3e880047a0 con 0x7f3e98073990 2019-10-13T00:52:03.867 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2019-10-13T00:52:03.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.870+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3e78017c80 msgr2=0x7f3e7801a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.870+0000 7f3e9f142700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3e78017c80 0x7f3e7801a110 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.870+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 msgr2=0x7f3e9808ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:03.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.870+0000 7f3e9f142700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f3e880106e0 tx=0x7f3e88010a80).stop 2019-10-13T00:52:03.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.870+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 shutdown_connections 2019-10-13T00:52:03.869 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.870+0000 7f3e9f142700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3e78017c80 0x7f3e7801a110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.870 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.870+0000 7f3e9f142700 1 --2- 172.21.15.155:0/1075681864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e98073990 0x7f3e9808ee60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:03.870 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.871+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 shutdown_connections 2019-10-13T00:52:03.870 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.871+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 wait complete. 2019-10-13T00:52:03.870 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.871+0000 7f3e9f142700 1 -- 172.21.15.155:0/1075681864 >> 172.21.15.155:0/1075681864 conn(0x7f3e9806d080 msgr2=0x7f3e9806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:03.878 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2019-10-13T00:52:03.878 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-10-13T00:52:03.879 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:03.879 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-13T00:52:04.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.998+0000 7f585aa35700 1 Processor -- start 2019-10-13T00:52:04.005 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.998+0000 7f585aa35700 1 -- start start 2019-10-13T00:52:04.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f585aa35700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585409a700 0x7f585409c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f585aa35700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5854064f00 con 0x7f585409d620 2019-10-13T00:52:04.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f585aa35700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58540987e0 con 0x7f5854099b10 2019-10-13T00:52:04.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f585aa35700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585409b680 con 0x7f585409a700 2019-10-13T00:52:04.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f5858fd2700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5854099b10 0x7f5854099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49370/0 (socket says 172.21.15.155:49370) 2019-10-13T00:52:04.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f5858fd2700 1 -- 172.21.15.155:0/4117197746 learned_addr learned my addr 172.21.15.155:0/4117197746 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:04.006 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585409a700 0x7f585409c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440474663 0 0) 0x7f5844001150 con 0x7f5854099b10 2019-10-13T00:52:04.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:03.999+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f583c002e70 con 0x7f5854099b10 2019-10-13T00:52:04.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2725875112 0 0) 0x7f5844001130 con 0x7f5854099b10 2019-10-13T00:52:04.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585409a700 msgr2=0x7f585409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:04.007 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585409a700 0x7f585409c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 >> v1:172.21.15.142:6789/0 conn(0x7f585409d620 legacy=0x7f5854152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585409bd10 con 0x7f5854099b10 2019-10-13T00:52:04.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f5844001660 con 0x7f5854099b10 2019-10-13T00:52:04.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 >> v1:172.21.15.155:6789/0 conn(0x7f5854099b10 legacy=0x7f5854099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.008 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f583c003b10 0x7f583c005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f583c009430 0x7f583c00b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c00bfe0 con 0x7f583c003b10 2019-10-13T00:52:04.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0501e0 con 0x7f583c006700 2019-10-13T00:52:04.009 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c050410 con 0x7f583c009430 2019-10-13T00:52:04.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5858fd2700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f583c003b10 0x7f583c005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58597d3700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f583c009430 0x7f583c00b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.010 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 -- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f583c009430 msgr2=0x7f583c00b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:04.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f583c009430 0x7f583c00b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 -- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f583c003b10 msgr2=0x7f583c005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:04.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f583c003b10 0x7f583c005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 -- 172.21.15.155:0/4117197746 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0506c0 con 0x7f583c006700 2019-10-13T00:52:04.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5848002c70 tx=0x7f584800f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.011 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5848002c70 tx=0x7f584800f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 secure :-1 s=READY pgs=151 cs=0 l=1 rx=0x7f5848002c70 tx=0x7f584800f9b0).ready entity=mon.1 client_cookie=15265f083fb9294e server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7f5848002c70 tx=0x7f584800f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5848000940 con 0x7f583c006700 2019-10-13T00:52:04.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7f5848002c70 tx=0x7f584800f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.012 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f5853fff700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7f5848002c70 tx=0x7f584800f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.013 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5848000ea0 con 0x7f583c006700 2019-10-13T00:52:04.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58537fe700 1 -- 172.21.15.155:0/4117197746 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5848001570 con 0x7f583c006700 2019-10-13T00:52:04.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f585aa35700 1 -- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 msgr2=0x7f583c008bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f585aa35700 1 --2- 172.21.15.155:0/4117197746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f583c006700 0x7f583c008bd0 secure :-1 s=READY pgs=151 cs=0 l=1 rx=0x7f5848002c70 tx=0x7f584800f9b0).stop 2019-10-13T00:52:04.014 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.000+0000 7f58597d3700 1 -- 172.21.15.155:0/4117197746 reap_dead start 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.001+0000 7f585aa35700 1 -- 172.21.15.155:0/4117197746 shutdown_connections 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.001+0000 7f585aa35700 1 -- 172.21.15.155:0/4117197746 shutdown_connections 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.001+0000 7f585aa35700 1 -- 172.21.15.155:0/4117197746 wait complete. 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.001+0000 7f585aa35700 1 -- 172.21.15.155:0/4117197746 >> 172.21.15.155:0/4117197746 conn(0x7f5854099550 msgr2=0x7f5854061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.003+0000 7f585aa35700 1 Processor -- start 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.003+0000 7f585aa35700 1 -- start start 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f585aa35700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5854088da0 0x7f5854092450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.015 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f585aa35700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5854093420 0x7f5854092990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f585aa35700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58540901f0 con 0x7f5854092ed0 2019-10-13T00:52:04.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f585aa35700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5854090a50 con 0x7f5854093420 2019-10-13T00:52:04.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f585aa35700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5854090d20 con 0x7f5854088da0 2019-10-13T00:52:04.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f5853fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5854093420 0x7f5854092990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f5853fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5854093420 0x7f5854092990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59306/0 (socket says 172.21.15.155:59306) 2019-10-13T00:52:04.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f58597d3700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5854092ed0 0x7f585408e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49382/0 (socket says 172.21.15.155:49382) 2019-10-13T00:52:04.016 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f5853fff700 1 -- 172.21.15.155:0/729566152 learned_addr learned my addr 172.21.15.155:0/729566152 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:04.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.004+0000 7f5858fd2700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5854088da0 0x7f5854092450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 -- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5854088da0 msgr2=0x7f5854092450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5854088da0 0x7f5854092450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 -- 172.21.15.155:0/729566152 >> v1:172.21.15.155:6789/0 conn(0x7f5854092ed0 legacy=0x7f585408e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984826365 0 0) 0x7f584c0099a0 con 0x7f5854092ed0 2019-10-13T00:52:04.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 -- 172.21.15.155:0/729566152 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5854091b50 con 0x7f5854093420 2019-10-13T00:52:04.017 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> v2:172.21.15.155:3300/0 conn(0x7f5854093420 0x7f5854092990 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> v2:172.21.15.155:3300/0 conn(0x7f5854093420 0x7f5854092990 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5848000940 con 0x7f5854093420 2019-10-13T00:52:04.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.018 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5848001090 con 0x7f5854093420 2019-10-13T00:52:04.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5848001760 con 0x7f5854093420 2019-10-13T00:52:04.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5854091b50 con 0x7f5854093420 2019-10-13T00:52:04.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5854091b50 con 0x7f5854093420 2019-10-13T00:52:04.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.005+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.006+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5848000c90 con 0x7f5854093420 2019-10-13T00:52:04.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.006+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.019 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.006+0000 7f58517fa700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5834017c60 0x7f583401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.006+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f58480300e0 con 0x7f5854093420 2019-10-13T00:52:04.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.006+0000 7f5858fd2700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5834017c60 0x7f583401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.020 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.006+0000 7f5858fd2700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5834017c60 0x7f583401a0f0 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.013+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5854091b50 con 0x7f5854093420 2019-10-13T00:52:04.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.017+0000 7f5853fff700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.021 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.017+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f5848000940 con 0x7f5854093420 2019-10-13T00:52:04.185 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.187+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 --> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5854091b50 con 0x7f5834017c60 2019-10-13T00:52:04.189 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.190+0000 7f58517fa700 1 -- 172.21.15.155:0/729566152 <== mgr.4110 v2:172.21.15.142:6800/5662 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80478 (crc 0 0 0) 0x7f5844000ef0 con 0x7f5834017c60 2019-10-13T00:52:04.189 INFO:teuthology.orchestra.run.smithi155.stderr:dumped all 2019-10-13T00:52:04.189 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:52:04.191 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5834017c60 msgr2=0x7f583401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.191 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5834017c60 0x7f583401a0f0 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.191 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 msgr2=0x7f5854092990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.191 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7f5848010440 tx=0x7f58480107e0).stop 2019-10-13T00:52:04.191 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 shutdown_connections 2019-10-13T00:52:04.191 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5854093420 0x7f5854092990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.192 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f5834017c60 0x7f583401a0f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.192 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 --2- 172.21.15.155:0/729566152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5854088da0 0x7f5854092450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.192 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 shutdown_connections 2019-10-13T00:52:04.193 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 wait complete. 2019-10-13T00:52:04.193 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.192+0000 7f585aa35700 1 -- 172.21.15.155:0/729566152 >> 172.21.15.155:0/729566152 conn(0x7f585406bfe0 msgr2=0x7f585406d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:04.201 INFO:teuthology.orchestra.run.smithi155.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2019-10-13T00:52:03.780651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8395136,"kb_used_data":6400,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746579584,"statfs":{"total":773094113280,"available":764497494016,"internally_reserved":8589934592,"allocated":6553600,"data_stored":719444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853835+0000","last_change":"2019-10-13T00:51:46.833481+0000","last_active":"2019-10-13T00:51:52.853835+0000","last_peered":"2019-10-13T00:51:52.853835+0000","last_clean":"2019-10-13T00:51:52.853835+0000","last_became_active":"2019-10-13T00:51:46.833140+0000","last_became_peered":"2019-10-13T00:51:46.833140+0000","last_unstale":"2019-10-13T00:51:52.853835+0000","last_undegraded":"2019-10-13T00:51:52.853835+0000","last_fullsized":"2019-10-13T00:51:52.853835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"23'8","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.896759+0000","last_change":"2019-10-13T00:51:49.850206+0000","last_active":"2019-10-13T00:51:59.896759+0000","last_peered":"2019-10-13T00:51:59.896759+0000","last_clean":"2019-10-13T00:51:59.896759+0000","last_became_active":"2019-10-13T00:51:49.849825+0000","last_became_peered":"2019-10-13T00:51:49.849825+0000","last_unstale":"2019-10-13T00:51:59.896759+0000","last_undegraded":"2019-10-13T00:51:59.896759+0000","last_fullsized":"2019-10-13T00:51:59.896759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.860011+0000","last_change":"2019-10-13T00:51:52.859165+0000","last_active":"2019-10-13T00:51:52.860011+0000","last_peered":"2019-10-13T00:51:52.860011+0000","last_clean":"2019-10-13T00:51:52.860011+0000","last_became_active":"2019-10-13T00:51:52.858864+0000","last_became_peered":"2019-10-13T00:51:52.858864+0000","last_unstale":"2019-10-13T00:51:52.860011+0000","last_undegraded":"2019-10-13T00:51:52.860011+0000","last_fullsized":"2019-10-13T00:51:52.860011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4,3],"acting":[2,6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853169+0000","last_change":"2019-10-13T00:51:46.830036+0000","last_active":"2019-10-13T00:51:52.853169+0000","last_peered":"2019-10-13T00:51:52.853169+0000","last_clean":"2019-10-13T00:51:52.853169+0000","last_became_active":"2019-10-13T00:51:46.829735+0000","last_became_peered":"2019-10-13T00:51:46.829735+0000","last_unstale":"2019-10-13T00:51:52.853169+0000","last_undegraded":"2019-10-13T00:51:52.853169+0000","last_fullsized":"2019-10-13T00:51:52.853169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"23'6","reported_seq":"18","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901377+0000","last_change":"2019-10-13T00:51:49.852008+0000","last_active":"2019-10-13T00:51:59.901377+0000","last_peered":"2019-10-13T00:51:59.901377+0000","last_clean":"2019-10-13T00:51:59.901377+0000","last_became_active":"2019-10-13T00:51:49.851773+0000","last_became_peered":"2019-10-13T00:51:49.851773+0000","last_unstale":"2019-10-13T00:51:59.901377+0000","last_undegraded":"2019-10-13T00:51:59.901377+0000","last_fullsized":"2019-10-13T00:51:59.901377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.862466+0000","last_change":"2019-10-13T00:51:52.859449+0000","last_active":"2019-10-13T00:51:52.862466+0000","last_peered":"2019-10-13T00:51:52.862466+0000","last_clean":"2019-10-13T00:51:52.862466+0000","last_became_active":"2019-10-13T00:51:52.858761+0000","last_became_peered":"2019-10-13T00:51:52.858761+0000","last_unstale":"2019-10-13T00:51:52.862466+0000","last_undegraded":"2019-10-13T00:51:52.862466+0000","last_fullsized":"2019-10-13T00:51:52.862466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4,2],"acting":[6,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990614+0000","last_change":"2019-10-13T00:51:52.989809+0000","last_active":"2019-10-13T00:51:52.990614+0000","last_peered":"2019-10-13T00:51:52.990614+0000","last_clean":"2019-10-13T00:51:52.990614+0000","last_became_active":"2019-10-13T00:51:52.989281+0000","last_became_peered":"2019-10-13T00:51:52.989281+0000","last_unstale":"2019-10-13T00:51:52.990614+0000","last_undegraded":"2019-10-13T00:51:52.990614+0000","last_fullsized":"2019-10-13T00:51:52.990614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,7],"acting":[4,6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.858065+0000","last_change":"2019-10-13T00:51:46.833584+0000","last_active":"2019-10-13T00:51:52.858065+0000","last_peered":"2019-10-13T00:51:52.858065+0000","last_clean":"2019-10-13T00:51:52.858065+0000","last_became_active":"2019-10-13T00:51:46.833260+0000","last_became_peered":"2019-10-13T00:51:46.833260+0000","last_unstale":"2019-10-13T00:51:52.858065+0000","last_undegraded":"2019-10-13T00:51:52.858065+0000","last_fullsized":"2019-10-13T00:51:52.858065+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"23'3","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.896156+0000","last_change":"2019-10-13T00:51:49.988514+0000","last_active":"2019-10-13T00:51:59.896156+0000","last_peered":"2019-10-13T00:51:59.896156+0000","last_clean":"2019-10-13T00:51:59.896156+0000","last_became_active":"2019-10-13T00:51:49.988273+0000","last_became_peered":"2019-10-13T00:51:49.988273+0000","last_unstale":"2019-10-13T00:51:59.896156+0000","last_undegraded":"2019-10-13T00:51:59.896156+0000","last_fullsized":"2019-10-13T00:51:59.896156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990765+0000","last_change":"2019-10-13T00:51:52.990020+0000","last_active":"2019-10-13T00:51:52.990765+0000","last_peered":"2019-10-13T00:51:52.990765+0000","last_clean":"2019-10-13T00:51:52.990765+0000","last_became_active":"2019-10-13T00:51:52.989097+0000","last_became_peered":"2019-10-13T00:51:52.989097+0000","last_unstale":"2019-10-13T00:51:52.990765+0000","last_undegraded":"2019-10-13T00:51:52.990765+0000","last_fullsized":"2019-10-13T00:51:52.990765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,0],"acting":[4,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.988091+0000","last_change":"2019-10-13T00:51:46.833662+0000","last_active":"2019-10-13T00:51:52.988091+0000","last_peered":"2019-10-13T00:51:52.988091+0000","last_clean":"2019-10-13T00:51:52.988091+0000","last_became_active":"2019-10-13T00:51:46.833342+0000","last_became_peered":"2019-10-13T00:51:46.833342+0000","last_unstale":"2019-10-13T00:51:52.988091+0000","last_undegraded":"2019-10-13T00:51:52.988091+0000","last_fullsized":"2019-10-13T00:51:52.988091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"23'7","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901582+0000","last_change":"2019-10-13T00:51:49.988964+0000","last_active":"2019-10-13T00:51:59.901582+0000","last_peered":"2019-10-13T00:51:59.901582+0000","last_clean":"2019-10-13T00:51:59.901582+0000","last_became_active":"2019-10-13T00:51:49.988658+0000","last_became_peered":"2019-10-13T00:51:49.988658+0000","last_unstale":"2019-10-13T00:51:59.901582+0000","last_undegraded":"2019-10-13T00:51:59.901582+0000","last_fullsized":"2019-10-13T00:51:59.901582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853815+0000","last_change":"2019-10-13T00:51:47.624840+0000","last_active":"2019-10-13T00:51:52.853815+0000","last_peered":"2019-10-13T00:51:52.853815+0000","last_clean":"2019-10-13T00:51:52.853815+0000","last_became_active":"2019-10-13T00:51:47.624553+0000","last_became_peered":"2019-10-13T00:51:47.624553+0000","last_unstale":"2019-10-13T00:51:52.853815+0000","last_undegraded":"2019-10-13T00:51:52.853815+0000","last_fullsized":"2019-10-13T00:51:52.853815+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"23'5","reported_seq":"17","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901420+0000","last_change":"2019-10-13T00:51:50.036823+0000","last_active":"2019-10-13T00:51:59.901420+0000","last_peered":"2019-10-13T00:51:59.901420+0000","last_clean":"2019-10-13T00:51:59.901420+0000","last_became_active":"2019-10-13T00:51:50.036566+0000","last_became_peered":"2019-10-13T00:51:50.036566+0000","last_unstale":"2019-10-13T00:51:59.901420+0000","last_undegraded":"2019-10-13T00:51:59.901420+0000","last_fullsized":"2019-10-13T00:51:59.901420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.862478+0000","last_change":"2019-10-13T00:51:52.859232+0000","last_active":"2019-10-13T00:51:52.862478+0000","last_peered":"2019-10-13T00:51:52.862478+0000","last_clean":"2019-10-13T00:51:52.862478+0000","last_became_active":"2019-10-13T00:51:52.858702+0000","last_became_peered":"2019-10-13T00:51:52.858702+0000","last_unstale":"2019-10-13T00:51:52.862478+0000","last_undegraded":"2019-10-13T00:51:52.862478+0000","last_fullsized":"2019-10-13T00:51:52.862478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0,1],"acting":[4,2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853279+0000","last_change":"2019-10-13T00:51:46.829935+0000","last_active":"2019-10-13T00:51:52.853279+0000","last_peered":"2019-10-13T00:51:52.853279+0000","last_clean":"2019-10-13T00:51:52.853279+0000","last_became_active":"2019-10-13T00:51:46.829636+0000","last_became_peered":"2019-10-13T00:51:46.829636+0000","last_unstale":"2019-10-13T00:51:52.853279+0000","last_undegraded":"2019-10-13T00:51:52.853279+0000","last_fullsized":"2019-10-13T00:51:52.853279+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"23'1","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:53.192918+0000","last_change":"2019-10-13T00:51:49.851510+0000","last_active":"2019-10-13T00:51:53.192918+0000","last_peered":"2019-10-13T00:51:53.192918+0000","last_clean":"2019-10-13T00:51:53.192918+0000","last_became_active":"2019-10-13T00:51:49.851278+0000","last_became_peered":"2019-10-13T00:51:49.851278+0000","last_unstale":"2019-10-13T00:51:53.192918+0000","last_undegraded":"2019-10-13T00:51:53.192918+0000","last_fullsized":"2019-10-13T00:51:53.192918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990704+0000","last_change":"2019-10-13T00:51:52.990436+0000","last_active":"2019-10-13T00:51:52.990704+0000","last_peered":"2019-10-13T00:51:52.990704+0000","last_clean":"2019-10-13T00:51:52.990704+0000","last_became_active":"2019-10-13T00:51:52.989893+0000","last_became_peered":"2019-10-13T00:51:52.989893+0000","last_unstale":"2019-10-13T00:51:52.990704+0000","last_undegraded":"2019-10-13T00:51:52.990704+0000","last_fullsized":"2019-10-13T00:51:52.990704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[0,2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.988094+0000","last_change":"2019-10-13T00:51:49.851993+0000","last_active":"2019-10-13T00:51:52.988094+0000","last_peered":"2019-10-13T00:51:52.988094+0000","last_clean":"2019-10-13T00:51:52.988094+0000","last_became_active":"2019-10-13T00:51:49.851710+0000","last_became_peered":"2019-10-13T00:51:49.851710+0000","last_unstale":"2019-10-13T00:51:52.988094+0000","last_undegraded":"2019-10-13T00:51:52.988094+0000","last_fullsized":"2019-10-13T00:51:52.988094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.856978+0000","last_change":"2019-10-13T00:51:46.833384+0000","last_active":"2019-10-13T00:51:52.856978+0000","last_peered":"2019-10-13T00:51:52.856978+0000","last_clean":"2019-10-13T00:51:52.856978+0000","last_became_active":"2019-10-13T00:51:46.833046+0000","last_became_peered":"2019-10-13T00:51:46.833046+0000","last_unstale":"2019-10-13T00:51:52.856978+0000","last_undegraded":"2019-10-13T00:51:52.856978+0000","last_fullsized":"2019-10-13T00:51:52.856978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990338+0000","last_change":"2019-10-13T00:51:52.989799+0000","last_active":"2019-10-13T00:51:52.990338+0000","last_peered":"2019-10-13T00:51:52.990338+0000","last_clean":"2019-10-13T00:51:52.990338+0000","last_became_active":"2019-10-13T00:51:52.989243+0000","last_became_peered":"2019-10-13T00:51:52.989243+0000","last_unstale":"2019-10-13T00:51:52.990338+0000","last_undegraded":"2019-10-13T00:51:52.990338+0000","last_fullsized":"2019-10-13T00:51:52.990338+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"23'8","reported_seq":"20","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901628+0000","last_change":"2019-10-13T00:51:49.852213+0000","last_active":"2019-10-13T00:51:59.901628+0000","last_peered":"2019-10-13T00:51:59.901628+0000","last_clean":"2019-10-13T00:51:59.901628+0000","last_became_active":"2019-10-13T00:51:49.852019+0000","last_became_peered":"2019-10-13T00:51:49.852019+0000","last_unstale":"2019-10-13T00:51:59.901628+0000","last_undegraded":"2019-10-13T00:51:59.901628+0000","last_fullsized":"2019-10-13T00:51:59.901628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853873+0000","last_change":"2019-10-13T00:51:46.833165+0000","last_active":"2019-10-13T00:51:52.853873+0000","last_peered":"2019-10-13T00:51:52.853873+0000","last_clean":"2019-10-13T00:51:52.853873+0000","last_became_active":"2019-10-13T00:51:46.832767+0000","last_became_peered":"2019-10-13T00:51:46.832767+0000","last_unstale":"2019-10-13T00:51:52.853873+0000","last_undegraded":"2019-10-13T00:51:52.853873+0000","last_fullsized":"2019-10-13T00:51:52.853873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.862417+0000","last_change":"2019-10-13T00:51:52.859396+0000","last_active":"2019-10-13T00:51:52.862417+0000","last_peered":"2019-10-13T00:51:52.862417+0000","last_clean":"2019-10-13T00:51:52.862417+0000","last_became_active":"2019-10-13T00:51:52.858853+0000","last_became_peered":"2019-10-13T00:51:52.858853+0000","last_unstale":"2019-10-13T00:51:52.862417+0000","last_undegraded":"2019-10-13T00:51:52.862417+0000","last_fullsized":"2019-10-13T00:51:52.862417+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.264}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.279}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.384}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.333}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.246}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.322}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.396}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049408,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322432,"statfs":{"total":96636764160,"available":95562170368,"internally_reserved":1073741824,"allocated":835584,"data_stored":90885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.444}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.494}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.268}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.254}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.336}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049408,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322432,"statfs":{"total":96636764160,"available":95562170368,"internally_reserved":1073741824,"allocated":835584,"data_stored":90885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.340}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.260}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.395}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.290}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.213}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.235}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.380}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049392,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322448,"statfs":{"total":96636764160,"available":95562186752,"internally_reserved":1073741824,"allocated":819200,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.459}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.448}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.476}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.353}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.372}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.397}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049392,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322448,"statfs":{"total":96636764160,"available":95562186752,"internally_reserved":1073741824,"allocated":819200,"data_stored":89925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.425}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.469}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.456}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.513}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.603}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.365}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.349}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.431}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.556}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.577}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.592}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.536}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":89393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.207}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.244}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.278}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.562}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.492}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.575}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.505}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049440,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322400,"statfs":{"total":96636764160,"available":95562137600,"internally_reserved":1073741824,"allocated":868352,"data_stored":90119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.372}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.327}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.390}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.479}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.456}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.551}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.533}]}]}]}} 2019-10-13T00:52:04.205 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:04.205 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-13T00:52:04.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.325+0000 7f3f5898a700 1 Processor -- start 2019-10-13T00:52:04.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.325+0000 7f3f5898a700 1 -- start start 2019-10-13T00:52:04.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.325+0000 7f3f5898a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f54099b10 0x7f3f54099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5898a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f5409d620 0x7f3f5409a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5898a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3f5409c7f0 0x7f3f5409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5898a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5409ac50 con 0x7f3f5409c7f0 2019-10-13T00:52:04.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5898a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5409af20 con 0x7f3f5409d620 2019-10-13T00:52:04.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5898a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5409a980 con 0x7f3f54099b10 2019-10-13T00:52:04.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f5409d620 0x7f3f5409a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f5409d620 0x7f3f5409a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59312/0 (socket says 172.21.15.155:59312) 2019-10-13T00:52:04.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 -- 172.21.15.155:0/196925436 learned_addr learned my addr 172.21.15.155:0/196925436 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:04.328 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5359e700 1 --2- 172.21.15.155:0/196925436 >> v2:172.21.15.142:3300/0 conn(0x7f3f5409c7f0 0x7f3f5409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f54099b10 0x7f3f54099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 -- 172.21.15.155:0/196925436 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f54099b10 msgr2=0x7f3f54099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:04.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f54099b10 0x7f3f54099f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 -- 172.21.15.155:0/196925436 >> v2:172.21.15.142:3300/0 conn(0x7f3f5409c7f0 msgr2=0x7f3f5409cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:04.329 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> v2:172.21.15.142:3300/0 conn(0x7f3f5409c7f0 0x7f3f5409cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 -- 172.21.15.155:0/196925436 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5409bd10 con 0x7f3f5409d620 2019-10-13T00:52:04.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> v2:172.21.15.155:3300/0 conn(0x7f3f5409d620 0x7f3f5409a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f4c00f730 tx=0x7f3f4c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> v2:172.21.15.155:3300/0 conn(0x7f3f5409d620 0x7f3f5409a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f4c00f730 tx=0x7f3f4c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5409d620 0x7f3f5409a440 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7f3f4c00f730 tx=0x7f3f4c00fad0).ready entity=mon.1 client_cookie=3f92f0a1119cf7b2 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.330 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5409d620 0x7f3f5409a440 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7f3f4c00f730 tx=0x7f3f4c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5409d620 0x7f3f5409a440 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7f3f4c00f730 tx=0x7f3f4c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f51d9b700 1 -- 172.21.15.155:0/196925436 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f4c000960 con 0x7f3f5409d620 2019-10-13T00:52:04.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5259c700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5409d620 0x7f3f5409a440 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7f3f4c00f730 tx=0x7f3f4c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f51d9b700 1 -- 172.21.15.155:0/196925436 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4c000c90 con 0x7f3f5409d620 2019-10-13T00:52:04.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f51d9b700 1 -- 172.21.15.155:0/196925436 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f4c0013e0 con 0x7f3f5409d620 2019-10-13T00:52:04.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5898a700 1 -- 172.21.15.155:0/196925436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5409d620 msgr2=0x7f3f5409a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.331 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.326+0000 7f3f5898a700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5409d620 0x7f3f5409a440 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7f3f4c00f730 tx=0x7f3f4c00fad0).stop 2019-10-13T00:52:04.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.327+0000 7f3f5898a700 1 -- 172.21.15.155:0/196925436 shutdown_connections 2019-10-13T00:52:04.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.327+0000 7f3f5898a700 1 --2- 172.21.15.155:0/196925436 >> v2:172.21.15.142:3300/0 conn(0x7f3f5409c7f0 0x7f3f5409cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.327+0000 7f3f5898a700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5409d620 0x7f3f5409a440 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.327+0000 7f3f5898a700 1 --2- 172.21.15.155:0/196925436 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f54099b10 0x7f3f54099f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.327+0000 7f3f5898a700 1 -- 172.21.15.155:0/196925436 shutdown_connections 2019-10-13T00:52:04.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.327+0000 7f3f5898a700 1 -- 172.21.15.155:0/196925436 wait complete. 2019-10-13T00:52:04.332 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.327+0000 7f3f5898a700 1 -- 172.21.15.155:0/196925436 >> 172.21.15.155:0/196925436 conn(0x7f3f54099550 msgr2=0x7f3f54061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:04.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.329+0000 7f3f5898a700 1 Processor -- start 2019-10-13T00:52:04.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.329+0000 7f3f5898a700 1 -- start start 2019-10-13T00:52:04.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5898a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f54073990 0x7f3f54091710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5898a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f540926e0 0x7f3f54091c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5898a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3f54092190 0x7f3f54088ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.336 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5898a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f54090780 con 0x7f3f54092190 2019-10-13T00:52:04.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5898a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5408fd50 con 0x7f3f54073990 2019-10-13T00:52:04.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5898a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f54090020 con 0x7f3f540926e0 2019-10-13T00:52:04.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f52d9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f54073990 0x7f3f54091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f52d9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f54073990 0x7f3f54091710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59320/0 (socket says 172.21.15.155:59320) 2019-10-13T00:52:04.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f52d9d700 1 -- 172.21.15.155:0/2160297149 learned_addr learned my addr 172.21.15.155:0/2160297149 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:04.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5359e700 1 --2- 172.21.15.155:0/2160297149 >> v2:172.21.15.142:3300/0 conn(0x7f3f54092190 0x7f3f54088ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.337 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.330+0000 7f3f5259c700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f540926e0 0x7f3f54091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 -- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f540926e0 msgr2=0x7f3f54091c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f540926e0 0x7f3f54091c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 -- 172.21.15.155:0/2160297149 >> v2:172.21.15.142:3300/0 conn(0x7f3f54092190 msgr2=0x7f3f54088ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> v2:172.21.15.142:3300/0 conn(0x7f3f54092190 0x7f3f54088ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 -- 172.21.15.155:0/2160297149 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f54090e10 con 0x7f3f54073990 2019-10-13T00:52:04.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> v2:172.21.15.155:3300/0 conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.338 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> v2:172.21.15.155:3300/0 conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f4b7fe700 1 -- 172.21.15.155:0/2160297149 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f44000940 con 0x7f3f54073990 2019-10-13T00:52:04.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f4b7fe700 1 -- 172.21.15.155:0/2160297149 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f440084f0 con 0x7f3f54073990 2019-10-13T00:52:04.339 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f4b7fe700 1 -- 172.21.15.155:0/2160297149 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f44008bc0 con 0x7f3f54073990 2019-10-13T00:52:04.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f54090e10 con 0x7f3f54073990 2019-10-13T00:52:04.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f54090e10 con 0x7f3f54073990 2019-10-13T00:52:04.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.331+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.332+0000 7f3f4b7fe700 1 -- 172.21.15.155:0/2160297149 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f3f440086e0 con 0x7f3f54073990 2019-10-13T00:52:04.340 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.332+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.332+0000 7f3f4b7fe700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3f2c017c60 0x7f3f2c01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.332+0000 7f3f4b7fe700 1 -- 172.21.15.155:0/2160297149 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3f44030090 con 0x7f3f54073990 2019-10-13T00:52:04.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.332+0000 7f3f5259c700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3f2c017c60 0x7f3f2c01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.341 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.332+0000 7f3f5259c700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3f2c017c60 0x7f3f2c01a0f0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.342 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.340+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f54090e10 con 0x7f3f54073990 2019-10-13T00:52:04.342 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.343+0000 7f3f52d9d700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.342 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.343+0000 7f3f4b7fe700 1 -- 172.21.15.155:0/2160297149 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f3f44007510 con 0x7f3f54073990 2019-10-13T00:52:04.510 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.512+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 --> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f54149bb0 con 0x7f3f2c017c60 2019-10-13T00:52:04.518 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.515+0000 7f3f4b7fe700 1 -- 172.21.15.155:0/2160297149 <== mgr.4110 v2:172.21.15.142:6800/5662 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80478 (crc 0 0 0) 0x7f3f4c007ee0 con 0x7f3f2c017c60 2019-10-13T00:52:04.518 INFO:teuthology.orchestra.run.smithi155.stderr:dumped all 2019-10-13T00:52:04.518 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3f2c017c60 msgr2=0x7f3f2c01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.519 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3f2c017c60 0x7f3f2c01a0f0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.519 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 msgr2=0x7f3f54091710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.519 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7f3f44010440 tx=0x7f3f440107e0).stop 2019-10-13T00:52:04.520 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 shutdown_connections 2019-10-13T00:52:04.520 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 --2- 172.21.15.155:0/2160297149 >> v2:172.21.15.142:3300/0 conn(0x7f3f54092190 0x7f3f54088ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.520 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f3f2c017c60 0x7f3f2c01a0f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.520 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3f540926e0 0x7f3f54091c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.520 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 --2- 172.21.15.155:0/2160297149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54073990 0x7f3f54091710 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.520 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 shutdown_connections 2019-10-13T00:52:04.522 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 wait complete. 2019-10-13T00:52:04.522 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.517+0000 7f3f5898a700 1 -- 172.21.15.155:0/2160297149 >> 172.21.15.155:0/2160297149 conn(0x7f3f5406d080 msgr2=0x7f3f5406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:04.523 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:52:04.526 INFO:teuthology.orchestra.run.smithi155.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2019-10-13T00:52:03.780651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8395136,"kb_used_data":6400,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746579584,"statfs":{"total":773094113280,"available":764497494016,"internally_reserved":8589934592,"allocated":6553600,"data_stored":719444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853835+0000","last_change":"2019-10-13T00:51:46.833481+0000","last_active":"2019-10-13T00:51:52.853835+0000","last_peered":"2019-10-13T00:51:52.853835+0000","last_clean":"2019-10-13T00:51:52.853835+0000","last_became_active":"2019-10-13T00:51:46.833140+0000","last_became_peered":"2019-10-13T00:51:46.833140+0000","last_unstale":"2019-10-13T00:51:52.853835+0000","last_undegraded":"2019-10-13T00:51:52.853835+0000","last_fullsized":"2019-10-13T00:51:52.853835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"23'8","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.896759+0000","last_change":"2019-10-13T00:51:49.850206+0000","last_active":"2019-10-13T00:51:59.896759+0000","last_peered":"2019-10-13T00:51:59.896759+0000","last_clean":"2019-10-13T00:51:59.896759+0000","last_became_active":"2019-10-13T00:51:49.849825+0000","last_became_peered":"2019-10-13T00:51:49.849825+0000","last_unstale":"2019-10-13T00:51:59.896759+0000","last_undegraded":"2019-10-13T00:51:59.896759+0000","last_fullsized":"2019-10-13T00:51:59.896759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.860011+0000","last_change":"2019-10-13T00:51:52.859165+0000","last_active":"2019-10-13T00:51:52.860011+0000","last_peered":"2019-10-13T00:51:52.860011+0000","last_clean":"2019-10-13T00:51:52.860011+0000","last_became_active":"2019-10-13T00:51:52.858864+0000","last_became_peered":"2019-10-13T00:51:52.858864+0000","last_unstale":"2019-10-13T00:51:52.860011+0000","last_undegraded":"2019-10-13T00:51:52.860011+0000","last_fullsized":"2019-10-13T00:51:52.860011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4,3],"acting":[2,6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853169+0000","last_change":"2019-10-13T00:51:46.830036+0000","last_active":"2019-10-13T00:51:52.853169+0000","last_peered":"2019-10-13T00:51:52.853169+0000","last_clean":"2019-10-13T00:51:52.853169+0000","last_became_active":"2019-10-13T00:51:46.829735+0000","last_became_peered":"2019-10-13T00:51:46.829735+0000","last_unstale":"2019-10-13T00:51:52.853169+0000","last_undegraded":"2019-10-13T00:51:52.853169+0000","last_fullsized":"2019-10-13T00:51:52.853169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"23'6","reported_seq":"18","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901377+0000","last_change":"2019-10-13T00:51:49.852008+0000","last_active":"2019-10-13T00:51:59.901377+0000","last_peered":"2019-10-13T00:51:59.901377+0000","last_clean":"2019-10-13T00:51:59.901377+0000","last_became_active":"2019-10-13T00:51:49.851773+0000","last_became_peered":"2019-10-13T00:51:49.851773+0000","last_unstale":"2019-10-13T00:51:59.901377+0000","last_undegraded":"2019-10-13T00:51:59.901377+0000","last_fullsized":"2019-10-13T00:51:59.901377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.862466+0000","last_change":"2019-10-13T00:51:52.859449+0000","last_active":"2019-10-13T00:51:52.862466+0000","last_peered":"2019-10-13T00:51:52.862466+0000","last_clean":"2019-10-13T00:51:52.862466+0000","last_became_active":"2019-10-13T00:51:52.858761+0000","last_became_peered":"2019-10-13T00:51:52.858761+0000","last_unstale":"2019-10-13T00:51:52.862466+0000","last_undegraded":"2019-10-13T00:51:52.862466+0000","last_fullsized":"2019-10-13T00:51:52.862466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4,2],"acting":[6,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990614+0000","last_change":"2019-10-13T00:51:52.989809+0000","last_active":"2019-10-13T00:51:52.990614+0000","last_peered":"2019-10-13T00:51:52.990614+0000","last_clean":"2019-10-13T00:51:52.990614+0000","last_became_active":"2019-10-13T00:51:52.989281+0000","last_became_peered":"2019-10-13T00:51:52.989281+0000","last_unstale":"2019-10-13T00:51:52.990614+0000","last_undegraded":"2019-10-13T00:51:52.990614+0000","last_fullsized":"2019-10-13T00:51:52.990614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,7],"acting":[4,6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.858065+0000","last_change":"2019-10-13T00:51:46.833584+0000","last_active":"2019-10-13T00:51:52.858065+0000","last_peered":"2019-10-13T00:51:52.858065+0000","last_clean":"2019-10-13T00:51:52.858065+0000","last_became_active":"2019-10-13T00:51:46.833260+0000","last_became_peered":"2019-10-13T00:51:46.833260+0000","last_unstale":"2019-10-13T00:51:52.858065+0000","last_undegraded":"2019-10-13T00:51:52.858065+0000","last_fullsized":"2019-10-13T00:51:52.858065+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"23'3","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.896156+0000","last_change":"2019-10-13T00:51:49.988514+0000","last_active":"2019-10-13T00:51:59.896156+0000","last_peered":"2019-10-13T00:51:59.896156+0000","last_clean":"2019-10-13T00:51:59.896156+0000","last_became_active":"2019-10-13T00:51:49.988273+0000","last_became_peered":"2019-10-13T00:51:49.988273+0000","last_unstale":"2019-10-13T00:51:59.896156+0000","last_undegraded":"2019-10-13T00:51:59.896156+0000","last_fullsized":"2019-10-13T00:51:59.896156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990765+0000","last_change":"2019-10-13T00:51:52.990020+0000","last_active":"2019-10-13T00:51:52.990765+0000","last_peered":"2019-10-13T00:51:52.990765+0000","last_clean":"2019-10-13T00:51:52.990765+0000","last_became_active":"2019-10-13T00:51:52.989097+0000","last_became_peered":"2019-10-13T00:51:52.989097+0000","last_unstale":"2019-10-13T00:51:52.990765+0000","last_undegraded":"2019-10-13T00:51:52.990765+0000","last_fullsized":"2019-10-13T00:51:52.990765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1,0],"acting":[4,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.988091+0000","last_change":"2019-10-13T00:51:46.833662+0000","last_active":"2019-10-13T00:51:52.988091+0000","last_peered":"2019-10-13T00:51:52.988091+0000","last_clean":"2019-10-13T00:51:52.988091+0000","last_became_active":"2019-10-13T00:51:46.833342+0000","last_became_peered":"2019-10-13T00:51:46.833342+0000","last_unstale":"2019-10-13T00:51:52.988091+0000","last_undegraded":"2019-10-13T00:51:52.988091+0000","last_fullsized":"2019-10-13T00:51:52.988091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"23'7","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901582+0000","last_change":"2019-10-13T00:51:49.988964+0000","last_active":"2019-10-13T00:51:59.901582+0000","last_peered":"2019-10-13T00:51:59.901582+0000","last_clean":"2019-10-13T00:51:59.901582+0000","last_became_active":"2019-10-13T00:51:49.988658+0000","last_became_peered":"2019-10-13T00:51:49.988658+0000","last_unstale":"2019-10-13T00:51:59.901582+0000","last_undegraded":"2019-10-13T00:51:59.901582+0000","last_fullsized":"2019-10-13T00:51:59.901582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853815+0000","last_change":"2019-10-13T00:51:47.624840+0000","last_active":"2019-10-13T00:51:52.853815+0000","last_peered":"2019-10-13T00:51:52.853815+0000","last_clean":"2019-10-13T00:51:52.853815+0000","last_became_active":"2019-10-13T00:51:47.624553+0000","last_became_peered":"2019-10-13T00:51:47.624553+0000","last_unstale":"2019-10-13T00:51:52.853815+0000","last_undegraded":"2019-10-13T00:51:52.853815+0000","last_fullsized":"2019-10-13T00:51:52.853815+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"23'5","reported_seq":"17","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901420+0000","last_change":"2019-10-13T00:51:50.036823+0000","last_active":"2019-10-13T00:51:59.901420+0000","last_peered":"2019-10-13T00:51:59.901420+0000","last_clean":"2019-10-13T00:51:59.901420+0000","last_became_active":"2019-10-13T00:51:50.036566+0000","last_became_peered":"2019-10-13T00:51:50.036566+0000","last_unstale":"2019-10-13T00:51:59.901420+0000","last_undegraded":"2019-10-13T00:51:59.901420+0000","last_fullsized":"2019-10-13T00:51:59.901420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.862478+0000","last_change":"2019-10-13T00:51:52.859232+0000","last_active":"2019-10-13T00:51:52.862478+0000","last_peered":"2019-10-13T00:51:52.862478+0000","last_clean":"2019-10-13T00:51:52.862478+0000","last_became_active":"2019-10-13T00:51:52.858702+0000","last_became_peered":"2019-10-13T00:51:52.858702+0000","last_unstale":"2019-10-13T00:51:52.862478+0000","last_undegraded":"2019-10-13T00:51:52.862478+0000","last_fullsized":"2019-10-13T00:51:52.862478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0,1],"acting":[4,2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853279+0000","last_change":"2019-10-13T00:51:46.829935+0000","last_active":"2019-10-13T00:51:52.853279+0000","last_peered":"2019-10-13T00:51:52.853279+0000","last_clean":"2019-10-13T00:51:52.853279+0000","last_became_active":"2019-10-13T00:51:46.829636+0000","last_became_peered":"2019-10-13T00:51:46.829636+0000","last_unstale":"2019-10-13T00:51:52.853279+0000","last_undegraded":"2019-10-13T00:51:52.853279+0000","last_fullsized":"2019-10-13T00:51:52.853279+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"23'1","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:53.192918+0000","last_change":"2019-10-13T00:51:49.851510+0000","last_active":"2019-10-13T00:51:53.192918+0000","last_peered":"2019-10-13T00:51:53.192918+0000","last_clean":"2019-10-13T00:51:53.192918+0000","last_became_active":"2019-10-13T00:51:49.851278+0000","last_became_peered":"2019-10-13T00:51:49.851278+0000","last_unstale":"2019-10-13T00:51:53.192918+0000","last_undegraded":"2019-10-13T00:51:53.192918+0000","last_fullsized":"2019-10-13T00:51:53.192918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990704+0000","last_change":"2019-10-13T00:51:52.990436+0000","last_active":"2019-10-13T00:51:52.990704+0000","last_peered":"2019-10-13T00:51:52.990704+0000","last_clean":"2019-10-13T00:51:52.990704+0000","last_became_active":"2019-10-13T00:51:52.989893+0000","last_became_peered":"2019-10-13T00:51:52.989893+0000","last_unstale":"2019-10-13T00:51:52.990704+0000","last_undegraded":"2019-10-13T00:51:52.990704+0000","last_fullsized":"2019-10-13T00:51:52.990704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[0,2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.988094+0000","last_change":"2019-10-13T00:51:49.851993+0000","last_active":"2019-10-13T00:51:52.988094+0000","last_peered":"2019-10-13T00:51:52.988094+0000","last_clean":"2019-10-13T00:51:52.988094+0000","last_became_active":"2019-10-13T00:51:49.851710+0000","last_became_peered":"2019-10-13T00:51:49.851710+0000","last_unstale":"2019-10-13T00:51:52.988094+0000","last_undegraded":"2019-10-13T00:51:52.988094+0000","last_fullsized":"2019-10-13T00:51:52.988094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.856978+0000","last_change":"2019-10-13T00:51:46.833384+0000","last_active":"2019-10-13T00:51:52.856978+0000","last_peered":"2019-10-13T00:51:52.856978+0000","last_clean":"2019-10-13T00:51:52.856978+0000","last_became_active":"2019-10-13T00:51:46.833046+0000","last_became_peered":"2019-10-13T00:51:46.833046+0000","last_unstale":"2019-10-13T00:51:52.856978+0000","last_undegraded":"2019-10-13T00:51:52.856978+0000","last_fullsized":"2019-10-13T00:51:52.856978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.990338+0000","last_change":"2019-10-13T00:51:52.989799+0000","last_active":"2019-10-13T00:51:52.990338+0000","last_peered":"2019-10-13T00:51:52.990338+0000","last_clean":"2019-10-13T00:51:52.990338+0000","last_became_active":"2019-10-13T00:51:52.989243+0000","last_became_peered":"2019-10-13T00:51:52.989243+0000","last_unstale":"2019-10-13T00:51:52.990338+0000","last_undegraded":"2019-10-13T00:51:52.990338+0000","last_fullsized":"2019-10-13T00:51:52.990338+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"23'8","reported_seq":"20","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T00:51:59.901628+0000","last_change":"2019-10-13T00:51:49.852213+0000","last_active":"2019-10-13T00:51:59.901628+0000","last_peered":"2019-10-13T00:51:59.901628+0000","last_clean":"2019-10-13T00:51:59.901628+0000","last_became_active":"2019-10-13T00:51:49.852019+0000","last_became_peered":"2019-10-13T00:51:49.852019+0000","last_unstale":"2019-10-13T00:51:59.901628+0000","last_undegraded":"2019-10-13T00:51:59.901628+0000","last_fullsized":"2019-10-13T00:51:59.901628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:48.833628+0000","last_clean_scrub_stamp":"2019-10-13T00:51:48.833628+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.853873+0000","last_change":"2019-10-13T00:51:46.833165+0000","last_active":"2019-10-13T00:51:52.853873+0000","last_peered":"2019-10-13T00:51:52.853873+0000","last_clean":"2019-10-13T00:51:52.853873+0000","last_became_active":"2019-10-13T00:51:46.832767+0000","last_became_peered":"2019-10-13T00:51:46.832767+0000","last_unstale":"2019-10-13T00:51:52.853873+0000","last_undegraded":"2019-10-13T00:51:52.853873+0000","last_fullsized":"2019-10-13T00:51:52.853873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:45.817698+0000","last_clean_scrub_stamp":"2019-10-13T00:51:45.817698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T00:51:52.862417+0000","last_change":"2019-10-13T00:51:52.859396+0000","last_active":"2019-10-13T00:51:52.862417+0000","last_peered":"2019-10-13T00:51:52.862417+0000","last_clean":"2019-10-13T00:51:52.862417+0000","last_became_active":"2019-10-13T00:51:52.858853+0000","last_became_peered":"2019-10-13T00:51:52.858853+0000","last_unstale":"2019-10-13T00:51:52.862417+0000","last_undegraded":"2019-10-13T00:51:52.862417+0000","last_fullsized":"2019-10-13T00:51:52.862417+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T00:51:51.847258+0000","last_clean_scrub_stamp":"2019-10-13T00:51:51.847258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.264}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.279}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.384}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.333}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.246}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.322}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.396}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049408,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322432,"statfs":{"total":96636764160,"available":95562170368,"internally_reserved":1073741824,"allocated":835584,"data_stored":90885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.444}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.494}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.268}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.254}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.336}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049408,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322432,"statfs":{"total":96636764160,"available":95562170368,"internally_reserved":1073741824,"allocated":835584,"data_stored":90885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.340}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.260}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.395}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.290}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.213}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.235}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.380}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049392,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322448,"statfs":{"total":96636764160,"available":95562186752,"internally_reserved":1073741824,"allocated":819200,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.459}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.448}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.476}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.353}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.372}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.397}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049392,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322448,"statfs":{"total":96636764160,"available":95562186752,"internally_reserved":1073741824,"allocated":819200,"data_stored":89925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.425}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.469}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.456}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.513}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.603}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.365}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.349}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.431}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.556}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.577}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.592}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.536}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":89393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.207}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.244}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.278}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.562}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.492}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.575}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.505}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049440,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322400,"statfs":{"total":96636764160,"available":95562137600,"internally_reserved":1073741824,"allocated":868352,"data_stored":90119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.372}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.327}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.390}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.479}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.456}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.551}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.533}]}]}]}} 2019-10-13T00:52:04.530 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-10-13T00:52:04.530 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-10-13T00:52:04.530 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:04.530 INFO:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2019-10-13T00:52:04.643 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.640+0000 7ff9899a1700 1 Processor -- start 2019-10-13T00:52:04.643 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.640+0000 7ff9899a1700 1 -- start start 2019-10-13T00:52:04.643 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9899a1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff98409a700 0x7ff98409c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.643 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9899a1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff98409d620 0x7ff98409cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.643 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9899a1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff984064f00 con 0x7ff984099b10 2019-10-13T00:52:04.644 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9899a1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98409b680 con 0x7ff98409a700 2019-10-13T00:52:04.644 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9899a1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98409ac50 con 0x7ff98409d620 2019-10-13T00:52:04.644 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff983fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff98409d620 0x7ff98409cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.644 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9837fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff984099b10 0x7ff984099f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49402/0 (socket says 172.21.15.155:49402) 2019-10-13T00:52:04.644 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9837fe700 1 -- 172.21.15.155:0/1603219230 learned_addr learned my addr 172.21.15.155:0/1603219230 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:04.644 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff982ffd700 1 --2- 172.21.15.155:0/1603219230 >> v2:172.21.15.142:3300/0 conn(0x7ff98409a700 0x7ff98409c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.644 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9827fc700 1 -- 172.21.15.155:0/1603219230 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849636255 0 0) 0x7ff9740014d0 con 0x7ff984099b10 2019-10-13T00:52:04.645 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.641+0000 7ff9827fc700 1 -- 172.21.15.155:0/1603219230 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96c002e70 con 0x7ff984099b10 2019-10-13T00:52:04.645 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 -- 172.21.15.155:0/1603219230 >> v2:172.21.15.142:3300/0 conn(0x7ff98409a700 msgr2=0x7ff98409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:04.645 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 --2- 172.21.15.155:0/1603219230 >> v2:172.21.15.142:3300/0 conn(0x7ff98409a700 0x7ff98409c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.645 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 -- 172.21.15.155:0/1603219230 >> v1:172.21.15.155:6789/0 conn(0x7ff984099b10 legacy=0x7ff984099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.646 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 -- 172.21.15.155:0/1603219230 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98409bd10 con 0x7ff98409d620 2019-10-13T00:52:04.646 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 --2- 172.21.15.155:0/1603219230 >> v2:172.21.15.155:3300/0 conn(0x7ff98409d620 0x7ff98409cd30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff97c00af40 tx=0x7ff97c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.646 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 --2- 172.21.15.155:0/1603219230 >> v2:172.21.15.155:3300/0 conn(0x7ff98409d620 0x7ff98409cd30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff97c00af40 tx=0x7ff97c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.646 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 --2- 172.21.15.155:0/1603219230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff98409d620 0x7ff98409cd30 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7ff97c00af40 tx=0x7ff97c00b750).ready entity=mon.1 client_cookie=67310f10eee5c002 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.646 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 --2- 172.21.15.155:0/1603219230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff98409d620 0x7ff98409cd30 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7ff97c00af40 tx=0x7ff97c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.646 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff9827fc700 1 -- 172.21.15.155:0/1603219230 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff97c009b90 con 0x7ff98409d620 2019-10-13T00:52:04.647 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 --2- 172.21.15.155:0/1603219230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff98409d620 0x7ff98409cd30 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7ff97c00af40 tx=0x7ff97c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.647 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff983fff700 1 --2- 172.21.15.155:0/1603219230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff98409d620 0x7ff98409cd30 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7ff97c00af40 tx=0x7ff97c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.647 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff9827fc700 1 -- 172.21.15.155:0/1603219230 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c000a70 con 0x7ff98409d620 2019-10-13T00:52:04.647 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff9827fc700 1 -- 172.21.15.155:0/1603219230 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff97c00a170 con 0x7ff98409d620 2019-10-13T00:52:04.647 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff9899a1700 1 -- 172.21.15.155:0/1603219230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff98409d620 msgr2=0x7ff98409cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.647 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.642+0000 7ff9899a1700 1 --2- 172.21.15.155:0/1603219230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff98409d620 0x7ff98409cd30 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7ff97c00af40 tx=0x7ff97c00b750).stop 2019-10-13T00:52:04.648 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.643+0000 7ff9899a1700 1 -- 172.21.15.155:0/1603219230 shutdown_connections 2019-10-13T00:52:04.648 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.643+0000 7ff9899a1700 1 --2- 172.21.15.155:0/1603219230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff98409d620 0x7ff98409cd30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.648 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.643+0000 7ff9899a1700 1 --2- 172.21.15.155:0/1603219230 >> v2:172.21.15.142:3300/0 conn(0x7ff98409a700 0x7ff98409c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:04.648 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.643+0000 7ff9899a1700 1 -- 172.21.15.155:0/1603219230 shutdown_connections 2019-10-13T00:52:04.648 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.643+0000 7ff9899a1700 1 -- 172.21.15.155:0/1603219230 wait complete. 2019-10-13T00:52:04.648 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.643+0000 7ff9899a1700 1 -- 172.21.15.155:0/1603219230 >> 172.21.15.155:0/1603219230 conn(0x7ff984099550 msgr2=0x7ff984061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:04.648 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.645+0000 7ff9899a1700 1 Processor -- start 2019-10-13T00:52:04.649 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.645+0000 7ff9899a1700 1 -- start start 2019-10-13T00:52:04.652 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9899a1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff984073990 0x7ff98408ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.652 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9899a1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9840890d0 con 0x7ff98408fe30 2019-10-13T00:52:04.652 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9899a1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff984088bd0 con 0x7ff984153ef0 2019-10-13T00:52:04.652 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9899a1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff984093d40 con 0x7ff984073990 2019-10-13T00:52:04.653 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff983fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff984153ef0 0x7ff9841562f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49412/0 (socket says 172.21.15.155:49412) 2019-10-13T00:52:04.653 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff983fff700 1 -- 172.21.15.155:0/595647241 learned_addr learned my addr 172.21.15.155:0/595647241 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:04.653 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> v2:172.21.15.142:3300/0 conn(0x7ff984073990 0x7ff98408ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.653 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739249 0 0) 0x7ff97c000b50 con 0x7ff984153ef0 2019-10-13T00:52:04.654 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff964002e70 con 0x7ff984153ef0 2019-10-13T00:52:04.654 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 -- 172.21.15.155:0/595647241 >> v1:172.21.15.155:6789/0 conn(0x7ff984153ef0 legacy=0x7ff9841562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.654 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 -- 172.21.15.155:0/595647241 >> v1:172.21.15.142:6789/0 conn(0x7ff98408fe30 legacy=0x7ff98408f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.654 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 -- 172.21.15.155:0/595647241 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98408e560 con 0x7ff984073990 2019-10-13T00:52:04.654 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> v2:172.21.15.142:3300/0 conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.654 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> v2:172.21.15.142:3300/0 conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.654 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.655 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.655 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff97400f4a0 con 0x7ff984073990 2019-10-13T00:52:04.655 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.655 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.656 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.646+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97400f0c0 con 0x7ff984073990 2019-10-13T00:52:04.656 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff97400fb20 con 0x7ff984073990 2019-10-13T00:52:04.656 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98408e560 con 0x7ff984073990 2019-10-13T00:52:04.656 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98408e560 con 0x7ff984073990 2019-10-13T00:52:04.657 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.657 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff97400f2b0 con 0x7ff984073990 2019-10-13T00:52:04.657 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.657 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff96bfff700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff964017ca0 0x7ff96401a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.658 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff974030350 con 0x7ff984073990 2019-10-13T00:52:04.658 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.647+0000 7ff982ffd700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff964017ca0 0x7ff96401a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.658 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.648+0000 7ff982ffd700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff964017ca0 0x7ff96401a110 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.658 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.656+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98408e560 con 0x7ff984073990 2019-10-13T00:52:04.659 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.659+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.659 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.659+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff97400edc0 con 0x7ff984073990 2019-10-13T00:52:04.850 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.852+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7ff98408e560 con 0x7ff984073990 2019-10-13T00:52:04.851 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.853+0000 7ff9837fe700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.851 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.853+0000 7ff96bfff700 1 -- 172.21.15.155:0/595647241 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+10 (secure 0 0 0) 0x7ff97400f2b0 con 0x7ff984073990 2019-10-13T00:52:04.852 INFO:teuthology.misc.health.smithi155.stdout:HEALTH_OK 2019-10-13T00:52:04.855 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff964017ca0 msgr2=0x7ff96401a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.855 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff964017ca0 0x7ff96401a110 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.855 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 msgr2=0x7ff98408ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.855 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7ff9740106f0 tx=0x7ff974010a90).stop 2019-10-13T00:52:04.856 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 shutdown_connections 2019-10-13T00:52:04.856 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff964017ca0 0x7ff96401a110 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.856 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 --2- 172.21.15.155:0/595647241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff984073990 0x7ff98408ee60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.856 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 shutdown_connections 2019-10-13T00:52:04.856 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 wait complete. 2019-10-13T00:52:04.856 INFO:teuthology.misc.health.smithi155.stderr:2019-10-13T00:52:04.855+0000 7ff9899a1700 1 -- 172.21.15.155:0/595647241 >> 172.21.15.155:0/595647241 conn(0x7ff98406d080 msgr2=0x7ff98406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:04.862 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2019-10-13T00:52:04.862 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:04.863 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:52:04.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.982+0000 7ffba45eb700 1 Processor -- start 2019-10-13T00:52:04.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.982+0000 7ffba45eb700 1 -- start start 2019-10-13T00:52:04.984 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba45eb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb9c09d620 0x7ffb9c153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:04.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba45eb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c064f00 con 0x7ffb9c099b10 2019-10-13T00:52:04.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba45eb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c0987e0 con 0x7ffb9c09a700 2019-10-13T00:52:04.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba45eb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c09b680 con 0x7ffb9c09d620 2019-10-13T00:52:04.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb9c09d620 0x7ffb9c153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb9c09d620 0x7ffb9c153c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59342/0 (socket says 172.21.15.155:59342) 2019-10-13T00:52:04.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba2387700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ffb9c09a700 0x7ffb9c09c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49420/0 (socket says 172.21.15.155:49420) 2019-10-13T00:52:04.985 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 -- 172.21.15.155:0/996858916 learned_addr learned my addr 172.21.15.155:0/996858916 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba1b86700 1 -- 172.21.15.155:0/996858916 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374138170 0 0) 0x7ffb98001080 con 0x7ffb9c099b10 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 -- 172.21.15.155:0/996858916 >> v1:172.21.15.155:6789/0 conn(0x7ffb9c09a700 legacy=0x7ffb9c09c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 -- 172.21.15.155:0/996858916 >> v1:172.21.15.142:6789/0 conn(0x7ffb9c099b10 legacy=0x7ffb9c099f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 -- 172.21.15.155:0/996858916 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c09bd10 con 0x7ffb9c09d620 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 --2- 172.21.15.155:0/996858916 >> v2:172.21.15.155:3300/0 conn(0x7ffb9c09d620 0x7ffb9c153c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffb9400faa0 tx=0x7ffb9400fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 --2- 172.21.15.155:0/996858916 >> v2:172.21.15.155:3300/0 conn(0x7ffb9c09d620 0x7ffb9c153c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffb9400faa0 tx=0x7ffb9400fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.983+0000 7ffba3389700 1 --2- 172.21.15.155:0/996858916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c09d620 0x7ffb9c153c20 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7ffb9400faa0 tx=0x7ffb9400fe40).ready entity=mon.1 client_cookie=c3baa04713fa02c8 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.986 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba3389700 1 --2- 172.21.15.155:0/996858916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c09d620 0x7ffb9c153c20 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7ffb9400faa0 tx=0x7ffb9400fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba1b86700 1 -- 172.21.15.155:0/996858916 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb94001460 con 0x7ffb9c09d620 2019-10-13T00:52:04.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba3389700 1 --2- 172.21.15.155:0/996858916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c09d620 0x7ffb9c153c20 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7ffb9400faa0 tx=0x7ffb9400fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba3389700 1 --2- 172.21.15.155:0/996858916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c09d620 0x7ffb9c153c20 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7ffb9400faa0 tx=0x7ffb9400fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba1b86700 1 -- 172.21.15.155:0/996858916 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94000df0 con 0x7ffb9c09d620 2019-10-13T00:52:04.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba1b86700 1 -- 172.21.15.155:0/996858916 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb94001a40 con 0x7ffb9c09d620 2019-10-13T00:52:04.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba45eb700 1 -- 172.21.15.155:0/996858916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c09d620 msgr2=0x7ffb9c153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.987 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba45eb700 1 --2- 172.21.15.155:0/996858916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c09d620 0x7ffb9c153c20 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7ffb9400faa0 tx=0x7ffb9400fe40).stop 2019-10-13T00:52:04.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba45eb700 1 -- 172.21.15.155:0/996858916 shutdown_connections 2019-10-13T00:52:04.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba45eb700 1 --2- 172.21.15.155:0/996858916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c09d620 0x7ffb9c153c20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba45eb700 1 -- 172.21.15.155:0/996858916 shutdown_connections 2019-10-13T00:52:04.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba45eb700 1 -- 172.21.15.155:0/996858916 wait complete. 2019-10-13T00:52:04.988 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.984+0000 7ffba45eb700 1 -- 172.21.15.155:0/996858916 >> 172.21.15.155:0/996858916 conn(0x7ffb9c099550 msgr2=0x7ffb9c061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:04.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba45eb700 1 Processor -- start 2019-10-13T00:52:04.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba45eb700 1 -- start start 2019-10-13T00:52:04.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba45eb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ffb9c073990 0x7ffb9c091710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba45eb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb9c154020 0x7ffb9c156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba45eb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c08eac0 con 0x7ffb9c0926e0 2019-10-13T00:52:04.992 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba45eb700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c090780 con 0x7ffb9c073990 2019-10-13T00:52:04.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba45eb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c08fd50 con 0x7ffb9c154020 2019-10-13T00:52:04.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba3389700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb9c154020 0x7ffb9c156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba2387700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ffb9c0926e0 0x7ffb9c091c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49424/0 (socket says 172.21.15.155:49424) 2019-10-13T00:52:04.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.987+0000 7ffba2387700 1 -- 172.21.15.155:0/3546711507 learned_addr learned my addr 172.21.15.155:0/3546711507 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:04.993 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba2b88700 1 --2- 172.21.15.155:0/3546711507 >> v2:172.21.15.142:3300/0 conn(0x7ffb9c073990 0x7ffb9c091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 -- 172.21.15.155:0/3546711507 >> v2:172.21.15.142:3300/0 conn(0x7ffb9c073990 msgr2=0x7ffb9c091710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> v2:172.21.15.142:3300/0 conn(0x7ffb9c073990 0x7ffb9c091710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 -- 172.21.15.155:0/3546711507 >> v1:172.21.15.155:6789/0 conn(0x7ffb9c0926e0 legacy=0x7ffb9c091c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549498144 0 0) 0x7ffb8c001170 con 0x7ffb9c0926e0 2019-10-13T00:52:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 -- 172.21.15.155:0/3546711507 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c090e10 con 0x7ffb9c154020 2019-10-13T00:52:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> v2:172.21.15.155:3300/0 conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> v2:172.21.15.155:3300/0 conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb94001420 con 0x7ffb9c154020 2019-10-13T00:52:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94008080 con 0x7ffb9c154020 2019-10-13T00:52:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb94001a00 con 0x7ffb9c154020 2019-10-13T00:52:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c090e10 con 0x7ffb9c154020 2019-10-13T00:52:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c090e10 con 0x7ffb9c154020 2019-10-13T00:52:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.988+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.989+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ffb94001230 con 0x7ffb9c154020 2019-10-13T00:52:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.989+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.989+0000 7ffb937fe700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ffb84017c60 0x7ffb8401a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:04.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.989+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ffb940305c0 con 0x7ffb9c154020 2019-10-13T00:52:04.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.989+0000 7ffba2b88700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ffb84017c60 0x7ffb8401a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:04.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.989+0000 7ffba2b88700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ffb84017c60 0x7ffb8401a0f0 crc :-1 s=READY pgs=94 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:04.997 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:04.997+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c090e10 con 0x7ffb9c154020 2019-10-13T00:52:04.998 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.001+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:04.999 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.001+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ffb94007d70 con 0x7ffb9c154020 2019-10-13T00:52:05.166 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.168+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb9c090e10 con 0x7ffb9c154020 2019-10-13T00:52:05.167 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.169+0000 7ffba3389700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.167 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.169+0000 7ffb937fe700 1 -- 172.21.15.155:0/3546711507 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7ffb94008080 con 0x7ffb9c154020 2019-10-13T00:52:05.168 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:52:05.168 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:52:05.169 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ffb84017c60 msgr2=0x7ffb8401a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.170 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ffb84017c60 0x7ffb8401a0f0 crc :-1 s=READY pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.170 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 msgr2=0x7ffb9c156420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.170 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7ffb94010710 tx=0x7ffb94010ab0).stop 2019-10-13T00:52:05.170 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 shutdown_connections 2019-10-13T00:52:05.170 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb9c154020 0x7ffb9c156420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.170 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 --2- 172.21.15.155:0/3546711507 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ffb84017c60 0x7ffb8401a0f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.170 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 --2- 172.21.15.155:0/3546711507 >> v2:172.21.15.142:3300/0 conn(0x7ffb9c073990 0x7ffb9c091710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.171 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 shutdown_connections 2019-10-13T00:52:05.171 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 wait complete. 2019-10-13T00:52:05.171 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.171+0000 7ffba45eb700 1 -- 172.21.15.155:0/3546711507 >> 172.21.15.155:0/3546711507 conn(0x7ffb9c06d080 msgr2=0x7ffb9c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:05.179 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:44.894568+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:48.109887+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:50.855681+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T00:52:05.180 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:05.180 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T00:52:05.311 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.303+0000 7fd179318700 1 Processor -- start 2019-10-13T00:52:05.311 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd179318700 1 -- start start 2019-10-13T00:52:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd179318700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17409e9c0 0x7fd17414c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd179318700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd17409e1f0 0x7fd1740a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd179318700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17409ada0 con 0x7fd17414be40 2019-10-13T00:52:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd179318700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17409ca60 con 0x7fd17409e1f0 2019-10-13T00:52:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd179318700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17409c030 con 0x7fd17409e9c0 2019-10-13T00:52:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd1737fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd17414be40 0x7fd17409d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49430/0 (socket says 172.21.15.155:49430) 2019-10-13T00:52:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.304+0000 7fd1737fe700 1 -- 172.21.15.155:0/540015677 learned_addr learned my addr 172.21.15.155:0/540015677 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:05.313 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17409e9c0 0x7fd17414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.313 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd173fff700 1 --2- 172.21.15.155:0/540015677 >> v2:172.21.15.142:3300/0 conn(0x7fd17409e1f0 0x7fd1740a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.313 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192555747 0 0) 0x7fd164001150 con 0x7fd17414be40 2019-10-13T00:52:05.313 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd154002e70 con 0x7fd17414be40 2019-10-13T00:52:05.313 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2343474116 0 0) 0x7fd164001130 con 0x7fd17414be40 2019-10-13T00:52:05.313 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17409e9c0 msgr2=0x7fd17414c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17409e9c0 0x7fd17414c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 >> v2:172.21.15.142:3300/0 conn(0x7fd17409e1f0 msgr2=0x7fd1740a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 --2- 172.21.15.155:0/540015677 >> v2:172.21.15.142:3300/0 conn(0x7fd17409e1f0 0x7fd1740a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17409d0f0 con 0x7fd17414be40 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7fd164001660 con 0x7fd17414be40 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 >> v1:172.21.15.155:6789/0 conn(0x7fd17414be40 legacy=0x7fd17409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd154047b60 0x7fd154049fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.314 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15404d3f0 0x7fd15404f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540500d0 con 0x7fd15404d3f0 2019-10-13T00:52:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd154050300 con 0x7fd15404a720 2019-10-13T00:52:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd154050530 con 0x7fd154047b60 2019-10-13T00:52:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17409e9c0 0x7fd17414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd173fff700 1 --2- 172.21.15.155:0/540015677 >> v2:172.21.15.142:3300/0 conn(0x7fd17409e1f0 0x7fd1740a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd1737fe700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd154047b60 0x7fd154049fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.316 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.316 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd173fff700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15404d3f0 0x7fd15404f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.316 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 -- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd154047b60 msgr2=0x7fd154049fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:05.316 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd154047b60 0x7fd154049fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.316 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 -- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15404d3f0 msgr2=0x7fd15404f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:05.317 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd15404d3f0 0x7fd15404f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.317 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.305+0000 7fd172ffd700 1 -- 172.21.15.155:0/540015677 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1540507e0 con 0x7fd15404a720 2019-10-13T00:52:05.317 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd15c00b340 tx=0x7fd15c00bb20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.317 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd15c00b340 tx=0x7fd15c00bb20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.317 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7fd15c00b340 tx=0x7fd15c00bb20).ready entity=mon.1 client_cookie=8d9a4c93e3cc3b93 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.317 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7fd15c00b340 tx=0x7fd15c00bb20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.317 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd15c00a950 con 0x7fd15404a720 2019-10-13T00:52:05.318 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7fd15c00b340 tx=0x7fd15c00bb20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.318 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd172ffd700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7fd15c00b340 tx=0x7fd15c00bb20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.318 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c009cf0 con 0x7fd15404a720 2019-10-13T00:52:05.318 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd1727fc700 1 -- 172.21.15.155:0/540015677 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd15c009ee0 con 0x7fd15404a720 2019-10-13T00:52:05.319 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd179318700 1 -- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 msgr2=0x7fd15404cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.319 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd179318700 1 --2- 172.21.15.155:0/540015677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15404a720 0x7fd15404cbf0 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7fd15c00b340 tx=0x7fd15c00bb20).stop 2019-10-13T00:52:05.319 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd173fff700 1 -- 172.21.15.155:0/540015677 reap_dead start 2019-10-13T00:52:05.319 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd179318700 1 -- 172.21.15.155:0/540015677 shutdown_connections 2019-10-13T00:52:05.319 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd179318700 1 -- 172.21.15.155:0/540015677 shutdown_connections 2019-10-13T00:52:05.320 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd179318700 1 -- 172.21.15.155:0/540015677 wait complete. 2019-10-13T00:52:05.320 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.306+0000 7fd179318700 1 -- 172.21.15.155:0/540015677 >> 172.21.15.155:0/540015677 conn(0x7fd174059b50 msgr2=0x7fd174061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:05.320 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.309+0000 7fd179318700 1 Processor -- start 2019-10-13T00:52:05.320 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.309+0000 7fd179318700 1 -- start start 2019-10-13T00:52:05.320 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd179318700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.320 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd179318700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd174153ea0 0x7fd1741562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.320 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd179318700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17408b000 con 0x7fd17411b2f0 2019-10-13T00:52:05.321 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd179318700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd174088a40 con 0x7fd174153ea0 2019-10-13T00:52:05.321 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd179318700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd174088d10 con 0x7fd17412f930 2019-10-13T00:52:05.321 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd172ffd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd17411b2f0 0x7fd17411a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49444/0 (socket says 172.21.15.155:49444) 2019-10-13T00:52:05.321 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd172ffd700 1 -- 172.21.15.155:0/2359533391 learned_addr learned my addr 172.21.15.155:0/2359533391 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:05.322 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd173fff700 1 --2- 172.21.15.155:0/2359533391 >> v2:172.21.15.142:3300/0 conn(0x7fd174153ea0 0x7fd1741562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.322 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.322 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811205034 0 0) 0x7fd15c0098f0 con 0x7fd17411b2f0 2019-10-13T00:52:05.322 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c002e70 con 0x7fd17411b2f0 2019-10-13T00:52:05.322 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 -- 172.21.15.155:0/2359533391 >> v2:172.21.15.142:3300/0 conn(0x7fd174153ea0 msgr2=0x7fd1741562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.322 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> v2:172.21.15.142:3300/0 conn(0x7fd174153ea0 0x7fd1741562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.323 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 -- 172.21.15.155:0/2359533391 >> v1:172.21.15.155:6789/0 conn(0x7fd17411b2f0 legacy=0x7fd17411a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.323 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 -- 172.21.15.155:0/2359533391 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd174089b00 con 0x7fd17412f930 2019-10-13T00:52:05.323 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd173fff700 1 --2- 172.21.15.155:0/2359533391 >> v2:172.21.15.142:3300/0 conn(0x7fd174153ea0 0x7fd1741562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:05.323 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.323 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.323 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.323 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.324 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd164004440 con 0x7fd17412f930 2019-10-13T00:52:05.324 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.324 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.324 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1640016a0 con 0x7fd17412f930 2019-10-13T00:52:05.325 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd164004a20 con 0x7fd17412f930 2019-10-13T00:52:05.325 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd174089b00 con 0x7fd17412f930 2019-10-13T00:52:05.325 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.310+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd174089b00 con 0x7fd17412f930 2019-10-13T00:52:05.325 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.311+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.325 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.311+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd164001890 con 0x7fd17412f930 2019-10-13T00:52:05.325 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.311+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.311+0000 7fd16bfff700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd14c017ca0 0x7fd14c01a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.311+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fd16402fe80 con 0x7fd17412f930 2019-10-13T00:52:05.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.311+0000 7fd172ffd700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd14c017ca0 0x7fd14c01a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.326 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.312+0000 7fd172ffd700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd14c017ca0 0x7fd14c01a110 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.319+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd174089b00 con 0x7fd17412f930 2019-10-13T00:52:05.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.322+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.327 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.322+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd164001790 con 0x7fd17412f930 2019-10-13T00:52:05.480 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.482+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd174089b00 con 0x7fd17412f930 2019-10-13T00:52:05.481 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.484+0000 7fd1737fe700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.481 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.484+0000 7fd16bfff700 1 -- 172.21.15.155:0/2359533391 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd1640019a0 con 0x7fd17412f930 2019-10-13T00:52:05.481 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:52:05.483 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd14c017ca0 msgr2=0x7fd14c01a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.483 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd14c017ca0 0x7fd14c01a110 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.484 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 msgr2=0x7fd17411a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.484 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7fd164010350 tx=0x7fd1640106f0).stop 2019-10-13T00:52:05.484 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 shutdown_connections 2019-10-13T00:52:05.484 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 --2- 172.21.15.155:0/2359533391 >> v2:172.21.15.142:3300/0 conn(0x7fd174153ea0 0x7fd1741562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.485 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fd14c017ca0 0x7fd14c01a110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.485 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 --2- 172.21.15.155:0/2359533391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd17412f930 0x7fd17411a320 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.485 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 shutdown_connections 2019-10-13T00:52:05.485 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 wait complete. 2019-10-13T00:52:05.486 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.485+0000 7fd179318700 1 -- 172.21.15.155:0/2359533391 >> 172.21.15.155:0/2359533391 conn(0x7fd17412b150 msgr2=0x7fd17407bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:05.493 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:05.494 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T00:52:05.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.617+0000 7f14d9ce3700 1 Processor -- start 2019-10-13T00:52:05.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.617+0000 7f14d9ce3700 1 -- start start 2019-10-13T00:52:05.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d9ce3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d409e9c0 0x7f14d414c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d9ce3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f14d409e1f0 0x7f14d40a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.620 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d9ce3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d409ada0 con 0x7f14d414be40 2019-10-13T00:52:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d9ce3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d409ca60 con 0x7f14d409e1f0 2019-10-13T00:52:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d9ce3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d409c030 con 0x7f14d409e9c0 2019-10-13T00:52:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d3fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f14d414be40 0x7f14d409d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49450/0 (socket says 172.21.15.155:49450) 2019-10-13T00:52:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d3fff700 1 -- 172.21.15.155:0/1116797476 learned_addr learned my addr 172.21.15.155:0/1116797476 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14cbfff700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d409e9c0 0x7f14d414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> v2:172.21.15.142:3300/0 conn(0x7f14d409e1f0 0x7f14d40a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d37fe700 1 -- 172.21.15.155:0/1116797476 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385681829 0 0) 0x7f14c4001150 con 0x7f14d414be40 2019-10-13T00:52:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.618+0000 7f14d37fe700 1 -- 172.21.15.155:0/1116797476 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b4002e70 con 0x7f14d414be40 2019-10-13T00:52:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 -- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d409e9c0 msgr2=0x7f14d414c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d409e9c0 0x7f14d414c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 -- 172.21.15.155:0/1116797476 >> v1:172.21.15.155:6789/0 conn(0x7f14d414be40 legacy=0x7f14d409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.623 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 -- 172.21.15.155:0/1116797476 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d409d0f0 con 0x7f14d409e1f0 2019-10-13T00:52:05.623 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14cbfff700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d409e9c0 0x7f14d414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:05.623 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> v2:172.21.15.142:3300/0 conn(0x7f14d409e1f0 0x7f14d40a3550 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f14cc00b2f0 tx=0x7f14cc00bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.623 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> v2:172.21.15.142:3300/0 conn(0x7f14d409e1f0 0x7f14d40a3550 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f14cc00b2f0 tx=0x7f14cc00bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.623 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d409e1f0 0x7f14d40a3550 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f14cc00b2f0 tx=0x7f14cc00bb00).ready entity=mon.0 client_cookie=3ca2fb48edb3bf99 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.623 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d409e1f0 0x7f14d40a3550 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f14cc00b2f0 tx=0x7f14cc00bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.623 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d37fe700 1 -- 172.21.15.155:0/1116797476 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14cc0075b0 con 0x7f14d409e1f0 2019-10-13T00:52:05.624 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d409e1f0 0x7f14d40a3550 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f14cc00b2f0 tx=0x7f14cc00bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.624 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d8a81700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d409e1f0 0x7f14d40a3550 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f14cc00b2f0 tx=0x7f14cc00bb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.624 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d37fe700 1 -- 172.21.15.155:0/1116797476 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc000f30 con 0x7f14d409e1f0 2019-10-13T00:52:05.624 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d37fe700 1 -- 172.21.15.155:0/1116797476 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14cc00a1d0 con 0x7f14d409e1f0 2019-10-13T00:52:05.625 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d9ce3700 1 -- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d409e1f0 msgr2=0x7f14d40a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.625 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.619+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d409e1f0 0x7f14d40a3550 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f14cc00b2f0 tx=0x7f14cc00bb00).stop 2019-10-13T00:52:05.625 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.620+0000 7f14d9ce3700 1 -- 172.21.15.155:0/1116797476 shutdown_connections 2019-10-13T00:52:05.625 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.620+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d409e1f0 0x7f14d40a3550 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.625 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.620+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/1116797476 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d409e9c0 0x7f14d414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.625 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.620+0000 7f14d9ce3700 1 -- 172.21.15.155:0/1116797476 shutdown_connections 2019-10-13T00:52:05.626 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.620+0000 7f14d9ce3700 1 -- 172.21.15.155:0/1116797476 wait complete. 2019-10-13T00:52:05.626 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.620+0000 7f14d9ce3700 1 -- 172.21.15.155:0/1116797476 >> 172.21.15.155:0/1116797476 conn(0x7f14d4059b50 msgr2=0x7f14d4061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:05.626 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.622+0000 7f14d9ce3700 1 Processor -- start 2019-10-13T00:52:05.626 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.622+0000 7f14d9ce3700 1 -- start start 2019-10-13T00:52:05.630 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d412f930 0x7f14d408a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.630 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f14d411a220 0x7f14d408a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.630 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f14d408acf0 0x7f14d4085ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.630 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d40892e0 con 0x7f14d411a220 2019-10-13T00:52:05.630 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d40888b0 con 0x7f14d408acf0 2019-10-13T00:52:05.630 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d4088b80 con 0x7f14d412f930 2019-10-13T00:52:05.631 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f14d408acf0 0x7f14d4085ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.631 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f14d408acf0 0x7f14d4085ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59384/0 (socket says 172.21.15.155:59384) 2019-10-13T00:52:05.631 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 -- 172.21.15.155:0/4083554574 learned_addr learned my addr 172.21.15.155:0/4083554574 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:05.631 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14cbfff700 1 --2- 172.21.15.155:0/4083554574 >> v2:172.21.15.142:3300/0 conn(0x7f14d411a220 0x7f14d408a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.631 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d3fff700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d412f930 0x7f14d408a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.631 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 -- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d412f930 msgr2=0x7f14d408a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.631 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d412f930 0x7f14d408a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.632 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 -- 172.21.15.155:0/4083554574 >> v2:172.21.15.142:3300/0 conn(0x7f14d411a220 msgr2=0x7f14d408a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.632 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> v2:172.21.15.142:3300/0 conn(0x7f14d411a220 0x7f14d408a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.632 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 -- 172.21.15.155:0/4083554574 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4089970 con 0x7f14d408acf0 2019-10-13T00:52:05.632 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> v2:172.21.15.155:3300/0 conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.632 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> v2:172.21.15.155:3300/0 conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.633 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=READY pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.633 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.633 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.633 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d17fa700 1 -- 172.21.15.155:0/4083554574 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14cc00b350 con 0x7f14d408acf0 2019-10-13T00:52:05.633 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.633 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d17fa700 1 -- 172.21.15.155:0/4083554574 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc009d20 con 0x7f14d408acf0 2019-10-13T00:52:05.634 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d17fa700 1 -- 172.21.15.155:0/4083554574 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14cc00a3b0 con 0x7f14d408acf0 2019-10-13T00:52:05.634 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d4089970 con 0x7f14d408acf0 2019-10-13T00:52:05.634 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.623+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d4089970 con 0x7f14d408acf0 2019-10-13T00:52:05.634 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.624+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.635 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.624+0000 7f14d17fa700 1 -- 172.21.15.155:0/4083554574 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f14cc009920 con 0x7f14d408acf0 2019-10-13T00:52:05.635 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.624+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.635 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.624+0000 7f14d17fa700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f14ac017c60 0x7f14ac01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.635 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.624+0000 7f14d17fa700 1 -- 172.21.15.155:0/4083554574 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f14cc030440 con 0x7f14d408acf0 2019-10-13T00:52:05.635 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.624+0000 7f14d3fff700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f14ac017c60 0x7f14ac01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.636 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.624+0000 7f14d3fff700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f14ac017c60 0x7f14ac01a0f0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.636 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.632+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d4089970 con 0x7f14d408acf0 2019-10-13T00:52:05.637 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.636+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.637 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.636+0000 7f14d17fa700 1 -- 172.21.15.155:0/4083554574 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f14cc00b350 con 0x7f14d408acf0 2019-10-13T00:52:05.796 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.798+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f14d4086ab0 con 0x7f14d408acf0 2019-10-13T00:52:05.796 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.798+0000 7f14d8a81700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.797 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.798+0000 7f14d17fa700 1 -- 172.21.15.155:0/4083554574 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f14cc009920 con 0x7f14d408acf0 2019-10-13T00:52:05.797 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:52:05.799 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f14ac017c60 msgr2=0x7f14ac01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.799 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f14ac017c60 0x7f14ac01a0f0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.799 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 msgr2=0x7f14d4085ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.799 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 secure :-1 s=READY pgs=167 cs=0 l=1 rx=0x7f14cc0107c0 tx=0x7f14cc010b60).stop 2019-10-13T00:52:05.799 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 shutdown_connections 2019-10-13T00:52:05.800 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14d408acf0 0x7f14d4085ca0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.800 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/4083554574 >> v2:172.21.15.142:3300/0 conn(0x7f14d411a220 0x7f14d408a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.800 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f14ac017c60 0x7f14ac01a0f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.800 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.800+0000 7f14d9ce3700 1 --2- 172.21.15.155:0/4083554574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14d412f930 0x7f14d408a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.800 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.801+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 shutdown_connections 2019-10-13T00:52:05.801 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.801+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 wait complete. 2019-10-13T00:52:05.801 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.801+0000 7f14d9ce3700 1 -- 172.21.15.155:0/4083554574 >> 172.21.15.155:0/4083554574 conn(0x7f14d412b150 msgr2=0x7f14d407bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:05.808 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:05.809 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T00:52:05.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.933+0000 7fb2e4670700 1 Processor -- start 2019-10-13T00:52:05.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.933+0000 7fb2e4670700 1 -- start start 2019-10-13T00:52:05.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.934+0000 7fb2e4670700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb2dc14be40 0x7fb2dc14c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.934+0000 7fb2e4670700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:05.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.934+0000 7fb2e4670700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc09e650 con 0x7fb2dc09e9c0 2019-10-13T00:52:05.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.934+0000 7fb2e4670700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc09ca60 con 0x7fb2dc14be40 2019-10-13T00:52:05.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.934+0000 7fb2e4670700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc09c030 con 0x7fb2dc09e1f0 2019-10-13T00:52:05.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.934+0000 7fb2e340e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.934+0000 7fb2e240c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb2dc09e9c0 0x7fb2dc09d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49470/0 (socket says 172.21.15.155:49470) 2019-10-13T00:52:05.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e240c700 1 -- 172.21.15.155:0/4207730664 learned_addr learned my addr 172.21.15.155:0/4207730664 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:05.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/4207730664 >> v2:172.21.15.142:3300/0 conn(0x7fb2dc14be40 0x7fb2dc14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 -- 172.21.15.155:0/4207730664 >> v2:172.21.15.142:3300/0 conn(0x7fb2dc14be40 msgr2=0x7fb2dc14c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:05.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 --2- 172.21.15.155:0/4207730664 >> v2:172.21.15.142:3300/0 conn(0x7fb2dc14be40 0x7fb2dc14c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 -- 172.21.15.155:0/4207730664 >> v1:172.21.15.155:6789/0 conn(0x7fb2dc09e9c0 legacy=0x7fb2dc09d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.938 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e1c0b700 1 -- 172.21.15.155:0/4207730664 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146958395 0 0) 0x7fb2cc001070 con 0x7fb2dc09e9c0 2019-10-13T00:52:05.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 -- 172.21.15.155:0/4207730664 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc09d0f0 con 0x7fb2dc09e1f0 2019-10-13T00:52:05.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 --2- 172.21.15.155:0/4207730664 >> v2:172.21.15.155:3300/0 conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2d400af40 tx=0x7fb2d400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 --2- 172.21.15.155:0/4207730664 >> v2:172.21.15.155:3300/0 conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2d400af40 tx=0x7fb2d400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 --2- 172.21.15.155:0/4207730664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7fb2d400af40 tx=0x7fb2d400b750).ready entity=mon.1 client_cookie=dab9d2e550a5d7b0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 --2- 172.21.15.155:0/4207730664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7fb2d400af40 tx=0x7fb2d400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.939 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e1c0b700 1 -- 172.21.15.155:0/4207730664 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d4009b90 con 0x7fb2dc09e1f0 2019-10-13T00:52:05.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 --2- 172.21.15.155:0/4207730664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7fb2d400af40 tx=0x7fb2d400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e340e700 1 --2- 172.21.15.155:0/4207730664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7fb2d400af40 tx=0x7fb2d400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e1c0b700 1 -- 172.21.15.155:0/4207730664 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4000a70 con 0x7fb2dc09e1f0 2019-10-13T00:52:05.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e1c0b700 1 -- 172.21.15.155:0/4207730664 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d400a170 con 0x7fb2dc09e1f0 2019-10-13T00:52:05.940 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e4670700 1 -- 172.21.15.155:0/4207730664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2dc09e1f0 msgr2=0x7fb2dc0a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.935+0000 7fb2e4670700 1 --2- 172.21.15.155:0/4207730664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7fb2d400af40 tx=0x7fb2d400b750).stop 2019-10-13T00:52:05.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.936+0000 7fb2e4670700 1 -- 172.21.15.155:0/4207730664 shutdown_connections 2019-10-13T00:52:05.941 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.936+0000 7fb2e4670700 1 --2- 172.21.15.155:0/4207730664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2dc09e1f0 0x7fb2dc0a3550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.943 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.936+0000 7fb2e4670700 1 --2- 172.21.15.155:0/4207730664 >> v2:172.21.15.142:3300/0 conn(0x7fb2dc14be40 0x7fb2dc14c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:05.944 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.936+0000 7fb2e4670700 1 -- 172.21.15.155:0/4207730664 shutdown_connections 2019-10-13T00:52:05.944 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.936+0000 7fb2e4670700 1 -- 172.21.15.155:0/4207730664 wait complete. 2019-10-13T00:52:05.945 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.936+0000 7fb2e4670700 1 -- 172.21.15.155:0/4207730664 >> 172.21.15.155:0/4207730664 conn(0x7fb2dc059b50 msgr2=0x7fb2dc061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:05.945 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.938+0000 7fb2e4670700 1 Processor -- start 2019-10-13T00:52:05.945 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.938+0000 7fb2e4670700 1 -- start start 2019-10-13T00:52:05.949 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e4670700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.951 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e4670700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2dc153ea0 0x7fb2dc1562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.952 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e4670700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc13c980 con 0x7fb2dc1405a0 2019-10-13T00:52:05.952 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e4670700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc13e640 con 0x7fb2dc08aa40 2019-10-13T00:52:05.952 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e4670700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc13dc10 con 0x7fb2dc153ea0 2019-10-13T00:52:05.952 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e240c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb2dc1405a0 0x7fb2dc13fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49474/0 (socket says 172.21.15.155:49474) 2019-10-13T00:52:05.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e240c700 1 -- 172.21.15.155:0/3849157949 learned_addr learned my addr 172.21.15.155:0/3849157949 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:05.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> v2:172.21.15.142:3300/0 conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2e340e700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2dc153ea0 0x7fb2dc1562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292752531 0 0) 0x7fb2cc001180 con 0x7fb2dc1405a0 2019-10-13T00:52:05.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.939+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4002e70 con 0x7fb2dc1405a0 2019-10-13T00:52:05.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 -- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2dc153ea0 msgr2=0x7fb2dc1562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2dc153ea0 0x7fb2dc1562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:05.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 -- 172.21.15.155:0/3849157949 >> v1:172.21.15.155:6789/0 conn(0x7fb2dc1405a0 legacy=0x7fb2dc13fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:05.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 -- 172.21.15.155:0/3849157949 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc13ecd0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e340e700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2dc153ea0 0x7fb2dc1562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:05.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> v2:172.21.15.142:3300/0 conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> v2:172.21.15.142:3300/0 conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d8000cc0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d80058f0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d8000cc0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc13ecd0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc13ecd0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.940+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.941+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb2d80054f0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.941+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.941+0000 7fb2d37fe700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb2c4017ca0 0x7fb2c401a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:05.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.941+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb2d8030390 con 0x7fb2dc08aa40 2019-10-13T00:52:05.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.941+0000 7fb2e240c700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb2c4017ca0 0x7fb2c401a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:05.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.941+0000 7fb2e240c700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb2c4017ca0 0x7fb2c401a110 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:05.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.949+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc13ecd0 con 0x7fb2dc08aa40 2019-10-13T00:52:05.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.952+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:05.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:05.952+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb2d8000cc0 con 0x7fb2dc08aa40 2019-10-13T00:52:06.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.112+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb2dc13ecd0 con 0x7fb2dc08aa40 2019-10-13T00:52:06.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.113+0000 7fb2e2c0d700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.111 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.113+0000 7fb2d37fe700 1 -- 172.21.15.155:0/3849157949 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb2d80054f0 con 0x7fb2dc08aa40 2019-10-13T00:52:06.111 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:52:06.113 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb2c4017ca0 msgr2=0x7fb2c401a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.113 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb2c4017ca0 0x7fb2c401a110 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.113 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 msgr2=0x7fb2dc13f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fb2d8010700 tx=0x7fb2d8010aa0).stop 2019-10-13T00:52:06.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 shutdown_connections 2019-10-13T00:52:06.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fb2c4017ca0 0x7fb2c401a110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2dc153ea0 0x7fb2dc1562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 --2- 172.21.15.155:0/3849157949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2dc08aa40 0x7fb2dc13f5d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.114 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 shutdown_connections 2019-10-13T00:52:06.115 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 wait complete. 2019-10-13T00:52:06.115 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.115+0000 7fb2e4670700 1 -- 172.21.15.155:0/3849157949 >> 172.21.15.155:0/3849157949 conn(0x7fb2dc07bef0 msgr2=0x7fb2dc11a160 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:06.123 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:06.123 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T00:52:06.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.243+0000 7f7469f96700 1 Processor -- start 2019-10-13T00:52:06.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.243+0000 7f7469f96700 1 -- start start 2019-10-13T00:52:06.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7469f96700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:06.248 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7469f96700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7464064f00 con 0x7f746409d620 2019-10-13T00:52:06.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7469f96700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74640987e0 con 0x7f746409cff0 2019-10-13T00:52:06.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7469f96700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746409ac50 con 0x7f7464099b10 2019-10-13T00:52:06.249 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7468d34700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f746409cff0 0x7f7464152c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49482/0 (socket says 172.21.15.155:49482) 2019-10-13T00:52:06.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7468d34700 1 -- 172.21.15.155:0/133274549 learned_addr learned my addr 172.21.15.155:0/133274549 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:06.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7463fff700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:06.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f74637fe700 1 -- 172.21.15.155:0/133274549 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71970642 0 0) 0x7f745c0014d0 con 0x7f746409cff0 2019-10-13T00:52:06.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f74637fe700 1 -- 172.21.15.155:0/133274549 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7444002e70 con 0x7f746409cff0 2019-10-13T00:52:06.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f74637fe700 1 -- 172.21.15.155:0/133274549 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513177871 0 0) 0x7f744c001080 con 0x7f746409d620 2019-10-13T00:52:06.252 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f74637fe700 1 -- 172.21.15.155:0/133274549 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7444004160 con 0x7f746409d620 2019-10-13T00:52:06.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7463fff700 1 -- 172.21.15.155:0/133274549 >> v1:172.21.15.155:6789/0 conn(0x7f746409cff0 legacy=0x7f7464152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7463fff700 1 -- 172.21.15.155:0/133274549 >> v1:172.21.15.142:6789/0 conn(0x7f746409d620 legacy=0x7f746409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.253 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.244+0000 7f7463fff700 1 -- 172.21.15.155:0/133274549 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746409bd10 con 0x7f7464099b10 2019-10-13T00:52:06.254 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7463fff700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f745400af40 tx=0x7f745400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.254 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7463fff700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f745400af40 tx=0x7f745400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7463fff700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f745400af40 tx=0x7f745400b750).ready entity=mon.2 client_cookie=50083471d536d7a5 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:06.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7463fff700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f745400af40 tx=0x7f745400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f74637fe700 1 -- 172.21.15.155:0/133274549 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7454000960 con 0x7f7464099b10 2019-10-13T00:52:06.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7463fff700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f745400af40 tx=0x7f745400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7463fff700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f745400af40 tx=0x7f745400b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.255 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f74637fe700 1 -- 172.21.15.155:0/133274549 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7454009850 con 0x7f7464099b10 2019-10-13T00:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f74637fe700 1 -- 172.21.15.155:0/133274549 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7454000960 con 0x7f7464099b10 2019-10-13T00:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7469f96700 1 -- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 msgr2=0x7f7464099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.245+0000 7f7469f96700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f745400af40 tx=0x7f745400b750).stop 2019-10-13T00:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.246+0000 7f7469f96700 1 -- 172.21.15.155:0/133274549 shutdown_connections 2019-10-13T00:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.246+0000 7f7469f96700 1 --2- 172.21.15.155:0/133274549 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7464099b10 0x7f7464099f00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.246+0000 7f7469f96700 1 -- 172.21.15.155:0/133274549 shutdown_connections 2019-10-13T00:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.246+0000 7f7469f96700 1 -- 172.21.15.155:0/133274549 wait complete. 2019-10-13T00:52:06.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.246+0000 7f7469f96700 1 -- 172.21.15.155:0/133274549 >> 172.21.15.155:0/133274549 conn(0x7f7464099550 msgr2=0x7f7464061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:06.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.248+0000 7f7469f96700 1 Processor -- start 2019-10-13T00:52:06.257 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.248+0000 7f7469f96700 1 -- start start 2019-10-13T00:52:06.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7469f96700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f746408c8b0 0x7f746414bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:06.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7469f96700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74641367e0 con 0x7f7464073770 2019-10-13T00:52:06.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7469f96700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7464073b60 con 0x7f746414b7f0 2019-10-13T00:52:06.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7469f96700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74640727e0 con 0x7f746408c8b0 2019-10-13T00:52:06.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7463fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f746414b7f0 0x7f746406fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49486/0 (socket says 172.21.15.155:49486) 2019-10-13T00:52:06.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7463fff700 1 -- 172.21.15.155:0/8398411 learned_addr learned my addr 172.21.15.155:0/8398411 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:06.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> v2:172.21.15.155:3300/0 conn(0x7f746408c8b0 0x7f746414bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:06.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359504338 0 0) 0x7f7454000940 con 0x7f746414b7f0 2019-10-13T00:52:06.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f743c002e70 con 0x7f746414b7f0 2019-10-13T00:52:06.265 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 -- 172.21.15.155:0/8398411 >> v1:172.21.15.155:6789/0 conn(0x7f746414b7f0 legacy=0x7f746406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.265 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 -- 172.21.15.155:0/8398411 >> v1:172.21.15.142:6789/0 conn(0x7f7464073770 legacy=0x7f7464070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.265 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 -- 172.21.15.155:0/8398411 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7464072e70 con 0x7f746408c8b0 2019-10-13T00:52:06.265 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> v2:172.21.15.155:3300/0 conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.265 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> v2:172.21.15.155:3300/0 conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.265 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f744c000aa0 con 0x7f746408c8b0 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c00efc0 con 0x7f746408c8b0 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f744c00f2b0 con 0x7f746408c8b0 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7464072e70 con 0x7f746408c8b0 2019-10-13T00:52:06.266 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.249+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7464072e70 con 0x7f746408c8b0 2019-10-13T00:52:06.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.250+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.250+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f744c000aa0 con 0x7f746408c8b0 2019-10-13T00:52:06.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.250+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.250+0000 7f74617fa700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f743c017ca0 0x7f743c01a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:06.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.250+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f744c030020 con 0x7f746408c8b0 2019-10-13T00:52:06.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.250+0000 7f7463fff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f743c017ca0 0x7f743c01a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:06.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.250+0000 7f7463fff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f743c017ca0 0x7f743c01a110 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:06.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.258+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746408c570 con 0x7f746408c8b0 2019-10-13T00:52:06.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.262+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.262+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f744c00eb90 con 0x7f746408c8b0 2019-10-13T00:52:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.427+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7464072e70 con 0x7f746408c8b0 2019-10-13T00:52:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.427+0000 7f745bfff700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.427+0000 7f74617fa700 1 -- 172.21.15.155:0/8398411 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2108 (secure 0 0 0) 0x7f744c000aa0 con 0x7f746408c8b0 2019-10-13T00:52:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:dumped fsmap epoch 6 2019-10-13T00:52:06.427 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:52:06.427 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.155:6833/1182541438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1182541438},{"type":"v1","addr":"172.21.15.155:6833","nonce":1182541438}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T00:51:53.188987+0000","modified":"2019-10-13T00:52:00.887832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4217":{"gid":4217,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6835/1835893052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":1835893052},{"type":"v1","addr":"172.21.15.142:6835","nonce":1835893052}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T00:52:06.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f743c017ca0 msgr2=0x7f743c01a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f743c017ca0 0x7f743c01a110 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.428 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 msgr2=0x7f746414bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f744c0103b0 tx=0x7f744c010750).stop 2019-10-13T00:52:06.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 shutdown_connections 2019-10-13T00:52:06.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f746408c8b0 0x7f746414bbe0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 --2- 172.21.15.155:0/8398411 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f743c017ca0 0x7f743c01a110 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.429 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 shutdown_connections 2019-10-13T00:52:06.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 wait complete. 2019-10-13T00:52:06.430 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.430+0000 7f7469f96700 1 -- 172.21.15.155:0/8398411 >> 172.21.15.155:0/8398411 conn(0x7f746406c410 msgr2=0x7f7464074840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:06.640 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4217}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4217': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 3, u'state': u'up:active', u'gid': 4217, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 1835893052, u'type': u'v2', u'addr': u'172.21.15.142:6834'}, {u'nonce': 1835893052, u'type': u'v1', u'addr': u'172.21.15.142:6835'}]}, u'addr': u'172.21.15.142:6835/1835893052'}}, u'fs_name': u'cephfs', u'created': u'2019-10-13T00:51:53.188987+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-10-13T00:52:00.887832+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-10-13T00:52:06.640 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-10-13T00:52:06.640 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:52:06.640 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-10-13T00:52:06.776 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2019-10-13T00:52:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "cluster_fsid": "9c6ba46f-16c8-4ed6-8640-ff20acbbecb0", 2019-10-13T00:52:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "whoami": 0, 2019-10-13T00:52:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "id": 4217, 2019-10-13T00:52:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "want_state": "up:active", 2019-10-13T00:52:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "up:active", 2019-10-13T00:52:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "rank_uptime": 6.8906804599999996, 2019-10-13T00:52:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "mdsmap_epoch": 6, 2019-10-13T00:52:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_epoch": 23, 2019-10-13T00:52:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_epoch_barrier": 23, 2019-10-13T00:52:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "uptime": 11.122040180999999 2019-10-13T00:52:06.779 INFO:teuthology.orchestra.run.smithi142.stdout:} 2019-10-13T00:52:06.781 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "9c6ba46f-16c8-4ed6-8640-ff20acbbecb0", "whoami": 0, "id": 4217, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.8906804599999996, "mdsmap_epoch": 6, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "uptime": 11.122040180999999 } 2019-10-13T00:52:06.782 INFO:teuthology.run_tasks:Running task ceph-fuse... 2019-10-13T00:52:06.810 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2019-10-13T00:52:06.810 INFO:tasks.ceph_fuse:config is None 2019-10-13T00:52:06.811 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2019-10-13T00:52:06.811 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2019-10-13T00:52:06.811 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:06.811 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T00:52:06.945 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.931+0000 7f607cba3700 1 Processor -- start 2019-10-13T00:52:06.946 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.932+0000 7f607cba3700 1 -- start start 2019-10-13T00:52:06.947 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.932+0000 7f607cba3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6078099b10 0x7f6078099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:06.947 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.932+0000 7f607cba3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078064f00 con 0x7f607809cff0 2019-10-13T00:52:06.947 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.932+0000 7f607cba3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60780987e0 con 0x7f607809d620 2019-10-13T00:52:06.947 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.932+0000 7f607cba3700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607809b680 con 0x7f6078099b10 2019-10-13T00:52:06.947 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6078099b10 0x7f6078099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:06.947 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6078099b10 0x7f6078099f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59418/0 (socket says 172.21.15.155:59418) 2019-10-13T00:52:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 -- 172.21.15.155:0/3023412623 learned_addr learned my addr 172.21.15.155:0/3023412623 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6075d9b700 1 -- 172.21.15.155:0/3023412623 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480745545 0 0) 0x7f60700009b0 con 0x7f607809cff0 2019-10-13T00:52:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 -- 172.21.15.155:0/3023412623 >> v1:172.21.15.155:6789/0 conn(0x7f607809d620 legacy=0x7f607809c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 -- 172.21.15.155:0/3023412623 >> v1:172.21.15.142:6789/0 conn(0x7f607809cff0 legacy=0x7f6078152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 -- 172.21.15.155:0/3023412623 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607809bd10 con 0x7f6078099b10 2019-10-13T00:52:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- 172.21.15.155:0/3023412623 >> v2:172.21.15.155:3300/0 conn(0x7f6078099b10 0x7f6078099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6060008770 tx=0x7f606000fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- 172.21.15.155:0/3023412623 >> v2:172.21.15.155:3300/0 conn(0x7f6078099b10 0x7f6078099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6060008770 tx=0x7f606000fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- 172.21.15.155:0/3023412623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6078099b10 0x7f6078099f00 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f6060008770 tx=0x7f606000fe40).ready entity=mon.1 client_cookie=8596e4f354c8dbd0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- 172.21.15.155:0/3023412623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6078099b10 0x7f6078099f00 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f6060008770 tx=0x7f606000fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6075d9b700 1 -- 172.21.15.155:0/3023412623 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6060000ce0 con 0x7f6078099b10 2019-10-13T00:52:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- 172.21.15.155:0/3023412623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6078099b10 0x7f6078099f00 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f6060008770 tx=0x7f606000fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6076d9d700 1 --2- 172.21.15.155:0/3023412623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6078099b10 0x7f6078099f00 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f6060008770 tx=0x7f606000fe40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6075d9b700 1 -- 172.21.15.155:0/3023412623 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6060001010 con 0x7f6078099b10 2019-10-13T00:52:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f6075d9b700 1 -- 172.21.15.155:0/3023412623 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60600016e0 con 0x7f6078099b10 2019-10-13T00:52:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f607cba3700 1 -- 172.21.15.155:0/3023412623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6078099b10 msgr2=0x7f6078099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.933+0000 7f607cba3700 1 --2- 172.21.15.155:0/3023412623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6078099b10 0x7f6078099f00 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f6060008770 tx=0x7f606000fe40).stop 2019-10-13T00:52:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.934+0000 7f607cba3700 1 -- 172.21.15.155:0/3023412623 shutdown_connections 2019-10-13T00:52:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.934+0000 7f607cba3700 1 --2- 172.21.15.155:0/3023412623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6078099b10 0x7f6078099f00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.934+0000 7f607cba3700 1 -- 172.21.15.155:0/3023412623 shutdown_connections 2019-10-13T00:52:06.951 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.934+0000 7f607cba3700 1 -- 172.21.15.155:0/3023412623 wait complete. 2019-10-13T00:52:06.951 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.934+0000 7f607cba3700 1 -- 172.21.15.155:0/3023412623 >> 172.21.15.155:0/3023412623 conn(0x7f6078099550 msgr2=0x7f6078061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.936+0000 7f607cba3700 1 Processor -- start 2019-10-13T00:52:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.936+0000 7f607cba3700 1 -- start start 2019-10-13T00:52:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f607cba3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:06.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f607cba3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607808d4b0 con 0x7f6078091690 2019-10-13T00:52:06.953 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f607cba3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607808de90 con 0x7f6078152ee0 2019-10-13T00:52:06.954 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f607cba3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607808ed00 con 0x7f6078073990 2019-10-13T00:52:06.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f607759e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6078152ee0 0x7f60781552e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49502/0 (socket says 172.21.15.155:49502) 2019-10-13T00:52:06.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f607759e700 1 -- 172.21.15.155:0/2379463938 learned_addr learned my addr 172.21.15.155:0/2379463938 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:06.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:06.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683620114 0 0) 0x7f6070001200 con 0x7f6078152ee0 2019-10-13T00:52:06.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6050002e70 con 0x7f6078152ee0 2019-10-13T00:52:06.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846249352 0 0) 0x7f6068000c40 con 0x7f6078091690 2019-10-13T00:52:06.955 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.937+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60500030a0 con 0x7f6078091690 2019-10-13T00:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 -- 172.21.15.155:0/2379463938 >> v1:172.21.15.155:6789/0 conn(0x7f6078152ee0 legacy=0x7f60781552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 -- 172.21.15.155:0/2379463938 >> v1:172.21.15.142:6789/0 conn(0x7f6078091690 legacy=0x7f6078090c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 -- 172.21.15.155:0/2379463938 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607808fdc0 con 0x7f6078073990 2019-10-13T00:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:06.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6060007630 con 0x7f6078073990 2019-10-13T00:52:06.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60600056e0 con 0x7f6078073990 2019-10-13T00:52:06.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6060005e30 con 0x7f6078073990 2019-10-13T00:52:06.957 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607808fdc0 con 0x7f6078073990 2019-10-13T00:52:06.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607808fdc0 con 0x7f6078073990 2019-10-13T00:52:06.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f60600058d0 con 0x7f6078073990 2019-10-13T00:52:06.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.938+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.939+0000 7f606f7fe700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f6050019ea0 0x7f605001c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:06.958 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.939+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f6060030330 con 0x7f6078073990 2019-10-13T00:52:06.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.939+0000 7f607659c700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f6050019ea0 0x7f605001c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:06.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.939+0000 7f607659c700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f6050019ea0 0x7f605001c330 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:06.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.947+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607808fdc0 con 0x7f6078073990 2019-10-13T00:52:06.959 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.951+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:06.960 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:06.951+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f6060000cc0 con 0x7f6078073990 2019-10-13T00:52:07.119 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.120+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f607808fdc0 con 0x7f6078073990 2019-10-13T00:52:07.119 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.121+0000 7f6076d9d700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.119 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.121+0000 7f606f7fe700 1 -- 172.21.15.155:0/2379463938 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f6060016e30 con 0x7f6078073990 2019-10-13T00:52:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:52:07.121 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":23,"fsid":"9c6ba46f-16c8-4ed6-8640-ff20acbbecb0","created":"2019-10-13T00:51:30.205399+0000","modified":"2019-10-13T00:51:53.188793+0000","last_up_change":"2019-10-13T00:51:40.798196+0000","last_in_change":"2019-10-13T00:51:40.798196+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T00:51:44.894568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T00:51:48.109887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T00:51:50.855681+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"90b46d14-0a9a-4f76-b5d7-c1e4716ea22d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":7109},{"type":"v1","addr":"172.21.15.155:6801","nonce":7109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":7109},{"type":"v1","addr":"172.21.15.155:6803","nonce":7109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":7109},{"type":"v1","addr":"172.21.15.155:6807","nonce":7109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":7109},{"type":"v1","addr":"172.21.15.155:6805","nonce":7109}]},"public_addr":"172.21.15.155:6801/7109","cluster_addr":"172.21.15.155:6803/7109","heartbeat_back_addr":"172.21.15.155:6807/7109","heartbeat_front_addr":"172.21.15.155:6805/7109","state":["exists","up"]},{"osd":1,"uuid":"9fbcb085-3256-410b-85fc-dfd7fc572b50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":7110},{"type":"v1","addr":"172.21.15.155:6809","nonce":7110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":7110},{"type":"v1","addr":"172.21.15.155:6811","nonce":7110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":7110},{"type":"v1","addr":"172.21.15.155:6815","nonce":7110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":7110},{"type":"v1","addr":"172.21.15.155:6813","nonce":7110}]},"public_addr":"172.21.15.155:6809/7110","cluster_addr":"172.21.15.155:6811/7110","heartbeat_back_addr":"172.21.15.155:6815/7110","heartbeat_front_addr":"172.21.15.155:6813/7110","state":["exists","up"]},{"osd":2,"uuid":"df99f110-da25-4a6d-93d8-20d8ab22b345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6817","nonce":7112},{"type":"v1","addr":"172.21.15.155:6819","nonce":7112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":7112},{"type":"v1","addr":"172.21.15.155:6824","nonce":7112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":7112},{"type":"v1","addr":"172.21.15.155:6831","nonce":7112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":7112},{"type":"v1","addr":"172.21.15.155:6829","nonce":7112}]},"public_addr":"172.21.15.155:6819/7112","cluster_addr":"172.21.15.155:6824/7112","heartbeat_back_addr":"172.21.15.155:6831/7112","heartbeat_front_addr":"172.21.15.155:6829/7112","state":["exists","up"]},{"osd":3,"uuid":"7cbcdaa5-ac79-4661-aa09-faf34ceb2662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":7111},{"type":"v1","addr":"172.21.15.155:6818","nonce":7111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":7111},{"type":"v1","addr":"172.21.15.155:6821","nonce":7111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":7111},{"type":"v1","addr":"172.21.15.155:6828","nonce":7111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6823","nonce":7111},{"type":"v1","addr":"172.21.15.155:6825","nonce":7111}]},"public_addr":"172.21.15.155:6818/7111","cluster_addr":"172.21.15.155:6821/7111","heartbeat_back_addr":"172.21.15.155:6828/7111","heartbeat_front_addr":"172.21.15.155:6825/7111","state":["exists","up"]},{"osd":4,"uuid":"83c00323-8853-4f30-84a0-711565f5a804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":7037},{"type":"v1","addr":"172.21.15.142:6803","nonce":7037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":7037},{"type":"v1","addr":"172.21.15.142:6805","nonce":7037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":7037},{"type":"v1","addr":"172.21.15.142:6809","nonce":7037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":7037},{"type":"v1","addr":"172.21.15.142:6807","nonce":7037}]},"public_addr":"172.21.15.142:6803/7037","cluster_addr":"172.21.15.142:6805/7037","heartbeat_back_addr":"172.21.15.142:6809/7037","heartbeat_front_addr":"172.21.15.142:6807/7037","state":["exists","up"]},{"osd":5,"uuid":"d1142116-2074-437f-bb80-2cdef5bcd668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":7038},{"type":"v1","addr":"172.21.15.142:6811","nonce":7038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":7038},{"type":"v1","addr":"172.21.15.142:6813","nonce":7038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":7038},{"type":"v1","addr":"172.21.15.142:6817","nonce":7038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":7038},{"type":"v1","addr":"172.21.15.142:6815","nonce":7038}]},"public_addr":"172.21.15.142:6811/7038","cluster_addr":"172.21.15.142:6813/7038","heartbeat_back_addr":"172.21.15.142:6817/7038","heartbeat_front_addr":"172.21.15.142:6815/7038","state":["exists","up"]},{"osd":6,"uuid":"410ababe-9879-4664-941a-5f5764c1eb0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6819","nonce":7039},{"type":"v1","addr":"172.21.15.142:6821","nonce":7039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6823","nonce":7039},{"type":"v1","addr":"172.21.15.142:6825","nonce":7039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6831","nonce":7039},{"type":"v1","addr":"172.21.15.142:6833","nonce":7039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6827","nonce":7039},{"type":"v1","addr":"172.21.15.142:6829","nonce":7039}]},"public_addr":"172.21.15.142:6821/7039","cluster_addr":"172.21.15.142:6825/7039","heartbeat_back_addr":"172.21.15.142:6833/7039","heartbeat_front_addr":"172.21.15.142:6829/7039","state":["exists","up"]},{"osd":7,"uuid":"1bf99cb1-c304-47f3-9fde-1ca95c7a8266","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":7040},{"type":"v1","addr":"172.21.15.142:6820","nonce":7040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":7040},{"type":"v1","addr":"172.21.15.142:6824","nonce":7040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":7040},{"type":"v1","addr":"172.21.15.142:6832","nonce":7040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":7040},{"type":"v1","addr":"172.21.15.142:6828","nonce":7040}]},"public_addr":"172.21.15.142:6820/7040","cluster_addr":"172.21.15.142:6824/7040","heartbeat_back_addr":"172.21.15.142:6832/7040","heartbeat_front_addr":"172.21.15.142:6828/7040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.966013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.947070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.919198+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.905825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.929376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.917678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.983427+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T00:51:38.934912+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T00:52:07.123 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f6050019ea0 msgr2=0x7f605001c330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.123 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f6050019ea0 0x7f605001c330 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.123 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 msgr2=0x7f60780906c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.123 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f6060010700 tx=0x7f6060010aa0).stop 2019-10-13T00:52:07.124 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 shutdown_connections 2019-10-13T00:52:07.124 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f6050019ea0 0x7f605001c330 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.124 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 --2- 172.21.15.155:0/2379463938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6078073990 0x7f60780906c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.124 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 shutdown_connections 2019-10-13T00:52:07.124 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 wait complete. 2019-10-13T00:52:07.124 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.125+0000 7f607cba3700 1 -- 172.21.15.155:0/2379463938 >> 172.21.15.155:0/2379463938 conn(0x7f607806d080 msgr2=0x7f607806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:07.133 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:44.894568+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:48.109887+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T00:51:50.855681+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T00:52:07.133 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:07.134 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T00:52:07.259 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.257+0000 7f8733aaf700 1 Processor -- start 2019-10-13T00:52:07.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.257+0000 7f8733aaf700 1 -- start start 2019-10-13T00:52:07.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f8733aaf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:07.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f8733aaf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f872c09e1f0 0x7f872c0a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:07.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f8733aaf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872c09ada0 con 0x7f872c14be40 2019-10-13T00:52:07.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f8733aaf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c09ca60 con 0x7f872c09e1f0 2019-10-13T00:52:07.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f8733aaf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c09c030 con 0x7f872c09e9c0 2019-10-13T00:52:07.260 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f873184b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f873284d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f872c09e1f0 0x7f872c0a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f873204c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f872c14be40 0x7f872c09d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:57524/0 (socket says 172.21.15.155:57524) 2019-10-13T00:52:07.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.258+0000 7f873204c700 1 -- 172.21.15.155:0/702375185 learned_addr learned my addr 172.21.15.155:0/702375185 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:07.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 -- 172.21.15.155:0/702375185 >> v2:172.21.15.142:3300/0 conn(0x7f872c09e1f0 msgr2=0x7f872c0a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:07.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 --2- 172.21.15.155:0/702375185 >> v2:172.21.15.142:3300/0 conn(0x7f872c09e1f0 0x7f872c0a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:07.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 -- 172.21.15.155:0/702375185 >> v1:172.21.15.142:6789/0 conn(0x7f872c14be40 legacy=0x7f872c09d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.261 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 -- 172.21.15.155:0/702375185 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c09d0f0 con 0x7f872c09e9c0 2019-10-13T00:52:07.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873284d700 1 --2- 172.21.15.155:0/702375185 >> v2:172.21.15.142:3300/0 conn(0x7f872c09e1f0 0x7f872c0a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:07.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f871c00af40 tx=0x7f871c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f871c00af40 tx=0x7f871c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7f871c00af40 tx=0x7f871c00b750).ready entity=mon.2 client_cookie=29cbbc6a564f62cf server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f871c00af40 tx=0x7f871c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f871c00af40 tx=0x7f871c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.262 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873104a700 1 -- 172.21.15.155:0/702375185 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f871c000960 con 0x7f872c09e9c0 2019-10-13T00:52:07.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873184b700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f871c00af40 tx=0x7f871c00b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873104a700 1 -- 172.21.15.155:0/702375185 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c009850 con 0x7f872c09e9c0 2019-10-13T00:52:07.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f873104a700 1 -- 172.21.15.155:0/702375185 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f871c009f20 con 0x7f872c09e9c0 2019-10-13T00:52:07.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f8733aaf700 1 -- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 msgr2=0x7f872c14c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.259+0000 7f8733aaf700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7f871c00af40 tx=0x7f871c00b750).stop 2019-10-13T00:52:07.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.260+0000 7f8733aaf700 1 -- 172.21.15.155:0/702375185 shutdown_connections 2019-10-13T00:52:07.263 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.260+0000 7f8733aaf700 1 --2- 172.21.15.155:0/702375185 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09e9c0 0x7f872c14c230 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.260+0000 7f8733aaf700 1 --2- 172.21.15.155:0/702375185 >> v2:172.21.15.142:3300/0 conn(0x7f872c09e1f0 0x7f872c0a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:07.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.260+0000 7f8733aaf700 1 -- 172.21.15.155:0/702375185 shutdown_connections 2019-10-13T00:52:07.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.260+0000 7f8733aaf700 1 -- 172.21.15.155:0/702375185 wait complete. 2019-10-13T00:52:07.264 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.260+0000 7f8733aaf700 1 -- 172.21.15.155:0/702375185 >> 172.21.15.155:0/702375185 conn(0x7f872c059b50 msgr2=0x7f872c061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:07.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.262+0000 7f8733aaf700 1 Processor -- start 2019-10-13T00:52:07.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.262+0000 7f8733aaf700 1 -- start start 2019-10-13T00:52:07.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f8733aaf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f872c08aa70 0x7f872c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f8733aaf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872c08aed0 con 0x7f872c11a220 2019-10-13T00:52:07.267 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f8733aaf700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872c087440 con 0x7f872c12f930 2019-10-13T00:52:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f8733aaf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c0892e0 con 0x7f872c08aa70 2019-10-13T00:52:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f873204c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f872c12f930 0x7f872c08a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49514/0 (socket says 172.21.15.155:49514) 2019-10-13T00:52:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f873204c700 1 -- 172.21.15.155:0/302477115 learned_addr learned my addr 172.21.15.155:0/302477115 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> v2:172.21.15.142:3300/0 conn(0x7f872c08aa70 0x7f872c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293808585 0 0) 0x7f8728000fe0 con 0x7f872c12f930 2019-10-13T00:52:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8714002e70 con 0x7f872c12f930 2019-10-13T00:52:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413148370 0 0) 0x7f871c000940 con 0x7f872c11a220 2019-10-13T00:52:07.269 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f873284d700 1 -- 172.21.15.155:0/302477115 >> v1:172.21.15.155:6789/0 conn(0x7f872c12f930 legacy=0x7f872c08a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.269 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f873284d700 1 -- 172.21.15.155:0/302477115 >> v1:172.21.15.142:6789/0 conn(0x7f872c11a220 legacy=0x7f872c152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.269 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f873284d700 1 -- 172.21.15.155:0/302477115 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c089970 con 0x7f872c08aa70 2019-10-13T00:52:07.269 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.263+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> v2:172.21.15.142:3300/0 conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.269 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> v2:172.21.15.142:3300/0 conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.269 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.269 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.270 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8724000bd0 con 0x7f872c08aa70 2019-10-13T00:52:07.270 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.270 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8724000b30 con 0x7f872c08aa70 2019-10-13T00:52:07.270 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.270 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c089970 con 0x7f872c08aa70 2019-10-13T00:52:07.270 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87240055a0 con 0x7f872c08aa70 2019-10-13T00:52:07.270 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c089970 con 0x7f872c08aa70 2019-10-13T00:52:07.271 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.271 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f87240051c0 con 0x7f872c08aa70 2019-10-13T00:52:07.271 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.271 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8722ffd700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8714017cb0 0x7f871401a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.272 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.264+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f872402ffe0 con 0x7f872c08aa70 2019-10-13T00:52:07.272 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.265+0000 7f873204c700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8714017cb0 0x7f871401a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.272 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.265+0000 7f873204c700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8714017cb0 0x7f871401a120 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.272 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.273+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872c089970 con 0x7f872c08aa70 2019-10-13T00:52:07.274 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.276+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.274 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.276+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f87240092c0 con 0x7f872c08aa70 2019-10-13T00:52:07.435 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.437+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f872c089970 con 0x7f872c08aa70 2019-10-13T00:52:07.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.438+0000 7f873284d700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.436 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.438+0000 7f8722ffd700 1 -- 172.21.15.155:0/302477115 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8724000b10 con 0x7f872c08aa70 2019-10-13T00:52:07.437 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:52:07.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.440+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8714017cb0 msgr2=0x7f871401a120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.440+0000 7f8733aaf700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8714017cb0 0x7f871401a120 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.440+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 msgr2=0x7f872c1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.440+0000 7f8733aaf700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7f8724010380 tx=0x7f8724010720).stop 2019-10-13T00:52:07.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.440+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 shutdown_connections 2019-10-13T00:52:07.439 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.440+0000 7f8733aaf700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c08aa70 0x7f872c1566b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.440+0000 7f8733aaf700 1 --2- 172.21.15.155:0/302477115 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f8714017cb0 0x7f871401a120 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.441+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 shutdown_connections 2019-10-13T00:52:07.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.441+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 wait complete. 2019-10-13T00:52:07.440 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.441+0000 7f8733aaf700 1 -- 172.21.15.155:0/302477115 >> 172.21.15.155:0/302477115 conn(0x7f872c12b150 msgr2=0x7f872c07bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:07.450 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:07.450 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T00:52:07.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.575+0000 7ff04eb34700 1 Processor -- start 2019-10-13T00:52:07.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.575+0000 7ff04eb34700 1 -- start start 2019-10-13T00:52:07.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04eb34700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff04814be40 0x7ff04814c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:07.577 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04eb34700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04809e650 con 0x7ff04809e9c0 2019-10-13T00:52:07.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04eb34700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04809ada0 con 0x7ff04809e1f0 2019-10-13T00:52:07.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04eb34700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04809ca60 con 0x7ff04814be40 2019-10-13T00:52:07.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff04814be40 0x7ff04814c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff04814be40 0x7ff04814c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59446/0 (socket says 172.21.15.155:59446) 2019-10-13T00:52:07.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 -- 172.21.15.155:0/2439607049 learned_addr learned my addr 172.21.15.155:0/2439607049 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:07.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d8d2700 1 --1- 172.21.15.155:0/2439607049 >> v1:172.21.15.155:6789/0 conn(0x7ff04809e1f0 0x7ff048152b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49524/0 (socket says 172.21.15.155:49524) 2019-10-13T00:52:07.578 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 -- 172.21.15.155:0/2439607049 >> v1:172.21.15.155:6789/0 conn(0x7ff04809e1f0 legacy=0x7ff048152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 -- 172.21.15.155:0/2439607049 >> v1:172.21.15.142:6789/0 conn(0x7ff04809e9c0 legacy=0x7ff04809d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff047fff700 1 -- 172.21.15.155:0/2439607049 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817114596 0 0) 0x7ff0400009b0 con 0x7ff04809e1f0 2019-10-13T00:52:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 -- 172.21.15.155:0/2439607049 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04809d0f0 con 0x7ff04814be40 2019-10-13T00:52:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2439607049 >> v2:172.21.15.155:3300/0 conn(0x7ff04814be40 0x7ff04814c230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff03800fab0 tx=0x7ff03800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2439607049 >> v2:172.21.15.155:3300/0 conn(0x7ff04814be40 0x7ff04814c230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff03800fab0 tx=0x7ff03800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2439607049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff04814be40 0x7ff04814c230 secure :-1 s=READY pgs=179 cs=0 l=1 rx=0x7ff03800fab0 tx=0x7ff03800fe50).ready entity=mon.1 client_cookie=f4ff250c61b9e88e server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2439607049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff04814be40 0x7ff04814c230 secure :-1 s=THROTTLE_DONE pgs=179 cs=0 l=1 rx=0x7ff03800fab0 tx=0x7ff03800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff047fff700 1 -- 172.21.15.155:0/2439607049 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff038000ce0 con 0x7ff04814be40 2019-10-13T00:52:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2439607049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff04814be40 0x7ff04814c230 secure :-1 s=THROTTLE_DONE pgs=179 cs=0 l=1 rx=0x7ff03800fab0 tx=0x7ff03800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2439607049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff04814be40 0x7ff04814c230 secure :-1 s=THROTTLE_DONE pgs=179 cs=0 l=1 rx=0x7ff03800fab0 tx=0x7ff03800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff047fff700 1 -- 172.21.15.155:0/2439607049 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff038001010 con 0x7ff04814be40 2019-10-13T00:52:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff047fff700 1 -- 172.21.15.155:0/2439607049 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0380016e0 con 0x7ff04814be40 2019-10-13T00:52:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04eb34700 1 -- 172.21.15.155:0/2439607049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff04814be40 msgr2=0x7ff04814c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.576+0000 7ff04eb34700 1 --2- 172.21.15.155:0/2439607049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff04814be40 0x7ff04814c230 secure :-1 s=READY pgs=179 cs=0 l=1 rx=0x7ff03800fab0 tx=0x7ff03800fe50).stop 2019-10-13T00:52:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.577+0000 7ff04eb34700 1 -- 172.21.15.155:0/2439607049 shutdown_connections 2019-10-13T00:52:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.577+0000 7ff04eb34700 1 --2- 172.21.15.155:0/2439607049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff04814be40 0x7ff04814c230 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.577+0000 7ff04eb34700 1 -- 172.21.15.155:0/2439607049 shutdown_connections 2019-10-13T00:52:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.577+0000 7ff04eb34700 1 -- 172.21.15.155:0/2439607049 wait complete. 2019-10-13T00:52:07.582 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.577+0000 7ff04eb34700 1 -- 172.21.15.155:0/2439607049 >> 172.21.15.155:0/2439607049 conn(0x7ff048059b50 msgr2=0x7ff048061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:07.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.579+0000 7ff04eb34700 1 Processor -- start 2019-10-13T00:52:07.586 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.580+0000 7ff04eb34700 1 -- start start 2019-10-13T00:52:07.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.580+0000 7ff04eb34700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff04808aa40 0x7ff04813f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.580+0000 7ff04eb34700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff048153ea0 0x7ff0481562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.580+0000 7ff04eb34700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04813c980 con 0x7ff0481405a0 2019-10-13T00:52:07.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.580+0000 7ff04eb34700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04813e640 con 0x7ff04808aa40 2019-10-13T00:52:07.587 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.580+0000 7ff04eb34700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04813dc10 con 0x7ff048153ea0 2019-10-13T00:52:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04c8d0700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff0481405a0 0x7ff04813fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.155:57544/0 (socket says 172.21.15.155:57544) 2019-10-13T00:52:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04c8d0700 1 -- 172.21.15.155:0/2146800673 learned_addr learned my addr 172.21.15.155:0/2146800673 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> v2:172.21.15.142:3300/0 conn(0x7ff04808aa40 0x7ff04813f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d8d2700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff048153ea0 0x7ff0481562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 -- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff048153ea0 msgr2=0x7ff0481562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff048153ea0 0x7ff0481562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 -- 172.21.15.155:0/2146800673 >> v1:172.21.15.142:6789/0 conn(0x7ff0481405a0 legacy=0x7ff04813fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 -- 172.21.15.155:0/2146800673 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04813ecd0 con 0x7ff04808aa40 2019-10-13T00:52:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347947717 0 0) 0x7ff03c000fe0 con 0x7ff0481405a0 2019-10-13T00:52:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d8d2700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff048153ea0 0x7ff0481562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T00:52:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> v2:172.21.15.142:3300/0 conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> v2:172.21.15.142:3300/0 conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0380084f0 con 0x7ff04808aa40 2019-10-13T00:52:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0380058f0 con 0x7ff04808aa40 2019-10-13T00:52:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff038005ae0 con 0x7ff04808aa40 2019-10-13T00:52:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04813ecd0 con 0x7ff04808aa40 2019-10-13T00:52:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.581+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04813ecd0 con 0x7ff04808aa40 2019-10-13T00:52:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.582+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.582+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff0380054f0 con 0x7ff04808aa40 2019-10-13T00:52:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.582+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.582+0000 7ff045ffb700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff028017c60 0x7ff02801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.582+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff038030370 con 0x7ff04808aa40 2019-10-13T00:52:07.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.582+0000 7ff04c8d0700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff028017c60 0x7ff02801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.594 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.582+0000 7ff04c8d0700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff028017c60 0x7ff02801a0f0 crc :-1 s=READY pgs=102 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.590+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04813ecd0 con 0x7ff04808aa40 2019-10-13T00:52:07.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.594+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.595 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.594+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff0380084f0 con 0x7ff04808aa40 2019-10-13T00:52:07.753 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.755+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff04813ecd0 con 0x7ff04808aa40 2019-10-13T00:52:07.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.756+0000 7ff04d0d1700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.754 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.756+0000 7ff045ffb700 1 -- 172.21.15.155:0/2146800673 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff0380054f0 con 0x7ff04808aa40 2019-10-13T00:52:07.755 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:52:07.756 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff028017c60 msgr2=0x7ff02801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.757 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff028017c60 0x7ff02801a0f0 crc :-1 s=READY pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.757 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 msgr2=0x7ff04813f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.757 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7ff0380077c0 tx=0x7ff038010a70).stop 2019-10-13T00:52:07.757 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 shutdown_connections 2019-10-13T00:52:07.757 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7ff028017c60 0x7ff02801a0f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff048153ea0 0x7ff0481562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 --2- 172.21.15.155:0/2146800673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff04808aa40 0x7ff04813f5d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 shutdown_connections 2019-10-13T00:52:07.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 wait complete. 2019-10-13T00:52:07.758 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.758+0000 7ff04eb34700 1 -- 172.21.15.155:0/2146800673 >> 172.21.15.155:0/2146800673 conn(0x7ff04807bef0 msgr2=0x7ff04811a160 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:07.766 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:07.766 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T00:52:07.897 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.890+0000 7f49bbafd700 1 Processor -- start 2019-10-13T00:52:07.897 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.890+0000 7f49bbafd700 1 -- start start 2019-10-13T00:52:07.897 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.891+0000 7f49bbafd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49b414c230 0x7f49b409e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:07.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.891+0000 7f49bbafd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b409ada0 con 0x7f49b409e9c0 2019-10-13T00:52:07.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.891+0000 7f49bbafd700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b409b250 con 0x7f49b414be40 2019-10-13T00:52:07.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.891+0000 7f49bbafd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b409ca60 con 0x7f49b414c230 2019-10-13T00:52:07.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.891+0000 7f49ba09a700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f49b414be40 0x7f49b409d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49536/0 (socket says 172.21.15.155:49536) 2019-10-13T00:52:07.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.891+0000 7f49ba09a700 1 -- 172.21.15.155:0/2603091369 learned_addr learned my addr 172.21.15.155:0/2603091369 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:07.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49b414c230 0x7f49b409e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.898 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349402868 0 0) 0x7f49b00014d0 con 0x7f49b414be40 2019-10-13T00:52:07.899 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a0002e70 con 0x7f49b414be40 2019-10-13T00:52:07.899 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595710292 0 0) 0x7f49a4001080 con 0x7f49b409e9c0 2019-10-13T00:52:07.899 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a00030a0 con 0x7f49b409e9c0 2019-10-13T00:52:07.899 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3449075149 0 0) 0x7f49b00014b0 con 0x7f49b414be40 2019-10-13T00:52:07.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49b414c230 msgr2=0x7f49b409e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:07.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49b414c230 0x7f49b409e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:07.913 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 >> v1:172.21.15.142:6789/0 conn(0x7f49b409e9c0 legacy=0x7f49b4150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b409d0f0 con 0x7f49b414be40 2019-10-13T00:52:07.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 972502067 0 0) 0x7f49b00019e0 con 0x7f49b414be40 2019-10-13T00:52:07.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 >> v1:172.21.15.155:6789/0 conn(0x7f49b414be40 legacy=0x7f49b409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a0003bb0 0x7f49a0006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:07.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a0006740 0x7f49a0008b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:07.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:07.914 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a004ffd0 con 0x7f49a0003bb0 2019-10-13T00:52:07.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00501c0 con 0x7f49a004d2f0 2019-10-13T00:52:07.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00503f0 con 0x7f49a0006740 2019-10-13T00:52:07.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba09a700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a0003bb0 0x7f49a0006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49b9899700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a0006740 0x7f49a0008b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba89b700 1 -- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a0006740 msgr2=0x7f49a0008b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:07.915 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a0006740 0x7f49a0008b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:07.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba89b700 1 -- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a0003bb0 msgr2=0x7f49a0006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T00:52:07.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a0003bb0 0x7f49a0006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T00:52:07.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.892+0000 7f49ba89b700 1 -- 172.21.15.155:0/2603091369 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00506a0 con 0x7f49a004d2f0 2019-10-13T00:52:07.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f49ac002e20 tx=0x7f49ac00fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f49ac002e20 tx=0x7f49ac00fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 secure :-1 s=READY pgs=181 cs=0 l=1 rx=0x7f49ac002e20 tx=0x7f49ac00fa00).ready entity=mon.1 client_cookie=3e15f076ac652a62 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.916 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f49ac002e20 tx=0x7f49ac00fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.923 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49ac000b10 con 0x7f49a004d2f0 2019-10-13T00:52:07.924 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f49ac002e20 tx=0x7f49ac00fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.924 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49ba89b700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f49ac002e20 tx=0x7f49ac00fa00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.924 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ac001060 con 0x7f49a004d2f0 2019-10-13T00:52:07.924 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49b9098700 1 -- 172.21.15.155:0/2603091369 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49ac001730 con 0x7f49a004d2f0 2019-10-13T00:52:07.924 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49bbafd700 1 -- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 msgr2=0x7f49a004f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.924 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49bbafd700 1 --2- 172.21.15.155:0/2603091369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49a004d2f0 0x7f49a004f7c0 secure :-1 s=READY pgs=181 cs=0 l=1 rx=0x7f49ac002e20 tx=0x7f49ac00fa00).stop 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49ba89b700 1 -- 172.21.15.155:0/2603091369 reap_dead start 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.893+0000 7f49bbafd700 1 -- 172.21.15.155:0/2603091369 shutdown_connections 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.894+0000 7f49bbafd700 1 -- 172.21.15.155:0/2603091369 shutdown_connections 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.894+0000 7f49bbafd700 1 -- 172.21.15.155:0/2603091369 wait complete. 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.894+0000 7f49bbafd700 1 -- 172.21.15.155:0/2603091369 >> 172.21.15.155:0/2603091369 conn(0x7f49b4059b50 msgr2=0x7f49b4061fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.896+0000 7f49bbafd700 1 Processor -- start 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.896+0000 7f49bbafd700 1 -- start start 2019-10-13T00:52:07.925 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.896+0000 7f49bbafd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f49b4078b40 0x7f49b414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.926 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.896+0000 7f49bbafd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f49b4131b50 0x7f49b4120710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.926 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49bbafd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b4079590 con 0x7f49b4132430 2019-10-13T00:52:07.926 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49bbafd700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b414b6a0 con 0x7f49b4131b50 2019-10-13T00:52:07.926 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49bbafd700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b414a1d0 con 0x7f49b4078b40 2019-10-13T00:52:07.926 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f49b4078b40 0x7f49b414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.926 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f49b4078b40 0x7f49b414b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59472/0 (socket says 172.21.15.155:59472) 2019-10-13T00:52:07.926 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 -- 172.21.15.155:0/43598017 learned_addr learned my addr 172.21.15.155:0/43598017 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:07.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba89b700 1 --2- 172.21.15.155:0/43598017 >> v2:172.21.15.142:3300/0 conn(0x7f49b4131b50 0x7f49b4120710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 -- 172.21.15.155:0/43598017 >> v2:172.21.15.142:3300/0 conn(0x7f49b4131b50 msgr2=0x7f49b4120710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> v2:172.21.15.142:3300/0 conn(0x7f49b4131b50 0x7f49b4120710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:07.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 -- 172.21.15.155:0/43598017 >> v1:172.21.15.142:6789/0 conn(0x7f49b4132430 legacy=0x7f49b411ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:07.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 -- 172.21.15.155:0/43598017 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b414a860 con 0x7f49b4078b40 2019-10-13T00:52:07.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> v2:172.21.15.155:3300/0 conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.927 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> v2:172.21.15.155:3300/0 conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=READY pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.928 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49aaffd700 1 -- 172.21.15.155:0/43598017 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b00047a0 con 0x7f49b4078b40 2019-10-13T00:52:07.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49aaffd700 1 -- 172.21.15.155:0/43598017 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b0001270 con 0x7f49b4078b40 2019-10-13T00:52:07.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49aaffd700 1 -- 172.21.15.155:0/43598017 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b0004d80 con 0x7f49b4078b40 2019-10-13T00:52:07.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b414a860 con 0x7f49b4078b40 2019-10-13T00:52:07.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.897+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b414a860 con 0x7f49b4078b40 2019-10-13T00:52:07.934 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.898+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.935 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.898+0000 7f49aaffd700 1 -- 172.21.15.155:0/43598017 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f49b00045b0 con 0x7f49b4078b40 2019-10-13T00:52:07.935 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.898+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.935 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.898+0000 7f49aaffd700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f499c017c60 0x7f499c01a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:07.935 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.898+0000 7f49aaffd700 1 -- 172.21.15.155:0/43598017 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f49b00303f0 con 0x7f49b4078b40 2019-10-13T00:52:07.935 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.898+0000 7f49b9899700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f499c017c60 0x7f499c01a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:07.936 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.898+0000 7f49b9899700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f499c017c60 0x7f499c01a0f0 crc :-1 s=READY pgs=103 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:07.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.906+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b414a860 con 0x7f49b4078b40 2019-10-13T00:52:07.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.909+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:07.937 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:07.909+0000 7f49aaffd700 1 -- 172.21.15.155:0/43598017 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f49b0001690 con 0x7f49b4078b40 2019-10-13T00:52:08.069 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.071+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f49b414a860 con 0x7f49b4078b40 2019-10-13T00:52:08.069 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.071+0000 7f49ba09a700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.069 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.071+0000 7f49aaffd700 1 -- 172.21.15.155:0/43598017 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f49b00045b0 con 0x7f49b4078b40 2019-10-13T00:52:08.070 INFO:teuthology.orchestra.run.smithi155.stdout:pg_num: 8 2019-10-13T00:52:08.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.073+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f499c017c60 msgr2=0x7f499c01a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.073+0000 7f49bbafd700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f499c017c60 0x7f499c01a0f0 crc :-1 s=READY pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.073+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 msgr2=0x7f49b414b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.073+0000 7f49bbafd700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 secure :-1 s=READY pgs=182 cs=0 l=1 rx=0x7f49b00107a0 tx=0x7f49b0010b40).stop 2019-10-13T00:52:08.072 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.074+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 shutdown_connections 2019-10-13T00:52:08.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.074+0000 7f49bbafd700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7f499c017c60 0x7f499c01a0f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.074+0000 7f49bbafd700 1 --2- 172.21.15.155:0/43598017 >> v2:172.21.15.142:3300/0 conn(0x7f49b4131b50 0x7f49b4120710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.074+0000 7f49bbafd700 1 --2- 172.21.15.155:0/43598017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49b4078b40 0x7f49b414b160 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.073 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.074+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 shutdown_connections 2019-10-13T00:52:08.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.074+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 wait complete. 2019-10-13T00:52:08.074 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.074+0000 7f49bbafd700 1 -- 172.21.15.155:0/43598017 >> 172.21.15.155:0/43598017 conn(0x7f49b407beb0 msgr2=0x7f49b4130b50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:08.081 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2019-10-13T00:52:08.081 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:08.082 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T00:52:08.203 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.201+0000 7fc0f5d45700 1 Processor -- start 2019-10-13T00:52:08.203 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.201+0000 7fc0f5d45700 1 -- start start 2019-10-13T00:52:08.203 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0f5d45700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc0f0099b10 0x7fc0f0099f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T00:52:08.203 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0f5d45700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f0064f00 con 0x7fc0f009d620 2019-10-13T00:52:08.204 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0f5d45700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00987e0 con 0x7fc0f009cff0 2019-10-13T00:52:08.204 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0f5d45700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f009b680 con 0x7fc0f0099b10 2019-10-13T00:52:08.204 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0f4ae3700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc0f009cff0 0x7fc0f0152c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49560/0 (socket says 172.21.15.155:49560) 2019-10-13T00:52:08.204 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0f4ae3700 1 -- 172.21.15.155:0/3882946143 learned_addr learned my addr 172.21.15.155:0/3882946143 (peer_addr_for_me v1:172.21.15.155:0/0) 2019-10-13T00:52:08.204 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0effff700 1 --2- 172.21.15.155:0/3882946143 >> v2:172.21.15.142:3300/0 conn(0x7fc0f0099b10 0x7fc0f0099f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:08.204 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0ef7fe700 1 -- 172.21.15.155:0/3882946143 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147313023 0 0) 0x7fc0e8001480 con 0x7fc0f009cff0 2019-10-13T00:52:08.205 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0ef7fe700 1 -- 172.21.15.155:0/3882946143 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d0002e70 con 0x7fc0f009cff0 2019-10-13T00:52:08.205 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0ef7fe700 1 -- 172.21.15.155:0/3882946143 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638968159 0 0) 0x7fc0d8001080 con 0x7fc0f009d620 2019-10-13T00:52:08.205 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0ef7fe700 1 -- 172.21.15.155:0/3882946143 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d0004160 con 0x7fc0f009d620 2019-10-13T00:52:08.205 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0effff700 1 -- 172.21.15.155:0/3882946143 >> v1:172.21.15.155:6789/0 conn(0x7fc0f009cff0 legacy=0x7fc0f0152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.205 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0effff700 1 -- 172.21.15.155:0/3882946143 >> v1:172.21.15.142:6789/0 conn(0x7fc0f009d620 legacy=0x7fc0f009c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.205 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0effff700 1 -- 172.21.15.155:0/3882946143 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f009bd10 con 0x7fc0f0099b10 2019-10-13T00:52:08.205 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.202+0000 7fc0effff700 1 --2- 172.21.15.155:0/3882946143 >> v2:172.21.15.142:3300/0 conn(0x7fc0f0099b10 0x7fc0f0099f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc0e000af40 tx=0x7fc0e000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.206 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0effff700 1 --2- 172.21.15.155:0/3882946143 >> v2:172.21.15.142:3300/0 conn(0x7fc0f0099b10 0x7fc0f0099f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc0e000af40 tx=0x7fc0e000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.206 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0effff700 1 --2- 172.21.15.155:0/3882946143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0f0099b10 0x7fc0f0099f00 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7fc0e000af40 tx=0x7fc0e000b750).ready entity=mon.0 client_cookie=b091017ebdb0ec5e server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:08.206 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0effff700 1 --2- 172.21.15.155:0/3882946143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0f0099b10 0x7fc0f0099f00 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fc0e000af40 tx=0x7fc0e000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.206 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0effff700 1 --2- 172.21.15.155:0/3882946143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0f0099b10 0x7fc0f0099f00 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fc0e000af40 tx=0x7fc0e000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.206 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0ef7fe700 1 -- 172.21.15.155:0/3882946143 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0e0000960 con 0x7fc0f0099b10 2019-10-13T00:52:08.206 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0effff700 1 --2- 172.21.15.155:0/3882946143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0f0099b10 0x7fc0f0099f00 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fc0e000af40 tx=0x7fc0e000b750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.206 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0ef7fe700 1 -- 172.21.15.155:0/3882946143 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e0009850 con 0x7fc0f0099b10 2019-10-13T00:52:08.207 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0ef7fe700 1 -- 172.21.15.155:0/3882946143 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0e0009f20 con 0x7fc0f0099b10 2019-10-13T00:52:08.207 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0f5d45700 1 -- 172.21.15.155:0/3882946143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0f0099b10 msgr2=0x7fc0f0099f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.207 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0f5d45700 1 --2- 172.21.15.155:0/3882946143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0f0099b10 0x7fc0f0099f00 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7fc0e000af40 tx=0x7fc0e000b750).stop 2019-10-13T00:52:08.207 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0f5d45700 1 -- 172.21.15.155:0/3882946143 shutdown_connections 2019-10-13T00:52:08.207 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0f5d45700 1 --2- 172.21.15.155:0/3882946143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0f0099b10 0x7fc0f0099f00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.207 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.203+0000 7fc0f5d45700 1 -- 172.21.15.155:0/3882946143 shutdown_connections 2019-10-13T00:52:08.207 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.204+0000 7fc0f5d45700 1 -- 172.21.15.155:0/3882946143 wait complete. 2019-10-13T00:52:08.208 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.204+0000 7fc0f5d45700 1 -- 172.21.15.155:0/3882946143 >> 172.21.15.155:0/3882946143 conn(0x7fc0f0099550 msgr2=0x7fc0f0061f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:08.214 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0f5d45700 1 Processor -- start 2019-10-13T00:52:08.214 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0f5d45700 1 -- start start 2019-10-13T00:52:08.215 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0f5d45700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc0f0088dd0 0x7fc0f008d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:08.215 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0f5d45700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0f0198430 0x7fc0f019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:08.215 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0f5d45700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00894a0 con 0x7fc0f008e800 2019-10-13T00:52:08.215 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0f5d45700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f0072e90 con 0x7fc0f0088dd0 2019-10-13T00:52:08.215 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0f5d45700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0072460 con 0x7fc0f0198430 2019-10-13T00:52:08.215 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.206+0000 7fc0effff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc0f0088dd0 0x7fc0f008d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:08.215 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc0f0088dd0 0x7fc0f008d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59486/0 (socket says 172.21.15.155:59486) 2019-10-13T00:52:08.216 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 -- 172.21.15.155:0/291666985 learned_addr learned my addr 172.21.15.155:0/291666985 (peer_addr_for_me v2:172.21.15.155:0/0) 2019-10-13T00:52:08.216 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0e7fff700 1 --1- 172.21.15.155:0/291666985 >> v1:172.21.15.155:6789/0 conn(0x7fc0f008e800 0x7fc0f008dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49564/0 (socket says 172.21.15.155:49564) 2019-10-13T00:52:08.216 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0f4ae3700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0f0198430 0x7fc0f019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:08.216 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 -- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0f0198430 msgr2=0x7fc0f019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.216 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0f0198430 0x7fc0f019a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.216 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 -- 172.21.15.155:0/291666985 >> v1:172.21.15.155:6789/0 conn(0x7fc0f008e800 legacy=0x7fc0f008dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.216 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 -- 172.21.15.155:0/291666985 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f0073520 con 0x7fc0f0088dd0 2019-10-13T00:52:08.217 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> v2:172.21.15.155:3300/0 conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.217 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> v2:172.21.15.155:3300/0 conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.217 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:08.217 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.217 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0ed7fa700 1 -- 172.21.15.155:0/291666985 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0e000b020 con 0x7fc0f0088dd0 2019-10-13T00:52:08.217 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f0073520 con 0x7fc0f0088dd0 2019-10-13T00:52:08.217 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.218 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.218 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0ed7fa700 1 -- 172.21.15.155:0/291666985 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e000af80 con 0x7fc0f0088dd0 2019-10-13T00:52:08.218 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0ed7fa700 1 -- 172.21.15.155:0/291666985 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0e000b170 con 0x7fc0f0088dd0 2019-10-13T00:52:08.218 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f0073520 con 0x7fc0f0088dd0 2019-10-13T00:52:08.219 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.207+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.219 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.208+0000 7fc0ed7fa700 1 -- 172.21.15.155:0/291666985 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fc0e0009570 con 0x7fc0f0088dd0 2019-10-13T00:52:08.219 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.208+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.219 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.208+0000 7fc0ed7fa700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc0c8017c60 0x7fc0c801a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T00:52:08.219 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.208+0000 7fc0ed7fa700 1 -- 172.21.15.155:0/291666985 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc0e0030070 con 0x7fc0f0088dd0 2019-10-13T00:52:08.219 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.208+0000 7fc0e7fff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc0c8017c60 0x7fc0c801a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T00:52:08.220 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.208+0000 7fc0e7fff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc0c8017c60 0x7fc0c801a0f0 crc :-1 s=READY pgs=104 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T00:52:08.220 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.216+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f0073520 con 0x7fc0f0088dd0 2019-10-13T00:52:08.221 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.220+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.221 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.220+0000 7fc0ed7fa700 1 -- 172.21.15.155:0/291666985 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fc0e00256b0 con 0x7fc0f0088dd0 2019-10-13T00:52:08.383 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.385+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f0073520 con 0x7fc0f0088dd0 2019-10-13T00:52:08.383 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.385+0000 7fc0effff700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T00:52:08.384 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.385+0000 7fc0ed7fa700 1 -- 172.21.15.155:0/291666985 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2108 (secure 0 0 0) 0x7fc0e00256b0 con 0x7fc0f0088dd0 2019-10-13T00:52:08.384 INFO:teuthology.orchestra.run.smithi155.stderr:dumped fsmap epoch 6 2019-10-13T00:52:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: 2019-10-13T00:52:08.384 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.155:6833/1182541438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1182541438},{"type":"v1","addr":"172.21.15.155:6833","nonce":1182541438}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T00:51:53.188987+0000","modified":"2019-10-13T00:52:00.887832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4217":{"gid":4217,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6835/1835893052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":1835893052},{"type":"v1","addr":"172.21.15.142:6835","nonce":1835893052}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T00:52:08.386 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.387+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc0c8017c60 msgr2=0x7fc0c801a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.386 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.387+0000 7fc0f5d45700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc0c8017c60 0x7fc0c801a0f0 crc :-1 s=READY pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.386 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.387+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 msgr2=0x7fc0f008d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T00:52:08.386 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.387+0000 7fc0f5d45700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7fc0e0010420 tx=0x7fc0e00107c0).stop 2019-10-13T00:52:08.386 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.387+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 shutdown_connections 2019-10-13T00:52:08.386 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.388+0000 7fc0f5d45700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:6800/5662,v1:172.21.15.142:6801/5662] conn(0x7fc0c8017c60 0x7fc0c801a0f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.387 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.388+0000 7fc0f5d45700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0f0198430 0x7fc0f019a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.387 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.388+0000 7fc0f5d45700 1 --2- 172.21.15.155:0/291666985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0f0088dd0 0x7fc0f008d830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T00:52:08.387 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.388+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 shutdown_connections 2019-10-13T00:52:08.387 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.388+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 wait complete. 2019-10-13T00:52:08.387 INFO:teuthology.orchestra.run.smithi155.stderr:2019-10-13T00:52:08.388+0000 7fc0f5d45700 1 -- 172.21.15.155:0/291666985 >> 172.21.15.155:0/291666985 conn(0x7fc0f006c410 msgr2=0x7fc0f0074840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T00:52:08.598 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4217}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4217': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 3, u'state': u'up:active', u'gid': 4217, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 1835893052, u'type': u'v2', u'addr': u'172.21.15.142:6834'}, {u'nonce': 1835893052, u'type': u'v1', u'addr': u'172.21.15.142:6835'}]}, u'addr': u'172.21.15.142:6835/1835893052'}}, u'fs_name': u'cephfs', u'created': u'2019-10-13T00:51:53.188987+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-10-13T00:52:00.887832+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-10-13T00:52:08.598 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-10-13T00:52:08.598 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:52:08.598 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-10-13T00:52:08.726 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2019-10-13T00:52:08.726 INFO:teuthology.orchestra.run.smithi142.stdout: "cluster_fsid": "9c6ba46f-16c8-4ed6-8640-ff20acbbecb0", 2019-10-13T00:52:08.726 INFO:teuthology.orchestra.run.smithi142.stdout: "whoami": 0, 2019-10-13T00:52:08.726 INFO:teuthology.orchestra.run.smithi142.stdout: "id": 4217, 2019-10-13T00:52:08.726 INFO:teuthology.orchestra.run.smithi142.stdout: "want_state": "up:active", 2019-10-13T00:52:08.727 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "up:active", 2019-10-13T00:52:08.727 INFO:teuthology.orchestra.run.smithi142.stdout: "rank_uptime": 8.8407044199999998, 2019-10-13T00:52:08.727 INFO:teuthology.orchestra.run.smithi142.stdout: "mdsmap_epoch": 6, 2019-10-13T00:52:08.727 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_epoch": 23, 2019-10-13T00:52:08.727 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_epoch_barrier": 23, 2019-10-13T00:52:08.727 INFO:teuthology.orchestra.run.smithi142.stdout: "uptime": 13.072064532000001 2019-10-13T00:52:08.727 INFO:teuthology.orchestra.run.smithi142.stdout:} 2019-10-13T00:52:08.731 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "9c6ba46f-16c8-4ed6-8640-ff20acbbecb0", "whoami": 0, "id": 4217, "want_state": "up:active", "state": "up:active", "rank_uptime": 8.8407044199999998, "mdsmap_epoch": 6, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "uptime": 13.072064532000001 } 2019-10-13T00:52:08.732 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2019-10-13T00:52:08.732 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2019-10-13T00:52:08.732 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi155.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2019-10-13T00:52:08.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:08.733 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:08.733 INFO:teuthology.orchestra.run.smithi155:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-10-13T00:52:08.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:08.803 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:08.804 INFO:teuthology.orchestra.run.smithi155:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-10-13T00:52:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-10-13T00:52:08.892 INFO:teuthology.orchestra.run.smithi155.stderr:mount: mount point /sys/fs/fuse/connections does not exist 2019-10-13T00:52:08.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:08.893 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:08.893 INFO:teuthology.orchestra.run.smithi155:> ls /sys/fs/fuse/connections 2019-10-13T00:52:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 2 2019-10-13T00:52:08.963 INFO:teuthology.orchestra.run.smithi155.stderr:ls: cannot access /sys/fs/fuse/connections: No such file or directory 2019-10-13T00:52:08.963 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2019-10-13T00:52:08.963 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:08.964 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0 2019-10-13T00:52:08.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:08.977 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:08.977 INFO:teuthology.orchestra.run.smithi155:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-10-13T00:52:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-10-13T00:52:09.045 INFO:teuthology.orchestra.run.smithi155.stderr:mount: mount point /sys/fs/fuse/connections does not exist 2019-10-13T00:52:09.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:09.047 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:09.047 INFO:teuthology.orchestra.run.smithi155:> ls /sys/fs/fuse/connections 2019-10-13T00:52:09.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi155.stderr:2019-10-13T00:52:09.066+0000 7fb8c46abe00 -1 init, newargv = 0x55c36b9a7040 newargc=7 2019-10-13T00:52:09.064 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi155.stderr:ceph-fuse[10942]: starting ceph client 2019-10-13T00:52:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 2 2019-10-13T00:52:09.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi155.stderr:ceph-fuse[10942]: starting fuse 2019-10-13T00:52:09.116 INFO:teuthology.orchestra.run.smithi155.stderr:ls: cannot access /sys/fs/fuse/connections: No such file or directory 2019-10-13T00:52:10.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:10.219 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.219 INFO:teuthology.orchestra.run.smithi155:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-10-13T00:52:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-10-13T00:52:10.342 INFO:teuthology.orchestra.run.smithi155.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2019-10-13T00:52:10.342 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:10.342 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.342 INFO:teuthology.orchestra.run.smithi155:> ls /sys/fs/fuse/connections 2019-10-13T00:52:10.411 INFO:teuthology.orchestra.run.smithi155.stdout:43 2019-10-13T00:52:10.412 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2019-10-13T00:52:10.412 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:10.412 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.412 INFO:teuthology.orchestra.run.smithi155:> sudo python2 -c ' 2019-10-13T00:52:10.412 INFO:teuthology.orchestra.run.smithi155:> import glob 2019-10-13T00:52:10.413 INFO:teuthology.orchestra.run.smithi155:> import re 2019-10-13T00:52:10.413 INFO:teuthology.orchestra.run.smithi155:> import os 2019-10-13T00:52:10.413 INFO:teuthology.orchestra.run.smithi155:> import subprocess 2019-10-13T00:52:10.413 INFO:teuthology.orchestra.run.smithi155:> 2019-10-13T00:52:10.413 INFO:teuthology.orchestra.run.smithi155:> def find_socket(client_name): 2019-10-13T00:52:10.413 INFO:teuthology.orchestra.run.smithi155:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2019-10-13T00:52:10.413 INFO:teuthology.orchestra.run.smithi155:> files = glob.glob(asok_path) 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> # Given a non-glob path, it better be there 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> if "*" not in asok_path: 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> assert(len(files) == 1) 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> return files[0] 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> for f in files: 2019-10-13T00:52:10.414 INFO:teuthology.orchestra.run.smithi155:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2019-10-13T00:52:10.415 INFO:teuthology.orchestra.run.smithi155:> if os.path.exists("/proc/{0}".format(pid)): 2019-10-13T00:52:10.415 INFO:teuthology.orchestra.run.smithi155:> return f 2019-10-13T00:52:10.415 INFO:teuthology.orchestra.run.smithi155:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2019-10-13T00:52:10.415 INFO:teuthology.orchestra.run.smithi155:> 2019-10-13T00:52:10.415 INFO:teuthology.orchestra.run.smithi155:> print find_socket("client.0") 2019-10-13T00:52:10.415 INFO:teuthology.orchestra.run.smithi155:> ' 2019-10-13T00:52:10.501 INFO:teuthology.orchestra.run.smithi155.stdout:/var/run/ceph/ceph-client.0.10942.asok 2019-10-13T00:52:10.501 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.10942.asok 2019-10-13T00:52:10.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:10.501 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.501 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.10942.asok status 2019-10-13T00:52:10.621 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2019-10-13T00:52:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "metadata": { 2019-10-13T00:52:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_sha1": "a8ee8ac95ca822967c9dffe2a705f1d5502812bd", 2019-10-13T00:52:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev)", 2019-10-13T00:52:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "entity_id": "0", 2019-10-13T00:52:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "hostname": "smithi155", 2019-10-13T00:52:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2019-10-13T00:52:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "pid": "10942", 2019-10-13T00:52:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "root": "/" 2019-10-13T00:52:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T00:52:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "dentry_count": 0, 2019-10-13T00:52:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "dentry_pinned_count": 0, 2019-10-13T00:52:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "id": 4361, 2019-10-13T00:52:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "inst": { 2019-10-13T00:52:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "name": { 2019-10-13T00:52:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "client", 2019-10-13T00:52:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "num": 4361 2019-10-13T00:52:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T00:52:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": { 2019-10-13T00:52:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "v1", 2019-10-13T00:52:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": "172.21.15.155:0", 2019-10-13T00:52:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "nonce": 2169511560 2019-10-13T00:52:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: } 2019-10-13T00:52:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T00:52:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": { 2019-10-13T00:52:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "v1", 2019-10-13T00:52:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": "172.21.15.155:0", 2019-10-13T00:52:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "nonce": 2169511560 2019-10-13T00:52:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T00:52:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "inst_str": "client.4361 172.21.15.155:0/2169511560", 2019-10-13T00:52:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "addr_str": "172.21.15.155:0/2169511560", 2019-10-13T00:52:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "inode_count": 1, 2019-10-13T00:52:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_epoch": 6, 2019-10-13T00:52:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_epoch": 23, 2019-10-13T00:52:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_epoch_barrier": 0, 2019-10-13T00:52:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "blacklisted": false 2019-10-13T00:52:10.627 INFO:teuthology.orchestra.run.smithi155.stdout:} 2019-10-13T00:52:10.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:10.627 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.628 INFO:teuthology.orchestra.run.smithi155:> stat --file-system '--printf=%T 2019-10-13T00:52:10.628 INFO:teuthology.orchestra.run.smithi155:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T00:52:10.697 INFO:teuthology.orchestra.run.smithi155.stdout:fuseblk 2019-10-13T00:52:10.697 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2019-10-13T00:52:10.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T00:52:10.698 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.698 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2019-10-13T00:52:10.782 INFO:teuthology.run_tasks:Running task check-counter... 2019-10-13T00:52:10.794 INFO:teuthology.run_tasks:Running task workunit... 2019-10-13T00:52:10.810 INFO:tasks.workunit:Pulling workunits from ref a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T00:52:10.811 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-10-13T00:52:10.811 INFO:tasks.workunit:timeout=3h 2019-10-13T00:52:10.811 INFO:tasks.workunit:cleanup=True 2019-10-13T00:52:10.811 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.811 INFO:teuthology.orchestra.run.smithi155:> stat -- /home/ubuntu/cephtest/mnt.0 2019-10-13T00:52:10.854 INFO:teuthology.orchestra.run.smithi155.stdout: File: ‘/home/ubuntu/cephtest/mnt.0’ 2019-10-13T00:52:10.854 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2019-10-13T00:52:10.854 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 2bh/43d Inode: 1 Links: 2 2019-10-13T00:52:10.855 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T00:52:10.855 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:fusefs_t:s0 2019-10-13T00:52:10.855 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2019-10-13T00:52:10.855 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2019-10-13 00:51:59.888887883 +0000 2019-10-13T00:52:10.855 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2019-10-13 00:52:10.769916070 +0000 2019-10-13T00:52:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2019-10-13T00:52:10.856 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2019-10-13T00:52:10.856 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.856 INFO:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2019-10-13T00:52:10.980 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:10.980 INFO:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T00:52:11.168 INFO:tasks.workunit.client.0.smithi155.stdout:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-10-13T00:52:20.647 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:52:20.647 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:52:20.651 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:20.651 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:52:50.740 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:52:50.740 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:52:50.745 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:50.745 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:52:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:Note: checking out 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'. 2019-10-13T00:52:57.690 INFO:tasks.workunit.client.0.smithi155.stderr: 2019-10-13T00:52:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-10-13T00:52:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:changes and commit them, and you can discard any commits you make in this 2019-10-13T00:52:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:state without impacting any branches by performing another checkout. 2019-10-13T00:52:57.690 INFO:tasks.workunit.client.0.smithi155.stderr: 2019-10-13T00:52:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:If you want to create a new branch to retain commits you create, you may 2019-10-13T00:52:57.691 INFO:tasks.workunit.client.0.smithi155.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-10-13T00:52:57.691 INFO:tasks.workunit.client.0.smithi155.stderr: 2019-10-13T00:52:57.691 INFO:tasks.workunit.client.0.smithi155.stderr: git checkout -b new_branch_name 2019-10-13T00:52:57.691 INFO:tasks.workunit.client.0.smithi155.stderr: 2019-10-13T00:52:57.691 INFO:tasks.workunit.client.0.smithi155.stderr:HEAD is now at a8ee8ac... Merge PR #29626 into wip-pdonnell-testing-20191012.025926 2019-10-13T00:52:57.692 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:52:57.692 INFO:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-10-13T00:52:57.744 INFO:tasks.workunit.client.0.smithi155.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2019-10-13T00:52:57.746 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-10-13T00:52:57.746 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2019-10-13T00:52:58.103 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2019-10-13T00:52:58.179 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2019-10-13T00:52:58.208 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-10-13T00:52:58.210 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-10-13T00:52:58.210 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2019-10-13T00:52:58.240 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-10-13T00:52:58.285 DEBUG:teuthology.orchestra.remote:smithi155:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2019-10-13T00:52:58.338 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2019-10-13T00:52:58.338 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2019-10-13T00:52:58.339 INFO:teuthology.orchestra.run.smithi155:Running (workunit test suites/ffsb.sh): 2019-10-13T00:52:58.339 INFO:teuthology.orchestra.run.smithi155:> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8ee8ac95ca822967c9dffe2a705f1d5502812bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2019-10-13T00:52:58.411 INFO:tasks.workunit.client.0.smithi155.stderr:--2019-10-13 00:52:58-- http://download.ceph.com/qa/ffsb.tar.bz2 2019-10-13T00:52:58.412 INFO:tasks.workunit.client.0.smithi155.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2019-10-13T00:52:58.433 INFO:tasks.workunit.client.0.smithi155.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2019-10-13T00:52:58.453 INFO:tasks.workunit.client.0.smithi155.stderr:HTTP request sent, awaiting response... 200 OK 2019-10-13T00:52:58.453 INFO:tasks.workunit.client.0.smithi155.stderr:Length: 128931 (126K) [application/octet-stream] 2019-10-13T00:52:58.455 INFO:tasks.workunit.client.0.smithi155.stderr:Saving to: ‘ffsb.tar.bz2’ 2019-10-13T00:52:58.458 INFO:tasks.workunit.client.0.smithi155.stderr: 2019-10-13T00:52:58.496 INFO:tasks.workunit.client.0.smithi155.stderr: 0K .......... .......... .......... .......... .......... 39% 1.28M 0s 2019-10-13T00:52:58.514 INFO:tasks.workunit.client.0.smithi155.stderr: 50K .......... .......... .......... .......... .......... 79% 2.47M 0s 2019-10-13T00:52:58.514 INFO:tasks.workunit.client.0.smithi155.stderr: 100K .......... .......... ..... 100% 20.6M=0.06s 2019-10-13T00:52:58.515 INFO:tasks.workunit.client.0.smithi155.stderr: 2019-10-13T00:52:58.516 INFO:tasks.workunit.client.0.smithi155.stderr:2019-10-13 00:52:58 (2.08 MB/s) - ‘ffsb.tar.bz2’ saved [128931/128931] 2019-10-13T00:52:58.517 INFO:tasks.workunit.client.0.smithi155.stderr: 2019-10-13T00:52:58.561 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ 2019-10-13T00:52:58.562 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_fs.c 2019-10-13T00:52:58.566 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/INSTALL 2019-10-13T00:52:58.571 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/fileops.c 2019-10-13T00:52:58.575 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/missing 2019-10-13T00:52:58.579 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/config.h.in 2019-10-13T00:52:58.583 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/config.guess 2019-10-13T00:52:58.589 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/fh.h 2019-10-13T00:52:58.593 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/COPYING 2019-10-13T00:52:58.597 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/util.h 2019-10-13T00:52:58.600 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/main.c 2019-10-13T00:52:58.604 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ChangeLog 2019-10-13T00:52:58.608 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/rand.c 2019-10-13T00:52:58.612 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_fs.h 2019-10-13T00:52:58.616 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/install-sh 2019-10-13T00:52:58.619 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/rwlock.h 2019-10-13T00:52:58.623 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/parser.h 2019-10-13T00:52:58.627 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/filelist.h 2019-10-13T00:52:58.631 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/LICENSE 2019-10-13T00:52:58.635 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/Makefile.in 2019-10-13T00:52:58.640 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/filelist.c 2019-10-13T00:52:58.643 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/configure 2019-10-13T00:52:58.654 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/README 2019-10-13T00:52:58.658 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/cirlist.c 2019-10-13T00:52:58.662 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/metaops.c 2019-10-13T00:52:58.666 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/registration-description 2019-10-13T00:52:58.670 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/list.c 2019-10-13T00:52:58.674 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/list.h 2019-10-13T00:52:58.677 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/rand.h 2019-10-13T00:52:58.681 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/parser.c 2019-10-13T00:52:58.685 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_tg.c 2019-10-13T00:52:58.688 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/cirlist.h 2019-10-13T00:52:58.692 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/AUTHORS 2019-10-13T00:52:58.696 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/aclocal.m4 2019-10-13T00:52:58.700 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/configure.in 2019-10-13T00:52:58.704 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/examples/ 2019-10-13T00:52:58.706 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/examples/profile_everything 2019-10-13T00:52:58.710 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/depcomp 2019-10-13T00:52:58.715 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_stats.c 2019-10-13T00:52:58.719 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/util.c 2019-10-13T00:52:58.723 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/Makefile.am 2019-10-13T00:52:58.727 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_thread.c 2019-10-13T00:52:58.730 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/rbt.c 2019-10-13T00:52:58.735 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/rbt.h 2019-10-13T00:52:58.739 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_tg.h 2019-10-13T00:52:58.743 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/stamp-h.in 2019-10-13T00:52:58.746 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/config.sub 2019-10-13T00:52:58.751 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_thread.h 2019-10-13T00:52:58.754 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/fileops.h 2019-10-13T00:52:58.758 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_op.h 2019-10-13T00:52:58.762 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/public-description 2019-10-13T00:52:58.766 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/mkinstalldirs 2019-10-13T00:52:58.769 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb.h 2019-10-13T00:52:58.773 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/metaops.h 2019-10-13T00:52:58.777 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_op.c 2019-10-13T00:52:58.781 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/fh.c 2019-10-13T00:52:58.785 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/NEWS 2019-10-13T00:52:58.788 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/rwlock.c 2019-10-13T00:52:58.792 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_stats.h 2019-10-13T00:52:58.795 INFO:tasks.workunit.client.0.smithi155.stdout:ffsb-6.0-rc2/ffsb_fc.c 2019-10-13T00:52:58.815 INFO:tasks.workunit.client.0.smithi155.stdout:patching file parser.c 2019-10-13T00:52:58.994 INFO:tasks.workunit.client.0.smithi155.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2019-10-13T00:53:00.004 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether build environment is sane... yes 2019-10-13T00:53:00.020 INFO:tasks.workunit.client.0.smithi155.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2019-10-13T00:53:00.021 INFO:tasks.workunit.client.0.smithi155.stdout:checking for gawk... gawk 2019-10-13T00:53:00.043 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether make sets $(MAKE)... yes 2019-10-13T00:53:00.087 INFO:tasks.workunit.client.0.smithi155.stdout:checking build system type... x86_64-unknown-linux-gnu 2019-10-13T00:53:00.087 INFO:tasks.workunit.client.0.smithi155.stdout:checking host system type... x86_64-unknown-linux-gnu 2019-10-13T00:53:00.108 INFO:tasks.workunit.client.0.smithi155.stdout:checking for style of include used by make... GNU 2019-10-13T00:53:00.113 INFO:tasks.workunit.client.0.smithi155.stdout:checking for gcc... gcc 2019-10-13T00:53:00.192 INFO:tasks.workunit.client.0.smithi155.stdout:checking for C compiler default output file name... a.out 2019-10-13T00:53:00.195 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether the C compiler works... yes 2019-10-13T00:53:00.199 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether we are cross compiling... no 2019-10-13T00:53:00.247 INFO:tasks.workunit.client.0.smithi155.stdout:checking for suffix of executables... 2019-10-13T00:53:00.289 INFO:tasks.workunit.client.0.smithi155.stdout:checking for suffix of object files... o 2019-10-13T00:53:00.332 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether we are using the GNU C compiler... yes 2019-10-13T00:53:00.373 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether gcc accepts -g... yes 2019-10-13T00:53:00.430 INFO:tasks.workunit.client.0.smithi155.stdout:checking for gcc option to accept ISO C89... none needed 2019-10-13T00:53:00.595 INFO:tasks.workunit.client.0.smithi155.stdout:checking dependency style of gcc... gcc3 2019-10-13T00:53:00.655 INFO:tasks.workunit.client.0.smithi155.stdout:checking how to run the C preprocessor... gcc -E 2019-10-13T00:53:00.715 INFO:tasks.workunit.client.0.smithi155.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2019-10-13T00:53:00.716 INFO:tasks.workunit.client.0.smithi155.stdout:checking for egrep... /usr/bin/grep -E 2019-10-13T00:53:00.728 INFO:tasks.workunit.client.0.smithi155.stdout:checking for AIX... no 2019-10-13T00:53:00.745 INFO:tasks.workunit.client.0.smithi155.stdout:checking for gcc... (cached) gcc 2019-10-13T00:53:00.756 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether we are using the GNU C compiler... (cached) yes 2019-10-13T00:53:00.757 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether gcc accepts -g... (cached) yes 2019-10-13T00:53:00.758 INFO:tasks.workunit.client.0.smithi155.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2019-10-13T00:53:00.759 INFO:tasks.workunit.client.0.smithi155.stdout:checking dependency style of gcc... (cached) gcc3 2019-10-13T00:53:00.825 INFO:tasks.workunit.client.0.smithi155.stdout:checking for main in -lm... yes 2019-10-13T00:53:00.896 INFO:tasks.workunit.client.0.smithi155.stdout:checking for main in -lpthread... yes 2019-10-13T00:53:01.098 INFO:tasks.workunit.client.0.smithi155.stdout:checking for ANSI C header files... yes 2019-10-13T00:53:01.152 INFO:tasks.workunit.client.0.smithi155.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2019-10-13T00:53:01.211 INFO:tasks.workunit.client.0.smithi155.stdout:checking for sys/types.h... yes 2019-10-13T00:53:01.274 INFO:tasks.workunit.client.0.smithi155.stdout:checking for sys/stat.h... yes 2019-10-13T00:53:01.338 INFO:tasks.workunit.client.0.smithi155.stdout:checking for stdlib.h... yes 2019-10-13T00:53:01.407 INFO:tasks.workunit.client.0.smithi155.stdout:checking for string.h... yes 2019-10-13T00:53:01.482 INFO:tasks.workunit.client.0.smithi155.stdout:checking for memory.h... yes 2019-10-13T00:53:01.546 INFO:tasks.workunit.client.0.smithi155.stdout:checking for strings.h... yes 2019-10-13T00:53:01.611 INFO:tasks.workunit.client.0.smithi155.stdout:checking for inttypes.h... yes 2019-10-13T00:53:01.676 INFO:tasks.workunit.client.0.smithi155.stdout:checking for stdint.h... yes 2019-10-13T00:53:01.744 INFO:tasks.workunit.client.0.smithi155.stdout:checking for unistd.h... yes 2019-10-13T00:53:01.815 INFO:tasks.workunit.client.0.smithi155.stdout:checking pthread.h usability... yes 2019-10-13T00:53:01.842 INFO:tasks.workunit.client.0.smithi155.stdout:checking pthread.h presence... yes 2019-10-13T00:53:01.843 INFO:tasks.workunit.client.0.smithi155.stdout:checking for pthread.h... yes 2019-10-13T00:53:01.914 INFO:tasks.workunit.client.0.smithi155.stdout:checking fcntl.h usability... yes 2019-10-13T00:53:01.946 INFO:tasks.workunit.client.0.smithi155.stdout:checking fcntl.h presence... yes 2019-10-13T00:53:01.947 INFO:tasks.workunit.client.0.smithi155.stdout:checking for fcntl.h... yes 2019-10-13T00:53:02.027 INFO:tasks.workunit.client.0.smithi155.stdout:checking limits.h usability... yes 2019-10-13T00:53:02.054 INFO:tasks.workunit.client.0.smithi155.stdout:checking limits.h presence... yes 2019-10-13T00:53:02.054 INFO:tasks.workunit.client.0.smithi155.stdout:checking for limits.h... yes 2019-10-13T00:53:02.076 INFO:tasks.workunit.client.0.smithi155.stdout:checking for stdint.h... (cached) yes 2019-10-13T00:53:02.141 INFO:tasks.workunit.client.0.smithi155.stdout:checking sys/time.h usability... yes 2019-10-13T00:53:02.168 INFO:tasks.workunit.client.0.smithi155.stdout:checking sys/time.h presence... yes 2019-10-13T00:53:02.169 INFO:tasks.workunit.client.0.smithi155.stdout:checking for sys/time.h... yes 2019-10-13T00:53:02.188 INFO:tasks.workunit.client.0.smithi155.stdout:checking for unistd.h... (cached) yes 2019-10-13T00:53:02.252 INFO:tasks.workunit.client.0.smithi155.stdout:checking sys/vfs.h usability... yes 2019-10-13T00:53:02.282 INFO:tasks.workunit.client.0.smithi155.stdout:checking sys/vfs.h presence... yes 2019-10-13T00:53:02.282 INFO:tasks.workunit.client.0.smithi155.stdout:checking for sys/vfs.h... yes 2019-10-13T00:53:02.393 INFO:tasks.workunit.client.0.smithi155.stdout:checking sys/limits.h usability... no 2019-10-13T00:53:02.423 INFO:tasks.workunit.client.0.smithi155.stdout:checking sys/limits.h presence... no 2019-10-13T00:53:02.423 INFO:tasks.workunit.client.0.smithi155.stdout:checking for sys/limits.h... no 2019-10-13T00:53:02.467 INFO:tasks.workunit.client.0.smithi155.stdout:checking for an ANSI C-conforming const... yes 2019-10-13T00:53:02.512 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether time.h and sys/time.h may both be included... yes 2019-10-13T00:53:02.515 INFO:tasks.workunit.client.0.smithi155.stdout:checking for function prototypes... yes 2019-10-13T00:53:02.559 INFO:tasks.workunit.client.0.smithi155.stdout:checking whether setvbuf arguments are reversed... no 2019-10-13T00:53:02.669 INFO:tasks.workunit.client.0.smithi155.stdout:checking for system... yes 2019-10-13T00:53:02.754 INFO:tasks.workunit.client.0.smithi155.stdout:checking for gettimeofday... yes 2019-10-13T00:53:02.839 INFO:tasks.workunit.client.0.smithi155.stdout:checking for mkdir... yes 2019-10-13T00:53:02.928 INFO:tasks.workunit.client.0.smithi155.stdout:checking for strerror... yes 2019-10-13T00:53:03.027 INFO:tasks.workunit.client.0.smithi155.stdout:checking for open64... yes 2019-10-13T00:53:03.117 INFO:tasks.workunit.client.0.smithi155.stdout:checking for stat64... yes 2019-10-13T00:53:03.207 INFO:tasks.workunit.client.0.smithi155.stdout:checking for fseeko64... yes 2019-10-13T00:53:03.294 INFO:tasks.workunit.client.0.smithi155.stdout:checking for lrand48_r... yes 2019-10-13T00:53:03.382 INFO:tasks.workunit.client.0.smithi155.stdout:checking for srand48_r... yes 2019-10-13T00:53:03.414 INFO:tasks.workunit.client.0.smithi155.stdout:configure: creating ./config.status 2019-10-13T00:53:03.573 INFO:tasks.workunit.client.0.smithi155.stdout:config.status: creating Makefile 2019-10-13T00:53:03.591 INFO:tasks.workunit.client.0.smithi155.stdout:config.status: creating config.h 2019-10-13T00:53:03.619 INFO:tasks.workunit.client.0.smithi155.stdout:config.status: executing depfiles commands 2019-10-13T00:53:03.704 INFO:tasks.workunit.client.0.smithi155.stdout:config.status: executing default commands 2019-10-13T00:53:03.800 INFO:tasks.workunit.client.0.smithi155.stdout:make all-am 2019-10-13T00:53:03.825 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2019-10-13T00:53:03.825 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2019-10-13T00:53:03.962 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2019-10-13T00:53:03.968 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2019-10-13T00:53:04.042 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2019-10-13T00:53:04.047 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2019-10-13T00:53:04.138 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/main.Tpo .deps/main.Po 2019-10-13T00:53:04.144 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2019-10-13T00:53:04.244 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2019-10-13T00:53:04.250 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2019-10-13T00:53:04.365 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2019-10-13T00:53:04.371 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2019-10-13T00:53:04.436 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2019-10-13T00:53:04.441 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2019-10-13T00:53:04.487 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2019-10-13T00:53:04.493 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2019-10-13T00:53:04.549 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2019-10-13T00:53:04.555 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2019-10-13T00:53:04.684 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2019-10-13T00:53:04.690 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2019-10-13T00:53:04.808 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2019-10-13T00:53:04.814 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2019-10-13T00:53:04.856 INFO:tasks.workunit.client.0.smithi155.stderr:ffsb_fs.c: In function ‘fs_needs_stats’: 2019-10-13T00:53:04.857 INFO:tasks.workunit.client.0.smithi155.stderr:ffsb_fs.c:627:24: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:04.857 INFO:tasks.workunit.client.0.smithi155.stderr: return (fs != NULL) ? (int)fs->fsd.config : 0; 2019-10-13T00:53:04.857 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:04.965 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2019-10-13T00:53:04.973 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2019-10-13T00:53:05.049 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2019-10-13T00:53:05.055 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2019-10-13T00:53:05.139 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2019-10-13T00:53:05.145 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2019-10-13T00:53:05.249 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/util.Tpo .deps/util.Po 2019-10-13T00:53:05.254 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2019-10-13T00:53:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:In file included from parser.h:22:0, 2019-10-13T00:53:05.299 INFO:tasks.workunit.client.0.smithi155.stderr: from parser.c:25: 2019-10-13T00:53:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c: In function ‘init_filesys’: 2019-10-13T00:53:05.302 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.302 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.302 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.302 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.302 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.302 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.303 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.303 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.303 INFO:tasks.workunit.client.0.smithi155.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T00:53:05.303 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:934:3: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.304 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2019-10-13T00:53:05.304 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.304 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.304 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.305 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.305 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.305 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.305 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.305 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.305 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.306 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.306 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.306 INFO:tasks.workunit.client.0.smithi155.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T00:53:05.306 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.306 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:934:3: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.306 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2019-10-13T00:53:05.306 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.307 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.307 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.307 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.308 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.308 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.308 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.308 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.308 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.308 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.308 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.309 INFO:tasks.workunit.client.0.smithi155.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T00:53:05.309 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.309 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:941:3: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.309 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2019-10-13T00:53:05.309 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.310 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.310 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.310 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.310 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.311 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.311 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.311 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.311 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.311 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.311 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.311 INFO:tasks.workunit.client.0.smithi155.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T00:53:05.312 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.312 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:941:3: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.312 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2019-10-13T00:53:05.312 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.312 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c: In function ‘init_tg_stats’: 2019-10-13T00:53:05.313 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.313 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.313 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.313 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.313 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.314 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.314 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.314 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.314 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.314 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.314 INFO:tasks.workunit.client.0.smithi155.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T00:53:05.314 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.315 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:971:6: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.315 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, 2019-10-13T00:53:05.315 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.316 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.316 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.316 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.316 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.316 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.316 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:971:6: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.317 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, 2019-10-13T00:53:05.318 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.318 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.319 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.319 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.319 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.319 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.319 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.319 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.319 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.320 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.320 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.320 INFO:tasks.workunit.client.0.smithi155.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T00:53:05.320 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.320 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:980:6: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.320 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, 2019-10-13T00:53:05.320 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.321 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T00:53:05.321 INFO:tasks.workunit.client.0.smithi155.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T00:53:05.321 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.321 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2019-10-13T00:53:05.322 INFO:tasks.workunit.client.0.smithi155.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T00:53:05.322 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.322 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2019-10-13T00:53:05.322 INFO:tasks.workunit.client.0.smithi155.stderr: container_of(ptr, type, member) 2019-10-13T00:53:05.322 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.322 INFO:tasks.workunit.client.0.smithi155.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2019-10-13T00:53:05.323 INFO:tasks.workunit.client.0.smithi155.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T00:53:05.323 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.323 INFO:tasks.workunit.client.0.smithi155.stderr:parser.c:980:6: note: in expansion of macro ‘list_for_each_entry’ 2019-10-13T00:53:05.323 INFO:tasks.workunit.client.0.smithi155.stderr: list_for_each_entry(tmp_list, 2019-10-13T00:53:05.323 INFO:tasks.workunit.client.0.smithi155.stderr: ^ 2019-10-13T00:53:05.515 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2019-10-13T00:53:05.521 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2019-10-13T00:53:05.595 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2019-10-13T00:53:05.600 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2019-10-13T00:53:05.718 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2019-10-13T00:53:05.723 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2019-10-13T00:53:05.757 INFO:tasks.workunit.client.0.smithi155.stdout:mv -f .deps/list.Tpo .deps/list.Po 2019-10-13T00:53:05.769 INFO:tasks.workunit.client.0.smithi155.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2019-10-13T00:53:05.864 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2019-10-13T00:53:05.881 INFO:tasks.workunit.client.0.smithi155.stdout:FFSB version 6.0-RC2 started 2019-10-13T00:53:05.882 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.885 INFO:tasks.workunit.client.0.smithi155.stdout:benchmark time = 300 2019-10-13T00:53:05.886 INFO:tasks.workunit.client.0.smithi155.stdout:ThreadGroup 0 2019-10-13T00:53:05.886 INFO:tasks.workunit.client.0.smithi155.stdout:================ 2019-10-13T00:53:05.886 INFO:tasks.workunit.client.0.smithi155.stdout: num_threads = 32 2019-10-13T00:53:05.886 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.886 INFO:tasks.workunit.client.0.smithi155.stdout: read_random = off 2019-10-13T00:53:05.886 INFO:tasks.workunit.client.0.smithi155.stdout: read_size = 0 (0B) 2019-10-13T00:53:05.886 INFO:tasks.workunit.client.0.smithi155.stdout: read_blocksize = 0 (0B) 2019-10-13T00:53:05.887 INFO:tasks.workunit.client.0.smithi155.stdout: read_skip = off 2019-10-13T00:53:05.887 INFO:tasks.workunit.client.0.smithi155.stdout: read_skipsize = 0 (0B) 2019-10-13T00:53:05.887 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.887 INFO:tasks.workunit.client.0.smithi155.stdout: write_random = on 2019-10-13T00:53:05.887 INFO:tasks.workunit.client.0.smithi155.stdout: write_size = 5242880 (5MB) 2019-10-13T00:53:05.887 INFO:tasks.workunit.client.0.smithi155.stdout: fsync_file = 0 2019-10-13T00:53:05.887 INFO:tasks.workunit.client.0.smithi155.stdout: write_blocksize = 4096 (4KB) 2019-10-13T00:53:05.888 INFO:tasks.workunit.client.0.smithi155.stdout: wait time = 0 2019-10-13T00:53:05.888 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.888 INFO:tasks.workunit.client.0.smithi155.stdout: op weights 2019-10-13T00:53:05.888 INFO:tasks.workunit.client.0.smithi155.stdout: read = 0 (0.00%) 2019-10-13T00:53:05.888 INFO:tasks.workunit.client.0.smithi155.stdout: readall = 0 (0.00%) 2019-10-13T00:53:05.888 INFO:tasks.workunit.client.0.smithi155.stdout: write = 1 (100.00%) 2019-10-13T00:53:05.888 INFO:tasks.workunit.client.0.smithi155.stdout: create = 0 (0.00%) 2019-10-13T00:53:05.889 INFO:tasks.workunit.client.0.smithi155.stdout: append = 0 (0.00%) 2019-10-13T00:53:05.889 INFO:tasks.workunit.client.0.smithi155.stdout: delete = 0 (0.00%) 2019-10-13T00:53:05.889 INFO:tasks.workunit.client.0.smithi155.stdout: metaop = 0 (0.00%) 2019-10-13T00:53:05.889 INFO:tasks.workunit.client.0.smithi155.stdout: createdir = 0 (0.00%) 2019-10-13T00:53:05.889 INFO:tasks.workunit.client.0.smithi155.stdout: stat = 0 (0.00%) 2019-10-13T00:53:05.889 INFO:tasks.workunit.client.0.smithi155.stdout: writeall = 0 (0.00%) 2019-10-13T00:53:05.889 INFO:tasks.workunit.client.0.smithi155.stdout: writeall_fsync = 0 (0.00%) 2019-10-13T00:53:05.890 INFO:tasks.workunit.client.0.smithi155.stdout: open_close = 0 (0.00%) 2019-10-13T00:53:05.890 INFO:tasks.workunit.client.0.smithi155.stdout: write_fsync = 0 (0.00%) 2019-10-13T00:53:05.890 INFO:tasks.workunit.client.0.smithi155.stdout: create_fsync = 0 (0.00%) 2019-10-13T00:53:05.890 INFO:tasks.workunit.client.0.smithi155.stdout: append_fsync = 0 (0.00%) 2019-10-13T00:53:05.890 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.890 INFO:tasks.workunit.client.0.smithi155.stdout:FileSystem . 2019-10-13T00:53:05.890 INFO:tasks.workunit.client.0.smithi155.stdout:========== 2019-10-13T00:53:05.891 INFO:tasks.workunit.client.0.smithi155.stdout: num_dirs = 0 2019-10-13T00:53:05.891 INFO:tasks.workunit.client.0.smithi155.stdout: starting files = 128 2019-10-13T00:53:05.891 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.891 INFO:tasks.workunit.client.0.smithi155.stdout: min file size = 104857600 (100MB) 2019-10-13T00:53:05.891 INFO:tasks.workunit.client.0.smithi155.stdout: max file size = 104857600 (100MB) 2019-10-13T00:53:05.891 INFO:tasks.workunit.client.0.smithi155.stdout: directio = off 2019-10-13T00:53:05.891 INFO:tasks.workunit.client.0.smithi155.stdout: alignedio = on 2019-10-13T00:53:05.892 INFO:tasks.workunit.client.0.smithi155.stdout: bufferedio = off 2019-10-13T00:53:05.892 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.892 INFO:tasks.workunit.client.0.smithi155.stdout: aging is off 2019-10-13T00:53:05.892 INFO:tasks.workunit.client.0.smithi155.stdout: current utilization = 0.00% 2019-10-13T00:53:05.892 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T00:53:05.892 INFO:tasks.workunit.client.0.smithi155.stdout:checking existing fs: . 2019-10-13T00:53:05.893 INFO:tasks.workunit.client.0.smithi155.stdout:recreating new fileset 2019-10-13T00:53:05.893 INFO:tasks.workunit.client.0.smithi155.stderr:opendir: No such file or directory 2019-10-13T00:53:21.044 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:53:21.044 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:53:21.049 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:53:21.049 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:53:51.141 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:53:51.141 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:53:51.145 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:53:51.145 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:54:21.231 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:54:21.231 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:54:21.235 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:54:21.235 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:54:51.328 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:54:51.328 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:54:51.332 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:54:51.332 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:55:21.417 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:55:21.417 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:55:21.421 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:55:21.421 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:55:51.519 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:55:51.519 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:55:51.527 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:55:51.528 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:56:21.619 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:56:21.619 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:56:21.625 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:56:21.625 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:56:51.711 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:56:51.712 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:56:51.716 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:56:51.716 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:57:21.909 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:57:21.910 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:57:21.916 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:57:21.916 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:57:52.020 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:57:52.020 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:57:52.024 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:57:52.025 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:58:23.025 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:58:23.026 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:58:23.029 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:58:23.030 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:58:56.472 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:58:56.472 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:58:56.476 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:58:56.476 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:59:29.710 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T00:59:29.710 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T00:59:29.714 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T00:59:29.714 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:00:03.324 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:00:03.324 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:00:03.329 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:00:03.330 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:00:36.738 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:00:36.738 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:00:36.742 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:00:36.742 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:01:08.539 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:01:08.539 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:01:08.543 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:01:08.543 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:01:25.719 INFO:tasks.workunit.client.0.smithi155.stdout:fs setup took 499 secs 2019-10-13T01:01:27.542 INFO:tasks.workunit.client.0.smithi155.stdout:Syncing()...1 sec 2019-10-13T01:01:27.542 INFO:tasks.workunit.client.0.smithi155.stdout:Starting Actual Benchmark At: Sun Oct 13 01:01:27 2019 2019-10-13T01:01:27.543 INFO:tasks.workunit.client.0.smithi155.stdout: 2019-10-13T01:01:28.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:01:28.105+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=1041, to_read={3:d4b0c92a:::1000000016e.00000005:head=319488,4096,0,937984,4096,0,1462272,4096,0,2080768,4096,0}, subchunks={3:d4b0c92a:::1000000016e.00000005:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:01:28.682 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:01:28.689+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=1820, attrs_read=0)) 2019-10-13T01:01:31.303 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:01:31.312+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=1439, attrs_read=0)) 2019-10-13T01:01:38.643 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:01:38.644 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:01:38.648 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:01:38.648 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:02:09.219 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:02:09.219 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:02:09.225 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:02:09.225 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:02:31.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:02:31.740+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=9118, to_read={3:bf34e975:::10000000159.00000008:head=720896,4096,0,757760,4096,0,868352,4096,0,978944,4096,0}, subchunks={3:bf34e975:::10000000159.00000008:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:35.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:02:35.614+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 5 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=9675, attrs_read=0)) 2019-10-13T01:02:35.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:02:35.823+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=9546, attrs_read=0)) 2019-10-13T01:02:36.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:02:36.568+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=9849, to_read={3:c08f0f9e:::10000000197.00000007:head=634880,4096,0,667648,4096,0,1155072,4096,0}, subchunks={3:c08f0f9e:::10000000197.00000007:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:40.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:02:40.880+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=10186, attrs_read=0)) 2019-10-13T01:02:41.973 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:02:41.973 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:02:41.977 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:02:41.977 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:02:44.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:02:44.831+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=10689, to_read={3:ae7f1c39:::10000000181.00000001:head=991232,4096,0,1159168,4096,0,1507328,4096,0,1720320,4096,0}, subchunks={3:ae7f1c39:::10000000181.00000001:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:45.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:02:45.806+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=10798, to_read={3:b7d3ae62:::100000001c3.00000015:head=1245184,4096,0,1400832,4096,0}, subchunks={3:b7d3ae62:::100000001c3.00000015:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:51.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:02:51.758+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=11563, to_read={3:bea1de3b:::10000000163.00000000:head=106496,4096,0,122880,4096,0,536576,4096,0,544768,4096,0,1880064,4096,0,2064384,4096,0}, subchunks={3:bea1de3b:::10000000163.00000000:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:52.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:02:52.618+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=11900, to_read={3:c0cf53ce:::100000001b2.00000008:head=495616,4096,0,1327104,4096,0,1470464,4096,0}, subchunks={3:c0cf53ce:::100000001b2.00000008:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:54.271 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:02:54.281+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=23940, to_read={3:3416e1ca:::100000001a9.00000000:head=413696,4096,0,626688,4096,0,659456,4096,0,1015808,4096,0}, subchunks={3:3416e1ca:::100000001a9.00000000:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:54.876 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:02:54.884+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=13130, attrs_read=0)) 2019-10-13T01:02:55.305 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:02:55.316+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=24111, attrs_read=0)) 2019-10-13T01:02:56.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:02:56.639+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=12364, to_read={3:dffb15ff:::10000000166.00000004:head=499712,4096,0,753664,4096,0,1011712,8192,0,1392640,4096,0,1576960,4096,0,1626112,4096,0,1654784,4096,0,1945600,4096,0}, subchunks={3:dffb15ff:::10000000166.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:02:58.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:02:58.784+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=12428, attrs_read=0)) 2019-10-13T01:02:59.280 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:02:59.291+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=24972, attrs_read=0)) 2019-10-13T01:03:00.810 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:00.820+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=13799, attrs_read=0)) 2019-10-13T01:03:02.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:02.804+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=12894, to_read={3:a5c63c55:::100000001c5.00000006:head=933888,4096,0,1478656,4096,0,2002944,4096,0}, subchunks={3:a5c63c55:::100000001c5.00000006:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:04.770 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:04.775+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=14315, attrs_read=0)) 2019-10-13T01:03:05.243 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:05.252+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=26143, to_read={3:85a3fcf4:::100000001ce.0000000d:head=659456,4096,0,1617920,4096,0,1806336,4096,0,2031616,4096,0}, subchunks={3:85a3fcf4:::100000001ce.0000000d:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:06.237 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:06.246+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=26367, to_read={3:223fa5d4:::100000001c0.00000001:head=1429504,4096,0,1716224,4096,0,1748992,4096,0}, subchunks={3:223fa5d4:::100000001c0.00000001:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:06.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:06.792+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=13414, to_read={3:ae58533e:::100000001c0.00000012:head=512000,4096,0,1323008,4096,0}, subchunks={3:ae58533e:::100000001c0.00000012:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:07.256 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:07.267+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=26592, attrs_read=0)) 2019-10-13T01:03:08.302 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:08.310+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=26855, to_read={3:9766623a:::100000001bd.00000011:head=516096,4096,0,663552,4096,0,987136,4096,0,1261568,4096,0,1716224,4096,0,2060288,4096,0}, subchunks={3:9766623a:::100000001bd.00000011:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:09.342 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:09.353+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=27079, to_read={3:3a3a86d1:::1000000018e.00000017:head=1081344,8192,0,1638400,4096,0}, subchunks={3:3a3a86d1:::1000000018e.00000017:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:11.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:11.751+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=14007, to_read={3:a6503063:::100000001b2.00000009:head=0,4096,0,385024,4096,0,487424,4096,0}, subchunks={3:a6503063:::100000001b2.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:15.402 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:15.414+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=28180, to_read={3:3820fe3a:::100000001a5.00000003:head=286720,4096,0,811008,4096,0,856064,4096,0}, subchunks={3:3820fe3a:::100000001a5.00000003:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:15.963 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:15.974+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=15518, to_read={3:0a5df730:::10000000178.00000006:head=1748992,4096,0,1818624,4096,0,1953792,4096,0,2068480,4096,0}, subchunks={3:0a5df730:::10000000178.00000006:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:16.367 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:16.378+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=28425, attrs_read=0)) 2019-10-13T01:03:17.836 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:03:17.836 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:03:17.850 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:03:17.851 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:03:18.320 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:18.331+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=28765, to_read={3:2f0bb507:::100000001a5.0000000c:head=962560,4096,0,1081344,4096,0,1236992,4096,0,1380352,4096,0}, subchunks={3:2f0bb507:::100000001a5.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:18.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:03:18.696+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=14733, to_read={3:dec09cb6:::1000000016c.0000000c:head=1642496,4096,0,1875968,4096,0}, subchunks={3:dec09cb6:::1000000016c.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:20.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:20.821+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=14966, to_read={3:a108c019:::1000000017f.00000003:head=192512,4096,0,729088,4096,0,913408,4096,0,929792,4096,0}, subchunks={3:a108c019:::1000000017f.00000003:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:21.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:03:21.696+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=15097, to_read={3:d871cbf2:::100000001bb.00000004:head=344064,4096,0,421888,4096,0,884736,4096,0,1691648,4096,0}, subchunks={3:d871cbf2:::100000001bb.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:22.350 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:22.361+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=29621, to_read={3:98a8a8b3:::100000001c0.00000000:head=49152,4096,0,217088,4096,0,262144,4096,0}, subchunks={3:98a8a8b3:::100000001c0.00000000:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:22.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:22.854+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=15227, to_read={3:a32408b4:::10000000166.0000000c:head=1396736,4096,0,1961984,4096,0}, subchunks={3:a32408b4:::10000000166.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:26.911 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:26.922+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=16649, attrs_read=0)) 2019-10-13T01:03:27.450 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:27.461+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=30667, to_read={3:987959db:::1000000019c.00000011:head=1695744,4096,0,1851392,4096,0}, subchunks={3:987959db:::1000000019c.00000011:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:28.501 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:28.503+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=30920, attrs_read=0)) 2019-10-13T01:03:29.003 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:29.005+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=16874, to_read={3:05823a84:::100000001d4.0000000d:head=1687552,4096,0,1896448,4096,0}, subchunks={3:05823a84:::100000001d4.0000000d:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:29.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:03:29.660+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=16037, attrs_read=0)) 2019-10-13T01:03:29.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:29.932+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=16072, attrs_read=0)) 2019-10-13T01:03:30.446 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:30.456+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=31367, attrs_read=0)) 2019-10-13T01:03:32.373 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:32.385+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=31842, attrs_read=0)) 2019-10-13T01:03:32.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:33.000+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=16496, attrs_read=0)) 2019-10-13T01:03:33.338 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:33.342+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=32042, attrs_read=0)) 2019-10-13T01:03:34.042 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:34.053+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=17433, to_read={3:0afb720c:::1000000018a.00000012:head=557056,4096,0,593920,4096,0,684032,4096,0,1236992,4096,0}, subchunks={3:0afb720c:::1000000018a.00000012:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:34.364 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:34.376+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=32196, to_read={3:9eecface:::100000001d4.0000000f:head=667648,4096,0,917504,4096,0}, subchunks={3:9eecface:::100000001d4.0000000f:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:35.375 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:35.386+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=32435, to_read={3:2bdb434f:::100000001c2.0000000f:head=778240,4096,0,831488,4096,0,839680,4096,0}, subchunks={3:2bdb434f:::100000001c2.0000000f:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:36.100 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:36.109+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=17679, to_read={3:1af564b4:::10000000180.00000009:head=454656,4096,0,1134592,4096,0,1261568,4096,0}, subchunks={3:1af564b4:::10000000180.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:37.084 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:37.095+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=17795, to_read={3:1dbcdec2:::100000001ce.00000005:head=221184,4096,0,344064,4096,0,495616,4096,0,1159168,4096,0,1212416,4096,0,1392640,4096,0,1413120,4096,0,2039808,4096,0}, subchunks={3:1dbcdec2:::100000001ce.00000005:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:38.354 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:38.365+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=33027, to_read={3:8d170fb7:::10000000189.00000009:head=1458176,4096,0,1744896,4096,0}, subchunks={3:8d170fb7:::10000000189.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:40.156 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:40.163+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=18046, to_read={3:0d7577c3:::100000001d4.00000008:head=147456,4096,0,241664,4096,0,364544,4096,0}, subchunks={3:0d7577c3:::100000001d4.00000008:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:40.361 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:40.363+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=33438, attrs_read=0)) 2019-10-13T01:03:40.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:03:40.524+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=17263, to_read={3:d5d84b29:::10000000171.00000010:head=163840,4096,0,319488,4096,0,663552,4096,0,724992,4096,0,741376,4096,0,778240,4096,0,1470464,4096,0,1949696,4096,0}, subchunks={3:d5d84b29:::10000000171.00000010:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:41.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:41.346+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=17352, attrs_read=0)) 2019-10-13T01:03:41.337 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:41.349+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=33659, to_read={3:92c3b581:::1000000019c.00000001:head=372736,4096,0,446464,4096,0,573440,4096,0,1781760,4096,0,1794048,4096,0,1830912,4096,0,2052096,4096,0}, subchunks={3:92c3b581:::1000000019c.00000001:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:42.326 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:42.336+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=33903, attrs_read=0)) 2019-10-13T01:03:43.362 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:43.373+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 7 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=34175, to_read={3:26a98f59:::10000000159.00000017:head=163840,4096,0,749568,4096,0,761856,4096,0,925696,4096,0,1175552,4096,0,1744896,4096,0,1974272,4096,0}, subchunks={3:26a98f59:::10000000159.00000017:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:44.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:44.326+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=17749, to_read={3:a5c63c55:::100000001c5.00000006:head=1470464,4096,0,1622016,4096,0,2056192,4096,0}, subchunks={3:a5c63c55:::100000001c5.00000006:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:44.317 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:44.329+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=34419, attrs_read=0)) 2019-10-13T01:03:44.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:03:44.668+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=17803, attrs_read=0)) 2019-10-13T01:03:45.313 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:45.324+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=34679, to_read={3:2f62025a:::1000000018a.0000000b:head=405504,4096,0,557056,4096,0,1941504,4096,0}, subchunks={3:2f62025a:::1000000018a.0000000b:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:50.473 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:50.484+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=19224, attrs_read=0)) 2019-10-13T01:03:52.158 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:03:52.158 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:03:52.164 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:03:52.164 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:03:52.308 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:52.320+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=36325, attrs_read=0)) 2019-10-13T01:03:52.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:03:52.785+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=18850, to_read={3:bdf20ca7:::1000000016b.00000018:head=1032192,4096,0,2007040,4096,0}, subchunks={3:bdf20ca7:::1000000016b.00000018:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:53.298 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:53.302+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=36589, to_read={3:932e2413:::10000000180.0000000c:head=1458176,4096,0,1515520,4096,0,1761280,4096,0,1871872,4096,0}, subchunks={3:932e2413:::10000000180.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:54.263 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:54.273+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=36831, attrs_read=0)) 2019-10-13T01:03:55.279 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:55.290+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=36991, to_read={3:362a1e90:::1000000019b.0000000e:head=651264,4096,0,1101824,4096,0,1671168,4096,0}, subchunks={3:362a1e90:::1000000019b.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:03:57.289 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:03:57.301+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=37431, attrs_read=0)) 2019-10-13T01:03:57.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:03:57.704+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=19302, attrs_read=0)) 2019-10-13T01:03:57.767 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:03:57.778+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=20062, attrs_read=0)) 2019-10-13T01:04:00.311 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:00.323+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=38258, attrs_read=0)) 2019-10-13T01:04:00.365 INFO:tasks.ceph.mon.b.smithi142.stderr:2019-10-13T01:04:00.371+0000 7f869d63e700 -1 log_channel(cluster) log [ERR] : Health check failed: mons b,c are very low on available space (MON_DISK_CRIT) 2019-10-13T01:04:00.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:04:00.656+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=19664, to_read={3:cf1657a2:::10000000180.00000002:head=1392640,4096,0,1478656,4096,0,1519616,4096,0,1765376,4096,0,2043904,4096,0}, subchunks={3:cf1657a2:::10000000180.00000002:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:01.813 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:01.822+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=20596, attrs_read=0)) 2019-10-13T01:04:02.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:04:02.579+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=19865, to_read={3:ce5316c2:::10000000180.00000018:head=40960,4096,0}, subchunks={3:ce5316c2:::10000000180.00000018:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:06.322 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:06.334+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.1s0 23/21 ECSubRead(tid=39642, to_read={3:9940a6f5:::100000001d3.0000000c:head=614400,4096,0,655360,4096,0,688128,4096,0,958464,4096,0,1404928,4096,0,1724416,4096,0,2056192,4096,0}, subchunks={3:9940a6f5:::100000001d3.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:06.682 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:06.694+0000 7f5199ddc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 9295) UID: 0 2019-10-13T01:04:06.682 INFO:tasks.ceph.osd.7.smithi142.stderr:2019-10-13T01:04:06.694+0000 7ff21ea10700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 9295) UID: 0 2019-10-13T01:04:06.682 INFO:tasks.ceph.osd.4.smithi142.stderr:2019-10-13T01:04:06.693+0000 7fc0c7768700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 9295) UID: 0 2019-10-13T01:04:06.683 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:06.694+0000 7efcd642c700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 9295) UID: 0 2019-10-13T01:04:07.315 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:07.327+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=39872, to_read={3:310cfbb9:::100000001c7.00000004:head=147456,4096,0,172032,4096,0,798720,4096,0}, subchunks={3:310cfbb9:::100000001c7.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:10.346 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:10.359+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=40518, attrs_read=0)) 2019-10-13T01:04:10.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:04:10.978+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=21104, to_read={3:a1fcecf5:::1000000016f.00000004:head=1409024,4096,0,1830912,4096,0}, subchunks={3:a1fcecf5:::1000000016f.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:11.313 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:11.325+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=40741, to_read={3:3e06200a:::100000001c4.0000000b:head=438272,4096,0,1142784,4096,0,1183744,4096,0}, subchunks={3:3e06200a:::100000001c4.0000000b:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:12.280 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:12.292+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=40862, attrs_read=0)) 2019-10-13T01:04:12.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:04:12.608+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=20673, to_read={3:c3f946cf:::10000000159.00000014:head=258048,8192,0,909312,4096,0}, subchunks={3:c3f946cf:::10000000159.00000014:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:12.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:04:12.962+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=21262, attrs_read=0)) 2019-10-13T01:04:13.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:04:13.573+0000 7fdde2779700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=20796, to_read={3:df9dd252:::10000000162.0000000d:head=212992,4096,0,262144,4096,0}, subchunks={3:df9dd252:::10000000162.0000000d:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:16.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:04:16.972+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=21734, to_read={3:ac73c8bc:::100000001c4.0000000c:head=1261568,4096,0,1556480,4096,0,1634304,4096,0,1691648,4096,0}, subchunks={3:ac73c8bc:::100000001c4.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:17.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:04:17.937+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=21849, to_read={3:a8e21f97:::10000000162.00000004:head=659456,4096,0,794624,4096,0,835584,4096,0,1605632,4096,0,1732608,4096,0}, subchunks={3:a8e21f97:::10000000162.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:18.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:04:18.937+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=21977, attrs_read=0)) 2019-10-13T01:04:19.677 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:19.689+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=22537, to_read={3:0c4aa90b:::10000000175.0000000a:head=143360,4096,0,307200,4096,0,344064,4096,0,843776,4096,0,1511424,4096,0,1785856,4096,0,2076672,4096,0}, subchunks={3:0c4aa90b:::10000000175.0000000a:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:19.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:04:19.945+0000 7f5e9787c700 -1 osd.2 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=22100, to_read={3:b3a9ba91:::10000000162.00000008:head=995328,4096,0,1196032,4096,0,1388544,4096,0,1400832,4096,0}, subchunks={3:b3a9ba91:::10000000162.00000008:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:21.386 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:21.397+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=42831, attrs_read=0)) 2019-10-13T01:04:23.680 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:23.689+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=22954, attrs_read=0)) 2019-10-13T01:04:29.565 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:29.578+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=44578, to_read={3:32d26f31:::10000000168.00000010:head=372736,4096,0,450560,4096,0,770048,4096,0,1171456,4096,0,1449984,4096,0}, subchunks={3:32d26f31:::10000000168.00000010:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:30.579 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:30.589+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=44770, to_read={3:2f437b88:::10000000177.00000018:head=630784,4096,0,1372160,4096,0,1810432,4096,0}, subchunks={3:2f437b88:::10000000177.00000018:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:30.733 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:30.745+0000 7efcd4399700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=23621, attrs_read=0)) 2019-10-13T01:04:35.833 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:35.843+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=45756, attrs_read=0)) 2019-10-13T01:04:36.613 INFO:tasks.ceph.osd.7.smithi142.stderr:2019-10-13T01:04:36.626+0000 7ff21a178700 -1 bdev(0x55b6b9f8e000 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:36.615 INFO:tasks.ceph.osd.4.smithi142.stderr:2019-10-13T01:04:36.627+0000 7fc0c2ed0700 -1 bdev(0x55879dfe2000 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:36.615 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:36.628+0000 7f5195544700 -1 bdev(0x55c5bed46000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:36.879 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:36.892+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=45909, to_read={3:29b66dc8:::100000001d4.0000000e:head=368640,4096,0,1339392,4096,0}, subchunks={3:29b66dc8:::100000001d4.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:37.201 INFO:tasks.ceph.osd.4.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc0c2ed0700 time 2019-10-13T01:04:36.788010+0000 2019-10-13T01:04:37.201 INFO:tasks.ceph.osd.4.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.201 INFO:tasks.ceph.osd.6.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5195544700 time 2019-10-13T01:04:36.787979+0000 2019-10-13T01:04:37.201 INFO:tasks.ceph.osd.6.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.202 INFO:tasks.ceph.osd.7.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff21a178700 time 2019-10-13T01:04:36.788019+0000 2019-10-13T01:04:37.202 INFO:tasks.ceph.osd.7.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.525 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.526 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x558791aabc7e] 2019-10-13T01:04:37.526 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.526 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.526 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.526 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr:*** Caught signal (Aborted) ** 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr: in thread 7fc0c2ed0700 thread_name:bstore_aio 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr:2019-10-13T01:04:37.538+0000 7fc0c2ed0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc0c2ed0700 time 2019-10-13T01:04:36.788010+0000 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x558791aabc7e] 2019-10-13T01:04:37.527 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.528 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.528 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.528 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.528 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.528 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.528 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55b6ad5eac7e] 2019-10-13T01:04:37.529 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.529 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.529 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.529 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.529 INFO:tasks.ceph.osd.7.smithi142.stderr:2019-10-13T01:04:37.538+0000 7ff21a178700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff21a178700 time 2019-10-13T01:04:36.788019+0000 2019-10-13T01:04:37.529 INFO:tasks.ceph.osd.7.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.529 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.530 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.530 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55b6ad5eac7e] 2019-10-13T01:04:37.530 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.530 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.530 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.530 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.530 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.531 INFO:tasks.ceph.osd.7.smithi142.stderr:*** Caught signal (Aborted) ** 2019-10-13T01:04:37.531 INFO:tasks.ceph.osd.7.smithi142.stderr: in thread 7ff21a178700 thread_name:bstore_aio 2019-10-13T01:04:37.531 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.531 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55c5b28a6c7e] 2019-10-13T01:04:37.531 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.532 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.532 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.532 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.532 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:37.538+0000 7f5195544700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5195544700 time 2019-10-13T01:04:36.787979+0000 2019-10-13T01:04:37.532 INFO:tasks.ceph.osd.6.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.532 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.532 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55c5b28a6c7e] 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr:*** Caught signal (Aborted) ** 2019-10-13T01:04:37.533 INFO:tasks.ceph.osd.6.smithi142.stderr: in thread 7f5195544700 thread_name:bstore_aio 2019-10-13T01:04:37.535 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.535 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (()+0xf630) [0x7fc0cdf22630] 2019-10-13T01:04:37.535 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (gsignal()+0x37) [0x7fc0ccd16377] 2019-10-13T01:04:37.535 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (abort()+0x148) [0x7fc0ccd17a68] 2019-10-13T01:04:37.535 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x558791aabd46] 2019-10-13T01:04:37.535 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.535 INFO:tasks.ceph.osd.4.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.536 INFO:tasks.ceph.osd.4.smithi142.stderr: 7: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.536 INFO:tasks.ceph.osd.4.smithi142.stderr: 8: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.536 INFO:tasks.ceph.osd.4.smithi142.stderr:2019-10-13T01:04:37.539+0000 7fc0c2ed0700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.536 INFO:tasks.ceph.osd.4.smithi142.stderr: in thread 7fc0c2ed0700 thread_name:bstore_aio 2019-10-13T01:04:37.536 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.536 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.536 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (()+0xf630) [0x7fc0cdf22630] 2019-10-13T01:04:37.537 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (gsignal()+0x37) [0x7fc0ccd16377] 2019-10-13T01:04:37.537 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (abort()+0x148) [0x7fc0ccd17a68] 2019-10-13T01:04:37.537 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x558791aabd46] 2019-10-13T01:04:37.537 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.537 INFO:tasks.ceph.osd.4.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.537 INFO:tasks.ceph.osd.4.smithi142.stderr: 7: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.537 INFO:tasks.ceph.osd.4.smithi142.stderr: 8: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.538 INFO:tasks.ceph.osd.4.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.538 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.538 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.538 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (()+0xf630) [0x7f51a0596630] 2019-10-13T01:04:37.538 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (gsignal()+0x37) [0x7f519f38a377] 2019-10-13T01:04:37.538 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (abort()+0x148) [0x7f519f38ba68] 2019-10-13T01:04:37.538 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55c5b28a6d46] 2019-10-13T01:04:37.539 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.539 INFO:tasks.ceph.osd.6.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.539 INFO:tasks.ceph.osd.6.smithi142.stderr: 7: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.539 INFO:tasks.ceph.osd.6.smithi142.stderr: 8: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.539 INFO:tasks.ceph.osd.6.smithi142.stderr:2019-10-13T01:04:37.539+0000 7f5195544700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.539 INFO:tasks.ceph.osd.6.smithi142.stderr: in thread 7f5195544700 thread_name:bstore_aio 2019-10-13T01:04:37.539 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.540 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.540 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (()+0xf630) [0x7f51a0596630] 2019-10-13T01:04:37.540 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (gsignal()+0x37) [0x7f519f38a377] 2019-10-13T01:04:37.540 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (abort()+0x148) [0x7f519f38ba68] 2019-10-13T01:04:37.540 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55c5b28a6d46] 2019-10-13T01:04:37.540 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.540 INFO:tasks.ceph.osd.6.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.541 INFO:tasks.ceph.osd.6.smithi142.stderr: 7: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.541 INFO:tasks.ceph.osd.6.smithi142.stderr: 8: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.541 INFO:tasks.ceph.osd.6.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.541 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.541 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.541 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (()+0xf630) [0x7ff2251ca630] 2019-10-13T01:04:37.541 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (gsignal()+0x37) [0x7ff223fbe377] 2019-10-13T01:04:37.542 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (abort()+0x148) [0x7ff223fbfa68] 2019-10-13T01:04:37.542 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55b6ad5ead46] 2019-10-13T01:04:37.542 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.542 INFO:tasks.ceph.osd.7.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.542 INFO:tasks.ceph.osd.7.smithi142.stderr: 7: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.542 INFO:tasks.ceph.osd.7.smithi142.stderr: 8: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.542 INFO:tasks.ceph.osd.7.smithi142.stderr:2019-10-13T01:04:37.539+0000 7ff21a178700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.543 INFO:tasks.ceph.osd.7.smithi142.stderr: in thread 7ff21a178700 thread_name:bstore_aio 2019-10-13T01:04:37.543 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.543 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.543 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (()+0xf630) [0x7ff2251ca630] 2019-10-13T01:04:37.543 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (gsignal()+0x37) [0x7ff223fbe377] 2019-10-13T01:04:37.544 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (abort()+0x148) [0x7ff223fbfa68] 2019-10-13T01:04:37.544 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55b6ad5ead46] 2019-10-13T01:04:37.544 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.545 INFO:tasks.ceph.osd.7.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.545 INFO:tasks.ceph.osd.7.smithi142.stderr: 7: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.545 INFO:tasks.ceph.osd.7.smithi142.stderr: 8: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.546 INFO:tasks.ceph.osd.7.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.546 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.699 INFO:tasks.ceph.osd.4.smithi142.stderr: -2337> 2019-10-13T01:04:36.627+0000 7fc0c2ed0700 -1 bdev(0x55879dfe2000 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.699 INFO:tasks.ceph.osd.7.smithi142.stderr: -289> 2019-10-13T01:04:36.626+0000 7ff21a178700 -1 bdev(0x55b6b9f8e000 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.699 INFO:tasks.ceph.osd.7.smithi142.stderr: -1> 2019-10-13T01:04:37.538+0000 7ff21a178700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff21a178700 time 2019-10-13T01:04:36.788019+0000 2019-10-13T01:04:37.700 INFO:tasks.ceph.osd.7.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.700 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.700 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.700 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55b6ad5eac7e] 2019-10-13T01:04:37.700 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.700 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.701 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.701 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.701 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.701 INFO:tasks.ceph.osd.7.smithi142.stderr: 0> 2019-10-13T01:04:37.539+0000 7ff21a178700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.701 INFO:tasks.ceph.osd.7.smithi142.stderr: in thread 7ff21a178700 thread_name:bstore_aio 2019-10-13T01:04:37.701 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.702 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.702 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (()+0xf630) [0x7ff2251ca630] 2019-10-13T01:04:37.702 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (gsignal()+0x37) [0x7ff223fbe377] 2019-10-13T01:04:37.702 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (abort()+0x148) [0x7ff223fbfa68] 2019-10-13T01:04:37.702 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55b6ad5ead46] 2019-10-13T01:04:37.702 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.702 INFO:tasks.ceph.osd.7.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.703 INFO:tasks.ceph.osd.7.smithi142.stderr: 7: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.703 INFO:tasks.ceph.osd.7.smithi142.stderr: 8: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.703 INFO:tasks.ceph.osd.7.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.703 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.704 INFO:tasks.ceph.osd.6.smithi142.stderr: -382> 2019-10-13T01:04:36.628+0000 7f5195544700 -1 bdev(0x55c5bed46000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.704 INFO:tasks.ceph.osd.6.smithi142.stderr: -155> 2019-10-13T01:04:36.892+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=45909, to_read={3:29b66dc8:::100000001d4.0000000e:head=368640,4096,0,1339392,4096,0}, subchunks={3:29b66dc8:::100000001d4.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:37.705 INFO:tasks.ceph.osd.6.smithi142.stderr: -1> 2019-10-13T01:04:37.538+0000 7f5195544700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5195544700 time 2019-10-13T01:04:36.787979+0000 2019-10-13T01:04:37.705 INFO:tasks.ceph.osd.6.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.705 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.705 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.705 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55c5b28a6c7e] 2019-10-13T01:04:37.705 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.705 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.706 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.706 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.707 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.707 INFO:tasks.ceph.osd.6.smithi142.stderr: 0> 2019-10-13T01:04:37.539+0000 7f5195544700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.707 INFO:tasks.ceph.osd.6.smithi142.stderr: in thread 7f5195544700 thread_name:bstore_aio 2019-10-13T01:04:37.707 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.707 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.707 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (()+0xf630) [0x7f51a0596630] 2019-10-13T01:04:37.708 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (gsignal()+0x37) [0x7f519f38a377] 2019-10-13T01:04:37.708 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (abort()+0x148) [0x7f519f38ba68] 2019-10-13T01:04:37.708 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55c5b28a6d46] 2019-10-13T01:04:37.708 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.708 INFO:tasks.ceph.osd.6.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.708 INFO:tasks.ceph.osd.6.smithi142.stderr: 7: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.708 INFO:tasks.ceph.osd.6.smithi142.stderr: 8: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.709 INFO:tasks.ceph.osd.6.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.709 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.709 INFO:tasks.ceph.osd.7.smithi142.stderr: -9999> 2019-10-13T01:04:36.626+0000 7ff21a178700 -1 bdev(0x55b6b9f8e000 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.709 INFO:tasks.ceph.osd.7.smithi142.stderr: -9998> 2019-10-13T01:04:37.538+0000 7ff21a178700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7ff21a178700 time 2019-10-13T01:04:36.788019+0000 2019-10-13T01:04:37.709 INFO:tasks.ceph.osd.7.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.709 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.709 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.710 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55b6ad5eac7e] 2019-10-13T01:04:37.710 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.710 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.710 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.710 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.710 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.711 INFO:tasks.ceph.osd.7.smithi142.stderr: -9997> 2019-10-13T01:04:37.539+0000 7ff21a178700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.711 INFO:tasks.ceph.osd.7.smithi142.stderr: in thread 7ff21a178700 thread_name:bstore_aio 2019-10-13T01:04:37.711 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.711 INFO:tasks.ceph.osd.7.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.711 INFO:tasks.ceph.osd.7.smithi142.stderr: 1: (()+0xf630) [0x7ff2251ca630] 2019-10-13T01:04:37.711 INFO:tasks.ceph.osd.7.smithi142.stderr: 2: (gsignal()+0x37) [0x7ff223fbe377] 2019-10-13T01:04:37.711 INFO:tasks.ceph.osd.7.smithi142.stderr: 3: (abort()+0x148) [0x7ff223fbfa68] 2019-10-13T01:04:37.712 INFO:tasks.ceph.osd.7.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55b6ad5ead46] 2019-10-13T01:04:37.712 INFO:tasks.ceph.osd.7.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55b6adc5aeae] 2019-10-13T01:04:37.712 INFO:tasks.ceph.osd.7.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55b6adc5d4ad] 2019-10-13T01:04:37.712 INFO:tasks.ceph.osd.7.smithi142.stderr: 7: (()+0x7ea5) [0x7ff2251c2ea5] 2019-10-13T01:04:37.712 INFO:tasks.ceph.osd.7.smithi142.stderr: 8: (clone()+0x6d) [0x7ff2240868cd] 2019-10-13T01:04:37.713 INFO:tasks.ceph.osd.7.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.713 INFO:tasks.ceph.osd.7.smithi142.stderr: 2019-10-13T01:04:37.713 INFO:tasks.ceph.osd.4.smithi142.stderr: -1> 2019-10-13T01:04:37.538+0000 7fc0c2ed0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc0c2ed0700 time 2019-10-13T01:04:36.788010+0000 2019-10-13T01:04:37.713 INFO:tasks.ceph.osd.4.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.713 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.713 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.714 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x558791aabc7e] 2019-10-13T01:04:37.714 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.714 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.714 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.714 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.714 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.715 INFO:tasks.ceph.osd.4.smithi142.stderr: 0> 2019-10-13T01:04:37.539+0000 7fc0c2ed0700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.715 INFO:tasks.ceph.osd.4.smithi142.stderr: in thread 7fc0c2ed0700 thread_name:bstore_aio 2019-10-13T01:04:37.715 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.715 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.715 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (()+0xf630) [0x7fc0cdf22630] 2019-10-13T01:04:37.715 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (gsignal()+0x37) [0x7fc0ccd16377] 2019-10-13T01:04:37.715 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (abort()+0x148) [0x7fc0ccd17a68] 2019-10-13T01:04:37.716 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x558791aabd46] 2019-10-13T01:04:37.716 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.716 INFO:tasks.ceph.osd.4.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.716 INFO:tasks.ceph.osd.4.smithi142.stderr: 7: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.717 INFO:tasks.ceph.osd.4.smithi142.stderr: 8: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.717 INFO:tasks.ceph.osd.4.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.717 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.718 INFO:tasks.ceph.osd.6.smithi142.stderr: -9999> 2019-10-13T01:04:36.628+0000 7f5195544700 -1 bdev(0x55c5bed46000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.718 INFO:tasks.ceph.osd.6.smithi142.stderr: -9998> 2019-10-13T01:04:36.892+0000 7f5197d49700 -1 osd.6 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=45909, to_read={3:29b66dc8:::100000001d4.0000000e:head=368640,4096,0,1339392,4096,0}, subchunks={3:29b66dc8:::100000001d4.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:37.718 INFO:tasks.ceph.osd.6.smithi142.stderr: -9997> 2019-10-13T01:04:37.538+0000 7f5195544700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5195544700 time 2019-10-13T01:04:36.787979+0000 2019-10-13T01:04:37.718 INFO:tasks.ceph.osd.6.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.718 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.718 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.719 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55c5b28a6c7e] 2019-10-13T01:04:37.719 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.719 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.719 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.719 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.719 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.720 INFO:tasks.ceph.osd.6.smithi142.stderr: -9996> 2019-10-13T01:04:37.539+0000 7f5195544700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.720 INFO:tasks.ceph.osd.6.smithi142.stderr: in thread 7f5195544700 thread_name:bstore_aio 2019-10-13T01:04:37.720 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.720 INFO:tasks.ceph.osd.6.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.720 INFO:tasks.ceph.osd.6.smithi142.stderr: 1: (()+0xf630) [0x7f51a0596630] 2019-10-13T01:04:37.720 INFO:tasks.ceph.osd.6.smithi142.stderr: 2: (gsignal()+0x37) [0x7f519f38a377] 2019-10-13T01:04:37.721 INFO:tasks.ceph.osd.6.smithi142.stderr: 3: (abort()+0x148) [0x7f519f38ba68] 2019-10-13T01:04:37.721 INFO:tasks.ceph.osd.6.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55c5b28a6d46] 2019-10-13T01:04:37.721 INFO:tasks.ceph.osd.6.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55c5b2f16eae] 2019-10-13T01:04:37.722 INFO:tasks.ceph.osd.6.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55c5b2f194ad] 2019-10-13T01:04:37.722 INFO:tasks.ceph.osd.6.smithi142.stderr: 7: (()+0x7ea5) [0x7f51a058eea5] 2019-10-13T01:04:37.722 INFO:tasks.ceph.osd.6.smithi142.stderr: 8: (clone()+0x6d) [0x7f519f4528cd] 2019-10-13T01:04:37.722 INFO:tasks.ceph.osd.6.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.722 INFO:tasks.ceph.osd.6.smithi142.stderr: 2019-10-13T01:04:37.722 INFO:tasks.ceph.osd.4.smithi142.stderr: -9999> 2019-10-13T01:04:36.627+0000 7fc0c2ed0700 -1 bdev(0x55879dfe2000 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.723 INFO:tasks.ceph.osd.4.smithi142.stderr: -9998> 2019-10-13T01:04:37.538+0000 7fc0c2ed0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc0c2ed0700 time 2019-10-13T01:04:36.788010+0000 2019-10-13T01:04:37.723 INFO:tasks.ceph.osd.4.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.723 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.723 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.723 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x558791aabc7e] 2019-10-13T01:04:37.723 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.723 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.724 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.724 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.724 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.724 INFO:tasks.ceph.osd.4.smithi142.stderr: -9997> 2019-10-13T01:04:37.539+0000 7fc0c2ed0700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.724 INFO:tasks.ceph.osd.4.smithi142.stderr: in thread 7fc0c2ed0700 thread_name:bstore_aio 2019-10-13T01:04:37.724 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.724 INFO:tasks.ceph.osd.4.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 1: (()+0xf630) [0x7fc0cdf22630] 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 2: (gsignal()+0x37) [0x7fc0ccd16377] 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 3: (abort()+0x148) [0x7fc0ccd17a68] 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x558791aabd46] 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x55879211beae] 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55879211e4ad] 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 7: (()+0x7ea5) [0x7fc0cdf1aea5] 2019-10-13T01:04:37.725 INFO:tasks.ceph.osd.4.smithi142.stderr: 8: (clone()+0x6d) [0x7fc0ccdde8cd] 2019-10-13T01:04:37.726 INFO:tasks.ceph.osd.4.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.726 INFO:tasks.ceph.osd.4.smithi142.stderr: 2019-10-13T01:04:37.807 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:37.820+0000 7efcd1b94700 -1 bdev(0x561ea1bd4000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.808 INFO:tasks.ceph.osd.5.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7efcd1b94700 time 2019-10-13T01:04:37.820932+0000 2019-10-13T01:04:37.808 INFO:tasks.ceph.osd.5.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.809 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.809 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x561e95f86c7e] 2019-10-13T01:04:37.809 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.809 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.809 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.809 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.809 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:37.821+0000 7efcd1b94700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7efcd1b94700 time 2019-10-13T01:04:37.820932+0000 2019-10-13T01:04:37.810 INFO:tasks.ceph.osd.5.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.810 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.810 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.810 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x561e95f86c7e] 2019-10-13T01:04:37.810 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.810 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.810 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.811 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.811 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.811 INFO:tasks.ceph.osd.5.smithi142.stderr:*** Caught signal (Aborted) ** 2019-10-13T01:04:37.811 INFO:tasks.ceph.osd.5.smithi142.stderr: in thread 7efcd1b94700 thread_name:bstore_aio 2019-10-13T01:04:37.812 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.812 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (()+0xf630) [0x7efcdcbe6630] 2019-10-13T01:04:37.812 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (gsignal()+0x37) [0x7efcdb9da377] 2019-10-13T01:04:37.812 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (abort()+0x148) [0x7efcdb9dba68] 2019-10-13T01:04:37.812 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x561e95f86d46] 2019-10-13T01:04:37.812 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.812 INFO:tasks.ceph.osd.5.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.813 INFO:tasks.ceph.osd.5.smithi142.stderr: 7: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.813 INFO:tasks.ceph.osd.5.smithi142.stderr: 8: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.813 INFO:tasks.ceph.osd.5.smithi142.stderr:2019-10-13T01:04:37.822+0000 7efcd1b94700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.813 INFO:tasks.ceph.osd.5.smithi142.stderr: in thread 7efcd1b94700 thread_name:bstore_aio 2019-10-13T01:04:37.813 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.813 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.813 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (()+0xf630) [0x7efcdcbe6630] 2019-10-13T01:04:37.814 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (gsignal()+0x37) [0x7efcdb9da377] 2019-10-13T01:04:37.814 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (abort()+0x148) [0x7efcdb9dba68] 2019-10-13T01:04:37.814 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x561e95f86d46] 2019-10-13T01:04:37.814 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.814 INFO:tasks.ceph.osd.5.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.814 INFO:tasks.ceph.osd.5.smithi142.stderr: 7: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.814 INFO:tasks.ceph.osd.5.smithi142.stderr: 8: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.815 INFO:tasks.ceph.osd.5.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.815 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.821 INFO:tasks.ceph.osd.5.smithi142.stderr: -2> 2019-10-13T01:04:37.820+0000 7efcd1b94700 -1 bdev(0x561ea1bd4000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.821 INFO:tasks.ceph.osd.5.smithi142.stderr: -1> 2019-10-13T01:04:37.821+0000 7efcd1b94700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7efcd1b94700 time 2019-10-13T01:04:37.820932+0000 2019-10-13T01:04:37.821 INFO:tasks.ceph.osd.5.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.821 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.821 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.822 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x561e95f86c7e] 2019-10-13T01:04:37.822 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.822 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.822 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.822 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.822 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.822 INFO:tasks.ceph.osd.5.smithi142.stderr: 0> 2019-10-13T01:04:37.822+0000 7efcd1b94700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.823 INFO:tasks.ceph.osd.5.smithi142.stderr: in thread 7efcd1b94700 thread_name:bstore_aio 2019-10-13T01:04:37.823 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.823 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.824 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (()+0xf630) [0x7efcdcbe6630] 2019-10-13T01:04:37.824 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (gsignal()+0x37) [0x7efcdb9da377] 2019-10-13T01:04:37.825 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (abort()+0x148) [0x7efcdb9dba68] 2019-10-13T01:04:37.825 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x561e95f86d46] 2019-10-13T01:04:37.825 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.825 INFO:tasks.ceph.osd.5.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.825 INFO:tasks.ceph.osd.5.smithi142.stderr: 7: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.825 INFO:tasks.ceph.osd.5.smithi142.stderr: 8: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.826 INFO:tasks.ceph.osd.5.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.826 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.826 INFO:tasks.ceph.osd.5.smithi142.stderr: -9999> 2019-10-13T01:04:37.820+0000 7efcd1b94700 -1 bdev(0x561ea1bd4000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T01:04:37.826 INFO:tasks.ceph.osd.5.smithi142.stderr: -9998> 2019-10-13T01:04:37.821+0000 7efcd1b94700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7efcd1b94700 time 2019-10-13T01:04:37.820932+0000 2019-10-13T01:04:37.826 INFO:tasks.ceph.osd.5.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T01:04:37.827 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.827 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.827 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x561e95f86c7e] 2019-10-13T01:04:37.827 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.827 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.827 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.827 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.828 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.828 INFO:tasks.ceph.osd.5.smithi142.stderr: -9997> 2019-10-13T01:04:37.822+0000 7efcd1b94700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:04:37.828 INFO:tasks.ceph.osd.5.smithi142.stderr: in thread 7efcd1b94700 thread_name:bstore_aio 2019-10-13T01:04:37.828 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.828 INFO:tasks.ceph.osd.5.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:04:37.828 INFO:tasks.ceph.osd.5.smithi142.stderr: 1: (()+0xf630) [0x7efcdcbe6630] 2019-10-13T01:04:37.828 INFO:tasks.ceph.osd.5.smithi142.stderr: 2: (gsignal()+0x37) [0x7efcdb9da377] 2019-10-13T01:04:37.829 INFO:tasks.ceph.osd.5.smithi142.stderr: 3: (abort()+0x148) [0x7efcdb9dba68] 2019-10-13T01:04:37.829 INFO:tasks.ceph.osd.5.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x561e95f86d46] 2019-10-13T01:04:37.829 INFO:tasks.ceph.osd.5.smithi142.stderr: 5: (KernelDevice::_aio_thread()+0xebe) [0x561e965f6eae] 2019-10-13T01:04:37.829 INFO:tasks.ceph.osd.5.smithi142.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x561e965f94ad] 2019-10-13T01:04:37.829 INFO:tasks.ceph.osd.5.smithi142.stderr: 7: (()+0x7ea5) [0x7efcdcbdeea5] 2019-10-13T01:04:37.829 INFO:tasks.ceph.osd.5.smithi142.stderr: 8: (clone()+0x6d) [0x7efcdbaa28cd] 2019-10-13T01:04:37.829 INFO:tasks.ceph.osd.5.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:04:37.830 INFO:tasks.ceph.osd.5.smithi142.stderr: 2019-10-13T01:04:37.906 INFO:tasks.ceph.osd.6.smithi142.stderr:daemon-helper: command failed with exit status 5 2019-10-13T01:04:37.917 INFO:tasks.ceph.osd.7.smithi142.stderr:daemon-helper: command failed with exit status 5 2019-10-13T01:04:37.999 INFO:tasks.ceph.osd.5.smithi142.stderr:daemon-helper: command failed with exit status 5 2019-10-13T01:04:38.003 INFO:tasks.ceph.osd.4.smithi142.stderr:daemon-helper: command failed with exit status 5 2019-10-13T01:04:38.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2019-10-13T01:04:38.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2019-10-13T01:04:38.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2019-10-13T01:04:38.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2019-10-13T01:04:40.337 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T01:04:40.350+0000 7f74be998700 -1 osd.1 26 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpReadReply(3.6s2 26/25 ECSubReadReply(tid=39, attrs_read=0)) 2019-10-13T01:04:40.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:04:40.570+0000 7fdde2779700 -1 osd.0 26 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpRead(3.3s0 26/25 ECSubRead(tid=23023, to_read={3:d4ba5905:::10000000159.00000018:head=1028096,4096,0}, subchunks={3:d4ba5905:::10000000159.00000018:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:44.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T01:04:44.493+0000 7f74be998700 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.6s2 26/25 ECSubRead(tid=432, to_read={3:732a4a78:::10000000165.00000001:head=806912,4096,0,925696,4096,0,1040384,4096,0}, subchunks={3:732a4a78:::10000000165.00000001:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:44.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2019-10-13T01:04:44.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2019-10-13T01:04:44.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~6s 2019-10-13T01:04:44.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~6s 2019-10-13T01:04:51.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~12s 2019-10-13T01:04:51.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~12s 2019-10-13T01:04:51.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~12s 2019-10-13T01:04:51.069 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~12s 2019-10-13T01:04:56.230 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:04:56.244+0000 7f5e9787c700 -1 osd.2 26 get_health_metrics reporting 11 slow ops, oldest is MOSDECSubOpRead(3.2s1 26/25 ECSubRead(tid=32990, to_read={3:4e94f364:::10000000181.00000010:head=806912,4096,0,962560,4096,0,1667072,4096,0,1843200,4096,0,1925120,4096,0,2072576,4096,0,2084864,4096,0}, subchunks={3:4e94f364:::10000000181.00000010:head=[0,1]}, attrs_to_read=)) 2019-10-13T01:04:57.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~18s 2019-10-13T01:04:57.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~18s 2019-10-13T01:04:57.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~18s 2019-10-13T01:04:57.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~18s 2019-10-13T01:04:59.503 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:04:59.517+0000 7fdde2779700 -1 osd.0 26 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 26/25 ECSubReadReply(tid=24594, attrs_read=0)) 2019-10-13T01:05:03.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~24s 2019-10-13T01:05:03.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~24s 2019-10-13T01:05:03.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~24s 2019-10-13T01:05:03.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~24s 2019-10-13T01:05:06.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T01:05:06.556+0000 7fdde2779700 -1 osd.0 26 get_health_metrics reporting 7 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 26/25 ECSubReadReply(tid=24833, attrs_read=0)) 2019-10-13T01:05:09.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~31s 2019-10-13T01:05:09.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~31s 2019-10-13T01:05:09.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~31s 2019-10-13T01:05:09.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~31s 2019-10-13T01:05:15.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~37s 2019-10-13T01:05:15.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~37s 2019-10-13T01:05:15.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~37s 2019-10-13T01:05:15.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~37s 2019-10-13T01:05:21.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~43s 2019-10-13T01:05:21.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~43s 2019-10-13T01:05:21.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~43s 2019-10-13T01:05:21.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~43s 2019-10-13T01:05:27.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~49s 2019-10-13T01:05:27.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~49s 2019-10-13T01:05:27.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~49s 2019-10-13T01:05:27.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~49s 2019-10-13T01:05:33.065 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T01:05:33.077+0000 7f5e9787c700 -1 osd.2 26 get_health_metrics reporting 7 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 26/25 ECSubReadReply(tid=35196, attrs_read=0)) 2019-10-13T01:05:33.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~55s 2019-10-13T01:05:33.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~55s 2019-10-13T01:05:33.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~55s 2019-10-13T01:05:33.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~55s 2019-10-13T01:05:39.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~61s 2019-10-13T01:05:39.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~61s 2019-10-13T01:05:39.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~61s 2019-10-13T01:05:39.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~61s 2019-10-13T01:05:46.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~67s 2019-10-13T01:05:46.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~67s 2019-10-13T01:05:46.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~67s 2019-10-13T01:05:46.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~67s 2019-10-13T01:05:52.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~73s 2019-10-13T01:05:52.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~73s 2019-10-13T01:05:52.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~73s 2019-10-13T01:05:52.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~73s 2019-10-13T01:05:58.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~79s 2019-10-13T01:05:58.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~79s 2019-10-13T01:05:58.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~79s 2019-10-13T01:05:58.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~79s 2019-10-13T01:06:04.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~86s 2019-10-13T01:06:04.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~86s 2019-10-13T01:06:04.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~86s 2019-10-13T01:06:04.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~86s 2019-10-13T01:06:10.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~92s 2019-10-13T01:06:10.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~92s 2019-10-13T01:06:10.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~92s 2019-10-13T01:06:10.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~92s 2019-10-13T01:06:16.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~98s 2019-10-13T01:06:16.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~98s 2019-10-13T01:06:16.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~98s 2019-10-13T01:06:16.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~98s 2019-10-13T01:06:22.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~104s 2019-10-13T01:06:22.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~104s 2019-10-13T01:06:22.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~104s 2019-10-13T01:06:22.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~104s 2019-10-13T01:06:28.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~110s 2019-10-13T01:06:28.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~110s 2019-10-13T01:06:28.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~110s 2019-10-13T01:06:28.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~110s 2019-10-13T01:06:29.141 INFO:tasks.ceph.mon.b.smithi142.stderr:2019-10-13T01:06:29.153+0000 7f869d63e700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T01:06:29.141 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x00313030'5' Value size = 29063) 2019-10-13T01:06:29.141 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T01:06:29.141 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T01:06:29.382 INFO:tasks.ceph.mon.b.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f869d63e700 time 2019-10-13T01:06:29.386122+0000 2019-10-13T01:06:29.382 INFO:tasks.ceph.mon.b.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:29.383 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.383 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f86a867ecfc] 2019-10-13T01:06:29.383 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.383 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.383 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.383 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.383 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.384 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.384 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.384 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.384 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.384 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.384 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.384 INFO:tasks.ceph.mon.b.smithi142.stderr:*** Caught signal (Aborted) ** 2019-10-13T01:06:29.385 INFO:tasks.ceph.mon.b.smithi142.stderr: in thread 7f869d63e700 thread_name:safe_timer 2019-10-13T01:06:29.385 INFO:tasks.ceph.mon.b.smithi142.stderr:2019-10-13T01:06:29.396+0000 7f869d63e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f869d63e700 time 2019-10-13T01:06:29.386122+0000 2019-10-13T01:06:29.385 INFO:tasks.ceph.mon.b.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:29.385 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.385 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.385 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f86a867ecfc] 2019-10-13T01:06:29.385 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.386 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.387 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.387 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.387 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.400 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.401 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (()+0xf630) [0x7f86a6484630] 2019-10-13T01:06:29.401 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (gsignal()+0x37) [0x7f86a5278377] 2019-10-13T01:06:29.401 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (abort()+0x148) [0x7f86a5279a68] 2019-10-13T01:06:29.401 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f86a867edc4] 2019-10-13T01:06:29.401 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.401 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.401 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 13: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 14: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.402 INFO:tasks.ceph.mon.b.smithi142.stderr: 15: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.403 INFO:tasks.ceph.mon.b.smithi142.stderr:2019-10-13T01:06:29.414+0000 7f869d63e700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:06:29.403 INFO:tasks.ceph.mon.b.smithi142.stderr: in thread 7f869d63e700 thread_name:safe_timer 2019-10-13T01:06:29.403 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.403 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.403 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (()+0xf630) [0x7f86a6484630] 2019-10-13T01:06:29.403 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (gsignal()+0x37) [0x7f86a5278377] 2019-10-13T01:06:29.403 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (abort()+0x148) [0x7f86a5279a68] 2019-10-13T01:06:29.404 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f86a867edc4] 2019-10-13T01:06:29.404 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.404 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.404 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.404 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.404 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.404 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.405 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.405 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.405 INFO:tasks.ceph.mon.b.smithi142.stderr: 13: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.405 INFO:tasks.ceph.mon.b.smithi142.stderr: 14: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.405 INFO:tasks.ceph.mon.b.smithi142.stderr: 15: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.405 INFO:tasks.ceph.mon.b.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:06:29.405 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.411 INFO:tasks.ceph.mon.b.smithi142.stderr: -2> 2019-10-13T01:06:29.153+0000 7f869d63e700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T01:06:29.411 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x00313030'5' Value size = 29063) 2019-10-13T01:06:29.411 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T01:06:29.411 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T01:06:29.411 INFO:tasks.ceph.mon.b.smithi142.stderr: -1> 2019-10-13T01:06:29.396+0000 7f869d63e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f869d63e700 time 2019-10-13T01:06:29.386122+0000 2019-10-13T01:06:29.412 INFO:tasks.ceph.mon.b.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:29.412 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.412 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.412 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f86a867ecfc] 2019-10-13T01:06:29.412 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.412 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.412 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.413 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: 0> 2019-10-13T01:06:29.414+0000 7f869d63e700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: in thread 7f869d63e700 thread_name:safe_timer 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (()+0xf630) [0x7f86a6484630] 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (gsignal()+0x37) [0x7f86a5278377] 2019-10-13T01:06:29.414 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (abort()+0x148) [0x7f86a5279a68] 2019-10-13T01:06:29.415 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f86a867edc4] 2019-10-13T01:06:29.415 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.415 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.415 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.415 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.416 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.416 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.416 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.417 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.417 INFO:tasks.ceph.mon.b.smithi142.stderr: 13: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.417 INFO:tasks.ceph.mon.b.smithi142.stderr: 14: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.417 INFO:tasks.ceph.mon.b.smithi142.stderr: 15: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.417 INFO:tasks.ceph.mon.b.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:06:29.417 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.418 INFO:tasks.ceph.mon.b.smithi142.stderr: -9999> 2019-10-13T01:06:29.153+0000 7f869d63e700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T01:06:29.418 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x00313030'5' Value size = 29063) 2019-10-13T01:06:29.418 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T01:06:29.418 INFO:tasks.ceph.mon.b.smithi142.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T01:06:29.418 INFO:tasks.ceph.mon.b.smithi142.stderr: -9998> 2019-10-13T01:06:29.396+0000 7f869d63e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f869d63e700 time 2019-10-13T01:06:29.386122+0000 2019-10-13T01:06:29.418 INFO:tasks.ceph.mon.b.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:29.418 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.419 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.419 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f86a867ecfc] 2019-10-13T01:06:29.419 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.419 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.419 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.419 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.419 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.420 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.420 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.420 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.420 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.420 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.420 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.420 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.421 INFO:tasks.ceph.mon.b.smithi142.stderr: -9997> 2019-10-13T01:06:29.414+0000 7f869d63e700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:06:29.421 INFO:tasks.ceph.mon.b.smithi142.stderr: in thread 7f869d63e700 thread_name:safe_timer 2019-10-13T01:06:29.421 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.421 INFO:tasks.ceph.mon.b.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:29.423 INFO:tasks.ceph.mon.b.smithi142.stderr: 1: (()+0xf630) [0x7f86a6484630] 2019-10-13T01:06:29.424 INFO:tasks.ceph.mon.b.smithi142.stderr: 2: (gsignal()+0x37) [0x7f86a5278377] 2019-10-13T01:06:29.424 INFO:tasks.ceph.mon.b.smithi142.stderr: 3: (abort()+0x148) [0x7f86a5279a68] 2019-10-13T01:06:29.424 INFO:tasks.ceph.mon.b.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f86a867edc4] 2019-10-13T01:06:29.424 INFO:tasks.ceph.mon.b.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55bc13650aa1] 2019-10-13T01:06:29.424 INFO:tasks.ceph.mon.b.smithi142.stderr: 6: (Paxos::begin(ceph::buffer::v14_2_0::list&)+0x4f7) [0x55bc13759ab7] 2019-10-13T01:06:29.424 INFO:tasks.ceph.mon.b.smithi142.stderr: 7: (Paxos::propose_pending()+0x11e) [0x55bc1375b2ae] 2019-10-13T01:06:29.425 INFO:tasks.ceph.mon.b.smithi142.stderr: 8: (Paxos::trigger_propose()+0x2fa) [0x55bc1375dd5a] 2019-10-13T01:06:29.425 INFO:tasks.ceph.mon.b.smithi142.stderr: 9: (PaxosService::propose_pending()+0x259) [0x55bc13763f49] 2019-10-13T01:06:29.425 INFO:tasks.ceph.mon.b.smithi142.stderr: 10: (()+0x339239) [0x55bc13764239] 2019-10-13T01:06:29.425 INFO:tasks.ceph.mon.b.smithi142.stderr: 11: (Context::complete(int)+0x9) [0x55bc1368e599] 2019-10-13T01:06:29.425 INFO:tasks.ceph.mon.b.smithi142.stderr: 12: (SafeTimer::timer_thread()+0x1a8) [0x7f86a8751878] 2019-10-13T01:06:29.426 INFO:tasks.ceph.mon.b.smithi142.stderr: 13: (SafeTimerThread::entry()+0xd) [0x7f86a8752c6d] 2019-10-13T01:06:29.426 INFO:tasks.ceph.mon.b.smithi142.stderr: 14: (()+0x7ea5) [0x7f86a647cea5] 2019-10-13T01:06:29.426 INFO:tasks.ceph.mon.b.smithi142.stderr: 15: (clone()+0x6d) [0x7f86a53408cd] 2019-10-13T01:06:29.426 INFO:tasks.ceph.mon.b.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:06:29.427 INFO:tasks.ceph.mon.b.smithi142.stderr: 2019-10-13T01:06:29.714 INFO:tasks.ceph.mon.b.smithi142.stderr:daemon-helper: command crashed with signal 6 2019-10-13T01:06:34.871 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~116s 2019-10-13T01:06:34.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~116s 2019-10-13T01:06:34.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~116s 2019-10-13T01:06:34.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~116s 2019-10-13T01:06:34.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2019-10-13T01:06:38.142 INFO:tasks.ceph.mon.c.smithi142.stderr:2019-10-13T01:06:38.156+0000 7feaaf0f6700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T01:06:38.142 INFO:tasks.ceph.mon.c.smithi142.stderr:Put( Prefix = m key = 'nitor'0x00656c65'ction_epoch' Value size = 8) 2019-10-13T01:06:38.143 INFO:tasks.ceph.mon.c.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7feaaf0f6700 time 2019-10-13T01:06:38.158031+0000 2019-10-13T01:06:38.144 INFO:tasks.ceph.mon.c.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:38.144 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.144 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7feabc93bcfc] 2019-10-13T01:06:38.144 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.144 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.144 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.145 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.145 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.145 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.145 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.145 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.145 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.145 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr:2019-10-13T01:06:38.157+0000 7feaaf0f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7feaaf0f6700 time 2019-10-13T01:06:38.158031+0000 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.146 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7feabc93bcfc] 2019-10-13T01:06:38.147 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.147 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.147 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.147 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.147 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.147 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.147 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.148 INFO:tasks.ceph.mon.c.smithi142.stderr:*** Caught signal (Aborted) ** 2019-10-13T01:06:38.151 INFO:tasks.ceph.mon.c.smithi142.stderr: in thread 7feaaf0f6700 thread_name:ms_dispatch 2019-10-13T01:06:38.152 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.152 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (()+0xf630) [0x7feaba741630] 2019-10-13T01:06:38.152 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (gsignal()+0x37) [0x7feab9535377] 2019-10-13T01:06:38.152 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (abort()+0x148) [0x7feab9536a68] 2019-10-13T01:06:38.152 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7feabc93bdc4] 2019-10-13T01:06:38.152 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.153 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.154 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.154 INFO:tasks.ceph.mon.c.smithi142.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.156 INFO:tasks.ceph.mon.c.smithi142.stderr: 16: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.156 INFO:tasks.ceph.mon.c.smithi142.stderr: 17: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.156 INFO:tasks.ceph.mon.c.smithi142.stderr:2019-10-13T01:06:38.158+0000 7feaaf0f6700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:06:38.156 INFO:tasks.ceph.mon.c.smithi142.stderr: in thread 7feaaf0f6700 thread_name:ms_dispatch 2019-10-13T01:06:38.156 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.156 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.157 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (()+0xf630) [0x7feaba741630] 2019-10-13T01:06:38.157 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (gsignal()+0x37) [0x7feab9535377] 2019-10-13T01:06:38.157 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (abort()+0x148) [0x7feab9536a68] 2019-10-13T01:06:38.157 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7feabc93bdc4] 2019-10-13T01:06:38.157 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.157 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.158 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.158 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.158 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.158 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.158 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.158 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.158 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.159 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.159 INFO:tasks.ceph.mon.c.smithi142.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.159 INFO:tasks.ceph.mon.c.smithi142.stderr: 16: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.159 INFO:tasks.ceph.mon.c.smithi142.stderr: 17: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.159 INFO:tasks.ceph.mon.c.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:06:38.160 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.160 INFO:tasks.ceph.mon.c.smithi142.stderr: -2> 2019-10-13T01:06:38.156+0000 7feaaf0f6700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T01:06:38.160 INFO:tasks.ceph.mon.c.smithi142.stderr:Put( Prefix = m key = 'nitor'0x00656c65'ction_epoch' Value size = 8) 2019-10-13T01:06:38.161 INFO:tasks.ceph.mon.c.smithi142.stderr: -1> 2019-10-13T01:06:38.157+0000 7feaaf0f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7feaaf0f6700 time 2019-10-13T01:06:38.158031+0000 2019-10-13T01:06:38.170 INFO:tasks.ceph.mon.c.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:38.170 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.170 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.170 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7feabc93bcfc] 2019-10-13T01:06:38.170 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.170 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.171 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.171 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.171 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.171 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.171 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.172 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.172 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.172 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.172 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.172 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.173 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.173 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.173 INFO:tasks.ceph.mon.c.smithi142.stderr: 0> 2019-10-13T01:06:38.158+0000 7feaaf0f6700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:06:38.173 INFO:tasks.ceph.mon.c.smithi142.stderr: in thread 7feaaf0f6700 thread_name:ms_dispatch 2019-10-13T01:06:38.174 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.174 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.174 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (()+0xf630) [0x7feaba741630] 2019-10-13T01:06:38.174 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (gsignal()+0x37) [0x7feab9535377] 2019-10-13T01:06:38.175 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (abort()+0x148) [0x7feab9536a68] 2019-10-13T01:06:38.175 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7feabc93bdc4] 2019-10-13T01:06:38.175 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.175 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.175 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.176 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.176 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.176 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.176 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.177 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.177 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.177 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.177 INFO:tasks.ceph.mon.c.smithi142.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.178 INFO:tasks.ceph.mon.c.smithi142.stderr: 16: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.178 INFO:tasks.ceph.mon.c.smithi142.stderr: 17: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.178 INFO:tasks.ceph.mon.c.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:06:38.178 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.179 INFO:tasks.ceph.mon.c.smithi142.stderr: -9999> 2019-10-13T01:06:38.156+0000 7feaaf0f6700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000015.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T01:06:38.179 INFO:tasks.ceph.mon.c.smithi142.stderr:Put( Prefix = m key = 'nitor'0x00656c65'ction_epoch' Value size = 8) 2019-10-13T01:06:38.179 INFO:tasks.ceph.mon.c.smithi142.stderr: -9998> 2019-10-13T01:06:38.157+0000 7feaaf0f6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7feaaf0f6700 time 2019-10-13T01:06:38.158031+0000 2019-10-13T01:06:38.179 INFO:tasks.ceph.mon.c.smithi142.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-6132-ga8ee8ac/rpm/el7/BUILD/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T01:06:38.180 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.180 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.180 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7feabc93bcfc] 2019-10-13T01:06:38.180 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.181 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.181 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.181 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.181 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.181 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.182 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.182 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.182 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.182 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.183 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.183 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.183 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.183 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.184 INFO:tasks.ceph.mon.c.smithi142.stderr: -9997> 2019-10-13T01:06:38.158+0000 7feaaf0f6700 -1 *** Caught signal (Aborted) ** 2019-10-13T01:06:38.184 INFO:tasks.ceph.mon.c.smithi142.stderr: in thread 7feaaf0f6700 thread_name:ms_dispatch 2019-10-13T01:06:38.184 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.184 INFO:tasks.ceph.mon.c.smithi142.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T01:06:38.184 INFO:tasks.ceph.mon.c.smithi142.stderr: 1: (()+0xf630) [0x7feaba741630] 2019-10-13T01:06:38.185 INFO:tasks.ceph.mon.c.smithi142.stderr: 2: (gsignal()+0x37) [0x7feab9535377] 2019-10-13T01:06:38.185 INFO:tasks.ceph.mon.c.smithi142.stderr: 3: (abort()+0x148) [0x7feab9536a68] 2019-10-13T01:06:38.185 INFO:tasks.ceph.mon.c.smithi142.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7feabc93bdc4] 2019-10-13T01:06:38.185 INFO:tasks.ceph.mon.c.smithi142.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0x901) [0x55f346e8caa1] 2019-10-13T01:06:38.186 INFO:tasks.ceph.mon.c.smithi142.stderr: 6: (Elector::persist_epoch(unsigned int)+0x16d) [0x55f346f3e3bd] 2019-10-13T01:06:38.186 INFO:tasks.ceph.mon.c.smithi142.stderr: 7: (ElectionLogic::bump_epoch(unsigned int)+0x43) [0x55f346f42403] 2019-10-13T01:06:38.186 INFO:tasks.ceph.mon.c.smithi142.stderr: 8: (ElectionLogic::receive_propose(int, unsigned int)+0x247) [0x55f346f43cd7] 2019-10-13T01:06:38.186 INFO:tasks.ceph.mon.c.smithi142.stderr: 9: (Elector::handle_propose(boost::intrusive_ptr)+0x1e6) [0x55f346f3d716] 2019-10-13T01:06:38.187 INFO:tasks.ceph.mon.c.smithi142.stderr: 10: (Elector::dispatch(boost::intrusive_ptr)+0x995) [0x55f346f3ef35] 2019-10-13T01:06:38.187 INFO:tasks.ceph.mon.c.smithi142.stderr: 11: (Monitor::dispatch_op(boost::intrusive_ptr)+0x110f) [0x55f346ec84af] 2019-10-13T01:06:38.187 INFO:tasks.ceph.mon.c.smithi142.stderr: 12: (Monitor::_ms_dispatch(Message*)+0x9ff) [0x55f346ec909f] 2019-10-13T01:06:38.188 INFO:tasks.ceph.mon.c.smithi142.stderr: 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55f346ef7078] 2019-10-13T01:06:38.189 INFO:tasks.ceph.mon.c.smithi142.stderr: 14: (DispatchQueue::entry()+0x11ba) [0x7feabcb3cf1a] 2019-10-13T01:06:38.190 INFO:tasks.ceph.mon.c.smithi142.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7feabcbd6f1d] 2019-10-13T01:06:38.190 INFO:tasks.ceph.mon.c.smithi142.stderr: 16: (()+0x7ea5) [0x7feaba739ea5] 2019-10-13T01:06:38.190 INFO:tasks.ceph.mon.c.smithi142.stderr: 17: (clone()+0x6d) [0x7feab95fd8cd] 2019-10-13T01:06:38.191 INFO:tasks.ceph.mon.c.smithi142.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T01:06:38.191 INFO:tasks.ceph.mon.c.smithi142.stderr: 2019-10-13T01:06:38.289 INFO:tasks.ceph.mon.c.smithi142.stderr:daemon-helper: command crashed with signal 6 2019-10-13T01:06:40.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~122s 2019-10-13T01:06:40.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~122s 2019-10-13T01:06:40.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~122s 2019-10-13T01:06:40.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~122s 2019-10-13T01:06:40.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2019-10-13T01:06:40.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~6s 2019-10-13T01:06:46.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~128s 2019-10-13T01:06:46.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~128s 2019-10-13T01:06:46.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~128s 2019-10-13T01:06:46.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~128s 2019-10-13T01:06:46.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~6s 2019-10-13T01:06:46.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~12s 2019-10-13T01:06:52.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~134s 2019-10-13T01:06:52.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~134s 2019-10-13T01:06:52.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~134s 2019-10-13T01:06:52.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~134s 2019-10-13T01:06:52.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~12s 2019-10-13T01:06:52.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~18s 2019-10-13T01:06:58.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~140s 2019-10-13T01:06:58.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~140s 2019-10-13T01:06:58.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~140s 2019-10-13T01:06:58.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~140s 2019-10-13T01:06:58.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~18s 2019-10-13T01:06:58.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~24s 2019-10-13T01:07:04.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~146s 2019-10-13T01:07:04.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~146s 2019-10-13T01:07:04.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~146s 2019-10-13T01:07:04.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~146s 2019-10-13T01:07:04.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~24s 2019-10-13T01:07:04.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~30s 2019-10-13T01:07:10.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~152s 2019-10-13T01:07:10.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~152s 2019-10-13T01:07:10.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~152s 2019-10-13T01:07:10.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~152s 2019-10-13T01:07:10.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~30s 2019-10-13T01:07:10.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~36s 2019-10-13T01:07:16.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~158s 2019-10-13T01:07:16.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~158s 2019-10-13T01:07:16.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~158s 2019-10-13T01:07:16.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~158s 2019-10-13T01:07:16.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~35s 2019-10-13T01:07:16.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~41s 2019-10-13T01:07:22.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~164s 2019-10-13T01:07:22.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~164s 2019-10-13T01:07:22.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~164s 2019-10-13T01:07:22.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~164s 2019-10-13T01:07:22.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~41s 2019-10-13T01:07:22.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~47s 2019-10-13T01:07:28.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~169s 2019-10-13T01:07:28.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~169s 2019-10-13T01:07:28.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~169s 2019-10-13T01:07:28.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~169s 2019-10-13T01:07:28.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~47s 2019-10-13T01:07:28.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~53s 2019-10-13T01:07:33.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~175s 2019-10-13T01:07:33.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~175s 2019-10-13T01:07:33.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~175s 2019-10-13T01:07:33.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~175s 2019-10-13T01:07:33.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~53s 2019-10-13T01:07:33.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~59s 2019-10-13T01:07:39.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~181s 2019-10-13T01:07:39.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~181s 2019-10-13T01:07:39.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~181s 2019-10-13T01:07:39.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~181s 2019-10-13T01:07:39.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~59s 2019-10-13T01:07:39.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~65s 2019-10-13T01:07:45.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~187s 2019-10-13T01:07:45.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~187s 2019-10-13T01:07:45.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~187s 2019-10-13T01:07:45.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~187s 2019-10-13T01:07:45.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~65s 2019-10-13T01:07:45.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~71s 2019-10-13T01:07:51.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~193s 2019-10-13T01:07:51.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~193s 2019-10-13T01:07:51.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~193s 2019-10-13T01:07:51.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~193s 2019-10-13T01:07:51.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~71s 2019-10-13T01:07:51.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~77s 2019-10-13T01:07:57.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~199s 2019-10-13T01:07:57.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~199s 2019-10-13T01:07:57.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~199s 2019-10-13T01:07:57.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~199s 2019-10-13T01:07:57.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~77s 2019-10-13T01:07:57.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~83s 2019-10-13T01:08:03.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~205s 2019-10-13T01:08:03.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~205s 2019-10-13T01:08:03.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~205s 2019-10-13T01:08:03.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~205s 2019-10-13T01:08:03.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~83s 2019-10-13T01:08:03.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~89s 2019-10-13T01:08:09.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~211s 2019-10-13T01:08:09.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~211s 2019-10-13T01:08:09.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~211s 2019-10-13T01:08:09.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~211s 2019-10-13T01:08:09.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~89s 2019-10-13T01:08:09.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2019-10-13T01:08:15.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~217s 2019-10-13T01:08:15.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~217s 2019-10-13T01:08:15.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~217s 2019-10-13T01:08:15.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~217s 2019-10-13T01:08:15.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2019-10-13T01:08:15.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~101s 2019-10-13T01:08:21.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~223s 2019-10-13T01:08:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~223s 2019-10-13T01:08:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~223s 2019-10-13T01:08:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~223s 2019-10-13T01:08:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~100s 2019-10-13T01:08:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~106s 2019-10-13T01:08:27.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~229s 2019-10-13T01:08:27.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~229s 2019-10-13T01:08:27.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~229s 2019-10-13T01:08:27.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~229s 2019-10-13T01:08:27.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~106s 2019-10-13T01:08:27.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~112s 2019-10-13T01:08:33.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~234s 2019-10-13T01:08:33.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~234s 2019-10-13T01:08:33.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~234s 2019-10-13T01:08:33.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~234s 2019-10-13T01:08:33.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~112s 2019-10-13T01:08:33.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~118s 2019-10-13T01:08:35.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:08:35.120+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:08:38.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~240s 2019-10-13T01:08:38.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~240s 2019-10-13T01:08:38.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~240s 2019-10-13T01:08:38.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~240s 2019-10-13T01:08:38.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~118s 2019-10-13T01:08:38.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~124s 2019-10-13T01:08:40.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:08:40.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:08:44.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~246s 2019-10-13T01:08:44.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~246s 2019-10-13T01:08:44.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~246s 2019-10-13T01:08:44.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~246s 2019-10-13T01:08:44.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~124s 2019-10-13T01:08:44.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~130s 2019-10-13T01:08:45.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:08:45.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:08:50.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:08:50.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:08:50.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~252s 2019-10-13T01:08:50.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~252s 2019-10-13T01:08:50.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~252s 2019-10-13T01:08:50.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~252s 2019-10-13T01:08:50.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~130s 2019-10-13T01:08:50.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~136s 2019-10-13T01:08:55.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:08:55.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:08:56.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~258s 2019-10-13T01:08:56.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~258s 2019-10-13T01:08:56.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~258s 2019-10-13T01:08:56.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~258s 2019-10-13T01:08:56.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~136s 2019-10-13T01:08:56.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~142s 2019-10-13T01:09:00.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:00.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:02.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~264s 2019-10-13T01:09:02.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~264s 2019-10-13T01:09:02.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~264s 2019-10-13T01:09:02.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~264s 2019-10-13T01:09:02.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~142s 2019-10-13T01:09:02.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~148s 2019-10-13T01:09:05.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:05.124+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:08.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~270s 2019-10-13T01:09:08.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~270s 2019-10-13T01:09:08.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~270s 2019-10-13T01:09:08.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~270s 2019-10-13T01:09:08.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~148s 2019-10-13T01:09:08.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~154s 2019-10-13T01:09:10.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:10.124+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:14.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~276s 2019-10-13T01:09:14.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~276s 2019-10-13T01:09:14.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~276s 2019-10-13T01:09:14.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~276s 2019-10-13T01:09:14.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~154s 2019-10-13T01:09:14.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~160s 2019-10-13T01:09:15.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:15.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:15.868 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:09:15.868 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:09:15.873 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:09:15.873 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:09:20.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:20.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:20.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~282s 2019-10-13T01:09:20.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~282s 2019-10-13T01:09:20.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~282s 2019-10-13T01:09:20.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~282s 2019-10-13T01:09:20.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~160s 2019-10-13T01:09:20.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~166s 2019-10-13T01:09:25.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:25.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:26.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~288s 2019-10-13T01:09:26.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~288s 2019-10-13T01:09:26.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~288s 2019-10-13T01:09:26.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~288s 2019-10-13T01:09:26.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~165s 2019-10-13T01:09:26.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~172s 2019-10-13T01:09:30.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:30.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:32.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~294s 2019-10-13T01:09:32.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~294s 2019-10-13T01:09:32.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~294s 2019-10-13T01:09:32.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~294s 2019-10-13T01:09:32.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~171s 2019-10-13T01:09:32.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~177s 2019-10-13T01:09:35.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:35.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:38.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~299s 2019-10-13T01:09:38.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~299s 2019-10-13T01:09:38.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~299s 2019-10-13T01:09:38.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~299s 2019-10-13T01:09:38.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~177s 2019-10-13T01:09:38.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~183s 2019-10-13T01:09:40.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:40.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:44.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~305s 2019-10-13T01:09:44.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~305s 2019-10-13T01:09:44.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~305s 2019-10-13T01:09:44.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~305s 2019-10-13T01:09:44.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~183s 2019-10-13T01:09:44.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~189s 2019-10-13T01:09:44.004 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2019-10-13T01:09:44.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T01:09:44.004 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:09:44.005 INFO:teuthology.orchestra.run.smithi155:> stat --file-system '--printf=%T 2019-10-13T01:09:44.005 INFO:teuthology.orchestra.run.smithi155:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T01:09:45.108 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:45.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:46.047 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:09:46.048 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:09:46.052 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:09:46.052 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:09:50.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:50.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 248 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:09:55.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:09:55.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 263 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:00.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:00.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:05.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:05.129+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:10.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:10.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:15.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:15.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:16.143 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:10:16.143 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:10:16.147 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:10:16.147 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:10:20.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:20.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 352 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:25.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:25.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:30.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:30.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:35.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:35.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 402 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:40.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:40.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:45.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:45.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:46.258 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:10:46.258 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:10:46.263 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:10:46.263 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:10:50.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:50.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:10:55.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:10:55.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 472 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:00.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:00.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:05.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:05.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 508 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:10.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:10.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:15.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:15.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 542 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:16.397 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:11:16.397 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:11:16.401 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:11:16.402 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:11:20.117 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:20.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 561 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:25.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:25.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:30.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:30.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:35.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:35.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:40.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:40.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:45.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:45.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:46.537 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:11:46.537 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:11:46.541 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:11:46.541 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:11:50.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:50.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:11:55.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:11:55.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 686 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:00.124 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:00.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:05.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:05.141+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 724 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:10.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:10.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:15.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:15.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 759 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:16.685 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:12:16.686 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:12:16.690 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:12:16.691 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:12:20.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:20.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 779 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:25.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:25.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:30.129 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:30.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:35.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:35.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:40.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:40.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:45.131 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:45.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:46.792 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:12:46.792 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:12:46.796 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:12:46.797 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:12:50.132 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:50.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 889 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:12:55.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:12:55.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:00.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:00.147+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:05.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:05.147+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:10.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:10.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:15.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:15.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:16.889 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:13:16.890 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:13:16.894 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:13:16.894 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:13:20.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:20.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:25.138 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:25.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:30.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:30.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:35.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:35.150+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:40.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:40.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:45.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:45.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:47.012 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:13:47.013 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:13:47.018 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:13:47.018 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:13:50.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:50.152+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:13:55.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:13:55.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1129 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:00.144 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:00.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1148 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:05.144 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:05.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:10.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:10.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:15.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:15.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:17.152 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:14:17.152 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:14:17.157 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:14:17.157 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:14:20.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:20.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1222 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:25.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:25.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:30.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:30.157+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:35.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:35.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1277 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:40.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:40.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1296 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:45.150 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:45.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:47.309 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:14:47.309 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:14:47.313 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:14:47.314 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:14:50.151 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:50.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:14:55.151 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:14:55.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:00.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:00.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1370 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:05.153 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:05.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1389 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:10.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:10.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:15.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:15.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:17.405 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:15:17.406 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:15:17.409 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:15:17.409 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:15:20.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:20.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1444 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:25.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:25.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:30.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:30.164+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1480 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:35.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:35.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1499 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:40.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:40.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:45.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:45.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:47.702 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:15:47.702 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:15:47.706 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:15:47.706 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:15:50.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:50.166+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:15:55.159 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:15:55.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:00.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:00.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1592 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:05.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:05.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:10.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:10.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1628 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:15.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:15.169+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:17.837 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:16:17.837 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:16:17.845 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:16:17.845 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:16:20.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:20.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:25.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:25.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1685 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:30.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:30.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1702 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:35.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:35.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:40.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:40.172+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1740 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:45.167 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:45.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1759 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:47.986 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:16:47.986 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:16:47.990 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:16:47.990 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:16:50.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:50.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1776 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:16:55.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:16:55.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:00.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:00.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:05.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:05.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:10.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:10.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1850 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:15.171 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:15.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:18.081 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:17:18.081 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:17:18.085 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:17:18.085 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:17:20.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:20.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:25.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:25.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:30.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:30.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:35.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:35.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:40.174 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:40.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:45.175 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:45.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:48.273 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:17:48.273 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:17:48.277 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:17:48.277 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:17:50.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:50.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 1998 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:17:55.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:17:55.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:00.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:00.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:05.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:05.183+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:10.178 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:10.183+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2072 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:15.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:15.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:18.418 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:18:18.418 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:18:18.423 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:18:18.423 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:18:20.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:20.183+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:25.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:25.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2129 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:30.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:30.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:35.181 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:35.186+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:40.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:40.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:45.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:45.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:48.532 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:18:48.532 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:18:48.537 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:18:48.537 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:18:50.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:50.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2220 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:18:55.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:18:55.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:00.185 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:00.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:05.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:05.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2277 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:10.200 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:10.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:15.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:15.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:18.680 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:19:18.680 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:19:18.685 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:19:18.685 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:19:20.188 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:20.192+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:25.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:25.192+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:30.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:30.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:35.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:35.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:40.191 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:40.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:45.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:45.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:48.794 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:19:48.794 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:19:48.799 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:19:48.799 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:19:50.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:50.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:19:55.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:19:55.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:00.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:00.197+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2480 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:05.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:05.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2499 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:10.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:10.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:15.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:15.199+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:18.901 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:20:18.901 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:20:18.907 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:20:18.907 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:20:20.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:20.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:25.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:25.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:30.199 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:30.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2590 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:35.200 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:35.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:40.200 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:40.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2628 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:45.201 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:45.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:49.093 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:20:49.093 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:20:49.097 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:20:49.098 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:20:50.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:50.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:20:55.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:20:55.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:00.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:00.206+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2702 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:05.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:05.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:10.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:10.206+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:15.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:15.208+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:19.232 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:21:19.233 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:21:19.237 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:21:19.237 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:21:20.207 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:20.207+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2776 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:25.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:25.208+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:30.209 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:30.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:35.210 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:35.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:40.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:40.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2850 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:45.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:45.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:49.322 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:21:49.322 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:21:49.327 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:21:49.327 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:21:50.212 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:50.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:21:55.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:21:55.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:00.214 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:00.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:05.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:05.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:10.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:10.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2960 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:15.217 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:15.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:19.458 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:22:19.459 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:22:19.464 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:22:19.464 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:22:20.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:20.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 2998 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:25.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:25.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:30.219 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:30.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:35.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:35.219+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:40.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:40.219+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3072 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:45.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:45.219+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:49.680 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:22:49.680 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:22:49.684 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:22:49.684 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:22:50.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:50.220+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:22:55.223 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:22:55.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:00.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:00.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:05.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:05.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:10.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:10.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3182 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:15.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:15.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:19.918 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:23:19.918 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:23:19.922 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:23:19.922 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:23:20.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:20.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3220 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:25.228 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:25.226+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:30.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:30.226+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3256 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:35.230 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:35.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:40.231 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:40.228+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:45.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:45.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:50.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:50.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3330 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:23:50.301 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:23:50.302 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:23:50.305 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:23:50.306 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:23:55.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:23:55.230+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:00.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:00.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:05.239 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:05.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:10.236 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:10.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:15.237 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:15.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:20.238 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:20.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:20.392 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:24:20.393 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:24:20.396 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:24:20.396 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:24:25.238 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:25.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:30.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:30.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3478 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:35.240 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:35.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:40.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:40.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:44.012 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1355, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2019-10-13T01:24:44.013 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1343, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2019-10-13T01:24:45.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:45.237+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:50.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:50.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:24:50.540 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:24:50.541 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:24:50.545 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:24:50.545 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:24:55.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:24:55.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3571 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:00.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:00.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3590 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:05.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:05.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:10.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:10.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:15.247 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:15.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:20.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:20.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:20.688 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:25:20.689 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:25:20.694 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:25:20.694 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:25:25.249 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:25.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:30.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:30.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:35.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:35.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:40.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:40.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:45.253 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:45.247+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:50.253 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:50.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:25:51.057 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:25:51.057 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:25:51.061 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:25:51.062 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:25:55.255 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:25:55.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:00.256 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:00.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:05.256 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:05.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:10.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:10.251+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:15.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:15.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3866 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:20.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:20.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:21.321 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:26:21.321 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:26:21.325 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:26:21.326 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:26:25.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:25.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3904 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:30.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:30.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:35.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:35.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:40.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:40.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:45.263 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:45.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3978 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:50.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:50.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 3995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:26:51.423 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:26:51.423 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:26:51.427 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:26:51.427 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:26:55.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:26:55.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:00.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:00.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:05.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:05.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:10.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:10.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:15.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:15.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:20.270 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:20.263+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:21.575 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:27:21.576 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:27:21.580 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:27:21.580 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:27:25.270 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:25.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:30.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:30.263+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:35.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:35.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:40.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:40.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4181 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:45.272 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:45.266+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:50.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:50.266+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:27:51.776 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:27:51.777 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:27:51.780 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:27:51.781 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:27:55.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:27:55.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4236 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:00.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:00.268+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:05.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:05.268+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4274 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:10.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:10.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:15.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:15.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:20.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:20.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:21.948 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:28:21.949 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:28:21.953 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:28:21.953 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:28:25.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:25.273+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4348 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:30.280 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:30.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:35.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:35.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:40.280 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:40.275+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:45.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:45.275+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:50.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:50.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4439 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:28:52.183 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:28:52.183 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:28:52.192 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:28:52.192 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:28:55.283 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:28:55.277+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:00.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:00.278+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:05.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:05.279+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:10.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:10.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:15.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:15.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4532 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:20.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:20.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:22.371 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:29:22.371 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:29:22.376 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:29:22.376 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:29:25.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:25.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4570 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:30.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:30.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:35.289 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:35.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:40.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:40.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:45.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:45.286+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4642 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:50.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:50.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:29:52.568 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:29:52.568 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:29:52.578 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:29:52.578 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:29:55.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:29:55.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4677 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:00.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:00.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:05.294 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:05.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:10.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:10.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4730 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:15.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:15.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4749 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:20.296 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:20.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4767 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:22.952 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:30:22.952 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:30:22.957 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:30:22.957 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:30:25.297 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:25.292+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:30.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:30.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:35.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:35.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:40.300 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:40.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:45.301 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:45.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4858 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:50.302 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:50.297+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4874 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:30:53.055 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:30:53.055 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:30:53.060 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:30:53.060 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:30:55.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:30:55.298+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4893 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:00.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:00.298+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:05.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:05.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:10.305 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:10.300+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:15.306 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:15.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:20.307 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:20.302+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 4983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:23.258 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:31:23.258 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:31:23.262 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:31:23.262 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:31:25.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:25.303+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:30.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:30.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:35.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:35.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:40.311 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:40.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:45.311 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:45.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5074 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:50.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:50.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5090 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:31:53.495 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:31:53.495 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:31:53.502 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:31:53.502 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:31:55.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:31:55.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:00.314 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:00.309+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:05.314 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:05.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:10.315 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:10.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:15.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:15.311+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5181 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:20.317 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:20.312+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:23.692 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:32:23.692 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:32:23.696 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:32:23.696 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:32:25.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:25.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5218 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:30.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:30.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:35.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:35.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:40.321 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:40.317+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5271 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:45.321 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:45.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5290 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:50.322 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:50.317+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:32:53.913 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:32:53.913 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:32:53.917 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:32:53.917 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:32:55.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:32:55.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:00.324 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:00.319+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:05.325 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:05.320+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5362 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:10.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:10.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5378 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:15.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:15.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5397 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:20.328 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:20.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:24.035 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:33:24.035 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:33:24.040 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:33:24.040 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:33:25.328 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:25.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5434 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:30.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:30.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:35.330 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:35.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5469 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:40.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:40.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:45.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:45.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5506 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:50.341 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:50.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:33:54.207 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:33:54.207 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:33:54.211 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:33:54.211 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:33:55.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:33:55.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5541 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:00.335 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:00.330+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5559 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:05.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:05.331+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:10.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:10.332+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5594 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:15.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:15.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:20.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:20.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:24.379 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:34:24.380 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:34:24.397 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:34:24.397 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:34:25.339 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:25.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:30.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:30.337+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:35.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:35.337+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5685 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:40.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:40.337+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:45.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:45.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5722 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:50.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:50.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:34:54.597 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:34:54.597 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:34:54.602 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:34:54.602 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:34:55.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:34:55.340+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:00.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:00.341+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5775 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:05.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:05.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5794 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:10.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:10.345+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:15.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:15.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:20.350 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:20.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:24.807 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:35:24.808 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:35:24.812 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:35:24.812 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:35:25.350 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:25.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5866 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:30.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:30.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:35.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:35.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5901 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:40.353 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:40.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:45.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:45.350+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5938 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:50.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:50.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:35:54.963 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:35:54.963 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:35:54.967 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:35:54.967 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:35:55.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:35:55.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:00.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:00.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 5991 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:05.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:05.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:10.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:10.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6026 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:15.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:15.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6045 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:20.359 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:20.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6063 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:25.094 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:36:25.094 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:36:25.099 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:36:25.099 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:36:25.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:25.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6082 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:30.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:30.357+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6098 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:35.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:35.358+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6117 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:40.363 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:40.360+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:45.364 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:45.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6154 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:50.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:50.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:36:55.318 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:36:55.318 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:36:55.323 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:36:55.323 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:36:55.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:36:55.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6189 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:00.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:00.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6207 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:05.368 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:05.364+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:10.369 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:10.364+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:15.370 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:15.366+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6261 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:20.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:20.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6279 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:25.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:25.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:25.753 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:37:25.754 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:37:25.758 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:37:25.758 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:37:30.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:30.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:35.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:35.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:40.375 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:40.372+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:45.376 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:45.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6370 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:50.376 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:50.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6386 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:55.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:37:55.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6405 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:37:56.167 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:37:56.167 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:37:56.171 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:37:56.172 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:38:00.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:00.374+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:05.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:05.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:10.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:10.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:15.381 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:15.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:20.382 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:20.379+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:25.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:25.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6514 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:26.454 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:38:26.454 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:38:26.459 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:38:26.459 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:38:30.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:30.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:35.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:35.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:40.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:40.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:45.387 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:45.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6586 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:50.388 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:50.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6602 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:55.390 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:38:55.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:38:56.815 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:38:56.815 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:38:56.819 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:38:56.819 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:39:00.391 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:00.387+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:05.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:05.389+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:10.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:10.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:15.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:15.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6693 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:20.395 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:20.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6711 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:25.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:25.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6730 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:27.065 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:39:27.065 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:39:27.069 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:39:27.069 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:39:30.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:30.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:35.398 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:35.394+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:40.399 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:40.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:45.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:45.396+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:50.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:50.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:55.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:39:55.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:39:57.151 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:39:57.151 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:39:57.155 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:39:57.155 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:40:00.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:00.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6855 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:05.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:05.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6874 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:10.405 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:10.402+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:15.406 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:15.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:20.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:20.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:25.408 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:25.406+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:27.289 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:40:27.289 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:40:27.295 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:40:27.295 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:40:30.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:30.406+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:35.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:35.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:40.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:40.408+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 6999 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:45.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:45.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:50.413 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:50.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:55.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:40:55.411+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:40:57.706 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:40:57.706 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:40:57.710 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:40:57.710 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:41:00.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:00.412+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7071 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:05.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:05.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7090 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:10.417 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:10.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:15.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:15.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:20.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:20.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:25.420 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:25.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:27.790 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:41:27.790 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:41:27.795 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:41:27.795 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:41:30.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:30.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:35.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:35.420+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7197 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:40.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:40.421+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:45.425 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:45.422+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:50.426 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:50.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7250 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:55.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:41:55.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:41:57.921 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:41:57.921 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:41:57.926 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:41:57.927 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:42:00.428 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:00.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:05.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:05.426+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:10.430 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:10.428+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:15.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:15.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7344 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:20.432 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:20.430+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7362 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:25.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:25.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:28.070 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:42:28.070 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:42:28.075 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:42:28.075 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:42:30.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:30.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:35.438 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:35.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7418 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:40.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:40.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:45.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:45.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:50.438 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:50.436+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7472 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:55.439 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:42:55.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:42:58.236 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:42:58.237 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:42:58.247 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:42:58.247 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:43:00.440 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:00.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:05.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:05.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:10.442 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:10.440+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:15.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:15.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:20.444 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:20.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:25.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:25.444+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7604 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:28.507 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:43:28.507 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:43:28.511 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:43:28.512 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:43:30.446 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:30.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:35.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:35.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:40.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:40.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:45.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:45.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:50.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:50.448+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7694 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:55.452 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:43:55.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:43:58.596 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:43:58.597 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:43:58.601 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:43:58.601 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:44:00.452 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:00.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7732 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:05.454 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:05.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:10.455 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:10.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7768 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:15.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:15.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:20.458 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:20.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:25.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:25.458+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:28.710 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:44:28.710 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:44:28.715 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:44:28.715 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:44:30.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:30.458+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7842 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:35.461 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:35.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:40.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:40.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7880 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:45.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:45.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:50.464 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:50.462+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:55.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:44:55.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7936 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:44:58.830 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:44:58.830 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:44:58.834 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:44:58.835 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:45:00.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:00.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:05.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:05.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7974 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:10.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:10.467+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 7990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:15.469 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:15.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:20.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:20.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:25.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:25.470+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:28.972 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:45:28.972 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:45:28.976 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:45:28.977 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:45:30.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:30.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:35.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:35.472+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8084 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:40.474 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:40.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8102 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:45.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:45.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8122 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:50.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:50.475+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8138 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:55.478 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:45:55.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:45:59.136 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:45:59.137 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:45:59.141 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:45:59.141 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:46:00.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:00.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8176 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:05.480 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:05.480+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:10.481 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:10.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8212 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:15.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:15.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8232 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:20.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:20.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8250 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:25.486 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:25.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:29.284 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:46:29.284 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:46:29.288 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:46:29.288 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:46:30.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:30.484+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:35.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:35.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:40.488 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:40.487+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:45.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:45.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8344 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:50.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:50.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:55.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:46:55.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8380 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:46:59.456 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:46:59.456 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:46:59.460 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:46:59.460 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:47:00.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:00.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:05.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:05.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8418 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:10.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:10.495+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:15.496 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:15.495+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:20.497 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:20.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:25.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:25.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:29.612 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:47:29.612 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:47:29.617 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:47:29.617 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:47:30.499 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:30.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:35.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:35.499+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:40.501 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:40.500+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8547 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:45.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:45.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:50.504 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:50.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:55.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:47:55.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8602 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:47:59.795 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:47:59.795 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:47:59.800 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:47:59.800 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:48:00.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:00.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:05.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:05.507+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:10.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:10.507+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8657 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:15.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:15.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:20.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:20.510+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:25.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:25.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:29.966 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:48:29.966 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:48:29.971 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:48:29.971 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:48:30.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:30.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:35.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:35.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8750 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:40.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:40.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:45.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:45.517+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:50.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:50.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8805 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:48:55.519 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:48:55.519+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:00.268 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:49:00.268 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:49:00.273 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:49:00.273 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:49:00.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:00.519+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8842 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:05.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:05.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:10.523 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:10.522+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8877 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:15.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:15.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8896 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:20.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:20.525+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:25.526 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:25.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:30.383 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:49:30.384 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:49:30.389 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:49:30.389 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:49:30.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:30.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:35.529 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:35.529+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8970 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:40.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:40.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 8989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:45.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:45.531+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9008 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:50.532 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:50.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:49:55.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:49:55.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:00.534 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:00.534+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9063 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:00.546 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:50:00.546 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:50:00.550 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:50:00.550 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:50:05.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:05.535+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9082 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:10.537 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:10.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:15.538 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:15.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:20.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:20.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:25.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:25.542+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:30.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:30.543+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9173 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:30.656 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:50:30.656 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:50:30.660 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:50:30.660 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:50:35.544 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:35.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:40.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:40.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:45.546 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:45.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:50.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:50.547+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:50:55.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:50:55.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9266 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:00.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:00.549+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:00.821 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:51:00.821 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:51:00.825 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:51:00.826 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:51:05.551 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:05.552+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:10.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:10.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:15.554 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:15.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:20.555 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:20.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9359 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:25.556 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:25.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9378 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:30.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:30.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:31.035 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:51:31.035 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:51:31.039 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:51:31.040 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:51:35.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:35.558+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:40.560 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:40.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9433 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:45.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:45.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:50.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:50.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9466 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:51:55.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:51:55.563+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9488 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:00.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:00.565+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:01.149 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:52:01.149 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:52:01.153 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:52:01.154 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:52:05.566 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:05.565+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:10.567 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:10.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:15.570 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:15.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:20.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:20.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:25.570 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:25.570+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9600 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:30.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:30.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:31.364 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:52:31.364 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:52:31.369 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:52:31.369 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:52:35.573 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:35.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:40.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:40.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9655 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:45.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:45.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:50.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:50.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:52:55.579 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:52:55.578+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9710 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:00.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:00.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:01.556 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:53:01.556 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:53:01.561 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:53:01.561 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:53:05.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:05.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9748 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:10.582 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:10.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:15.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:15.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:20.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:20.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9803 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:25.587 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:25.587+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9822 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:30.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:30.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:31.681 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:53:31.682 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:53:31.686 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:53:31.687 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:53:35.589 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:35.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9858 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:40.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:40.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9877 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:45.592 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:45.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9896 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:50.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:50.593+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9913 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:53:55.595 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:53:55.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:00.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:00.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:01.887 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:54:01.887 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:54:01.891 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:54:01.891 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:54:05.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:05.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9970 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:10.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:10.599+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 9987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:15.600 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:15.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:20.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:20.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:25.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:25.602+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:30.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:30.603+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10061 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:31.978 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:54:31.978 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:54:31.982 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:54:31.982 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:54:35.605 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:35.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:40.606 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:40.605+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:45.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:45.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:50.609 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:50.609+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:54:55.610 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:54:55.610+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10154 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:00.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:00.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10173 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:02.167 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:55:02.167 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:55:02.171 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:55:02.171 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:55:05.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:05.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:10.614 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:10.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:15.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:15.614+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10228 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:20.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:20.616+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:25.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:25.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10266 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:30.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:30.620+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:32.340 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:55:32.340 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:55:32.345 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:55:32.345 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:55:35.621 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:35.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:40.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:40.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:45.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:45.624+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:50.625 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:50.624+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:55:55.626 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:55:55.625+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:00.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:00.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:02.503 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:56:02.504 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:56:02.508 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:56:02.508 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:56:05.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:05.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:10.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:10.629+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:15.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:15.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:20.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:20.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10469 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:25.634 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:25.634+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10488 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:30.636 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:30.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:32.718 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:56:32.719 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:56:32.723 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:56:32.724 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:56:35.637 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:35.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:40.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:40.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:45.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:45.639+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:50.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:50.640+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:56:55.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:56:55.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:00.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:00.644+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:02.993 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:57:02.993 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:57:02.997 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:57:02.997 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:57:05.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:05.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:10.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:10.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:15.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:15.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:20.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:20.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:25.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:25.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10710 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:30.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:30.651+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:33.171 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:57:33.171 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:57:33.175 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:57:33.176 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:57:35.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:35.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:40.654 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:40.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:45.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:45.655+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:50.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:50.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:57:55.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:57:55.658+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:00.660 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:00.660+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:03.402 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:58:03.402 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:58:03.414 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:58:03.414 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:58:05.661 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:05.660+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10858 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:10.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:10.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:15.664 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:15.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10894 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:20.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:20.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10913 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:25.667 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:25.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:30.668 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:30.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:33.729 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:58:33.729 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:58:33.734 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:58:33.734 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:58:35.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:35.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:40.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:40.671+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 10987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:45.672 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:45.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:50.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:50.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11023 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:58:55.675 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:58:55.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11042 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:00.677 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:00.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11061 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:04.096 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:59:04.096 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:59:04.101 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:59:04.101 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:59:05.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:05.677+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:10.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:10.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:15.681 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:15.681+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:20.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:20.683+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:25.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:25.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11154 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:30.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:30.685+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:34.502 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T01:59:34.503 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:59:34.507 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T01:59:34.507 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T01:59:35.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:35.686+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11190 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:40.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:40.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:45.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:45.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11228 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:50.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:50.690+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T01:59:55.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T01:59:55.691+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11264 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:00.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:00.694+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:04.591 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:00:04.592 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:00:04.599 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:00:04.600 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:00:05.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:05.695+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:10.696 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:10.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11319 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:15.698 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:15.698+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11338 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:20.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:20.698+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:25.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:25.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:30.702 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:30.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:34.722 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:00:34.722 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:00:34.726 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:00:34.726 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:00:35.703 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:35.703+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11412 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:40.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:40.704+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:45.706 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:45.706+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:50.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:50.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:00:55.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:00:55.709+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:00.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:00.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:04.811 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:01:04.811 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:01:04.816 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:01:04.816 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:01:05.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:05.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:10.713 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:10.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11541 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:15.714 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:15.713+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:20.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:20.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:25.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:25.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:30.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:30.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:34.918 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:01:34.918 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:01:34.922 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:01:34.923 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:01:35.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:35.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:40.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:40.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:45.723 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:45.723+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:50.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:50.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11689 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:01:55.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:01:55.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11708 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:00.728 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:00.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:05.110 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:02:05.110 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:02:05.117 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:02:05.117 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:02:05.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:05.728+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:10.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:10.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:15.733 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:15.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:20.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:20.734+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:25.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:25.735+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:30.736 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:30.737+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:35.299 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:02:35.299 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:02:35.303 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:02:35.304 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:02:35.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:35.737+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11856 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:40.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:40.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:45.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:45.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11894 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:50.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:50.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:02:55.744 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:02:55.743+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:00.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:00.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:05.409 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:03:05.409 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:03:05.413 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:03:05.413 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:03:05.746 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:05.746+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:10.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:10.748+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 11985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:15.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:15.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:20.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:20.750+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12023 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:25.752 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:25.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12042 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:30.754 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:30.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12059 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:35.582 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:03:35.582 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:03:35.586 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:03:35.586 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:03:35.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:35.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:40.756 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:40.756+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:45.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:45.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:50.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:50.759+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12133 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:03:55.761 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:03:55.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:00.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:00.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:05.670 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:04:05.670 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:04:05.674 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:04:05.675 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:04:05.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:05.763+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12190 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:10.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:10.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12207 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:15.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:15.766+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:20.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:20.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:25.769 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:25.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12264 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:30.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:30.770+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:35.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:35.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:35.825 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:04:35.826 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:04:35.830 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:04:35.830 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:04:40.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:40.774+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12319 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:45.775 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:45.775+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12338 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:50.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:50.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:04:55.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:04:55.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:00.780 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:00.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:05.785 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:05.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12412 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:05.990 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:05:05.990 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:05:05.994 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:05:05.994 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:05:10.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:10.782+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:15.784 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:15.783+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:20.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:20.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:25.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:25.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:30.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:30.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:35.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:35.791+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:36.203 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:05:36.203 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:05:36.207 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:05:36.208 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:05:40.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:40.791+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12541 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:45.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:45.793+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:50.795 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:50.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:05:55.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:05:55.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:00.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:00.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:05.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:05.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:06.369 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:06:06.370 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:06:06.374 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:06:06.374 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:06:10.801 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:10.801+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:15.802 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:15.802+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:20.804 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:20.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12689 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:25.805 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:25.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:30.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:30.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:35.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:35.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:36.508 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:06:36.508 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:06:36.512 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:06:36.512 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:06:40.810 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:40.809+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:45.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:45.810+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:50.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:50.813+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:06:55.814 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:06:55.814+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:00.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:00.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:05.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:05.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12856 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:06.696 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:07:06.696 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:07:06.701 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:07:06.701 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:07:10.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:10.818+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12873 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:15.820 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:15.819+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:20.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:20.821+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:25.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:25.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:30.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:30.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:35.826 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:35.826+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:36.869 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:07:36.869 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:07:36.874 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:07:36.874 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:07:40.828 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:40.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 12985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:45.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:45.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:50.831 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:50.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:07:55.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:07:55.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13040 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:00.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:00.833+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13059 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:05.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:05.834+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:07.049 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:08:07.049 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:08:07.053 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:08:07.053 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:08:10.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:10.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:15.838 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:15.837+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:20.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:20.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13133 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:25.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:25.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:30.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:30.843+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13169 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:35.844 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:35.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:37.254 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:08:37.254 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:08:37.260 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:08:37.261 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:08:40.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:40.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13207 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:45.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:45.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:50.850 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:50.849+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13243 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:08:55.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:08:55.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:00.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:00.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:05.855 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:05.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:07.485 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:09:07.485 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:09:07.489 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:09:07.489 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:09:10.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:10.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:15.858 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:15.858+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:20.859 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:20.859+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:25.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:25.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:30.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:30.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:35.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:35.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:37.682 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:09:37.682 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:09:37.686 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:09:37.687 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:09:40.865 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:40.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:45.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:45.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:50.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:50.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:09:55.870 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:09:55.870+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:00.871 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:00.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:05.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:05.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:08.043 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:10:08.043 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:10:08.048 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:10:08.048 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:10:10.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:10.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:15.876 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:15.875+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:20.877 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:20.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:25.879 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:25.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:30.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:30.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:35.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:35.882+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:38.143 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:10:38.144 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:10:38.149 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:10:38.149 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:10:40.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:40.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:45.885 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:45.885+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:50.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:50.886+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13687 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:10:55.889 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:10:55.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13708 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:00.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:00.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:05.900 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:05.891+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:08.325 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:11:08.325 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:11:08.329 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:11:08.329 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:11:10.893 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:10.892+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:15.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:15.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:20.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:20.896+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:25.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:25.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:30.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:30.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13835 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:35.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:35.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13856 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:38.555 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:11:38.555 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:11:38.560 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:11:38.560 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:11:40.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:40.901+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13873 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:45.904 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:45.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:50.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:50.905+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:11:55.907 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:11:55.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:00.909 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:00.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:05.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:05.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:08.746 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:12:08.746 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:12:08.751 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:12:08.751 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:12:10.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:10.912+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 13983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:15.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:15.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:20.915 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:20.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:25.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:25.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14040 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:30.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:30.917+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:35.920 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:35.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:38.935 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:12:38.936 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:12:38.941 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:12:38.941 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:12:40.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:40.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:45.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:45.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:50.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:50.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:12:55.927 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:12:55.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:00.928 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:00.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14169 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:05.929 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:05.928+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:09.155 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:13:09.155 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:13:09.159 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:13:09.160 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:13:10.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:10.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:15.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:15.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:20.934 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:20.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14243 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:25.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:25.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:30.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:30.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14279 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:35.938 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:35.938+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:39.336 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:13:39.336 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:13:39.340 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:13:39.340 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:13:40.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:40.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:45.941 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:45.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:50.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:50.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14353 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:13:55.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:13:55.943+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:00.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:00.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:05.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:05.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:09.490 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:14:09.491 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:14:09.502 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:14:09.502 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:14:10.949 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:10.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:15.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:15.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:20.953 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:20.953+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:25.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:25.954+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:30.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:30.955+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14501 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:35.958 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:35.957+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:39.688 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:14:39.688 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:14:39.692 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:14:39.692 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:14:40.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:40.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:45.961 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:45.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:50.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:50.962+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14575 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:14:55.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:14:55.963+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:00.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:00.965+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:05.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:05.967+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14632 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:09.860 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:15:09.860 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:15:09.865 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:15:09.865 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:15:10.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:10.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14649 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:15.971 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:15.969+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:20.972 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:20.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14687 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:25.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:25.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:30.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:30.974+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14723 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:35.977 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:35.975+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:40.100 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:15:40.100 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:15:40.104 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:15:40.104 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:15:40.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:40.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:45.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:45.979+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:50.981 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:50.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:15:55.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:15:55.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:00.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:00.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14835 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:05.986 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:05.985+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:10.490 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:16:10.491 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:16:10.495 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:16:10.495 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:16:10.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:10.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:15.989 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:15.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:20.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:20.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:25.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:25.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:30.995 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:30.994+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:35.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:35.996+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:40.568 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:16:40.568 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:16:40.598 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:16:40.598 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:16:41.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:40.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 14983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:46.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:45.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:51.001 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:51.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:16:56.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:16:56.001+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15040 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:01.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:01.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:06.006 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:06.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:10.721 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:17:10.721 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:17:10.734 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:17:10.734 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:17:11.008 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:11.007+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:16.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:16.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:21.011 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:21.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:26.012 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:26.012+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:31.014 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:31.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:36.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:36.014+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:40.905 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:17:40.906 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:17:40.910 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:17:40.910 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:17:41.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:41.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:46.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:46.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:51.021 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:51.019+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:17:56.022 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:17:56.022+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:01.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:01.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15280 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:06.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:06.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:11.027 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:11.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:11.070 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:18:11.071 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:18:11.077 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:18:11.077 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:18:16.030 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:16.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:21.030 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:21.028+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15354 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:26.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:26.030+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:31.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:31.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:36.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:36.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:41.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:41.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:41.185 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:18:41.185 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:18:41.191 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:18:41.191 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:18:46.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:46.038+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:51.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:51.040+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:18:56.042 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:18:56.041+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:01.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:01.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:06.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:06.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:11.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:11.045+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:11.281 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:19:11.281 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:19:11.285 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:19:11.286 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:19:16.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:16.047+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:21.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:21.049+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:26.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:26.051+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:31.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:31.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:36.055 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:36.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15632 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:41.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:41.056+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:41.427 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:19:41.427 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:19:41.432 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:19:41.432 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:19:46.058 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:46.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:51.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:51.058+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15687 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:19:56.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:19:56.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:01.063 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:01.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:06.065 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:06.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:11.067 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:11.066+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15760 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:11.607 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:20:11.607 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:20:11.612 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:20:11.612 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:20:16.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:16.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:21.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:21.070+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:26.072 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:26.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:31.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:31.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15834 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:36.075 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:36.074+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:41.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:41.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:41.779 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:20:41.779 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:20:41.783 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:20:41.783 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:20:46.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:46.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:51.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:51.079+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15908 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:20:56.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:20:56.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:01.084 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:01.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:06.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:06.085+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15964 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:11.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:11.087+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 15982 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:12.044 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:21:12.047 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:21:12.053 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:21:12.054 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:21:16.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:16.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:21.091 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:21.089+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16020 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:26.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:26.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:31.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:31.093+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16056 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:36.096 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:36.094+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:41.097 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:41.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16094 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:42.140 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:21:42.140 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:21:42.145 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:21:42.145 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:21:46.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:46.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:51.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:51.100+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16130 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:21:56.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:21:56.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:01.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:01.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:06.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:06.104+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:11.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:11.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16204 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:12.367 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:22:12.367 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:22:12.371 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:22:12.371 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:22:16.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:16.107+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:21.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:21.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:26.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:26.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16260 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:31.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:31.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:36.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:36.116+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:41.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:41.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:42.505 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:22:42.505 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:22:42.509 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:22:42.509 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:22:46.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:46.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:51.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:51.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16352 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:22:56.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:22:56.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:01.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:01.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:06.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:06.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:11.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:11.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:12.592 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:23:12.592 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:23:12.597 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:23:12.597 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:23:16.129 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:16.129+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:21.132 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:21.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16464 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:26.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:26.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16482 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:31.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:31.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:36.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:36.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16520 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:41.138 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:41.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16538 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:42.964 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:23:42.965 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:23:42.969 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:23:42.969 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:23:46.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:46.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:51.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:51.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:23:56.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:23:56.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16594 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:01.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:01.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:06.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:06.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16630 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:11.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:11.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:13.060 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:24:13.061 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:24:13.065 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:24:13.065 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:24:16.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:16.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:21.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:21.150+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16685 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:26.153 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:26.152+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:31.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:31.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:36.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:36.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:41.159 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:41.157+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16759 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:43.183 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:24:43.184 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:24:43.188 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:24:43.188 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:24:46.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:46.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:51.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:51.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:24:56.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:24:56.164+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:01.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:01.164+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:06.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:06.166+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:11.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:11.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:13.328 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:25:13.329 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:25:13.333 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:25:13.333 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:25:16.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:16.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16889 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:21.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:21.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:26.175 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:26.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16925 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:31.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:31.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:36.178 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:36.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:41.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:41.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:43.582 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:25:43.582 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:25:43.586 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:25:43.586 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:25:46.181 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:46.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 16999 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:51.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:51.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:25:56.185 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:25:56.183+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:01.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:01.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:06.188 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:06.186+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:11.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:11.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:13.670 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:26:13.671 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:26:13.675 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:26:13.676 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:26:16.191 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:16.190+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:21.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:21.192+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17129 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:26.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:26.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:31.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:31.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:36.199 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:36.197+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:41.201 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:41.199+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:43.848 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:26:43.848 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:26:43.852 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:26:43.852 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:26:46.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:46.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17221 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:51.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:51.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:26:56.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:26:56.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17259 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:01.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:01.207+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17277 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:06.209 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:06.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17295 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:11.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:11.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:14.012 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:27:14.012 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:27:14.016 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:27:14.016 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:27:16.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:16.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:21.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:21.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:26.217 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:26.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:31.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:31.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:36.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:36.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:41.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:41.220+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:44.116 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:27:44.117 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:27:44.121 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:27:44.121 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:27:46.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:46.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17443 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:51.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:51.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:27:56.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:27:56.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:01.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:01.228+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17499 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:06.230 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:06.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:11.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:11.230+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:14.281 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:28:14.281 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:28:14.285 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:28:14.285 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:28:16.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:16.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17555 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:21.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:21.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:26.237 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:26.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:31.240 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:31.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:36.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:36.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17629 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:41.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:41.242+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:44.386 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:28:44.386 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:28:44.391 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:28:44.391 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:28:46.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:46.244+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:51.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:51.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:28:56.249 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:28:56.247+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:01.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:01.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:06.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:06.251+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17739 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:11.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:11.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:14.473 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:29:14.474 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:29:14.478 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:29:14.478 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:29:16.256 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:16.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:21.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:21.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:26.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:26.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17813 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:31.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:31.261+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:36.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:36.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:41.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:41.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:44.589 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:29:44.589 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:29:44.593 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:29:44.593 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:29:46.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:46.266+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:51.269 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:51.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:29:56.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:29:56.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17925 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:01.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:01.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:06.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:06.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:11.277 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:11.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:14.711 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:30:14.711 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:30:14.717 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:30:14.718 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:30:16.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:16.278+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 17999 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:21.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:21.279+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:26.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:26.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18035 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:31.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:31.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:36.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:36.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:41.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:41.286+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:44.844 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:30:44.844 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:30:44.848 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:30:44.848 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:30:46.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:46.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:51.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:51.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:30:56.294 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:30:56.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:01.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:01.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:06.297 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:06.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18183 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:11.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:11.297+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:14.949 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:31:14.949 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:31:14.964 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:31:14.965 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:31:16.301 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:16.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18221 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:21.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:21.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:26.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:26.303+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:31.307 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:31.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:36.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:36.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18295 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:41.311 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:41.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:45.384 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:31:45.384 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:31:45.388 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:31:45.389 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:31:46.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:46.312+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18331 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:51.314 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:51.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:31:56.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:31:56.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:01.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:01.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:06.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:06.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18405 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:11.322 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:11.320+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:15.474 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:32:15.474 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:32:15.479 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:32:15.479 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:32:16.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:16.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18443 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:21.325 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:21.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:26.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:26.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18479 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:31.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:31.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:36.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:36.330+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:41.333 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:41.331+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:45.859 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:32:45.859 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:32:45.907 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:32:45.907 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:32:46.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:46.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:51.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:51.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18571 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:32:56.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:32:56.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:01.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:01.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:06.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:06.341+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:11.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:11.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:16.015 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:33:16.016 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:33:16.020 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:33:16.020 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:33:16.345 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:16.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:21.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:21.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:26.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:26.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:31.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:31.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18719 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:36.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:36.351+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18739 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:41.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:41.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:46.143 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:33:46.144 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:33:46.149 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:33:46.149 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:33:46.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:46.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18775 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:51.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:51.357+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:33:56.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:33:56.359+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18813 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:01.362 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:01.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:06.364 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:06.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18849 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:11.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:11.364+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:16.259 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:34:16.259 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:34:16.265 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:34:16.265 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:34:16.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:16.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:21.369 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:21.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:26.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:26.369+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:31.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:31.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18941 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:36.375 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:36.374+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:41.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:41.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:46.347 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:34:46.348 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:34:46.352 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:34:46.352 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:34:46.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:46.379+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 18997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:51.382 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:51.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:34:56.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:34:56.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19035 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:01.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:01.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:06.387 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:06.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19071 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:11.389 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:11.387+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19089 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:16.390 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:16.389+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:16.453 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:35:16.453 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:35:16.457 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:35:16.458 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:35:21.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:21.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:26.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:26.394+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:31.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:31.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:36.398 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:36.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19183 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:41.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:41.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:46.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:46.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:46.540 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:35:46.541 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:35:46.548 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:35:46.548 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:35:51.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:51.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:35:56.405 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:35:56.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:01.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:01.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:06.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:06.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19293 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:11.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:11.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19311 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:16.413 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:16.412+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19331 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:16.663 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:36:16.663 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:36:16.667 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:36:16.667 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:36:21.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:21.414+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:26.432 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:26.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:31.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:31.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:36.420 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:36.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19405 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:41.423 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:41.420+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:46.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:46.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:46.771 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:36:46.771 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:36:46.778 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:36:46.778 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:36:51.426 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:51.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:36:56.428 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:36:56.427+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19479 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:01.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:01.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:06.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:06.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:11.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:11.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19533 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:16.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:16.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:17.153 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:37:17.153 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:37:17.156 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:37:17.157 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:37:21.438 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:21.436+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19571 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:26.440 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:26.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:31.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:31.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:36.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:36.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:41.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:41.444+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:46.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:46.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:47.546 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:37:47.547 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:37:47.552 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:37:47.552 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:37:51.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:51.448+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19681 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:37:56.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:37:56.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:01.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:01.451+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19719 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:06.455 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:06.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:11.457 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:11.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:16.458 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:16.457+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19775 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:17.628 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:38:17.628 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:38:17.632 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:38:17.632 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:38:21.461 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:21.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:26.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:26.462+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:31.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:31.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:36.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:36.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19849 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:41.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:41.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:46.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:46.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:47.715 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:38:47.716 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:38:47.720 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:38:47.720 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:38:51.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:51.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:38:56.475 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:38:56.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:01.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:01.475+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19941 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:06.478 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:06.477+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:11.481 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:11.480+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:16.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:16.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 19997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:17.804 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:39:17.804 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:39:17.815 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:39:17.816 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:39:21.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:21.484+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:26.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:26.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:31.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:31.487+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:36.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:36.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20071 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:41.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:41.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20089 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:46.494 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:46.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:47.908 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:39:47.908 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:39:47.913 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:39:47.913 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:39:51.496 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:51.495+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:39:56.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:39:56.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:01.501 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:01.500+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:06.504 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:06.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20181 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:11.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:11.504+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:16.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:16.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:18.041 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:40:18.042 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:40:18.046 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:40:18.046 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:40:21.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:21.507+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:26.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:26.509+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:31.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:31.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:36.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:36.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20293 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:41.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:41.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20311 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:46.519 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:46.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:48.172 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:40:48.173 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:40:48.177 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:40:48.178 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:40:51.521 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:51.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:40:56.523 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:40:56.522+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:01.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:01.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:06.526 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:06.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:11.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:11.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20421 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:16.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:16.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:18.293 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:41:18.293 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:41:18.297 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:41:18.298 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:41:21.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:21.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:26.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:26.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:31.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:31.534+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20495 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:36.537 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:36.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:41.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:41.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20533 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:46.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:46.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:48.383 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:41:48.383 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:41:48.387 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:41:48.387 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:41:51.543 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:51.541+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20569 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:41:56.544 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:41:56.542+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:01.546 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:01.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:06.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:06.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:11.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:11.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:16.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:16.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:18.497 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:42:18.497 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:42:18.501 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:42:18.502 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:42:21.554 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:21.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20681 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:26.556 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:26.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:31.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:31.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20717 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:36.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:36.558+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:41.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:41.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:46.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:46.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:48.620 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:42:48.620 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:42:48.624 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:42:48.624 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:42:51.566 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:51.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20791 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:42:56.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:42:56.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:01.570 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:01.569+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:06.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:06.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:11.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:11.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20865 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:16.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:16.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:18.734 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:43:18.734 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:43:18.739 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:43:18.739 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:43:21.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:21.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:26.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:26.578+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:31.582 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:31.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:36.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:36.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:41.586 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:41.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:46.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:46.587+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 20995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:48.953 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:43:48.953 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:43:48.958 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:43:48.958 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:43:51.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:51.589+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:43:56.592 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:43:56.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:01.594 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:01.593+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:06.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:06.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:11.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:11.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21087 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:16.600 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:16.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:19.049 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:44:19.050 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:44:19.055 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:44:19.055 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:44:21.603 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:21.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:26.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:26.602+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:31.606 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:31.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21161 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:36.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:36.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21181 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:41.610 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:41.609+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:46.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:46.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:49.195 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:44:49.196 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:44:49.200 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:44:49.200 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:44:51.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:51.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:44:56.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:44:56.615+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:01.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:01.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:06.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:06.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:11.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:11.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:16.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:16.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:19.322 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:45:19.323 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:45:19.327 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:45:19.327 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:45:21.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:21.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21348 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:26.629 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:26.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:31.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:31.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:36.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:36.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:41.635 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:41.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:46.637 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:46.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21439 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:49.445 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:45:49.445 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:45:49.450 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:45:49.450 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:45:51.639 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:51.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:45:56.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:45:56.639+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:01.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:01.640+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:06.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:06.643+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:11.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:11.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21532 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:16.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:16.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:19.526 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:46:19.526 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:46:19.531 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:46:19.531 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:46:21.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:21.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21570 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:26.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:26.650+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:31.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:31.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:36.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:36.655+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:41.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:41.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:46.661 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:46.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21661 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:49.656 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:46:49.656 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:46:49.660 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:46:49.661 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:46:51.663 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:51.662+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:46:56.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:46:56.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:01.667 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:01.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:06.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:06.667+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:11.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:11.669+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21754 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:16.673 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:16.671+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:19.753 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:47:19.753 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:47:19.757 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:47:19.758 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:47:21.675 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:21.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:26.681 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:26.675+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:31.679 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:31.677+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:36.681 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:36.680+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:41.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:41.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21866 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:46.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:46.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21883 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:49.917 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:47:49.918 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:47:49.922 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:47:49.922 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:47:51.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:51.686+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21902 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:47:56.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:47:56.688+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:01.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:01.690+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:06.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:06.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21957 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:11.697 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:11.695+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21976 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:16.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:16.697+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 21995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:20.115 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:48:20.115 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:48:20.122 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:48:20.122 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:48:21.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:21.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:26.703 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:26.702+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22031 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:31.706 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:31.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:36.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:36.706+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:41.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:41.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:46.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:46.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22105 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:50.292 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:48:50.292 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:48:50.296 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:48:50.297 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:48:51.714 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:51.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22124 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:48:56.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:48:56.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:01.718 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:01.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:06.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:06.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:11.723 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:11.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22198 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:16.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:16.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:20.494 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:49:20.494 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:49:20.498 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:49:20.498 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:49:21.727 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:21.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22236 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:26.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:26.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:31.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:31.729+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22272 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:36.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:36.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:41.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:41.733+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:46.737 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:46.735+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22327 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:50.642 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:49:50.643 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:49:50.647 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:49:50.647 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:49:51.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:51.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22346 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:49:56.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:49:56.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:01.744 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:01.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22383 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:06.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:06.743+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:11.747 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:11.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:16.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:16.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:20.762 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:50:20.763 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:50:20.768 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:50:20.768 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:50:21.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:21.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:26.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:26.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22474 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:31.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:31.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22493 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:36.757 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:36.756+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22512 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:41.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:41.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:46.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:46.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:50.939 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:50:50.939 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:50:50.943 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:50:50.944 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:50:51.764 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:51.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22567 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:50:56.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:50:56.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22586 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:01.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:01.766+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:06.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:06.769+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22622 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:11.773 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:11.770+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:16.775 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:16.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22660 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:21.153 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:51:21.154 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:51:21.158 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:51:21.158 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:51:21.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:21.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22679 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:26.779 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:26.778+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:31.781 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:31.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:36.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:36.781+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22734 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:41.785 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:41.783+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22753 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:46.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:46.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22770 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:51.433 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:51:51.433 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:51:51.438 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:51:51.438 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:51:51.791 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:51.788+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22789 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:51:56.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:51:56.790+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22808 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:01.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:01.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22827 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:06.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:06.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:11.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:11.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:16.801 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:16.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:21.637 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:52:21.637 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:52:21.643 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:52:21.643 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:52:21.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:21.801+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22901 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:26.805 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:26.803+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:31.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:31.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:36.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:36.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:41.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:41.809+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:46.814 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:46.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 22992 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:51.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:51.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23011 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:52:51.825 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:52:51.825 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:52:51.830 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:52:51.830 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:52:56.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:52:56.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23030 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:01.821 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:01.820+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:06.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:06.821+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:11.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:11.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:16.828 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:16.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:21.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:21.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:22.029 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:53:22.029 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:53:22.033 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:53:22.034 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:53:26.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:26.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:31.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:31.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23159 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:36.836 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:36.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:41.838 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:41.837+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23197 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:46.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:46.839+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:51.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:51.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:53:52.126 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:53:52.126 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:53:52.130 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:53:52.130 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:53:56.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:53:56.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23252 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:01.847 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:01.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23279 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:06.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:06.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:11.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:11.849+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:16.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:16.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:21.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:21.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:22.305 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:54:22.306 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:54:22.310 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:54:22.310 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:54:26.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:26.857+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:31.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:31.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:36.863 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:36.862+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:41.866 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:41.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:46.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:46.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23541 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:51.870 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:51.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:54:52.492 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:54:52.492 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:54:52.497 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:54:52.497 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:54:56.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:54:56.870+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23601 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:01.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:01.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:06.877 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:06.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:11.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:11.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23692 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:16.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:16.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23719 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:21.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:21.883+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:22.866 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:55:22.867 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:55:22.871 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:55:22.871 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:55:26.886 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:26.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:31.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:31.886+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:36.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:36.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:41.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:41.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23870 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:46.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:46.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:51.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:51.896+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:55:52.950 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:55:52.951 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:55:52.954 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:55:52.955 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:55:56.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:55:56.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23955 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:01.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:01.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 23988 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:06.904 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:06.902+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:11.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:11.904+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:16.908 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:16.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:21.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:21.908+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:23.066 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:56:23.066 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:56:23.072 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:56:23.072 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:56:26.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:26.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:31.915 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:31.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:36.917 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:36.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24191 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:41.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:41.918+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:46.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:46.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:51.924 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:51.922+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:56:53.197 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:56:53.197 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:56:53.201 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:56:53.201 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:56:56.926 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:56:56.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:01.928 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:01.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24342 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:06.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:06.928+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:11.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:11.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:16.934 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:16.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:21.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:21.936+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:23.344 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:57:23.344 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:57:23.348 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:57:23.349 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:57:26.939 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:26.938+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:31.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:31.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:36.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:36.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:41.946 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:41.944+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:46.948 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:46.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:51.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:51.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:57:53.461 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:57:53.461 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:57:53.465 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:57:53.465 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:57:56.953 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:57:56.950+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:01.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:01.952+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:06.957 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:06.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:11.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:11.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24754 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:16.962 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:16.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24781 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:21.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:21.963+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:23.563 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:58:23.563 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:58:23.568 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:58:23.568 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:58:26.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:26.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:31.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:31.965+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:36.970 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:36.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24899 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:41.972 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:41.970+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:46.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:46.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24957 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:51.977 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:51.975+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 24990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:58:53.887 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:58:53.887 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:58:53.892 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:58:53.892 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:58:56.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:58:56.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:01.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:01.980+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:06.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:06.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:11.986 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:11.984+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:16.989 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:16.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:21.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:21.989+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:24.310 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:59:24.310 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:59:24.314 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:59:24.314 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:59:26.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:26.991+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25193 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:31.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:31.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:36.998 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:36.995+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:42.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:41.999+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:47.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:47.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25311 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:52.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:52.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25346 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T02:59:54.406 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T02:59:54.406 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:59:54.410 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T02:59:54.410 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T02:59:57.007 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T02:59:57.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:02.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:02.006+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:07.011 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:07.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:12.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:12.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25464 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:17.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:17.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:22.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:22.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:24.731 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:00:24.731 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:00:24.736 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:00:24.736 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:00:27.021 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:27.019+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25547 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:32.022 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:32.021+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25582 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:37.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:37.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:42.027 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:42.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:47.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:47.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:52.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:52.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:00:54.981 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:00:54.981 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:00:54.997 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:00:54.997 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:00:57.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:00:57.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:02.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:02.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:07.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:07.035+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:12.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:12.039+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:17.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:17.041+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:22.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:22.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25876 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:25.408 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:01:25.408 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:01:25.413 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:01:25.413 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:01:27.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:27.045+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25901 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:32.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:32.047+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25936 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:37.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:37.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:42.055 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:42.052+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 25994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:47.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:47.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:52.059 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:52.056+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26054 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:01:55.515 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:01:55.515 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:01:55.519 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:01:55.519 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:01:57.061 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:01:57.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26079 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:02.063 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:02.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:07.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:07.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:12.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:12.066+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:17.072 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:17.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26197 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:22.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:22.071+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:25.619 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:02:25.619 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:02:25.623 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:02:25.623 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:02:27.076 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:27.074+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:32.078 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:32.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26290 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:37.080 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:37.078+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:42.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:42.081+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26348 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:47.085 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:47.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26373 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:52.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:52.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:02:55.747 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:02:55.747 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:02:55.751 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:02:55.751 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:02:57.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:02:57.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26433 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:02.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:02.089+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26466 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:07.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:07.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:12.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:12.094+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:17.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:17.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:22.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:22.099+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:26.142 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:03:26.142 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:03:26.147 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:03:26.147 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:03:27.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:27.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:32.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:32.105+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:37.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:37.108+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:42.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:42.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26702 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:47.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:47.111+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:52.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:52.113+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:03:56.233 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:03:56.233 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:03:56.237 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:03:56.238 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:03:57.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:03:57.116+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26787 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:02.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:02.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26819 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:07.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:07.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:12.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:12.124+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:17.129 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:17.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:22.144 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:22.129+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:26.379 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:04:26.379 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:04:26.383 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:04:26.383 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:04:27.132 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:27.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26960 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:32.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:32.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 26994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:37.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:37.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:42.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:42.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:47.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:47.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:52.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:52.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:04:56.542 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:04:56.542 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:04:56.547 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:04:56.548 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:04:57.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:04:57.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:02.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:02.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:07.153 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:07.150+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:12.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:12.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:17.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:17.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:22.159 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:22.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:26.721 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:05:26.721 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:05:26.726 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:05:26.726 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:05:27.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:27.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:32.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:32.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27342 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:37.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:37.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:42.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:42.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:47.171 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:47.169+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:52.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:52.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:05:56.876 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:05:56.876 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:05:56.880 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:05:56.880 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:05:57.175 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:05:57.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:02.178 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:02.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:07.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:07.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:12.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:12.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:17.185 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:17.182+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27599 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:22.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:22.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:26.979 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:06:26.979 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:06:26.984 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:06:26.984 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:06:27.188 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:27.186+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27656 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:32.191 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:32.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:37.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:37.192+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:42.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:42.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:47.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:47.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27772 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:52.200 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:52.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:06:57.135 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:06:57.135 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:06:57.139 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:06:57.139 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:06:57.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:06:57.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:02.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:02.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:07.207 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:07.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:12.210 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:12.207+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27922 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:17.212 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:17.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:22.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:22.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 27979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:27.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:27.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:27.306 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:07:27.306 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:07:27.310 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:07:27.310 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:07:32.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:32.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28035 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:37.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:37.220+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:42.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:42.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28089 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:47.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:47.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:52.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:52.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:57.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:07:57.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:07:57.467 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:07:57.468 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:07:57.472 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:07:57.472 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:08:02.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:02.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:07.236 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:07.235+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:12.239 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:12.237+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:17.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:17.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28280 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:22.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:22.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:27.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:27.244+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:27.551 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:08:27.552 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:08:27.570 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:08:27.571 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:08:32.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:32.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:37.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:37.248+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:42.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:42.251+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:47.255 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:47.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28444 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:52.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:52.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28475 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:57.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:08:57.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:08:57.709 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:08:57.710 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:08:57.714 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:08:57.714 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:09:02.263 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:02.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28529 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:07.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:07.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:12.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:12.266+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:17.272 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:17.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28610 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:22.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:22.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:27.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:27.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:27.835 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:09:27.835 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:09:27.839 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:09:27.839 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:09:32.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:32.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:37.280 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:37.279+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28720 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:42.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:42.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28749 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:47.285 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:47.283+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28774 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:52.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:52.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28805 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:57.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:09:57.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:09:57.949 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:09:57.949 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:09:57.953 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:09:57.954 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:10:02.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:02.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:07.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:07.292+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28884 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:12.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:12.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:17.300 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:17.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:22.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:22.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:27.305 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:27.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 28994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:28.096 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:10:28.097 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:10:28.102 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:10:28.103 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:10:32.307 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:32.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:37.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:37.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:42.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:42.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29079 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:47.315 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:47.312+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:52.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:52.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:57.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:10:57.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29160 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:10:58.253 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:10:58.253 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:10:58.257 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:10:58.257 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:11:02.322 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:02.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29189 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:07.325 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:07.323+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:12.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:12.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:17.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:17.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:22.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:22.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:27.335 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:27.332+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:28.343 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:11:28.343 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:11:28.347 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:11:28.347 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:11:32.339 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:32.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29356 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:37.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:37.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:42.341 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:42.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:47.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:47.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29437 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:52.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:52.345+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29468 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:57.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:11:57.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:11:58.495 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:11:58.495 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:11:58.499 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:11:58.500 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:12:02.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:02.350+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29523 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:07.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:07.351+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:12.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:12.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:17.359 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:17.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:22.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:22.359+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:27.364 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:27.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29661 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:28.632 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:12:28.632 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:12:28.637 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:12:28.637 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:12:32.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:32.364+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29692 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:37.369 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:37.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:42.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:42.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:47.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:47.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:52.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:52.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:57.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:12:57.377+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:12:58.777 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:12:58.778 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:12:58.781 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:12:58.782 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:13:02.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:02.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:07.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:07.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:12.389 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:12.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:17.406 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:17.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29942 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:22.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:22.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29971 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:27.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:27.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 29997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:28.924 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:13:28.925 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:13:28.932 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:13:28.932 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:13:32.399 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:32.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:37.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:37.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30054 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:42.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:42.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:47.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:47.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:52.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:52.406+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:57.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:13:57.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:13:59.095 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:13:59.096 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:13:59.108 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:13:59.108 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:14:02.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:02.412+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:07.417 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:07.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30221 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:12.420 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:12.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30252 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:17.423 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:17.421+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:22.425 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:22.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30307 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:27.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:27.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:29.242 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:14:29.242 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:14:29.247 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:14:29.247 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:14:32.430 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:32.427+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30364 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:37.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:37.430+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:42.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:42.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:47.438 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:47.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30445 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:52.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:52.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30476 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:57.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:14:57.440+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:14:59.406 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:14:59.407 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:14:59.424 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:14:59.424 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:15:02.444 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:02.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:07.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:07.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30557 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:12.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:12.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30588 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:17.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:17.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30614 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:22.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:22.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:27.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:27.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:29.591 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:15:29.592 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:15:29.595 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:15:29.596 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:15:32.461 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:32.458+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:37.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:37.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30726 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:42.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:42.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:47.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:47.467+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30781 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:52.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:52.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:57.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:15:57.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:15:59.745 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:15:59.745 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:15:59.754 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:15:59.754 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:16:02.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:02.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30865 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:07.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:07.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30891 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:12.481 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:12.479+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:17.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:17.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:22.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:22.485+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 30975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:27.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:27.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:29.909 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:16:29.909 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:16:29.914 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:16:29.914 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:16:32.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:32.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31031 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:37.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:37.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:42.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:42.495+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:47.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:47.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:52.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:52.500+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:16:57.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:16:57.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:00.047 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:17:00.047 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:17:00.051 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:17:00.052 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:17:02.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:02.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:07.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:07.509+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31221 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:12.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:12.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:17.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:17.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31277 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:22.519 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:22.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:27.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:27.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31331 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:30.193 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:17:30.193 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:17:30.198 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:17:30.198 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:17:32.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:32.522+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:37.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:37.525+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:42.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:42.527+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:47.534 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:47.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:52.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:52.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31471 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:17:57.537 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:17:57.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:00.338 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:18:00.339 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:18:00.355 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:18:00.355 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:18:02.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:02.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:07.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:07.541+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:12.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:12.543+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:17.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:17.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:22.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:22.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:27.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:27.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31661 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:30.494 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:18:30.494 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:18:30.547 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:18:30.547 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:18:32.555 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:32.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:37.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:37.556+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31717 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:42.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:42.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:47.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:47.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:52.566 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:52.565+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:18:57.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:18:57.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31827 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:00.875 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:19:00.876 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:19:00.880 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:19:00.880 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:19:02.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:02.569+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31855 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:07.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:07.572+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31881 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:12.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:12.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:17.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:17.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:22.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:22.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:27.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:27.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 31991 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:30.987 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:19:30.987 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:19:30.992 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:19:30.993 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:19:32.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:32.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:37.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:37.589+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:42.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:42.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:47.595 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:47.593+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32101 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:52.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:52.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:19:57.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:19:57.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32157 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:01.072 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:20:01.073 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:20:01.077 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:20:01.077 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:20:02.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:02.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:07.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:07.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:12.609 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:12.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:17.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:17.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:22.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:22.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32296 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:27.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:27.616+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32322 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:31.286 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:20:31.287 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:20:31.290 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:20:31.291 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:20:32.621 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:32.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32353 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:37.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:37.620+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32379 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:42.625 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:42.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:47.628 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:47.625+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32434 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:52.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:52.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:20:57.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:20:57.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:01.576 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:21:01.576 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:21:01.580 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:21:01.581 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:21:02.635 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:02.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32520 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:07.637 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:07.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:12.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:12.639+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:17.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:17.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:22.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:22.644+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32632 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:27.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:27.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:31.736 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:21:31.736 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:21:31.740 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:21:31.740 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:21:32.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:32.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32689 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:37.654 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:37.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:42.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:42.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:47.660 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:47.658+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32770 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:52.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:52.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:21:57.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:21:57.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32827 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:01.882 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:22:01.882 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:22:01.886 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:22:01.887 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:22:02.668 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:02.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32856 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:07.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:07.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:12.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:12.671+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32913 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:17.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:17.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:22.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:22.677+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:27.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:27.680+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 32994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:32.044 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:22:32.044 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:22:32.049 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:22:32.049 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:22:32.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:32.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:37.697 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:37.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:42.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:42.690+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:47.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:47.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:52.697 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:52.695+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:22:57.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:22:57.697+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:02.132 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:23:02.132 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:23:02.142 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:23:02.142 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:23:02.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:02.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:07.706 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:07.703+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33218 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:12.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:12.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:17.712 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:17.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:22.715 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:22.713+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:27.718 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:27.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33330 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:32.278 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:23:32.278 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:23:32.283 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:23:32.283 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:23:32.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:32.718+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:37.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:37.721+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:42.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:42.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33416 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:47.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:47.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:52.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:52.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:23:57.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:23:57.733+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33499 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:02.365 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:24:02.366 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:24:02.369 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:24:02.369 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:24:02.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:02.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:07.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:07.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:12.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:12.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:17.746 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:17.743+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:22.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:22.746+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:27.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:27.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:32.458 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:24:32.458 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:24:32.462 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:24:32.462 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:24:32.754 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:32.752+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:37.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:37.755+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33723 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:42.760 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:42.759+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:47.764 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:47.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33778 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:52.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:52.765+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:24:57.769 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:24:57.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33835 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:02.641 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:25:02.641 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:25:02.648 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:25:02.649 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:25:02.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:02.769+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:07.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:07.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:12.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:12.774+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:17.780 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:17.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:22.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:22.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 33976 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:27.785 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:27.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:32.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:32.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:32.795 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:25:32.795 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:25:32.799 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:25:32.799 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:25:37.791 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:37.790+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34059 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:42.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:42.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:47.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:47.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:52.800 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:52.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:25:57.802 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:25:57.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:02.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:02.803+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:02.888 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:26:02.889 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:26:02.899 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:26:02.899 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:26:07.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:07.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:12.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:12.810+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:17.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:17.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:22.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:22.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34312 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:27.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:27.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34338 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:32.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:32.819+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:33.046 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:26:33.046 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:26:33.052 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:26:33.052 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:26:37.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:37.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:42.828 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:42.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:47.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:47.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:52.833 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:52.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:26:57.836 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:26:57.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:02.839 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:02.838+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34536 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:03.260 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:27:03.261 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:27:03.267 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:27:03.267 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:27:07.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:07.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:12.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:12.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34593 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:17.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:17.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:22.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:22.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34648 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:27.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:27.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:32.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:32.854+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:33.421 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:27:33.421 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:27:33.425 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:27:33.425 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:27:37.859 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:37.857+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:42.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:42.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34760 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:47.865 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:47.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:52.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:52.867+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34817 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:27:57.871 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:27:57.869+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:02.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:02.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:03.600 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:28:03.601 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:28:03.605 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:28:03.605 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:28:07.877 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:07.874+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:12.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:12.877+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34929 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:17.882 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:17.880+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34955 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:22.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:22.883+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 34984 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:27.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:27.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:32.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:32.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:33.755 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:28:33.755 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:28:33.766 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:28:33.766 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:28:37.894 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:37.892+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35067 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:42.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:42.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35096 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:47.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:47.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35122 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:52.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:52.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35153 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:28:57.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:28:57.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:02.909 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:02.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:03.934 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:29:03.935 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:29:03.939 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:29:03.939 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:29:07.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:07.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:12.915 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:12.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:17.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:17.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:22.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:22.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35320 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:27.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:27.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35346 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:32.927 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:32.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35377 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:34.090 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:29:34.091 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:29:34.101 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:29:34.101 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:29:37.929 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:37.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:42.933 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:42.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35432 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:47.935 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:47.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:52.938 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:52.936+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:29:57.941 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:29:57.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:02.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:02.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:04.183 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:30:04.183 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:30:04.188 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:30:04.188 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:30:07.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:07.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35570 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:12.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:12.947+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35601 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:17.952 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:17.950+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:22.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:22.953+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35656 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:27.958 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:27.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:32.962 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:32.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:34.355 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:30:34.355 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:30:34.360 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:30:34.360 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:30:37.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:37.961+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35739 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:42.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:42.965+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35768 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:47.970 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:47.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35794 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:52.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:52.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:30:57.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:30:57.973+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:02.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:02.976+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35880 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:04.527 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:31:04.527 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:31:04.533 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:31:04.533 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:31:07.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:07.979+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35906 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:12.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:12.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:17.989 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:17.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:22.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:22.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 35992 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:27.995 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:27.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:32.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:32.996+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:34.723 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:31:34.723 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:31:34.727 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:31:34.727 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:31:38.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:37.999+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:43.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:43.001+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:48.006 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:48.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36130 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:53.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:53.006+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36161 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:31:58.011 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:31:58.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:03.014 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:03.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36216 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:04.902 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:32:04.903 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:32:04.907 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:32:04.907 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:32:08.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:08.015+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:13.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:13.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:18.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:18.021+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:23.026 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:23.024+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:28.029 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:28.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36354 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:33.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:33.030+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:35.049 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:32:35.049 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:32:35.053 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:32:35.053 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:32:38.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:38.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:43.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:43.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36440 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:48.042 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:48.038+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36466 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:53.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:53.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:32:58.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:32:58.045+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:03.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:03.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:05.168 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:33:05.168 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:33:05.173 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:33:05.173 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:33:08.053 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:08.052+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:13.056 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:13.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:18.059 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:18.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:23.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:23.059+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:28.065 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:28.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:33.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:33.065+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:35.315 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:33:35.316 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:33:35.320 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:33:35.320 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:33:38.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:38.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36748 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:43.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:43.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36776 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:48.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:48.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36803 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:53.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:53.078+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:33:58.085 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:33:58.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36860 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:03.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:03.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:05.461 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:34:05.461 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:34:05.466 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:34:05.466 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:34:08.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:08.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36917 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:13.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:13.089+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:18.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:18.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 36972 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:23.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:23.096+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:28.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:28.099+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:33.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:33.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:35.631 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:34:35.632 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:34:35.636 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:34:35.636 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:34:38.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:38.104+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37084 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:43.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:43.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:48.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:48.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:53.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:53.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37169 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:34:58.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:34:58.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:03.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:03.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:05.803 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:35:05.803 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:35:05.807 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:35:05.807 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:35:08.124 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:08.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:13.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:13.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:18.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:18.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:23.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:23.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:28.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:28.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:33.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:33.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:35.980 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:35:35.980 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:35:35.984 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:35:35.984 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:35:38.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:38.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37420 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:43.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:43.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:48.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:48.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:53.151 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:53.150+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:35:58.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:35:58.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37532 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:03.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:03.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:06.136 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:36:06.136 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:36:06.140 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:36:06.140 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:36:08.160 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:08.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37588 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:13.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:13.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37616 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:18.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:18.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:23.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:23.166+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:28.171 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:28.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37702 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:33.174 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:33.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37730 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:36.292 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:36:36.292 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:36:36.298 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:36:36.298 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:36:38.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:38.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:43.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:43.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:48.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:48.182+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:53.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:53.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:36:58.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:36:58.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:03.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:03.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:06.462 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:37:06.462 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:37:06.467 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:37:06.467 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:37:08.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:08.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:13.199 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:13.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:18.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:18.199+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 37986 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:23.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:23.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:28.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:28.206+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:33.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:33.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38072 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:36.671 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:37:36.671 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:37:36.675 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:37:36.675 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:37:38.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:38.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38100 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:43.217 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:43.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38128 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:48.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:48.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:53.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:53.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:37:58.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:37:58.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:03.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:03.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:06.835 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:38:06.835 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:38:06.839 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:38:06.839 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:38:08.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:08.230+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38272 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:13.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:13.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:18.238 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:18.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:23.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:23.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38356 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:28.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:28.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38386 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:33.247 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:33.244+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:36.917 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:38:36.917 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:38:36.921 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:38:36.921 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:38:38.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:38.247+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:43.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:43.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38470 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:48.256 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:48.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:53.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:53.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:38:58.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:38:58.261+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:03.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:03.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:07.032 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:39:07.032 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:39:07.041 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:39:07.041 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:39:08.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:08.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:13.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:13.268+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:18.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:18.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:23.277 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:23.275+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:28.280 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:28.277+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:33.283 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:33.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:37.163 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:39:37.163 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:39:37.167 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:39:37.167 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:39:38.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:38.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:43.289 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:43.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:48.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:48.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:53.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:53.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:39:58.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:39:58.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:03.301 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:03.298+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:07.253 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:40:07.253 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:40:07.257 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:40:07.257 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:40:08.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:08.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 38977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:13.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:13.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39005 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:18.310 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:18.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:23.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:23.311+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:28.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:28.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:33.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:33.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:37.351 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:40:37.351 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:40:37.355 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:40:37.356 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:40:38.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:38.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:43.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:43.323+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:48.328 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:48.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:53.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:53.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:40:58.335 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:40:58.332+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:03.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:03.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:07.555 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:41:07.556 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:41:07.560 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:41:07.560 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:41:08.341 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:08.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:13.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:13.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:18.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:18.345+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:23.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:23.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:28.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:28.351+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:33.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:33.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:37.752 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:41:37.752 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:41:37.756 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:41:37.757 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:41:38.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:38.358+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:43.364 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:43.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:48.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:48.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:53.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:53.369+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:41:58.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:41:58.372+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:03.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:03.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:07.849 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:42:07.849 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:42:07.853 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:42:07.853 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:42:08.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:08.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:13.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:13.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:18.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:18.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:23.389 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:23.386+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:28.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:28.389+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:33.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:33.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39845 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:37.964 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:42:37.964 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:42:37.968 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:42:37.968 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:42:38.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:38.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39877 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:43.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:43.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39904 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:48.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:48.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39938 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:53.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:53.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:42:58.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:42:58.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 39997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:03.413 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:03.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40024 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:08.103 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:43:08.103 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:43:08.107 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:43:08.107 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:43:08.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:08.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40058 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:13.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:13.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:18.423 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:18.420+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40117 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:23.426 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:23.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40144 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:28.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:28.428+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:33.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:33.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:38.260 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:43:38.261 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:43:38.265 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:43:38.265 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:43:38.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:38.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:43.439 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:43.436+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40264 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:48.442 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:48.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:53.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:53.442+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:43:58.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:43:58.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:03.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:03.448+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:08.431 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:44:08.431 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:44:08.435 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:44:08.435 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:44:08.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:08.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40418 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:13.457 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:13.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40445 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:18.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:18.458+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:23.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:23.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40504 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:28.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:28.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40538 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:33.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:33.467+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40565 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:38.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:38.470+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:38.513 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:44:38.513 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:44:38.517 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:44:38.517 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:44:43.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:43.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:48.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:48.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40657 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:53.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:53.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:44:58.486 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:44:58.484+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40716 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:03.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:03.487+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:08.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:08.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:08.657 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:45:08.657 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:45:08.661 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:45:08.661 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:45:13.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:13.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:18.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:18.495+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:23.501 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:23.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:28.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:28.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:33.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:33.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:38.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:38.509+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:38.803 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:45:38.803 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:45:38.808 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:45:38.808 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:45:43.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:43.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 40983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:48.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:48.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:53.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:53.517+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:45:58.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:45:58.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:03.527 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:03.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:08.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:08.527+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:08.949 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:46:08.949 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:46:08.953 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:46:08.953 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:46:13.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:13.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:18.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:18.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:23.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:23.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:28.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:28.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:33.546 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:33.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:38.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:38.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:39.086 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:46:39.086 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:46:39.090 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:46:39.090 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:46:43.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:43.549+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:48.556 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:48.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41377 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:53.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:53.556+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:46:58.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:46:58.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:03.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:03.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:08.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:08.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:09.225 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:47:09.225 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:47:09.228 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:47:09.229 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:47:13.571 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:13.570+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:18.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:18.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:23.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:23.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:28.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:28.578+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:33.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:33.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:38.591 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:38.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:39.316 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:47:39.317 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:47:39.321 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:47:39.321 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:47:43.591 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:43.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:48.595 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:48.592+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:53.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:53.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:47:58.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:47:58.599+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:03.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:03.602+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41823 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:08.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:08.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:09.475 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:48:09.475 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:48:09.480 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:48:09.480 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:48:13.610 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:13.607+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41884 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:18.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:18.610+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:23.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:23.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:28.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:28.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 41977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:33.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:33.620+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:38.626 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:38.625+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:39.646 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:48:39.646 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:48:39.650 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:48:39.650 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:48:43.637 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:43.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42063 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:48.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:48.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:53.636 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:53.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42124 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:48:58.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:48:58.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:03.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:03.639+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42183 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:08.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:08.643+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:09.817 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:49:09.817 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:49:09.821 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:49:09.821 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:49:13.649 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:13.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:18.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:18.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:23.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:23.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42303 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:28.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:28.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:33.663 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:33.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42364 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:38.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:38.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:40.146 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:49:40.146 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:49:40.150 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:49:40.151 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:49:43.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:43.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:48.672 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:48.669+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:53.675 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:53.672+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:49:58.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:49:58.675+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:03.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:03.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:08.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:08.683+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:10.301 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:50:10.302 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:50:10.307 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:50:10.307 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:50:13.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:13.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42604 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:18.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:18.690+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:23.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:23.693+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:28.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:28.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:33.702 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:33.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42724 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:38.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:38.702+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:40.405 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:50:40.405 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:50:40.410 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:50:40.410 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:50:43.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:43.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:48.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:48.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42817 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:53.714 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:53.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:50:58.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:50:58.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42876 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:03.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:03.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:08.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:08.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:10.566 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:51:10.566 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:51:10.570 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:51:10.571 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:51:13.728 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:13.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42964 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:18.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:18.728+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 42996 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:23.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:23.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43023 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:28.738 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:28.735+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:32.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:32.577+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:32.578013+0000) 2019-10-13T03:51:33.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:33.736+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.738089+0000) 2019-10-13T03:51:33.740 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:33.736+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.738219+0000) 2019-10-13T03:51:33.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:33.733+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.734212+0000) 2019-10-13T03:51:33.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:33.737+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.738674+0000) 2019-10-13T03:51:33.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:33.737+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.738915+0000) 2019-10-13T03:51:33.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:33.737+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.739063+0000) 2019-10-13T03:51:33.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:33.737+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.739340+0000) 2019-10-13T03:51:33.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:33.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43084 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:33.742 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:33.734+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:33.735360+0000) 2019-10-13T03:51:34.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:34.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:34.954413+0000) 2019-10-13T03:51:34.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:34.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:34.954386+0000) 2019-10-13T03:51:34.956 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:34.953+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:34.954395+0000) 2019-10-13T03:51:35.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:35.827+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:35.828338+0000) 2019-10-13T03:51:37.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:37.318+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:37.319186+0000) 2019-10-13T03:51:38.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:38.109+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.110006+0000) 2019-10-13T03:51:38.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:38.741+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.741646+0000) 2019-10-13T03:51:38.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:38.736+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.737082+0000) 2019-10-13T03:51:38.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:38.741+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.742053+0000) 2019-10-13T03:51:38.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:38.741+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.742295+0000) 2019-10-13T03:51:38.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:38.742+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.742568+0000) 2019-10-13T03:51:38.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:38.742+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.742764+0000) 2019-10-13T03:51:38.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:38.742+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.742941+0000) 2019-10-13T03:51:38.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:38.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:38.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:38.739+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.739894+0000) 2019-10-13T03:51:38.920 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:38.913+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:38.913603+0000) 2019-10-13T03:51:40.714 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:51:40.714 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:51:40.718 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:51:40.719 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:51:42.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:42.578+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:42.578197+0000) 2019-10-13T03:51:43.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:43.744+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.745070+0000) 2019-10-13T03:51:43.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:43.744+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.745360+0000) 2019-10-13T03:51:43.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:43.745+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.745440+0000) 2019-10-13T03:51:43.748 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:43.745+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.745681+0000) 2019-10-13T03:51:43.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:43.745+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.745885+0000) 2019-10-13T03:51:43.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:43.738+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.740520+0000) 2019-10-13T03:51:43.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:43.745+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.745992+0000) 2019-10-13T03:51:43.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:43.746+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:43.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:43.742+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:43.744240+0000) 2019-10-13T03:51:44.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:44.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:44.954625+0000) 2019-10-13T03:51:44.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:44.954+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:44.954625+0000) 2019-10-13T03:51:44.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:44.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:44.954621+0000) 2019-10-13T03:51:45.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:45.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:45.828558+0000) 2019-10-13T03:51:47.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:47.319+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:47.319367+0000) 2019-10-13T03:51:48.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:48.110+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.110183+0000) 2019-10-13T03:51:48.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:48.741+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.743283+0000) 2019-10-13T03:51:48.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:48.748+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.748666+0000) 2019-10-13T03:51:48.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:48.748+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.748800+0000) 2019-10-13T03:51:48.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:48.748+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.748924+0000) 2019-10-13T03:51:48.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:48.748+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.749185+0000) 2019-10-13T03:51:48.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:48.749+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.749490+0000) 2019-10-13T03:51:48.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:48.749+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.749628+0000) 2019-10-13T03:51:48.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:48.743+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.745608+0000) 2019-10-13T03:51:48.752 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:48.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43177 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:48.922 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:48.914+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:48.915567+0000) 2019-10-13T03:51:52.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:52.577+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:52.578384+0000) 2019-10-13T03:51:53.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:53.745+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.746750+0000) 2019-10-13T03:51:53.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:53.751+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.751974+0000) 2019-10-13T03:51:53.758 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:53.751+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.752060+0000) 2019-10-13T03:51:53.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:53.752+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.752479+0000) 2019-10-13T03:51:53.759 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:53.752+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.752685+0000) 2019-10-13T03:51:53.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:53.752+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.752835+0000) 2019-10-13T03:51:53.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:53.752+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.753279+0000) 2019-10-13T03:51:53.760 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:53.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43204 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:53.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:53.748+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:53.750472+0000) 2019-10-13T03:51:54.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:54.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:54.954817+0000) 2019-10-13T03:51:54.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:54.953+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:54.954817+0000) 2019-10-13T03:51:54.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:54.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:54.954816+0000) 2019-10-13T03:51:55.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:55.827+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:55.828732+0000) 2019-10-13T03:51:57.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:57.318+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:57.319520+0000) 2019-10-13T03:51:58.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:58.109+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.110384+0000) 2019-10-13T03:51:58.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:51:58.754+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.755546+0000) 2019-10-13T03:51:58.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:51:58.754+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.755791+0000) 2019-10-13T03:51:58.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:51:58.754+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.755810+0000) 2019-10-13T03:51:58.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:51:58.754+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.755958+0000) 2019-10-13T03:51:58.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:51:58.750+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.751065+0000) 2019-10-13T03:51:58.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:58.751+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.752154+0000) 2019-10-13T03:51:58.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:51:58.755+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.756462+0000) 2019-10-13T03:51:58.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:51:58.755+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.756642+0000) 2019-10-13T03:51:58.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:51:58.755+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43236 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:51:58.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:51:58.916+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:51:58.917678+0000) 2019-10-13T03:52:02.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:02.577+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:02.578563+0000) 2019-10-13T03:52:03.759 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:03.756+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.758315+0000) 2019-10-13T03:52:03.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:03.757+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.758938+0000) 2019-10-13T03:52:03.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:03.757+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.759139+0000) 2019-10-13T03:52:03.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:03.757+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.759236+0000) 2019-10-13T03:52:03.761 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:03.758+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.759391+0000) 2019-10-13T03:52:03.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:03.758+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.759509+0000) 2019-10-13T03:52:03.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:03.754+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.754728+0000) 2019-10-13T03:52:03.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:03.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43263 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:03.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:03.755+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:03.756647+0000) 2019-10-13T03:52:04.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:04.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:04.955049+0000) 2019-10-13T03:52:04.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:04.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:04.955028+0000) 2019-10-13T03:52:04.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:04.954+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:04.955049+0000) 2019-10-13T03:52:05.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:05.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:05.828924+0000) 2019-10-13T03:52:07.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:07.318+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:07.319672+0000) 2019-10-13T03:52:08.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:08.109+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.110609+0000) 2019-10-13T03:52:08.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:08.759+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.761245+0000) 2019-10-13T03:52:08.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:08.756+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.757703+0000) 2019-10-13T03:52:08.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:08.760+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.762390+0000) 2019-10-13T03:52:08.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:08.761+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.762453+0000) 2019-10-13T03:52:08.766 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:08.761+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.762713+0000) 2019-10-13T03:52:08.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:08.761+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.763056+0000) 2019-10-13T03:52:08.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:08.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:08.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:08.761+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.762554+0000) 2019-10-13T03:52:08.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:08.759+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.760254+0000) 2019-10-13T03:52:08.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:08.919+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:08.919996+0000) 2019-10-13T03:52:10.894 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:52:10.894 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:52:10.899 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:52:10.899 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:52:12.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:12.578+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:12.578717+0000) 2019-10-13T03:52:13.766 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:13.763+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.764778+0000) 2019-10-13T03:52:13.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:13.763+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.765115+0000) 2019-10-13T03:52:13.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:13.763+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.765260+0000) 2019-10-13T03:52:13.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:13.760+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.760999+0000) 2019-10-13T03:52:13.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:13.764+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.765439+0000) 2019-10-13T03:52:13.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:13.765+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:13.768 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:13.765+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.766596+0000) 2019-10-13T03:52:13.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:13.765+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.766595+0000) 2019-10-13T03:52:13.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:13.762+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:13.763133+0000) 2019-10-13T03:52:14.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:14.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:14.955245+0000) 2019-10-13T03:52:14.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:14.954+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:14.955246+0000) 2019-10-13T03:52:14.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:14.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:14.955245+0000) 2019-10-13T03:52:15.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:15.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:15.829098+0000) 2019-10-13T03:52:17.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:17.319+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:17.319887+0000) 2019-10-13T03:52:18.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:18.110+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.110815+0000) 2019-10-13T03:52:18.770 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:18.767+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.768600+0000) 2019-10-13T03:52:18.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:18.767+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.768893+0000) 2019-10-13T03:52:18.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:18.767+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.769009+0000) 2019-10-13T03:52:18.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:18.767+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.769162+0000) 2019-10-13T03:52:18.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:18.768+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.769423+0000) 2019-10-13T03:52:18.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:18.763+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.764639+0000) 2019-10-13T03:52:18.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:18.765+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.765810+0000) 2019-10-13T03:52:18.772 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:18.768+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.770065+0000) 2019-10-13T03:52:18.773 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:18.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43356 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:18.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:18.921+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:18.922545+0000) 2019-10-13T03:52:22.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:22.578+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:22.578907+0000) 2019-10-13T03:52:23.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:23.771+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.772170+0000) 2019-10-13T03:52:23.774 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:23.771+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.772364+0000) 2019-10-13T03:52:23.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:23.766+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.767412+0000) 2019-10-13T03:52:23.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:23.772+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.772911+0000) 2019-10-13T03:52:23.775 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:23.772+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.773115+0000) 2019-10-13T03:52:23.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:23.772+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.773223+0000) 2019-10-13T03:52:23.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:23.773+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.773498+0000) 2019-10-13T03:52:23.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:23.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43383 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:23.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:23.770+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:23.770805+0000) 2019-10-13T03:52:24.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:24.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:24.955471+0000) 2019-10-13T03:52:24.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:24.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:24.955474+0000) 2019-10-13T03:52:24.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:24.955+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:24.955474+0000) 2019-10-13T03:52:25.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:25.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:25.829269+0000) 2019-10-13T03:52:27.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:27.319+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:27.320057+0000) 2019-10-13T03:52:28.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:28.110+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.111005+0000) 2019-10-13T03:52:28.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:28.769+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.770328+0000) 2019-10-13T03:52:28.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:28.775+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.775725+0000) 2019-10-13T03:52:28.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:28.775+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.775832+0000) 2019-10-13T03:52:28.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:28.775+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.776237+0000) 2019-10-13T03:52:28.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:28.775+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.776208+0000) 2019-10-13T03:52:28.778 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:28.776+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.776830+0000) 2019-10-13T03:52:28.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:28.776+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.776602+0000) 2019-10-13T03:52:28.779 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:28.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43417 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:28.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:28.771+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.772680+0000) 2019-10-13T03:52:28.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:28.924+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:28.924728+0000) 2019-10-13T03:52:32.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:32.578+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:32.579101+0000) 2019-10-13T03:52:33.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:33.778+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.778747+0000) 2019-10-13T03:52:33.780 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:33.778+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.778859+0000) 2019-10-13T03:52:33.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:33.778+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.779188+0000) 2019-10-13T03:52:33.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:33.778+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.779287+0000) 2019-10-13T03:52:33.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:33.772+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.774274+0000) 2019-10-13T03:52:33.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:33.773+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.775409+0000) 2019-10-13T03:52:33.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:33.779+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.779687+0000) 2019-10-13T03:52:33.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:33.779+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:33.780049+0000) 2019-10-13T03:52:33.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:33.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43444 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:34.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:34.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:34.955663+0000) 2019-10-13T03:52:34.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:34.955+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:34.955662+0000) 2019-10-13T03:52:34.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:34.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:34.955657+0000) 2019-10-13T03:52:35.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:35.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:35.829476+0000) 2019-10-13T03:52:37.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:37.320+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:37.320228+0000) 2019-10-13T03:52:38.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:38.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.111186+0000) 2019-10-13T03:52:38.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:38.775+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.776831+0000) 2019-10-13T03:52:38.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:38.781+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.782256+0000) 2019-10-13T03:52:38.784 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:38.782+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.782702+0000) 2019-10-13T03:52:38.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:38.782+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.782719+0000) 2019-10-13T03:52:38.785 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:38.782+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.783040+0000) 2019-10-13T03:52:38.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:38.782+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.783163+0000) 2019-10-13T03:52:38.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:38.777+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.779415+0000) 2019-10-13T03:52:38.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:38.783+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.783601+0000) 2019-10-13T03:52:38.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:38.783+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43476 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:38.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:38.926+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:38.926974+0000) 2019-10-13T03:52:41.055 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:52:41.055 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:52:41.060 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:52:41.060 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:52:42.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:42.578+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:42.579288+0000) 2019-10-13T03:52:43.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:43.779+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.780539+0000) 2019-10-13T03:52:43.788 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:43.784+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.785548+0000) 2019-10-13T03:52:43.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:43.784+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.785831+0000) 2019-10-13T03:52:43.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:43.784+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.786014+0000) 2019-10-13T03:52:43.789 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:43.784+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.786281+0000) 2019-10-13T03:52:43.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:43.784+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.786365+0000) 2019-10-13T03:52:43.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:43.785+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.786572+0000) 2019-10-13T03:52:43.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:43.782+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:43.782902+0000) 2019-10-13T03:52:43.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:43.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:44.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:44.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:44.955814+0000) 2019-10-13T03:52:44.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:44.954+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:44.955813+0000) 2019-10-13T03:52:44.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:44.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:44.955816+0000) 2019-10-13T03:52:45.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:45.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:45.829674+0000) 2019-10-13T03:52:47.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:47.319+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:47.320454+0000) 2019-10-13T03:52:48.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:48.109+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.111375+0000) 2019-10-13T03:52:48.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:48.787+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.788807+0000) 2019-10-13T03:52:48.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:48.783+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.783971+0000) 2019-10-13T03:52:48.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:48.787+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.789281+0000) 2019-10-13T03:52:48.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:48.788+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.789420+0000) 2019-10-13T03:52:48.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:48.788+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.789955+0000) 2019-10-13T03:52:48.792 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:48.788+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.790078+0000) 2019-10-13T03:52:48.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:48.788+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.790259+0000) 2019-10-13T03:52:48.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:48.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:48.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:48.786+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.786738+0000) 2019-10-13T03:52:48.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:48.928+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:48.929005+0000) 2019-10-13T03:52:52.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:52.578+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:52.579477+0000) 2019-10-13T03:52:53.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:53.790+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.792355+0000) 2019-10-13T03:52:53.796 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:53.791+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.792534+0000) 2019-10-13T03:52:53.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:53.791+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.792705+0000) 2019-10-13T03:52:53.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:53.791+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.792861+0000) 2019-10-13T03:52:53.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:53.791+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.793155+0000) 2019-10-13T03:52:53.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:53.792+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.793707+0000) 2019-10-13T03:52:53.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:53.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:53.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:53.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.788177+0000) 2019-10-13T03:52:53.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:53.790+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:53.791174+0000) 2019-10-13T03:52:54.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:54.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:54.956025+0000) 2019-10-13T03:52:54.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:54.955+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:54.956025+0000) 2019-10-13T03:52:54.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:54.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:54.956024+0000) 2019-10-13T03:52:55.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:55.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:55.829858+0000) 2019-10-13T03:52:57.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:57.320+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:57.320665+0000) 2019-10-13T03:52:58.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:58.110+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.111573+0000) 2019-10-13T03:52:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 124 2019-10-13T03:52:58.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:52:58.793+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.795258+0000) 2019-10-13T03:52:58.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:52:58.794+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.795594+0000) 2019-10-13T03:52:58.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:52:58.794+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.795616+0000) 2019-10-13T03:52:58.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:52:58.789+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.790715+0000) 2019-10-13T03:52:58.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:52:58.794+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.795890+0000) 2019-10-13T03:52:58.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:52:58.794+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.796210+0000) 2019-10-13T03:52:58.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:52:58.795+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.796496+0000) 2019-10-13T03:52:58.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:52:58.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:52:58.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:58.792+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.793142+0000) 2019-10-13T03:52:58.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:52:58.930+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:52:58.931274+0000) 2019-10-13T03:53:02.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:02.579+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:02.579659+0000) 2019-10-13T03:53:03.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:03.796+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.798039+0000) 2019-10-13T03:53:03.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:03.796+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.798209+0000) 2019-10-13T03:53:03.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:03.792+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.793186+0000) 2019-10-13T03:53:03.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:03.797+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.798843+0000) 2019-10-13T03:53:03.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:03.797+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.799109+0000) 2019-10-13T03:53:03.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:03.797+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.799261+0000) 2019-10-13T03:53:03.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:03.798+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.799540+0000) 2019-10-13T03:53:03.802 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:03.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:03.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:03.797+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:03.797791+0000) 2019-10-13T03:53:04.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:04.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:04.956194+0000) 2019-10-13T03:53:04.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:04.955+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:04.956194+0000) 2019-10-13T03:53:04.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:04.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:04.956214+0000) 2019-10-13T03:53:05.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:05.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:05.830037+0000) 2019-10-13T03:53:07.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:07.320+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:07.320809+0000) 2019-10-13T03:53:08.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:08.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.111763+0000) 2019-10-13T03:53:08.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:08.801+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.801668+0000) 2019-10-13T03:53:08.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:08.801+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.801876+0000) 2019-10-13T03:53:08.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:08.796+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.797238+0000) 2019-10-13T03:53:08.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:08.801+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.802220+0000) 2019-10-13T03:53:08.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:08.802+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.802579+0000) 2019-10-13T03:53:08.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:08.802+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.802847+0000) 2019-10-13T03:53:08.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:08.802+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.803155+0000) 2019-10-13T03:53:08.805 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:08.803+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43657 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:08.806 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:08.798+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.799686+0000) 2019-10-13T03:53:08.940 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:08.932+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:08.933590+0000) 2019-10-13T03:53:11.163 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:53:11.163 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:53:11.167 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:53:11.167 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:53:12.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:12.579+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:12.579835+0000) 2019-10-13T03:53:13.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:13.804+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.805365+0000) 2019-10-13T03:53:13.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:13.805+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.805526+0000) 2019-10-13T03:53:13.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:13.805+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.805652+0000) 2019-10-13T03:53:13.808 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:13.805+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.805786+0000) 2019-10-13T03:53:13.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:13.805+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.806034+0000) 2019-10-13T03:53:13.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:13.800+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.800902+0000) 2019-10-13T03:53:13.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:13.806+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.806394+0000) 2019-10-13T03:53:13.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:13.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:13.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:13.802+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:13.803225+0000) 2019-10-13T03:53:14.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:14.956+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:14.956421+0000) 2019-10-13T03:53:14.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:14.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:14.956394+0000) 2019-10-13T03:53:14.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:14.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:14.956421+0000) 2019-10-13T03:53:15.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:15.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:15.830243+0000) 2019-10-13T03:53:17.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:17.320+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:17.321011+0000) 2019-10-13T03:53:18.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:18.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.111938+0000) 2019-10-13T03:53:18.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:18.807+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.808170+0000) 2019-10-13T03:53:18.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:18.808+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.808512+0000) 2019-10-13T03:53:18.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:18.801+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.803516+0000) 2019-10-13T03:53:18.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:18.808+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.808936+0000) 2019-10-13T03:53:18.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:18.808+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.809184+0000) 2019-10-13T03:53:18.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:18.808+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.809389+0000) 2019-10-13T03:53:18.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:18.809+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.809582+0000) 2019-10-13T03:53:18.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:18.804+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.805956+0000) 2019-10-13T03:53:18.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:18.809+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43716 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:18.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:18.934+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:18.935821+0000) 2019-10-13T03:53:22.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:22.579+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:22.580023+0000) 2019-10-13T03:53:23.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:23.811+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.811769+0000) 2019-10-13T03:53:23.815 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:23.811+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.811816+0000) 2019-10-13T03:53:23.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:23.811+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.812144+0000) 2019-10-13T03:53:23.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:23.812+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.812465+0000) 2019-10-13T03:53:23.816 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:23.812+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.812642+0000) 2019-10-13T03:53:23.816 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:23.812+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.812940+0000) 2019-10-13T03:53:23.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:23.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:23.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:23.805+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.807567+0000) 2019-10-13T03:53:23.818 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:23.807+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:23.808957+0000) 2019-10-13T03:53:24.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:24.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:24.956583+0000) 2019-10-13T03:53:24.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:24.955+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:24.956583+0000) 2019-10-13T03:53:24.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:24.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:24.956579+0000) 2019-10-13T03:53:25.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:25.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:25.830428+0000) 2019-10-13T03:53:27.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:27.320+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:27.321167+0000) 2019-10-13T03:53:28.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:28.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.112073+0000) 2019-10-13T03:53:28.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:28.808+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.809809+0000) 2019-10-13T03:53:28.817 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:28.813+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.815245+0000) 2019-10-13T03:53:28.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:28.813+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.815388+0000) 2019-10-13T03:53:28.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:28.814+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.815507+0000) 2019-10-13T03:53:28.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:28.814+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.815751+0000) 2019-10-13T03:53:28.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:28.814+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.815818+0000) 2019-10-13T03:53:28.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:28.814+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.816344+0000) 2019-10-13T03:53:28.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:28.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:28.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:28.811+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.812896+0000) 2019-10-13T03:53:28.944 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:28.937+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:28.937869+0000) 2019-10-13T03:53:32.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:32.579+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:32.580303+0000) 2019-10-13T03:53:33.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:33.816+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.818280+0000) 2019-10-13T03:53:33.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:33.817+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.818605+0000) 2019-10-13T03:53:33.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:33.817+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.818959+0000) 2019-10-13T03:53:33.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:33.817+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.819165+0000) 2019-10-13T03:53:33.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:33.812+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.813546+0000) 2019-10-13T03:53:33.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:33.817+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.819268+0000) 2019-10-13T03:53:33.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:33.818+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.819779+0000) 2019-10-13T03:53:33.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:33.818+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:33.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:33.815+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:33.816481+0000) 2019-10-13T03:53:34.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:34.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:34.956799+0000) 2019-10-13T03:53:34.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:34.955+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:34.956799+0000) 2019-10-13T03:53:34.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:34.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:34.956801+0000) 2019-10-13T03:53:35.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:35.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:35.830608+0000) 2019-10-13T03:53:37.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:37.320+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:37.321346+0000) 2019-10-13T03:53:38.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:38.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.112268+0000) 2019-10-13T03:53:38.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:38.815+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.816551+0000) 2019-10-13T03:53:38.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:38.820+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.821883+0000) 2019-10-13T03:53:38.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:38.820+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.822200+0000) 2019-10-13T03:53:38.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:38.820+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.822294+0000) 2019-10-13T03:53:38.825 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:38.821+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.822809+0000) 2019-10-13T03:53:38.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:38.821+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.823002+0000) 2019-10-13T03:53:38.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:38.821+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.822645+0000) 2019-10-13T03:53:38.826 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:38.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:38.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:38.818+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.819499+0000) 2019-10-13T03:53:38.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:38.939+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:38.939983+0000) 2019-10-13T03:53:41.352 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:53:41.352 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:53:41.356 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:53:41.357 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:53:42.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:42.579+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:42.580518+0000) 2019-10-13T03:53:43.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:43.823+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.825146+0000) 2019-10-13T03:53:43.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:43.823+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.825297+0000) 2019-10-13T03:53:43.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:43.819+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.820245+0000) 2019-10-13T03:53:43.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:43.824+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.825800+0000) 2019-10-13T03:53:43.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:43.824+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.826182+0000) 2019-10-13T03:53:43.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:43.824+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.826339+0000) 2019-10-13T03:53:43.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:43.825+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.826515+0000) 2019-10-13T03:53:43.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:43.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:43.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:43.821+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:43.822593+0000) 2019-10-13T03:53:44.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:44.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:44.956983+0000) 2019-10-13T03:53:44.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:44.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:44.956981+0000) 2019-10-13T03:53:44.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:44.956+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:44.956982+0000) 2019-10-13T03:53:45.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:45.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:45.830835+0000) 2019-10-13T03:53:47.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:47.321+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:47.321559+0000) 2019-10-13T03:53:48.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:48.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.112461+0000) 2019-10-13T03:53:48.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:48.827+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.828669+0000) 2019-10-13T03:53:48.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:48.822+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.823672+0000) 2019-10-13T03:53:48.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:48.827+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.829267+0000) 2019-10-13T03:53:48.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:48.827+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.829314+0000) 2019-10-13T03:53:48.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:48.828+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.829593+0000) 2019-10-13T03:53:48.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:48.828+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.829753+0000) 2019-10-13T03:53:48.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:48.828+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.830064+0000) 2019-10-13T03:53:48.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:48.825+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.825863+0000) 2019-10-13T03:53:48.833 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:48.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:48.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:48.941+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:48.942324+0000) 2019-10-13T03:53:52.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:52.580+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:52.580689+0000) 2019-10-13T03:53:53.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:53.831+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.832162+0000) 2019-10-13T03:53:53.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:53.832+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.832604+0000) 2019-10-13T03:53:53.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:53.832+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.832753+0000) 2019-10-13T03:53:53.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:53.832+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.833208+0000) 2019-10-13T03:53:53.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:53.833+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.833402+0000) 2019-10-13T03:53:53.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:53.827+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.827839+0000) 2019-10-13T03:53:53.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:53.833+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.833542+0000) 2019-10-13T03:53:53.836 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:53.833+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:53.836 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:53.829+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:53.829816+0000) 2019-10-13T03:53:54.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:54.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:54.957161+0000) 2019-10-13T03:53:54.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:54.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:54.957162+0000) 2019-10-13T03:53:54.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:54.956+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:54.957162+0000) 2019-10-13T03:53:55.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:55.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:55.831047+0000) 2019-10-13T03:53:57.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:57.321+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:57.321732+0000) 2019-10-13T03:53:58.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:58.112+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.112640+0000) 2019-10-13T03:53:58.497 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2019-10-13T03:53:58.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:53:58.835+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.835571+0000) 2019-10-13T03:53:58.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:53:58.835+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.835674+0000) 2019-10-13T03:53:58.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:53:58.835+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.835817+0000) 2019-10-13T03:53:58.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:53:58.835+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.836055+0000) 2019-10-13T03:53:58.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:53:58.835+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.836215+0000) 2019-10-13T03:53:58.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:53:58.830+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.830782+0000) 2019-10-13T03:53:58.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:53:58.836+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.836460+0000) 2019-10-13T03:53:58.839 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:53:58.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:53:58.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:58.832+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.832859+0000) 2019-10-13T03:53:58.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:53:58.944+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:53:58.944383+0000) 2019-10-13T03:54:02.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:02.580+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:02.580866+0000) 2019-10-13T03:54:03.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:03.837+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.838037+0000) 2019-10-13T03:54:03.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:03.837+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.838327+0000) 2019-10-13T03:54:03.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:03.838+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.838631+0000) 2019-10-13T03:54:03.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:03.838+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.838759+0000) 2019-10-13T03:54:03.841 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:03.838+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.838820+0000) 2019-10-13T03:54:03.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:03.838+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.838942+0000) 2019-10-13T03:54:03.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:03.838+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 43983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:03.842 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:03.833+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.834116+0000) 2019-10-13T03:54:03.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:03.835+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:03.836285+0000) 2019-10-13T03:54:04.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:04.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:04.957368+0000) 2019-10-13T03:54:04.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:04.957+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:04.957368+0000) 2019-10-13T03:54:04.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:04.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:04.957368+0000) 2019-10-13T03:54:05.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:05.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:05.831248+0000) 2019-10-13T03:54:07.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:07.321+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:07.321910+0000) 2019-10-13T03:54:08.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:08.112+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.112831+0000) 2019-10-13T03:54:08.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:08.840+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.841031+0000) 2019-10-13T03:54:08.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:08.841+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.841488+0000) 2019-10-13T03:54:08.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:08.840+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.841265+0000) 2019-10-13T03:54:08.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:08.841+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.841783+0000) 2019-10-13T03:54:08.843 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:08.841+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.842001+0000) 2019-10-13T03:54:08.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:08.835+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.837012+0000) 2019-10-13T03:54:08.844 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:08.842+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.842464+0000) 2019-10-13T03:54:08.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:08.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:08.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:08.837+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.839339+0000) 2019-10-13T03:54:08.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:08.945+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:08.946391+0000) 2019-10-13T03:54:11.552 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:54:11.553 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:54:11.567 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:54:11.567 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:54:12.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:12.580+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:12.581009+0000) 2019-10-13T03:54:13.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:13.843+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.844439+0000) 2019-10-13T03:54:13.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:13.843+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.844726+0000) 2019-10-13T03:54:13.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:13.843+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.844834+0000) 2019-10-13T03:54:13.847 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:13.843+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.844597+0000) 2019-10-13T03:54:13.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:13.843+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.844965+0000) 2019-10-13T03:54:13.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:13.843+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.845375+0000) 2019-10-13T03:54:13.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:13.838+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.839801+0000) 2019-10-13T03:54:13.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:13.844+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:13.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:13.840+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:13.842035+0000) 2019-10-13T03:54:14.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:14.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:14.957579+0000) 2019-10-13T03:54:14.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:14.956+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:14.957580+0000) 2019-10-13T03:54:14.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:14.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:14.957574+0000) 2019-10-13T03:54:15.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:15.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:15.831394+0000) 2019-10-13T03:54:17.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:17.321+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:17.322098+0000) 2019-10-13T03:54:18.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:18.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.112978+0000) 2019-10-13T03:54:18.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:18.845+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.847087+0000) 2019-10-13T03:54:18.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:18.845+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.847153+0000) 2019-10-13T03:54:18.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:18.841+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.842319+0000) 2019-10-13T03:54:18.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:18.846+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.847799+0000) 2019-10-13T03:54:18.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:18.846+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.847937+0000) 2019-10-13T03:54:18.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:18.846+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.848301+0000) 2019-10-13T03:54:18.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:18.847+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.848459+0000) 2019-10-13T03:54:18.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:18.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:18.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:18.843+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.844097+0000) 2019-10-13T03:54:18.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:18.947+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:18.948457+0000) 2019-10-13T03:54:22.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:22.580+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:22.581190+0000) 2019-10-13T03:54:23.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:23.844+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.845560+0000) 2019-10-13T03:54:23.853 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:23.849+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.850722+0000) 2019-10-13T03:54:23.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:23.849+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.851005+0000) 2019-10-13T03:54:23.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:23.849+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.851340+0000) 2019-10-13T03:54:23.853 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:23.850+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.851490+0000) 2019-10-13T03:54:23.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:23.850+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.851997+0000) 2019-10-13T03:54:23.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:23.850+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.852058+0000) 2019-10-13T03:54:23.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:23.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:23.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:23.848+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:23.849233+0000) 2019-10-13T03:54:24.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:24.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:24.957763+0000) 2019-10-13T03:54:24.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:24.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:24.957785+0000) 2019-10-13T03:54:24.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:24.957+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:24.957786+0000) 2019-10-13T03:54:25.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:25.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:25.831550+0000) 2019-10-13T03:54:27.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:27.321+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:27.322287+0000) 2019-10-13T03:54:28.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:28.111+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.113140+0000) 2019-10-13T03:54:28.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:28.847+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.848509+0000) 2019-10-13T03:54:28.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:28.852+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.853980+0000) 2019-10-13T03:54:28.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:28.852+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.854138+0000) 2019-10-13T03:54:28.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:28.852+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.854284+0000) 2019-10-13T03:54:28.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:28.852+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.854368+0000) 2019-10-13T03:54:28.857 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:28.853+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.854744+0000) 2019-10-13T03:54:28.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:28.853+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.854965+0000) 2019-10-13T03:54:28.857 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:28.854+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:28.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:28.851+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.851826+0000) 2019-10-13T03:54:28.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:28.950+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:28.950940+0000) 2019-10-13T03:54:32.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:32.580+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:32.581337+0000) 2019-10-13T03:54:33.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:33.855+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.857247+0000) 2019-10-13T03:54:33.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:33.856+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.857653+0000) 2019-10-13T03:54:33.859 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:33.856+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.857717+0000) 2019-10-13T03:54:33.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:33.852+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.852921+0000) 2019-10-13T03:54:33.860 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:33.856+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.858335+0000) 2019-10-13T03:54:33.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:33.857+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.858486+0000) 2019-10-13T03:54:33.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:33.857+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.858640+0000) 2019-10-13T03:54:33.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:33.857+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:33.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:33.854+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:33.855094+0000) 2019-10-13T03:54:34.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:34.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:34.957985+0000) 2019-10-13T03:54:34.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:34.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:34.958006+0000) 2019-10-13T03:54:34.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:34.957+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:34.958005+0000) 2019-10-13T03:54:35.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:35.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:35.831678+0000) 2019-10-13T03:54:37.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:37.322+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:37.322480+0000) 2019-10-13T03:54:38.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:38.112+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.113314+0000) 2019-10-13T03:54:38.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:38.859+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.860307+0000) 2019-10-13T03:54:38.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:38.860+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.860634+0000) 2019-10-13T03:54:38.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:38.860+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.860727+0000) 2019-10-13T03:54:38.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:38.855+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.855790+0000) 2019-10-13T03:54:38.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:38.860+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.861223+0000) 2019-10-13T03:54:38.864 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:38.860+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.860891+0000) 2019-10-13T03:54:38.864 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:38.861+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.861586+0000) 2019-10-13T03:54:38.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:38.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44197 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:38.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:38.856+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.857266+0000) 2019-10-13T03:54:38.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:38.952+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:38.953029+0000) 2019-10-13T03:54:41.737 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:54:41.737 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:54:41.741 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:54:41.742 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:54:42.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:42.581+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:42.581508+0000) 2019-10-13T03:54:43.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:43.863+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.863463+0000) 2019-10-13T03:54:43.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:43.863+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.863779+0000) 2019-10-13T03:54:43.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:43.863+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.863979+0000) 2019-10-13T03:54:43.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:43.863+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.864304+0000) 2019-10-13T03:54:43.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:43.858+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.859476+0000) 2019-10-13T03:54:43.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:43.864+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.864425+0000) 2019-10-13T03:54:43.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:43.864+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.864731+0000) 2019-10-13T03:54:43.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:43.865+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:43.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:43.861+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:43.861739+0000) 2019-10-13T03:54:44.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:44.957+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:44.958200+0000) 2019-10-13T03:54:44.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:44.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:44.958176+0000) 2019-10-13T03:54:44.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:44.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:44.958198+0000) 2019-10-13T03:54:45.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:45.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:45.831843+0000) 2019-10-13T03:54:47.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:47.322+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:47.322646+0000) 2019-10-13T03:54:48.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:48.113+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.113471+0000) 2019-10-13T03:54:48.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:48.861+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.861815+0000) 2019-10-13T03:54:48.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:48.867+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.867448+0000) 2019-10-13T03:54:48.869 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:48.867+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.867674+0000) 2019-10-13T03:54:48.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:48.867+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.867747+0000) 2019-10-13T03:54:48.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:48.867+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.868152+0000) 2019-10-13T03:54:48.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:48.868+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.868547+0000) 2019-10-13T03:54:48.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:48.868+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.868565+0000) 2019-10-13T03:54:48.871 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:48.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:48.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:48.865+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.865901+0000) 2019-10-13T03:54:48.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:48.955+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:48.955095+0000) 2019-10-13T03:54:52.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:52.580+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:52.581753+0000) 2019-10-13T03:54:53.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:53.870+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.870714+0000) 2019-10-13T03:54:53.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:53.864+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.866167+0000) 2019-10-13T03:54:53.873 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:53.870+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.871271+0000) 2019-10-13T03:54:53.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:53.871+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.871883+0000) 2019-10-13T03:54:53.873 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:53.871+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.872090+0000) 2019-10-13T03:54:53.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:53.871+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.872288+0000) 2019-10-13T03:54:53.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:53.872+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.872544+0000) 2019-10-13T03:54:53.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:53.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:53.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:53.866+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:53.868564+0000) 2019-10-13T03:54:54.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:54.957+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:54.958367+0000) 2019-10-13T03:54:54.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:54.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:54.958366+0000) 2019-10-13T03:54:54.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:54.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:54.958367+0000) 2019-10-13T03:54:55.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:55.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:55.831969+0000) 2019-10-13T03:54:57.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:57.321+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:57.322835+0000) 2019-10-13T03:54:58.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:58.112+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.113645+0000) 2019-10-13T03:54:58.498 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2019-10-13T03:54:58.498 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2019-10-13T03:54:58.498 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:54:58.498 INFO:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2019-10-13T03:54:58.844 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 136, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 409, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi155 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8ee8ac95ca822967c9dffe2a705f1d5502812bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2019-10-13T03:54:58.845 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2019-10-13T03:54:58.854 DEBUG:tasks.check_counter:Getting stats from a 2019-10-13T03:54:58.854 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:54:58.854 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump 2019-10-13T03:54:58.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:54:58.872+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.873943+0000) 2019-10-13T03:54:58.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:54:58.872+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.874123+0000) 2019-10-13T03:54:58.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:54:58.873+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.874483+0000) 2019-10-13T03:54:58.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:54:58.873+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.874420+0000) 2019-10-13T03:54:58.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:54:58.873+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.874988+0000) 2019-10-13T03:54:58.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:54:58.873+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.875089+0000) 2019-10-13T03:54:58.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:54:58.867+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.869566+0000) 2019-10-13T03:54:58.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:54:58.874+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:54:58.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:58.869+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.871446+0000) 2019-10-13T03:54:58.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:54:58.956+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:54:58.957086+0000) 2019-10-13T03:54:59.214 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2019-10-13T03:54:59.214 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2019-10-13T03:54:59.215 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 2045, 2019-10-13T03:54:59.215 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 3955, 2019-10-13T03:54:59.215 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 22601242, 2019-10-13T03:54:59.215 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 674485, 2019-10-13T03:54:59.215 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 2022, 2019-10-13T03:54:59.215 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 18446744073709551615, 2019-10-13T03:54:59.215 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.861155866, 2019-10-13T03:54:59.216 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.061410992, 2019-10-13T03:54:59.216 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.982421086, 2019-10-13T03:54:59.216 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2019-10-13T03:54:59.216 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T03:54:59.216 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2618, 2019-10-13T03:54:59.216 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 1.898647470, 2019-10-13T03:54:59.216 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000725228 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: } 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.217 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2019-10-13T03:54:59.218 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 9024, 2019-10-13T03:54:59.218 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 18035, 2019-10-13T03:54:59.218 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 23267254, 2019-10-13T03:54:59.218 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 435022193, 2019-10-13T03:54:59.218 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 8995, 2019-10-13T03:54:59.218 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 18446744073709551615, 2019-10-13T03:54:59.218 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 3.617026718, 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.528498966, 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 3.107111830, 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 16698, 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 7.081640595, 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000424101 2019-10-13T03:54:59.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2019-10-13T03:54:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2019-10-13T03:54:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: } 2019-10-13T03:54:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2019-10-13T03:54:59.221 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 5605, 2019-10-13T03:54:59.221 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 10901, 2019-10-13T03:54:59.221 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 22078448, 2019-10-13T03:54:59.221 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 213402933, 2019-10-13T03:54:59.221 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 5435, 2019-10-13T03:54:59.222 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 18446744073709551615, 2019-10-13T03:54:59.222 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 2.204787737, 2019-10-13T03:54:59.222 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.296207335, 2019-10-13T03:54:59.222 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 1.986288384, 2019-10-13T03:54:59.222 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.004450704, 2019-10-13T03:54:59.222 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T03:54:59.223 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 9536, 2019-10-13T03:54:59.224 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 4.243971921, 2019-10-13T03:54:59.224 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000445047 2019-10-13T03:54:59.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.224 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2019-10-13T03:54:59.224 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2019-10-13T03:54:59.224 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.225 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.225 INFO:teuthology.orchestra.run.smithi155.stdout: } 2019-10-13T03:54:59.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.225 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2019-10-13T03:54:59.225 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 0, 2019-10-13T03:54:59.226 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2019-10-13T03:54:59.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.226 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2019-10-13T03:54:59.226 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2019-10-13T03:54:59.226 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2019-10-13T03:54:59.226 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2019-10-13T03:54:59.226 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2019-10-13T03:54:59.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2019-10-13T03:54:59.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2019-10-13T03:54:59.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2019-10-13T03:54:59.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2019-10-13T03:54:59.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2019-10-13T03:54:59.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2019-10-13T03:54:59.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2019-10-13T03:54:59.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2019-10-13T03:54:59.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2019-10-13T03:54:59.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2019-10-13T03:54:59.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2019-10-13T03:54:59.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2019-10-13T03:54:59.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2019-10-13T03:54:59.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 78671, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 44, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2019-10-13T03:54:59.229 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2019-10-13T03:54:59.230 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2019-10-13T03:54:59.230 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2019-10-13T03:54:59.230 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2019-10-13T03:54:59.230 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2019-10-13T03:54:59.230 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 0, 2019-10-13T03:54:59.230 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 0, 2019-10-13T03:54:59.230 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2019-10-13T03:54:59.231 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2019-10-13T03:54:59.231 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2019-10-13T03:54:59.231 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2019-10-13T03:54:59.231 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 0, 2019-10-13T03:54:59.231 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 0, 2019-10-13T03:54:59.231 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2019-10-13T03:54:59.232 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2019-10-13T03:54:59.232 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2019-10-13T03:54:59.232 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2019-10-13T03:54:59.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2019-10-13T03:54:59.232 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2019-10-13T03:54:59.232 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2019-10-13T03:54:59.232 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2019-10-13T03:54:59.233 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2019-10-13T03:54:59.233 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 16670, 2019-10-13T03:54:59.233 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 66712265, 2019-10-13T03:54:59.233 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2019-10-13T03:54:59.233 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 16670, 2019-10-13T03:54:59.234 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2019-10-13T03:54:59.234 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2019-10-13T03:54:59.234 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 16670, 2019-10-13T03:54:59.234 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 66712265, 2019-10-13T03:54:59.234 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2019-10-13T03:54:59.235 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2019-10-13T03:54:59.235 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.235 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.235 INFO:teuthology.orchestra.run.smithi155.stdout: } 2019-10-13T03:54:59.235 INFO:teuthology.orchestra.run.smithi155.stdout: } 2019-10-13T03:54:59.235 INFO:teuthology.orchestra.run.smithi155.stdout:} 2019-10-13T03:54:59.236 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2019-10-13T03:54:59.236 DEBUG:tasks.check_counter:Getting stats from b 2019-10-13T03:54:59.237 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:54:59.237 INFO:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok perf dump 2019-10-13T03:54:59.443 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2019-10-13T03:54:59.443 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2019-10-13T03:54:59.443 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 7610, 2019-10-13T03:54:59.443 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 12450, 2019-10-13T03:54:59.444 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 28906696, 2019-10-13T03:54:59.444 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 7769760, 2019-10-13T03:54:59.444 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 2560, 2019-10-13T03:54:59.444 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 1, 2019-10-13T03:54:59.444 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 1.294998463, 2019-10-13T03:54:59.444 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.190454032, 2019-10-13T03:54:59.444 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 1.198593571, 2019-10-13T03:54:59.445 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.023316675, 2019-10-13T03:54:59.445 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T03:54:59.445 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 10733, 2019-10-13T03:54:59.445 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 2.744936384, 2019-10-13T03:54:59.445 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000255747 2019-10-13T03:54:59.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.445 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2019-10-13T03:54:59.446 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 9918, 2019-10-13T03:54:59.446 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.001581404, 2019-10-13T03:54:59.446 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000159 2019-10-13T03:54:59.446 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.446 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2019-10-13T03:54:59.446 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 3492, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 6292, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 29054368, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 7648027, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 2608, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 1, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 1.061572446, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.103469408, 2019-10-13T03:54:59.447 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 1.021582212, 2019-10-13T03:54:59.448 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.018570876, 2019-10-13T03:54:59.448 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T03:54:59.448 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 4585, 2019-10-13T03:54:59.448 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 2.272536225, 2019-10-13T03:54:59.448 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000495645 2019-10-13T03:54:59.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.448 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2019-10-13T03:54:59.449 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.449 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.449 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.449 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2019-10-13T03:54:59.449 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 1287, 2019-10-13T03:54:59.450 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 3661, 2019-10-13T03:54:59.450 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 10790662, 2019-10-13T03:54:59.450 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 6143046, 2019-10-13T03:54:59.450 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 962, 2019-10-13T03:54:59.450 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 0, 2019-10-13T03:54:59.450 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.472694249, 2019-10-13T03:54:59.451 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.096436635, 2019-10-13T03:54:59.451 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.379849180, 2019-10-13T03:54:59.451 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.004975937, 2019-10-13T03:54:59.451 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T03:54:59.451 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 3042, 2019-10-13T03:54:59.451 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.929875374, 2019-10-13T03:54:59.451 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000305678 2019-10-13T03:54:59.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.452 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2019-10-13T03:54:59.452 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.452 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.453 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.453 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2019-10-13T03:54:59.453 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2019-10-13T03:54:59.453 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2019-10-13T03:54:59.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.454 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2019-10-13T03:54:59.454 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2019-10-13T03:54:59.454 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2019-10-13T03:54:59.454 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 3073, 2019-10-13T03:54:59.454 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 1.055665591, 2019-10-13T03:54:59.455 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000343529 2019-10-13T03:54:59.455 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.455 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2019-10-13T03:54:59.455 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2019-10-13T03:54:59.455 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2019-10-13T03:54:59.455 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 311, 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.020621968, 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000066308 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 1094, 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 1094, 2019-10-13T03:54:59.456 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2019-10-13T03:54:59.457 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1094, 2019-10-13T03:54:59.457 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 1.731822599, 2019-10-13T03:54:59.457 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001583018 2019-10-13T03:54:59.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.457 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 0, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch": 0, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 21, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 1, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 0, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "inode_max": 2147483647, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 248, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 0, 2019-10-13T03:54:59.458 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 248, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 0, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 248, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 0, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 239, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 239, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 2, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 1679, 2019-10-13T03:54:59.459 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 1280, 2019-10-13T03:54:59.460 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 0, 2019-10-13T03:54:59.460 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 0, 2019-10-13T03:54:59.460 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 0, 2019-10-13T03:54:59.460 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2019-10-13T03:54:59.460 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 0, 2019-10-13T03:54:59.460 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 0, 2019-10-13T03:54:59.460 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2019-10-13T03:54:59.461 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 0, 2019-10-13T03:54:59.461 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 0, 2019-10-13T03:54:59.461 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 0, 2019-10-13T03:54:59.461 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 0, 2019-10-13T03:54:59.461 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2019-10-13T03:54:59.461 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 0, 2019-10-13T03:54:59.461 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0 2019-10-13T03:54:59.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.462 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2019-10-13T03:54:59.462 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 0, 2019-10-13T03:54:59.462 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2019-10-13T03:54:59.462 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2019-10-13T03:54:59.462 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 232, 2019-10-13T03:54:59.462 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 232, 2019-10-13T03:54:59.463 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2019-10-13T03:54:59.463 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2019-10-13T03:54:59.463 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2019-10-13T03:54:59.463 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 0, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 0, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 1, 2019-10-13T03:54:59.464 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2019-10-13T03:54:59.465 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2019-10-13T03:54:59.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.465 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2019-10-13T03:54:59.465 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 3285, 2019-10-13T03:54:59.465 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 0, 2019-10-13T03:54:59.465 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 0, 2019-10-13T03:54:59.465 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 3285, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 0, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 4, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 0, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 0, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 4, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2019-10-13T03:54:59.466 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 0, 2019-10-13T03:54:59.467 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 4194304, 2019-10-13T03:54:59.467 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 11739306, 2019-10-13T03:54:59.467 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 0, 2019-10-13T03:54:59.467 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2019-10-13T03:54:59.467 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1482, 2019-10-13T03:54:59.467 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 3.792249728, 2019-10-13T03:54:59.467 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.002558872 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 0 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 251, 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 483, 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 232, 2019-10-13T03:54:59.468 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 27, 2019-10-13T03:54:59.469 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 34, 2019-10-13T03:54:59.469 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 7, 2019-10-13T03:54:59.469 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 287, 2019-10-13T03:54:59.469 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 547, 2019-10-13T03:54:59.469 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 260, 2019-10-13T03:54:59.469 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 239, 2019-10-13T03:54:59.469 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 471, 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 232, 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 43628, 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 315572 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 1258, 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 0, 2019-10-13T03:54:59.470 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 1094, 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 46, 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_slave_request": 0, 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 452, 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.498208159, 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001102230 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.471 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2019-10-13T03:54:59.472 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 7, 2019-10-13T03:54:59.472 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.001790293, 2019-10-13T03:54:59.472 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000255756 2019-10-13T03:54:59.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.472 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2019-10-13T03:54:59.472 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.472 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.473 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.473 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2019-10-13T03:54:59.473 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.474 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.474 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.474 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2019-10-13T03:54:59.474 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 19, 2019-10-13T03:54:59.474 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.056543097, 2019-10-13T03:54:59.474 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.002975952 2019-10-13T03:54:59.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.475 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2019-10-13T03:54:59.475 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.475 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.475 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.475 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.476 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.477 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2019-10-13T03:54:59.477 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.477 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.477 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.478 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.479 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.479 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2019-10-13T03:54:59.479 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 14, 2019-10-13T03:54:59.479 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.015253403, 2019-10-13T03:54:59.479 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001089528 2019-10-13T03:54:59.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.480 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.481 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.481 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2019-10-13T03:54:59.481 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 235, 2019-10-13T03:54:59.481 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.420386844, 2019-10-13T03:54:59.481 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001788880 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 43, 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.330055816, 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.007675716 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2019-10-13T03:54:59.482 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 22, 2019-10-13T03:54:59.483 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.037817852, 2019-10-13T03:54:59.483 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001718993 2019-10-13T03:54:59.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.483 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2019-10-13T03:54:59.484 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.484 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.484 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.484 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2019-10-13T03:54:59.484 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 6, 2019-10-13T03:54:59.484 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.040293439, 2019-10-13T03:54:59.485 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.006715573 2019-10-13T03:54:59.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.485 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2019-10-13T03:54:59.485 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.485 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.485 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 84, 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.094830711, 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001128937 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1, 2019-10-13T03:54:59.486 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000507394, 2019-10-13T03:54:59.487 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000507394 2019-10-13T03:54:59.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.487 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2019-10-13T03:54:59.487 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.487 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.487 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.488 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.489 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.489 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2019-10-13T03:54:59.489 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.489 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.489 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.490 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2019-10-13T03:54:59.490 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 4, 2019-10-13T03:54:59.490 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.003739667, 2019-10-13T03:54:59.490 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000934916 2019-10-13T03:54:59.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.490 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2019-10-13T03:54:59.490 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 207, 2019-10-13T03:54:59.491 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.232395924, 2019-10-13T03:54:59.491 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001122685 2019-10-13T03:54:59.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.491 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0 2019-10-13T03:54:59.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.492 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2019-10-13T03:54:59.492 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 1, 2019-10-13T03:54:59.492 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 1, 2019-10-13T03:54:59.493 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 0, 2019-10-13T03:54:59.493 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 1, 2019-10-13T03:54:59.493 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2019-10-13T03:54:59.493 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 46, 2019-10-13T03:54:59.493 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 46, 2019-10-13T03:54:59.493 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 871 2019-10-13T03:54:59.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.494 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2019-10-13T03:54:59.494 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2019-10-13T03:54:59.494 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2019-10-13T03:54:59.494 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2019-10-13T03:54:59.495 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2019-10-13T03:54:59.496 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2019-10-13T03:54:59.496 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2019-10-13T03:54:59.496 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2019-10-13T03:54:59.496 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2019-10-13T03:54:59.496 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2019-10-13T03:54:59.496 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2019-10-13T03:54:59.496 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 135387, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 597, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2019-10-13T03:54:59.497 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2019-10-13T03:54:59.498 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2019-10-13T03:54:59.498 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2019-10-13T03:54:59.498 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2019-10-13T03:54:59.498 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2019-10-13T03:54:59.498 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 4872, 2019-10-13T03:54:59.498 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 76, 2019-10-13T03:54:59.498 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 761999, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 15168, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2019-10-13T03:54:59.499 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2019-10-13T03:54:59.500 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2019-10-13T03:54:59.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.500 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2019-10-13T03:54:59.500 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 2, 2019-10-13T03:54:59.500 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2019-10-13T03:54:59.500 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 1992, 2019-10-13T03:54:59.500 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 7721200, 2019-10-13T03:54:59.501 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 31, 2019-10-13T03:54:59.501 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 1992, 2019-10-13T03:54:59.501 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 1994, 2019-10-13T03:54:59.501 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 19, 2019-10-13T03:54:59.501 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 1975, 2019-10-13T03:54:59.501 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2019-10-13T03:54:59.501 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 0, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 11, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 19, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 1501, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 65, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2019-10-13T03:54:59.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 1, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 245, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 0, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 22, 2019-10-13T03:54:59.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2019-10-13T03:54:59.504 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2019-10-13T03:54:59.504 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2019-10-13T03:54:59.504 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2019-10-13T03:54:59.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2019-10-13T03:54:59.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2019-10-13T03:54:59.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2019-10-13T03:54:59.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2019-10-13T03:54:59.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2019-10-13T03:54:59.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2019-10-13T03:54:59.505 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2019-10-13T03:54:59.506 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2019-10-13T03:54:59.507 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 26, 2019-10-13T03:54:59.508 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2019-10-13T03:54:59.508 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 3, 2019-10-13T03:54:59.508 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 3, 2019-10-13T03:54:59.508 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 44, 2019-10-13T03:54:59.508 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 41, 2019-10-13T03:54:59.508 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2019-10-13T03:54:59.508 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 291, 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 0, 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 104 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 232, 2019-10-13T03:54:59.509 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2019-10-13T03:54:59.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.510 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2019-10-13T03:54:59.510 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2019-10-13T03:54:59.510 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2019-10-13T03:54:59.510 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2019-10-13T03:54:59.510 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 12385, 2019-10-13T03:54:59.510 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 67834137, 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 12385, 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 12385, 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 67834137, 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2019-10-13T03:54:59.511 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.512 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.512 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.512 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2019-10-13T03:54:59.512 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 180, 2019-10-13T03:54:59.512 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2019-10-13T03:54:59.513 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2019-10-13T03:54:59.513 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2019-10-13T03:54:59.513 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2019-10-13T03:54:59.513 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2019-10-13T03:54:59.513 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2019-10-13T03:54:59.513 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 1994, 2019-10-13T03:54:59.514 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 7744812, 2019-10-13T03:54:59.514 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1746, 2019-10-13T03:54:59.514 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 7744632, 2019-10-13T03:54:59.514 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2019-10-13T03:54:59.515 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.515 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.515 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.515 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2019-10-13T03:54:59.515 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 2, 2019-10-13T03:54:59.516 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2019-10-13T03:54:59.516 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2019-10-13T03:54:59.516 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2019-10-13T03:54:59.516 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2019-10-13T03:54:59.516 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2019-10-13T03:54:59.516 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2019-10-13T03:54:59.516 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 1994, 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 1994, 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1992, 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1992, 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.517 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.518 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2019-10-13T03:54:59.518 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2019-10-13T03:54:59.518 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2019-10-13T03:54:59.518 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2019-10-13T03:54:59.518 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 232, 2019-10-13T03:54:59.518 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 23432, 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 232, 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 19, 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 23432, 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2019-10-13T03:54:59.519 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.520 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.520 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.520 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2019-10-13T03:54:59.520 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x559179737b90": { 2019-10-13T03:54:59.520 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2019-10-13T03:54:59.520 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 3285, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 7545002, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 3285, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2019-10-13T03:54:59.521 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1482, 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 7545002, 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2019-10-13T03:54:59.522 INFO:teuthology.orchestra.run.smithi142.stdout:} 2019-10-13T03:54:59.524 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2019-10-13T03:54:59.525 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2019-10-13T03:54:59.538 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2019-10-13T03:54:59.539 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T03:54:59.539 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:54:59.539 INFO:teuthology.orchestra.run.smithi155:> stat --file-system '--printf=%T 2019-10-13T03:54:59.539 INFO:teuthology.orchestra.run.smithi155:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T03:55:02.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:02.581+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:02.581927+0000) 2019-10-13T03:55:03.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:03.875+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.876775+0000) 2019-10-13T03:55:03.878 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:03.875+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.876962+0000) 2019-10-13T03:55:03.879 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:03.875+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.877163+0000) 2019-10-13T03:55:03.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:03.870+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.871945+0000) 2019-10-13T03:55:03.880 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:03.876+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.877593+0000) 2019-10-13T03:55:03.880 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:03.876+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.877843+0000) 2019-10-13T03:55:03.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:03.876+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.878069+0000) 2019-10-13T03:55:03.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:03.877+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:03.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:03.872+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:03.874625+0000) 2019-10-13T03:55:04.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:04.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:04.958580+0000) 2019-10-13T03:55:04.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:04.957+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:04.958581+0000) 2019-10-13T03:55:04.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:04.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:04.958576+0000) 2019-10-13T03:55:05.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:05.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:05.832181+0000) 2019-10-13T03:55:07.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:07.322+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:07.323020+0000) 2019-10-13T03:55:08.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:08.113+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.113850+0000) 2019-10-13T03:55:08.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:08.878+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.879817+0000) 2019-10-13T03:55:08.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:08.878+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.879962+0000) 2019-10-13T03:55:08.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:08.878+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.880156+0000) 2019-10-13T03:55:08.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:08.878+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.880231+0000) 2019-10-13T03:55:08.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:08.879+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.880456+0000) 2019-10-13T03:55:08.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:08.879+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.880539+0000) 2019-10-13T03:55:08.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:08.874+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.875035+0000) 2019-10-13T03:55:08.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:08.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44378 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:08.884 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:08.876+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.877517+0000) 2019-10-13T03:55:08.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:08.958+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:08.959065+0000) 2019-10-13T03:55:11.893 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:55:11.894 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:55:11.898 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:55:11.898 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:55:12.589 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:12.581+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:12.582117+0000) 2019-10-13T03:55:13.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:13.881+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.882882+0000) 2019-10-13T03:55:13.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:13.881+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.883027+0000) 2019-10-13T03:55:13.885 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:13.881+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.883283+0000) 2019-10-13T03:55:13.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:13.877+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.877845+0000) 2019-10-13T03:55:13.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:13.882+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.883624+0000) 2019-10-13T03:55:13.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:13.882+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.883967+0000) 2019-10-13T03:55:13.887 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:13.882+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.884078+0000) 2019-10-13T03:55:13.887 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:13.883+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:13.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:13.879+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:13.880258+0000) 2019-10-13T03:55:14.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:14.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:14.958778+0000) 2019-10-13T03:55:14.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:14.958+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:14.958778+0000) 2019-10-13T03:55:14.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:14.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:14.958778+0000) 2019-10-13T03:55:15.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:15.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:15.832366+0000) 2019-10-13T03:55:17.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:17.322+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:17.323156+0000) 2019-10-13T03:55:18.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:18.113+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.114040+0000) 2019-10-13T03:55:18.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:18.884+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.886346+0000) 2019-10-13T03:55:18.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:18.885+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.886708+0000) 2019-10-13T03:55:18.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:18.880+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.881200+0000) 2019-10-13T03:55:18.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:18.885+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.887004+0000) 2019-10-13T03:55:18.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:18.885+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.887089+0000) 2019-10-13T03:55:18.890 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:18.885+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.887308+0000) 2019-10-13T03:55:18.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:18.886+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.887497+0000) 2019-10-13T03:55:18.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:18.886+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44437 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:18.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:18.883+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.884707+0000) 2019-10-13T03:55:18.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:18.960+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:18.961428+0000) 2019-10-13T03:55:22.589 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:22.581+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:22.582291+0000) 2019-10-13T03:55:23.891 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:23.889+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.889819+0000) 2019-10-13T03:55:23.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:23.889+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.889895+0000) 2019-10-13T03:55:23.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:23.889+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.890129+0000) 2019-10-13T03:55:23.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:23.884+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.885022+0000) 2019-10-13T03:55:23.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:23.890+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.890545+0000) 2019-10-13T03:55:23.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:23.890+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.891059+0000) 2019-10-13T03:55:23.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:23.890+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.891381+0000) 2019-10-13T03:55:23.893 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:23.891+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:23.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:23.885+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:23.886437+0000) 2019-10-13T03:55:24.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:24.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:24.958983+0000) 2019-10-13T03:55:24.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:24.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:24.958978+0000) 2019-10-13T03:55:24.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:24.958+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:24.958983+0000) 2019-10-13T03:55:25.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:25.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:25.832572+0000) 2019-10-13T03:55:27.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:27.322+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:27.323304+0000) 2019-10-13T03:55:28.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:28.113+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.114225+0000) 2019-10-13T03:55:28.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:28.892+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.893319+0000) 2019-10-13T03:55:28.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:28.887+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.888115+0000) 2019-10-13T03:55:28.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:28.893+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.894057+0000) 2019-10-13T03:55:28.896 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:28.893+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.894265+0000) 2019-10-13T03:55:28.896 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:28.893+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.894306+0000) 2019-10-13T03:55:28.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:28.894+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.894471+0000) 2019-10-13T03:55:28.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:28.894+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.894884+0000) 2019-10-13T03:55:28.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:28.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44498 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:28.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:28.889+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.890507+0000) 2019-10-13T03:55:28.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:28.962+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:28.963674+0000) 2019-10-13T03:55:32.589 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:32.582+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:32.582470+0000) 2019-10-13T03:55:33.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:33.895+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.896351+0000) 2019-10-13T03:55:33.900 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:33.896+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.896511+0000) 2019-10-13T03:55:33.900 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:33.896+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.896832+0000) 2019-10-13T03:55:33.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:33.896+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.896993+0000) 2019-10-13T03:55:33.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:33.897+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.897670+0000) 2019-10-13T03:55:33.902 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:33.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:33.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:33.897+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.897973+0000) 2019-10-13T03:55:33.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:33.891+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.891986+0000) 2019-10-13T03:55:33.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:33.893+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:33.893924+0000) 2019-10-13T03:55:34.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:34.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:34.959188+0000) 2019-10-13T03:55:34.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:34.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:34.959216+0000) 2019-10-13T03:55:34.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:34.958+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:34.959217+0000) 2019-10-13T03:55:35.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:35.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:35.832777+0000) 2019-10-13T03:55:37.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:37.323+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:37.323517+0000) 2019-10-13T03:55:38.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:38.114+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.114416+0000) 2019-10-13T03:55:38.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:38.899+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.899816+0000) 2019-10-13T03:55:38.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:38.899+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.899921+0000) 2019-10-13T03:55:38.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:38.894+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.895349+0000) 2019-10-13T03:55:38.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:38.899+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.900324+0000) 2019-10-13T03:55:38.903 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:38.900+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.900605+0000) 2019-10-13T03:55:38.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:38.900+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.901032+0000) 2019-10-13T03:55:38.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:38.900+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.901143+0000) 2019-10-13T03:55:38.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:38.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44557 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:38.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:38.895+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.896215+0000) 2019-10-13T03:55:38.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:38.965+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:38.965860+0000) 2019-10-13T03:55:42.194 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:55:42.194 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:55:42.198 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:55:42.198 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:55:42.589 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:42.581+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:42.582657+0000) 2019-10-13T03:55:43.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:43.896+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.897960+0000) 2019-10-13T03:55:43.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:43.900+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.902216+0000) 2019-10-13T03:55:43.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:43.901+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.903080+0000) 2019-10-13T03:55:43.913 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:43.902+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.903630+0000) 2019-10-13T03:55:43.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:43.902+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.903728+0000) 2019-10-13T03:55:43.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:43.902+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.903888+0000) 2019-10-13T03:55:43.920 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:43.902+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.904090+0000) 2019-10-13T03:55:43.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:43.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:43.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:43.903+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:43.904596+0000) 2019-10-13T03:55:44.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:44.958+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:44.959393+0000) 2019-10-13T03:55:44.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:44.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:44.959371+0000) 2019-10-13T03:55:44.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:44.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:44.959392+0000) 2019-10-13T03:55:45.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:45.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:45.833002+0000) 2019-10-13T03:55:47.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:47.322+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:47.323715+0000) 2019-10-13T03:55:48.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:48.113+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.114576+0000) 2019-10-13T03:55:48.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:48.904+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.906190+0000) 2019-10-13T03:55:48.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:48.905+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.906746+0000) 2019-10-13T03:55:48.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:48.905+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.906898+0000) 2019-10-13T03:55:48.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:48.905+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.907082+0000) 2019-10-13T03:55:48.908 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:48.905+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.907257+0000) 2019-10-13T03:55:48.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:48.900+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.902386+0000) 2019-10-13T03:55:48.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:48.906+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.907615+0000) 2019-10-13T03:55:48.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:48.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44618 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:48.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:48.902+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.904138+0000) 2019-10-13T03:55:48.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:48.966+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:48.967749+0000) 2019-10-13T03:55:52.589 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:52.582+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:52.582880+0000) 2019-10-13T03:55:53.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:53.903+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.904049+0000) 2019-10-13T03:55:53.911 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:53.908+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.909833+0000) 2019-10-13T03:55:53.911 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:53.908+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.910046+0000) 2019-10-13T03:55:53.911 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:53.908+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.910375+0000) 2019-10-13T03:55:53.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:53.909+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.910631+0000) 2019-10-13T03:55:53.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:53.909+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.910869+0000) 2019-10-13T03:55:53.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:53.909+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.911013+0000) 2019-10-13T03:55:53.913 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:53.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:53.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:53.905+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:53.906299+0000) 2019-10-13T03:55:54.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:54.958+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:54.959598+0000) 2019-10-13T03:55:54.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:54.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:54.959594+0000) 2019-10-13T03:55:54.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:54.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:54.959599+0000) 2019-10-13T03:55:55.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:55.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:55.833186+0000) 2019-10-13T03:55:57.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:57.322+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:57.323931+0000) 2019-10-13T03:55:58.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:58.114+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.114794+0000) 2019-10-13T03:55:58.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:55:58.911+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.913113+0000) 2019-10-13T03:55:58.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:55:58.907+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.908307+0000) 2019-10-13T03:55:58.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:55:58.912+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.913853+0000) 2019-10-13T03:55:58.917 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:55:58.912+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.914078+0000) 2019-10-13T03:55:58.917 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:55:58.912+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.914208+0000) 2019-10-13T03:55:58.917 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:55:58.913+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.914556+0000) 2019-10-13T03:55:58.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:55:58.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44677 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:55:58.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:55:58.913+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.914797+0000) 2019-10-13T03:55:58.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:58.910+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.911426+0000) 2019-10-13T03:55:58.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:55:58.969+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:55:58.969999+0000) 2019-10-13T03:56:02.589 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:02.582+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:02.583051+0000) 2019-10-13T03:56:03.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:03.915+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.916514+0000) 2019-10-13T03:56:03.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:03.915+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.916639+0000) 2019-10-13T03:56:03.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:03.915+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.916938+0000) 2019-10-13T03:56:03.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:03.911+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.912324+0000) 2019-10-13T03:56:03.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:03.915+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.917177+0000) 2019-10-13T03:56:03.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:03.916+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.917869+0000) 2019-10-13T03:56:03.920 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:03.916+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.917990+0000) 2019-10-13T03:56:03.920 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:03.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:03.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:03.913+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:03.914152+0000) 2019-10-13T03:56:04.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:04.959+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:04.959769+0000) 2019-10-13T03:56:04.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:04.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:04.959769+0000) 2019-10-13T03:56:04.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:04.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:04.959769+0000) 2019-10-13T03:56:05.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:05.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:05.833281+0000) 2019-10-13T03:56:07.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:07.323+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:07.324142+0000) 2019-10-13T03:56:08.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:08.114+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.115013+0000) 2019-10-13T03:56:08.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:08.919+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.919565+0000) 2019-10-13T03:56:08.921 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:08.919+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.919726+0000) 2019-10-13T03:56:08.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:08.919+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.919886+0000) 2019-10-13T03:56:08.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:08.913+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.914666+0000) 2019-10-13T03:56:08.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:08.920+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.920404+0000) 2019-10-13T03:56:08.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:08.920+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.920554+0000) 2019-10-13T03:56:08.923 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:08.920+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.920759+0000) 2019-10-13T03:56:08.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:08.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:08.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:08.916+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.917104+0000) 2019-10-13T03:56:08.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:08.971+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:08.972352+0000) 2019-10-13T03:56:12.337 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:56:12.337 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:56:12.341 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:56:12.341 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:56:12.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:12.582+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:12.583207+0000) 2019-10-13T03:56:13.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:13.921+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.922119+0000) 2019-10-13T03:56:13.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:13.922+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.922422+0000) 2019-10-13T03:56:13.924 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:13.922+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.922519+0000) 2019-10-13T03:56:13.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:13.922+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.922734+0000) 2019-10-13T03:56:13.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:13.916+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.917694+0000) 2019-10-13T03:56:13.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:13.922+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.922913+0000) 2019-10-13T03:56:13.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:13.922+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.923337+0000) 2019-10-13T03:56:13.926 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:13.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:13.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:13.919+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:13.920351+0000) 2019-10-13T03:56:14.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:14.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:14.959958+0000) 2019-10-13T03:56:14.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:14.959+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:14.959957+0000) 2019-10-13T03:56:14.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:14.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:14.959952+0000) 2019-10-13T03:56:15.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:15.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:15.833469+0000) 2019-10-13T03:56:17.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:17.324+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:17.324347+0000) 2019-10-13T03:56:18.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:18.115+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.115204+0000) 2019-10-13T03:56:18.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:18.924+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.925078+0000) 2019-10-13T03:56:18.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:18.924+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.925389+0000) 2019-10-13T03:56:18.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:18.925+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.925813+0000) 2019-10-13T03:56:18.930 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:18.925+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.925960+0000) 2019-10-13T03:56:18.931 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:18.925+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.926227+0000) 2019-10-13T03:56:18.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:18.925+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.926085+0000) 2019-10-13T03:56:18.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:18.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:18.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:18.920+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.920978+0000) 2019-10-13T03:56:18.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:18.922+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.923342+0000) 2019-10-13T03:56:18.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:18.974+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:18.974480+0000) 2019-10-13T03:56:22.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:22.583+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:22.583393+0000) 2019-10-13T03:56:23.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:23.928+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.928450+0000) 2019-10-13T03:56:23.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:23.922+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.923458+0000) 2019-10-13T03:56:23.931 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:23.928+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.929125+0000) 2019-10-13T03:56:23.931 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:23.928+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.929343+0000) 2019-10-13T03:56:23.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:23.929+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.929529+0000) 2019-10-13T03:56:23.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:23.929+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.929901+0000) 2019-10-13T03:56:23.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:23.929+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.930012+0000) 2019-10-13T03:56:23.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:23.929+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44823 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:23.932 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:23.924+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:23.925340+0000) 2019-10-13T03:56:24.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:24.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:24.960175+0000) 2019-10-13T03:56:24.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:24.960+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:24.960174+0000) 2019-10-13T03:56:24.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:24.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:24.960162+0000) 2019-10-13T03:56:25.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:25.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:25.833615+0000) 2019-10-13T03:56:27.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:27.323+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:27.324559+0000) 2019-10-13T03:56:28.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:28.114+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.115415+0000) 2019-10-13T03:56:28.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:28.925+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.926889+0000) 2019-10-13T03:56:28.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:28.930+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.932177+0000) 2019-10-13T03:56:28.937 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:28.931+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.932775+0000) 2019-10-13T03:56:28.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:28.931+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.932912+0000) 2019-10-13T03:56:28.938 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:28.931+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.933172+0000) 2019-10-13T03:56:28.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:28.931+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.933303+0000) 2019-10-13T03:56:28.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:28.932+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.933728+0000) 2019-10-13T03:56:28.938 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:28.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44858 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:28.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:28.928+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.930326+0000) 2019-10-13T03:56:28.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:28.975+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:28.976453+0000) 2019-10-13T03:56:32.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:32.581+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:32.583591+0000) 2019-10-13T03:56:33.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:33.934+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.935390+0000) 2019-10-13T03:56:33.937 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:33.934+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.936299+0000) 2019-10-13T03:56:33.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:33.935+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.936536+0000) 2019-10-13T03:56:33.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:33.929+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.931340+0000) 2019-10-13T03:56:33.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:33.935+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.936983+0000) 2019-10-13T03:56:33.940 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:33.935+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.937171+0000) 2019-10-13T03:56:33.941 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:33.935+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44884 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:33.942 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:33.935+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.937355+0000) 2019-10-13T03:56:33.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:33.931+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:33.933425+0000) 2019-10-13T03:56:34.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:34.959+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:34.960391+0000) 2019-10-13T03:56:34.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:34.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:34.960387+0000) 2019-10-13T03:56:34.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:34.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:34.960391+0000) 2019-10-13T03:56:35.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:35.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:35.833805+0000) 2019-10-13T03:56:37.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:37.323+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:37.324739+0000) 2019-10-13T03:56:38.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:38.114+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.115623+0000) 2019-10-13T03:56:38.949 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:38.937+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.939156+0000) 2019-10-13T03:56:38.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:38.937+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.939318+0000) 2019-10-13T03:56:38.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:38.938+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.939535+0000) 2019-10-13T03:56:38.950 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:38.938+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.939821+0000) 2019-10-13T03:56:38.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:38.938+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.940064+0000) 2019-10-13T03:56:38.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:38.939+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.940506+0000) 2019-10-13T03:56:38.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:38.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44917 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:38.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:38.931+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.933364+0000) 2019-10-13T03:56:38.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:38.933+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.935592+0000) 2019-10-13T03:56:38.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:38.977+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:38.978572+0000) 2019-10-13T03:56:42.475 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:56:42.475 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:56:42.481 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:56:42.481 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:56:42.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:42.583+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:42.583764+0000) 2019-10-13T03:56:43.944 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:43.941+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.942463+0000) 2019-10-13T03:56:43.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:43.936+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.937373+0000) 2019-10-13T03:56:43.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:43.941+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.943010+0000) 2019-10-13T03:56:43.945 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:43.941+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.943167+0000) 2019-10-13T03:56:43.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:43.941+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.943324+0000) 2019-10-13T03:56:43.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:43.942+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.943480+0000) 2019-10-13T03:56:43.946 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:43.942+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.943765+0000) 2019-10-13T03:56:43.946 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:43.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:43.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:43.939+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:43.940521+0000) 2019-10-13T03:56:44.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:44.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:44.960619+0000) 2019-10-13T03:56:44.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:44.959+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:44.960619+0000) 2019-10-13T03:56:44.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:44.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:44.960598+0000) 2019-10-13T03:56:45.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:45.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:45.834011+0000) 2019-10-13T03:56:47.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:47.324+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:47.324932+0000) 2019-10-13T03:56:48.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:48.115+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.115834+0000) 2019-10-13T03:56:48.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:48.944+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.946082+0000) 2019-10-13T03:56:48.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:48.944+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.946275+0000) 2019-10-13T03:56:48.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:48.945+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.946518+0000) 2019-10-13T03:56:48.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:48.940+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.941627+0000) 2019-10-13T03:56:48.949 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:48.945+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.946853+0000) 2019-10-13T03:56:48.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:48.946+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.947629+0000) 2019-10-13T03:56:48.950 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:48.945+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.946937+0000) 2019-10-13T03:56:48.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:48.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 44978 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:48.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:48.943+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.944002+0000) 2019-10-13T03:56:48.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:48.980+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:48.980622+0000) 2019-10-13T03:56:52.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:52.583+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:52.583938+0000) 2019-10-13T03:56:53.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:53.944+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.944751+0000) 2019-10-13T03:56:53.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:53.950+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.950530+0000) 2019-10-13T03:56:53.952 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:53.950+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.950734+0000) 2019-10-13T03:56:53.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:53.950+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.950932+0000) 2019-10-13T03:56:53.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:53.950+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.951106+0000) 2019-10-13T03:56:53.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:53.950+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.951325+0000) 2019-10-13T03:56:53.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:53.945+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.946625+0000) 2019-10-13T03:56:53.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:53.951+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:53.951484+0000) 2019-10-13T03:56:53.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:53.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:54.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:54.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:54.960790+0000) 2019-10-13T03:56:54.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:54.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:54.960834+0000) 2019-10-13T03:56:54.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:54.960+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:54.960834+0000) 2019-10-13T03:56:55.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:55.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:55.834171+0000) 2019-10-13T03:56:57.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:57.324+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:57.325121+0000) 2019-10-13T03:56:58.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:58.115+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.116014+0000) 2019-10-13T03:56:58.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:56:58.953+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.953456+0000) 2019-10-13T03:56:58.955 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:56:58.953+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.953655+0000) 2019-10-13T03:56:58.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:56:58.947+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.948602+0000) 2019-10-13T03:56:58.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:56:58.953+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.954336+0000) 2019-10-13T03:56:58.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:56:58.954+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.954572+0000) 2019-10-13T03:56:58.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:56:58.954+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.954705+0000) 2019-10-13T03:56:58.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:56:58.954+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.954913+0000) 2019-10-13T03:56:58.958 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:56:58.954+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:56:58.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:58.949+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.950314+0000) 2019-10-13T03:56:58.991 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:56:58.982+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:56:58.983121+0000) 2019-10-13T03:57:02.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:02.583+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:02.584129+0000) 2019-10-13T03:57:03.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:03.950+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.951592+0000) 2019-10-13T03:57:03.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:03.956+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.957288+0000) 2019-10-13T03:57:03.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:03.957+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.957494+0000) 2019-10-13T03:57:03.959 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:03.957+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.957957+0000) 2019-10-13T03:57:03.960 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:03.957+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.958255+0000) 2019-10-13T03:57:03.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:03.957+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.958344+0000) 2019-10-13T03:57:03.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:03.958+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.958488+0000) 2019-10-13T03:57:03.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:03.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:03.962 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:03.954+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:03.955289+0000) 2019-10-13T03:57:04.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:04.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:04.960908+0000) 2019-10-13T03:57:04.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:04.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:04.960965+0000) 2019-10-13T03:57:04.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:04.960+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:04.960989+0000) 2019-10-13T03:57:05.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:05.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:05.834322+0000) 2019-10-13T03:57:07.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:07.325+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:07.325312+0000) 2019-10-13T03:57:08.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:08.116+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.116188+0000) 2019-10-13T03:57:08.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:08.954+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.955396+0000) 2019-10-13T03:57:08.966 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:08.957+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.958283+0000) 2019-10-13T03:57:08.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:08.960+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.960461+0000) 2019-10-13T03:57:08.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:08.960+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.960630+0000) 2019-10-13T03:57:08.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:08.960+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.960951+0000) 2019-10-13T03:57:08.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:08.960+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.961160+0000) 2019-10-13T03:57:08.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:08.961+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.961406+0000) 2019-10-13T03:57:08.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:08.961+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.961573+0000) 2019-10-13T03:57:08.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:08.961+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45101 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:08.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:08.985+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:08.985207+0000) 2019-10-13T03:57:12.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:12.583+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:12.584337+0000) 2019-10-13T03:57:12.740 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:57:12.740 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:57:12.745 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:57:12.745 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:57:13.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:13.962+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.964065+0000) 2019-10-13T03:57:13.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:13.962+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.964204+0000) 2019-10-13T03:57:13.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:13.962+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.964375+0000) 2019-10-13T03:57:13.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:13.958+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.959101+0000) 2019-10-13T03:57:13.967 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:13.963+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.965020+0000) 2019-10-13T03:57:13.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:13.963+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.965245+0000) 2019-10-13T03:57:13.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:13.963+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.965327+0000) 2019-10-13T03:57:13.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:13.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45128 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:13.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:13.960+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:13.961548+0000) 2019-10-13T03:57:14.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:14.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:14.961078+0000) 2019-10-13T03:57:14.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:14.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:14.961137+0000) 2019-10-13T03:57:14.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:14.960+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:14.961148+0000) 2019-10-13T03:57:15.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:15.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:15.834523+0000) 2019-10-13T03:57:17.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:17.324+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:17.325508+0000) 2019-10-13T03:57:18.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:18.115+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.116382+0000) 2019-10-13T03:57:18.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:18.960+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.962233+0000) 2019-10-13T03:57:18.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:18.966+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.967877+0000) 2019-10-13T03:57:18.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:18.966+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.968034+0000) 2019-10-13T03:57:18.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:18.966+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.968229+0000) 2019-10-13T03:57:18.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:18.967+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.968412+0000) 2019-10-13T03:57:18.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:18.967+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.968994+0000) 2019-10-13T03:57:18.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:18.967+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.969192+0000) 2019-10-13T03:57:18.971 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:18.967+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:18.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:18.965+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.966790+0000) 2019-10-13T03:57:18.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:18.986+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:18.987303+0000) 2019-10-13T03:57:22.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:22.582+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:22.584495+0000) 2019-10-13T03:57:23.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:23.969+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.971202+0000) 2019-10-13T03:57:23.973 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:23.970+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.971600+0000) 2019-10-13T03:57:23.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:23.970+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.971780+0000) 2019-10-13T03:57:23.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:23.970+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.972042+0000) 2019-10-13T03:57:23.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:23.970+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.972296+0000) 2019-10-13T03:57:23.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:23.965+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.966760+0000) 2019-10-13T03:57:23.974 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:23.971+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.972559+0000) 2019-10-13T03:57:23.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:23.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45189 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:23.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:23.967+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:23.969301+0000) 2019-10-13T03:57:24.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:24.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:24.961270+0000) 2019-10-13T03:57:24.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:24.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:24.961351+0000) 2019-10-13T03:57:24.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:24.960+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:24.961327+0000) 2019-10-13T03:57:25.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:25.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:25.834717+0000) 2019-10-13T03:57:27.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:27.325+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:27.325686+0000) 2019-10-13T03:57:28.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:28.115+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.116586+0000) 2019-10-13T03:57:28.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:28.973+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.974755+0000) 2019-10-13T03:57:28.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:28.973+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.975046+0000) 2019-10-13T03:57:28.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:28.973+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.975340+0000) 2019-10-13T03:57:28.977 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:28.974+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.975467+0000) 2019-10-13T03:57:28.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:28.974+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.975761+0000) 2019-10-13T03:57:28.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:28.974+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.975815+0000) 2019-10-13T03:57:28.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:28.968+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.970707+0000) 2019-10-13T03:57:28.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:28.975+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:28.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:28.970+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.972650+0000) 2019-10-13T03:57:28.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:28.988+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:28.989573+0000) 2019-10-13T03:57:32.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:32.583+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:32.584650+0000) 2019-10-13T03:57:33.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:33.977+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.978757+0000) 2019-10-13T03:57:33.981 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:33.977+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.978967+0000) 2019-10-13T03:57:33.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:33.977+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.979042+0000) 2019-10-13T03:57:33.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:33.978+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.979497+0000) 2019-10-13T03:57:33.982 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:33.978+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.979670+0000) 2019-10-13T03:57:33.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:33.978+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.979793+0000) 2019-10-13T03:57:33.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:33.972+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.972888+0000) 2019-10-13T03:57:33.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:33.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45252 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:33.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:33.975+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:33.976145+0000) 2019-10-13T03:57:34.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:34.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:34.961444+0000) 2019-10-13T03:57:34.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:34.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:34.961541+0000) 2019-10-13T03:57:34.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:34.960+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:34.961521+0000) 2019-10-13T03:57:35.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:35.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:35.834924+0000) 2019-10-13T03:57:37.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:37.324+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:37.325894+0000) 2019-10-13T03:57:38.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:38.116+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.116803+0000) 2019-10-13T03:57:38.983 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:38.981+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.981998+0000) 2019-10-13T03:57:38.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:38.982+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.982562+0000) 2019-10-13T03:57:38.984 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:38.982+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.982481+0000) 2019-10-13T03:57:38.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:38.976+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.977344+0000) 2019-10-13T03:57:38.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:38.982+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.983264+0000) 2019-10-13T03:57:38.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:38.983+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.983414+0000) 2019-10-13T03:57:38.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:38.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45287 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:38.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:38.983+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.983572+0000) 2019-10-13T03:57:38.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:38.978+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.979613+0000) 2019-10-13T03:57:38.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:38.990+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:38.991682+0000) 2019-10-13T03:57:42.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:42.584+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:42.584831+0000) 2019-10-13T03:57:42.938 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:57:42.938 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:57:42.965 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:57:42.965 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:57:43.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:43.986+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.986427+0000) 2019-10-13T03:57:43.988 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:43.985+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.986381+0000) 2019-10-13T03:57:43.988 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:43.986+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.986616+0000) 2019-10-13T03:57:43.989 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:43.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:43.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:43.980+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.981297+0000) 2019-10-13T03:57:43.990 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:43.986+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.987135+0000) 2019-10-13T03:57:43.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:43.986+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.987316+0000) 2019-10-13T03:57:43.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:43.987+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.987494+0000) 2019-10-13T03:57:43.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:43.984+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:43.985338+0000) 2019-10-13T03:57:44.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:44.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:44.961618+0000) 2019-10-13T03:57:44.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:44.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:44.961687+0000) 2019-10-13T03:57:44.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:44.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:44.961688+0000) 2019-10-13T03:57:45.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:45.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:45.835123+0000) 2019-10-13T03:57:47.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:47.325+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:47.326076+0000) 2019-10-13T03:57:48.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:48.116+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.116991+0000) 2019-10-13T03:57:48.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:48.988+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.988771+0000) 2019-10-13T03:57:48.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:48.988+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.988998+0000) 2019-10-13T03:57:48.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:48.988+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.989176+0000) 2019-10-13T03:57:48.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:48.988+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.989231+0000) 2019-10-13T03:57:48.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:48.989+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.989532+0000) 2019-10-13T03:57:48.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:48.984+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.984783+0000) 2019-10-13T03:57:48.993 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:48.989+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.989782+0000) 2019-10-13T03:57:48.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:48.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45350 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:48.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:48.985+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.986153+0000) 2019-10-13T03:57:49.000 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:48.993+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:48.993769+0000) 2019-10-13T03:57:52.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:52.584+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:52.585020+0000) 2019-10-13T03:57:53.994 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:53.992+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.992434+0000) 2019-10-13T03:57:53.994 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:53.992+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.992735+0000) 2019-10-13T03:57:53.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:53.986+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.987602+0000) 2019-10-13T03:57:53.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:53.992+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.993260+0000) 2019-10-13T03:57:53.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:53.993+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.993488+0000) 2019-10-13T03:57:53.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:53.993+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.993599+0000) 2019-10-13T03:57:53.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:53.993+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.993814+0000) 2019-10-13T03:57:53.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:53.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:53.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:53.988+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:53.989250+0000) 2019-10-13T03:57:54.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:54.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:54.961763+0000) 2019-10-13T03:57:54.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:54.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:54.961831+0000) 2019-10-13T03:57:54.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:54.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:54.961822+0000) 2019-10-13T03:57:55.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:55.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:55.835328+0000) 2019-10-13T03:57:57.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:57.325+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:57.326266+0000) 2019-10-13T03:57:58.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:58.116+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.117190+0000) 2019-10-13T03:57:58.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:57:58.993+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.995195+0000) 2019-10-13T03:57:58.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:57:58.994+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.995427+0000) 2019-10-13T03:57:58.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:57:58.994+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.995637+0000) 2019-10-13T03:57:58.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:57:58.990+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.990868+0000) 2019-10-13T03:57:58.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:57:58.994+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.996137+0000) 2019-10-13T03:57:58.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:57:58.994+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.996366+0000) 2019-10-13T03:57:58.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:57:58.995+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.996546+0000) 2019-10-13T03:57:58.998 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:57:58.995+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:57:58.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:58.991+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.992692+0000) 2019-10-13T03:57:59.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:57:58.995+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:57:58.995971+0000) 2019-10-13T03:58:02.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:02.585+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:02.585172+0000) 2019-10-13T03:58:04.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:03.997+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.998417+0000) 2019-10-13T03:58:04.001 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:03.997+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.998556+0000) 2019-10-13T03:58:04.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:03.992+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.993281+0000) 2019-10-13T03:58:04.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:03.997+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.999224+0000) 2019-10-13T03:58:04.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:03.998+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.999441+0000) 2019-10-13T03:58:04.003 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:03.994+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.995677+0000) 2019-10-13T03:58:04.005 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:03.998+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.999532+0000) 2019-10-13T03:58:04.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:03.998+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:03.999857+0000) 2019-10-13T03:58:04.006 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:03.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45437 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:04.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:04.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:04.961845+0000) 2019-10-13T03:58:04.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:04.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:04.961968+0000) 2019-10-13T03:58:04.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:04.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:04.961979+0000) 2019-10-13T03:58:05.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:05.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:05.835514+0000) 2019-10-13T03:58:07.328 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:07.325+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:07.326453+0000) 2019-10-13T03:58:08.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:08.116+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:08.117368+0000) 2019-10-13T03:58:09.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:08.995+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:08.996404+0000) 2019-10-13T03:58:09.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:09.000+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:09.002046+0000) 2019-10-13T03:58:09.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:09.000+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:09.002312+0000) 2019-10-13T03:58:09.007 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:09.001+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:09.002450+0000) 2019-10-13T03:58:09.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:09.001+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:09.002778+0000) 2019-10-13T03:58:09.008 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:09.001+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:09.003230+0000) 2019-10-13T03:58:09.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:09.001+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:09.003347+0000) 2019-10-13T03:58:09.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:09.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45474 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:09.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:08.998+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:08.999494+0000) 2019-10-13T03:58:09.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:08.998+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:08.999544+0000) 2019-10-13T03:58:12.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:12.584+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:12.585375+0000) 2019-10-13T03:58:13.161 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:58:13.161 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:58:13.166 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:58:13.166 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:58:14.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:13.998+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.000611+0000) 2019-10-13T03:58:14.008 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:14.004+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.005814+0000) 2019-10-13T03:58:14.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:14.004+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.005929+0000) 2019-10-13T03:58:14.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:14.005+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.006433+0000) 2019-10-13T03:58:14.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:14.005+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.006620+0000) 2019-10-13T03:58:14.009 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:14.005+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.006766+0000) 2019-10-13T03:58:14.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:14.005+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.007007+0000) 2019-10-13T03:58:14.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:14.001+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.002972+0000) 2019-10-13T03:58:14.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:14.005+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:14.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:14.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.961972+0000) 2019-10-13T03:58:14.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:14.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.962080+0000) 2019-10-13T03:58:14.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:14.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:14.962113+0000) 2019-10-13T03:58:15.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:15.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:15.835703+0000) 2019-10-13T03:58:17.328 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:17.326+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:17.326640+0000) 2019-10-13T03:58:18.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:18.116+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:18.117557+0000) 2019-10-13T03:58:19.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:18.998+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:18.999655+0000) 2019-10-13T03:58:19.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:19.007+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.008566+0000) 2019-10-13T03:58:19.010 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:19.007+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.008828+0000) 2019-10-13T03:58:19.010 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:19.007+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.009037+0000) 2019-10-13T03:58:19.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:19.007+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.009064+0000) 2019-10-13T03:58:19.011 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:19.007+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.009364+0000) 2019-10-13T03:58:19.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:19.002+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.004618+0000) 2019-10-13T03:58:19.012 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:19.008+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.009820+0000) 2019-10-13T03:58:19.012 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:19.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:19.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:19.005+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:19.006842+0000) 2019-10-13T03:58:22.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:22.584+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:22.585559+0000) 2019-10-13T03:58:24.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:24.012+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.012734+0000) 2019-10-13T03:58:24.015 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:24.012+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.013046+0000) 2019-10-13T03:58:24.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:24.012+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.013341+0000) 2019-10-13T03:58:24.015 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:24.013+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.013408+0000) 2019-10-13T03:58:24.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:24.013+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.013512+0000) 2019-10-13T03:58:24.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:24.006+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.007522+0000) 2019-10-13T03:58:24.016 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:24.013+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.013869+0000) 2019-10-13T03:58:24.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:24.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45561 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:24.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:24.009+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.010565+0000) 2019-10-13T03:58:24.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:24.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.962136+0000) 2019-10-13T03:58:24.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:24.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.962226+0000) 2019-10-13T03:58:24.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:24.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:24.962233+0000) 2019-10-13T03:58:25.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:25.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:25.835888+0000) 2019-10-13T03:58:27.328 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:27.326+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:27.326759+0000) 2019-10-13T03:58:28.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:28.117+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:28.117733+0000) 2019-10-13T03:58:29.008 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:29.001+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.001875+0000) 2019-10-13T03:58:29.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:29.015+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.015725+0000) 2019-10-13T03:58:29.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:29.015+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.015923+0000) 2019-10-13T03:58:29.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:29.009+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.010140+0000) 2019-10-13T03:58:29.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:29.015+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.016086+0000) 2019-10-13T03:58:29.018 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:29.016+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.016408+0000) 2019-10-13T03:58:29.018 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:29.016+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.016605+0000) 2019-10-13T03:58:29.019 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:29.016+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.017326+0000) 2019-10-13T03:58:29.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:29.012+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:29.012767+0000) 2019-10-13T03:58:29.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:29.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:32.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:32.585+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:32.585751+0000) 2019-10-13T03:58:34.021 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:34.019+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.019390+0000) 2019-10-13T03:58:34.022 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:34.019+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.019500+0000) 2019-10-13T03:58:34.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:34.019+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.019843+0000) 2019-10-13T03:58:34.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:34.013+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.014648+0000) 2019-10-13T03:58:34.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:34.015+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.015880+0000) 2019-10-13T03:58:34.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:34.019+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.020289+0000) 2019-10-13T03:58:34.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:34.019+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.020327+0000) 2019-10-13T03:58:34.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:34.020+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.020419+0000) 2019-10-13T03:58:34.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:34.020+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45624 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:34.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:34.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.962310+0000) 2019-10-13T03:58:34.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:34.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.962425+0000) 2019-10-13T03:58:34.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:34.962+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:34.962406+0000) 2019-10-13T03:58:35.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:35.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:35.836036+0000) 2019-10-13T03:58:37.328 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:37.326+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:37.326965+0000) 2019-10-13T03:58:38.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:38.117+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:38.117928+0000) 2019-10-13T03:58:39.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:39.003+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.004235+0000) 2019-10-13T03:58:39.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:39.022+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.022815+0000) 2019-10-13T03:58:39.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:39.022+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.023064+0000) 2019-10-13T03:58:39.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:39.017+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.017944+0000) 2019-10-13T03:58:39.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:39.019+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.020575+0000) 2019-10-13T03:58:39.035 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:39.023+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.023441+0000) 2019-10-13T03:58:39.036 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:39.023+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.023671+0000) 2019-10-13T03:58:39.036 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:39.023+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.024002+0000) 2019-10-13T03:58:39.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:39.023+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:39.024158+0000) 2019-10-13T03:58:39.036 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:39.024+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:42.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:42.585+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:42.585937+0000) 2019-10-13T03:58:43.386 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:58:43.387 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:58:43.394 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:58:43.395 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:58:44.028 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:44.024+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.026337+0000) 2019-10-13T03:58:44.029 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:44.025+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.026575+0000) 2019-10-13T03:58:44.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:44.025+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.026816+0000) 2019-10-13T03:58:44.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:44.025+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.026896+0000) 2019-10-13T03:58:44.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:44.025+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.027097+0000) 2019-10-13T03:58:44.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:44.021+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.022264+0000) 2019-10-13T03:58:44.031 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:44.026+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.027461+0000) 2019-10-13T03:58:44.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:44.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45685 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:44.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:44.023+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.024034+0000) 2019-10-13T03:58:44.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:44.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.962496+0000) 2019-10-13T03:58:44.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:44.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.962552+0000) 2019-10-13T03:58:44.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:44.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:44.962586+0000) 2019-10-13T03:58:45.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:45.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:45.836200+0000) 2019-10-13T03:58:47.328 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:47.326+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:47.327180+0000) 2019-10-13T03:58:48.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:48.117+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:48.118113+0000) 2019-10-13T03:58:49.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:49.006+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.006490+0000) 2019-10-13T03:58:49.034 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:49.028+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.030224+0000) 2019-10-13T03:58:49.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:49.029+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.030411+0000) 2019-10-13T03:58:49.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:49.029+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.030730+0000) 2019-10-13T03:58:49.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:49.029+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.031042+0000) 2019-10-13T03:58:49.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:49.029+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.031169+0000) 2019-10-13T03:58:49.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:49.029+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.031359+0000) 2019-10-13T03:58:49.036 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:49.030+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45722 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:49.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:49.023+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.024571+0000) 2019-10-13T03:58:49.036 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:49.026+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:49.027262+0000) 2019-10-13T03:58:52.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:52.585+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:52.586115+0000) 2019-10-13T03:58:54.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:54.032+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.033521+0000) 2019-10-13T03:58:54.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:54.032+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.033848+0000) 2019-10-13T03:58:54.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:54.027+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.028688+0000) 2019-10-13T03:58:54.036 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:54.032+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.034324+0000) 2019-10-13T03:58:54.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:54.033+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.034561+0000) 2019-10-13T03:58:54.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:54.033+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.034788+0000) 2019-10-13T03:58:54.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:54.033+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.035313+0000) 2019-10-13T03:58:54.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:54.034+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45748 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:54.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:54.029+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.030414+0000) 2019-10-13T03:58:54.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:54.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.962700+0000) 2019-10-13T03:58:54.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:54.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.962677+0000) 2019-10-13T03:58:54.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:54.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:54.962749+0000) 2019-10-13T03:58:55.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:55.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:55.836370+0000) 2019-10-13T03:58:57.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:57.326+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:57.327370+0000) 2019-10-13T03:58:58.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:58.117+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:58.118313+0000) 2019-10-13T03:58:59.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:59.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.008863+0000) 2019-10-13T03:58:59.038 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:58:59.035+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.037146+0000) 2019-10-13T03:58:59.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:58:59.031+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.032033+0000) 2019-10-13T03:58:59.040 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:58:59.036+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.037655+0000) 2019-10-13T03:58:59.040 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:58:59.036+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.037840+0000) 2019-10-13T03:58:59.040 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:58:59.036+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.038085+0000) 2019-10-13T03:58:59.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:58:59.036+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.038332+0000) 2019-10-13T03:58:59.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:58:59.037+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.038723+0000) 2019-10-13T03:58:59.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:58:59.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:58:59.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:58:59.034+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:58:59.035488+0000) 2019-10-13T03:59:02.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:02.585+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:02.586298+0000) 2019-10-13T03:59:04.042 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:04.039+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.040897+0000) 2019-10-13T03:59:04.042 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:04.039+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.041132+0000) 2019-10-13T03:59:04.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:04.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.035904+0000) 2019-10-13T03:59:04.043 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:04.040+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.041495+0000) 2019-10-13T03:59:04.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:04.040+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.041591+0000) 2019-10-13T03:59:04.044 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:04.040+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.042041+0000) 2019-10-13T03:59:04.044 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:04.041+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:04.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:04.041+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.042530+0000) 2019-10-13T03:59:04.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:04.036+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.038529+0000) 2019-10-13T03:59:04.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:04.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.962837+0000) 2019-10-13T03:59:04.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:04.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.962854+0000) 2019-10-13T03:59:04.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:04.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:04.962929+0000) 2019-10-13T03:59:05.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:05.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:05.836543+0000) 2019-10-13T03:59:07.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:07.327+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:07.327551+0000) 2019-10-13T03:59:08.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:08.118+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:08.118543+0000) 2019-10-13T03:59:09.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:09.010+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.011162+0000) 2019-10-13T03:59:09.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:09.036+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.038369+0000) 2019-10-13T03:59:09.045 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:09.043+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.043959+0000) 2019-10-13T03:59:09.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:09.043+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.044110+0000) 2019-10-13T03:59:09.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:09.043+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.044336+0000) 2019-10-13T03:59:09.046 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:09.044+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.044506+0000) 2019-10-13T03:59:09.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:09.044+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.044700+0000) 2019-10-13T03:59:09.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:09.044+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.044887+0000) 2019-10-13T03:59:09.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:09.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:09.048 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:09.039+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:09.041445+0000) 2019-10-13T03:59:12.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:12.585+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:12.586482+0000) 2019-10-13T03:59:13.563 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:59:13.564 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:59:13.570 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:59:13.570 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:59:14.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:14.046+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.047221+0000) 2019-10-13T03:59:14.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:14.047+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.047428+0000) 2019-10-13T03:59:14.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:14.040+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.042668+0000) 2019-10-13T03:59:14.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:14.047+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.047839+0000) 2019-10-13T03:59:14.051 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:14.047+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.048212+0000) 2019-10-13T03:59:14.052 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:14.047+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.048382+0000) 2019-10-13T03:59:14.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:14.048+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.048637+0000) 2019-10-13T03:59:14.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:14.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:14.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:14.042+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.044428+0000) 2019-10-13T03:59:14.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:14.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.963013+0000) 2019-10-13T03:59:14.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:14.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.963029+0000) 2019-10-13T03:59:14.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:14.962+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:14.963089+0000) 2019-10-13T03:59:15.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:15.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:15.836693+0000) 2019-10-13T03:59:17.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:17.327+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:17.327733+0000) 2019-10-13T03:59:18.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:18.118+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:18.118756+0000) 2019-10-13T03:59:19.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:19.012+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.013518+0000) 2019-10-13T03:59:19.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:19.050+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.050389+0000) 2019-10-13T03:59:19.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:19.050+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.050625+0000) 2019-10-13T03:59:19.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:19.050+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.050752+0000) 2019-10-13T03:59:19.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:19.044+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.045626+0000) 2019-10-13T03:59:19.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:19.050+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.051233+0000) 2019-10-13T03:59:19.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:19.051+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.051458+0000) 2019-10-13T03:59:19.054 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:19.051+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.051646+0000) 2019-10-13T03:59:19.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:19.051+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45904 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:19.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:19.046+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:19.047472+0000) 2019-10-13T03:59:22.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:22.586+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:22.586662+0000) 2019-10-13T03:59:24.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:24.052+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.053255+0000) 2019-10-13T03:59:24.055 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:24.053+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.053487+0000) 2019-10-13T03:59:24.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:24.047+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.047963+0000) 2019-10-13T03:59:24.056 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:24.054+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.054437+0000) 2019-10-13T03:59:24.057 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:24.054+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.054691+0000) 2019-10-13T03:59:24.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:24.054+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.054775+0000) 2019-10-13T03:59:24.057 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:24.054+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.055013+0000) 2019-10-13T03:59:24.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:24.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:24.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:24.051+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.052263+0000) 2019-10-13T03:59:24.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:24.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.963224+0000) 2019-10-13T03:59:24.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:24.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.963223+0000) 2019-10-13T03:59:24.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:24.962+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:24.963228+0000) 2019-10-13T03:59:25.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:25.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:25.836866+0000) 2019-10-13T03:59:27.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:27.326+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:27.327924+0000) 2019-10-13T03:59:28.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:28.117+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:28.118971+0000) 2019-10-13T03:59:29.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:29.015+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.015693+0000) 2019-10-13T03:59:29.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:29.050+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.051667+0000) 2019-10-13T03:59:29.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:29.055+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.057200+0000) 2019-10-13T03:59:29.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:29.055+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.057354+0000) 2019-10-13T03:59:29.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:29.056+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.057471+0000) 2019-10-13T03:59:29.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:29.056+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.057617+0000) 2019-10-13T03:59:29.061 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:29.056+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.058007+0000) 2019-10-13T03:59:29.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:29.056+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.058389+0000) 2019-10-13T03:59:29.061 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:29.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45964 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:29.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:29.053+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:29.054333+0000) 2019-10-13T03:59:32.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:32.586+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:32.586882+0000) 2019-10-13T03:59:34.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:34.058+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.060130+0000) 2019-10-13T03:59:34.063 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:34.058+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.060267+0000) 2019-10-13T03:59:34.063 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:34.059+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.060627+0000) 2019-10-13T03:59:34.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:34.059+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.060501+0000) 2019-10-13T03:59:34.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:34.059+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.060755+0000) 2019-10-13T03:59:34.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:34.059+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.060875+0000) 2019-10-13T03:59:34.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:34.055+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.055845+0000) 2019-10-13T03:59:34.064 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:34.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 45990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:34.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:34.057+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.058094+0000) 2019-10-13T03:59:34.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:34.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.963371+0000) 2019-10-13T03:59:34.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:34.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.963321+0000) 2019-10-13T03:59:34.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:34.961+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:34.963361+0000) 2019-10-13T03:59:35.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:35.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:35.837072+0000) 2019-10-13T03:59:37.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:37.327+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:37.328145+0000) 2019-10-13T03:59:38.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:38.118+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:38.119141+0000) 2019-10-13T03:59:39.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:39.017+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.018016+0000) 2019-10-13T03:59:39.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:39.062+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.063503+0000) 2019-10-13T03:59:39.065 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:39.062+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.063714+0000) 2019-10-13T03:59:39.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:39.058+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.058915+0000) 2019-10-13T03:59:39.066 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:39.062+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.064058+0000) 2019-10-13T03:59:39.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:39.062+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.064047+0000) 2019-10-13T03:59:39.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:39.063+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.064573+0000) 2019-10-13T03:59:39.067 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:39.063+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.064840+0000) 2019-10-13T03:59:39.067 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:39.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:39.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:39.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:39.061490+0000) 2019-10-13T03:59:42.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:42.586+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:42.587047+0000) 2019-10-13T03:59:43.749 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T03:59:43.750 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:59:43.754 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T03:59:43.754 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T03:59:44.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:44.061+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.062557+0000) 2019-10-13T03:59:44.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:44.065+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.067070+0000) 2019-10-13T03:59:44.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:44.066+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.067491+0000) 2019-10-13T03:59:44.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:44.066+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.067659+0000) 2019-10-13T03:59:44.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:44.066+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.067795+0000) 2019-10-13T03:59:44.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:44.066+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.068194+0000) 2019-10-13T03:59:44.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:44.067+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.068653+0000) 2019-10-13T03:59:44.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:44.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:44.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:44.064+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.065108+0000) 2019-10-13T03:59:44.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:44.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.963530+0000) 2019-10-13T03:59:44.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:44.962+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.963531+0000) 2019-10-13T03:59:44.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:44.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:44.963532+0000) 2019-10-13T03:59:45.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:45.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:45.837287+0000) 2019-10-13T03:59:47.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:47.327+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:47.328334+0000) 2019-10-13T03:59:48.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:48.118+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:48.119317+0000) 2019-10-13T03:59:49.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:49.020+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.020467+0000) 2019-10-13T03:59:49.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:49.069+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.070969+0000) 2019-10-13T03:59:49.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:49.069+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.071097+0000) 2019-10-13T03:59:49.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:49.069+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.071317+0000) 2019-10-13T03:59:49.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:49.065+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.065980+0000) 2019-10-13T03:59:49.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:49.070+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.071790+0000) 2019-10-13T03:59:49.074 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:49.070+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.072078+0000) 2019-10-13T03:59:49.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:49.070+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.072075+0000) 2019-10-13T03:59:49.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:49.071+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46082 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:49.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:49.067+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:49.068224+0000) 2019-10-13T03:59:52.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:52.586+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:52.587213+0000) 2019-10-13T03:59:54.076 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:54.073+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.074360+0000) 2019-10-13T03:59:54.076 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:54.074+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.074550+0000) 2019-10-13T03:59:54.076 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:54.074+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.074688+0000) 2019-10-13T03:59:54.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:54.067+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.069476+0000) 2019-10-13T03:59:54.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:54.074+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.075160+0000) 2019-10-13T03:59:54.077 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:54.074+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.075225+0000) 2019-10-13T03:59:54.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:54.075+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.075481+0000) 2019-10-13T03:59:54.078 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:54.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T03:59:54.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:54.069+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.071016+0000) 2019-10-13T03:59:54.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:54.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.963718+0000) 2019-10-13T03:59:54.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:54.963+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.963719+0000) 2019-10-13T03:59:54.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:54.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:54.963724+0000) 2019-10-13T03:59:55.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:55.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:55.837455+0000) 2019-10-13T03:59:57.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:57.328+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:57.328499+0000) 2019-10-13T03:59:58.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:58.119+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:58.119513+0000) 2019-10-13T03:59:59.029 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:59.021+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.022660+0000) 2019-10-13T03:59:59.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T03:59:59.070+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.072474+0000) 2019-10-13T03:59:59.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T03:59:59.073+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.074948+0000) 2019-10-13T03:59:59.083 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T03:59:59.076+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.077341+0000) 2019-10-13T03:59:59.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T03:59:59.077+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.077476+0000) 2019-10-13T03:59:59.084 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T03:59:59.077+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.077702+0000) 2019-10-13T03:59:59.084 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T03:59:59.078+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.078844+0000) 2019-10-13T03:59:59.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T03:59:59.078+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.079002+0000) 2019-10-13T03:59:59.084 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T03:59:59.078+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T02:59:59.079303+0000) 2019-10-13T03:59:59.084 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T03:59:59.079+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:02.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:02.586+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:02.587423+0000) 2019-10-13T04:00:04.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:04.080+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.081354+0000) 2019-10-13T04:00:04.083 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:04.081+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.081579+0000) 2019-10-13T04:00:04.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:04.074+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.076382+0000) 2019-10-13T04:00:04.084 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:04.081+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.082035+0000) 2019-10-13T04:00:04.084 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:04.081+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.082121+0000) 2019-10-13T04:00:04.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:04.082+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.082412+0000) 2019-10-13T04:00:04.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:04.082+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.083035+0000) 2019-10-13T04:00:04.085 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:04.083+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:04.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:04.077+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.079019+0000) 2019-10-13T04:00:04.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:04.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.963909+0000) 2019-10-13T04:00:04.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:04.963+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.963908+0000) 2019-10-13T04:00:04.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:04.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:04.963910+0000) 2019-10-13T04:00:05.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:05.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:05.837674+0000) 2019-10-13T04:00:07.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:07.328+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:07.328682+0000) 2019-10-13T04:00:08.121 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:08.119+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:08.119676+0000) 2019-10-13T04:00:09.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:09.024+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.024831+0000) 2019-10-13T04:00:09.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:09.084+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.085284+0000) 2019-10-13T04:00:09.087 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:09.085+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.085645+0000) 2019-10-13T04:00:09.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:09.079+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.080157+0000) 2019-10-13T04:00:09.088 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:09.085+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.085941+0000) 2019-10-13T04:00:09.088 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:09.085+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.086251+0000) 2019-10-13T04:00:09.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:09.086+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.086393+0000) 2019-10-13T04:00:09.089 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:09.086+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.086627+0000) 2019-10-13T04:00:09.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:09.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:09.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:09.082+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:09.083180+0000) 2019-10-13T04:00:12.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:12.586+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:12.587609+0000) 2019-10-13T04:00:13.949 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:00:13.949 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:00:13.958 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:00:13.958 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:00:14.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:14.082+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.082978+0000) 2019-10-13T04:00:14.091 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:14.087+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.088532+0000) 2019-10-13T04:00:14.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:14.087+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.088738+0000) 2019-10-13T04:00:14.092 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:14.087+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.088872+0000) 2019-10-13T04:00:14.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:14.087+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.089245+0000) 2019-10-13T04:00:14.092 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:14.088+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.089487+0000) 2019-10-13T04:00:14.092 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:14.088+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.089592+0000) 2019-10-13T04:00:14.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:14.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:14.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:14.085+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.085935+0000) 2019-10-13T04:00:14.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:14.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.964083+0000) 2019-10-13T04:00:14.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:14.963+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.964080+0000) 2019-10-13T04:00:14.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:14.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:14.964080+0000) 2019-10-13T04:00:15.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:15.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:15.837857+0000) 2019-10-13T04:00:17.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:17.327+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:17.328914+0000) 2019-10-13T04:00:18.121 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:18.118+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:18.119773+0000) 2019-10-13T04:00:19.034 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:19.026+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.027269+0000) 2019-10-13T04:00:19.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:19.085+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.086178+0000) 2019-10-13T04:00:19.093 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:19.090+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.091841+0000) 2019-10-13T04:00:19.094 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:19.090+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.092231+0000) 2019-10-13T04:00:19.094 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:19.090+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.092227+0000) 2019-10-13T04:00:19.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:19.090+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.092335+0000) 2019-10-13T04:00:19.094 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:19.091+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.092523+0000) 2019-10-13T04:00:19.094 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:19.091+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.092668+0000) 2019-10-13T04:00:19.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:19.087+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:19.088316+0000) 2019-10-13T04:00:19.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:19.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:22.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:22.587+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:22.587790+0000) 2019-10-13T04:00:24.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:24.093+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.095002+0000) 2019-10-13T04:00:24.096 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:24.093+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.095142+0000) 2019-10-13T04:00:24.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:24.089+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.089942+0000) 2019-10-13T04:00:24.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:24.094+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.095628+0000) 2019-10-13T04:00:24.098 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:24.094+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.095912+0000) 2019-10-13T04:00:24.098 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:24.094+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.096331+0000) 2019-10-13T04:00:24.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:24.095+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.096476+0000) 2019-10-13T04:00:24.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:24.095+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:24.099 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:24.091+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.092147+0000) 2019-10-13T04:00:24.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:24.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.964190+0000) 2019-10-13T04:00:24.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:24.963+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.964237+0000) 2019-10-13T04:00:24.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:24.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:24.964260+0000) 2019-10-13T04:00:25.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:25.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:25.838028+0000) 2019-10-13T04:00:27.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:27.328+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:27.329038+0000) 2019-10-13T04:00:28.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:28.119+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:28.119927+0000) 2019-10-13T04:00:29.036 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:29.029+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.029416+0000) 2019-10-13T04:00:29.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:29.092+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.092855+0000) 2019-10-13T04:00:29.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:29.097+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.098607+0000) 2019-10-13T04:00:29.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:29.097+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.098800+0000) 2019-10-13T04:00:29.103 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:29.097+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.098931+0000) 2019-10-13T04:00:29.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:29.097+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.099089+0000) 2019-10-13T04:00:29.104 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:29.097+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.099283+0000) 2019-10-13T04:00:29.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:29.098+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.099424+0000) 2019-10-13T04:00:29.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:29.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46318 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:29.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:29.094+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:29.094845+0000) 2019-10-13T04:00:32.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:32.587+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:32.588017+0000) 2019-10-13T04:00:34.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:34.095+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.095811+0000) 2019-10-13T04:00:34.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:34.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.101607+0000) 2019-10-13T04:00:34.103 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:34.100+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.101815+0000) 2019-10-13T04:00:34.104 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:34.100+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.102050+0000) 2019-10-13T04:00:34.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:34.100+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.102090+0000) 2019-10-13T04:00:34.104 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:34.100+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.102229+0000) 2019-10-13T04:00:34.104 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:34.101+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.102464+0000) 2019-10-13T04:00:34.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:34.097+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.098136+0000) 2019-10-13T04:00:34.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:34.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46344 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:34.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:34.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.964291+0000) 2019-10-13T04:00:34.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:34.962+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.964353+0000) 2019-10-13T04:00:34.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:34.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:34.964375+0000) 2019-10-13T04:00:35.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:35.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:35.838187+0000) 2019-10-13T04:00:37.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:37.328+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:37.329185+0000) 2019-10-13T04:00:38.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:38.119+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:38.120122+0000) 2019-10-13T04:00:39.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:39.031+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.031688+0000) 2019-10-13T04:00:39.106 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:39.103+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.104961+0000) 2019-10-13T04:00:39.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:39.103+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.105204+0000) 2019-10-13T04:00:39.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:39.103+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.105367+0000) 2019-10-13T04:00:39.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:39.099+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.100174+0000) 2019-10-13T04:00:39.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:39.104+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.105634+0000) 2019-10-13T04:00:39.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:39.104+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.105953+0000) 2019-10-13T04:00:39.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:39.104+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.106036+0000) 2019-10-13T04:00:39.108 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:39.105+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46377 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:39.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:39.101+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:39.102225+0000) 2019-10-13T04:00:42.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:42.587+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:42.588213+0000) 2019-10-13T04:00:44.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:44.101+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.103341+0000) 2019-10-13T04:00:44.111 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:44.108+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.108474+0000) 2019-10-13T04:00:44.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:44.108+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.108929+0000) 2019-10-13T04:00:44.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:44.108+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.109089+0000) 2019-10-13T04:00:44.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:44.108+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.109272+0000) 2019-10-13T04:00:44.111 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:44.109+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.109438+0000) 2019-10-13T04:00:44.112 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:44.109+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.109784+0000) 2019-10-13T04:00:44.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:44.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:44.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:44.104+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.105996+0000) 2019-10-13T04:00:44.114 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:00:44.114 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:00:44.118 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:00:44.118 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:00:44.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:44.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.964414+0000) 2019-10-13T04:00:44.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:44.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.964485+0000) 2019-10-13T04:00:44.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:44.964+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:44.964507+0000) 2019-10-13T04:00:45.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:45.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:45.838394+0000) 2019-10-13T04:00:47.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:47.329+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:47.329364+0000) 2019-10-13T04:00:48.121 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:48.120+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:48.120296+0000) 2019-10-13T04:00:49.041 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:49.032+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.034192+0000) 2019-10-13T04:00:49.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:49.104+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.106710+0000) 2019-10-13T04:00:49.115 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:49.111+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.112232+0000) 2019-10-13T04:00:49.115 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:49.112+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.112416+0000) 2019-10-13T04:00:49.115 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:49.112+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.112844+0000) 2019-10-13T04:00:49.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:49.112+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.113001+0000) 2019-10-13T04:00:49.116 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:49.112+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.113377+0000) 2019-10-13T04:00:49.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:49.113+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.113433+0000) 2019-10-13T04:00:49.117 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:49.113+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:49.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:49.107+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:49.109636+0000) 2019-10-13T04:00:52.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:52.587+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:52.588395+0000) 2019-10-13T04:00:54.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:54.108+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.109915+0000) 2019-10-13T04:00:54.117 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:54.115+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.115621+0000) 2019-10-13T04:00:54.117 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:54.115+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.115871+0000) 2019-10-13T04:00:54.117 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:54.115+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.116085+0000) 2019-10-13T04:00:54.118 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:54.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.116392+0000) 2019-10-13T04:00:54.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:54.116+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.116457+0000) 2019-10-13T04:00:54.118 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:54.116+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.116708+0000) 2019-10-13T04:00:54.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:54.116+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46464 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:54.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:54.111+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.112855+0000) 2019-10-13T04:00:54.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:54.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.964565+0000) 2019-10-13T04:00:54.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:54.964+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.964630+0000) 2019-10-13T04:00:54.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:54.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:54.964632+0000) 2019-10-13T04:00:55.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:55.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:55.838535+0000) 2019-10-13T04:00:57.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:57.328+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:57.329555+0000) 2019-10-13T04:00:58.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:58.119+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:58.120482+0000) 2019-10-13T04:00:59.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:59.035+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.036497+0000) 2019-10-13T04:00:59.120 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:00:59.112+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.113094+0000) 2019-10-13T04:00:59.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:00:59.117+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.118809+0000) 2019-10-13T04:00:59.129 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:00:59.117+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.118945+0000) 2019-10-13T04:00:59.129 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:00:59.117+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.118995+0000) 2019-10-13T04:00:59.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:00:59.117+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.119388+0000) 2019-10-13T04:00:59.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:00:59.118+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.119550+0000) 2019-10-13T04:00:59.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:00:59.118+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.119728+0000) 2019-10-13T04:00:59.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:00:59.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:00:59.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:00:59.115+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:00:59.115894+0000) 2019-10-13T04:01:02.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:02.587+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:02.588557+0000) 2019-10-13T04:01:04.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:04.115+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.116028+0000) 2019-10-13T04:01:04.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:04.120+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.121655+0000) 2019-10-13T04:01:04.123 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:04.120+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.122132+0000) 2019-10-13T04:01:04.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:04.121+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.122598+0000) 2019-10-13T04:01:04.124 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:04.121+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.122731+0000) 2019-10-13T04:01:04.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:04.121+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.122838+0000) 2019-10-13T04:01:04.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:04.121+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.122992+0000) 2019-10-13T04:01:04.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:04.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46523 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:04.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:04.118+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.119528+0000) 2019-10-13T04:01:04.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:04.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.964749+0000) 2019-10-13T04:01:04.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:04.963+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.964764+0000) 2019-10-13T04:01:04.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:04.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:04.964763+0000) 2019-10-13T04:01:05.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:05.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:05.838727+0000) 2019-10-13T04:01:07.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:07.328+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:07.329750+0000) 2019-10-13T04:01:08.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:08.119+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:08.120642+0000) 2019-10-13T04:01:09.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:09.038+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.038849+0000) 2019-10-13T04:01:09.126 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:09.123+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.125139+0000) 2019-10-13T04:01:09.127 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:09.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.125305+0000) 2019-10-13T04:01:09.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:09.124+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.125497+0000) 2019-10-13T04:01:09.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:09.124+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.125630+0000) 2019-10-13T04:01:09.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:09.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.120563+0000) 2019-10-13T04:01:09.128 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:09.124+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.126364+0000) 2019-10-13T04:01:09.128 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:09.125+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.126526+0000) 2019-10-13T04:01:09.129 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:09.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:09.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:09.121+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:09.122646+0000) 2019-10-13T04:01:12.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:12.588+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:12.588756+0000) 2019-10-13T04:01:14.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:14.122+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.123119+0000) 2019-10-13T04:01:14.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:14.127+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.129179+0000) 2019-10-13T04:01:14.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:14.127+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.129301+0000) 2019-10-13T04:01:14.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:14.128+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.129446+0000) 2019-10-13T04:01:14.131 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:14.128+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.129630+0000) 2019-10-13T04:01:14.132 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:14.128+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.129826+0000) 2019-10-13T04:01:14.132 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:14.128+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.129893+0000) 2019-10-13T04:01:14.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:14.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:14.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:14.124+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.125402+0000) 2019-10-13T04:01:14.215 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:01:14.215 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:01:14.233 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:01:14.233 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:01:14.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:14.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.964901+0000) 2019-10-13T04:01:14.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:14.964+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.964937+0000) 2019-10-13T04:01:14.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:14.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:14.964939+0000) 2019-10-13T04:01:15.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:15.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:15.838919+0000) 2019-10-13T04:01:17.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:17.329+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:17.329945+0000) 2019-10-13T04:01:18.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:18.119+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:18.120848+0000) 2019-10-13T04:01:19.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:19.040+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.041001+0000) 2019-10-13T04:01:19.134 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:19.130+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.131636+0000) 2019-10-13T04:01:19.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:19.130+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.132161+0000) 2019-10-13T04:01:19.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:19.131+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.132394+0000) 2019-10-13T04:01:19.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:19.131+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.132599+0000) 2019-10-13T04:01:19.136 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:19.131+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.132901+0000) 2019-10-13T04:01:19.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:19.131+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.133096+0000) 2019-10-13T04:01:19.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:19.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:19.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:19.126+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.127090+0000) 2019-10-13T04:01:19.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:19.127+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:19.128638+0000) 2019-10-13T04:01:22.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:22.588+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:22.588942+0000) 2019-10-13T04:01:24.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:24.133+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.134964+0000) 2019-10-13T04:01:24.138 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:24.129+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.130602+0000) 2019-10-13T04:01:24.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:24.134+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.135562+0000) 2019-10-13T04:01:24.139 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:24.134+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.135727+0000) 2019-10-13T04:01:24.139 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:24.134+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.136094+0000) 2019-10-13T04:01:24.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:24.134+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.136224+0000) 2019-10-13T04:01:24.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:24.135+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.136942+0000) 2019-10-13T04:01:24.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:24.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:24.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:24.132+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.133292+0000) 2019-10-13T04:01:24.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:24.964+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.965104+0000) 2019-10-13T04:01:24.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:24.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.965107+0000) 2019-10-13T04:01:24.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:24.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:24.965098+0000) 2019-10-13T04:01:25.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:25.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:25.839103+0000) 2019-10-13T04:01:27.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:27.329+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:27.330140+0000) 2019-10-13T04:01:28.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:28.120+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:28.121038+0000) 2019-10-13T04:01:29.050 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:29.043+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.043267+0000) 2019-10-13T04:01:29.140 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:29.132+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.133481+0000) 2019-10-13T04:01:29.141 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:29.138+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.139337+0000) 2019-10-13T04:01:29.141 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:29.139+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.139707+0000) 2019-10-13T04:01:29.141 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:29.139+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.139759+0000) 2019-10-13T04:01:29.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:29.139+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.139899+0000) 2019-10-13T04:01:29.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:29.134+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.135406+0000) 2019-10-13T04:01:29.142 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:29.139+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.140085+0000) 2019-10-13T04:01:29.142 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:29.140+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:29.140440+0000) 2019-10-13T04:01:29.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:29.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:32.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:32.589+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:32.589149+0000) 2019-10-13T04:01:34.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:34.141+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.142151+0000) 2019-10-13T04:01:34.143 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:34.141+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.142292+0000) 2019-10-13T04:01:34.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:34.136+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.137221+0000) 2019-10-13T04:01:34.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:34.142+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.142712+0000) 2019-10-13T04:01:34.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:34.142+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.142885+0000) 2019-10-13T04:01:34.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:34.142+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.142942+0000) 2019-10-13T04:01:34.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:34.142+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.143332+0000) 2019-10-13T04:01:34.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:34.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:34.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:34.139+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.140088+0000) 2019-10-13T04:01:34.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:34.965+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.965304+0000) 2019-10-13T04:01:34.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:34.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.965303+0000) 2019-10-13T04:01:34.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:34.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:34.965303+0000) 2019-10-13T04:01:35.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:35.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:35.839290+0000) 2019-10-13T04:01:37.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:37.330+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:37.330331+0000) 2019-10-13T04:01:38.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:38.120+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:38.121235+0000) 2019-10-13T04:01:39.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:39.044+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.045263+0000) 2019-10-13T04:01:39.146 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:39.144+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.145012+0000) 2019-10-13T04:01:39.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:39.144+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.145157+0000) 2019-10-13T04:01:39.147 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:39.145+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.145432+0000) 2019-10-13T04:01:39.147 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:39.137+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.139499+0000) 2019-10-13T04:01:39.148 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:39.145+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.145557+0000) 2019-10-13T04:01:39.148 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:39.145+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.145941+0000) 2019-10-13T04:01:39.148 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:39.145+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.146035+0000) 2019-10-13T04:01:39.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:39.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:39.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:39.140+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:39.141924+0000) 2019-10-13T04:01:42.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:42.588+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:42.589320+0000) 2019-10-13T04:01:44.149 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:44.146+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.147983+0000) 2019-10-13T04:01:44.149 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:44.146+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.148242+0000) 2019-10-13T04:01:44.150 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:44.146+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.148366+0000) 2019-10-13T04:01:44.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:44.141+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.143539+0000) 2019-10-13T04:01:44.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:44.147+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.148628+0000) 2019-10-13T04:01:44.151 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:44.147+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.149315+0000) 2019-10-13T04:01:44.151 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:44.148+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.149474+0000) 2019-10-13T04:01:44.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:44.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:44.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:44.143+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.145623+0000) 2019-10-13T04:01:44.367 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:01:44.367 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:01:44.372 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:01:44.372 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:01:44.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:44.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.965470+0000) 2019-10-13T04:01:44.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:44.964+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.965464+0000) 2019-10-13T04:01:44.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:44.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:44.965466+0000) 2019-10-13T04:01:45.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:45.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:45.839474+0000) 2019-10-13T04:01:47.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:47.329+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:47.330519+0000) 2019-10-13T04:01:48.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:48.120+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:48.121426+0000) 2019-10-13T04:01:49.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:49.046+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.047492+0000) 2019-10-13T04:01:49.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:49.150+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.152020+0000) 2019-10-13T04:01:49.154 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:49.150+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.152350+0000) 2019-10-13T04:01:49.154 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:49.151+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.152504+0000) 2019-10-13T04:01:49.155 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:49.146+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.147586+0000) 2019-10-13T04:01:49.155 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:49.151+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.152801+0000) 2019-10-13T04:01:49.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:49.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:49.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:49.151+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.152886+0000) 2019-10-13T04:01:49.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:49.151+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.153223+0000) 2019-10-13T04:01:49.156 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:49.148+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:49.149056+0000) 2019-10-13T04:01:52.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:52.588+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:52.589467+0000) 2019-10-13T04:01:54.155 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:54.153+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.154403+0000) 2019-10-13T04:01:54.156 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:54.148+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.149275+0000) 2019-10-13T04:01:54.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:54.153+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.155199+0000) 2019-10-13T04:01:54.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:54.153+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.155352+0000) 2019-10-13T04:01:54.157 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:54.154+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.155549+0000) 2019-10-13T04:01:54.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:54.154+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.156134+0000) 2019-10-13T04:01:54.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:54.154+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.156299+0000) 2019-10-13T04:01:54.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:54.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:54.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:54.151+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.152671+0000) 2019-10-13T04:01:54.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:54.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.965653+0000) 2019-10-13T04:01:54.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:54.964+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.965657+0000) 2019-10-13T04:01:54.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:54.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:54.965658+0000) 2019-10-13T04:01:55.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:55.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:55.839679+0000) 2019-10-13T04:01:57.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:57.329+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:57.330696+0000) 2019-10-13T04:01:58.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:58.120+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:58.121610+0000) 2019-10-13T04:01:59.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:59.049+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.049804+0000) 2019-10-13T04:01:59.159 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:01:59.157+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.158492+0000) 2019-10-13T04:01:59.161 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:01:59.152+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.153358+0000) 2019-10-13T04:01:59.161 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:01:59.157+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.158886+0000) 2019-10-13T04:01:59.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:01:59.157+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.159080+0000) 2019-10-13T04:01:59.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:01:59.157+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.159142+0000) 2019-10-13T04:01:59.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:01:59.157+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.159239+0000) 2019-10-13T04:01:59.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:01:59.158+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.159524+0000) 2019-10-13T04:01:59.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:01:59.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:01:59.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:01:59.155+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:01:59.155779+0000) 2019-10-13T04:02:02.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:02.589+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:02.589638+0000) 2019-10-13T04:02:04.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:04.160+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.162050+0000) 2019-10-13T04:02:04.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:04.160+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.162165+0000) 2019-10-13T04:02:04.164 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:04.160+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.162306+0000) 2019-10-13T04:02:04.164 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:04.161+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.162460+0000) 2019-10-13T04:02:04.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:04.156+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.157535+0000) 2019-10-13T04:02:04.165 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:04.161+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.162748+0000) 2019-10-13T04:02:04.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:04.161+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.163066+0000) 2019-10-13T04:02:04.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:04.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:04.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:04.159+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.160073+0000) 2019-10-13T04:02:04.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:04.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.965834+0000) 2019-10-13T04:02:04.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:04.965+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.965842+0000) 2019-10-13T04:02:04.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:04.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:04.965842+0000) 2019-10-13T04:02:05.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:05.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:05.839849+0000) 2019-10-13T04:02:07.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:07.330+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:07.330855+0000) 2019-10-13T04:02:08.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:08.121+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:08.121843+0000) 2019-10-13T04:02:09.060 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:09.051+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.051917+0000) 2019-10-13T04:02:09.167 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:09.159+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.160708+0000) 2019-10-13T04:02:09.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:09.164+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.165912+0000) 2019-10-13T04:02:09.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:09.164+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.166210+0000) 2019-10-13T04:02:09.168 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:09.164+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.166322+0000) 2019-10-13T04:02:09.168 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:09.165+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.166613+0000) 2019-10-13T04:02:09.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:09.161+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.162040+0000) 2019-10-13T04:02:09.169 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:09.165+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.167095+0000) 2019-10-13T04:02:09.169 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:09.165+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:09.167266+0000) 2019-10-13T04:02:09.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:09.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:12.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:12.589+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:12.589811+0000) 2019-10-13T04:02:14.170 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:14.168+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.169073+0000) 2019-10-13T04:02:14.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:14.168+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.169384+0000) 2019-10-13T04:02:14.171 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:14.163+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.164449+0000) 2019-10-13T04:02:14.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:14.169+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.169734+0000) 2019-10-13T04:02:14.172 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:14.169+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.169906+0000) 2019-10-13T04:02:14.172 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:14.169+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.170172+0000) 2019-10-13T04:02:14.172 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:14.170+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.170418+0000) 2019-10-13T04:02:14.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:14.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:14.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:14.166+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.167217+0000) 2019-10-13T04:02:14.507 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:02:14.508 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:02:14.512 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:02:14.512 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:02:14.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:14.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.965987+0000) 2019-10-13T04:02:14.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:14.965+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.966001+0000) 2019-10-13T04:02:14.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:14.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:14.966001+0000) 2019-10-13T04:02:15.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:15.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:15.840033+0000) 2019-10-13T04:02:17.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:17.330+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:17.331012+0000) 2019-10-13T04:02:18.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:18.121+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:18.121992+0000) 2019-10-13T04:02:19.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:19.053+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.054116+0000) 2019-10-13T04:02:19.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:19.173+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.173454+0000) 2019-10-13T04:02:19.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:19.173+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.173697+0000) 2019-10-13T04:02:19.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:19.173+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.174019+0000) 2019-10-13T04:02:19.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:19.167+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.168499+0000) 2019-10-13T04:02:19.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:19.173+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.174242+0000) 2019-10-13T04:02:19.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:19.173+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.174365+0000) 2019-10-13T04:02:19.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:19.174+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.174560+0000) 2019-10-13T04:02:19.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:19.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 46977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:19.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:19.171+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:19.171900+0000) 2019-10-13T04:02:22.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:22.589+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:22.589986+0000) 2019-10-13T04:02:24.178 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:24.170+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.171354+0000) 2019-10-13T04:02:24.178 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:24.176+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.176918+0000) 2019-10-13T04:02:24.179 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:24.176+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.177176+0000) 2019-10-13T04:02:24.179 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:24.176+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.177317+0000) 2019-10-13T04:02:24.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:24.177+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.177557+0000) 2019-10-13T04:02:24.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:24.177+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.177759+0000) 2019-10-13T04:02:24.180 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:24.177+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.178035+0000) 2019-10-13T04:02:24.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:24.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47005 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:24.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:24.174+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.175538+0000) 2019-10-13T04:02:24.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:24.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.966179+0000) 2019-10-13T04:02:24.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:24.966+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.966190+0000) 2019-10-13T04:02:24.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:24.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:24.966188+0000) 2019-10-13T04:02:25.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:25.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:25.840221+0000) 2019-10-13T04:02:27.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:27.331+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:27.331210+0000) 2019-10-13T04:02:28.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:28.121+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:28.122202+0000) 2019-10-13T04:02:29.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:29.055+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.056474+0000) 2019-10-13T04:02:29.181 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:29.178+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.179909+0000) 2019-10-13T04:02:29.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:29.172+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.174645+0000) 2019-10-13T04:02:29.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:29.178+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.180249+0000) 2019-10-13T04:02:29.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:29.178+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.180368+0000) 2019-10-13T04:02:29.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:29.179+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.180675+0000) 2019-10-13T04:02:29.183 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:29.179+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.180856+0000) 2019-10-13T04:02:29.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:29.179+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.181310+0000) 2019-10-13T04:02:29.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:29.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:29.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:29.175+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:29.177563+0000) 2019-10-13T04:02:32.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:32.589+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:32.590168+0000) 2019-10-13T04:02:34.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:34.176+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.178268+0000) 2019-10-13T04:02:34.185 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:34.182+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.183493+0000) 2019-10-13T04:02:34.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:34.182+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.183784+0000) 2019-10-13T04:02:34.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:34.182+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.183933+0000) 2019-10-13T04:02:34.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:34.182+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.184077+0000) 2019-10-13T04:02:34.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:34.182+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.184338+0000) 2019-10-13T04:02:34.187 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:34.183+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.184794+0000) 2019-10-13T04:02:34.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:34.183+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:34.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:34.179+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.181144+0000) 2019-10-13T04:02:34.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:34.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.966340+0000) 2019-10-13T04:02:34.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:34.964+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.966344+0000) 2019-10-13T04:02:34.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:34.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:34.966343+0000) 2019-10-13T04:02:35.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:35.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:35.840372+0000) 2019-10-13T04:02:37.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:37.330+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:37.331433+0000) 2019-10-13T04:02:38.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:38.121+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:38.122388+0000) 2019-10-13T04:02:39.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:39.057+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.058754+0000) 2019-10-13T04:02:39.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:39.185+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.186980+0000) 2019-10-13T04:02:39.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:39.185+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.187249+0000) 2019-10-13T04:02:39.190 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:39.186+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.187548+0000) 2019-10-13T04:02:39.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:39.186+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.187729+0000) 2019-10-13T04:02:39.202 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:39.186+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.187939+0000) 2019-10-13T04:02:39.202 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:39.186+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.188138+0000) 2019-10-13T04:02:39.203 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:39.180+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.182031+0000) 2019-10-13T04:02:39.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:39.182+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:39.183876+0000) 2019-10-13T04:02:39.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:39.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:42.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:42.589+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:42.590373+0000) 2019-10-13T04:02:44.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:44.189+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.190582+0000) 2019-10-13T04:02:44.193 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:44.189+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.190680+0000) 2019-10-13T04:02:44.193 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:44.189+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.190862+0000) 2019-10-13T04:02:44.194 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:44.189+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.190967+0000) 2019-10-13T04:02:44.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:44.189+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.191337+0000) 2019-10-13T04:02:44.194 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:44.189+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.191389+0000) 2019-10-13T04:02:44.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:44.190+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:44.195 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:44.185+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.185794+0000) 2019-10-13T04:02:44.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:44.186+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.187372+0000) 2019-10-13T04:02:44.629 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:02:44.629 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:02:44.633 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:02:44.633 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:02:44.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:44.965+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.966506+0000) 2019-10-13T04:02:44.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:44.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.966506+0000) 2019-10-13T04:02:44.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:44.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:44.966501+0000) 2019-10-13T04:02:45.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:45.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:45.840527+0000) 2019-10-13T04:02:47.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:47.331+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:47.331619+0000) 2019-10-13T04:02:48.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:48.122+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:48.122610+0000) 2019-10-13T04:02:49.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:49.060+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.061124+0000) 2019-10-13T04:02:49.194 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:49.191+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.192861+0000) 2019-10-13T04:02:49.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:49.191+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.192994+0000) 2019-10-13T04:02:49.195 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:49.191+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.193295+0000) 2019-10-13T04:02:49.195 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:49.187+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.188489+0000) 2019-10-13T04:02:49.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:49.188+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.189663+0000) 2019-10-13T04:02:49.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:49.192+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.193681+0000) 2019-10-13T04:02:49.205 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:49.192+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.194038+0000) 2019-10-13T04:02:49.205 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:49.192+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:49.194209+0000) 2019-10-13T04:02:49.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:49.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:52.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:52.590+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:52.590568+0000) 2019-10-13T04:02:54.197 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:54.194+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.196177+0000) 2019-10-13T04:02:54.197 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:54.195+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.196499+0000) 2019-10-13T04:02:54.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:54.195+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.196844+0000) 2019-10-13T04:02:54.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:54.195+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.197056+0000) 2019-10-13T04:02:54.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:54.195+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.197235+0000) 2019-10-13T04:02:54.199 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:54.196+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.197417+0000) 2019-10-13T04:02:54.201 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:54.191+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.192414+0000) 2019-10-13T04:02:54.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:54.192+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.193667+0000) 2019-10-13T04:02:54.201 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:54.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:54.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:54.966+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.966672+0000) 2019-10-13T04:02:54.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:54.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.966672+0000) 2019-10-13T04:02:54.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:54.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:54.966678+0000) 2019-10-13T04:02:55.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:55.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:55.840704+0000) 2019-10-13T04:02:57.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:57.331+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:57.331834+0000) 2019-10-13T04:02:58.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:58.122+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:58.122800+0000) 2019-10-13T04:02:59.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:59.063+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.063428+0000) 2019-10-13T04:02:59.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:02:59.199+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.199736+0000) 2019-10-13T04:02:59.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:02:59.199+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.199928+0000) 2019-10-13T04:02:59.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:02:59.200+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.200666+0000) 2019-10-13T04:02:59.202 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:02:59.200+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.200926+0000) 2019-10-13T04:02:59.202 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:02:59.195+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.195982+0000) 2019-10-13T04:02:59.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:02:59.201+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.201797+0000) 2019-10-13T04:02:59.203 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:02:59.201+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.201925+0000) 2019-10-13T04:02:59.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:02:59.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:02:59.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:02:59.197+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:02:59.198260+0000) 2019-10-13T04:03:02.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:02.590+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:02.590767+0000) 2019-10-13T04:03:04.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:04.197+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.198674+0000) 2019-10-13T04:03:04.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:04.203+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.204263+0000) 2019-10-13T04:03:04.206 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:04.204+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.204609+0000) 2019-10-13T04:03:04.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:04.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:04.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:04.204+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.204739+0000) 2019-10-13T04:03:04.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:04.204+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.204911+0000) 2019-10-13T04:03:04.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:04.204+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.205064+0000) 2019-10-13T04:03:04.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:04.205+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.205408+0000) 2019-10-13T04:03:04.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:04.200+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.201231+0000) 2019-10-13T04:03:04.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:04.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.966826+0000) 2019-10-13T04:03:04.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:04.966+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.966811+0000) 2019-10-13T04:03:04.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:04.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:04.966826+0000) 2019-10-13T04:03:05.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:05.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:05.840893+0000) 2019-10-13T04:03:07.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:07.331+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:07.332045+0000) 2019-10-13T04:03:08.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:08.122+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:08.122998+0000) 2019-10-13T04:03:09.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:09.065+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.065756+0000) 2019-10-13T04:03:09.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:09.205+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.206257+0000) 2019-10-13T04:03:09.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:09.201+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.201905+0000) 2019-10-13T04:03:09.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:09.207+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.207580+0000) 2019-10-13T04:03:09.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:09.207+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.207633+0000) 2019-10-13T04:03:09.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:09.207+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.208099+0000) 2019-10-13T04:03:09.210 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:09.207+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:09.210 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:09.208+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.208526+0000) 2019-10-13T04:03:09.210 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:09.208+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.208744+0000) 2019-10-13T04:03:09.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:09.203+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:09.204555+0000) 2019-10-13T04:03:12.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:12.590+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:12.590969+0000) 2019-10-13T04:03:14.211 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:14.204+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.204825+0000) 2019-10-13T04:03:14.212 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:14.209+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.210517+0000) 2019-10-13T04:03:14.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:14.209+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.210638+0000) 2019-10-13T04:03:14.213 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:14.209+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.210778+0000) 2019-10-13T04:03:14.213 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:14.209+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.211087+0000) 2019-10-13T04:03:14.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:14.209+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.210978+0000) 2019-10-13T04:03:14.213 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:14.209+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.211191+0000) 2019-10-13T04:03:14.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:14.206+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.207098+0000) 2019-10-13T04:03:14.214 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:14.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:14.751 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:03:14.751 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:03:14.756 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:03:14.756 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:03:14.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:14.965+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.966963+0000) 2019-10-13T04:03:14.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:14.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.966961+0000) 2019-10-13T04:03:14.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:14.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:14.966973+0000) 2019-10-13T04:03:15.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:15.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:15.841083+0000) 2019-10-13T04:03:17.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:17.331+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:17.332234+0000) 2019-10-13T04:03:18.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:18.121+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:18.123220+0000) 2019-10-13T04:03:19.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:19.066+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.068038+0000) 2019-10-13T04:03:19.215 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:19.212+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.213540+0000) 2019-10-13T04:03:19.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:19.206+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.208497+0000) 2019-10-13T04:03:19.216 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:19.212+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.213925+0000) 2019-10-13T04:03:19.216 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:19.212+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.214061+0000) 2019-10-13T04:03:19.217 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:19.213+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.214589+0000) 2019-10-13T04:03:19.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:19.213+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.214879+0000) 2019-10-13T04:03:19.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:19.213+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.215109+0000) 2019-10-13T04:03:19.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:19.208+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:19.210628+0000) 2019-10-13T04:03:19.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:19.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:22.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:22.590+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:22.591151+0000) 2019-10-13T04:03:24.219 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:24.216+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.217662+0000) 2019-10-13T04:03:24.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:24.216+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.217819+0000) 2019-10-13T04:03:24.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:24.216+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.218139+0000) 2019-10-13T04:03:24.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:24.210+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.212692+0000) 2019-10-13T04:03:24.220 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:24.217+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.218503+0000) 2019-10-13T04:03:24.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:24.217+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.218741+0000) 2019-10-13T04:03:24.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:24.217+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.219127+0000) 2019-10-13T04:03:24.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:24.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:24.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:24.213+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.215057+0000) 2019-10-13T04:03:24.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:24.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.967122+0000) 2019-10-13T04:03:24.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:24.966+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.967121+0000) 2019-10-13T04:03:24.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:24.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:24.967121+0000) 2019-10-13T04:03:25.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:25.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:25.841273+0000) 2019-10-13T04:03:27.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:27.331+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:27.332414+0000) 2019-10-13T04:03:28.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:28.122+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:28.123387+0000) 2019-10-13T04:03:29.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:29.069+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.070374+0000) 2019-10-13T04:03:29.223 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:29.214+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.216318+0000) 2019-10-13T04:03:29.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:29.219+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.221350+0000) 2019-10-13T04:03:29.224 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:29.220+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.221431+0000) 2019-10-13T04:03:29.224 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:29.220+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.222253+0000) 2019-10-13T04:03:29.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:29.220+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.222344+0000) 2019-10-13T04:03:29.225 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:29.221+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.222544+0000) 2019-10-13T04:03:29.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:29.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:29.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:29.221+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.222841+0000) 2019-10-13T04:03:29.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:29.217+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:29.219451+0000) 2019-10-13T04:03:32.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:32.590+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:32.591313+0000) 2019-10-13T04:03:34.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:34.223+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.224414+0000) 2019-10-13T04:03:34.226 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:34.218+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.219430+0000) 2019-10-13T04:03:34.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:34.223+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.225014+0000) 2019-10-13T04:03:34.227 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:34.223+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.225274+0000) 2019-10-13T04:03:34.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:34.224+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.225581+0000) 2019-10-13T04:03:34.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:34.224+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.225893+0000) 2019-10-13T04:03:34.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:34.224+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.226052+0000) 2019-10-13T04:03:34.228 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:34.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:34.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:34.220+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.221297+0000) 2019-10-13T04:03:34.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:34.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.967277+0000) 2019-10-13T04:03:34.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:34.966+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.967279+0000) 2019-10-13T04:03:34.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:34.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:34.967276+0000) 2019-10-13T04:03:35.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:35.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:35.841433+0000) 2019-10-13T04:03:37.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:37.332+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:37.332608+0000) 2019-10-13T04:03:38.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:38.123+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:38.123540+0000) 2019-10-13T04:03:39.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:39.072+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.072669+0000) 2019-10-13T04:03:39.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:39.222+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.222865+0000) 2019-10-13T04:03:39.230 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:39.227+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.228460+0000) 2019-10-13T04:03:39.230 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:39.227+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.229029+0000) 2019-10-13T04:03:39.231 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:39.227+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.229351+0000) 2019-10-13T04:03:39.231 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:39.228+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.229539+0000) 2019-10-13T04:03:39.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:39.228+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.229734+0000) 2019-10-13T04:03:39.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:39.228+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.229935+0000) 2019-10-13T04:03:39.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:39.228+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:39.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:39.225+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:39.225825+0000) 2019-10-13T04:03:42.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:42.590+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:42.591461+0000) 2019-10-13T04:03:44.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:44.231+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.232068+0000) 2019-10-13T04:03:44.234 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:44.226+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.227114+0000) 2019-10-13T04:03:44.234 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:44.232+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.232564+0000) 2019-10-13T04:03:44.235 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:44.232+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.232686+0000) 2019-10-13T04:03:44.235 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:44.232+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.232817+0000) 2019-10-13T04:03:44.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:44.232+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.233234+0000) 2019-10-13T04:03:44.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:44.233+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.233394+0000) 2019-10-13T04:03:44.236 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:44.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:44.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:44.228+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.229564+0000) 2019-10-13T04:03:44.874 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:03:44.874 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:03:44.889 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:03:44.889 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:03:44.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:44.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.967486+0000) 2019-10-13T04:03:44.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:44.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.967487+0000) 2019-10-13T04:03:44.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:44.967+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:44.967491+0000) 2019-10-13T04:03:45.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:45.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:45.841628+0000) 2019-10-13T04:03:47.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:47.332+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:47.332831+0000) 2019-10-13T04:03:48.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:48.123+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:48.123725+0000) 2019-10-13T04:03:49.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:49.074+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.074848+0000) 2019-10-13T04:03:49.236 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:49.229+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.230238+0000) 2019-10-13T04:03:49.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:49.235+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.235969+0000) 2019-10-13T04:03:49.237 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:49.235+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.236150+0000) 2019-10-13T04:03:49.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:49.235+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.236278+0000) 2019-10-13T04:03:49.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:49.236+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.236466+0000) 2019-10-13T04:03:49.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:49.236+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.237054+0000) 2019-10-13T04:03:49.239 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:49.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:49.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:49.237+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.237515+0000) 2019-10-13T04:03:49.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:49.231+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:49.232453+0000) 2019-10-13T04:03:52.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:52.590+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:52.591647+0000) 2019-10-13T04:03:54.240 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:54.233+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.233778+0000) 2019-10-13T04:03:54.241 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:54.239+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.239502+0000) 2019-10-13T04:03:54.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:54.238+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.239356+0000) 2019-10-13T04:03:54.241 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:54.239+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.239821+0000) 2019-10-13T04:03:54.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:54.239+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.239984+0000) 2019-10-13T04:03:54.242 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:54.239+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.240219+0000) 2019-10-13T04:03:54.242 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:54.240+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.240627+0000) 2019-10-13T04:03:54.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:54.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:03:54.243 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:54.236+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.237372+0000) 2019-10-13T04:03:54.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:54.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.967659+0000) 2019-10-13T04:03:54.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:54.967+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.967663+0000) 2019-10-13T04:03:54.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:54.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:54.967658+0000) 2019-10-13T04:03:55.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:55.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:55.841813+0000) 2019-10-13T04:03:57.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:57.332+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:57.333057+0000) 2019-10-13T04:03:58.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:58.123+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:58.123898+0000) 2019-10-13T04:03:59.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:59.076+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.077123+0000) 2019-10-13T04:03:59.243 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:03:59.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.237217+0000) 2019-10-13T04:03:59.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:03:59.241+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.242863+0000) 2019-10-13T04:03:59.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:03:59.241+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.242965+0000) 2019-10-13T04:03:59.245 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:03:59.241+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.242735+0000) 2019-10-13T04:03:59.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:03:59.241+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.243264+0000) 2019-10-13T04:03:59.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:03:59.242+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.243721+0000) 2019-10-13T04:03:59.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:03:59.239+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.239739+0000) 2019-10-13T04:03:59.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:03:59.242+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:03:59.244075+0000) 2019-10-13T04:03:59.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:03:59.242+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:02.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:02.591+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:02.591825+0000) 2019-10-13T04:04:04.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:04.239+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.240542+0000) 2019-10-13T04:04:04.248 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:04.244+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.246123+0000) 2019-10-13T04:04:04.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:04.244+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.246206+0000) 2019-10-13T04:04:04.248 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:04.245+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.246475+0000) 2019-10-13T04:04:04.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:04.245+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.246792+0000) 2019-10-13T04:04:04.249 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:04.245+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.246856+0000) 2019-10-13T04:04:04.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:04.245+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.247204+0000) 2019-10-13T04:04:04.249 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:04.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:04.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:04.243+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.244411+0000) 2019-10-13T04:04:04.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:04.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.967754+0000) 2019-10-13T04:04:04.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:04.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.967792+0000) 2019-10-13T04:04:04.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:04.966+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:04.967801+0000) 2019-10-13T04:04:05.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:05.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:05.841975+0000) 2019-10-13T04:04:07.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:07.332+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:07.333252+0000) 2019-10-13T04:04:08.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:08.123+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:08.124072+0000) 2019-10-13T04:04:09.086 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:09.077+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.079338+0000) 2019-10-13T04:04:09.250 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:09.242+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.243913+0000) 2019-10-13T04:04:09.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:09.248+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.249679+0000) 2019-10-13T04:04:09.251 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:09.248+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.249852+0000) 2019-10-13T04:04:09.251 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:09.248+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.249998+0000) 2019-10-13T04:04:09.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:09.248+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.250310+0000) 2019-10-13T04:04:09.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:09.249+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.250631+0000) 2019-10-13T04:04:09.252 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:09.249+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.250883+0000) 2019-10-13T04:04:09.253 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:09.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47638 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:09.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:09.246+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:09.247867+0000) 2019-10-13T04:04:12.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:12.591+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:12.591954+0000) 2019-10-13T04:04:14.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:14.251+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.253164+0000) 2019-10-13T04:04:14.256 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:14.251+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.253357+0000) 2019-10-13T04:04:14.256 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:14.252+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.253501+0000) 2019-10-13T04:04:14.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:14.252+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.253763+0000) 2019-10-13T04:04:14.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:14.252+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.254258+0000) 2019-10-13T04:04:14.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:14.253+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.254672+0000) 2019-10-13T04:04:14.257 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:14.247+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.249107+0000) 2019-10-13T04:04:14.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:14.248+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.250722+0000) 2019-10-13T04:04:14.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:14.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:14.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:14.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.967928+0000) 2019-10-13T04:04:14.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:14.967+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.967938+0000) 2019-10-13T04:04:14.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:14.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:14.967936+0000) 2019-10-13T04:04:14.996 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:04:14.996 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:15.001 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:04:15.001 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:15.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:15.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:15.842155+0000) 2019-10-13T04:04:17.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:17.332+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:17.333412+0000) 2019-10-13T04:04:18.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:18.123+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:18.124252+0000) 2019-10-13T04:04:19.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:19.080+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.081773+0000) 2019-10-13T04:04:19.258 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:19.255+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.257036+0000) 2019-10-13T04:04:19.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:19.255+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.257188+0000) 2019-10-13T04:04:19.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:19.256+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.257393+0000) 2019-10-13T04:04:19.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:19.256+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.257605+0000) 2019-10-13T04:04:19.260 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:19.251+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.252824+0000) 2019-10-13T04:04:19.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:19.256+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.257823+0000) 2019-10-13T04:04:19.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:19.256+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:19.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:19.256+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.258273+0000) 2019-10-13T04:04:19.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:19.252+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:19.254353+0000) 2019-10-13T04:04:22.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:22.591+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:22.592115+0000) 2019-10-13T04:04:24.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:24.258+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.259858+0000) 2019-10-13T04:04:24.261 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:24.258+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.260125+0000) 2019-10-13T04:04:24.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:24.258+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.260370+0000) 2019-10-13T04:04:24.263 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:24.254+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.255319+0000) 2019-10-13T04:04:24.263 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:24.259+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.260475+0000) 2019-10-13T04:04:24.263 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:24.259+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.260959+0000) 2019-10-13T04:04:24.264 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:24.260+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.261566+0000) 2019-10-13T04:04:24.265 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:24.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:24.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:24.257+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.258017+0000) 2019-10-13T04:04:24.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:24.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.968071+0000) 2019-10-13T04:04:24.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:24.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.968071+0000) 2019-10-13T04:04:24.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:24.967+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:24.968071+0000) 2019-10-13T04:04:25.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:25.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:25.842363+0000) 2019-10-13T04:04:27.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:27.333+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:27.333589+0000) 2019-10-13T04:04:28.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:28.124+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:28.124450+0000) 2019-10-13T04:04:29.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:29.083+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.083986+0000) 2019-10-13T04:04:29.265 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:29.263+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.263900+0000) 2019-10-13T04:04:29.266 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:29.263+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.264024+0000) 2019-10-13T04:04:29.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:29.263+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.264248+0000) 2019-10-13T04:04:29.266 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:29.263+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.264360+0000) 2019-10-13T04:04:29.266 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:29.264+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.264605+0000) 2019-10-13T04:04:29.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:29.264+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.264679+0000) 2019-10-13T04:04:29.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:29.258+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.259493+0000) 2019-10-13T04:04:29.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:29.261+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:29.261864+0000) 2019-10-13T04:04:29.269 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:29.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:32.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:32.591+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:32.592302+0000) 2019-10-13T04:04:34.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:34.261+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.262029+0000) 2019-10-13T04:04:34.269 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:34.266+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.267258+0000) 2019-10-13T04:04:34.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:34.267+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.267547+0000) 2019-10-13T04:04:34.269 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:34.267+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.267808+0000) 2019-10-13T04:04:34.270 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:34.267+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.268039+0000) 2019-10-13T04:04:34.270 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:34.268+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.268537+0000) 2019-10-13T04:04:34.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:34.268+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.268811+0000) 2019-10-13T04:04:34.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:34.268+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:34.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:34.264+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.265643+0000) 2019-10-13T04:04:34.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:34.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.968226+0000) 2019-10-13T04:04:34.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:34.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.968227+0000) 2019-10-13T04:04:34.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:34.968+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:34.968226+0000) 2019-10-13T04:04:35.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:35.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:35.842560+0000) 2019-10-13T04:04:37.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:37.333+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:37.333782+0000) 2019-10-13T04:04:38.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:38.124+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:38.124637+0000) 2019-10-13T04:04:39.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:39.085+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.086369+0000) 2019-10-13T04:04:39.272 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:39.270+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.270587+0000) 2019-10-13T04:04:39.273 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:39.270+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.270714+0000) 2019-10-13T04:04:39.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:39.270+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.270910+0000) 2019-10-13T04:04:39.273 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:39.270+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.271031+0000) 2019-10-13T04:04:39.273 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:39.270+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.271239+0000) 2019-10-13T04:04:39.273 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:39.271+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.271440+0000) 2019-10-13T04:04:39.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:39.265+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.266159+0000) 2019-10-13T04:04:39.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:39.267+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:39.267857+0000) 2019-10-13T04:04:39.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:39.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47817 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:42.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:42.591+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:42.592488+0000) 2019-10-13T04:04:44.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:44.272+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.273705+0000) 2019-10-13T04:04:44.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:44.272+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.273884+0000) 2019-10-13T04:04:44.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:44.272+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.273991+0000) 2019-10-13T04:04:44.277 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:44.272+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.274190+0000) 2019-10-13T04:04:44.277 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:44.273+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.274443+0000) 2019-10-13T04:04:44.278 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:44.273+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.274603+0000) 2019-10-13T04:04:44.279 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:44.269+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.269869+0000) 2019-10-13T04:04:44.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:44.270+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.271014+0000) 2019-10-13T04:04:44.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:44.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47845 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:44.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:44.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.968419+0000) 2019-10-13T04:04:44.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:44.967+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.968420+0000) 2019-10-13T04:04:44.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:44.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:44.968423+0000) 2019-10-13T04:04:45.127 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:04:45.127 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:45.131 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:04:45.132 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:45.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:45.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:45.842775+0000) 2019-10-13T04:04:47.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:47.332+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:47.333986+0000) 2019-10-13T04:04:48.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:48.123+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:48.124826+0000) 2019-10-13T04:04:49.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:49.088+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.088684+0000) 2019-10-13T04:04:49.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:49.276+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.277419+0000) 2019-10-13T04:04:49.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:49.276+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.277542+0000) 2019-10-13T04:04:49.280 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:49.276+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.278013+0000) 2019-10-13T04:04:49.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:49.276+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.278083+0000) 2019-10-13T04:04:49.280 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:49.276+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.278202+0000) 2019-10-13T04:04:49.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:49.277+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.278412+0000) 2019-10-13T04:04:49.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:49.271+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.272300+0000) 2019-10-13T04:04:49.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:49.273+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:49.274480+0000) 2019-10-13T04:04:49.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:49.277+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:52.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:52.592+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:52.592643+0000) 2019-10-13T04:04:54.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:54.275+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.275735+0000) 2019-10-13T04:04:54.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:54.277+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.277953+0000) 2019-10-13T04:04:54.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:54.279+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.280701+0000) 2019-10-13T04:04:54.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:54.279+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.280830+0000) 2019-10-13T04:04:54.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:54.279+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.280940+0000) 2019-10-13T04:04:54.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:54.280+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.281509+0000) 2019-10-13T04:04:54.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:54.280+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.281670+0000) 2019-10-13T04:04:54.290 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:54.280+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.281954+0000) 2019-10-13T04:04:54.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:54.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47904 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:54.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:54.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.968596+0000) 2019-10-13T04:04:54.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:54.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.968608+0000) 2019-10-13T04:04:54.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:54.967+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:54.968594+0000) 2019-10-13T04:04:55.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:55.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:55.842970+0000) 2019-10-13T04:04:57.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:57.333+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:57.334111+0000) 2019-10-13T04:04:58.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:58.124+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:58.125015+0000) 2019-10-13T04:04:59.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:59.090+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.090942+0000) 2019-10-13T04:04:59.285 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:04:59.282+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.283902+0000) 2019-10-13T04:04:59.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:04:59.283+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.284500+0000) 2019-10-13T04:04:59.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:04:59.283+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.284736+0000) 2019-10-13T04:04:59.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:04:59.283+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.284835+0000) 2019-10-13T04:04:59.287 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:04:59.277+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.279582+0000) 2019-10-13T04:04:59.287 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:04:59.283+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.285103+0000) 2019-10-13T04:04:59.287 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:04:59.283+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.285247+0000) 2019-10-13T04:04:59.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:04:59.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:04:59.288 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:04:59.278+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:04:59.280688+0000) 2019-10-13T04:05:02.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:02.591+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:02.592826+0000) 2019-10-13T04:05:04.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:04.286+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.288220+0000) 2019-10-13T04:05:04.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:04.286+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.288352+0000) 2019-10-13T04:05:04.290 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:04.287+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.288682+0000) 2019-10-13T04:05:04.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:04.281+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.283107+0000) 2019-10-13T04:05:04.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:04.287+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.289121+0000) 2019-10-13T04:05:04.291 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:04.288+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.289403+0000) 2019-10-13T04:05:04.292 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:04.287+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.288897+0000) 2019-10-13T04:05:04.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:04.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:04.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:04.285+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.287116+0000) 2019-10-13T04:05:04.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:04.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.968720+0000) 2019-10-13T04:05:04.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:04.968+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.968770+0000) 2019-10-13T04:05:04.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:04.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:04.968803+0000) 2019-10-13T04:05:05.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:05.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:05.843156+0000) 2019-10-13T04:05:07.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:07.333+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:07.334270+0000) 2019-10-13T04:05:08.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:08.123+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:08.125217+0000) 2019-10-13T04:05:09.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:09.092+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.093447+0000) 2019-10-13T04:05:09.292 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:09.289+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.291211+0000) 2019-10-13T04:05:09.293 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:09.284+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.286212+0000) 2019-10-13T04:05:09.293 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:09.290+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.291637+0000) 2019-10-13T04:05:09.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:09.290+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.291844+0000) 2019-10-13T04:05:09.294 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:09.290+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.291986+0000) 2019-10-13T04:05:09.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:09.290+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.292108+0000) 2019-10-13T04:05:09.294 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:09.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 47998 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:09.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:09.286+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.287945+0000) 2019-10-13T04:05:09.295 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:09.291+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:09.292873+0000) 2019-10-13T04:05:12.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:12.592+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:12.592996+0000) 2019-10-13T04:05:14.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:14.293+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.294260+0000) 2019-10-13T04:05:14.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:14.294+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.294688+0000) 2019-10-13T04:05:14.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:14.288+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.289487+0000) 2019-10-13T04:05:14.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:14.294+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.294978+0000) 2019-10-13T04:05:14.297 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:14.294+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.295301+0000) 2019-10-13T04:05:14.297 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:14.295+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.295466+0000) 2019-10-13T04:05:14.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:14.295+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.295636+0000) 2019-10-13T04:05:14.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:14.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48024 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:14.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:14.290+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.291249+0000) 2019-10-13T04:05:14.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:14.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.968860+0000) 2019-10-13T04:05:14.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:14.968+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.968915+0000) 2019-10-13T04:05:14.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:14.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:14.968924+0000) 2019-10-13T04:05:15.254 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:05:15.254 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:15.258 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:05:15.258 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:15.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:15.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:15.843304+0000) 2019-10-13T04:05:17.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:17.334+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:17.334448+0000) 2019-10-13T04:05:18.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:18.125+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:18.125441+0000) 2019-10-13T04:05:19.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:19.095+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.095764+0000) 2019-10-13T04:05:19.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:19.297+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.297863+0000) 2019-10-13T04:05:19.299 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:19.297+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.298135+0000) 2019-10-13T04:05:19.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:19.291+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.292672+0000) 2019-10-13T04:05:19.300 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:19.298+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.298487+0000) 2019-10-13T04:05:19.301 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:19.298+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.298540+0000) 2019-10-13T04:05:19.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:19.298+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.298928+0000) 2019-10-13T04:05:19.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:19.298+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.299049+0000) 2019-10-13T04:05:19.301 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:19.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:19.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:19.294+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:19.295546+0000) 2019-10-13T04:05:22.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:22.592+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:22.593152+0000) 2019-10-13T04:05:24.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:24.295+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.296025+0000) 2019-10-13T04:05:24.303 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:24.301+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.301578+0000) 2019-10-13T04:05:24.304 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:24.301+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.301741+0000) 2019-10-13T04:05:24.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:24.301+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.301973+0000) 2019-10-13T04:05:24.304 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:24.301+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.302295+0000) 2019-10-13T04:05:24.304 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:24.302+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.302467+0000) 2019-10-13T04:05:24.304 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:24.302+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.302663+0000) 2019-10-13T04:05:24.305 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:24.302+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:24.305 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:24.298+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.299033+0000) 2019-10-13T04:05:24.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:24.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.969031+0000) 2019-10-13T04:05:24.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:24.968+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.969053+0000) 2019-10-13T04:05:24.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:24.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:24.969101+0000) 2019-10-13T04:05:25.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:25.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:25.843461+0000) 2019-10-13T04:05:27.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:27.334+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:27.334645+0000) 2019-10-13T04:05:28.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:28.125+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:28.125623+0000) 2019-10-13T04:05:29.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:29.097+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.098152+0000) 2019-10-13T04:05:29.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:29.304+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.304535+0000) 2019-10-13T04:05:29.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:29.304+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.304694+0000) 2019-10-13T04:05:29.306 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:29.304+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.304960+0000) 2019-10-13T04:05:29.307 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:29.298+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.299193+0000) 2019-10-13T04:05:29.307 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:29.304+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.305248+0000) 2019-10-13T04:05:29.308 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:29.304+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.305377+0000) 2019-10-13T04:05:29.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:29.305+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.305663+0000) 2019-10-13T04:05:29.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:29.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:29.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:29.300+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:29.301311+0000) 2019-10-13T04:05:32.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:32.592+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:32.593312+0000) 2019-10-13T04:05:34.308 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:34.305+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.307324+0000) 2019-10-13T04:05:34.309 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:34.305+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.307384+0000) 2019-10-13T04:05:34.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:34.301+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.302183+0000) 2019-10-13T04:05:34.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:34.306+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.308125+0000) 2019-10-13T04:05:34.311 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:34.306+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.308258+0000) 2019-10-13T04:05:34.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:34.307+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.308426+0000) 2019-10-13T04:05:34.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:34.307+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.308681+0000) 2019-10-13T04:05:34.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:34.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48144 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:34.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:34.304+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.305613+0000) 2019-10-13T04:05:34.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:34.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.969251+0000) 2019-10-13T04:05:34.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:34.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.969261+0000) 2019-10-13T04:05:34.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:34.968+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:34.969251+0000) 2019-10-13T04:05:35.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:35.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:35.843655+0000) 2019-10-13T04:05:37.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:37.333+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:37.334841+0000) 2019-10-13T04:05:38.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:38.124+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:38.125801+0000) 2019-10-13T04:05:39.107 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:39.100+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.100494+0000) 2019-10-13T04:05:39.312 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:39.308+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.310209+0000) 2019-10-13T04:05:39.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:39.309+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.310412+0000) 2019-10-13T04:05:39.312 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:39.309+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.310640+0000) 2019-10-13T04:05:39.313 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:39.309+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.310749+0000) 2019-10-13T04:05:39.313 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:39.309+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.310908+0000) 2019-10-13T04:05:39.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:39.309+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.311121+0000) 2019-10-13T04:05:39.314 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:39.304+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.305227+0000) 2019-10-13T04:05:39.314 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:39.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48177 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:39.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:39.307+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:39.307977+0000) 2019-10-13T04:05:42.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:42.593+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:42.593481+0000) 2019-10-13T04:05:44.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:44.311+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.313206+0000) 2019-10-13T04:05:44.316 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:44.307+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.308192+0000) 2019-10-13T04:05:44.316 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:44.312+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.313818+0000) 2019-10-13T04:05:44.317 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:44.312+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.313909+0000) 2019-10-13T04:05:44.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:44.312+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.314032+0000) 2019-10-13T04:05:44.317 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:44.312+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.314318+0000) 2019-10-13T04:05:44.318 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:44.313+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.314546+0000) 2019-10-13T04:05:44.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:44.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:44.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:44.309+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.310090+0000) 2019-10-13T04:05:44.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:44.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.969430+0000) 2019-10-13T04:05:44.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:44.968+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.969425+0000) 2019-10-13T04:05:44.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:44.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:44.969433+0000) 2019-10-13T04:05:45.379 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:05:45.380 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:45.384 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:05:45.384 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:45.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:45.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:45.843847+0000) 2019-10-13T04:05:47.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:47.334+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:47.335017+0000) 2019-10-13T04:05:48.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:48.125+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:48.125987+0000) 2019-10-13T04:05:49.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:49.102+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.102891+0000) 2019-10-13T04:05:49.317 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:49.310+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.311009+0000) 2019-10-13T04:05:49.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:49.315+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.316565+0000) 2019-10-13T04:05:49.319 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:49.315+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.316766+0000) 2019-10-13T04:05:49.319 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:49.315+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.317025+0000) 2019-10-13T04:05:49.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:49.315+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.317243+0000) 2019-10-13T04:05:49.319 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:49.315+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.317379+0000) 2019-10-13T04:05:49.320 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:49.316+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.317480+0000) 2019-10-13T04:05:49.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:49.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48238 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:49.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:49.312+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:49.313252+0000) 2019-10-13T04:05:52.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:52.591+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:52.593687+0000) 2019-10-13T04:05:54.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:54.312+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.313887+0000) 2019-10-13T04:05:54.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:54.318+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.319418+0000) 2019-10-13T04:05:54.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:54.318+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.319664+0000) 2019-10-13T04:05:54.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:54.318+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.319902+0000) 2019-10-13T04:05:54.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:54.318+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.320127+0000) 2019-10-13T04:05:54.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:54.318+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.320297+0000) 2019-10-13T04:05:54.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:54.319+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.320563+0000) 2019-10-13T04:05:54.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:54.320+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48264 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:54.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:54.316+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.318553+0000) 2019-10-13T04:05:54.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:54.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.969620+0000) 2019-10-13T04:05:54.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:54.969+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.969613+0000) 2019-10-13T04:05:54.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:54.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:54.969621+0000) 2019-10-13T04:05:55.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:55.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:55.844072+0000) 2019-10-13T04:05:57.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:57.334+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:57.335217+0000) 2019-10-13T04:05:58.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:58.125+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:58.126169+0000) 2019-10-13T04:05:59.118 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:59.104+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.105252+0000) 2019-10-13T04:05:59.324 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:05:59.316+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.317818+0000) 2019-10-13T04:05:59.325 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:05:59.323+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.323525+0000) 2019-10-13T04:05:59.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:05:59.323+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.323618+0000) 2019-10-13T04:05:59.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:05:59.323+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.323851+0000) 2019-10-13T04:05:59.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:05:59.323+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.324126+0000) 2019-10-13T04:05:59.326 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:05:59.323+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.324331+0000) 2019-10-13T04:05:59.326 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:05:59.324+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.324448+0000) 2019-10-13T04:05:59.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:05:59.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:05:59.328 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:05:59.319+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:05:59.321285+0000) 2019-10-13T04:06:02.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:02.593+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:02.593860+0000) 2019-10-13T04:06:04.328 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:04.326+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.326857+0000) 2019-10-13T04:06:04.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:04.321+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.322280+0000) 2019-10-13T04:06:04.329 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:04.326+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.327328+0000) 2019-10-13T04:06:04.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:04.327+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.327463+0000) 2019-10-13T04:06:04.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:04.327+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.327844+0000) 2019-10-13T04:06:04.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:04.327+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.328061+0000) 2019-10-13T04:06:04.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:04.327+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.328247+0000) 2019-10-13T04:06:04.330 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:04.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:04.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:04.323+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.324497+0000) 2019-10-13T04:06:04.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:04.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.969773+0000) 2019-10-13T04:06:04.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:04.969+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.969759+0000) 2019-10-13T04:06:04.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:04.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:04.969771+0000) 2019-10-13T04:06:05.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:05.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:05.844259+0000) 2019-10-13T04:06:07.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:07.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:07.335387+0000) 2019-10-13T04:06:08.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:08.126+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:08.126411+0000) 2019-10-13T04:06:09.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:09.106+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.107472+0000) 2019-10-13T04:06:09.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:09.329+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.330290+0000) 2019-10-13T04:06:09.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:09.330+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.330517+0000) 2019-10-13T04:06:09.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:09.324+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.325709+0000) 2019-10-13T04:06:09.333 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:09.330+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.330813+0000) 2019-10-13T04:06:09.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:09.330+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.331160+0000) 2019-10-13T04:06:09.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:09.330+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.331253+0000) 2019-10-13T04:06:09.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:09.331+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.331800+0000) 2019-10-13T04:06:09.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:09.332+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48358 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:09.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:09.327+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:09.328098+0000) 2019-10-13T04:06:12.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:12.593+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:12.594041+0000) 2019-10-13T04:06:14.335 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:14.333+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.333953+0000) 2019-10-13T04:06:14.335 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:14.333+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.334185+0000) 2019-10-13T04:06:14.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:14.334+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.334414+0000) 2019-10-13T04:06:14.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:14.334+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.335079+0000) 2019-10-13T04:06:14.336 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:14.334+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.335202+0000) 2019-10-13T04:06:14.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:14.329+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.330202+0000) 2019-10-13T04:06:14.337 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:14.335+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.335876+0000) 2019-10-13T04:06:14.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:14.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:14.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:14.330+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.331428+0000) 2019-10-13T04:06:14.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:14.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.969954+0000) 2019-10-13T04:06:14.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:14.968+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.969943+0000) 2019-10-13T04:06:14.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:14.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:14.969964+0000) 2019-10-13T04:06:15.493 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:06:15.493 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:06:15.498 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:06:15.498 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:06:15.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:15.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:15.844478+0000) 2019-10-13T04:06:17.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:17.334+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:17.335589+0000) 2019-10-13T04:06:18.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:18.125+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:18.126611+0000) 2019-10-13T04:06:19.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:19.109+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.109803+0000) 2019-10-13T04:06:19.340 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:19.337+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.338489+0000) 2019-10-13T04:06:19.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:19.337+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.338708+0000) 2019-10-13T04:06:19.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:19.337+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.338929+0000) 2019-10-13T04:06:19.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:19.337+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.339048+0000) 2019-10-13T04:06:19.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:19.337+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.339280+0000) 2019-10-13T04:06:19.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:19.338+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.339438+0000) 2019-10-13T04:06:19.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:19.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48417 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:19.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:19.333+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.334366+0000) 2019-10-13T04:06:19.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:19.335+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:19.336291+0000) 2019-10-13T04:06:22.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:22.593+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:22.594236+0000) 2019-10-13T04:06:24.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:24.340+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.342084+0000) 2019-10-13T04:06:24.344 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:24.336+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.336840+0000) 2019-10-13T04:06:24.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:24.341+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.342551+0000) 2019-10-13T04:06:24.345 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:24.341+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.342826+0000) 2019-10-13T04:06:24.345 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:24.341+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.342748+0000) 2019-10-13T04:06:24.345 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:24.341+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.342987+0000) 2019-10-13T04:06:24.345 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:24.341+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.343135+0000) 2019-10-13T04:06:24.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:24.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48445 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:24.346 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:24.339+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.340186+0000) 2019-10-13T04:06:24.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:24.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.970124+0000) 2019-10-13T04:06:24.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:24.969+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.970125+0000) 2019-10-13T04:06:24.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:24.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:24.970140+0000) 2019-10-13T04:06:25.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:25.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:25.844674+0000) 2019-10-13T04:06:27.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:27.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:27.335774+0000) 2019-10-13T04:06:28.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:28.126+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:28.126776+0000) 2019-10-13T04:06:29.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:29.111+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.112176+0000) 2019-10-13T04:06:29.346 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:29.339+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.340138+0000) 2019-10-13T04:06:29.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:29.344+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.345623+0000) 2019-10-13T04:06:29.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:29.344+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.345880+0000) 2019-10-13T04:06:29.348 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:29.344+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.346117+0000) 2019-10-13T04:06:29.348 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:29.345+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.346564+0000) 2019-10-13T04:06:29.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:29.345+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.346518+0000) 2019-10-13T04:06:29.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:29.345+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.346782+0000) 2019-10-13T04:06:29.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:29.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48478 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:29.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:29.341+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:29.342660+0000) 2019-10-13T04:06:32.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:32.594+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:32.594418+0000) 2019-10-13T04:06:34.353 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:34.343+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.344213+0000) 2019-10-13T04:06:34.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:34.347+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.348849+0000) 2019-10-13T04:06:34.353 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:34.347+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.349068+0000) 2019-10-13T04:06:34.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:34.347+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.349203+0000) 2019-10-13T04:06:34.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:34.348+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.349423+0000) 2019-10-13T04:06:34.354 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:34.348+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.350180+0000) 2019-10-13T04:06:34.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:34.348+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.350385+0000) 2019-10-13T04:06:34.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:34.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48504 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:34.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:34.346+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.347561+0000) 2019-10-13T04:06:34.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:34.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.970312+0000) 2019-10-13T04:06:34.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:34.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.970314+0000) 2019-10-13T04:06:34.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:34.969+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:34.970308+0000) 2019-10-13T04:06:35.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:35.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:35.844829+0000) 2019-10-13T04:06:37.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:37.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:37.335939+0000) 2019-10-13T04:06:38.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:38.126+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:38.126970+0000) 2019-10-13T04:06:39.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:39.113+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.114511+0000) 2019-10-13T04:06:39.354 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:39.351+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.352753+0000) 2019-10-13T04:06:39.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:39.351+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.352898+0000) 2019-10-13T04:06:39.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:39.351+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.353253+0000) 2019-10-13T04:06:39.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:39.352+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.353495+0000) 2019-10-13T04:06:39.355 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:39.352+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.353681+0000) 2019-10-13T04:06:39.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:39.352+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.354141+0000) 2019-10-13T04:06:39.356 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:39.347+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.348575+0000) 2019-10-13T04:06:39.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:39.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:39.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:39.350+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:39.350944+0000) 2019-10-13T04:06:42.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:42.593+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:42.594587+0000) 2019-10-13T04:06:44.358 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:44.355+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.356124+0000) 2019-10-13T04:06:44.358 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:44.355+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.356291+0000) 2019-10-13T04:06:44.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:44.356+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.356757+0000) 2019-10-13T04:06:44.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:44.356+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.356947+0000) 2019-10-13T04:06:44.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:44.356+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.356514+0000) 2019-10-13T04:06:44.359 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:44.356+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.357133+0000) 2019-10-13T04:06:44.359 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:44.350+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.351932+0000) 2019-10-13T04:06:44.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:44.357+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48565 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:44.360 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:44.352+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.354054+0000) 2019-10-13T04:06:44.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:44.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.970427+0000) 2019-10-13T04:06:44.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:44.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.970474+0000) 2019-10-13T04:06:44.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:44.970+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:44.970481+0000) 2019-10-13T04:06:45.623 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:06:45.624 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:06:45.628 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:06:45.628 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:06:45.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:45.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:45.844963+0000) 2019-10-13T04:06:47.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:47.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:47.336143+0000) 2019-10-13T04:06:48.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:48.126+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:48.127149+0000) 2019-10-13T04:06:49.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:49.115+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.116871+0000) 2019-10-13T04:06:49.361 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:49.359+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.359747+0000) 2019-10-13T04:06:49.361 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:49.359+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.359976+0000) 2019-10-13T04:06:49.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:49.359+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.360200+0000) 2019-10-13T04:06:49.362 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:49.360+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.360532+0000) 2019-10-13T04:06:49.362 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:49.360+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.360674+0000) 2019-10-13T04:06:49.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:49.360+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.361229+0000) 2019-10-13T04:06:49.363 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:49.354+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.356152+0000) 2019-10-13T04:06:49.363 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:49.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:49.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:49.356+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:49.358012+0000) 2019-10-13T04:06:52.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:52.593+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:52.594770+0000) 2019-10-13T04:06:54.365 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:54.357+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.358565+0000) 2019-10-13T04:06:54.366 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:54.363+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.363952+0000) 2019-10-13T04:06:54.367 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:54.363+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.364138+0000) 2019-10-13T04:06:54.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:54.364+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.364461+0000) 2019-10-13T04:06:54.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:54.364+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.364486+0000) 2019-10-13T04:06:54.367 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:54.364+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.364876+0000) 2019-10-13T04:06:54.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:54.364+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.365059+0000) 2019-10-13T04:06:54.368 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:54.364+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48624 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:06:54.369 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:54.360+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.361718+0000) 2019-10-13T04:06:54.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:54.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.970609+0000) 2019-10-13T04:06:54.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:54.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.970619+0000) 2019-10-13T04:06:54.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:54.970+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:54.970620+0000) 2019-10-13T04:06:55.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:55.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:55.845153+0000) 2019-10-13T04:06:57.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:57.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:57.336267+0000) 2019-10-13T04:06:58.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:58.126+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:58.127358+0000) 2019-10-13T04:06:59.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:59.118+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.119380+0000) 2019-10-13T04:06:59.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:06:59.366+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.366936+0000) 2019-10-13T04:06:59.368 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:06:59.366+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.367309+0000) 2019-10-13T04:06:59.369 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:06:59.361+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.362453+0000) 2019-10-13T04:06:59.370 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:06:59.367+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.367677+0000) 2019-10-13T04:06:59.370 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:06:59.367+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.367987+0000) 2019-10-13T04:06:59.370 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:06:59.367+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.368187+0000) 2019-10-13T04:06:59.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:06:59.368+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.368635+0000) 2019-10-13T04:06:59.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:06:59.363+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:06:59.363906+0000) 2019-10-13T04:06:59.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:06:59.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48657 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:02.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:02.594+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:02.594932+0000) 2019-10-13T04:07:04.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:04.369+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.370709+0000) 2019-10-13T04:07:04.372 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:04.369+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.370875+0000) 2019-10-13T04:07:04.373 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:04.369+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.371135+0000) 2019-10-13T04:07:04.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:04.369+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.371284+0000) 2019-10-13T04:07:04.373 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:04.370+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.371427+0000) 2019-10-13T04:07:04.373 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:04.364+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.365686+0000) 2019-10-13T04:07:04.374 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:04.370+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.371626+0000) 2019-10-13T04:07:04.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:04.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48685 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:04.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:04.367+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.368222+0000) 2019-10-13T04:07:04.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:04.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.970740+0000) 2019-10-13T04:07:04.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:04.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.970776+0000) 2019-10-13T04:07:04.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:04.969+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:04.970760+0000) 2019-10-13T04:07:05.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:05.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:05.845368+0000) 2019-10-13T04:07:07.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:07.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:07.336428+0000) 2019-10-13T04:07:08.129 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:08.126+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:08.127570+0000) 2019-10-13T04:07:09.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:09.121+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.121881+0000) 2019-10-13T04:07:09.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:09.372+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.374012+0000) 2019-10-13T04:07:09.375 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:09.372+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.374376+0000) 2019-10-13T04:07:09.376 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:09.368+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.369199+0000) 2019-10-13T04:07:09.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:09.373+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.374805+0000) 2019-10-13T04:07:09.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:09.373+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.375090+0000) 2019-10-13T04:07:09.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:09.373+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.375262+0000) 2019-10-13T04:07:09.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:09.374+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.375669+0000) 2019-10-13T04:07:09.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:09.370+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:09.370784+0000) 2019-10-13T04:07:09.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:09.374+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:12.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:12.594+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:12.595080+0000) 2019-10-13T04:07:14.378 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:14.371+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.371979+0000) 2019-10-13T04:07:14.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:14.376+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.378014+0000) 2019-10-13T04:07:14.379 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:14.376+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.378155+0000) 2019-10-13T04:07:14.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:14.377+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.378491+0000) 2019-10-13T04:07:14.380 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:14.377+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.378522+0000) 2019-10-13T04:07:14.380 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:14.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.378560+0000) 2019-10-13T04:07:14.380 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:14.377+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.379008+0000) 2019-10-13T04:07:14.381 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:14.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:14.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:14.375+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.376390+0000) 2019-10-13T04:07:14.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:14.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.970848+0000) 2019-10-13T04:07:14.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:14.970+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.970919+0000) 2019-10-13T04:07:14.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:14.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:14.970923+0000) 2019-10-13T04:07:15.796 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:07:15.796 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:15.800 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:07:15.800 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:15.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:15.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:15.845504+0000) 2019-10-13T04:07:17.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:17.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:17.336580+0000) 2019-10-13T04:07:18.129 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:18.127+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:18.127764+0000) 2019-10-13T04:07:19.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:19.123+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.124118+0000) 2019-10-13T04:07:19.382 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:19.379+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.381317+0000) 2019-10-13T04:07:19.383 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:19.375+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.376129+0000) 2019-10-13T04:07:19.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:19.380+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.381627+0000) 2019-10-13T04:07:19.384 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:19.380+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.381979+0000) 2019-10-13T04:07:19.384 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:19.380+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.382184+0000) 2019-10-13T04:07:19.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:19.380+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.382365+0000) 2019-10-13T04:07:19.384 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:19.381+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.382590+0000) 2019-10-13T04:07:19.385 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:19.377+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:19.378550+0000) 2019-10-13T04:07:19.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:19.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:22.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:22.595+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:22.595250+0000) 2019-10-13T04:07:24.390 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:24.380+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.380966+0000) 2019-10-13T04:07:24.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:24.382+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.383578+0000) 2019-10-13T04:07:24.391 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:24.383+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.385328+0000) 2019-10-13T04:07:24.391 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:24.384+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.385752+0000) 2019-10-13T04:07:24.391 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:24.384+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.386027+0000) 2019-10-13T04:07:24.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:24.384+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.386333+0000) 2019-10-13T04:07:24.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:24.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48805 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:24.392 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:24.385+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.386965+0000) 2019-10-13T04:07:24.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:24.385+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.387098+0000) 2019-10-13T04:07:24.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:24.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.971032+0000) 2019-10-13T04:07:24.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:24.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.971067+0000) 2019-10-13T04:07:24.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:24.970+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:24.971076+0000) 2019-10-13T04:07:25.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:25.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:25.845635+0000) 2019-10-13T04:07:27.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:27.336+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:27.336781+0000) 2019-10-13T04:07:28.129 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:28.127+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:28.127937+0000) 2019-10-13T04:07:29.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:29.126+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.126356+0000) 2019-10-13T04:07:29.390 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:29.388+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.389035+0000) 2019-10-13T04:07:29.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:29.388+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.389339+0000) 2019-10-13T04:07:29.391 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:29.389+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.389440+0000) 2019-10-13T04:07:29.391 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:29.389+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.389589+0000) 2019-10-13T04:07:29.392 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:29.383+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.383826+0000) 2019-10-13T04:07:29.392 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:29.384+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.385325+0000) 2019-10-13T04:07:29.392 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:29.389+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.389989+0000) 2019-10-13T04:07:29.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:29.389+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:29.390133+0000) 2019-10-13T04:07:29.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:29.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48838 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:32.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:32.594+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:32.595448+0000) 2019-10-13T04:07:34.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:34.391+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.392268+0000) 2019-10-13T04:07:34.394 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:34.385+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.387452+0000) 2019-10-13T04:07:34.394 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:34.392+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.392923+0000) 2019-10-13T04:07:34.395 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:34.392+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.393073+0000) 2019-10-13T04:07:34.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:34.392+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.393268+0000) 2019-10-13T04:07:34.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:34.392+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.393384+0000) 2019-10-13T04:07:34.395 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:34.393+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.393544+0000) 2019-10-13T04:07:34.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:34.387+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.389430+0000) 2019-10-13T04:07:34.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:34.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:34.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:34.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.971218+0000) 2019-10-13T04:07:34.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:34.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.971220+0000) 2019-10-13T04:07:34.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:34.970+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:34.971218+0000) 2019-10-13T04:07:35.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:35.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:35.845825+0000) 2019-10-13T04:07:37.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:37.336+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:37.337005+0000) 2019-10-13T04:07:38.129 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:38.127+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:38.128119+0000) 2019-10-13T04:07:39.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:39.127+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.128681+0000) 2019-10-13T04:07:39.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:39.395+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.395948+0000) 2019-10-13T04:07:39.402 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:39.395+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.396047+0000) 2019-10-13T04:07:39.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:39.395+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.396270+0000) 2019-10-13T04:07:39.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:39.396+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.396725+0000) 2019-10-13T04:07:39.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:39.396+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.396889+0000) 2019-10-13T04:07:39.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:39.396+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.397051+0000) 2019-10-13T04:07:39.405 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:39.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:39.406 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:39.389+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.391036+0000) 2019-10-13T04:07:39.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:39.390+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:39.392282+0000) 2019-10-13T04:07:42.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:42.594+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:42.595610+0000) 2019-10-13T04:07:44.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:44.398+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.399134+0000) 2019-10-13T04:07:44.401 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:44.399+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.399621+0000) 2019-10-13T04:07:44.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:44.399+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.399853+0000) 2019-10-13T04:07:44.402 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:44.399+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.399988+0000) 2019-10-13T04:07:44.402 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:44.399+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.400182+0000) 2019-10-13T04:07:44.403 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:44.399+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.400347+0000) 2019-10-13T04:07:44.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:44.393+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.395231+0000) 2019-10-13T04:07:44.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:44.395+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.396835+0000) 2019-10-13T04:07:44.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:44.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48925 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:44.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:44.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.971369+0000) 2019-10-13T04:07:44.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:44.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.971375+0000) 2019-10-13T04:07:44.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:44.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:44.971401+0000) 2019-10-13T04:07:45.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:45.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:45.846011+0000) 2019-10-13T04:07:45.935 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:07:45.936 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:45.941 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:07:45.941 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:47.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:47.335+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:47.337173+0000) 2019-10-13T04:07:48.129 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:48.127+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:48.128314+0000) 2019-10-13T04:07:49.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:49.130+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.130896+0000) 2019-10-13T04:07:49.404 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:49.401+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.402852+0000) 2019-10-13T04:07:49.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:49.401+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.403106+0000) 2019-10-13T04:07:49.405 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:49.401+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.403367+0000) 2019-10-13T04:07:49.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:49.402+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.403501+0000) 2019-10-13T04:07:49.406 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:49.397+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.397784+0000) 2019-10-13T04:07:49.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:49.402+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.403632+0000) 2019-10-13T04:07:49.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:49.402+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.403954+0000) 2019-10-13T04:07:49.406 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:49.402+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:49.409 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:49.399+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:49.400633+0000) 2019-10-13T04:07:52.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:52.595+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:52.595792+0000) 2019-10-13T04:07:54.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:54.404+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.405724+0000) 2019-10-13T04:07:54.407 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:54.404+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.405872+0000) 2019-10-13T04:07:54.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:54.399+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.400719+0000) 2019-10-13T04:07:54.408 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:54.404+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.406286+0000) 2019-10-13T04:07:54.408 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:54.405+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.406729+0000) 2019-10-13T04:07:54.409 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:54.405+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.406575+0000) 2019-10-13T04:07:54.409 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:54.405+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.407088+0000) 2019-10-13T04:07:54.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:54.406+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 48984 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:07:54.410 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:54.402+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.403556+0000) 2019-10-13T04:07:54.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:54.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.971534+0000) 2019-10-13T04:07:54.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:54.970+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.971538+0000) 2019-10-13T04:07:54.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:54.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:54.971534+0000) 2019-10-13T04:07:55.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:55.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:55.846219+0000) 2019-10-13T04:07:57.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:57.336+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:57.337341+0000) 2019-10-13T04:07:58.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:58.127+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:58.128506+0000) 2019-10-13T04:07:59.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:59.132+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.133243+0000) 2019-10-13T04:07:59.411 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:07:59.407+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.409089+0000) 2019-10-13T04:07:59.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:07:59.407+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.409277+0000) 2019-10-13T04:07:59.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:07:59.408+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.409671+0000) 2019-10-13T04:07:59.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:07:59.408+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.409853+0000) 2019-10-13T04:07:59.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:07:59.408+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.410151+0000) 2019-10-13T04:07:59.414 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:07:59.404+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.404967+0000) 2019-10-13T04:07:59.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:07:59.405+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.406114+0000) 2019-10-13T04:07:59.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:07:59.409+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:07:59.410573+0000) 2019-10-13T04:07:59.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:07:59.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:02.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:02.595+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:02.595962+0000) 2019-10-13T04:08:04.415 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:04.411+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.413307+0000) 2019-10-13T04:08:04.415 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:04.412+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.413620+0000) 2019-10-13T04:08:04.416 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:04.412+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.413501+0000) 2019-10-13T04:08:04.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:04.412+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.413782+0000) 2019-10-13T04:08:04.416 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:04.412+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.414072+0000) 2019-10-13T04:08:04.417 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:04.407+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.408680+0000) 2019-10-13T04:08:04.417 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:04.412+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.414219+0000) 2019-10-13T04:08:04.417 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:04.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49045 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:04.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:04.410+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.411471+0000) 2019-10-13T04:08:04.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:04.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.971725+0000) 2019-10-13T04:08:04.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:04.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.971722+0000) 2019-10-13T04:08:04.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:04.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:04.971724+0000) 2019-10-13T04:08:05.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:05.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:05.846391+0000) 2019-10-13T04:08:07.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:07.336+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:07.337546+0000) 2019-10-13T04:08:08.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:08.128+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:08.128693+0000) 2019-10-13T04:08:09.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:09.134+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.135662+0000) 2019-10-13T04:08:09.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:09.415+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.416998+0000) 2019-10-13T04:08:09.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:09.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.412293+0000) 2019-10-13T04:08:09.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:09.415+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.417332+0000) 2019-10-13T04:08:09.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:09.416+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.417514+0000) 2019-10-13T04:08:09.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:09.416+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.417770+0000) 2019-10-13T04:08:09.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:09.416+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.418068+0000) 2019-10-13T04:08:09.421 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:09.416+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.418233+0000) 2019-10-13T04:08:09.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:09.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:09.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:09.412+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:09.413509+0000) 2019-10-13T04:08:12.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:12.595+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:12.596120+0000) 2019-10-13T04:08:14.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:14.419+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.420140+0000) 2019-10-13T04:08:14.422 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:14.414+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.415034+0000) 2019-10-13T04:08:14.422 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:14.420+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.421263+0000) 2019-10-13T04:08:14.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:14.421+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.421525+0000) 2019-10-13T04:08:14.423 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:14.421+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.421705+0000) 2019-10-13T04:08:14.423 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:14.421+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.421919+0000) 2019-10-13T04:08:14.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:14.421+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.422128+0000) 2019-10-13T04:08:14.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:14.422+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:14.426 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:14.419+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.420063+0000) 2019-10-13T04:08:14.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:14.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.971920+0000) 2019-10-13T04:08:14.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:14.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.971918+0000) 2019-10-13T04:08:14.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:14.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:14.971916+0000) 2019-10-13T04:08:15.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:15.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:15.846591+0000) 2019-10-13T04:08:16.016 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:08:16.017 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:16.021 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:08:16.021 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:17.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:17.337+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:17.337702+0000) 2019-10-13T04:08:18.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:18.128+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:18.128853+0000) 2019-10-13T04:08:19.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:19.137+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.138042+0000) 2019-10-13T04:08:19.425 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:19.417+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.418650+0000) 2019-10-13T04:08:19.426 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:19.423+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.424275+0000) 2019-10-13T04:08:19.426 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:19.424+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.424395+0000) 2019-10-13T04:08:19.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:19.424+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.424606+0000) 2019-10-13T04:08:19.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:19.424+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.425082+0000) 2019-10-13T04:08:19.427 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:19.424+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.425205+0000) 2019-10-13T04:08:19.427 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:19.424+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.425330+0000) 2019-10-13T04:08:19.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:19.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:19.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:19.420+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:19.421438+0000) 2019-10-13T04:08:22.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:22.595+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:22.596311+0000) 2019-10-13T04:08:24.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:24.426+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.427364+0000) 2019-10-13T04:08:24.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:24.427+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.427511+0000) 2019-10-13T04:08:24.430 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:24.420+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.422300+0000) 2019-10-13T04:08:24.431 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:24.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.427939+0000) 2019-10-13T04:08:24.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:24.427+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.428109+0000) 2019-10-13T04:08:24.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:24.427+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.428386+0000) 2019-10-13T04:08:24.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:24.428+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.428449+0000) 2019-10-13T04:08:24.432 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:24.428+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:24.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:24.422+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.424583+0000) 2019-10-13T04:08:24.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:24.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.972066+0000) 2019-10-13T04:08:24.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:24.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.972067+0000) 2019-10-13T04:08:24.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:24.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:24.972068+0000) 2019-10-13T04:08:25.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:25.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:25.846802+0000) 2019-10-13T04:08:27.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:27.337+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:27.337894+0000) 2019-10-13T04:08:28.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:28.128+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:28.129086+0000) 2019-10-13T04:08:29.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:29.139+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.140272+0000) 2019-10-13T04:08:29.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:29.429+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.430279+0000) 2019-10-13T04:08:29.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:29.429+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.430381+0000) 2019-10-13T04:08:29.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:29.430+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.430684+0000) 2019-10-13T04:08:29.432 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:29.430+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.430896+0000) 2019-10-13T04:08:29.433 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:29.423+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.425265+0000) 2019-10-13T04:08:29.434 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:29.430+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.431291+0000) 2019-10-13T04:08:29.434 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:29.430+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.431375+0000) 2019-10-13T04:08:29.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:29.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49198 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:29.435 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:29.426+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:29.428675+0000) 2019-10-13T04:08:32.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:32.595+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:32.596486+0000) 2019-10-13T04:08:34.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:34.426+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.428371+0000) 2019-10-13T04:08:34.435 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:34.432+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.434098+0000) 2019-10-13T04:08:34.436 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:34.432+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.434284+0000) 2019-10-13T04:08:34.436 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:34.433+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.434664+0000) 2019-10-13T04:08:34.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:34.433+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.434786+0000) 2019-10-13T04:08:34.437 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:34.433+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.435081+0000) 2019-10-13T04:08:34.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:34.433+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.434920+0000) 2019-10-13T04:08:34.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:34.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:34.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:34.430+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.432042+0000) 2019-10-13T04:08:34.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:34.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.972222+0000) 2019-10-13T04:08:34.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:34.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.972220+0000) 2019-10-13T04:08:34.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:34.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:34.972223+0000) 2019-10-13T04:08:35.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:35.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:35.846991+0000) 2019-10-13T04:08:37.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:37.337+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:37.338074+0000) 2019-10-13T04:08:38.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:38.128+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:38.129298+0000) 2019-10-13T04:08:39.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:39.141+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.142523+0000) 2019-10-13T04:08:39.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:39.436+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.437568+0000) 2019-10-13T04:08:39.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:39.436+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.437659+0000) 2019-10-13T04:08:39.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:39.436+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.437928+0000) 2019-10-13T04:08:39.440 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:39.432+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.432746+0000) 2019-10-13T04:08:39.440 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:39.436+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.438331+0000) 2019-10-13T04:08:39.441 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:39.437+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.438519+0000) 2019-10-13T04:08:39.441 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:39.437+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.438600+0000) 2019-10-13T04:08:39.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:39.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:39.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:39.433+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:39.434560+0000) 2019-10-13T04:08:42.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:42.596+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:42.596646+0000) 2019-10-13T04:08:44.444 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:44.435+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.435913+0000) 2019-10-13T04:08:44.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:44.439+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.440945+0000) 2019-10-13T04:08:44.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:44.440+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.441483+0000) 2019-10-13T04:08:44.445 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:44.440+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.441609+0000) 2019-10-13T04:08:44.445 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:44.440+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.441821+0000) 2019-10-13T04:08:44.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:44.440+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.442324+0000) 2019-10-13T04:08:44.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:44.441+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.442774+0000) 2019-10-13T04:08:44.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:44.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:44.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:44.438+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.439289+0000) 2019-10-13T04:08:44.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:44.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.972418+0000) 2019-10-13T04:08:44.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:44.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.972418+0000) 2019-10-13T04:08:44.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:44.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:44.972420+0000) 2019-10-13T04:08:45.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:45.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:45.847171+0000) 2019-10-13T04:08:46.139 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:08:46.139 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:46.143 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:08:46.144 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:47.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:47.337+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:47.338270+0000) 2019-10-13T04:08:48.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:48.128+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:48.129456+0000) 2019-10-13T04:08:49.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:49.144+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.144881+0000) 2019-10-13T04:08:49.446 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:49.443+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.445014+0000) 2019-10-13T04:08:49.447 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:49.439+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.440151+0000) 2019-10-13T04:08:49.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:49.443+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.445311+0000) 2019-10-13T04:08:49.447 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:49.444+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.445784+0000) 2019-10-13T04:08:49.448 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:49.444+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.445659+0000) 2019-10-13T04:08:49.448 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:49.444+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.445901+0000) 2019-10-13T04:08:49.448 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:49.444+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.446389+0000) 2019-10-13T04:08:49.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:49.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49318 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:49.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:49.442+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:49.443655+0000) 2019-10-13T04:08:52.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:52.596+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:52.596811+0000) 2019-10-13T04:08:54.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:54.447+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.448926+0000) 2019-10-13T04:08:54.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:54.447+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.449223+0000) 2019-10-13T04:08:54.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:54.442+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.443690+0000) 2019-10-13T04:08:54.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:54.448+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.449449+0000) 2019-10-13T04:08:54.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:54.448+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.449586+0000) 2019-10-13T04:08:54.452 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:54.448+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.449930+0000) 2019-10-13T04:08:54.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:54.448+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.450213+0000) 2019-10-13T04:08:54.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:54.444+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.445095+0000) 2019-10-13T04:08:54.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:54.449+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49344 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:54.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:54.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.972550+0000) 2019-10-13T04:08:54.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:54.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.972564+0000) 2019-10-13T04:08:54.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:54.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:54.972568+0000) 2019-10-13T04:08:55.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:55.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:55.847327+0000) 2019-10-13T04:08:57.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:57.338+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:57.338477+0000) 2019-10-13T04:08:58.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:58.129+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:58.129650+0000) 2019-10-13T04:08:59.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:59.147+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.147442+0000) 2019-10-13T04:08:59.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:08:59.446+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.446900+0000) 2019-10-13T04:08:59.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:08:59.452+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.452532+0000) 2019-10-13T04:08:59.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:08:59.452+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.452879+0000) 2019-10-13T04:08:59.455 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:08:59.452+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.453115+0000) 2019-10-13T04:08:59.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:08:59.452+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.453368+0000) 2019-10-13T04:08:59.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:08:59.453+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.453520+0000) 2019-10-13T04:08:59.456 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:08:59.453+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.454087+0000) 2019-10-13T04:08:59.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:08:59.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49377 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:08:59.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:08:59.449+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:08:59.450290+0000) 2019-10-13T04:09:02.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:02.596+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:02.596960+0000) 2019-10-13T04:09:04.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:04.449+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.450721+0000) 2019-10-13T04:09:04.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:04.456+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.456446+0000) 2019-10-13T04:09:04.458 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:04.456+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.456847+0000) 2019-10-13T04:09:04.458 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:04.456+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.456976+0000) 2019-10-13T04:09:04.458 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:04.456+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.457152+0000) 2019-10-13T04:09:04.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:04.456+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.457312+0000) 2019-10-13T04:09:04.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:04.457+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.457605+0000) 2019-10-13T04:09:04.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:04.457+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49405 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:04.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:04.452+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.452993+0000) 2019-10-13T04:09:04.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:04.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.972721+0000) 2019-10-13T04:09:04.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:04.972+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.972718+0000) 2019-10-13T04:09:04.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:04.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:04.972720+0000) 2019-10-13T04:09:05.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:05.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:05.847500+0000) 2019-10-13T04:09:07.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:07.338+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:07.338689+0000) 2019-10-13T04:09:08.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:08.129+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:08.129873+0000) 2019-10-13T04:09:09.156 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:09.149+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.149784+0000) 2019-10-13T04:09:09.461 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:09.453+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.454359+0000) 2019-10-13T04:09:09.462 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:09.459+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.460044+0000) 2019-10-13T04:09:09.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:09.460+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.460416+0000) 2019-10-13T04:09:09.463 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:09.460+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.460629+0000) 2019-10-13T04:09:09.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:09.460+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.461123+0000) 2019-10-13T04:09:09.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:09.460+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.461223+0000) 2019-10-13T04:09:09.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:09.461+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.461402+0000) 2019-10-13T04:09:09.464 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:09.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:09.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:09.456+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:09.456976+0000) 2019-10-13T04:09:12.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:12.596+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:12.597164+0000) 2019-10-13T04:09:14.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:14.463+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.464180+0000) 2019-10-13T04:09:14.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:14.463+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.464366+0000) 2019-10-13T04:09:14.466 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:14.457+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.459473+0000) 2019-10-13T04:09:14.468 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:14.464+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.464590+0000) 2019-10-13T04:09:14.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:14.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49464 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:14.472 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:14.464+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.465308+0000) 2019-10-13T04:09:14.472 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:14.465+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.465782+0000) 2019-10-13T04:09:14.473 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:14.460+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.462658+0000) 2019-10-13T04:09:14.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:14.466+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.466827+0000) 2019-10-13T04:09:14.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:14.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.972858+0000) 2019-10-13T04:09:14.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:14.972+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.972853+0000) 2019-10-13T04:09:14.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:14.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:14.972837+0000) 2019-10-13T04:09:15.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:15.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:15.847717+0000) 2019-10-13T04:09:16.311 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:09:16.311 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:16.323 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:09:16.323 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:17.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:17.337+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:17.338863+0000) 2019-10-13T04:09:18.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:18.128+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:18.130087+0000) 2019-10-13T04:09:19.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:19.151+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.152122+0000) 2019-10-13T04:09:19.468 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:19.465+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.467332+0000) 2019-10-13T04:09:19.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:19.466+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.467466+0000) 2019-10-13T04:09:19.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:19.466+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.468272+0000) 2019-10-13T04:09:19.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:19.467+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.468488+0000) 2019-10-13T04:09:19.470 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:19.467+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.468670+0000) 2019-10-13T04:09:19.470 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:19.461+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.463447+0000) 2019-10-13T04:09:19.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:19.467+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:19.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:19.467+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.469038+0000) 2019-10-13T04:09:19.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:19.463+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:19.464930+0000) 2019-10-13T04:09:22.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:22.596+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:22.597378+0000) 2019-10-13T04:09:24.473 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:24.469+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.470929+0000) 2019-10-13T04:09:24.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:24.469+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.471141+0000) 2019-10-13T04:09:24.473 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:24.470+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.471457+0000) 2019-10-13T04:09:24.474 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:24.470+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.471533+0000) 2019-10-13T04:09:24.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:24.470+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.471624+0000) 2019-10-13T04:09:24.475 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:24.464+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.465969+0000) 2019-10-13T04:09:24.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:24.466+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.468485+0000) 2019-10-13T04:09:24.476 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:24.470+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.472156+0000) 2019-10-13T04:09:24.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:24.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:24.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:24.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.973006+0000) 2019-10-13T04:09:24.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:24.971+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.973002+0000) 2019-10-13T04:09:24.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:24.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:24.972997+0000) 2019-10-13T04:09:25.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:25.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:25.847887+0000) 2019-10-13T04:09:27.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:27.338+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:27.339070+0000) 2019-10-13T04:09:28.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:28.129+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:28.130320+0000) 2019-10-13T04:09:29.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:29.153+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.154578+0000) 2019-10-13T04:09:29.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:29.472+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.473986+0000) 2019-10-13T04:09:29.475 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:29.472+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.474381+0000) 2019-10-13T04:09:29.476 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:29.468+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.469330+0000) 2019-10-13T04:09:29.476 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:29.473+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.474716+0000) 2019-10-13T04:09:29.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:29.473+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.474976+0000) 2019-10-13T04:09:29.477 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:29.473+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.475269+0000) 2019-10-13T04:09:29.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:29.474+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.475748+0000) 2019-10-13T04:09:29.478 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:29.475+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:29.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:29.470+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:29.471596+0000) 2019-10-13T04:09:32.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:32.596+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:32.597579+0000) 2019-10-13T04:09:34.481 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:34.477+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.478429+0000) 2019-10-13T04:09:34.481 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:34.477+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.478590+0000) 2019-10-13T04:09:34.481 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:34.477+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.478789+0000) 2019-10-13T04:09:34.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:34.477+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.478993+0000) 2019-10-13T04:09:34.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:34.477+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.479265+0000) 2019-10-13T04:09:34.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:34.478+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.479604+0000) 2019-10-13T04:09:34.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:34.471+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.472694+0000) 2019-10-13T04:09:34.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:34.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:34.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:34.475+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.475838+0000) 2019-10-13T04:09:34.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:34.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.973166+0000) 2019-10-13T04:09:34.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:34.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.973164+0000) 2019-10-13T04:09:34.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:34.972+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:34.973164+0000) 2019-10-13T04:09:35.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:35.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:35.848076+0000) 2019-10-13T04:09:37.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:37.338+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:37.339281+0000) 2019-10-13T04:09:38.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:38.129+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:38.130515+0000) 2019-10-13T04:09:39.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:39.156+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.156810+0000) 2019-10-13T04:09:39.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:39.475+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.476201+0000) 2019-10-13T04:09:39.485 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:39.480+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.481765+0000) 2019-10-13T04:09:39.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:39.480+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.482020+0000) 2019-10-13T04:09:39.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:39.481+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.482431+0000) 2019-10-13T04:09:39.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:39.481+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.482481+0000) 2019-10-13T04:09:39.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:39.481+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.482847+0000) 2019-10-13T04:09:39.486 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:39.481+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.482765+0000) 2019-10-13T04:09:39.486 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:39.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:39.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:39.478+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:39.479264+0000) 2019-10-13T04:09:42.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:42.597+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:42.597763+0000) 2019-10-13T04:09:44.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:44.484+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.485166+0000) 2019-10-13T04:09:44.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:44.484+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.485149+0000) 2019-10-13T04:09:44.488 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:44.479+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.479955+0000) 2019-10-13T04:09:44.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:44.485+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.485857+0000) 2019-10-13T04:09:44.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:44.485+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.485946+0000) 2019-10-13T04:09:44.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:44.485+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.486170+0000) 2019-10-13T04:09:44.488 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:44.485+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.486291+0000) 2019-10-13T04:09:44.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:44.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:44.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:44.481+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.482390+0000) 2019-10-13T04:09:44.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:44.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.973332+0000) 2019-10-13T04:09:44.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:44.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.973331+0000) 2019-10-13T04:09:44.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:44.972+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:44.973349+0000) 2019-10-13T04:09:45.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:45.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:45.848297+0000) 2019-10-13T04:09:46.490 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:09:46.491 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:46.495 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:09:46.495 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:47.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:47.339+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:47.339476+0000) 2019-10-13T04:09:48.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:48.130+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:48.130690+0000) 2019-10-13T04:09:49.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:49.159+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.159344+0000) 2019-10-13T04:09:49.489 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:49.482+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.483095+0000) 2019-10-13T04:09:49.490 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:49.488+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.488659+0000) 2019-10-13T04:09:49.491 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:49.488+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.488860+0000) 2019-10-13T04:09:49.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:49.488+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.489324+0000) 2019-10-13T04:09:49.491 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:49.489+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.489402+0000) 2019-10-13T04:09:49.491 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:49.489+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.489608+0000) 2019-10-13T04:09:49.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:49.489+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.489723+0000) 2019-10-13T04:09:49.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:49.484+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:49.485143+0000) 2019-10-13T04:09:49.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:49.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:52.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:52.597+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:52.597948+0000) 2019-10-13T04:09:54.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:54.491+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.492014+0000) 2019-10-13T04:09:54.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:54.491+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.492383+0000) 2019-10-13T04:09:54.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:54.492+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.492533+0000) 2019-10-13T04:09:54.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:54.492+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.492976+0000) 2019-10-13T04:09:54.506 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:54.492+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.493162+0000) 2019-10-13T04:09:54.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:54.492+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.493168+0000) 2019-10-13T04:09:54.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:54.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:54.507 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:54.485+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.486447+0000) 2019-10-13T04:09:54.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:54.489+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.489900+0000) 2019-10-13T04:09:54.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:54.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.973506+0000) 2019-10-13T04:09:54.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:54.973+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.973503+0000) 2019-10-13T04:09:54.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:54.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:54.973505+0000) 2019-10-13T04:09:55.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:55.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:55.848493+0000) 2019-10-13T04:09:57.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:57.339+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:57.339678+0000) 2019-10-13T04:09:58.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:58.130+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:58.130936+0000) 2019-10-13T04:09:59.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:59.161+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.161721+0000) 2019-10-13T04:09:59.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:09:59.489+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.489974+0000) 2019-10-13T04:09:59.498 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:09:59.495+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.495565+0000) 2019-10-13T04:09:59.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:09:59.495+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.495675+0000) 2019-10-13T04:09:59.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:09:59.495+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.495808+0000) 2019-10-13T04:09:59.499 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:09:59.495+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.496097+0000) 2019-10-13T04:09:59.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:09:59.495+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.496291+0000) 2019-10-13T04:09:59.500 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:09:59.495+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.496389+0000) 2019-10-13T04:09:59.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:09:59.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:09:59.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:09:59.490+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:09:59.491517+0000) 2019-10-13T04:09:59.544 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1355, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2019-10-13T04:09:59.545 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1343, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2019-10-13T04:10:02.605 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:02.597+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:02.598115+0000) 2019-10-13T04:10:04.500 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:04.497+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.498771+0000) 2019-10-13T04:10:04.500 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:04.497+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.499019+0000) 2019-10-13T04:10:04.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:04.497+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.498848+0000) 2019-10-13T04:10:04.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:04.492+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.494196+0000) 2019-10-13T04:10:04.502 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:04.498+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.499540+0000) 2019-10-13T04:10:04.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:04.498+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.499833+0000) 2019-10-13T04:10:04.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:04.498+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.499992+0000) 2019-10-13T04:10:04.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:04.499+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:04.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:04.494+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.496423+0000) 2019-10-13T04:10:04.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:04.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.973662+0000) 2019-10-13T04:10:04.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:04.972+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.973665+0000) 2019-10-13T04:10:04.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:04.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:04.973663+0000) 2019-10-13T04:10:05.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:05.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:05.848679+0000) 2019-10-13T04:10:07.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:07.338+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:07.339870+0000) 2019-10-13T04:10:08.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:08.130+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:08.131114+0000) 2019-10-13T04:10:09.170 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:09.163+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.163996+0000) 2019-10-13T04:10:09.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:09.501+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.502636+0000) 2019-10-13T04:10:09.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:09.501+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.502886+0000) 2019-10-13T04:10:09.504 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:09.501+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.502938+0000) 2019-10-13T04:10:09.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:09.501+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.503097+0000) 2019-10-13T04:10:09.505 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:09.496+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.498395+0000) 2019-10-13T04:10:09.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:09.502+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.503538+0000) 2019-10-13T04:10:09.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:09.502+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.503939+0000) 2019-10-13T04:10:09.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:09.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:09.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:09.498+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:09.500696+0000) 2019-10-13T04:10:12.605 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:12.597+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:12.598288+0000) 2019-10-13T04:10:14.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:14.504+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.505774+0000) 2019-10-13T04:10:14.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:14.504+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.506065+0000) 2019-10-13T04:10:14.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:14.504+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.506158+0000) 2019-10-13T04:10:14.508 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:14.504+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.506272+0000) 2019-10-13T04:10:14.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:14.499+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.501036+0000) 2019-10-13T04:10:14.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:14.505+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.506789+0000) 2019-10-13T04:10:14.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:14.505+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.507074+0000) 2019-10-13T04:10:14.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:14.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:14.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:14.502+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.504083+0000) 2019-10-13T04:10:14.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:14.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.973837+0000) 2019-10-13T04:10:14.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:14.973+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.973836+0000) 2019-10-13T04:10:14.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:14.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:14.973845+0000) 2019-10-13T04:10:15.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:15.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:15.848852+0000) 2019-10-13T04:10:16.679 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:10:16.679 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:16.684 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:10:16.684 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:17.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:17.339+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:17.340068+0000) 2019-10-13T04:10:18.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:18.130+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:18.131290+0000) 2019-10-13T04:10:19.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:19.165+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.166300+0000) 2019-10-13T04:10:19.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:19.503+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.503871+0000) 2019-10-13T04:10:19.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:19.507+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.509378+0000) 2019-10-13T04:10:19.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:19.508+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.509698+0000) 2019-10-13T04:10:19.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:19.508+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.510113+0000) 2019-10-13T04:10:19.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:19.508+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.510159+0000) 2019-10-13T04:10:19.512 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:19.508+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.510294+0000) 2019-10-13T04:10:19.512 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:19.509+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.510506+0000) 2019-10-13T04:10:19.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:19.505+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:19.506279+0000) 2019-10-13T04:10:19.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:19.509+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:22.605 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:22.597+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:22.598471+0000) 2019-10-13T04:10:24.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:24.511+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.513123+0000) 2019-10-13T04:10:24.515 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:24.507+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.508072+0000) 2019-10-13T04:10:24.515 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:24.512+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.513619+0000) 2019-10-13T04:10:24.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:24.512+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.513595+0000) 2019-10-13T04:10:24.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:24.512+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.513900+0000) 2019-10-13T04:10:24.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:24.512+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.514245+0000) 2019-10-13T04:10:24.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:24.513+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.514436+0000) 2019-10-13T04:10:24.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:24.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:24.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:24.510+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.511082+0000) 2019-10-13T04:10:24.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:24.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.973975+0000) 2019-10-13T04:10:24.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:24.973+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.973976+0000) 2019-10-13T04:10:24.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:24.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:24.973975+0000) 2019-10-13T04:10:25.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:25.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:25.849065+0000) 2019-10-13T04:10:27.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:27.338+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:27.340289+0000) 2019-10-13T04:10:28.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:28.131+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:28.131481+0000) 2019-10-13T04:10:29.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:29.167+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.168579+0000) 2019-10-13T04:10:29.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:29.510+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.511366+0000) 2019-10-13T04:10:29.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:29.516+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.516985+0000) 2019-10-13T04:10:29.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:29.516+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.517136+0000) 2019-10-13T04:10:29.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:29.517+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.517392+0000) 2019-10-13T04:10:29.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:29.517+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.517494+0000) 2019-10-13T04:10:29.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:29.517+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.517739+0000) 2019-10-13T04:10:29.520 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:29.517+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.517860+0000) 2019-10-13T04:10:29.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:29.512+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:29.513108+0000) 2019-10-13T04:10:29.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:29.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:32.605 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:32.598+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:32.598673+0000) 2019-10-13T04:10:34.521 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:34.519+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.520259+0000) 2019-10-13T04:10:34.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:34.520+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.520497+0000) 2019-10-13T04:10:34.522 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:34.520+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.520753+0000) 2019-10-13T04:10:34.523 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:34.514+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.515250+0000) 2019-10-13T04:10:34.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:34.520+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.521122+0000) 2019-10-13T04:10:34.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:34.521+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.521418+0000) 2019-10-13T04:10:34.523 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:34.520+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.521271+0000) 2019-10-13T04:10:34.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:34.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:34.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:34.516+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.517600+0000) 2019-10-13T04:10:34.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:34.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.974163+0000) 2019-10-13T04:10:34.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:34.973+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.974163+0000) 2019-10-13T04:10:34.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:34.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:34.974162+0000) 2019-10-13T04:10:35.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:35.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:35.849249+0000) 2019-10-13T04:10:37.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:37.340+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:37.340478+0000) 2019-10-13T04:10:38.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:38.131+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:38.131663+0000) 2019-10-13T04:10:39.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:39.170+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.170873+0000) 2019-10-13T04:10:39.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:39.523+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.523553+0000) 2019-10-13T04:10:39.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:39.523+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.523729+0000) 2019-10-13T04:10:39.526 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:39.523+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.523833+0000) 2019-10-13T04:10:39.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:39.523+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.523966+0000) 2019-10-13T04:10:39.526 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:39.523+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.524313+0000) 2019-10-13T04:10:39.527 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:39.517+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.518694+0000) 2019-10-13T04:10:39.527 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:39.524+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.524622+0000) 2019-10-13T04:10:39.527 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:39.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 49977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:39.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:39.520+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:39.521310+0000) 2019-10-13T04:10:42.605 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:42.598+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:42.598880+0000) 2019-10-13T04:10:44.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:44.526+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.526616+0000) 2019-10-13T04:10:44.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:44.526+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.526872+0000) 2019-10-13T04:10:44.529 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:44.520+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.521345+0000) 2019-10-13T04:10:44.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:44.526+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.527309+0000) 2019-10-13T04:10:44.529 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:44.527+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.527705+0000) 2019-10-13T04:10:44.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:44.527+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.527592+0000) 2019-10-13T04:10:44.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:44.527+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.527810+0000) 2019-10-13T04:10:44.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:44.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50005 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:44.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:44.524+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.524831+0000) 2019-10-13T04:10:44.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:44.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.974348+0000) 2019-10-13T04:10:44.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:44.974+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.974350+0000) 2019-10-13T04:10:44.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:44.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:44.974352+0000) 2019-10-13T04:10:45.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:45.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:45.849431+0000) 2019-10-13T04:10:46.842 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:10:46.842 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:46.847 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:10:46.848 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:47.342 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:47.339+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:47.340702+0000) 2019-10-13T04:10:48.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:48.130+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:48.131841+0000) 2019-10-13T04:10:49.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:49.173+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.173364+0000) 2019-10-13T04:10:49.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:49.528+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.530302+0000) 2019-10-13T04:10:49.533 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:49.529+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.530447+0000) 2019-10-13T04:10:49.533 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:49.529+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.530613+0000) 2019-10-13T04:10:49.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:49.529+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.530740+0000) 2019-10-13T04:10:49.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:49.529+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.531231+0000) 2019-10-13T04:10:49.534 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:49.529+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.531268+0000) 2019-10-13T04:10:49.534 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:49.524+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.525501+0000) 2019-10-13T04:10:49.534 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:49.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:49.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:49.527+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:49.528309+0000) 2019-10-13T04:10:52.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:52.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:52.599059+0000) 2019-10-13T04:10:54.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:54.531+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.533189+0000) 2019-10-13T04:10:54.536 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:54.531+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.533305+0000) 2019-10-13T04:10:54.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:54.531+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.533361+0000) 2019-10-13T04:10:54.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:54.532+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.533488+0000) 2019-10-13T04:10:54.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:54.532+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.533595+0000) 2019-10-13T04:10:54.539 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:54.532+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.534000+0000) 2019-10-13T04:10:54.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:54.527+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.528802+0000) 2019-10-13T04:10:54.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:54.528+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.530432+0000) 2019-10-13T04:10:54.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:54.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:54.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:54.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.974555+0000) 2019-10-13T04:10:54.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:54.973+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.974551+0000) 2019-10-13T04:10:54.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:54.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:54.974557+0000) 2019-10-13T04:10:55.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:55.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:55.849617+0000) 2019-10-13T04:10:57.342 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:57.340+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:57.340911+0000) 2019-10-13T04:10:58.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:58.130+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:58.132043+0000) 2019-10-13T04:10:59.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:59.174+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.175774+0000) 2019-10-13T04:10:59.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:10:59.534+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.536226+0000) 2019-10-13T04:10:59.538 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:10:59.535+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.536622+0000) 2019-10-13T04:10:59.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:10:59.535+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.536856+0000) 2019-10-13T04:10:59.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:10:59.529+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.531089+0000) 2019-10-13T04:10:59.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:10:59.535+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.537041+0000) 2019-10-13T04:10:59.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:10:59.536+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.537480+0000) 2019-10-13T04:10:59.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:10:59.536+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.537659+0000) 2019-10-13T04:10:59.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:10:59.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:10:59.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:10:59.533+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:10:59.535007+0000) 2019-10-13T04:11:02.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:02.598+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:02.599225+0000) 2019-10-13T04:11:04.543 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:04.532+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.534479+0000) 2019-10-13T04:11:04.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:04.538+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.539661+0000) 2019-10-13T04:11:04.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:04.538+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.539745+0000) 2019-10-13T04:11:04.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:04.538+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.539976+0000) 2019-10-13T04:11:04.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:04.538+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.540195+0000) 2019-10-13T04:11:04.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:04.538+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.540344+0000) 2019-10-13T04:11:04.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:04.539+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.540874+0000) 2019-10-13T04:11:04.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:04.539+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:04.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:04.535+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.537196+0000) 2019-10-13T04:11:04.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:04.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.974732+0000) 2019-10-13T04:11:04.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:04.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.974734+0000) 2019-10-13T04:11:04.978 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:04.974+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:04.974731+0000) 2019-10-13T04:11:05.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:05.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:05.849851+0000) 2019-10-13T04:11:07.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:07.340+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:07.341104+0000) 2019-10-13T04:11:08.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:08.131+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:08.132210+0000) 2019-10-13T04:11:09.185 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:09.177+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.178207+0000) 2019-10-13T04:11:09.544 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:09.541+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.542515+0000) 2019-10-13T04:11:09.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:09.541+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.542848+0000) 2019-10-13T04:11:09.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:09.541+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.542975+0000) 2019-10-13T04:11:09.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:09.537+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.537771+0000) 2019-10-13T04:11:09.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:09.541+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.543326+0000) 2019-10-13T04:11:09.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:09.542+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.543599+0000) 2019-10-13T04:11:09.546 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:09.542+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.544221+0000) 2019-10-13T04:11:09.546 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:09.542+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:09.547 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:09.539+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:09.539878+0000) 2019-10-13T04:11:12.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:12.598+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:12.599398+0000) 2019-10-13T04:11:14.547 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:14.539+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.540412+0000) 2019-10-13T04:11:14.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:14.544+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.545989+0000) 2019-10-13T04:11:14.548 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:14.544+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.546387+0000) 2019-10-13T04:11:14.548 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:14.545+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.546503+0000) 2019-10-13T04:11:14.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:14.545+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.546611+0000) 2019-10-13T04:11:14.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:14.545+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.547042+0000) 2019-10-13T04:11:14.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:14.545+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.547092+0000) 2019-10-13T04:11:14.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:14.542+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.542753+0000) 2019-10-13T04:11:14.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:14.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:14.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:14.974+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.974935+0000) 2019-10-13T04:11:14.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:14.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.974930+0000) 2019-10-13T04:11:14.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:14.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:14.974930+0000) 2019-10-13T04:11:15.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:15.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:15.850039+0000) 2019-10-13T04:11:16.963 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:11:16.963 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:16.967 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:11:16.967 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:17.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:17.340+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:17.341345+0000) 2019-10-13T04:11:18.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:18.132+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:18.132386+0000) 2019-10-13T04:11:19.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:19.180+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.180663+0000) 2019-10-13T04:11:19.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:19.549+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.549403+0000) 2019-10-13T04:11:19.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:19.549+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.549623+0000) 2019-10-13T04:11:19.551 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:19.549+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.549868+0000) 2019-10-13T04:11:19.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:19.549+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.549998+0000) 2019-10-13T04:11:19.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:19.543+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.544691+0000) 2019-10-13T04:11:19.552 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:19.549+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.550230+0000) 2019-10-13T04:11:19.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:19.550+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.550419+0000) 2019-10-13T04:11:19.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:19.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:19.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:19.545+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:19.546051+0000) 2019-10-13T04:11:22.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:22.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:22.599568+0000) 2019-10-13T04:11:24.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:24.547+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.548580+0000) 2019-10-13T04:11:24.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:24.552+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.553065+0000) 2019-10-13T04:11:24.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:24.553+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.553524+0000) 2019-10-13T04:11:24.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:24.553+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.553630+0000) 2019-10-13T04:11:24.560 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:24.553+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.554116+0000) 2019-10-13T04:11:24.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:24.554+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.554774+0000) 2019-10-13T04:11:24.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:24.554+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.554932+0000) 2019-10-13T04:11:24.560 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:24.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:24.561 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:24.550+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.551566+0000) 2019-10-13T04:11:24.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:24.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.975069+0000) 2019-10-13T04:11:24.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:24.974+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.975069+0000) 2019-10-13T04:11:24.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:24.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:24.975071+0000) 2019-10-13T04:11:25.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:25.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:25.850247+0000) 2019-10-13T04:11:27.342 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:27.341+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:27.341510+0000) 2019-10-13T04:11:28.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:28.132+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:28.132599+0000) 2019-10-13T04:11:29.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:29.182+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.182953+0000) 2019-10-13T04:11:29.559 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:29.556+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.556666+0000) 2019-10-13T04:11:29.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:29.556+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.556854+0000) 2019-10-13T04:11:29.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:29.556+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.556897+0000) 2019-10-13T04:11:29.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:29.557+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.557410+0000) 2019-10-13T04:11:29.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:29.557+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.557586+0000) 2019-10-13T04:11:29.563 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:29.557+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.557690+0000) 2019-10-13T04:11:29.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:29.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:29.564 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:29.551+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.552159+0000) 2019-10-13T04:11:29.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:29.552+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:29.553679+0000) 2019-10-13T04:11:32.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:32.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:32.599763+0000) 2019-10-13T04:11:34.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:34.553+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.554663+0000) 2019-10-13T04:11:34.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:34.559+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.560411+0000) 2019-10-13T04:11:34.562 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:34.559+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.560555+0000) 2019-10-13T04:11:34.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:34.559+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.560720+0000) 2019-10-13T04:11:34.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:34.559+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.560962+0000) 2019-10-13T04:11:34.563 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:34.559+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.561019+0000) 2019-10-13T04:11:34.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:34.560+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.561699+0000) 2019-10-13T04:11:34.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:34.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:34.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:34.557+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.558051+0000) 2019-10-13T04:11:34.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:34.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.975235+0000) 2019-10-13T04:11:34.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:34.974+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.975234+0000) 2019-10-13T04:11:34.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:34.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:34.975239+0000) 2019-10-13T04:11:35.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:35.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:35.850433+0000) 2019-10-13T04:11:37.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:37.340+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:37.341688+0000) 2019-10-13T04:11:38.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:38.131+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:38.132781+0000) 2019-10-13T04:11:39.195 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:39.185+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.185342+0000) 2019-10-13T04:11:39.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:39.562+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.563794+0000) 2019-10-13T04:11:39.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:39.562+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.563973+0000) 2019-10-13T04:11:39.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:39.562+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.564171+0000) 2019-10-13T04:11:39.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:39.558+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.559251+0000) 2019-10-13T04:11:39.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:39.560+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.561554+0000) 2019-10-13T04:11:39.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:39.563+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.564414+0000) 2019-10-13T04:11:39.584 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:39.562+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.564357+0000) 2019-10-13T04:11:39.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:39.563+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:39.565103+0000) 2019-10-13T04:11:39.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:39.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:42.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:42.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:42.599942+0000) 2019-10-13T04:11:44.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:44.566+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.568315+0000) 2019-10-13T04:11:44.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:44.567+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.568551+0000) 2019-10-13T04:11:44.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:44.562+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.564038+0000) 2019-10-13T04:11:44.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:44.567+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.568921+0000) 2019-10-13T04:11:44.571 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:44.567+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.569132+0000) 2019-10-13T04:11:44.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:44.567+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.569324+0000) 2019-10-13T04:11:44.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:44.567+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.569218+0000) 2019-10-13T04:11:44.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:44.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:44.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:44.563+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.565674+0000) 2019-10-13T04:11:44.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:44.974+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.975435+0000) 2019-10-13T04:11:44.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:44.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.975438+0000) 2019-10-13T04:11:44.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:44.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:44.975439+0000) 2019-10-13T04:11:45.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:45.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:45.850638+0000) 2019-10-13T04:11:47.146 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:11:47.147 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:47.151 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:11:47.151 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:47.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:47.341+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:47.341887+0000) 2019-10-13T04:11:48.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:48.132+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:48.132992+0000) 2019-10-13T04:11:49.197 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:49.186+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.187604+0000) 2019-10-13T04:11:49.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:49.564+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.565792+0000) 2019-10-13T04:11:49.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:49.569+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.571068+0000) 2019-10-13T04:11:49.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:49.570+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.571681+0000) 2019-10-13T04:11:49.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:49.570+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.571895+0000) 2019-10-13T04:11:49.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:49.570+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.572139+0000) 2019-10-13T04:11:49.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:49.570+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.572032+0000) 2019-10-13T04:11:49.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:49.570+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.572334+0000) 2019-10-13T04:11:49.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:49.566+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:49.568131+0000) 2019-10-13T04:11:49.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:49.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:52.607 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:52.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:52.600118+0000) 2019-10-13T04:11:54.576 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:54.573+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.574802+0000) 2019-10-13T04:11:54.576 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:54.573+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.575094+0000) 2019-10-13T04:11:54.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:54.573+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.575228+0000) 2019-10-13T04:11:54.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:54.573+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.575362+0000) 2019-10-13T04:11:54.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:54.568+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.570396+0000) 2019-10-13T04:11:54.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:54.574+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.575482+0000) 2019-10-13T04:11:54.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:54.574+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.575755+0000) 2019-10-13T04:11:54.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:54.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:54.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:54.571+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.573049+0000) 2019-10-13T04:11:54.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:54.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.975617+0000) 2019-10-13T04:11:54.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:54.975+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.975614+0000) 2019-10-13T04:11:54.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:54.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:54.975619+0000) 2019-10-13T04:11:55.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:55.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:55.850823+0000) 2019-10-13T04:11:57.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:57.341+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:57.342118+0000) 2019-10-13T04:11:58.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:58.132+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:58.133190+0000) 2019-10-13T04:11:59.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:59.189+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.189940+0000) 2019-10-13T04:11:59.579 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:11:59.577+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.578457+0000) 2019-10-13T04:11:59.580 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:11:59.572+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.573316+0000) 2019-10-13T04:11:59.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:11:59.577+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.579209+0000) 2019-10-13T04:11:59.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:11:59.578+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.579466+0000) 2019-10-13T04:11:59.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:11:59.578+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.579496+0000) 2019-10-13T04:11:59.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:11:59.578+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.579767+0000) 2019-10-13T04:11:59.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:11:59.578+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.579922+0000) 2019-10-13T04:11:59.582 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:11:59.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:11:59.582 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:11:59.574+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:11:59.575579+0000) 2019-10-13T04:12:02.607 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:02.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:02.600326+0000) 2019-10-13T04:12:04.583 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:04.576+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.576791+0000) 2019-10-13T04:12:04.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:04.582+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.583186+0000) 2019-10-13T04:12:04.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:04.583+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.583422+0000) 2019-10-13T04:12:04.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:04.583+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.583662+0000) 2019-10-13T04:12:04.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:04.583+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.583764+0000) 2019-10-13T04:12:04.587 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:04.578+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.578970+0000) 2019-10-13T04:12:04.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:04.583+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.583897+0000) 2019-10-13T04:12:04.588 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:04.583+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.584119+0000) 2019-10-13T04:12:04.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:04.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:04.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:04.975+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.975772+0000) 2019-10-13T04:12:04.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:04.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.975772+0000) 2019-10-13T04:12:04.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:04.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:04.975772+0000) 2019-10-13T04:12:05.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:05.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:05.851001+0000) 2019-10-13T04:12:07.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:07.342+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:07.342286+0000) 2019-10-13T04:12:08.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:08.133+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:08.133356+0000) 2019-10-13T04:12:09.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:09.191+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.192269+0000) 2019-10-13T04:12:09.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:09.579+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.580491+0000) 2019-10-13T04:12:09.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:09.585+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.586062+0000) 2019-10-13T04:12:09.588 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:09.585+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.586202+0000) 2019-10-13T04:12:09.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:09.585+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.586282+0000) 2019-10-13T04:12:09.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:09.586+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.586433+0000) 2019-10-13T04:12:09.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:09.586+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.586919+0000) 2019-10-13T04:12:09.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:09.586+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.587245+0000) 2019-10-13T04:12:09.589 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:09.587+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:09.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:09.583+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:09.583788+0000) 2019-10-13T04:12:12.607 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:12.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:12.600499+0000) 2019-10-13T04:12:14.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:14.583+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.583737+0000) 2019-10-13T04:12:14.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:14.589+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.589410+0000) 2019-10-13T04:12:14.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:14.589+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.589562+0000) 2019-10-13T04:12:14.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:14.589+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.589938+0000) 2019-10-13T04:12:14.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:14.589+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.590235+0000) 2019-10-13T04:12:14.592 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:14.589+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.590361+0000) 2019-10-13T04:12:14.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:14.590+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.590898+0000) 2019-10-13T04:12:14.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:14.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:14.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:14.585+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.586570+0000) 2019-10-13T04:12:14.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:14.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.975956+0000) 2019-10-13T04:12:14.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:14.975+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.975955+0000) 2019-10-13T04:12:14.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:14.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:14.975954+0000) 2019-10-13T04:12:15.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:15.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:15.851183+0000) 2019-10-13T04:12:17.305 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:12:17.305 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:17.309 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:12:17.309 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:17.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:17.342+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:17.342461+0000) 2019-10-13T04:12:18.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:18.132+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:18.133546+0000) 2019-10-13T04:12:19.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:19.193+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.194631+0000) 2019-10-13T04:12:19.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:19.587+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.587951+0000) 2019-10-13T04:12:19.596 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:19.591+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.592807+0000) 2019-10-13T04:12:19.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:19.591+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.593031+0000) 2019-10-13T04:12:19.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:19.591+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.593246+0000) 2019-10-13T04:12:19.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:19.592+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.593616+0000) 2019-10-13T04:12:19.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:19.592+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.593715+0000) 2019-10-13T04:12:19.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:19.592+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.593857+0000) 2019-10-13T04:12:19.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:19.593+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:19.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:19.589+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:19.589824+0000) 2019-10-13T04:12:22.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:22.599+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:22.600698+0000) 2019-10-13T04:12:24.598 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:24.594+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.595955+0000) 2019-10-13T04:12:24.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:24.595+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.596408+0000) 2019-10-13T04:12:24.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:24.595+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.596566+0000) 2019-10-13T04:12:24.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:24.595+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.596843+0000) 2019-10-13T04:12:24.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:24.595+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.597100+0000) 2019-10-13T04:12:24.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:24.591+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.592046+0000) 2019-10-13T04:12:24.601 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:24.593+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.594179+0000) 2019-10-13T04:12:24.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:24.596+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.597654+0000) 2019-10-13T04:12:24.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:24.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:24.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:24.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.976145+0000) 2019-10-13T04:12:24.998 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:24.974+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.976147+0000) 2019-10-13T04:12:24.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:24.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:24.976145+0000) 2019-10-13T04:12:25.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:25.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:25.851413+0000) 2019-10-13T04:12:27.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:27.341+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:27.342633+0000) 2019-10-13T04:12:28.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:28.132+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:28.133748+0000) 2019-10-13T04:12:29.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:29.196+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.196896+0000) 2019-10-13T04:12:29.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:29.598+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.599938+0000) 2019-10-13T04:12:29.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:29.598+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.600053+0000) 2019-10-13T04:12:29.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:29.594+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.595597+0000) 2019-10-13T04:12:29.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:29.599+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.600652+0000) 2019-10-13T04:12:29.603 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:29.599+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.600815+0000) 2019-10-13T04:12:29.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:29.599+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.600924+0000) 2019-10-13T04:12:29.604 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:29.599+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.601219+0000) 2019-10-13T04:12:29.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:29.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50638 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:29.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:29.596+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:29.596925+0000) 2019-10-13T04:12:32.607 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:32.600+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:32.600885+0000) 2019-10-13T04:12:34.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:34.602+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.603856+0000) 2019-10-13T04:12:34.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:34.596+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.598585+0000) 2019-10-13T04:12:34.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:34.603+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.604394+0000) 2019-10-13T04:12:34.606 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:34.603+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.604571+0000) 2019-10-13T04:12:34.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:34.603+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.604705+0000) 2019-10-13T04:12:34.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:34.603+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.604893+0000) 2019-10-13T04:12:34.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:34.603+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.605305+0000) 2019-10-13T04:12:34.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:34.598+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.600559+0000) 2019-10-13T04:12:34.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:34.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:34.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:34.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.976330+0000) 2019-10-13T04:12:34.978 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:34.975+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.976333+0000) 2019-10-13T04:12:34.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:34.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:34.976334+0000) 2019-10-13T04:12:35.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:35.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:35.851600+0000) 2019-10-13T04:12:37.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:37.342+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:37.342828+0000) 2019-10-13T04:12:38.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:38.133+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:38.133940+0000) 2019-10-13T04:12:39.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:39.198+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.199200+0000) 2019-10-13T04:12:39.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:39.606+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.607815+0000) 2019-10-13T04:12:39.610 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:39.606+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.608083+0000) 2019-10-13T04:12:39.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:39.607+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.608461+0000) 2019-10-13T04:12:39.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:39.607+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.608607+0000) 2019-10-13T04:12:39.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:39.607+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.608766+0000) 2019-10-13T04:12:39.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:39.601+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.602841+0000) 2019-10-13T04:12:39.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:39.602+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.604515+0000) 2019-10-13T04:12:39.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:39.607+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:39.609145+0000) 2019-10-13T04:12:39.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:39.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:42.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:42.600+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:42.601066+0000) 2019-10-13T04:12:44.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:44.610+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.611560+0000) 2019-10-13T04:12:44.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:44.610+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.611665+0000) 2019-10-13T04:12:44.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:44.610+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.611853+0000) 2019-10-13T04:12:44.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:44.604+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.606502+0000) 2019-10-13T04:12:44.614 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:44.610+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.612267+0000) 2019-10-13T04:12:44.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:44.611+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.612666+0000) 2019-10-13T04:12:44.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:44.611+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.613040+0000) 2019-10-13T04:12:44.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:44.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50726 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:44.616 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:44.608+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.609966+0000) 2019-10-13T04:12:44.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:44.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.976503+0000) 2019-10-13T04:12:44.978 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:44.976+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.976530+0000) 2019-10-13T04:12:44.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:44.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:44.976506+0000) 2019-10-13T04:12:45.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:45.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:45.851807+0000) 2019-10-13T04:12:47.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:47.342+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:47.343020+0000) 2019-10-13T04:12:47.426 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:12:47.426 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:47.431 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:12:47.431 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:48.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:48.133+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:48.134149+0000) 2019-10-13T04:12:49.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:49.200+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.201629+0000) 2019-10-13T04:12:49.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:49.609+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.609882+0000) 2019-10-13T04:12:49.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:49.615+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.615833+0000) 2019-10-13T04:12:49.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:49.615+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.615936+0000) 2019-10-13T04:12:49.618 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:49.615+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.616223+0000) 2019-10-13T04:12:49.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:49.615+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.616388+0000) 2019-10-13T04:12:49.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:49.616+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.616551+0000) 2019-10-13T04:12:49.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:49.616+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.616726+0000) 2019-10-13T04:12:49.619 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:49.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:49.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:49.613+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:49.614116+0000) 2019-10-13T04:12:52.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:52.600+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:52.601259+0000) 2019-10-13T04:12:54.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:54.618+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.619006+0000) 2019-10-13T04:12:54.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:54.618+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.619272+0000) 2019-10-13T04:12:54.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:54.613+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.614594+0000) 2019-10-13T04:12:54.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:54.619+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.619669+0000) 2019-10-13T04:12:54.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:54.619+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.619876+0000) 2019-10-13T04:12:54.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:54.619+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.620172+0000) 2019-10-13T04:12:54.622 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:54.619+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.620310+0000) 2019-10-13T04:12:54.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:54.620+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:54.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:54.615+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.616346+0000) 2019-10-13T04:12:54.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:54.976+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.976678+0000) 2019-10-13T04:12:54.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:54.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.976651+0000) 2019-10-13T04:12:54.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:54.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:54.976669+0000) 2019-10-13T04:12:55.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:55.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:55.852015+0000) 2019-10-13T04:12:57.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:57.343+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:57.343216+0000) 2019-10-13T04:12:58.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:58.134+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:58.134343+0000) 2019-10-13T04:12:59.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:59.203+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.203885+0000) 2019-10-13T04:12:59.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:12:59.616+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.616782+0000) 2019-10-13T04:12:59.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:12:59.621+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.622366+0000) 2019-10-13T04:12:59.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:12:59.622+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.622785+0000) 2019-10-13T04:12:59.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:12:59.622+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.623064+0000) 2019-10-13T04:12:59.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:12:59.622+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.623262+0000) 2019-10-13T04:12:59.625 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:12:59.623+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.623414+0000) 2019-10-13T04:12:59.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:12:59.623+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.623526+0000) 2019-10-13T04:12:59.626 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:12:59.624+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50817 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:12:59.631 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:12:59.619+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:12:59.619969+0000) 2019-10-13T04:13:02.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:02.600+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:02.601418+0000) 2019-10-13T04:13:04.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:04.619+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.620652+0000) 2019-10-13T04:13:04.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:04.625+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.626476+0000) 2019-10-13T04:13:04.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:04.625+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.626639+0000) 2019-10-13T04:13:04.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:04.625+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.627154+0000) 2019-10-13T04:13:04.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:04.625+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.627225+0000) 2019-10-13T04:13:04.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:04.626+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.627410+0000) 2019-10-13T04:13:04.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:04.626+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.627723+0000) 2019-10-13T04:13:04.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:04.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:04.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:04.622+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.623645+0000) 2019-10-13T04:13:04.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:04.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.976850+0000) 2019-10-13T04:13:04.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:04.975+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.976851+0000) 2019-10-13T04:13:04.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:04.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:04.976851+0000) 2019-10-13T04:13:05.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:05.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:05.852222+0000) 2019-10-13T04:13:07.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:07.342+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:07.343437+0000) 2019-10-13T04:13:08.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:08.133+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:08.134576+0000) 2019-10-13T04:13:09.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:09.205+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.206233+0000) 2019-10-13T04:13:09.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:09.624+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.624806+0000) 2019-10-13T04:13:09.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:09.629+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.630581+0000) 2019-10-13T04:13:09.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:09.629+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.630746+0000) 2019-10-13T04:13:09.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:09.629+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.630982+0000) 2019-10-13T04:13:09.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:09.626+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.627020+0000) 2019-10-13T04:13:09.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:09.629+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.631230+0000) 2019-10-13T04:13:09.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:09.629+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.631138+0000) 2019-10-13T04:13:09.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:09.629+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:09.631342+0000) 2019-10-13T04:13:09.654 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:09.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:12.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:12.601+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:12.601617+0000) 2019-10-13T04:13:14.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:14.632+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.633597+0000) 2019-10-13T04:13:14.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:14.632+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.633880+0000) 2019-10-13T04:13:14.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:14.632+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.634053+0000) 2019-10-13T04:13:14.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:14.633+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.634434+0000) 2019-10-13T04:13:14.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:14.633+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.634652+0000) 2019-10-13T04:13:14.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:14.633+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.634840+0000) 2019-10-13T04:13:14.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:14.628+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.628857+0000) 2019-10-13T04:13:14.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:14.634+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:14.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:14.631+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.631741+0000) 2019-10-13T04:13:14.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:14.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.976997+0000) 2019-10-13T04:13:14.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:14.975+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.977004+0000) 2019-10-13T04:13:14.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:14.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:14.977017+0000) 2019-10-13T04:13:15.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:15.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:15.852424+0000) 2019-10-13T04:13:17.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:17.342+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:17.343630+0000) 2019-10-13T04:13:17.597 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:13:17.598 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:17.603 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:13:17.604 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:18.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:18.134+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:18.134756+0000) 2019-10-13T04:13:19.215 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:19.207+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.208590+0000) 2019-10-13T04:13:19.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:19.631+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.632115+0000) 2019-10-13T04:13:19.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:19.636+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.637752+0000) 2019-10-13T04:13:19.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:19.636+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.637932+0000) 2019-10-13T04:13:19.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:19.636+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.638187+0000) 2019-10-13T04:13:19.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:19.637+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.638472+0000) 2019-10-13T04:13:19.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:19.637+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.638632+0000) 2019-10-13T04:13:19.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:19.637+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.638898+0000) 2019-10-13T04:13:19.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:19.634+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:19.634915+0000) 2019-10-13T04:13:19.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:19.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:22.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:22.601+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:22.601796+0000) 2019-10-13T04:13:24.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:24.639+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.641276+0000) 2019-10-13T04:13:24.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:24.640+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.641509+0000) 2019-10-13T04:13:24.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:24.635+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.636863+0000) 2019-10-13T04:13:24.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:24.640+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.641778+0000) 2019-10-13T04:13:24.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:24.640+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.641862+0000) 2019-10-13T04:13:24.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:24.640+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.642356+0000) 2019-10-13T04:13:24.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:24.641+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.642729+0000) 2019-10-13T04:13:24.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:24.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:24.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:24.637+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.639564+0000) 2019-10-13T04:13:24.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:24.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.977162+0000) 2019-10-13T04:13:24.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:24.976+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.977161+0000) 2019-10-13T04:13:24.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:24.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:24.977166+0000) 2019-10-13T04:13:25.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:25.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:25.852583+0000) 2019-10-13T04:13:27.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:27.343+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:27.343827+0000) 2019-10-13T04:13:28.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:28.134+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:28.134950+0000) 2019-10-13T04:13:29.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:29.210+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.210923+0000) 2019-10-13T04:13:29.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:29.642+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.644134+0000) 2019-10-13T04:13:29.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:29.642+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.644294+0000) 2019-10-13T04:13:29.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:29.643+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.644499+0000) 2019-10-13T04:13:29.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:29.643+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.644585+0000) 2019-10-13T04:13:29.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:29.637+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.639529+0000) 2019-10-13T04:13:29.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:29.643+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.645109+0000) 2019-10-13T04:13:29.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:29.643+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.645273+0000) 2019-10-13T04:13:29.647 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:29.644+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 50998 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:29.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:29.640+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:29.641878+0000) 2019-10-13T04:13:32.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:32.601+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:32.601979+0000) 2019-10-13T04:13:34.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:34.647+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.647697+0000) 2019-10-13T04:13:34.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:34.647+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.647747+0000) 2019-10-13T04:13:34.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:34.647+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.647991+0000) 2019-10-13T04:13:34.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:34.647+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.648206+0000) 2019-10-13T04:13:34.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:34.641+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.642841+0000) 2019-10-13T04:13:34.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:34.648+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.648471+0000) 2019-10-13T04:13:34.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:34.648+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.648581+0000) 2019-10-13T04:13:34.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:34.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:34.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:34.644+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.645924+0000) 2019-10-13T04:13:34.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:34.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.977353+0000) 2019-10-13T04:13:34.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:34.976+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.977347+0000) 2019-10-13T04:13:34.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:34.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:34.977350+0000) 2019-10-13T04:13:35.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:35.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:35.852721+0000) 2019-10-13T04:13:37.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:37.343+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:37.343996+0000) 2019-10-13T04:13:38.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:38.134+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:38.135132+0000) 2019-10-13T04:13:39.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:39.212+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.213300+0000) 2019-10-13T04:13:39.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:39.650+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.651348+0000) 2019-10-13T04:13:39.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:39.651+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.651699+0000) 2019-10-13T04:13:39.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:39.651+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.652310+0000) 2019-10-13T04:13:39.654 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:39.651+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.652379+0000) 2019-10-13T04:13:39.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:39.652+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.652525+0000) 2019-10-13T04:13:39.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:39.652+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.652852+0000) 2019-10-13T04:13:39.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:39.646+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.647261+0000) 2019-10-13T04:13:39.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:39.647+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:39.648120+0000) 2019-10-13T04:13:39.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:39.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:42.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:42.601+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:42.602135+0000) 2019-10-13T04:13:44.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:44.654+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.655166+0000) 2019-10-13T04:13:44.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:44.649+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.649999+0000) 2019-10-13T04:13:44.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:44.651+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.652017+0000) 2019-10-13T04:13:44.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:44.655+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.655709+0000) 2019-10-13T04:13:44.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:44.655+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.655817+0000) 2019-10-13T04:13:44.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:44.655+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.656118+0000) 2019-10-13T04:13:44.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:44.655+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.656269+0000) 2019-10-13T04:13:44.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:44.656+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.656429+0000) 2019-10-13T04:13:44.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:44.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:44.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:44.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.977552+0000) 2019-10-13T04:13:44.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:44.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.977554+0000) 2019-10-13T04:13:44.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:44.977+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:44.977552+0000) 2019-10-13T04:13:45.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:45.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:45.852929+0000) 2019-10-13T04:13:47.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:47.344+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:47.344173+0000) 2019-10-13T04:13:47.761 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:13:47.762 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:47.766 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:13:47.766 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:48.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:48.135+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:48.135295+0000) 2019-10-13T04:13:49.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:49.215+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.215646+0000) 2019-10-13T04:13:49.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:49.657+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.658635+0000) 2019-10-13T04:13:49.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:49.657+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.658895+0000) 2019-10-13T04:13:49.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:49.657+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.659096+0000) 2019-10-13T04:13:49.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:49.657+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.659221+0000) 2019-10-13T04:13:49.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:49.653+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.653896+0000) 2019-10-13T04:13:49.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:49.658+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.659487+0000) 2019-10-13T04:13:49.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:49.658+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.659574+0000) 2019-10-13T04:13:49.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:49.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:49.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:49.656+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:49.656830+0000) 2019-10-13T04:13:52.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:52.601+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:52.602316+0000) 2019-10-13T04:13:54.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:54.661+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.662673+0000) 2019-10-13T04:13:54.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:54.661+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.662547+0000) 2019-10-13T04:13:54.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:54.661+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.662791+0000) 2019-10-13T04:13:54.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:54.661+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.662916+0000) 2019-10-13T04:13:54.666 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:54.657+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.657799+0000) 2019-10-13T04:13:54.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:54.662+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.663430+0000) 2019-10-13T04:13:54.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:54.662+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.663690+0000) 2019-10-13T04:13:54.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:54.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:13:54.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:54.659+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.660415+0000) 2019-10-13T04:13:54.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:54.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.977709+0000) 2019-10-13T04:13:54.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:54.976+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.977707+0000) 2019-10-13T04:13:54.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:54.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:54.977703+0000) 2019-10-13T04:13:55.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:55.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:55.853146+0000) 2019-10-13T04:13:57.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:57.343+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:57.344370+0000) 2019-10-13T04:13:58.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:58.134+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:58.135497+0000) 2019-10-13T04:13:59.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:59.217+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.217910+0000) 2019-10-13T04:13:59.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:13:59.664+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.665990+0000) 2019-10-13T04:13:59.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:13:59.664+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.666324+0000) 2019-10-13T04:13:59.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:13:59.665+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.666676+0000) 2019-10-13T04:13:59.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:13:59.665+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.666886+0000) 2019-10-13T04:13:59.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:13:59.665+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.667127+0000) 2019-10-13T04:13:59.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:13:59.665+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.667283+0000) 2019-10-13T04:13:59.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:13:59.661+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.662066+0000) 2019-10-13T04:13:59.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:13:59.662+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:13:59.663099+0000) 2019-10-13T04:13:59.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:13:59.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51177 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:02.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:02.602+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:02.602535+0000) 2019-10-13T04:14:04.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:04.668+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.669913+0000) 2019-10-13T04:14:04.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:04.668+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.670235+0000) 2019-10-13T04:14:04.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:04.663+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.664606+0000) 2019-10-13T04:14:04.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:04.669+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.670498+0000) 2019-10-13T04:14:04.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:04.669+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.670790+0000) 2019-10-13T04:14:04.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:04.669+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.671322+0000) 2019-10-13T04:14:04.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:04.669+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.671323+0000) 2019-10-13T04:14:04.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:04.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:04.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:04.667+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.668009+0000) 2019-10-13T04:14:04.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:04.977+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.977823+0000) 2019-10-13T04:14:04.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:04.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.977823+0000) 2019-10-13T04:14:04.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:04.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:04.977823+0000) 2019-10-13T04:14:05.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:05.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:05.853336+0000) 2019-10-13T04:14:07.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:07.343+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:07.344574+0000) 2019-10-13T04:14:08.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:08.135+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:08.135694+0000) 2019-10-13T04:14:09.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:09.220+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.220375+0000) 2019-10-13T04:14:09.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:09.667+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.668039+0000) 2019-10-13T04:14:09.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:09.672+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.673788+0000) 2019-10-13T04:14:09.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:09.672+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.674318+0000) 2019-10-13T04:14:09.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:09.673+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.674643+0000) 2019-10-13T04:14:09.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:09.673+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.674897+0000) 2019-10-13T04:14:09.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:09.673+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.675131+0000) 2019-10-13T04:14:09.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:09.673+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.675297+0000) 2019-10-13T04:14:09.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:09.670+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:09.671165+0000) 2019-10-13T04:14:09.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:09.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51238 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:12.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:12.602+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:12.602723+0000) 2019-10-13T04:14:14.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:14.676+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.677793+0000) 2019-10-13T04:14:14.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:14.676+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.678127+0000) 2019-10-13T04:14:14.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:14.676+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.678196+0000) 2019-10-13T04:14:14.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:14.677+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.678678+0000) 2019-10-13T04:14:14.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:14.677+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.678832+0000) 2019-10-13T04:14:14.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:14.677+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.679066+0000) 2019-10-13T04:14:14.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:14.678+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:14.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:14.670+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.672130+0000) 2019-10-13T04:14:14.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:14.673+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.675215+0000) 2019-10-13T04:14:14.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:14.977+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.977968+0000) 2019-10-13T04:14:14.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:14.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.977970+0000) 2019-10-13T04:14:14.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:14.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:14.977968+0000) 2019-10-13T04:14:15.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:15.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:15.853509+0000) 2019-10-13T04:14:17.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:17.344+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:17.344778+0000) 2019-10-13T04:14:17.909 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:14:17.910 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:17.935 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:14:17.935 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:18.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:18.135+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:18.135888+0000) 2019-10-13T04:14:19.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:19.221+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.222693+0000) 2019-10-13T04:14:19.683 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:19.680+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.681334+0000) 2019-10-13T04:14:19.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:19.681+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.681552+0000) 2019-10-13T04:14:19.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:19.681+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.681593+0000) 2019-10-13T04:14:19.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:19.674+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.676348+0000) 2019-10-13T04:14:19.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:19.675+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.677555+0000) 2019-10-13T04:14:19.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:19.681+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.682147+0000) 2019-10-13T04:14:19.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:19.681+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.682322+0000) 2019-10-13T04:14:19.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:19.682+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:19.682597+0000) 2019-10-13T04:14:19.686 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:19.683+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:22.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:22.602+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:22.602942+0000) 2019-10-13T04:14:24.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:24.684+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.684771+0000) 2019-10-13T04:14:24.687 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:24.684+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.684903+0000) 2019-10-13T04:14:24.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:24.684+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.684991+0000) 2019-10-13T04:14:24.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:24.684+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.685356+0000) 2019-10-13T04:14:24.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:24.677+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.679686+0000) 2019-10-13T04:14:24.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:24.685+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.685526+0000) 2019-10-13T04:14:24.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:24.685+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.685903+0000) 2019-10-13T04:14:24.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:24.686+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:24.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:24.680+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.681996+0000) 2019-10-13T04:14:24.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:24.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.978148+0000) 2019-10-13T04:14:24.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:24.977+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.978143+0000) 2019-10-13T04:14:24.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:24.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:24.978150+0000) 2019-10-13T04:14:25.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:25.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:25.853691+0000) 2019-10-13T04:14:27.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:27.344+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:27.344972+0000) 2019-10-13T04:14:28.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:28.135+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:28.136095+0000) 2019-10-13T04:14:29.231 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:29.224+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.225059+0000) 2019-10-13T04:14:29.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:29.688+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.688671+0000) 2019-10-13T04:14:29.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:29.688+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.688802+0000) 2019-10-13T04:14:29.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:29.688+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.688941+0000) 2019-10-13T04:14:29.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:29.683+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.683740+0000) 2019-10-13T04:14:29.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:29.688+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.689376+0000) 2019-10-13T04:14:29.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:29.689+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.689750+0000) 2019-10-13T04:14:29.692 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:29.689+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.689970+0000) 2019-10-13T04:14:29.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:29.690+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51358 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:29.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:29.685+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:29.685779+0000) 2019-10-13T04:14:32.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:32.602+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:32.603099+0000) 2019-10-13T04:14:34.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:34.686+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.686993+0000) 2019-10-13T04:14:34.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:34.690+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.692297+0000) 2019-10-13T04:14:34.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:34.691+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.692651+0000) 2019-10-13T04:14:34.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:34.691+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.692786+0000) 2019-10-13T04:14:34.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:34.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.692867+0000) 2019-10-13T04:14:34.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:34.688+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.688988+0000) 2019-10-13T04:14:34.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:34.691+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.693155+0000) 2019-10-13T04:14:34.708 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:34.691+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.693278+0000) 2019-10-13T04:14:34.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:34.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:34.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:34.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.978337+0000) 2019-10-13T04:14:34.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:34.977+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.978343+0000) 2019-10-13T04:14:34.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:34.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:34.978337+0000) 2019-10-13T04:14:35.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:35.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:35.853877+0000) 2019-10-13T04:14:37.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:37.344+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:37.345163+0000) 2019-10-13T04:14:38.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:38.135+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:38.136284+0000) 2019-10-13T04:14:39.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:39.226+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.227335+0000) 2019-10-13T04:14:39.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:39.689+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.689986+0000) 2019-10-13T04:14:39.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:39.694+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.695730+0000) 2019-10-13T04:14:39.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:39.694+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.695974+0000) 2019-10-13T04:14:39.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:39.694+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.696138+0000) 2019-10-13T04:14:39.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:39.694+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.696235+0000) 2019-10-13T04:14:39.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:39.694+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.696386+0000) 2019-10-13T04:14:39.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:39.695+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.696565+0000) 2019-10-13T04:14:39.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:39.691+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:39.692155+0000) 2019-10-13T04:14:39.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:39.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51417 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:42.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:42.602+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:42.603283+0000) 2019-10-13T04:14:44.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:44.692+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.693510+0000) 2019-10-13T04:14:44.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:44.697+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.699216+0000) 2019-10-13T04:14:44.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:44.697+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.699332+0000) 2019-10-13T04:14:44.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:44.698+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.699651+0000) 2019-10-13T04:14:44.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:44.698+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.699986+0000) 2019-10-13T04:14:44.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:44.698+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.700203+0000) 2019-10-13T04:14:44.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:44.698+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.700316+0000) 2019-10-13T04:14:44.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:44.695+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.696722+0000) 2019-10-13T04:14:44.704 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:44.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:44.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:44.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.978510+0000) 2019-10-13T04:14:44.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:44.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.978544+0000) 2019-10-13T04:14:44.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:44.977+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:44.978547+0000) 2019-10-13T04:14:45.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:45.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:45.854068+0000) 2019-10-13T04:14:47.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:47.344+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:47.345350+0000) 2019-10-13T04:14:48.124 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:14:48.124 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:48.129 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:14:48.129 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:48.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:48.135+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:48.136433+0000) 2019-10-13T04:14:49.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:49.229+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.229606+0000) 2019-10-13T04:14:49.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:49.701+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.702969+0000) 2019-10-13T04:14:49.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:49.701+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.703223+0000) 2019-10-13T04:14:49.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:49.697+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.698354+0000) 2019-10-13T04:14:49.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:49.702+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.703635+0000) 2019-10-13T04:14:49.706 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:49.702+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.703901+0000) 2019-10-13T04:14:49.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:49.702+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.704011+0000) 2019-10-13T04:14:49.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:49.702+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.704145+0000) 2019-10-13T04:14:49.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:49.703+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51478 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:49.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:49.701+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:49.701965+0000) 2019-10-13T04:14:52.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:52.603+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:52.603493+0000) 2019-10-13T04:14:54.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:54.704+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.706118+0000) 2019-10-13T04:14:54.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:54.704+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.706365+0000) 2019-10-13T04:14:54.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:54.700+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.701329+0000) 2019-10-13T04:14:54.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:54.705+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.707224+0000) 2019-10-13T04:14:54.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:54.705+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.707330+0000) 2019-10-13T04:14:54.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:54.706+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.707526+0000) 2019-10-13T04:14:54.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:54.706+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.707452+0000) 2019-10-13T04:14:54.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:54.706+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:54.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:54.702+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.703605+0000) 2019-10-13T04:14:54.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:54.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.978701+0000) 2019-10-13T04:14:54.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:54.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.978743+0000) 2019-10-13T04:14:54.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:54.978+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:54.978721+0000) 2019-10-13T04:14:55.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:55.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:55.854224+0000) 2019-10-13T04:14:57.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:57.344+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:57.345492+0000) 2019-10-13T04:14:58.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:58.136+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:58.136594+0000) 2019-10-13T04:14:59.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:59.231+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.231956+0000) 2019-10-13T04:14:59.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:14:59.704+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.705074+0000) 2019-10-13T04:14:59.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:14:59.708+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.710171+0000) 2019-10-13T04:14:59.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:14:59.709+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.710503+0000) 2019-10-13T04:14:59.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:14:59.709+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.710650+0000) 2019-10-13T04:14:59.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:14:59.709+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.710900+0000) 2019-10-13T04:14:59.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:14:59.709+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.711317+0000) 2019-10-13T04:14:59.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:14:59.709+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.711276+0000) 2019-10-13T04:14:59.714 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:14:59.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:14:59.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:14:59.707+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:14:59.708224+0000) 2019-10-13T04:15:02.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:02.603+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:02.603675+0000) 2019-10-13T04:15:04.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:04.712+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.713002+0000) 2019-10-13T04:15:04.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:04.712+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.713368+0000) 2019-10-13T04:15:04.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:04.713+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.713503+0000) 2019-10-13T04:15:04.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:04.713+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.713723+0000) 2019-10-13T04:15:04.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:04.713+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.713841+0000) 2019-10-13T04:15:04.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:04.713+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.713605+0000) 2019-10-13T04:15:04.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:04.707+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.708680+0000) 2019-10-13T04:15:04.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:04.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:04.717 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:04.710+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.710870+0000) 2019-10-13T04:15:04.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:04.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.978858+0000) 2019-10-13T04:15:04.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:04.978+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.978866+0000) 2019-10-13T04:15:04.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:04.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:04.978895+0000) 2019-10-13T04:15:05.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:05.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:05.854390+0000) 2019-10-13T04:15:07.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:07.345+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:07.345672+0000) 2019-10-13T04:15:08.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:08.136+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:08.136791+0000) 2019-10-13T04:15:09.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:09.233+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.234299+0000) 2019-10-13T04:15:09.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:09.715+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.715816+0000) 2019-10-13T04:15:09.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:09.715+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.716007+0000) 2019-10-13T04:15:09.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:09.715+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.716215+0000) 2019-10-13T04:15:09.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:09.715+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.716324+0000) 2019-10-13T04:15:09.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:09.715+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.716376+0000) 2019-10-13T04:15:09.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:09.716+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.716562+0000) 2019-10-13T04:15:09.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:09.708+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.710652+0000) 2019-10-13T04:15:09.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:09.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:09.720 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:09.711+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:09.713357+0000) 2019-10-13T04:15:12.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:12.602+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:12.603850+0000) 2019-10-13T04:15:14.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:14.718+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.718659+0000) 2019-10-13T04:15:14.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:14.718+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.718839+0000) 2019-10-13T04:15:14.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:14.718+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.718947+0000) 2019-10-13T04:15:14.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:14.712+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.714244+0000) 2019-10-13T04:15:14.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:14.719+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.719619+0000) 2019-10-13T04:15:14.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:14.718+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.719234+0000) 2019-10-13T04:15:14.723 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:14.719+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.720100+0000) 2019-10-13T04:15:14.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:14.715+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.716774+0000) 2019-10-13T04:15:14.723 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:14.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:14.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:14.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.979042+0000) 2019-10-13T04:15:14.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:14.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.979024+0000) 2019-10-13T04:15:14.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:14.978+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:14.979024+0000) 2019-10-13T04:15:15.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:15.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:15.854575+0000) 2019-10-13T04:15:17.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:17.345+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:17.345870+0000) 2019-10-13T04:15:18.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:18.136+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:18.136989+0000) 2019-10-13T04:15:18.289 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:15:18.289 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:15:18.294 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:15:18.294 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:15:19.243 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:19.235+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.236534+0000) 2019-10-13T04:15:19.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:19.722+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.722768+0000) 2019-10-13T04:15:19.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:19.722+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.723075+0000) 2019-10-13T04:15:19.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:19.716+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.718085+0000) 2019-10-13T04:15:19.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:19.722+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.723317+0000) 2019-10-13T04:15:19.726 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:19.723+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.723729+0000) 2019-10-13T04:15:19.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:19.723+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.724015+0000) 2019-10-13T04:15:19.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:19.723+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.723972+0000) 2019-10-13T04:15:19.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:19.718+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:19.719906+0000) 2019-10-13T04:15:19.727 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:19.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51657 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:22.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:22.603+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:22.604031+0000) 2019-10-13T04:15:24.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:24.725+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.726780+0000) 2019-10-13T04:15:24.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:24.725+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.727048+0000) 2019-10-13T04:15:24.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:24.725+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.727203+0000) 2019-10-13T04:15:24.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:24.721+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.721882+0000) 2019-10-13T04:15:24.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:24.726+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.727541+0000) 2019-10-13T04:15:24.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:24.726+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.727659+0000) 2019-10-13T04:15:24.730 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:24.727+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.728500+0000) 2019-10-13T04:15:24.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:24.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51686 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:24.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:24.724+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.725018+0000) 2019-10-13T04:15:24.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:24.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.979231+0000) 2019-10-13T04:15:24.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:24.978+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.979227+0000) 2019-10-13T04:15:24.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:24.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:24.979227+0000) 2019-10-13T04:15:25.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:25.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:25.854783+0000) 2019-10-13T04:15:27.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:27.345+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:27.346040+0000) 2019-10-13T04:15:28.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:28.136+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:28.137193+0000) 2019-10-13T04:15:29.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:29.238+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.238768+0000) 2019-10-13T04:15:29.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:29.729+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.731245+0000) 2019-10-13T04:15:29.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:29.729+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.731269+0000) 2019-10-13T04:15:29.735 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:29.730+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.731470+0000) 2019-10-13T04:15:29.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:29.730+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.731793+0000) 2019-10-13T04:15:29.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:29.730+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.732222+0000) 2019-10-13T04:15:29.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:29.730+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.732359+0000) 2019-10-13T04:15:29.736 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:29.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:29.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:29.724+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.725635+0000) 2019-10-13T04:15:29.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:29.726+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:29.727549+0000) 2019-10-13T04:15:32.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:32.603+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:32.604215+0000) 2019-10-13T04:15:34.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:34.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.734693+0000) 2019-10-13T04:15:34.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:34.729+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.729869+0000) 2019-10-13T04:15:34.737 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:34.734+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.735445+0000) 2019-10-13T04:15:34.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:34.734+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.735607+0000) 2019-10-13T04:15:34.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:34.734+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.735782+0000) 2019-10-13T04:15:34.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:34.734+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.735962+0000) 2019-10-13T04:15:34.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:34.734+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.736138+0000) 2019-10-13T04:15:34.738 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:34.735+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:34.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:34.732+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.733051+0000) 2019-10-13T04:15:34.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:34.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.979365+0000) 2019-10-13T04:15:34.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:34.978+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.979365+0000) 2019-10-13T04:15:34.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:34.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:34.979363+0000) 2019-10-13T04:15:35.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:35.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:35.854955+0000) 2019-10-13T04:15:37.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:37.344+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:37.346214+0000) 2019-10-13T04:15:38.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:38.136+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:38.137359+0000) 2019-10-13T04:15:39.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:39.240+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.240979+0000) 2019-10-13T04:15:39.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:39.737+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.738576+0000) 2019-10-13T04:15:39.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:39.737+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.738801+0000) 2019-10-13T04:15:39.741 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:39.737+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.739047+0000) 2019-10-13T04:15:39.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:39.737+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.739322+0000) 2019-10-13T04:15:39.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:39.738+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.739583+0000) 2019-10-13T04:15:39.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:39.738+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.739788+0000) 2019-10-13T04:15:39.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:39.733+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.734057+0000) 2019-10-13T04:15:39.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:39.735+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:39.735889+0000) 2019-10-13T04:15:39.744 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:39.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:42.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:42.604+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:42.604402+0000) 2019-10-13T04:15:44.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:44.736+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.737274+0000) 2019-10-13T04:15:44.746 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:44.741+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.742817+0000) 2019-10-13T04:15:44.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:44.741+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.742968+0000) 2019-10-13T04:15:44.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:44.741+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.743090+0000) 2019-10-13T04:15:44.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:44.741+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.743238+0000) 2019-10-13T04:15:44.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:44.741+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.743388+0000) 2019-10-13T04:15:44.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:44.742+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.743516+0000) 2019-10-13T04:15:44.747 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:44.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:44.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:44.738+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.739265+0000) 2019-10-13T04:15:44.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:44.979+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.979524+0000) 2019-10-13T04:15:44.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:44.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.979525+0000) 2019-10-13T04:15:44.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:44.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:44.979523+0000) 2019-10-13T04:15:45.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:45.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:45.855106+0000) 2019-10-13T04:15:47.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:47.346+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:47.346423+0000) 2019-10-13T04:15:48.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:48.137+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:48.137543+0000) 2019-10-13T04:15:48.444 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:15:48.444 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:15:48.456 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:15:48.456 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:15:49.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:49.243+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.243281+0000) 2019-10-13T04:15:49.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:49.744+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.745098+0000) 2019-10-13T04:15:49.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:49.744+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.745322+0000) 2019-10-13T04:15:49.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:49.745+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.745430+0000) 2019-10-13T04:15:49.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:49.739+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.740377+0000) 2019-10-13T04:15:49.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:49.745+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.745855+0000) 2019-10-13T04:15:49.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:49.746+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.746501+0000) 2019-10-13T04:15:49.748 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:49.746+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.746666+0000) 2019-10-13T04:15:49.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:49.746+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51838 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:49.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:49.741+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:49.742370+0000) 2019-10-13T04:15:52.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:52.603+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:52.604642+0000) 2019-10-13T04:15:54.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:54.748+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.749109+0000) 2019-10-13T04:15:54.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:54.748+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.749227+0000) 2019-10-13T04:15:54.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:54.748+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.749388+0000) 2019-10-13T04:15:54.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:54.743+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.744167+0000) 2019-10-13T04:15:54.752 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:54.749+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.749731+0000) 2019-10-13T04:15:54.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:54.749+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.749887+0000) 2019-10-13T04:15:54.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:54.749+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.750311+0000) 2019-10-13T04:15:54.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:54.750+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51865 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:15:54.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:54.745+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.745770+0000) 2019-10-13T04:15:54.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:54.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.979669+0000) 2019-10-13T04:15:54.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:54.979+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.979668+0000) 2019-10-13T04:15:54.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:54.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:54.979669+0000) 2019-10-13T04:15:55.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:55.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:55.855276+0000) 2019-10-13T04:15:57.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:57.346+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:57.346668+0000) 2019-10-13T04:15:58.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:58.137+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:58.137695+0000) 2019-10-13T04:15:59.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:59.245+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.245589+0000) 2019-10-13T04:15:59.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:15:59.752+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.753004+0000) 2019-10-13T04:15:59.754 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:15:59.752+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.753304+0000) 2019-10-13T04:15:59.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:15:59.753+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.753489+0000) 2019-10-13T04:15:59.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:15:59.747+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.748399+0000) 2019-10-13T04:15:59.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:15:59.753+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.753666+0000) 2019-10-13T04:15:59.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:15:59.753+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.754166+0000) 2019-10-13T04:15:59.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:15:59.753+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.754255+0000) 2019-10-13T04:15:59.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:15:59.750+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:15:59.750944+0000) 2019-10-13T04:15:59.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:15:59.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:02.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:02.603+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:02.604855+0000) 2019-10-13T04:16:04.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:04.748+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.750199+0000) 2019-10-13T04:16:04.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:04.755+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.755439+0000) 2019-10-13T04:16:04.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:04.755+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.756090+0000) 2019-10-13T04:16:04.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:04.755+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.756244+0000) 2019-10-13T04:16:04.758 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:04.755+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.756362+0000) 2019-10-13T04:16:04.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:04.756+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.756455+0000) 2019-10-13T04:16:04.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:04.756+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.756672+0000) 2019-10-13T04:16:04.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:04.750+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.752487+0000) 2019-10-13T04:16:04.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:04.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:04.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:04.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.979847+0000) 2019-10-13T04:16:04.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:04.979+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.979845+0000) 2019-10-13T04:16:04.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:04.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:04.979844+0000) 2019-10-13T04:16:05.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:05.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:05.855427+0000) 2019-10-13T04:16:07.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:07.345+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:07.346896+0000) 2019-10-13T04:16:08.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:08.136+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:08.137904+0000) 2019-10-13T04:16:09.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:09.247+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.248045+0000) 2019-10-13T04:16:09.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:09.751+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.753554+0000) 2019-10-13T04:16:09.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:09.757+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.759122+0000) 2019-10-13T04:16:09.761 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:09.758+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.759655+0000) 2019-10-13T04:16:09.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:09.758+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.759691+0000) 2019-10-13T04:16:09.762 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:09.758+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.759959+0000) 2019-10-13T04:16:09.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:09.758+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.760004+0000) 2019-10-13T04:16:09.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:09.758+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.760162+0000) 2019-10-13T04:16:09.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:09.760+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:09.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:09.755+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:09.757317+0000) 2019-10-13T04:16:12.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:12.604+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:12.605042+0000) 2019-10-13T04:16:14.764 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:14.761+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.763192+0000) 2019-10-13T04:16:14.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:14.762+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.763477+0000) 2019-10-13T04:16:14.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:14.757+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.757878+0000) 2019-10-13T04:16:14.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:14.762+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.763714+0000) 2019-10-13T04:16:14.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:14.762+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.764087+0000) 2019-10-13T04:16:14.766 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:14.763+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.764411+0000) 2019-10-13T04:16:14.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:14.763+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.764879+0000) 2019-10-13T04:16:14.767 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:14.763+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 51985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:14.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:14.760+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.761459+0000) 2019-10-13T04:16:14.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:14.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.979947+0000) 2019-10-13T04:16:14.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:14.979+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.979975+0000) 2019-10-13T04:16:14.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:14.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:14.979989+0000) 2019-10-13T04:16:15.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:15.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:15.855614+0000) 2019-10-13T04:16:17.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:17.346+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:17.347094+0000) 2019-10-13T04:16:18.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:18.136+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:18.138061+0000) 2019-10-13T04:16:18.606 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:16:18.606 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:18.611 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:16:18.611 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:19.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:19.249+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.250429+0000) 2019-10-13T04:16:19.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:19.765+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.767138+0000) 2019-10-13T04:16:19.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:19.765+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.767321+0000) 2019-10-13T04:16:19.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:19.766+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.767443+0000) 2019-10-13T04:16:19.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:19.761+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.762249+0000) 2019-10-13T04:16:19.770 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:19.766+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.767925+0000) 2019-10-13T04:16:19.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:19.766+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.768080+0000) 2019-10-13T04:16:19.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:19.767+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.768434+0000) 2019-10-13T04:16:19.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:19.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:19.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:19.763+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:19.764714+0000) 2019-10-13T04:16:22.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:22.604+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:22.605224+0000) 2019-10-13T04:16:24.774 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:24.770+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.771621+0000) 2019-10-13T04:16:24.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:24.770+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.771912+0000) 2019-10-13T04:16:24.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:24.770+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.771808+0000) 2019-10-13T04:16:24.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:24.770+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.772127+0000) 2019-10-13T04:16:24.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:24.771+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.772405+0000) 2019-10-13T04:16:24.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:24.771+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.772764+0000) 2019-10-13T04:16:24.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:24.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:24.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:24.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.767241+0000) 2019-10-13T04:16:24.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:24.768+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.769503+0000) 2019-10-13T04:16:24.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:24.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.980136+0000) 2019-10-13T04:16:24.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:24.979+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.980139+0000) 2019-10-13T04:16:24.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:24.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:24.980145+0000) 2019-10-13T04:16:25.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:25.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:25.855799+0000) 2019-10-13T04:16:27.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:27.346+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:27.347300+0000) 2019-10-13T04:16:28.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:28.137+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:28.138230+0000) 2019-10-13T04:16:29.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:29.252+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.252866+0000) 2019-10-13T04:16:29.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:29.773+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.774847+0000) 2019-10-13T04:16:29.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:29.773+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.775135+0000) 2019-10-13T04:16:29.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:29.773+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.775339+0000) 2019-10-13T04:16:29.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:29.774+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.775439+0000) 2019-10-13T04:16:29.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:29.774+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.775636+0000) 2019-10-13T04:16:29.778 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:29.774+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.775909+0000) 2019-10-13T04:16:29.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:29.774+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:29.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:29.770+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.770837+0000) 2019-10-13T04:16:29.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:29.770+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:29.771465+0000) 2019-10-13T04:16:32.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:32.604+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:32.605434+0000) 2019-10-13T04:16:34.779 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:34.777+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.777663+0000) 2019-10-13T04:16:34.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:34.777+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.777990+0000) 2019-10-13T04:16:34.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:34.772+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.773524+0000) 2019-10-13T04:16:34.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:34.778+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.778452+0000) 2019-10-13T04:16:34.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:34.777+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.778324+0000) 2019-10-13T04:16:34.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:34.778+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.778600+0000) 2019-10-13T04:16:34.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:34.778+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.779051+0000) 2019-10-13T04:16:34.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:34.774+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.775166+0000) 2019-10-13T04:16:34.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:34.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52105 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:34.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:34.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.980312+0000) 2019-10-13T04:16:34.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:34.979+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.980297+0000) 2019-10-13T04:16:34.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:34.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:34.980310+0000) 2019-10-13T04:16:35.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:35.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:35.855987+0000) 2019-10-13T04:16:37.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:37.347+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:37.347488+0000) 2019-10-13T04:16:38.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:38.138+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:38.138458+0000) 2019-10-13T04:16:39.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:39.255+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.255426+0000) 2019-10-13T04:16:39.783 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:39.781+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.781905+0000) 2019-10-13T04:16:39.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:39.781+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.782044+0000) 2019-10-13T04:16:39.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:39.781+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.782141+0000) 2019-10-13T04:16:39.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:39.777+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.777746+0000) 2019-10-13T04:16:39.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:39.782+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.782597+0000) 2019-10-13T04:16:39.785 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:39.782+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.782714+0000) 2019-10-13T04:16:39.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:39.782+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.782847+0000) 2019-10-13T04:16:39.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:39.783+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:39.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:39.777+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:39.778305+0000) 2019-10-13T04:16:42.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:42.605+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:42.605612+0000) 2019-10-13T04:16:44.787 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:44.785+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.785763+0000) 2019-10-13T04:16:44.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:44.785+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.785906+0000) 2019-10-13T04:16:44.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:44.785+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.786047+0000) 2019-10-13T04:16:44.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:44.780+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.781188+0000) 2019-10-13T04:16:44.789 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:44.785+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.786383+0000) 2019-10-13T04:16:44.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:44.786+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.786635+0000) 2019-10-13T04:16:44.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:44.786+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.786977+0000) 2019-10-13T04:16:44.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:44.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:44.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:44.783+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.784269+0000) 2019-10-13T04:16:44.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:44.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.980497+0000) 2019-10-13T04:16:44.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:44.980+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.980480+0000) 2019-10-13T04:16:44.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:44.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:44.980498+0000) 2019-10-13T04:16:45.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:45.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:45.856191+0000) 2019-10-13T04:16:47.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:47.347+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:47.347707+0000) 2019-10-13T04:16:48.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:48.138+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:48.138638+0000) 2019-10-13T04:16:48.752 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:16:48.753 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:48.757 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:16:48.757 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:49.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:49.257+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.257788+0000) 2019-10-13T04:16:49.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:49.789+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.789671+0000) 2019-10-13T04:16:49.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:49.789+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.789746+0000) 2019-10-13T04:16:49.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:49.789+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.789870+0000) 2019-10-13T04:16:49.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:49.783+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.784669+0000) 2019-10-13T04:16:49.793 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:49.789+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.790165+0000) 2019-10-13T04:16:49.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:49.789+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.790221+0000) 2019-10-13T04:16:49.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:49.789+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.790374+0000) 2019-10-13T04:16:49.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:49.790+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52198 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:49.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:49.785+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:49.786187+0000) 2019-10-13T04:16:52.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:52.605+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:52.605791+0000) 2019-10-13T04:16:54.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:54.785+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.787579+0000) 2019-10-13T04:16:54.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:54.791+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.793072+0000) 2019-10-13T04:16:54.795 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:54.792+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.793411+0000) 2019-10-13T04:16:54.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:54.792+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.793576+0000) 2019-10-13T04:16:54.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:54.792+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.793687+0000) 2019-10-13T04:16:54.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:54.792+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.794057+0000) 2019-10-13T04:16:54.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:54.793+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.794565+0000) 2019-10-13T04:16:54.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:54.793+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:16:54.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:54.789+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.791288+0000) 2019-10-13T04:16:54.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:54.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.980689+0000) 2019-10-13T04:16:54.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:54.979+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.980665+0000) 2019-10-13T04:16:54.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:54.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:54.980687+0000) 2019-10-13T04:16:55.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:55.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:55.856349+0000) 2019-10-13T04:16:57.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:57.347+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:57.347921+0000) 2019-10-13T04:16:58.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:58.137+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:58.138834+0000) 2019-10-13T04:16:59.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:59.259+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.260329+0000) 2019-10-13T04:16:59.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:16:59.789+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.791308+0000) 2019-10-13T04:16:59.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:16:59.795+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.797251+0000) 2019-10-13T04:16:59.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:16:59.795+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.797356+0000) 2019-10-13T04:16:59.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:16:59.796+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.797632+0000) 2019-10-13T04:16:59.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:16:59.796+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.797839+0000) 2019-10-13T04:16:59.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:16:59.796+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.797966+0000) 2019-10-13T04:16:59.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:16:59.796+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.797686+0000) 2019-10-13T04:16:59.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:16:59.791+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:16:59.793659+0000) 2019-10-13T04:16:59.801 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:16:59.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:02.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:02.605+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:02.606009+0000) 2019-10-13T04:17:04.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:04.793+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.795026+0000) 2019-10-13T04:17:04.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:04.798+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.800215+0000) 2019-10-13T04:17:04.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:04.799+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.800465+0000) 2019-10-13T04:17:04.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:04.799+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.800655+0000) 2019-10-13T04:17:04.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:04.799+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.800906+0000) 2019-10-13T04:17:04.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:04.799+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.801114+0000) 2019-10-13T04:17:04.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:04.799+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.801298+0000) 2019-10-13T04:17:04.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:04.795+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.797472+0000) 2019-10-13T04:17:04.804 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:04.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:04.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:04.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.980870+0000) 2019-10-13T04:17:04.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:04.980+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.980866+0000) 2019-10-13T04:17:04.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:04.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:04.980904+0000) 2019-10-13T04:17:05.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:05.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:05.856513+0000) 2019-10-13T04:17:07.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:07.347+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:07.348113+0000) 2019-10-13T04:17:08.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:08.138+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:08.139021+0000) 2019-10-13T04:17:09.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:09.261+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.262516+0000) 2019-10-13T04:17:09.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:09.797+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.798065+0000) 2019-10-13T04:17:09.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:09.802+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.803543+0000) 2019-10-13T04:17:09.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:09.802+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.803703+0000) 2019-10-13T04:17:09.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:09.802+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.803910+0000) 2019-10-13T04:17:09.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:09.802+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.804109+0000) 2019-10-13T04:17:09.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:09.802+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.804056+0000) 2019-10-13T04:17:09.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:09.803+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.804473+0000) 2019-10-13T04:17:09.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:09.799+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:09.800089+0000) 2019-10-13T04:17:09.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:09.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52318 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:12.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:12.605+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:12.606191+0000) 2019-10-13T04:17:14.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:14.801+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.801875+0000) 2019-10-13T04:17:14.809 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:14.805+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.807264+0000) 2019-10-13T04:17:14.809 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:14.806+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.807585+0000) 2019-10-13T04:17:14.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:14.806+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.807790+0000) 2019-10-13T04:17:14.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:14.806+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.808038+0000) 2019-10-13T04:17:14.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:14.807+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.808411+0000) 2019-10-13T04:17:14.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:14.803+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.804237+0000) 2019-10-13T04:17:14.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:14.807+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.808683+0000) 2019-10-13T04:17:14.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:14.808+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:14.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:14.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.981019+0000) 2019-10-13T04:17:14.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:14.980+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.981017+0000) 2019-10-13T04:17:14.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:14.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:14.981045+0000) 2019-10-13T04:17:15.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:15.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:15.856721+0000) 2019-10-13T04:17:17.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:17.347+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:17.348306+0000) 2019-10-13T04:17:18.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:18.138+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:18.139230+0000) 2019-10-13T04:17:18.914 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:17:18.915 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:18.919 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:17:18.919 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:19.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:19.264+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.264859+0000) 2019-10-13T04:17:19.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:19.810+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.811287+0000) 2019-10-13T04:17:19.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:19.811+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.811486+0000) 2019-10-13T04:17:19.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:19.811+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.812019+0000) 2019-10-13T04:17:19.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:19.811+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.812128+0000) 2019-10-13T04:17:19.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:19.811+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.812354+0000) 2019-10-13T04:17:19.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:19.805+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.806441+0000) 2019-10-13T04:17:19.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:19.808+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.809297+0000) 2019-10-13T04:17:19.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:19.812+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:19.812635+0000) 2019-10-13T04:17:19.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:19.813+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52377 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:22.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:22.605+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:22.606350+0000) 2019-10-13T04:17:24.817 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:24.814+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.815292+0000) 2019-10-13T04:17:24.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:24.815+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.815661+0000) 2019-10-13T04:17:24.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:24.815+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.815854+0000) 2019-10-13T04:17:24.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:24.809+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.810167+0000) 2019-10-13T04:17:24.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:24.815+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.815999+0000) 2019-10-13T04:17:24.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:24.815+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.816204+0000) 2019-10-13T04:17:24.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:24.816+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.816463+0000) 2019-10-13T04:17:24.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:24.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:24.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:24.811+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.812481+0000) 2019-10-13T04:17:24.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:24.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.981206+0000) 2019-10-13T04:17:24.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:24.980+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.981206+0000) 2019-10-13T04:17:24.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:24.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:24.981210+0000) 2019-10-13T04:17:25.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:25.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:25.856914+0000) 2019-10-13T04:17:27.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:27.348+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:27.348497+0000) 2019-10-13T04:17:28.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:28.139+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:28.139439+0000) 2019-10-13T04:17:29.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:29.267+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.267443+0000) 2019-10-13T04:17:29.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:29.818+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.819314+0000) 2019-10-13T04:17:29.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:29.819+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.819615+0000) 2019-10-13T04:17:29.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:29.814+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.814814+0000) 2019-10-13T04:17:29.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:29.819+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.819826+0000) 2019-10-13T04:17:29.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:29.819+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.820204+0000) 2019-10-13T04:17:29.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:29.820+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.820661+0000) 2019-10-13T04:17:29.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:29.820+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.820829+0000) 2019-10-13T04:17:29.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:29.820+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:29.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:29.816+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:29.816964+0000) 2019-10-13T04:17:32.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:32.605+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:32.606517+0000) 2019-10-13T04:17:34.824 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:34.822+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.823012+0000) 2019-10-13T04:17:34.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:34.817+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.817962+0000) 2019-10-13T04:17:34.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:34.823+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.823705+0000) 2019-10-13T04:17:34.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:34.823+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.823878+0000) 2019-10-13T04:17:34.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:34.823+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.824075+0000) 2019-10-13T04:17:34.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:34.823+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.824238+0000) 2019-10-13T04:17:34.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:34.818+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.819665+0000) 2019-10-13T04:17:34.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:34.824+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.824661+0000) 2019-10-13T04:17:34.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:34.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:34.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:34.981+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.981390+0000) 2019-10-13T04:17:34.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:34.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.981390+0000) 2019-10-13T04:17:34.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:34.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:34.981390+0000) 2019-10-13T04:17:35.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:35.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:35.857105+0000) 2019-10-13T04:17:37.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:37.347+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:37.348689+0000) 2019-10-13T04:17:38.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:38.138+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:38.139621+0000) 2019-10-13T04:17:39.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:39.269+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.269686+0000) 2019-10-13T04:17:39.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:39.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.826682+0000) 2019-10-13T04:17:39.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:39.825+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.826811+0000) 2019-10-13T04:17:39.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:39.825+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.827274+0000) 2019-10-13T04:17:39.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:39.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.822620+0000) 2019-10-13T04:17:39.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:39.822+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.823341+0000) 2019-10-13T04:17:39.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:39.826+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.827701+0000) 2019-10-13T04:17:39.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:39.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.828053+0000) 2019-10-13T04:17:39.832 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:39.826+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:39.828281+0000) 2019-10-13T04:17:39.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:39.827+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:42.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:42.606+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:42.606721+0000) 2019-10-13T04:17:44.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:44.829+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.830959+0000) 2019-10-13T04:17:44.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:44.830+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.831585+0000) 2019-10-13T04:17:44.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:44.830+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:44.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:44.830+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.831649+0000) 2019-10-13T04:17:44.835 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:44.830+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.831955+0000) 2019-10-13T04:17:44.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:44.830+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.832123+0000) 2019-10-13T04:17:44.835 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:44.830+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.832305+0000) 2019-10-13T04:17:44.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:44.826+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.827251+0000) 2019-10-13T04:17:44.836 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:44.827+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.828033+0000) 2019-10-13T04:17:44.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:44.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.981581+0000) 2019-10-13T04:17:44.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:44.980+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.981581+0000) 2019-10-13T04:17:44.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:44.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:44.981583+0000) 2019-10-13T04:17:45.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:45.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:45.857248+0000) 2019-10-13T04:17:47.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:47.348+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:47.348878+0000) 2019-10-13T04:17:48.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:48.139+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:48.139812+0000) 2019-10-13T04:17:49.067 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:17:49.067 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:49.072 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:17:49.072 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:49.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:49.271+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.272049+0000) 2019-10-13T04:17:49.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:49.831+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.832998+0000) 2019-10-13T04:17:49.835 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:49.831+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.833028+0000) 2019-10-13T04:17:49.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:49.831+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.833201+0000) 2019-10-13T04:17:49.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:49.831+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.833347+0000) 2019-10-13T04:17:49.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:49.832+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.833537+0000) 2019-10-13T04:17:49.836 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:49.832+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.833777+0000) 2019-10-13T04:17:49.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:49.827+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.828872+0000) 2019-10-13T04:17:49.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:49.833+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:49.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:49.828+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:49.830647+0000) 2019-10-13T04:17:52.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:52.606+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:52.606924+0000) 2019-10-13T04:17:54.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:54.834+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.836116+0000) 2019-10-13T04:17:54.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:54.834+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.836223+0000) 2019-10-13T04:17:54.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:54.835+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.836523+0000) 2019-10-13T04:17:54.839 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:54.835+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.836655+0000) 2019-10-13T04:17:54.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:54.830+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.831980+0000) 2019-10-13T04:17:54.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:54.835+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.837323+0000) 2019-10-13T04:17:54.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:54.836+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.837687+0000) 2019-10-13T04:17:54.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:54.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52586 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:17:54.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:54.832+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.833760+0000) 2019-10-13T04:17:54.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:54.981+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.981758+0000) 2019-10-13T04:17:54.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:54.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.981759+0000) 2019-10-13T04:17:54.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:54.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:54.981758+0000) 2019-10-13T04:17:55.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:55.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:55.857426+0000) 2019-10-13T04:17:57.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:57.348+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:57.349033+0000) 2019-10-13T04:17:58.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:58.139+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:58.140007+0000) 2019-10-13T04:17:59.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:59.273+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.274206+0000) 2019-10-13T04:17:59.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:17:59.832+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.834646+0000) 2019-10-13T04:17:59.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:17:59.838+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.840224+0000) 2019-10-13T04:17:59.842 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:17:59.839+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.840582+0000) 2019-10-13T04:17:59.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:17:59.839+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.840661+0000) 2019-10-13T04:17:59.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:17:59.839+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.840847+0000) 2019-10-13T04:17:59.843 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:17:59.839+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.840971+0000) 2019-10-13T04:17:59.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:17:59.839+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.841159+0000) 2019-10-13T04:17:59.843 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:17:59.835+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:17:59.837286+0000) 2019-10-13T04:17:59.844 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:17:59.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:02.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:02.606+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:02.607111+0000) 2019-10-13T04:18:04.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:04.837+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.838426+0000) 2019-10-13T04:18:04.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:04.843+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.843838+0000) 2019-10-13T04:18:04.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:04.843+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.844110+0000) 2019-10-13T04:18:04.847 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:04.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.844543+0000) 2019-10-13T04:18:04.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:04.844+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.844552+0000) 2019-10-13T04:18:04.847 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:04.844+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.844950+0000) 2019-10-13T04:18:04.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:04.844+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.845027+0000) 2019-10-13T04:18:04.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:04.840+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.840827+0000) 2019-10-13T04:18:04.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:04.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:04.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:04.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.981922+0000) 2019-10-13T04:18:04.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:04.981+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.981920+0000) 2019-10-13T04:18:04.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:04.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:04.981929+0000) 2019-10-13T04:18:05.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:05.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:05.857615+0000) 2019-10-13T04:18:07.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:07.348+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:07.349200+0000) 2019-10-13T04:18:08.142 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:08.139+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:08.140141+0000) 2019-10-13T04:18:09.283 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:09.275+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.276509+0000) 2019-10-13T04:18:09.849 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:09.847+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.847733+0000) 2019-10-13T04:18:09.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:09.842+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.843331+0000) 2019-10-13T04:18:09.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:09.847+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.848276+0000) 2019-10-13T04:18:09.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:09.848+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.848434+0000) 2019-10-13T04:18:09.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:09.848+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.848729+0000) 2019-10-13T04:18:09.851 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:09.848+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.849310+0000) 2019-10-13T04:18:09.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:09.849+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.849701+0000) 2019-10-13T04:18:09.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:09.849+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:09.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:09.845+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:09.845995+0000) 2019-10-13T04:18:12.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:12.606+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:12.607267+0000) 2019-10-13T04:18:14.853 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:14.851+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.851454+0000) 2019-10-13T04:18:14.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:14.851+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.851742+0000) 2019-10-13T04:18:14.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:14.851+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.851882+0000) 2019-10-13T04:18:14.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:14.851+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.851975+0000) 2019-10-13T04:18:14.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:14.851+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.852305+0000) 2019-10-13T04:18:14.854 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:14.852+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.852410+0000) 2019-10-13T04:18:14.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:14.845+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.846527+0000) 2019-10-13T04:18:14.855 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:14.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:14.855 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:14.848+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.849087+0000) 2019-10-13T04:18:14.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:14.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.982068+0000) 2019-10-13T04:18:14.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:14.981+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.982068+0000) 2019-10-13T04:18:14.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:14.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:14.982075+0000) 2019-10-13T04:18:15.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:15.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:15.857796+0000) 2019-10-13T04:18:17.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:17.349+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:17.349354+0000) 2019-10-13T04:18:18.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:18.140+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:18.140356+0000) 2019-10-13T04:18:19.223 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:18:19.223 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:19.227 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:18:19.227 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:19.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:19.278+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.278876+0000) 2019-10-13T04:18:19.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:19.854+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.855086+0000) 2019-10-13T04:18:19.857 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:19.854+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.855215+0000) 2019-10-13T04:18:19.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:19.855+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.855621+0000) 2019-10-13T04:18:19.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:19.849+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.850146+0000) 2019-10-13T04:18:19.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:19.855+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.855888+0000) 2019-10-13T04:18:19.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:19.855+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.856029+0000) 2019-10-13T04:18:19.859 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:19.855+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.856188+0000) 2019-10-13T04:18:19.859 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:19.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:19.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:19.852+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:19.853357+0000) 2019-10-13T04:18:22.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:22.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:22.607435+0000) 2019-10-13T04:18:24.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:24.856+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.858389+0000) 2019-10-13T04:18:24.860 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:24.857+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.858573+0000) 2019-10-13T04:18:24.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:24.852+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.853515+0000) 2019-10-13T04:18:24.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:24.857+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.859043+0000) 2019-10-13T04:18:24.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:24.857+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.859158+0000) 2019-10-13T04:18:24.861 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:24.857+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.859292+0000) 2019-10-13T04:18:24.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:24.858+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.859511+0000) 2019-10-13T04:18:24.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:24.858+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52767 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:24.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:24.855+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.855798+0000) 2019-10-13T04:18:24.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:24.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.982205+0000) 2019-10-13T04:18:24.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:24.981+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.982205+0000) 2019-10-13T04:18:24.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:24.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:24.982214+0000) 2019-10-13T04:18:25.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:25.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:25.857963+0000) 2019-10-13T04:18:27.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:27.348+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:27.349588+0000) 2019-10-13T04:18:28.142 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:28.139+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:28.140564+0000) 2019-10-13T04:18:29.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:29.280+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.281068+0000) 2019-10-13T04:18:29.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:29.857+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.858001+0000) 2019-10-13T04:18:29.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:29.862+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.863390+0000) 2019-10-13T04:18:29.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:29.862+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.863526+0000) 2019-10-13T04:18:29.866 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:29.862+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.863690+0000) 2019-10-13T04:18:29.866 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:29.862+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:29.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:29.862+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.864154+0000) 2019-10-13T04:18:29.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:29.862+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.864206+0000) 2019-10-13T04:18:29.867 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:29.863+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.864665+0000) 2019-10-13T04:18:29.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:29.860+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:29.861231+0000) 2019-10-13T04:18:32.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:32.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:32.607606+0000) 2019-10-13T04:18:34.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:34.863+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.865160+0000) 2019-10-13T04:18:34.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:34.863+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.865344+0000) 2019-10-13T04:18:34.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:34.859+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.860375+0000) 2019-10-13T04:18:34.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:34.864+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.865757+0000) 2019-10-13T04:18:34.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:34.864+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.866064+0000) 2019-10-13T04:18:34.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:34.864+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.866228+0000) 2019-10-13T04:18:34.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:34.864+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.866294+0000) 2019-10-13T04:18:34.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:34.865+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:34.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:34.861+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.862523+0000) 2019-10-13T04:18:34.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:34.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.982362+0000) 2019-10-13T04:18:34.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:34.981+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.982363+0000) 2019-10-13T04:18:34.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:34.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:34.982366+0000) 2019-10-13T04:18:35.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:35.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:35.858125+0000) 2019-10-13T04:18:37.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:37.349+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:37.349803+0000) 2019-10-13T04:18:38.142 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:38.140+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:38.140754+0000) 2019-10-13T04:18:39.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:39.283+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.283589+0000) 2019-10-13T04:18:39.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:39.861+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.863518+0000) 2019-10-13T04:18:39.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:39.867+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.869054+0000) 2019-10-13T04:18:39.871 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:39.867+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.869218+0000) 2019-10-13T04:18:39.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:39.868+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.869474+0000) 2019-10-13T04:18:39.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:39.868+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.869749+0000) 2019-10-13T04:18:39.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:39.868+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.869920+0000) 2019-10-13T04:18:39.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:39.868+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.869956+0000) 2019-10-13T04:18:39.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:39.869+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:39.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:39.865+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:39.866919+0000) 2019-10-13T04:18:42.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:42.606+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:42.607765+0000) 2019-10-13T04:18:44.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:44.865+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.867552+0000) 2019-10-13T04:18:44.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:44.871+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.873123+0000) 2019-10-13T04:18:44.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:44.871+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.873370+0000) 2019-10-13T04:18:44.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:44.871+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.873194+0000) 2019-10-13T04:18:44.875 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:44.872+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.873449+0000) 2019-10-13T04:18:44.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:44.872+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.873721+0000) 2019-10-13T04:18:44.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:44.872+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.873863+0000) 2019-10-13T04:18:44.876 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:44.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:44.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:44.869+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.871494+0000) 2019-10-13T04:18:44.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:44.982+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.982588+0000) 2019-10-13T04:18:44.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:44.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.982592+0000) 2019-10-13T04:18:44.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:44.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:44.982589+0000) 2019-10-13T04:18:45.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:45.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:45.858310+0000) 2019-10-13T04:18:47.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:47.349+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:47.349998+0000) 2019-10-13T04:18:48.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:48.140+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:48.140948+0000) 2019-10-13T04:18:49.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:49.285+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.286090+0000) 2019-10-13T04:18:49.360 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:18:49.360 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:49.365 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:18:49.365 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:49.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:49.869+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.871189+0000) 2019-10-13T04:18:49.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:49.876+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.876563+0000) 2019-10-13T04:18:49.879 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:49.876+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.876885+0000) 2019-10-13T04:18:49.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:49.876+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.877203+0000) 2019-10-13T04:18:49.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:49.876+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.877080+0000) 2019-10-13T04:18:49.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:49.877+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.877513+0000) 2019-10-13T04:18:49.880 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:49.877+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.877701+0000) 2019-10-13T04:18:49.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:49.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:49.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:49.872+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:49.874279+0000) 2019-10-13T04:18:52.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:52.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:52.607923+0000) 2019-10-13T04:18:54.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:54.879+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.880062+0000) 2019-10-13T04:18:54.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:54.874+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.875217+0000) 2019-10-13T04:18:54.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:54.880+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.880775+0000) 2019-10-13T04:18:54.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:54.880+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.880928+0000) 2019-10-13T04:18:54.883 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:54.880+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.881150+0000) 2019-10-13T04:18:54.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:54.881+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.881420+0000) 2019-10-13T04:18:54.883 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:54.881+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.881587+0000) 2019-10-13T04:18:54.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:54.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:54.884 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:54.877+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.878057+0000) 2019-10-13T04:18:54.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:54.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.982781+0000) 2019-10-13T04:18:54.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:54.982+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.982781+0000) 2019-10-13T04:18:54.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:54.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:54.982775+0000) 2019-10-13T04:18:55.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:55.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:55.858490+0000) 2019-10-13T04:18:57.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:57.349+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:57.350169+0000) 2019-10-13T04:18:58.142 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:58.140+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:58.141141+0000) 2019-10-13T04:18:59.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:59.287+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.288531+0000) 2019-10-13T04:18:59.885 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:18:59.883+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.883819+0000) 2019-10-13T04:18:59.886 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:18:59.883+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.883937+0000) 2019-10-13T04:18:59.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:18:59.883+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.884241+0000) 2019-10-13T04:18:59.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:18:59.883+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.884331+0000) 2019-10-13T04:18:59.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:18:59.884+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.884521+0000) 2019-10-13T04:18:59.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:18:59.878+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.878919+0000) 2019-10-13T04:18:59.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:18:59.884+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.884938+0000) 2019-10-13T04:18:59.887 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:18:59.885+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 52977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:18:59.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:18:59.880+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:18:59.881301+0000) 2019-10-13T04:19:02.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:02.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:02.608107+0000) 2019-10-13T04:19:04.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:04.881+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.882034+0000) 2019-10-13T04:19:04.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:04.887+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.887439+0000) 2019-10-13T04:19:04.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:04.887+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.887624+0000) 2019-10-13T04:19:04.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:04.887+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.887850+0000) 2019-10-13T04:19:04.891 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:04.887+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.888344+0000) 2019-10-13T04:19:04.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:04.888+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.888480+0000) 2019-10-13T04:19:04.891 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:04.888+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.888845+0000) 2019-10-13T04:19:04.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:04.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53007 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:04.892 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:04.884+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.885041+0000) 2019-10-13T04:19:04.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:04.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.982915+0000) 2019-10-13T04:19:04.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:04.982+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.982915+0000) 2019-10-13T04:19:04.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:04.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:04.982918+0000) 2019-10-13T04:19:05.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:05.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:05.858611+0000) 2019-10-13T04:19:07.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:07.349+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:07.350352+0000) 2019-10-13T04:19:08.142 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:08.140+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:08.141330+0000) 2019-10-13T04:19:09.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:09.290+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.290918+0000) 2019-10-13T04:19:09.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:09.890+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.891663+0000) 2019-10-13T04:19:09.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:09.890+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.891839+0000) 2019-10-13T04:19:09.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:09.886+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.887207+0000) 2019-10-13T04:19:09.894 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:09.891+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.892698+0000) 2019-10-13T04:19:09.895 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:09.891+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.892984+0000) 2019-10-13T04:19:09.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:09.891+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.892875+0000) 2019-10-13T04:19:09.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:09.891+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.893143+0000) 2019-10-13T04:19:09.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:09.892+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:09.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:09.888+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:09.889504+0000) 2019-10-13T04:19:12.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:12.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:12.608220+0000) 2019-10-13T04:19:14.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:14.893+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.894946+0000) 2019-10-13T04:19:14.897 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:14.894+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.895462+0000) 2019-10-13T04:19:14.897 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:14.894+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.895798+0000) 2019-10-13T04:19:14.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:14.894+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.896113+0000) 2019-10-13T04:19:14.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:14.890+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.890838+0000) 2019-10-13T04:19:14.899 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:14.891+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.892169+0000) 2019-10-13T04:19:14.899 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:14.894+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.896356+0000) 2019-10-13T04:19:14.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:14.895+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.896490+0000) 2019-10-13T04:19:14.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:14.895+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:14.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:14.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.983075+0000) 2019-10-13T04:19:14.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:14.982+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.983076+0000) 2019-10-13T04:19:14.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:14.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:14.983078+0000) 2019-10-13T04:19:15.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:15.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:15.858775+0000) 2019-10-13T04:19:17.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:17.349+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:17.350529+0000) 2019-10-13T04:19:18.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:18.140+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:18.141526+0000) 2019-10-13T04:19:19.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:19.293+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.293420+0000) 2019-10-13T04:19:19.524 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:19:19.524 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:19.528 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:19:19.528 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:19.900 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:19.897+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.898970+0000) 2019-10-13T04:19:19.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:19.897+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.899158+0000) 2019-10-13T04:19:19.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:19.897+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.899272+0000) 2019-10-13T04:19:19.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:19.898+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.899561+0000) 2019-10-13T04:19:19.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:19.893+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.894295+0000) 2019-10-13T04:19:19.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:19.898+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.899835+0000) 2019-10-13T04:19:19.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:19.898+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.900349+0000) 2019-10-13T04:19:19.902 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:19.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:19.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:19.896+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:19.897372+0000) 2019-10-13T04:19:22.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:22.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:22.608408+0000) 2019-10-13T04:19:24.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:24.901+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.903074+0000) 2019-10-13T04:19:24.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:24.901+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.903080+0000) 2019-10-13T04:19:24.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:24.901+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.903300+0000) 2019-10-13T04:19:24.905 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:24.902+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.903393+0000) 2019-10-13T04:19:24.906 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:24.902+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.903600+0000) 2019-10-13T04:19:24.906 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:24.902+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.903911+0000) 2019-10-13T04:19:24.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:24.897+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.898613+0000) 2019-10-13T04:19:24.907 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:24.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:24.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:24.900+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.901062+0000) 2019-10-13T04:19:24.989 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:24.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.983256+0000) 2019-10-13T04:19:24.989 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:24.982+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.983256+0000) 2019-10-13T04:19:24.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:24.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:24.983256+0000) 2019-10-13T04:19:25.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:25.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:25.858911+0000) 2019-10-13T04:19:27.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:27.350+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:27.350695+0000) 2019-10-13T04:19:28.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:28.141+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:28.141701+0000) 2019-10-13T04:19:29.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:29.295+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.295760+0000) 2019-10-13T04:19:29.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:29.905+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.906877+0000) 2019-10-13T04:19:29.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:29.905+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.907132+0000) 2019-10-13T04:19:29.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:29.905+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.907239+0000) 2019-10-13T04:19:29.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:29.901+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.902668+0000) 2019-10-13T04:19:29.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:29.902+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.903661+0000) 2019-10-13T04:19:29.911 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:29.906+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.908034+0000) 2019-10-13T04:19:29.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:29.905+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.907382+0000) 2019-10-13T04:19:29.911 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:29.907+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:29.908512+0000) 2019-10-13T04:19:29.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:29.907+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:32.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:32.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:32.608593+0000) 2019-10-13T04:19:34.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:34.910+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.910517+0000) 2019-10-13T04:19:34.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:34.910+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.910630+0000) 2019-10-13T04:19:34.912 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:34.910+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.910780+0000) 2019-10-13T04:19:34.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:34.904+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.905812+0000) 2019-10-13T04:19:34.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:34.910+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.911177+0000) 2019-10-13T04:19:34.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:34.911+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.911625+0000) 2019-10-13T04:19:34.914 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:34.911+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.911892+0000) 2019-10-13T04:19:34.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:34.912+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:34.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:34.907+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.908920+0000) 2019-10-13T04:19:34.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:34.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.983465+0000) 2019-10-13T04:19:34.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:34.983+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.983464+0000) 2019-10-13T04:19:34.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:34.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:34.983464+0000) 2019-10-13T04:19:35.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:35.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:35.859073+0000) 2019-10-13T04:19:37.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:37.350+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:37.350874+0000) 2019-10-13T04:19:38.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:38.141+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:38.141926+0000) 2019-10-13T04:19:39.305 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:39.297+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.298279+0000) 2019-10-13T04:19:39.915 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:39.913+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.914076+0000) 2019-10-13T04:19:39.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:39.913+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.914287+0000) 2019-10-13T04:19:39.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:39.907+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.909245+0000) 2019-10-13T04:19:39.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:39.914+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.915031+0000) 2019-10-13T04:19:39.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:39.914+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.915341+0000) 2019-10-13T04:19:39.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:39.915+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.916019+0000) 2019-10-13T04:19:39.918 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:39.915+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.916333+0000) 2019-10-13T04:19:39.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:39.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:39.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:39.909+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:39.911453+0000) 2019-10-13T04:19:42.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:42.607+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:42.608798+0000) 2019-10-13T04:19:44.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:44.912+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.913997+0000) 2019-10-13T04:19:44.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:44.918+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.919300+0000) 2019-10-13T04:19:44.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:44.919+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.919555+0000) 2019-10-13T04:19:44.922 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:44.919+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.919894+0000) 2019-10-13T04:19:44.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:44.919+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.920119+0000) 2019-10-13T04:19:44.922 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:44.919+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.920261+0000) 2019-10-13T04:19:44.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:44.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:44.923 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:44.920+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.920426+0000) 2019-10-13T04:19:44.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:44.915+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.916759+0000) 2019-10-13T04:19:44.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:44.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.983640+0000) 2019-10-13T04:19:44.988 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:44.983+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.983643+0000) 2019-10-13T04:19:44.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:44.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:44.983638+0000) 2019-10-13T04:19:45.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:45.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:45.859238+0000) 2019-10-13T04:19:47.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:47.350+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:47.351054+0000) 2019-10-13T04:19:48.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:48.142+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:48.142117+0000) 2019-10-13T04:19:49.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:49.300+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.300819+0000) 2019-10-13T04:19:49.686 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:19:49.687 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:49.691 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:19:49.692 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:49.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:49.916+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.916791+0000) 2019-10-13T04:19:49.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:49.921+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.922143+0000) 2019-10-13T04:19:49.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:49.921+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.922287+0000) 2019-10-13T04:19:49.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:49.922+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.922552+0000) 2019-10-13T04:19:49.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:49.922+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.922767+0000) 2019-10-13T04:19:49.925 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:49.922+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.922999+0000) 2019-10-13T04:19:49.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:49.923+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.923454+0000) 2019-10-13T04:19:49.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:49.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:49.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:49.919+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:49.920443+0000) 2019-10-13T04:19:52.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:52.608+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:52.608982+0000) 2019-10-13T04:19:54.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:54.924+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.925457+0000) 2019-10-13T04:19:54.927 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:54.924+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.925669+0000) 2019-10-13T04:19:54.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:54.924+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.926098+0000) 2019-10-13T04:19:54.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:54.920+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.921132+0000) 2019-10-13T04:19:54.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:54.925+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.926408+0000) 2019-10-13T04:19:54.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:54.925+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.926489+0000) 2019-10-13T04:19:54.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:54.922+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.922792+0000) 2019-10-13T04:19:54.930 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:54.925+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.927067+0000) 2019-10-13T04:19:54.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:54.925+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:19:54.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:54.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.983826+0000) 2019-10-13T04:19:54.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:54.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.983829+0000) 2019-10-13T04:19:54.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:54.982+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:54.983825+0000) 2019-10-13T04:19:55.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:55.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:55.859363+0000) 2019-10-13T04:19:57.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:57.350+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:57.351239+0000) 2019-10-13T04:19:58.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:58.141+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:58.142312+0000) 2019-10-13T04:19:59.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:59.302+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.303064+0000) 2019-10-13T04:19:59.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:19:59.923+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.924081+0000) 2019-10-13T04:19:59.931 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:19:59.927+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.929322+0000) 2019-10-13T04:19:59.931 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:19:59.928+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.929595+0000) 2019-10-13T04:19:59.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:19:59.928+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.929620+0000) 2019-10-13T04:19:59.932 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:19:59.928+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.929831+0000) 2019-10-13T04:19:59.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:19:59.928+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.929901+0000) 2019-10-13T04:19:59.932 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:19:59.928+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.930208+0000) 2019-10-13T04:19:59.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:19:59.925+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:19:59.925927+0000) 2019-10-13T04:19:59.933 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:19:59.929+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:02.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:02.608+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:02.609150+0000) 2019-10-13T04:20:04.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:04.931+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.933023+0000) 2019-10-13T04:20:04.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:04.931+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.933199+0000) 2019-10-13T04:20:04.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:04.932+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.933416+0000) 2019-10-13T04:20:04.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:04.927+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.928693+0000) 2019-10-13T04:20:04.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:04.932+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.933933+0000) 2019-10-13T04:20:04.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:04.932+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.934149+0000) 2019-10-13T04:20:04.937 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:04.933+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.934563+0000) 2019-10-13T04:20:04.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:04.929+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.930020+0000) 2019-10-13T04:20:04.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:04.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:04.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:04.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.984005+0000) 2019-10-13T04:20:04.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:04.983+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.984011+0000) 2019-10-13T04:20:04.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:04.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:04.984006+0000) 2019-10-13T04:20:05.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:05.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:05.859509+0000) 2019-10-13T04:20:07.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:07.350+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:07.351431+0000) 2019-10-13T04:20:08.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:08.141+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:08.142509+0000) 2019-10-13T04:20:09.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:09.304+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.305594+0000) 2019-10-13T04:20:09.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:09.935+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.936758+0000) 2019-10-13T04:20:09.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:09.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.937034+0000) 2019-10-13T04:20:09.939 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:09.935+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.937066+0000) 2019-10-13T04:20:09.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:09.935+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.937241+0000) 2019-10-13T04:20:09.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:09.932+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.932818+0000) 2019-10-13T04:20:09.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:09.936+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.937663+0000) 2019-10-13T04:20:09.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:09.936+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.938112+0000) 2019-10-13T04:20:09.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:09.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:09.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:09.933+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:09.934697+0000) 2019-10-13T04:20:12.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:12.608+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:12.609311+0000) 2019-10-13T04:20:14.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:14.939+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.940721+0000) 2019-10-13T04:20:14.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:14.935+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.936313+0000) 2019-10-13T04:20:14.943 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:14.939+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.941180+0000) 2019-10-13T04:20:14.943 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:14.939+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.941052+0000) 2019-10-13T04:20:14.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:14.939+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.941272+0000) 2019-10-13T04:20:14.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:14.940+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.941776+0000) 2019-10-13T04:20:14.944 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:14.940+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.941936+0000) 2019-10-13T04:20:14.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:14.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:14.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:14.938+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.939024+0000) 2019-10-13T04:20:14.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:14.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.984206+0000) 2019-10-13T04:20:14.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:14.983+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.984204+0000) 2019-10-13T04:20:14.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:14.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:14.984205+0000) 2019-10-13T04:20:15.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:15.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:15.859664+0000) 2019-10-13T04:20:17.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:17.351+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:17.351627+0000) 2019-10-13T04:20:18.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:18.142+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:18.142715+0000) 2019-10-13T04:20:19.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:19.307+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.308018+0000) 2019-10-13T04:20:19.848 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:20:19.848 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:19.852 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:20:19.853 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:19.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:19.938+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.938972+0000) 2019-10-13T04:20:19.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:19.943+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.944369+0000) 2019-10-13T04:20:19.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:19.944+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.944479+0000) 2019-10-13T04:20:19.946 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:19.944+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.944876+0000) 2019-10-13T04:20:19.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:19.944+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.944967+0000) 2019-10-13T04:20:19.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:19.944+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.944827+0000) 2019-10-13T04:20:19.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:19.944+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.945241+0000) 2019-10-13T04:20:19.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:19.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:19.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:19.941+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:19.942498+0000) 2019-10-13T04:20:22.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:22.609+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:22.609499+0000) 2019-10-13T04:20:24.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:24.947+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.947561+0000) 2019-10-13T04:20:24.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:24.942+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.943008+0000) 2019-10-13T04:20:24.950 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:24.947+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.948331+0000) 2019-10-13T04:20:24.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:24.948+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.948474+0000) 2019-10-13T04:20:24.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:24.948+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.948859+0000) 2019-10-13T04:20:24.951 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:24.947+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.947905+0000) 2019-10-13T04:20:24.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:24.948+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.948749+0000) 2019-10-13T04:20:24.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:24.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:24.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:24.945+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.946207+0000) 2019-10-13T04:20:24.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:24.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.984388+0000) 2019-10-13T04:20:24.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:24.984+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.984389+0000) 2019-10-13T04:20:24.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:24.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:24.984390+0000) 2019-10-13T04:20:25.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:25.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:25.859805+0000) 2019-10-13T04:20:27.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:27.351+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:27.351829+0000) 2019-10-13T04:20:28.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:28.142+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:28.142947+0000) 2019-10-13T04:20:29.317 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:29.309+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.310526+0000) 2019-10-13T04:20:29.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:29.951+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.951698+0000) 2019-10-13T04:20:29.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:29.951+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.951984+0000) 2019-10-13T04:20:29.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:29.946+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.947754+0000) 2019-10-13T04:20:29.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:29.951+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.952245+0000) 2019-10-13T04:20:29.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:29.952+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.952557+0000) 2019-10-13T04:20:29.955 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:29.952+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.952697+0000) 2019-10-13T04:20:29.955 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:29.952+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.952991+0000) 2019-10-13T04:20:29.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:29.953+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:29.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:29.948+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:29.950462+0000) 2019-10-13T04:20:32.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:32.608+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:32.609689+0000) 2019-10-13T04:20:34.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:34.955+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.955447+0000) 2019-10-13T04:20:34.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:34.949+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.951449+0000) 2019-10-13T04:20:34.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:34.955+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.956344+0000) 2019-10-13T04:20:34.959 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:34.955+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.955682+0000) 2019-10-13T04:20:34.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:34.956+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.956664+0000) 2019-10-13T04:20:34.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:34.955+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.956011+0000) 2019-10-13T04:20:34.960 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:34.956+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.956815+0000) 2019-10-13T04:20:34.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:34.957+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:34.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:34.952+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.953978+0000) 2019-10-13T04:20:34.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:34.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.984582+0000) 2019-10-13T04:20:34.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:34.984+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.984586+0000) 2019-10-13T04:20:34.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:34.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:34.984592+0000) 2019-10-13T04:20:35.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:35.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:35.859949+0000) 2019-10-13T04:20:37.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:37.351+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:37.352023+0000) 2019-10-13T04:20:38.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:38.142+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:38.143133+0000) 2019-10-13T04:20:39.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:39.312+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.312911+0000) 2019-10-13T04:20:39.961 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:39.957+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.959079+0000) 2019-10-13T04:20:39.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:39.957+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.959186+0000) 2019-10-13T04:20:39.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:39.957+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.959351+0000) 2019-10-13T04:20:39.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:39.958+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.959504+0000) 2019-10-13T04:20:39.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:39.958+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.960034+0000) 2019-10-13T04:20:39.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:39.954+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.954829+0000) 2019-10-13T04:20:39.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:39.959+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.960497+0000) 2019-10-13T04:20:39.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:39.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53575 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:39.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:39.956+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:39.957535+0000) 2019-10-13T04:20:42.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:42.609+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:42.609828+0000) 2019-10-13T04:20:44.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:44.961+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.962652+0000) 2019-10-13T04:20:44.964 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:44.961+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.962914+0000) 2019-10-13T04:20:44.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:44.961+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.963167+0000) 2019-10-13T04:20:44.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:44.957+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.957937+0000) 2019-10-13T04:20:44.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:44.961+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.963332+0000) 2019-10-13T04:20:44.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:44.962+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.963694+0000) 2019-10-13T04:20:44.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:44.961+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.963260+0000) 2019-10-13T04:20:44.966 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:44.962+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:44.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:44.960+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.961122+0000) 2019-10-13T04:20:44.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:44.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.984746+0000) 2019-10-13T04:20:44.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:44.983+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.984743+0000) 2019-10-13T04:20:44.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:44.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:44.984744+0000) 2019-10-13T04:20:45.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:45.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:45.860100+0000) 2019-10-13T04:20:47.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:47.351+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:47.352221+0000) 2019-10-13T04:20:48.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:48.142+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:48.143289+0000) 2019-10-13T04:20:49.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:49.314+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.315192+0000) 2019-10-13T04:20:49.968 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:49.964+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.965779+0000) 2019-10-13T04:20:49.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:49.964+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.965909+0000) 2019-10-13T04:20:49.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:49.964+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.966297+0000) 2019-10-13T04:20:49.969 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:49.965+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.966709+0000) 2019-10-13T04:20:49.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:49.965+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.966830+0000) 2019-10-13T04:20:49.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:49.960+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.961352+0000) 2019-10-13T04:20:49.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:49.965+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.967171+0000) 2019-10-13T04:20:49.970 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:49.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:49.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:49.963+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:49.963998+0000) 2019-10-13T04:20:50.003 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:20:50.003 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:50.007 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:20:50.008 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:52.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:52.609+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:52.609982+0000) 2019-10-13T04:20:54.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:54.968+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.969958+0000) 2019-10-13T04:20:54.974 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:54.968+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.970355+0000) 2019-10-13T04:20:54.974 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:54.969+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.970581+0000) 2019-10-13T04:20:54.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:54.964+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.965222+0000) 2019-10-13T04:20:54.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:54.969+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.970634+0000) 2019-10-13T04:20:54.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:54.969+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.970791+0000) 2019-10-13T04:20:54.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:54.969+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.971074+0000) 2019-10-13T04:20:54.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:54.970+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:54.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:54.968+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.968765+0000) 2019-10-13T04:20:54.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:54.984+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.984927+0000) 2019-10-13T04:20:54.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:54.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.984933+0000) 2019-10-13T04:20:54.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:54.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:54.984930+0000) 2019-10-13T04:20:55.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:55.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:55.860271+0000) 2019-10-13T04:20:57.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:57.351+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:57.352433+0000) 2019-10-13T04:20:58.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:58.142+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:58.143477+0000) 2019-10-13T04:20:59.324 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:59.316+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.317477+0000) 2019-10-13T04:20:59.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:20:59.972+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.973825+0000) 2019-10-13T04:20:59.976 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:20:59.972+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.974230+0000) 2019-10-13T04:20:59.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:20:59.972+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.974385+0000) 2019-10-13T04:20:59.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:20:59.973+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.974648+0000) 2019-10-13T04:20:59.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:20:59.973+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.974806+0000) 2019-10-13T04:20:59.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:20:59.969+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.970532+0000) 2019-10-13T04:20:59.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:20:59.973+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.975249+0000) 2019-10-13T04:20:59.978 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:20:59.974+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53693 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:20:59.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:20:59.970+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:20:59.971492+0000) 2019-10-13T04:21:02.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:02.609+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:02.610164+0000) 2019-10-13T04:21:04.980 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:04.978+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.978453+0000) 2019-10-13T04:21:04.980 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:04.978+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.978811+0000) 2019-10-13T04:21:04.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:04.978+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.978921+0000) 2019-10-13T04:21:04.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:04.978+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.979155+0000) 2019-10-13T04:21:04.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:04.973+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.974617+0000) 2019-10-13T04:21:04.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:04.978+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.979379+0000) 2019-10-13T04:21:04.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:04.979+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.979582+0000) 2019-10-13T04:21:04.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:04.979+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53723 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:04.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:04.975+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.976012+0000) 2019-10-13T04:21:04.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:04.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.985061+0000) 2019-10-13T04:21:04.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:04.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.985100+0000) 2019-10-13T04:21:04.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:04.984+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:04.985066+0000) 2019-10-13T04:21:05.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:05.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:05.860452+0000) 2019-10-13T04:21:07.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:07.352+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:07.352570+0000) 2019-10-13T04:21:08.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:08.143+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:08.143645+0000) 2019-10-13T04:21:09.326 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:09.319+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.319840+0000) 2019-10-13T04:21:09.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:09.976+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.976839+0000) 2019-10-13T04:21:09.985 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:09.981+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.981827+0000) 2019-10-13T04:21:09.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:09.981+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.982068+0000) 2019-10-13T04:21:09.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:09.981+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.982201+0000) 2019-10-13T04:21:09.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:09.982+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.982463+0000) 2019-10-13T04:21:09.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:09.982+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.982549+0000) 2019-10-13T04:21:09.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:09.982+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.982741+0000) 2019-10-13T04:21:09.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:09.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53753 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:09.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:09.978+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:09.979267+0000) 2019-10-13T04:21:12.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:12.610+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:12.610348+0000) 2019-10-13T04:21:14.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:14.984+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.985336+0000) 2019-10-13T04:21:14.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:14.979+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.980445+0000) 2019-10-13T04:21:14.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:14.985+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.986192+0000) 2019-10-13T04:21:14.988 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:14.986+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.986591+0000) 2019-10-13T04:21:14.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:14.981+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.982186+0000) 2019-10-13T04:21:14.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:14.986+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.987143+0000) 2019-10-13T04:21:14.989 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:14.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53781 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:14.989 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:14.987+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.987606+0000) 2019-10-13T04:21:14.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:14.987+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:14.987441+0000) 2019-10-13T04:21:15.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:15.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:15.860560+0000) 2019-10-13T04:21:17.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:17.352+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:17.352747+0000) 2019-10-13T04:21:18.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:18.143+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:18.143835+0000) 2019-10-13T04:21:19.329 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:19.321+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.322301+0000) 2019-10-13T04:21:19.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:19.982+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.984079+0000) 2019-10-13T04:21:19.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:19.989+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.989393+0000) 2019-10-13T04:21:19.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:19.989+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.989741+0000) 2019-10-13T04:21:19.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:19.989+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.990174+0000) 2019-10-13T04:21:19.992 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:19.989+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.990336+0000) 2019-10-13T04:21:19.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:19.990+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.990473+0000) 2019-10-13T04:21:19.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:19.990+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.990973+0000) 2019-10-13T04:21:19.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:19.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:19.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:19.985+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:19.987036+0000) 2019-10-13T04:21:20.099 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:21:20.099 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:21:20.103 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:21:20.104 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:21:21.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:21.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:21.735479+0000) 2019-10-13T04:21:21.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:21.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:21.735479+0000) 2019-10-13T04:21:21.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:21.734+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:21.735484+0000) 2019-10-13T04:21:22.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:22.609+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:22.610537+0000) 2019-10-13T04:21:24.994 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:24.991+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.992953+0000) 2019-10-13T04:21:24.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:24.991+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.993194+0000) 2019-10-13T04:21:24.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:24.987+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.988871+0000) 2019-10-13T04:21:24.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:24.992+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.993480+0000) 2019-10-13T04:21:24.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:24.992+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.993705+0000) 2019-10-13T04:21:24.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:24.992+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.994122+0000) 2019-10-13T04:21:24.996 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:24.992+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.994301+0000) 2019-10-13T04:21:24.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:24.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:24.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:24.989+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:24.991175+0000) 2019-10-13T04:21:25.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:25.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:25.860733+0000) 2019-10-13T04:21:27.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:27.352+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:27.352935+0000) 2019-10-13T04:21:28.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:28.143+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:28.144025+0000) 2019-10-13T04:21:29.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:29.324+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.324836+0000) 2019-10-13T04:21:29.999 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:29.995+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.997149+0000) 2019-10-13T04:21:29.999 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:29.996+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.997600+0000) 2019-10-13T04:21:30.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:29.991+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.993381+0000) 2019-10-13T04:21:30.000 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:29.996+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.997979+0000) 2019-10-13T04:21:30.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:29.996+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.997969+0000) 2019-10-13T04:21:30.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:29.996+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.998109+0000) 2019-10-13T04:21:30.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:29.996+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.998237+0000) 2019-10-13T04:21:30.001 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:29.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:30.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:29.994+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:29.995978+0000) 2019-10-13T04:21:31.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:31.734+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:31.735622+0000) 2019-10-13T04:21:31.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:31.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:31.735604+0000) 2019-10-13T04:21:31.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:31.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:31.735667+0000) 2019-10-13T04:21:32.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:32.610+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:32.610753+0000) 2019-10-13T04:21:35.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:34.996+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:34.996825+0000) 2019-10-13T04:21:35.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:34.997+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:34.998054+0000) 2019-10-13T04:21:35.006 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:34.999+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:35.000888+0000) 2019-10-13T04:21:35.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:35.000+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:35.001706+0000) 2019-10-13T04:21:35.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:35.000+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:35.001973+0000) 2019-10-13T04:21:35.006 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:35.000+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:35.002135+0000) 2019-10-13T04:21:35.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:35.000+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:35.002330+0000) 2019-10-13T04:21:35.007 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:35.001+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:35.002610+0000) 2019-10-13T04:21:35.007 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:35.001+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53899 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:35.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:35.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:35.860907+0000) 2019-10-13T04:21:37.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:37.352+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:37.353121+0000) 2019-10-13T04:21:38.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:38.143+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:38.144202+0000) 2019-10-13T04:21:39.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:39.326+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:39.327112+0000) 2019-10-13T04:21:40.006 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:40.003+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.004749+0000) 2019-10-13T04:21:40.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:40.003+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.004969+0000) 2019-10-13T04:21:40.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:39.999+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.000664+0000) 2019-10-13T04:21:40.008 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:40.004+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.005501+0000) 2019-10-13T04:21:40.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:40.004+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.005671+0000) 2019-10-13T04:21:40.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:40.004+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.006056+0000) 2019-10-13T04:21:40.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:40.004+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.006234+0000) 2019-10-13T04:21:40.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:40.005+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53929 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:40.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:40.002+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:40.003702+0000) 2019-10-13T04:21:41.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:41.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:41.735824+0000) 2019-10-13T04:21:41.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:41.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:41.735805+0000) 2019-10-13T04:21:41.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:41.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:41.735913+0000) 2019-10-13T04:21:42.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:42.610+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:42.610947+0000) 2019-10-13T04:21:45.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:45.007+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.008717+0000) 2019-10-13T04:21:45.010 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:45.007+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.008859+0000) 2019-10-13T04:21:45.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:45.004+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.004758+0000) 2019-10-13T04:21:45.011 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:45.007+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.009303+0000) 2019-10-13T04:21:45.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:45.008+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.009577+0000) 2019-10-13T04:21:45.012 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:45.008+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.009834+0000) 2019-10-13T04:21:45.012 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:45.008+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.010260+0000) 2019-10-13T04:21:45.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:45.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:45.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:45.006+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.007128+0000) 2019-10-13T04:21:45.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:45.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:45.861075+0000) 2019-10-13T04:21:47.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:47.352+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:47.353328+0000) 2019-10-13T04:21:48.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:48.143+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:48.144386+0000) 2019-10-13T04:21:49.336 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:49.329+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:49.329586+0000) 2019-10-13T04:21:50.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:50.006+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.007596+0000) 2019-10-13T04:21:50.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:50.012+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.012967+0000) 2019-10-13T04:21:50.015 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:50.012+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.013294+0000) 2019-10-13T04:21:50.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:50.012+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.013359+0000) 2019-10-13T04:21:50.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:50.013+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.013646+0000) 2019-10-13T04:21:50.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:50.013+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.013888+0000) 2019-10-13T04:21:50.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:50.013+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.014022+0000) 2019-10-13T04:21:50.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:50.014+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 53989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:50.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:50.011+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:50.011992+0000) 2019-10-13T04:21:50.255 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:21:50.255 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:21:50.259 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:21:50.259 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:21:51.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:51.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:51.736032+0000) 2019-10-13T04:21:51.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:51.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:51.736023+0000) 2019-10-13T04:21:51.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:51.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:51.736095+0000) 2019-10-13T04:21:52.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:52.610+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:52.611129+0000) 2019-10-13T04:21:55.019 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:21:55.016+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.016790+0000) 2019-10-13T04:21:55.019 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:21:55.016+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.017329+0000) 2019-10-13T04:21:55.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:55.017+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.017532+0000) 2019-10-13T04:21:55.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:55.017+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.017617+0000) 2019-10-13T04:21:55.020 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:21:55.017+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.017734+0000) 2019-10-13T04:21:55.020 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:55.017+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.017986+0000) 2019-10-13T04:21:55.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:21:55.011+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.012072+0000) 2019-10-13T04:21:55.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:55.013+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.013851+0000) 2019-10-13T04:21:55.022 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:21:55.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:21:55.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:21:55.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:55.861219+0000) 2019-10-13T04:21:57.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:21:57.353+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:57.353517+0000) 2019-10-13T04:21:58.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:21:58.144+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:58.144563+0000) 2019-10-13T04:21:59.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:21:59.331+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:21:59.332054+0000) 2019-10-13T04:22:00.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:00.020+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.020624+0000) 2019-10-13T04:22:00.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:00.020+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.020889+0000) 2019-10-13T04:22:00.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:00.020+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.021212+0000) 2019-10-13T04:22:00.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:00.015+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.015914+0000) 2019-10-13T04:22:00.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:00.021+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.021536+0000) 2019-10-13T04:22:00.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:00.021+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.021960+0000) 2019-10-13T04:22:00.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:00.021+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.022106+0000) 2019-10-13T04:22:00.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:00.022+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:00.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:00.017+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:00.018523+0000) 2019-10-13T04:22:01.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:01.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:01.736224+0000) 2019-10-13T04:22:01.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:01.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:01.736224+0000) 2019-10-13T04:22:01.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:01.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:01.736336+0000) 2019-10-13T04:22:02.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:02.611+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:02.611293+0000) 2019-10-13T04:22:05.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:05.018+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.019148+0000) 2019-10-13T04:22:05.026 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:05.023+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.024296+0000) 2019-10-13T04:22:05.026 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:05.024+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.024735+0000) 2019-10-13T04:22:05.027 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:05.024+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.024491+0000) 2019-10-13T04:22:05.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:05.024+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.025012+0000) 2019-10-13T04:22:05.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:05.024+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.025297+0000) 2019-10-13T04:22:05.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:05.024+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.025365+0000) 2019-10-13T04:22:05.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:05.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:05.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:05.021+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.022575+0000) 2019-10-13T04:22:05.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:05.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:05.861412+0000) 2019-10-13T04:22:07.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:07.352+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:07.353675+0000) 2019-10-13T04:22:08.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:08.143+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:08.144748+0000) 2019-10-13T04:22:09.341 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:09.334+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:09.334366+0000) 2019-10-13T04:22:10.030 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:10.027+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.028390+0000) 2019-10-13T04:22:10.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:10.027+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.028838+0000) 2019-10-13T04:22:10.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:10.027+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.029087+0000) 2019-10-13T04:22:10.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:10.022+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.023168+0000) 2019-10-13T04:22:10.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:10.028+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.029564+0000) 2019-10-13T04:22:10.032 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:10.028+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.029658+0000) 2019-10-13T04:22:10.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:10.028+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.029809+0000) 2019-10-13T04:22:10.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:10.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:10.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:10.025+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:10.026458+0000) 2019-10-13T04:22:11.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:11.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:11.736438+0000) 2019-10-13T04:22:11.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:11.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:11.736457+0000) 2019-10-13T04:22:11.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:11.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:11.736496+0000) 2019-10-13T04:22:12.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:12.610+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:12.611422+0000) 2019-10-13T04:22:15.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:15.026+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.027844+0000) 2019-10-13T04:22:15.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:15.031+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.032907+0000) 2019-10-13T04:22:15.036 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:15.031+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.033163+0000) 2019-10-13T04:22:15.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:15.031+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.033269+0000) 2019-10-13T04:22:15.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:15.032+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.033408+0000) 2019-10-13T04:22:15.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:15.032+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.033649+0000) 2019-10-13T04:22:15.037 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:15.032+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.033853+0000) 2019-10-13T04:22:15.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:15.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:15.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:15.028+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.030672+0000) 2019-10-13T04:22:15.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:15.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:15.861581+0000) 2019-10-13T04:22:17.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:17.353+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:17.353896+0000) 2019-10-13T04:22:18.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:18.144+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:18.144922+0000) 2019-10-13T04:22:19.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:19.335+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:19.336798+0000) 2019-10-13T04:22:20.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:20.029+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.031161+0000) 2019-10-13T04:22:20.038 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:20.034+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.036368+0000) 2019-10-13T04:22:20.038 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:20.035+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.036462+0000) 2019-10-13T04:22:20.039 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:20.035+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.036641+0000) 2019-10-13T04:22:20.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:20.035+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.036776+0000) 2019-10-13T04:22:20.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:20.035+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.036873+0000) 2019-10-13T04:22:20.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:20.035+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.037037+0000) 2019-10-13T04:22:20.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:20.031+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:20.033081+0000) 2019-10-13T04:22:20.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:20.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:20.442 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:22:20.442 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:22:20.448 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:22:20.448 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:22:21.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:21.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:21.736676+0000) 2019-10-13T04:22:21.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:21.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:21.736676+0000) 2019-10-13T04:22:21.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:21.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:21.736691+0000) 2019-10-13T04:22:22.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:22.610+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:22.611583+0000) 2019-10-13T04:22:25.040 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:25.037+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.039113+0000) 2019-10-13T04:22:25.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:25.038+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.039419+0000) 2019-10-13T04:22:25.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:25.033+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.034543+0000) 2019-10-13T04:22:25.042 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:25.038+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.039922+0000) 2019-10-13T04:22:25.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:25.038+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.040108+0000) 2019-10-13T04:22:25.042 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:25.038+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.040382+0000) 2019-10-13T04:22:25.042 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:25.039+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.040526+0000) 2019-10-13T04:22:25.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:25.036+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.036827+0000) 2019-10-13T04:22:25.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:25.040+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:25.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:25.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:25.861714+0000) 2019-10-13T04:22:27.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:27.353+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:27.354025+0000) 2019-10-13T04:22:28.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:28.144+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:28.145176+0000) 2019-10-13T04:22:29.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:29.338+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:29.339038+0000) 2019-10-13T04:22:30.044 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:30.041+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.043207+0000) 2019-10-13T04:22:30.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:30.037+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.038577+0000) 2019-10-13T04:22:30.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:30.042+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.043865+0000) 2019-10-13T04:22:30.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:30.042+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.043985+0000) 2019-10-13T04:22:30.046 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:30.042+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.044107+0000) 2019-10-13T04:22:30.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:30.042+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.044288+0000) 2019-10-13T04:22:30.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:30.043+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.044567+0000) 2019-10-13T04:22:30.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:30.040+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:30.040784+0000) 2019-10-13T04:22:30.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:30.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:31.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:31.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:31.736878+0000) 2019-10-13T04:22:31.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:31.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:31.736878+0000) 2019-10-13T04:22:31.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:31.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:31.736878+0000) 2019-10-13T04:22:32.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:32.611+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:32.611752+0000) 2019-10-13T04:22:35.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:35.041+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.041754+0000) 2019-10-13T04:22:35.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:35.046+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.047010+0000) 2019-10-13T04:22:35.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:35.047+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.047489+0000) 2019-10-13T04:22:35.050 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:35.047+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.047600+0000) 2019-10-13T04:22:35.050 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:35.047+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.047764+0000) 2019-10-13T04:22:35.050 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:35.047+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.048095+0000) 2019-10-13T04:22:35.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:35.043+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.044180+0000) 2019-10-13T04:22:35.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:35.047+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.048307+0000) 2019-10-13T04:22:35.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:35.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:35.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:35.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:35.861847+0000) 2019-10-13T04:22:37.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:37.353+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:37.354221+0000) 2019-10-13T04:22:38.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:38.144+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:38.145358+0000) 2019-10-13T04:22:39.348 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:39.341+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:39.341536+0000) 2019-10-13T04:22:40.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:40.045+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.045859+0000) 2019-10-13T04:22:40.053 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:40.050+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.051110+0000) 2019-10-13T04:22:40.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:40.050+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.051248+0000) 2019-10-13T04:22:40.053 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:40.050+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.051363+0000) 2019-10-13T04:22:40.054 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:40.051+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.051626+0000) 2019-10-13T04:22:40.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:40.051+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.051810+0000) 2019-10-13T04:22:40.054 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:40.051+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.051840+0000) 2019-10-13T04:22:40.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:40.052+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:40.055 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:40.048+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:40.048801+0000) 2019-10-13T04:22:41.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:41.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:41.737005+0000) 2019-10-13T04:22:41.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:41.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:41.737009+0000) 2019-10-13T04:22:41.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:41.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:41.737022+0000) 2019-10-13T04:22:42.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:42.611+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:42.611884+0000) 2019-10-13T04:22:45.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:45.049+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.049852+0000) 2019-10-13T04:22:45.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:45.054+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.054747+0000) 2019-10-13T04:22:45.057 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:45.054+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.055091+0000) 2019-10-13T04:22:45.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:45.054+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.055268+0000) 2019-10-13T04:22:45.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:45.055+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.055517+0000) 2019-10-13T04:22:45.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:45.056+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.056566+0000) 2019-10-13T04:22:45.058 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:45.056+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.056846+0000) 2019-10-13T04:22:45.058 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:45.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:45.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:45.052+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.053018+0000) 2019-10-13T04:22:45.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:45.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:45.861984+0000) 2019-10-13T04:22:47.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:47.354+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:47.354447+0000) 2019-10-13T04:22:48.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:48.145+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:48.145600+0000) 2019-10-13T04:22:49.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:49.343+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:49.344010+0000) 2019-10-13T04:22:50.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:50.058+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.059232+0000) 2019-10-13T04:22:50.062 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:50.059+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.059609+0000) 2019-10-13T04:22:50.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:50.059+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.059704+0000) 2019-10-13T04:22:50.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:50.059+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.060091+0000) 2019-10-13T04:22:50.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:50.059+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.060151+0000) 2019-10-13T04:22:50.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:50.059+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.060368+0000) 2019-10-13T04:22:50.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:50.054+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.054909+0000) 2019-10-13T04:22:50.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:50.057+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:50.057988+0000) 2019-10-13T04:22:50.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:50.061+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:50.554 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:22:50.554 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:22:50.558 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:22:50.559 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:22:51.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:51.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:51.737158+0000) 2019-10-13T04:22:51.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:51.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:51.737187+0000) 2019-10-13T04:22:51.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:51.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:51.737220+0000) 2019-10-13T04:22:52.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:52.611+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:52.612002+0000) 2019-10-13T04:22:55.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:55.062+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.063453+0000) 2019-10-13T04:22:55.066 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:22:55.062+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.063765+0000) 2019-10-13T04:22:55.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:22:55.062+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.063944+0000) 2019-10-13T04:22:55.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:55.062+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.064161+0000) 2019-10-13T04:22:55.066 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:55.062+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.064381+0000) 2019-10-13T04:22:55.067 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:22:55.063+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.064466+0000) 2019-10-13T04:22:55.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:22:55.058+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.059092+0000) 2019-10-13T04:22:55.067 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:55.059+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.060611+0000) 2019-10-13T04:22:55.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:22:55.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:22:55.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:22:55.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:55.862159+0000) 2019-10-13T04:22:57.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:22:57.353+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:57.354632+0000) 2019-10-13T04:22:58.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:22:58.144+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:58.145782+0000) 2019-10-13T04:22:59.353 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:22:59.346+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:22:59.346353+0000) 2019-10-13T04:23:00.069 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:00.066+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.067615+0000) 2019-10-13T04:23:00.069 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:00.066+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.067789+0000) 2019-10-13T04:23:00.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:00.066+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.067968+0000) 2019-10-13T04:23:00.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:00.062+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.063279+0000) 2019-10-13T04:23:00.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:00.067+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.068460+0000) 2019-10-13T04:23:00.071 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:00.067+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.068630+0000) 2019-10-13T04:23:00.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:00.067+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.068943+0000) 2019-10-13T04:23:00.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:00.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54401 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:00.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:00.063+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:00.064218+0000) 2019-10-13T04:23:01.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:01.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:01.737392+0000) 2019-10-13T04:23:01.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:01.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:01.737418+0000) 2019-10-13T04:23:01.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:01.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:01.737437+0000) 2019-10-13T04:23:02.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:02.611+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:02.612155+0000) 2019-10-13T04:23:05.072 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:05.069+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.070962+0000) 2019-10-13T04:23:05.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:05.069+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.071173+0000) 2019-10-13T04:23:05.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:05.064+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.066315+0000) 2019-10-13T04:23:05.075 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:05.070+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.071540+0000) 2019-10-13T04:23:05.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:05.070+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.072377+0000) 2019-10-13T04:23:05.075 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:05.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.072623+0000) 2019-10-13T04:23:05.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:05.070+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.072119+0000) 2019-10-13T04:23:05.075 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:05.071+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:05.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:05.068+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.069809+0000) 2019-10-13T04:23:05.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:05.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:05.862327+0000) 2019-10-13T04:23:07.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:07.354+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:07.354847+0000) 2019-10-13T04:23:08.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:08.145+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:08.145975+0000) 2019-10-13T04:23:09.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:09.347+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:09.348681+0000) 2019-10-13T04:23:10.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:10.068+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.070209+0000) 2019-10-13T04:23:10.077 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:10.073+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.075353+0000) 2019-10-13T04:23:10.077 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:10.074+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.075524+0000) 2019-10-13T04:23:10.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:10.074+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.075699+0000) 2019-10-13T04:23:10.078 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:10.074+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.075828+0000) 2019-10-13T04:23:10.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:10.074+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.076133+0000) 2019-10-13T04:23:10.078 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:10.075+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.076457+0000) 2019-10-13T04:23:10.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:10.070+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:10.072060+0000) 2019-10-13T04:23:10.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:10.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:11.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:11.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:11.737601+0000) 2019-10-13T04:23:11.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:11.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:11.737616+0000) 2019-10-13T04:23:11.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:11.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:11.737603+0000) 2019-10-13T04:23:12.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:12.611+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:12.612327+0000) 2019-10-13T04:23:15.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:15.072+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.074401+0000) 2019-10-13T04:23:15.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:15.077+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.079275+0000) 2019-10-13T04:23:15.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:15.078+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.079482+0000) 2019-10-13T04:23:15.082 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:15.078+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.079693+0000) 2019-10-13T04:23:15.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:15.078+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.079829+0000) 2019-10-13T04:23:15.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:15.078+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.080177+0000) 2019-10-13T04:23:15.083 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:15.078+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.080335+0000) 2019-10-13T04:23:15.083 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:15.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:15.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:15.075+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.077599+0000) 2019-10-13T04:23:15.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:15.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:15.862502+0000) 2019-10-13T04:23:17.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:17.354+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:17.355013+0000) 2019-10-13T04:23:18.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:18.145+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:18.146177+0000) 2019-10-13T04:23:19.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:19.350+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:19.351019+0000) 2019-10-13T04:23:20.085 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:20.083+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.083440+0000) 2019-10-13T04:23:20.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:20.082+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.083268+0000) 2019-10-13T04:23:20.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:20.083+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.083519+0000) 2019-10-13T04:23:20.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:20.083+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.083870+0000) 2019-10-13T04:23:20.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:20.083+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.084128+0000) 2019-10-13T04:23:20.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:20.078+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.079287+0000) 2019-10-13T04:23:20.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:20.079+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.080250+0000) 2019-10-13T04:23:20.087 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:20.084+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:20.084429+0000) 2019-10-13T04:23:20.087 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:20.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54519 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:20.693 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:23:20.693 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:23:20.697 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:23:20.698 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:23:21.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:21.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:21.737826+0000) 2019-10-13T04:23:21.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:21.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:21.737829+0000) 2019-10-13T04:23:21.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:21.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:21.737828+0000) 2019-10-13T04:23:22.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:22.611+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:22.612509+0000) 2019-10-13T04:23:25.088 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:25.086+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.086574+0000) 2019-10-13T04:23:25.088 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:25.086+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.086742+0000) 2019-10-13T04:23:25.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:25.086+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.086878+0000) 2019-10-13T04:23:25.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:25.081+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.082190+0000) 2019-10-13T04:23:25.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:25.087+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.087412+0000) 2019-10-13T04:23:25.090 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:25.087+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.087721+0000) 2019-10-13T04:23:25.090 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:25.087+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.087909+0000) 2019-10-13T04:23:25.090 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:25.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:25.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:25.085+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.086078+0000) 2019-10-13T04:23:25.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:25.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:25.862653+0000) 2019-10-13T04:23:27.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:27.354+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:27.355191+0000) 2019-10-13T04:23:28.148 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:28.146+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:28.146412+0000) 2019-10-13T04:23:29.360 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:29.353+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:29.353541+0000) 2019-10-13T04:23:30.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:30.084+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.085069+0000) 2019-10-13T04:23:30.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:30.090+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.090590+0000) 2019-10-13T04:23:30.093 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:30.090+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.090462+0000) 2019-10-13T04:23:30.093 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:30.090+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.090808+0000) 2019-10-13T04:23:30.093 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:30.090+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.091061+0000) 2019-10-13T04:23:30.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:30.090+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.091333+0000) 2019-10-13T04:23:30.094 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:30.091+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.091659+0000) 2019-10-13T04:23:30.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:30.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54582 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:30.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:30.088+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:30.088734+0000) 2019-10-13T04:23:31.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:31.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:31.737997+0000) 2019-10-13T04:23:31.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:31.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:31.737996+0000) 2019-10-13T04:23:31.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:31.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:31.737997+0000) 2019-10-13T04:23:32.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:32.612+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:32.612680+0000) 2019-10-13T04:23:35.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:35.093+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.094185+0000) 2019-10-13T04:23:35.096 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:35.093+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.094261+0000) 2019-10-13T04:23:35.096 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:35.094+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.094752+0000) 2019-10-13T04:23:35.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:35.089+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.089724+0000) 2019-10-13T04:23:35.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:35.094+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.095107+0000) 2019-10-13T04:23:35.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:35.094+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.095330+0000) 2019-10-13T04:23:35.097 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:35.095+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.095632+0000) 2019-10-13T04:23:35.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:35.091+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.092075+0000) 2019-10-13T04:23:35.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:35.096+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:35.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:35.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:35.862808+0000) 2019-10-13T04:23:37.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:37.354+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:37.355355+0000) 2019-10-13T04:23:38.148 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:38.145+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:38.146603+0000) 2019-10-13T04:23:39.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:39.355+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:39.355846+0000) 2019-10-13T04:23:40.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:40.096+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.097863+0000) 2019-10-13T04:23:40.099 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:40.096+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.098121+0000) 2019-10-13T04:23:40.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:40.097+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.098495+0000) 2019-10-13T04:23:40.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:40.097+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.098620+0000) 2019-10-13T04:23:40.101 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:40.097+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.099053+0000) 2019-10-13T04:23:40.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:40.098+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.099471+0000) 2019-10-13T04:23:40.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:40.093+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.094242+0000) 2019-10-13T04:23:40.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:40.094+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:40.095721+0000) 2019-10-13T04:23:40.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:40.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:41.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:41.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:41.738185+0000) 2019-10-13T04:23:41.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:41.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:41.738200+0000) 2019-10-13T04:23:41.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:41.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:41.738202+0000) 2019-10-13T04:23:42.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:42.612+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:42.612793+0000) 2019-10-13T04:23:45.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:45.100+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.101662+0000) 2019-10-13T04:23:45.103 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:45.100+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.101971+0000) 2019-10-13T04:23:45.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:45.096+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.097238+0000) 2019-10-13T04:23:45.105 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:45.101+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.102417+0000) 2019-10-13T04:23:45.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:45.101+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.102533+0000) 2019-10-13T04:23:45.105 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:45.101+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.102735+0000) 2019-10-13T04:23:45.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:45.101+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.102986+0000) 2019-10-13T04:23:45.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:45.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:45.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:45.098+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.099035+0000) 2019-10-13T04:23:45.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:45.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:45.862945+0000) 2019-10-13T04:23:47.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:47.354+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:47.355530+0000) 2019-10-13T04:23:48.148 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:48.146+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:48.146796+0000) 2019-10-13T04:23:49.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:49.357+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:49.358197+0000) 2019-10-13T04:23:50.106 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:50.103+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.105181+0000) 2019-10-13T04:23:50.107 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:50.104+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.105526+0000) 2019-10-13T04:23:50.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:50.100+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.101287+0000) 2019-10-13T04:23:50.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:50.104+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.105759+0000) 2019-10-13T04:23:50.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:50.104+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.105711+0000) 2019-10-13T04:23:50.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:50.104+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.105911+0000) 2019-10-13T04:23:50.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:50.105+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.106406+0000) 2019-10-13T04:23:50.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:50.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:50.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:50.104+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:50.105040+0000) 2019-10-13T04:23:50.778 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:23:50.778 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:23:50.782 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:23:50.782 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:23:51.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:51.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:51.738418+0000) 2019-10-13T04:23:51.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:51.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:51.738420+0000) 2019-10-13T04:23:51.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:51.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:51.738419+0000) 2019-10-13T04:23:52.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:52.612+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:52.612961+0000) 2019-10-13T04:23:55.118 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:23:55.108+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.109461+0000) 2019-10-13T04:23:55.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:55.108+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.109680+0000) 2019-10-13T04:23:55.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:23:55.108+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.109869+0000) 2019-10-13T04:23:55.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:55.108+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.110177+0000) 2019-10-13T04:23:55.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:23:55.108+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.110255+0000) 2019-10-13T04:23:55.119 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:55.109+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.110554+0000) 2019-10-13T04:23:55.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:23:55.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:23:55.119 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:23:55.102+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.104386+0000) 2019-10-13T04:23:55.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:55.105+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.106798+0000) 2019-10-13T04:23:55.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:23:55.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:55.863085+0000) 2019-10-13T04:23:57.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:23:57.355+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:57.355723+0000) 2019-10-13T04:23:58.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:23:58.146+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:58.146988+0000) 2019-10-13T04:23:59.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:23:59.359+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:23:59.360539+0000) 2019-10-13T04:24:00.116 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:00.112+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.113895+0000) 2019-10-13T04:24:00.117 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:00.112+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.114224+0000) 2019-10-13T04:24:00.117 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:00.113+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.114438+0000) 2019-10-13T04:24:00.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:00.113+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.114632+0000) 2019-10-13T04:24:00.118 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:00.113+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.114778+0000) 2019-10-13T04:24:00.118 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:00.113+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.115016+0000) 2019-10-13T04:24:00.120 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:00.107+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.109134+0000) 2019-10-13T04:24:00.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:00.109+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:00.111482+0000) 2019-10-13T04:24:00.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:00.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54767 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:01.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:01.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:01.738625+0000) 2019-10-13T04:24:01.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:01.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:01.738622+0000) 2019-10-13T04:24:01.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:01.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:01.738625+0000) 2019-10-13T04:24:02.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:02.612+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:02.613143+0000) 2019-10-13T04:24:05.119 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:05.117+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.117505+0000) 2019-10-13T04:24:05.120 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:05.111+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.113150+0000) 2019-10-13T04:24:05.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:05.117+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.117898+0000) 2019-10-13T04:24:05.120 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:05.117+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.118197+0000) 2019-10-13T04:24:05.120 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:05.118+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.118508+0000) 2019-10-13T04:24:05.121 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:05.118+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.118683+0000) 2019-10-13T04:24:05.121 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:05.118+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.119074+0000) 2019-10-13T04:24:05.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:05.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:05.122 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:05.113+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.115558+0000) 2019-10-13T04:24:05.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:05.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:05.863243+0000) 2019-10-13T04:24:07.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:07.355+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:07.355890+0000) 2019-10-13T04:24:08.148 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:08.146+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:08.147180+0000) 2019-10-13T04:24:09.369 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:09.362+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:09.362872+0000) 2019-10-13T04:24:10.123 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:10.121+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.121647+0000) 2019-10-13T04:24:10.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:10.121+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.121868+0000) 2019-10-13T04:24:10.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:10.121+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.122139+0000) 2019-10-13T04:24:10.124 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:10.117+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.117758+0000) 2019-10-13T04:24:10.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:10.122+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.122425+0000) 2019-10-13T04:24:10.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:10.122+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.122867+0000) 2019-10-13T04:24:10.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:10.122+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.122750+0000) 2019-10-13T04:24:10.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:10.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:10.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:10.119+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:10.120157+0000) 2019-10-13T04:24:11.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:11.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:11.738837+0000) 2019-10-13T04:24:11.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:11.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:11.738820+0000) 2019-10-13T04:24:11.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:11.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:11.738843+0000) 2019-10-13T04:24:12.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:12.612+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:12.613318+0000) 2019-10-13T04:24:15.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:15.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.120191+0000) 2019-10-13T04:24:15.127 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:15.125+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.125391+0000) 2019-10-13T04:24:15.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:15.125+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.126041+0000) 2019-10-13T04:24:15.127 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:15.125+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.126066+0000) 2019-10-13T04:24:15.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:15.125+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.126376+0000) 2019-10-13T04:24:15.128 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:15.126+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.126535+0000) 2019-10-13T04:24:15.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:15.126+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.126743+0000) 2019-10-13T04:24:15.129 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:15.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:15.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:15.122+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.123689+0000) 2019-10-13T04:24:15.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:15.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:15.863431+0000) 2019-10-13T04:24:17.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:17.355+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:17.356067+0000) 2019-10-13T04:24:18.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:18.147+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:18.147376+0000) 2019-10-13T04:24:19.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:19.364+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:19.365239+0000) 2019-10-13T04:24:20.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:20.123+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.124310+0000) 2019-10-13T04:24:20.132 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:20.129+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.129415+0000) 2019-10-13T04:24:20.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:20.129+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.129555+0000) 2019-10-13T04:24:20.133 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:20.129+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.129849+0000) 2019-10-13T04:24:20.133 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:20.129+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.130048+0000) 2019-10-13T04:24:20.133 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:20.129+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.130330+0000) 2019-10-13T04:24:20.134 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:20.129+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.130369+0000) 2019-10-13T04:24:20.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:20.126+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:20.127681+0000) 2019-10-13T04:24:20.135 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:20.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54891 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:20.937 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:24:20.937 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:24:20.941 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:24:20.941 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:24:21.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:21.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:21.739026+0000) 2019-10-13T04:24:21.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:21.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:21.739011+0000) 2019-10-13T04:24:21.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:21.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:21.739031+0000) 2019-10-13T04:24:22.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:22.613+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:22.613504+0000) 2019-10-13T04:24:25.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:25.127+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.128293+0000) 2019-10-13T04:24:25.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:25.132+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.133406+0000) 2019-10-13T04:24:25.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:25.132+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.134346+0000) 2019-10-13T04:24:25.136 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:25.133+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.134698+0000) 2019-10-13T04:24:25.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:25.133+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.134854+0000) 2019-10-13T04:24:25.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:25.133+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.135041+0000) 2019-10-13T04:24:25.137 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:25.133+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.135312+0000) 2019-10-13T04:24:25.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:25.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:25.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:25.132+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.132911+0000) 2019-10-13T04:24:25.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:25.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:25.863617+0000) 2019-10-13T04:24:27.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:27.355+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:27.356257+0000) 2019-10-13T04:24:28.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:28.146+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:28.147603+0000) 2019-10-13T04:24:29.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:29.367+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:29.367575+0000) 2019-10-13T04:24:30.139 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:30.136+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.137758+0000) 2019-10-13T04:24:30.139 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:30.136+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.137871+0000) 2019-10-13T04:24:30.140 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:30.136+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.138010+0000) 2019-10-13T04:24:30.140 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:30.136+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.138196+0000) 2019-10-13T04:24:30.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:30.132+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.133446+0000) 2019-10-13T04:24:30.141 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:30.137+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.138524+0000) 2019-10-13T04:24:30.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:30.137+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.138712+0000) 2019-10-13T04:24:30.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:30.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:30.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:30.135+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:30.136152+0000) 2019-10-13T04:24:31.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:31.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:31.739180+0000) 2019-10-13T04:24:31.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:31.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:31.739188+0000) 2019-10-13T04:24:31.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:31.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:31.739194+0000) 2019-10-13T04:24:32.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:32.613+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:32.613696+0000) 2019-10-13T04:24:35.142 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:35.135+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.136192+0000) 2019-10-13T04:24:35.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:35.140+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.141482+0000) 2019-10-13T04:24:35.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:35.140+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.141844+0000) 2019-10-13T04:24:35.143 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:35.140+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.141996+0000) 2019-10-13T04:24:35.144 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:35.140+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.142245+0000) 2019-10-13T04:24:35.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:35.141+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.142490+0000) 2019-10-13T04:24:35.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:35.141+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.142941+0000) 2019-10-13T04:24:35.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:35.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 54985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:35.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:35.139+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.140032+0000) 2019-10-13T04:24:35.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:35.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:35.863809+0000) 2019-10-13T04:24:37.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:37.355+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:37.356444+0000) 2019-10-13T04:24:38.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:38.147+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:38.147791+0000) 2019-10-13T04:24:39.376 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:39.369+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:39.369967+0000) 2019-10-13T04:24:40.146 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:40.143+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.144825+0000) 2019-10-13T04:24:40.146 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:40.143+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.145135+0000) 2019-10-13T04:24:40.147 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:40.143+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.145227+0000) 2019-10-13T04:24:40.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:40.139+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.140572+0000) 2019-10-13T04:24:40.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:40.142+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.143056+0000) 2019-10-13T04:24:40.151 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:40.144+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.145644+0000) 2019-10-13T04:24:40.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:40.144+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.145881+0000) 2019-10-13T04:24:40.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:40.144+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:40.146176+0000) 2019-10-13T04:24:40.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:40.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:41.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:41.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:41.739370+0000) 2019-10-13T04:24:41.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:41.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:41.739369+0000) 2019-10-13T04:24:41.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:41.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:41.739369+0000) 2019-10-13T04:24:42.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:42.613+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:42.613870+0000) 2019-10-13T04:24:45.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:45.146+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.148150+0000) 2019-10-13T04:24:45.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:45.146+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.148333+0000) 2019-10-13T04:24:45.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:45.147+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.148630+0000) 2019-10-13T04:24:45.151 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:45.147+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.148788+0000) 2019-10-13T04:24:45.152 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:45.143+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.144028+0000) 2019-10-13T04:24:45.152 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:45.147+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.149089+0000) 2019-10-13T04:24:45.152 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:45.147+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.149219+0000) 2019-10-13T04:24:45.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:45.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:45.153 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:45.145+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.146467+0000) 2019-10-13T04:24:45.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:45.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:45.863993+0000) 2019-10-13T04:24:47.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:47.356+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:47.356628+0000) 2019-10-13T04:24:48.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:48.147+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:48.147999+0000) 2019-10-13T04:24:49.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:49.371+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:49.372308+0000) 2019-10-13T04:24:50.154 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:50.151+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.152314+0000) 2019-10-13T04:24:50.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:50.152+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.152501+0000) 2019-10-13T04:24:50.155 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:50.146+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.147777+0000) 2019-10-13T04:24:50.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:50.152+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.153011+0000) 2019-10-13T04:24:50.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:50.152+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.153180+0000) 2019-10-13T04:24:50.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:50.153+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.153594+0000) 2019-10-13T04:24:50.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:50.153+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.153653+0000) 2019-10-13T04:24:50.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:50.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:50.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:50.149+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:50.151324+0000) 2019-10-13T04:24:51.097 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:24:51.097 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:24:51.101 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:24:51.102 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:24:51.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:51.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:51.739559+0000) 2019-10-13T04:24:51.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:51.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:51.739561+0000) 2019-10-13T04:24:51.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:51.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:51.739559+0000) 2019-10-13T04:24:52.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:52.612+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:52.614009+0000) 2019-10-13T04:24:55.157 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:24:55.155+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.156006+0000) 2019-10-13T04:24:55.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:24:55.149+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.151382+0000) 2019-10-13T04:24:55.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:55.156+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.156494+0000) 2019-10-13T04:24:55.159 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:24:55.156+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.157003+0000) 2019-10-13T04:24:55.159 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:55.156+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.157193+0000) 2019-10-13T04:24:55.159 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:24:55.157+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.157413+0000) 2019-10-13T04:24:55.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:55.157+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.157517+0000) 2019-10-13T04:24:55.160 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:24:55.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:24:55.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:55.152+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.153985+0000) 2019-10-13T04:24:55.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:24:55.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:55.864138+0000) 2019-10-13T04:24:57.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:24:57.356+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:57.356806+0000) 2019-10-13T04:24:58.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:24:58.147+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:58.148190+0000) 2019-10-13T04:24:59.381 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:24:59.373+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:24:59.374621+0000) 2019-10-13T04:25:00.161 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:00.153+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.155155+0000) 2019-10-13T04:25:00.162 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:00.159+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.160188+0000) 2019-10-13T04:25:00.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:00.160+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.160775+0000) 2019-10-13T04:25:00.164 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:00.160+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.160887+0000) 2019-10-13T04:25:00.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:00.160+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.161011+0000) 2019-10-13T04:25:00.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:00.156+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.157721+0000) 2019-10-13T04:25:00.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:00.160+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.161324+0000) 2019-10-13T04:25:00.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:00.161+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:00.161525+0000) 2019-10-13T04:25:00.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:00.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:01.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:01.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:01.739772+0000) 2019-10-13T04:25:01.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:01.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:01.739777+0000) 2019-10-13T04:25:01.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:01.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:01.739772+0000) 2019-10-13T04:25:02.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:02.613+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:02.614193+0000) 2019-10-13T04:25:05.166 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:05.159+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.159802+0000) 2019-10-13T04:25:05.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:05.164+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.164912+0000) 2019-10-13T04:25:05.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:05.164+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.165280+0000) 2019-10-13T04:25:05.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:05.164+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.165004+0000) 2019-10-13T04:25:05.168 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:05.165+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.165513+0000) 2019-10-13T04:25:05.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:05.165+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.165832+0000) 2019-10-13T04:25:05.168 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:05.165+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.166005+0000) 2019-10-13T04:25:05.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:05.161+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.162554+0000) 2019-10-13T04:25:05.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:05.166+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:05.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:05.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:05.864307+0000) 2019-10-13T04:25:07.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:07.355+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:07.357002+0000) 2019-10-13T04:25:08.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:08.147+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:08.148383+0000) 2019-10-13T04:25:09.383 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:09.376+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:09.376857+0000) 2019-10-13T04:25:10.172 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:10.167+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.168807+0000) 2019-10-13T04:25:10.173 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:10.167+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.169090+0000) 2019-10-13T04:25:10.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:10.167+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.169320+0000) 2019-10-13T04:25:10.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:10.168+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.169547+0000) 2019-10-13T04:25:10.173 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:10.168+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.169702+0000) 2019-10-13T04:25:10.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:10.168+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.170044+0000) 2019-10-13T04:25:10.174 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:10.169+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55202 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:10.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:10.162+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.163701+0000) 2019-10-13T04:25:10.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:10.166+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:10.167058+0000) 2019-10-13T04:25:11.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:11.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:11.739942+0000) 2019-10-13T04:25:11.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:11.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:11.739945+0000) 2019-10-13T04:25:11.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:11.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:11.739944+0000) 2019-10-13T04:25:12.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:12.613+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:12.614376+0000) 2019-10-13T04:25:15.174 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:15.171+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.172684+0000) 2019-10-13T04:25:15.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:15.171+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.172959+0000) 2019-10-13T04:25:15.175 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:15.167+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.168393+0000) 2019-10-13T04:25:15.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:15.171+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.173145+0000) 2019-10-13T04:25:15.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:15.172+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.173496+0000) 2019-10-13T04:25:15.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:15.172+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.173927+0000) 2019-10-13T04:25:15.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:15.172+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.174125+0000) 2019-10-13T04:25:15.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:15.170+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.170744+0000) 2019-10-13T04:25:15.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:15.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:15.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:15.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:15.864474+0000) 2019-10-13T04:25:17.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:17.356+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:17.357167+0000) 2019-10-13T04:25:18.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:18.147+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:18.148616+0000) 2019-10-13T04:25:19.403 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:19.378+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:19.378828+0000) 2019-10-13T04:25:20.178 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:20.174+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.176205+0000) 2019-10-13T04:25:20.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:20.175+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.176546+0000) 2019-10-13T04:25:20.179 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:20.175+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.176661+0000) 2019-10-13T04:25:20.179 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:20.175+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.176810+0000) 2019-10-13T04:25:20.179 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:20.175+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.177338+0000) 2019-10-13T04:25:20.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:20.176+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.177502+0000) 2019-10-13T04:25:20.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:20.171+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.172151+0000) 2019-10-13T04:25:20.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:20.173+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:20.174163+0000) 2019-10-13T04:25:20.181 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:20.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55263 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:21.284 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:25:21.285 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:25:21.289 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:25:21.289 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:25:21.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:21.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:21.740158+0000) 2019-10-13T04:25:21.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:21.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:21.740146+0000) 2019-10-13T04:25:21.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:21.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:21.740141+0000) 2019-10-13T04:25:22.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:22.614+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:22.614579+0000) 2019-10-13T04:25:25.186 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:25.175+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.175864+0000) 2019-10-13T04:25:25.186 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:25.176+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.177195+0000) 2019-10-13T04:25:25.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:25.178+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.180181+0000) 2019-10-13T04:25:25.186 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:25.178+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.180329+0000) 2019-10-13T04:25:25.187 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:25.179+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.180565+0000) 2019-10-13T04:25:25.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:25.179+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.181376+0000) 2019-10-13T04:25:25.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:25.179+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.181273+0000) 2019-10-13T04:25:25.187 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:25.180+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.181526+0000) 2019-10-13T04:25:25.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:25.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55296 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:25.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:25.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:25.864638+0000) 2019-10-13T04:25:27.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:27.355+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:27.357343+0000) 2019-10-13T04:25:28.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:28.148+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:28.148830+0000) 2019-10-13T04:25:29.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:29.380+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:29.381183+0000) 2019-10-13T04:25:30.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:30.178+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.179355+0000) 2019-10-13T04:25:30.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:30.183+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.184486+0000) 2019-10-13T04:25:30.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:30.183+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.185053+0000) 2019-10-13T04:25:30.187 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:30.183+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.185259+0000) 2019-10-13T04:25:30.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:30.184+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.185511+0000) 2019-10-13T04:25:30.188 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:30.184+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.185618+0000) 2019-10-13T04:25:30.188 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:30.183+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.185259+0000) 2019-10-13T04:25:30.188 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:30.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:30.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:30.181+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:30.182082+0000) 2019-10-13T04:25:31.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:31.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:31.740341+0000) 2019-10-13T04:25:31.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:31.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:31.740333+0000) 2019-10-13T04:25:31.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:31.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:31.740334+0000) 2019-10-13T04:25:32.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:32.614+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:32.614791+0000) 2019-10-13T04:25:35.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:35.183+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.184172+0000) 2019-10-13T04:25:35.192 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:35.188+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.189298+0000) 2019-10-13T04:25:35.193 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:35.189+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.189681+0000) 2019-10-13T04:25:35.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:35.189+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.189869+0000) 2019-10-13T04:25:35.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:35.189+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.189950+0000) 2019-10-13T04:25:35.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:35.189+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.190221+0000) 2019-10-13T04:25:35.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:35.189+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.190181+0000) 2019-10-13T04:25:35.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:35.190+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:35.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:35.186+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.187505+0000) 2019-10-13T04:25:35.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:35.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:35.864803+0000) 2019-10-13T04:25:37.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:37.357+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:37.357520+0000) 2019-10-13T04:25:38.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:38.148+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:38.149021+0000) 2019-10-13T04:25:39.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:39.382+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:39.383191+0000) 2019-10-13T04:25:40.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:40.186+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.187864+0000) 2019-10-13T04:25:40.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:40.192+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.193039+0000) 2019-10-13T04:25:40.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:40.192+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.193296+0000) 2019-10-13T04:25:40.195 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:40.192+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.193352+0000) 2019-10-13T04:25:40.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:40.193+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.193514+0000) 2019-10-13T04:25:40.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:40.193+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.193641+0000) 2019-10-13T04:25:40.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:40.193+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.194184+0000) 2019-10-13T04:25:40.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:40.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:40.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:40.190+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:40.192154+0000) 2019-10-13T04:25:41.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:41.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:41.740517+0000) 2019-10-13T04:25:41.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:41.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:41.740517+0000) 2019-10-13T04:25:41.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:41.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:41.740524+0000) 2019-10-13T04:25:42.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:42.614+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:42.614952+0000) 2019-10-13T04:25:45.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:45.191+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.192737+0000) 2019-10-13T04:25:45.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:45.196+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.196824+0000) 2019-10-13T04:25:45.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:45.196+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.197128+0000) 2019-10-13T04:25:45.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:45.197+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.197487+0000) 2019-10-13T04:25:45.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:45.197+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.198122+0000) 2019-10-13T04:25:45.201 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:45.197+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.198332+0000) 2019-10-13T04:25:45.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:45.198+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.198583+0000) 2019-10-13T04:25:45.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:45.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55420 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:45.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:45.193+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.195173+0000) 2019-10-13T04:25:45.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:45.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:45.864968+0000) 2019-10-13T04:25:47.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:47.357+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:47.357658+0000) 2019-10-13T04:25:48.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:48.149+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:48.149192+0000) 2019-10-13T04:25:49.392 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:49.384+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:49.385647+0000) 2019-10-13T04:25:50.202 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:50.200+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.200762+0000) 2019-10-13T04:25:50.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:50.200+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.200980+0000) 2019-10-13T04:25:50.203 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:50.200+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.201307+0000) 2019-10-13T04:25:50.203 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:50.195+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.196934+0000) 2019-10-13T04:25:50.204 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:50.201+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.201950+0000) 2019-10-13T04:25:50.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:50.201+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.202376+0000) 2019-10-13T04:25:50.204 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:50.201+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.202108+0000) 2019-10-13T04:25:50.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:50.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:50.205 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:50.197+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:50.199050+0000) 2019-10-13T04:25:51.455 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:25:51.455 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:25:51.459 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:25:51.459 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:25:51.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:51.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:51.740702+0000) 2019-10-13T04:25:51.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:51.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:51.740707+0000) 2019-10-13T04:25:51.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:51.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:51.740702+0000) 2019-10-13T04:25:52.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:52.614+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:52.615126+0000) 2019-10-13T04:25:55.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:25:55.198+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.199282+0000) 2019-10-13T04:25:55.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:55.203+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.204529+0000) 2019-10-13T04:25:55.206 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:25:55.203+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.204757+0000) 2019-10-13T04:25:55.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:55.203+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.205221+0000) 2019-10-13T04:25:55.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:25:55.204+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.205477+0000) 2019-10-13T04:25:55.208 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:25:55.204+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.205688+0000) 2019-10-13T04:25:55.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:55.204+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.205862+0000) 2019-10-13T04:25:55.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:25:55.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:25:55.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:55.203+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.203922+0000) 2019-10-13T04:25:55.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:25:55.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:55.865158+0000) 2019-10-13T04:25:57.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:25:57.356+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:57.357871+0000) 2019-10-13T04:25:58.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:25:58.147+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:58.149315+0000) 2019-10-13T04:25:59.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:25:59.387+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:25:59.387954+0000) 2019-10-13T04:26:00.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:00.207+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.208896+0000) 2019-10-13T04:26:00.211 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:00.207+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.209041+0000) 2019-10-13T04:26:00.211 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:00.207+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.209131+0000) 2019-10-13T04:26:00.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:00.208+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.209405+0000) 2019-10-13T04:26:00.212 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:00.208+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.209492+0000) 2019-10-13T04:26:00.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:00.208+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.209746+0000) 2019-10-13T04:26:00.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:00.204+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.204941+0000) 2019-10-13T04:26:00.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:00.205+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:00.205836+0000) 2019-10-13T04:26:00.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:00.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55512 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:01.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:01.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:01.740921+0000) 2019-10-13T04:26:01.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:01.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:01.740918+0000) 2019-10-13T04:26:01.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:01.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:01.740918+0000) 2019-10-13T04:26:02.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:02.614+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:02.615308+0000) 2019-10-13T04:26:05.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:05.210+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.212362+0000) 2019-10-13T04:26:05.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:05.207+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.207789+0000) 2019-10-13T04:26:05.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:05.211+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.212871+0000) 2019-10-13T04:26:05.215 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:05.211+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.212976+0000) 2019-10-13T04:26:05.215 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:05.211+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.213249+0000) 2019-10-13T04:26:05.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:05.211+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.213350+0000) 2019-10-13T04:26:05.216 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:05.211+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.213379+0000) 2019-10-13T04:26:05.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:05.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:05.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:05.210+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.210993+0000) 2019-10-13T04:26:05.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:05.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:05.865302+0000) 2019-10-13T04:26:07.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:07.357+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:07.358070+0000) 2019-10-13T04:26:08.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:08.148+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:08.149510+0000) 2019-10-13T04:26:09.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:09.389+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:09.390391+0000) 2019-10-13T04:26:10.218 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:10.215+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.216447+0000) 2019-10-13T04:26:10.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:10.215+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.216602+0000) 2019-10-13T04:26:10.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:10.215+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.216726+0000) 2019-10-13T04:26:10.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:10.215+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.216927+0000) 2019-10-13T04:26:10.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:10.212+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.212725+0000) 2019-10-13T04:26:10.220 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:10.215+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.217292+0000) 2019-10-13T04:26:10.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:10.216+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.217839+0000) 2019-10-13T04:26:10.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:10.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:10.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:10.213+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:10.214502+0000) 2019-10-13T04:26:11.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:11.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:11.741143+0000) 2019-10-13T04:26:11.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:11.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:11.741136+0000) 2019-10-13T04:26:11.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:11.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:11.741136+0000) 2019-10-13T04:26:12.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:12.615+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:12.615496+0000) 2019-10-13T04:26:15.225 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:15.215+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.216238+0000) 2019-10-13T04:26:15.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:15.218+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.220348+0000) 2019-10-13T04:26:15.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:15.219+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.220808+0000) 2019-10-13T04:26:15.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:15.219+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.220933+0000) 2019-10-13T04:26:15.227 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:15.219+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.221276+0000) 2019-10-13T04:26:15.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:15.220+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.221479+0000) 2019-10-13T04:26:15.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:15.220+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.221614+0000) 2019-10-13T04:26:15.228 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:15.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:15.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:15.217+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.218718+0000) 2019-10-13T04:26:15.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:15.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:15.865425+0000) 2019-10-13T04:26:17.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:17.357+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:17.358257+0000) 2019-10-13T04:26:18.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:18.149+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:18.149717+0000) 2019-10-13T04:26:19.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:19.392+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:19.392738+0000) 2019-10-13T04:26:20.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:20.223+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.224389+0000) 2019-10-13T04:26:20.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:20.224+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.224430+0000) 2019-10-13T04:26:20.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:20.224+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.224577+0000) 2019-10-13T04:26:20.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:20.219+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.219992+0000) 2019-10-13T04:26:20.229 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:20.225+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.225536+0000) 2019-10-13T04:26:20.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:20.225+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.225700+0000) 2019-10-13T04:26:20.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:20.222+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.223522+0000) 2019-10-13T04:26:20.230 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:20.226+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:20.226504+0000) 2019-10-13T04:26:20.230 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:20.226+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:21.612 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:26:21.612 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:26:21.617 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:26:21.618 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:26:21.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:21.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:21.741276+0000) 2019-10-13T04:26:21.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:21.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:21.741324+0000) 2019-10-13T04:26:21.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:21.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:21.741330+0000) 2019-10-13T04:26:22.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:22.615+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:22.615663+0000) 2019-10-13T04:26:25.231 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:25.223+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.224419+0000) 2019-10-13T04:26:25.231 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:25.228+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.228510+0000) 2019-10-13T04:26:25.232 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:25.228+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.228908+0000) 2019-10-13T04:26:25.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:25.228+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.229032+0000) 2019-10-13T04:26:25.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:25.225+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.226000+0000) 2019-10-13T04:26:25.233 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:25.229+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.229420+0000) 2019-10-13T04:26:25.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:25.229+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.229624+0000) 2019-10-13T04:26:25.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:25.229+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.229920+0000) 2019-10-13T04:26:25.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:25.230+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:25.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:25.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:25.865548+0000) 2019-10-13T04:26:27.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:27.358+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:27.358464+0000) 2019-10-13T04:26:28.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:28.149+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:28.149911+0000) 2019-10-13T04:26:29.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:29.394+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:29.395038+0000) 2019-10-13T04:26:30.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:30.227+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.228039+0000) 2019-10-13T04:26:30.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:30.232+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.232711+0000) 2019-10-13T04:26:30.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:30.232+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.233098+0000) 2019-10-13T04:26:30.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:30.232+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.233286+0000) 2019-10-13T04:26:30.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:30.233+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.233435+0000) 2019-10-13T04:26:30.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:30.233+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.233846+0000) 2019-10-13T04:26:30.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:30.233+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.234155+0000) 2019-10-13T04:26:30.239 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:30.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:30.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:30.230+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:30.231169+0000) 2019-10-13T04:26:31.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:31.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:31.741411+0000) 2019-10-13T04:26:31.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:31.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:31.741497+0000) 2019-10-13T04:26:31.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:31.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:31.741497+0000) 2019-10-13T04:26:32.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:32.614+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:32.615860+0000) 2019-10-13T04:26:35.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:35.236+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.236695+0000) 2019-10-13T04:26:35.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:35.236+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.237084+0000) 2019-10-13T04:26:35.239 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:35.230+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.232398+0000) 2019-10-13T04:26:35.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:35.237+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.237431+0000) 2019-10-13T04:26:35.240 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:35.237+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.237873+0000) 2019-10-13T04:26:35.241 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:35.237+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.238041+0000) 2019-10-13T04:26:35.241 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:35.237+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.238237+0000) 2019-10-13T04:26:35.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:35.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:35.242 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:35.234+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.235727+0000) 2019-10-13T04:26:35.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:35.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:35.865737+0000) 2019-10-13T04:26:37.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:37.357+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:37.358641+0000) 2019-10-13T04:26:38.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:38.149+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:38.150121+0000) 2019-10-13T04:26:39.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:39.396+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:39.397457+0000) 2019-10-13T04:26:40.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:40.239+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.240531+0000) 2019-10-13T04:26:40.242 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:40.239+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.240747+0000) 2019-10-13T04:26:40.243 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:40.234+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.236378+0000) 2019-10-13T04:26:40.243 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:40.240+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.241594+0000) 2019-10-13T04:26:40.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:40.240+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.241742+0000) 2019-10-13T04:26:40.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:40.240+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.241810+0000) 2019-10-13T04:26:40.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:40.240+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.242166+0000) 2019-10-13T04:26:40.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:40.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:40.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:40.237+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:40.238825+0000) 2019-10-13T04:26:41.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:41.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:41.741557+0000) 2019-10-13T04:26:41.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:41.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:41.741635+0000) 2019-10-13T04:26:41.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:41.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:41.741635+0000) 2019-10-13T04:26:42.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:42.615+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:42.615997+0000) 2019-10-13T04:26:45.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:45.242+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.244297+0000) 2019-10-13T04:26:45.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:45.243+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.244782+0000) 2019-10-13T04:26:45.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:45.239+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.240337+0000) 2019-10-13T04:26:45.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:45.244+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.245423+0000) 2019-10-13T04:26:45.248 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:45.244+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.245565+0000) 2019-10-13T04:26:45.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:45.244+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.245716+0000) 2019-10-13T04:26:45.248 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:45.244+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.245993+0000) 2019-10-13T04:26:45.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:45.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:45.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:45.241+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.242728+0000) 2019-10-13T04:26:45.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:45.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:45.865904+0000) 2019-10-13T04:26:47.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:47.358+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:47.358837+0000) 2019-10-13T04:26:48.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:48.149+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:48.150290+0000) 2019-10-13T04:26:49.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:49.399+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:49.400575+0000) 2019-10-13T04:26:50.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:50.247+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.248528+0000) 2019-10-13T04:26:50.251 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:50.247+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.248712+0000) 2019-10-13T04:26:50.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:50.247+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.249012+0000) 2019-10-13T04:26:50.251 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:50.247+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.249158+0000) 2019-10-13T04:26:50.252 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:50.244+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.244974+0000) 2019-10-13T04:26:50.252 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:50.248+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.249698+0000) 2019-10-13T04:26:50.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:50.248+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.250067+0000) 2019-10-13T04:26:50.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:50.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55822 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:50.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:50.245+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:50.246487+0000) 2019-10-13T04:26:51.724 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:26:51.724 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:26:51.728 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:26:51.728 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:26:51.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:51.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:51.741676+0000) 2019-10-13T04:26:51.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:51.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:51.741745+0000) 2019-10-13T04:26:51.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:51.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:51.741745+0000) 2019-10-13T04:26:52.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:52.615+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:52.616179+0000) 2019-10-13T04:26:55.254 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:26:55.250+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.252375+0000) 2019-10-13T04:26:55.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:26:55.251+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.252632+0000) 2019-10-13T04:26:55.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:55.251+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.252768+0000) 2019-10-13T04:26:55.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:26:55.247+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.248445+0000) 2019-10-13T04:26:55.255 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:55.251+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.253002+0000) 2019-10-13T04:26:55.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:55.252+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.253882+0000) 2019-10-13T04:26:55.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:26:55.252+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.254053+0000) 2019-10-13T04:26:55.256 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:26:55.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:26:55.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:55.250+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.251004+0000) 2019-10-13T04:26:55.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:26:55.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:55.866102+0000) 2019-10-13T04:26:57.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:26:57.358+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:57.358998+0000) 2019-10-13T04:26:58.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:26:58.149+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:58.150510+0000) 2019-10-13T04:26:59.410 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:26:59.403+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:26:59.403763+0000) 2019-10-13T04:27:00.258 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:00.254+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.256259+0000) 2019-10-13T04:27:00.258 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:00.255+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.256459+0000) 2019-10-13T04:27:00.258 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:00.255+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.256621+0000) 2019-10-13T04:27:00.259 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:00.251+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.252181+0000) 2019-10-13T04:27:00.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:00.255+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.257318+0000) 2019-10-13T04:27:00.260 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:00.256+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.257530+0000) 2019-10-13T04:27:00.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:00.256+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.258031+0000) 2019-10-13T04:27:00.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:00.256+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:00.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:00.254+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:00.255046+0000) 2019-10-13T04:27:01.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:01.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:01.741848+0000) 2019-10-13T04:27:01.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:01.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:01.741914+0000) 2019-10-13T04:27:01.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:01.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:01.741937+0000) 2019-10-13T04:27:02.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:02.615+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:02.616360+0000) 2019-10-13T04:27:05.261 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:05.254+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.254978+0000) 2019-10-13T04:27:05.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:05.259+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.260165+0000) 2019-10-13T04:27:05.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:05.259+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.260346+0000) 2019-10-13T04:27:05.262 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:05.260+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.260563+0000) 2019-10-13T04:27:05.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:05.260+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.260808+0000) 2019-10-13T04:27:05.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:05.260+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.261102+0000) 2019-10-13T04:27:05.264 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:05.261+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.261460+0000) 2019-10-13T04:27:05.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:05.261+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:05.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:05.258+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.258888+0000) 2019-10-13T04:27:05.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:05.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:05.866247+0000) 2019-10-13T04:27:07.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:07.358+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:07.359181+0000) 2019-10-13T04:27:08.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:08.150+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:08.150729+0000) 2019-10-13T04:27:09.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:09.405+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:09.406115+0000) 2019-10-13T04:27:10.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:10.258+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.259237+0000) 2019-10-13T04:27:10.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:10.264+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.264445+0000) 2019-10-13T04:27:10.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:10.264+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.264706+0000) 2019-10-13T04:27:10.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:10.264+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.264842+0000) 2019-10-13T04:27:10.267 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:10.264+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.265051+0000) 2019-10-13T04:27:10.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:10.264+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.265213+0000) 2019-10-13T04:27:10.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:10.260+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.260939+0000) 2019-10-13T04:27:10.268 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:10.265+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:10.265506+0000) 2019-10-13T04:27:10.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:10.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:11.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:11.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:11.742016+0000) 2019-10-13T04:27:11.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:11.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:11.742115+0000) 2019-10-13T04:27:11.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:11.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:11.742114+0000) 2019-10-13T04:27:12.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:12.616+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:12.616522+0000) 2019-10-13T04:27:15.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:15.267+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.267940+0000) 2019-10-13T04:27:15.270 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:15.267+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.268141+0000) 2019-10-13T04:27:15.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:15.268+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.268504+0000) 2019-10-13T04:27:15.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:15.268+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.268878+0000) 2019-10-13T04:27:15.271 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:15.268+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.269066+0000) 2019-10-13T04:27:15.272 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:15.263+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.264538+0000) 2019-10-13T04:27:15.272 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:15.268+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.269205+0000) 2019-10-13T04:27:15.272 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:15.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 55977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:15.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:15.265+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.265990+0000) 2019-10-13T04:27:15.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:15.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:15.866435+0000) 2019-10-13T04:27:17.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:17.359+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:17.359365+0000) 2019-10-13T04:27:18.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:18.150+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:18.150943+0000) 2019-10-13T04:27:19.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:19.408+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:19.408532+0000) 2019-10-13T04:27:20.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:20.271+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.272250+0000) 2019-10-13T04:27:20.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:20.272+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.272486+0000) 2019-10-13T04:27:20.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:20.272+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.272568+0000) 2019-10-13T04:27:20.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:20.272+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.272771+0000) 2019-10-13T04:27:20.276 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:20.267+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.268103+0000) 2019-10-13T04:27:20.276 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:20.272+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.273370+0000) 2019-10-13T04:27:20.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:20.273+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.273570+0000) 2019-10-13T04:27:20.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:20.273+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:20.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:20.270+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:20.270736+0000) 2019-10-13T04:27:21.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:21.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:21.742192+0000) 2019-10-13T04:27:21.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:21.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:21.742287+0000) 2019-10-13T04:27:21.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:21.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:21.742291+0000) 2019-10-13T04:27:21.818 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:27:21.819 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:27:21.823 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:27:21.823 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:27:22.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:22.614+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:22.616696+0000) 2019-10-13T04:27:25.278 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:25.274+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.276290+0000) 2019-10-13T04:27:25.278 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:25.275+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.276494+0000) 2019-10-13T04:27:25.278 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:25.275+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.276805+0000) 2019-10-13T04:27:25.279 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:25.270+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.272374+0000) 2019-10-13T04:27:25.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:25.275+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.277015+0000) 2019-10-13T04:27:25.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:25.276+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.277599+0000) 2019-10-13T04:27:25.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:25.276+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.277880+0000) 2019-10-13T04:27:25.280 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:25.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56040 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:25.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:25.272+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.273940+0000) 2019-10-13T04:27:25.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:25.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:25.866597+0000) 2019-10-13T04:27:27.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:27.358+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:27.359525+0000) 2019-10-13T04:27:28.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:28.150+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:28.151132+0000) 2019-10-13T04:27:29.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:29.409+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:29.410808+0000) 2019-10-13T04:27:30.281 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:30.278+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.280024+0000) 2019-10-13T04:27:30.282 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:30.278+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.280198+0000) 2019-10-13T04:27:30.283 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:30.274+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.275802+0000) 2019-10-13T04:27:30.283 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:30.279+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.280570+0000) 2019-10-13T04:27:30.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:30.279+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.280751+0000) 2019-10-13T04:27:30.283 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:30.279+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.280887+0000) 2019-10-13T04:27:30.284 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:30.279+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.281232+0000) 2019-10-13T04:27:30.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:30.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:30.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:30.275+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:30.277699+0000) 2019-10-13T04:27:31.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:31.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:31.742382+0000) 2019-10-13T04:27:31.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:31.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:31.742481+0000) 2019-10-13T04:27:31.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:31.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:31.742473+0000) 2019-10-13T04:27:32.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:32.616+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:32.616888+0000) 2019-10-13T04:27:35.285 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:35.281+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.283125+0000) 2019-10-13T04:27:35.285 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:35.281+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.283245+0000) 2019-10-13T04:27:35.286 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:35.277+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.278834+0000) 2019-10-13T04:27:35.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:35.282+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.283792+0000) 2019-10-13T04:27:35.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:35.282+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.284111+0000) 2019-10-13T04:27:35.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:35.282+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.284274+0000) 2019-10-13T04:27:35.287 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:35.282+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.284361+0000) 2019-10-13T04:27:35.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:35.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56101 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:35.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:35.279+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.281244+0000) 2019-10-13T04:27:35.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:35.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:35.866762+0000) 2019-10-13T04:27:37.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:37.359+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:37.359717+0000) 2019-10-13T04:27:38.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:38.150+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:38.151278+0000) 2019-10-13T04:27:39.419 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:39.412+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:39.413205+0000) 2019-10-13T04:27:40.293 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:40.283+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.283985+0000) 2019-10-13T04:27:40.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:40.284+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.285700+0000) 2019-10-13T04:27:40.294 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:40.285+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.286949+0000) 2019-10-13T04:27:40.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:40.285+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.287138+0000) 2019-10-13T04:27:40.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:40.285+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.287300+0000) 2019-10-13T04:27:40.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:40.286+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.288066+0000) 2019-10-13T04:27:40.295 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:40.287+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.288659+0000) 2019-10-13T04:27:40.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:40.287+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:40.289098+0000) 2019-10-13T04:27:40.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:40.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56133 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:41.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:41.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:41.742548+0000) 2019-10-13T04:27:41.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:41.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:41.742644+0000) 2019-10-13T04:27:41.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:41.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:41.742648+0000) 2019-10-13T04:27:42.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:42.616+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:42.617067+0000) 2019-10-13T04:27:45.293 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:45.290+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.291988+0000) 2019-10-13T04:27:45.296 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:45.287+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.287868+0000) 2019-10-13T04:27:45.296 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:45.290+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.292158+0000) 2019-10-13T04:27:45.297 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:45.290+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.292229+0000) 2019-10-13T04:27:45.297 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:45.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:45.297 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:45.290+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.292326+0000) 2019-10-13T04:27:45.297 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:45.291+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.292574+0000) 2019-10-13T04:27:45.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:45.291+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.292893+0000) 2019-10-13T04:27:45.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:45.289+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.290665+0000) 2019-10-13T04:27:45.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:45.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:45.866926+0000) 2019-10-13T04:27:47.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:47.359+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:47.359930+0000) 2019-10-13T04:27:48.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:48.151+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:48.151480+0000) 2019-10-13T04:27:49.423 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:49.415+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:49.415633+0000) 2019-10-13T04:27:50.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:50.293+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.294187+0000) 2019-10-13T04:27:50.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:50.293+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.294356+0000) 2019-10-13T04:27:50.297 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:50.294+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.294605+0000) 2019-10-13T04:27:50.298 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:50.289+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.289808+0000) 2019-10-13T04:27:50.298 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:50.294+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.295056+0000) 2019-10-13T04:27:50.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:50.295+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.295477+0000) 2019-10-13T04:27:50.299 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:50.295+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.295752+0000) 2019-10-13T04:27:50.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:50.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56194 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:50.299 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:50.291+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:50.291906+0000) 2019-10-13T04:27:51.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:51.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:51.742751+0000) 2019-10-13T04:27:51.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:51.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:51.742888+0000) 2019-10-13T04:27:51.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:51.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:51.742890+0000) 2019-10-13T04:27:51.899 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:27:51.899 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:27:51.903 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:27:51.903 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:27:52.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:52.616+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:52.617270+0000) 2019-10-13T04:27:55.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:55.297+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.298326+0000) 2019-10-13T04:27:55.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:27:55.298+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.298471+0000) 2019-10-13T04:27:55.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:27:55.298+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.298760+0000) 2019-10-13T04:27:55.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:55.298+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.298991+0000) 2019-10-13T04:27:55.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:27:55.293+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.294242+0000) 2019-10-13T04:27:55.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:55.298+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.299213+0000) 2019-10-13T04:27:55.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:27:55.298+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.299350+0000) 2019-10-13T04:27:55.302 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:27:55.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:27:55.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:55.294+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.295550+0000) 2019-10-13T04:27:55.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:27:55.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:55.867073+0000) 2019-10-13T04:27:57.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:27:57.359+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:57.360153+0000) 2019-10-13T04:27:58.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:27:58.151+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:58.151669+0000) 2019-10-13T04:27:59.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:27:59.417+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:27:59.417912+0000) 2019-10-13T04:28:00.303 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:00.296+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.297078+0000) 2019-10-13T04:28:00.304 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:00.301+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.302257+0000) 2019-10-13T04:28:00.305 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:00.301+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.302303+0000) 2019-10-13T04:28:00.305 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:00.302+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.302553+0000) 2019-10-13T04:28:00.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:00.302+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.302638+0000) 2019-10-13T04:28:00.305 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:00.302+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.302802+0000) 2019-10-13T04:28:00.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:00.302+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.302977+0000) 2019-10-13T04:28:00.306 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:00.298+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:00.299385+0000) 2019-10-13T04:28:00.306 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:00.303+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:01.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:01.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:01.742969+0000) 2019-10-13T04:28:01.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:01.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:01.743104+0000) 2019-10-13T04:28:01.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:01.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:01.743136+0000) 2019-10-13T04:28:02.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:02.616+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:02.617472+0000) 2019-10-13T04:28:05.308 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:05.300+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.300946+0000) 2019-10-13T04:28:05.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:05.305+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.305673+0000) 2019-10-13T04:28:05.308 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:05.305+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.305977+0000) 2019-10-13T04:28:05.309 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:05.305+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.306043+0000) 2019-10-13T04:28:05.309 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:05.305+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.306266+0000) 2019-10-13T04:28:05.309 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:05.305+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.306288+0000) 2019-10-13T04:28:05.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:05.306+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.306900+0000) 2019-10-13T04:28:05.310 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:05.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:05.310 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:05.303+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.303833+0000) 2019-10-13T04:28:05.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:05.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:05.867256+0000) 2019-10-13T04:28:07.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:07.359+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:07.360310+0000) 2019-10-13T04:28:08.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:08.150+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:08.151861+0000) 2019-10-13T04:28:09.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:09.420+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:09.420326+0000) 2019-10-13T04:28:10.314 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:10.308+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.309611+0000) 2019-10-13T04:28:10.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:10.308+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.309782+0000) 2019-10-13T04:28:10.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:10.308+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.310141+0000) 2019-10-13T04:28:10.314 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:10.309+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.310585+0000) 2019-10-13T04:28:10.314 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:10.309+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.310878+0000) 2019-10-13T04:28:10.315 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:10.309+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.311198+0000) 2019-10-13T04:28:10.315 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:10.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56318 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:10.315 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:10.304+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.305154+0000) 2019-10-13T04:28:10.316 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:10.306+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:10.307514+0000) 2019-10-13T04:28:11.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:11.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:11.743207+0000) 2019-10-13T04:28:11.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:11.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:11.743286+0000) 2019-10-13T04:28:11.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:11.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:11.743322+0000) 2019-10-13T04:28:12.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:12.617+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:12.617659+0000) 2019-10-13T04:28:15.315 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:15.306+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.308718+0000) 2019-10-13T04:28:15.316 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:15.312+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.313675+0000) 2019-10-13T04:28:15.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:15.312+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.313924+0000) 2019-10-13T04:28:15.316 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:15.313+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.314426+0000) 2019-10-13T04:28:15.317 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:15.313+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.314604+0000) 2019-10-13T04:28:15.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:15.313+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.314848+0000) 2019-10-13T04:28:15.318 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:15.313+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.314987+0000) 2019-10-13T04:28:15.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:15.314+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:15.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:15.309+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.311577+0000) 2019-10-13T04:28:15.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:15.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:15.867418+0000) 2019-10-13T04:28:17.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:17.359+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:17.360489+0000) 2019-10-13T04:28:18.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:18.151+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:18.152036+0000) 2019-10-13T04:28:19.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:19.421+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:19.422755+0000) 2019-10-13T04:28:20.319 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:20.316+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.317624+0000) 2019-10-13T04:28:20.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:20.316+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.318085+0000) 2019-10-13T04:28:20.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:20.316+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.318227+0000) 2019-10-13T04:28:20.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:20.311+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.313553+0000) 2019-10-13T04:28:20.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:20.317+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.318521+0000) 2019-10-13T04:28:20.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:20.317+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.319060+0000) 2019-10-13T04:28:20.321 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:20.317+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.319291+0000) 2019-10-13T04:28:20.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:20.313+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:20.315468+0000) 2019-10-13T04:28:20.322 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:20.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56381 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:21.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:21.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:21.743450+0000) 2019-10-13T04:28:21.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:21.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:21.743490+0000) 2019-10-13T04:28:21.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:21.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:21.743557+0000) 2019-10-13T04:28:22.004 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:28:22.004 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:28:22.008 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:28:22.008 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:28:22.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:22.616+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:22.617833+0000) 2019-10-13T04:28:25.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:25.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.316985+0000) 2019-10-13T04:28:25.324 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:25.320+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.322014+0000) 2019-10-13T04:28:25.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:25.320+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.322216+0000) 2019-10-13T04:28:25.325 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:25.321+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.322731+0000) 2019-10-13T04:28:25.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:25.321+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.323037+0000) 2019-10-13T04:28:25.325 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:25.321+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.323168+0000) 2019-10-13T04:28:25.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:25.322+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.323437+0000) 2019-10-13T04:28:25.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:25.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56412 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:25.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:25.318+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.320562+0000) 2019-10-13T04:28:25.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:25.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:25.867564+0000) 2019-10-13T04:28:27.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:27.360+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:27.360695+0000) 2019-10-13T04:28:28.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:28.151+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:28.152234+0000) 2019-10-13T04:28:29.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:29.424+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:29.425122+0000) 2019-10-13T04:28:30.327 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:30.324+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.325518+0000) 2019-10-13T04:28:30.327 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:30.324+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.325670+0000) 2019-10-13T04:28:30.328 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:30.324+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.326057+0000) 2019-10-13T04:28:30.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:30.320+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.321055+0000) 2019-10-13T04:28:30.329 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:30.324+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.326305+0000) 2019-10-13T04:28:30.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:30.325+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.326754+0000) 2019-10-13T04:28:30.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:30.325+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.326966+0000) 2019-10-13T04:28:30.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:30.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:30.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:30.322+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:30.323248+0000) 2019-10-13T04:28:31.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:31.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:31.743599+0000) 2019-10-13T04:28:31.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:31.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:31.743687+0000) 2019-10-13T04:28:31.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:31.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:31.743716+0000) 2019-10-13T04:28:32.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:32.617+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:32.618020+0000) 2019-10-13T04:28:35.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:35.328+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.328807+0000) 2019-10-13T04:28:35.331 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:35.323+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.324147+0000) 2019-10-13T04:28:35.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:35.328+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.329363+0000) 2019-10-13T04:28:35.332 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:35.328+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.329108+0000) 2019-10-13T04:28:35.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:35.329+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.329442+0000) 2019-10-13T04:28:35.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:35.329+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.329764+0000) 2019-10-13T04:28:35.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:35.329+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.330107+0000) 2019-10-13T04:28:35.333 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:35.330+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:35.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:35.326+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.327318+0000) 2019-10-13T04:28:35.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:35.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:35.867731+0000) 2019-10-13T04:28:37.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:37.360+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:37.360889+0000) 2019-10-13T04:28:38.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:38.152+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:38.152438+0000) 2019-10-13T04:28:39.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:39.426+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:39.427495+0000) 2019-10-13T04:28:40.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:40.332+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.332896+0000) 2019-10-13T04:28:40.336 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:40.328+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.328901+0000) 2019-10-13T04:28:40.336 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:40.332+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.333239+0000) 2019-10-13T04:28:40.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:40.333+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.333502+0000) 2019-10-13T04:28:40.337 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:40.333+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.334060+0000) 2019-10-13T04:28:40.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:40.333+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.333948+0000) 2019-10-13T04:28:40.337 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:40.333+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.334233+0000) 2019-10-13T04:28:40.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:40.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:40.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:40.331+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:40.331854+0000) 2019-10-13T04:28:41.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:41.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:41.743774+0000) 2019-10-13T04:28:41.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:41.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:41.743845+0000) 2019-10-13T04:28:41.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:41.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:41.743840+0000) 2019-10-13T04:28:42.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:42.617+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:42.618177+0000) 2019-10-13T04:28:45.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:45.336+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.336830+0000) 2019-10-13T04:28:45.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:45.336+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.336753+0000) 2019-10-13T04:28:45.339 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:45.336+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.337035+0000) 2019-10-13T04:28:45.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:45.336+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.337188+0000) 2019-10-13T04:28:45.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:45.336+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.337295+0000) 2019-10-13T04:28:45.340 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:45.332+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.333250+0000) 2019-10-13T04:28:45.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:45.337+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.338271+0000) 2019-10-13T04:28:45.341 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:45.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56536 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:45.341 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:45.334+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.335445+0000) 2019-10-13T04:28:45.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:45.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:45.867894+0000) 2019-10-13T04:28:47.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:47.360+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:47.361043+0000) 2019-10-13T04:28:48.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:48.152+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:48.152622+0000) 2019-10-13T04:28:49.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:49.429+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:49.429813+0000) 2019-10-13T04:28:50.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:50.334+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.335714+0000) 2019-10-13T04:28:50.343 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:50.340+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.341043+0000) 2019-10-13T04:28:50.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:50.340+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.341166+0000) 2019-10-13T04:28:50.343 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:50.340+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.341291+0000) 2019-10-13T04:28:50.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:50.341+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.341475+0000) 2019-10-13T04:28:50.344 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:50.341+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.341567+0000) 2019-10-13T04:28:50.344 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:50.341+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.341726+0000) 2019-10-13T04:28:50.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:50.337+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:50.337944+0000) 2019-10-13T04:28:50.345 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:50.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:51.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:51.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:51.744057+0000) 2019-10-13T04:28:51.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:51.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:51.743985+0000) 2019-10-13T04:28:51.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:51.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:51.744059+0000) 2019-10-13T04:28:52.145 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:28:52.146 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:28:52.150 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:28:52.150 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:28:52.625 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:52.618+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:52.618359+0000) 2019-10-13T04:28:55.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:55.342+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.344286+0000) 2019-10-13T04:28:55.346 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:28:55.343+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.344411+0000) 2019-10-13T04:28:55.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:55.343+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.344690+0000) 2019-10-13T04:28:55.348 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:28:55.339+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.339913+0000) 2019-10-13T04:28:55.348 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:55.343+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.344930+0000) 2019-10-13T04:28:55.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:28:55.343+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.345036+0000) 2019-10-13T04:28:55.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:28:55.343+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.345281+0000) 2019-10-13T04:28:55.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:28:55.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56597 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:28:55.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:55.342+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.343130+0000) 2019-10-13T04:28:55.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:28:55.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:55.868077+0000) 2019-10-13T04:28:57.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:28:57.360+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:57.361178+0000) 2019-10-13T04:28:58.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:28:58.151+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:58.152820+0000) 2019-10-13T04:28:59.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:28:59.432+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:28:59.432202+0000) 2019-10-13T04:29:00.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:00.346+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.347929+0000) 2019-10-13T04:29:00.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:00.346+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.348264+0000) 2019-10-13T04:29:00.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:00.346+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.348330+0000) 2019-10-13T04:29:00.373 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:00.347+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.348475+0000) 2019-10-13T04:29:00.373 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:00.347+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.348717+0000) 2019-10-13T04:29:00.373 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:00.347+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.349146+0000) 2019-10-13T04:29:00.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:00.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56629 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:00.373 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:00.342+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.343219+0000) 2019-10-13T04:29:00.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:00.345+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:00.345941+0000) 2019-10-13T04:29:01.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:01.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:01.744174+0000) 2019-10-13T04:29:01.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:01.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:01.744265+0000) 2019-10-13T04:29:01.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:01.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:01.744246+0000) 2019-10-13T04:29:02.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:02.618+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:02.618557+0000) 2019-10-13T04:29:05.354 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:05.350+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.351693+0000) 2019-10-13T04:29:05.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:05.350+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.351950+0000) 2019-10-13T04:29:05.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:05.351+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.352496+0000) 2019-10-13T04:29:05.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:05.351+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.352647+0000) 2019-10-13T04:29:05.355 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:05.351+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.352906+0000) 2019-10-13T04:29:05.357 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:05.346+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.347985+0000) 2019-10-13T04:29:05.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:05.351+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.353285+0000) 2019-10-13T04:29:05.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:05.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56660 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:05.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:05.349+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.350910+0000) 2019-10-13T04:29:05.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:05.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:05.868247+0000) 2019-10-13T04:29:07.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:07.360+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:07.361335+0000) 2019-10-13T04:29:08.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:08.152+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:08.152984+0000) 2019-10-13T04:29:09.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:09.433+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:09.434570+0000) 2019-10-13T04:29:10.357 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:10.354+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.355615+0000) 2019-10-13T04:29:10.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:10.349+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.351257+0000) 2019-10-13T04:29:10.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:10.354+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.356208+0000) 2019-10-13T04:29:10.359 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:10.354+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.356351+0000) 2019-10-13T04:29:10.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:10.355+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.356547+0000) 2019-10-13T04:29:10.359 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:10.355+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.356595+0000) 2019-10-13T04:29:10.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:10.355+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.356752+0000) 2019-10-13T04:29:10.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:10.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:10.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:10.352+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:10.354398+0000) 2019-10-13T04:29:11.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:11.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:11.744363+0000) 2019-10-13T04:29:11.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:11.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:11.744421+0000) 2019-10-13T04:29:11.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:11.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:11.744443+0000) 2019-10-13T04:29:12.625 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:12.617+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:12.618745+0000) 2019-10-13T04:29:15.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:15.353+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.355063+0000) 2019-10-13T04:29:15.362 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:15.358+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.359651+0000) 2019-10-13T04:29:15.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:15.358+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.360381+0000) 2019-10-13T04:29:15.363 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:15.359+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.360759+0000) 2019-10-13T04:29:15.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:15.359+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.360494+0000) 2019-10-13T04:29:15.363 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:15.359+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.360949+0000) 2019-10-13T04:29:15.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:15.359+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.361085+0000) 2019-10-13T04:29:15.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:15.355+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.357513+0000) 2019-10-13T04:29:15.364 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:15.360+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:15.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:15.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:15.868430+0000) 2019-10-13T04:29:17.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:17.361+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:17.361557+0000) 2019-10-13T04:29:18.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:18.152+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:18.153163+0000) 2019-10-13T04:29:19.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:19.436+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:19.436880+0000) 2019-10-13T04:29:20.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:20.363+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.363674+0000) 2019-10-13T04:29:20.366 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:20.358+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.359387+0000) 2019-10-13T04:29:20.366 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:20.363+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.363948+0000) 2019-10-13T04:29:20.366 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:20.363+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.364288+0000) 2019-10-13T04:29:20.367 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:20.363+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.364351+0000) 2019-10-13T04:29:20.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:20.364+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.364924+0000) 2019-10-13T04:29:20.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:20.364+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.365165+0000) 2019-10-13T04:29:20.368 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:20.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56753 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:20.368 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:20.361+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:20.361899+0000) 2019-10-13T04:29:21.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:21.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:21.744603+0000) 2019-10-13T04:29:21.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:21.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:21.744664+0000) 2019-10-13T04:29:21.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:21.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:21.744630+0000) 2019-10-13T04:29:22.275 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:29:22.275 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:22.279 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:29:22.279 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:22.625 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:22.618+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:22.618935+0000) 2019-10-13T04:29:25.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:25.367+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.367822+0000) 2019-10-13T04:29:25.370 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:25.362+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.363474+0000) 2019-10-13T04:29:25.371 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:25.368+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.368643+0000) 2019-10-13T04:29:25.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:25.368+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.368539+0000) 2019-10-13T04:29:25.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:25.368+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.368737+0000) 2019-10-13T04:29:25.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:25.368+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.369058+0000) 2019-10-13T04:29:25.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:25.368+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.369274+0000) 2019-10-13T04:29:25.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:25.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:25.373 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:25.364+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.365558+0000) 2019-10-13T04:29:25.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:25.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:25.868576+0000) 2019-10-13T04:29:27.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:27.361+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:27.361728+0000) 2019-10-13T04:29:28.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:28.153+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:28.153373+0000) 2019-10-13T04:29:29.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:29.439+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:29.439948+0000) 2019-10-13T04:29:30.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:30.371+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.371764+0000) 2019-10-13T04:29:30.374 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:30.366+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.367319+0000) 2019-10-13T04:29:30.374 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:30.371+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.372291+0000) 2019-10-13T04:29:30.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:30.372+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.372461+0000) 2019-10-13T04:29:30.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:30.372+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.372683+0000) 2019-10-13T04:29:30.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:30.372+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.372800+0000) 2019-10-13T04:29:30.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:30.373+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.373423+0000) 2019-10-13T04:29:30.376 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:30.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:30.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:30.369+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:30.370300+0000) 2019-10-13T04:29:31.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:31.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:31.744806+0000) 2019-10-13T04:29:31.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:31.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:31.744820+0000) 2019-10-13T04:29:31.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:31.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:31.744843+0000) 2019-10-13T04:29:32.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:32.618+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:32.619140+0000) 2019-10-13T04:29:35.378 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:35.375+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.375815+0000) 2019-10-13T04:29:35.378 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:35.375+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.375950+0000) 2019-10-13T04:29:35.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:35.375+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.376284+0000) 2019-10-13T04:29:35.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:35.376+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.376482+0000) 2019-10-13T04:29:35.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:35.371+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.372110+0000) 2019-10-13T04:29:35.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:35.376+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.377156+0000) 2019-10-13T04:29:35.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:35.377+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.377445+0000) 2019-10-13T04:29:35.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:35.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56845 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:35.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:35.373+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.374350+0000) 2019-10-13T04:29:35.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:35.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:35.868717+0000) 2019-10-13T04:29:37.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:37.360+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:37.361887+0000) 2019-10-13T04:29:38.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:38.152+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:38.153606+0000) 2019-10-13T04:29:39.449 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:39.442+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:39.442963+0000) 2019-10-13T04:29:40.384 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:40.375+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.376288+0000) 2019-10-13T04:29:40.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:40.378+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.380092+0000) 2019-10-13T04:29:40.384 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:40.379+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.380400+0000) 2019-10-13T04:29:40.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:40.379+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.380466+0000) 2019-10-13T04:29:40.385 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:40.379+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.380629+0000) 2019-10-13T04:29:40.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:40.379+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.380652+0000) 2019-10-13T04:29:40.385 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:40.379+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.381162+0000) 2019-10-13T04:29:40.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:40.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56877 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:40.386 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:40.377+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:40.377803+0000) 2019-10-13T04:29:41.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:41.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:41.745022+0000) 2019-10-13T04:29:41.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:41.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:41.745025+0000) 2019-10-13T04:29:41.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:41.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:41.745074+0000) 2019-10-13T04:29:42.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:42.618+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:42.619290+0000) 2019-10-13T04:29:45.385 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:45.378+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.378820+0000) 2019-10-13T04:29:45.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:45.382+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.383821+0000) 2019-10-13T04:29:45.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:45.382+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.383982+0000) 2019-10-13T04:29:45.386 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:45.382+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.384091+0000) 2019-10-13T04:29:45.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:45.382+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.384309+0000) 2019-10-13T04:29:45.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:45.383+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.384554+0000) 2019-10-13T04:29:45.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:45.383+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.385017+0000) 2019-10-13T04:29:45.388 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:45.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56908 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:45.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:45.381+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.382524+0000) 2019-10-13T04:29:45.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:45.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:45.868854+0000) 2019-10-13T04:29:47.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:47.360+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:47.362074+0000) 2019-10-13T04:29:48.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:48.153+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:48.153802+0000) 2019-10-13T04:29:49.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:49.445+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:49.445338+0000) 2019-10-13T04:29:50.389 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:50.386+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.387720+0000) 2019-10-13T04:29:50.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:50.386+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.387883+0000) 2019-10-13T04:29:50.390 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:50.386+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.388051+0000) 2019-10-13T04:29:50.390 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:50.386+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.388314+0000) 2019-10-13T04:29:50.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:50.382+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.383706+0000) 2019-10-13T04:29:50.391 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:50.387+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.388680+0000) 2019-10-13T04:29:50.391 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:50.387+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.388944+0000) 2019-10-13T04:29:50.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:50.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56938 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:50.392 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:50.385+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:50.386549+0000) 2019-10-13T04:29:51.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:51.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:51.745229+0000) 2019-10-13T04:29:51.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:51.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:51.745215+0000) 2019-10-13T04:29:51.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:51.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:51.745213+0000) 2019-10-13T04:29:52.416 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:29:52.417 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:52.422 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:29:52.422 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:52.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:52.619+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:52.619439+0000) 2019-10-13T04:29:55.397 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:29:55.387+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.387921+0000) 2019-10-13T04:29:55.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:55.388+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.389203+0000) 2019-10-13T04:29:55.397 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:29:55.390+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.391454+0000) 2019-10-13T04:29:55.397 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:55.390+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.391656+0000) 2019-10-13T04:29:55.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:55.390+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.392022+0000) 2019-10-13T04:29:55.398 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:29:55.390+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.392299+0000) 2019-10-13T04:29:55.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:55.391+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.392514+0000) 2019-10-13T04:29:55.398 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:29:55.391+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.392983+0000) 2019-10-13T04:29:55.398 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:29:55.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 56969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:29:55.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:29:55.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:55.869020+0000) 2019-10-13T04:29:57.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:29:57.361+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:57.362229+0000) 2019-10-13T04:29:58.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:29:58.153+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:58.153976+0000) 2019-10-13T04:29:59.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:29:59.446+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:29:59.447760+0000) 2019-10-13T04:30:00.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:00.394+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.396231+0000) 2019-10-13T04:30:00.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:00.395+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.396452+0000) 2019-10-13T04:30:00.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:00.395+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.396607+0000) 2019-10-13T04:30:00.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:00.390+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.392087+0000) 2019-10-13T04:30:00.400 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:00.395+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.397098+0000) 2019-10-13T04:30:00.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:00.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:00.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:00.395+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.397339+0000) 2019-10-13T04:30:00.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:00.396+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.397970+0000) 2019-10-13T04:30:00.402 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:00.394+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:00.395951+0000) 2019-10-13T04:30:01.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:01.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:01.745439+0000) 2019-10-13T04:30:01.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:01.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:01.745436+0000) 2019-10-13T04:30:01.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:01.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:01.745435+0000) 2019-10-13T04:30:02.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:02.618+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:02.619606+0000) 2019-10-13T04:30:05.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:05.398+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.398652+0000) 2019-10-13T04:30:05.402 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:05.398+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.398923+0000) 2019-10-13T04:30:05.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:05.398+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.399167+0000) 2019-10-13T04:30:05.403 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:05.392+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.394721+0000) 2019-10-13T04:30:05.403 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:05.394+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.396115+0000) 2019-10-13T04:30:05.403 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:05.398+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.399330+0000) 2019-10-13T04:30:05.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:05.399+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.399776+0000) 2019-10-13T04:30:05.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:05.399+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.399944+0000) 2019-10-13T04:30:05.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:05.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57032 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:05.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:05.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:05.869220+0000) 2019-10-13T04:30:07.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:07.362+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:07.362407+0000) 2019-10-13T04:30:08.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:08.153+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:08.154181+0000) 2019-10-13T04:30:09.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:09.449+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:09.450192+0000) 2019-10-13T04:30:10.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:10.402+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.402736+0000) 2019-10-13T04:30:10.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:10.402+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.402517+0000) 2019-10-13T04:30:10.406 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:10.398+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.398795+0000) 2019-10-13T04:30:10.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:10.403+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.403438+0000) 2019-10-13T04:30:10.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:10.403+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.403815+0000) 2019-10-13T04:30:10.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:10.403+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.404086+0000) 2019-10-13T04:30:10.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:10.402+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.403331+0000) 2019-10-13T04:30:10.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:10.399+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:10.399738+0000) 2019-10-13T04:30:10.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:10.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57062 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:11.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:11.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:11.745639+0000) 2019-10-13T04:30:11.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:11.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:11.745642+0000) 2019-10-13T04:30:11.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:11.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:11.745664+0000) 2019-10-13T04:30:12.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:12.619+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:12.619791+0000) 2019-10-13T04:30:15.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:15.401+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.401913+0000) 2019-10-13T04:30:15.409 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:15.406+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.406974+0000) 2019-10-13T04:30:15.409 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:15.406+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.407178+0000) 2019-10-13T04:30:15.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:15.406+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.407356+0000) 2019-10-13T04:30:15.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:15.407+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.407745+0000) 2019-10-13T04:30:15.410 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:15.407+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.407850+0000) 2019-10-13T04:30:15.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:15.407+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.407996+0000) 2019-10-13T04:30:15.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:15.408+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:15.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:15.405+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.405840+0000) 2019-10-13T04:30:15.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:15.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:15.869382+0000) 2019-10-13T04:30:17.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:17.362+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:17.362529+0000) 2019-10-13T04:30:18.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:18.154+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:18.154340+0000) 2019-10-13T04:30:19.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:19.451+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:19.452623+0000) 2019-10-13T04:30:20.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:20.410+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.410921+0000) 2019-10-13T04:30:20.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:20.410+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.411341+0000) 2019-10-13T04:30:20.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:20.411+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.411728+0000) 2019-10-13T04:30:20.414 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:20.411+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.411973+0000) 2019-10-13T04:30:20.414 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:20.407+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.408215+0000) 2019-10-13T04:30:20.415 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:20.412+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.413175+0000) 2019-10-13T04:30:20.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:20.412+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.413379+0000) 2019-10-13T04:30:20.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:20.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:20.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:20.407+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:20.408483+0000) 2019-10-13T04:30:21.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:21.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:21.745850+0000) 2019-10-13T04:30:21.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:21.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:21.745846+0000) 2019-10-13T04:30:21.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:21.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:21.745844+0000) 2019-10-13T04:30:22.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:22.619+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:22.619946+0000) 2019-10-13T04:30:22.628 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:30:22.628 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:22.633 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:30:22.633 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:25.416 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:25.413+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.414977+0000) 2019-10-13T04:30:25.418 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:25.410+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.410975+0000) 2019-10-13T04:30:25.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:25.414+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.415408+0000) 2019-10-13T04:30:25.418 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:25.414+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.415600+0000) 2019-10-13T04:30:25.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:25.414+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.415897+0000) 2019-10-13T04:30:25.418 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:25.414+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.416095+0000) 2019-10-13T04:30:25.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:25.414+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.416296+0000) 2019-10-13T04:30:25.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:25.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:25.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:25.411+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.412717+0000) 2019-10-13T04:30:25.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:25.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:25.869555+0000) 2019-10-13T04:30:27.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:27.361+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:27.362705+0000) 2019-10-13T04:30:28.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:28.153+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:28.154547+0000) 2019-10-13T04:30:29.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:29.454+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:29.455148+0000) 2019-10-13T04:30:30.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:30.413+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.414125+0000) 2019-10-13T04:30:30.422 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:30.417+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.419108+0000) 2019-10-13T04:30:30.423 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:30.417+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.419320+0000) 2019-10-13T04:30:30.423 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:30.418+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.419436+0000) 2019-10-13T04:30:30.423 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:30.418+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.419604+0000) 2019-10-13T04:30:30.423 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:30.418+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.419855+0000) 2019-10-13T04:30:30.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:30.418+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.420064+0000) 2019-10-13T04:30:30.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:30.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:30.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:30.417+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:30.418154+0000) 2019-10-13T04:30:31.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:31.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:31.746057+0000) 2019-10-13T04:30:31.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:31.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:31.746057+0000) 2019-10-13T04:30:31.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:31.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:31.746059+0000) 2019-10-13T04:30:32.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:32.619+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:32.620155+0000) 2019-10-13T04:30:35.424 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:35.417+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.418090+0000) 2019-10-13T04:30:35.426 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:35.421+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.423011+0000) 2019-10-13T04:30:35.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:35.421+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.423271+0000) 2019-10-13T04:30:35.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:35.422+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.423487+0000) 2019-10-13T04:30:35.427 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:35.422+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.423765+0000) 2019-10-13T04:30:35.428 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:35.422+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.423877+0000) 2019-10-13T04:30:35.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:35.422+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.424059+0000) 2019-10-13T04:30:35.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:35.420+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.421671+0000) 2019-10-13T04:30:35.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:35.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:35.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:35.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:35.869715+0000) 2019-10-13T04:30:37.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:37.362+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:37.362900+0000) 2019-10-13T04:30:38.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:38.154+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:38.154739+0000) 2019-10-13T04:30:39.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:39.457+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:39.457455+0000) 2019-10-13T04:30:40.429 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:40.425+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.427195+0000) 2019-10-13T04:30:40.429 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:40.425+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.427362+0000) 2019-10-13T04:30:40.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:40.426+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.427591+0000) 2019-10-13T04:30:40.430 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:40.422+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.423144+0000) 2019-10-13T04:30:40.430 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:40.426+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.427831+0000) 2019-10-13T04:30:40.431 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:40.427+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.428415+0000) 2019-10-13T04:30:40.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:40.427+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.428561+0000) 2019-10-13T04:30:40.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:40.428+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:40.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:40.424+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:40.425461+0000) 2019-10-13T04:30:41.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:41.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:41.746281+0000) 2019-10-13T04:30:41.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:41.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:41.746274+0000) 2019-10-13T04:30:41.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:41.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:41.746275+0000) 2019-10-13T04:30:42.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:42.620+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:42.620320+0000) 2019-10-13T04:30:45.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:45.429+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.431324+0000) 2019-10-13T04:30:45.434 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:45.430+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.431876+0000) 2019-10-13T04:30:45.434 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:45.430+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.432089+0000) 2019-10-13T04:30:45.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:45.430+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.432160+0000) 2019-10-13T04:30:45.434 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:45.431+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.432557+0000) 2019-10-13T04:30:45.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:45.426+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.427594+0000) 2019-10-13T04:30:45.435 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:45.428+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.428794+0000) 2019-10-13T04:30:45.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:45.431+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.432898+0000) 2019-10-13T04:30:45.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:45.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57280 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:45.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:45.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:45.869899+0000) 2019-10-13T04:30:47.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:47.362+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:47.363065+0000) 2019-10-13T04:30:48.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:48.154+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:48.154896+0000) 2019-10-13T04:30:49.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:49.458+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:49.459760+0000) 2019-10-13T04:30:50.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:50.435+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.435450+0000) 2019-10-13T04:30:50.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:50.435+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.435571+0000) 2019-10-13T04:30:50.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:50.435+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.436099+0000) 2019-10-13T04:30:50.438 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:50.435+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.436277+0000) 2019-10-13T04:30:50.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:50.435+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.436337+0000) 2019-10-13T04:30:50.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:50.430+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.431773+0000) 2019-10-13T04:30:50.440 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:50.431+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.433148+0000) 2019-10-13T04:30:50.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:50.436+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:50.436688+0000) 2019-10-13T04:30:50.440 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:50.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:51.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:51.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:51.746427+0000) 2019-10-13T04:30:51.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:51.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:51.746458+0000) 2019-10-13T04:30:51.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:51.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:51.746468+0000) 2019-10-13T04:30:52.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:52.619+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:52.620525+0000) 2019-10-13T04:30:52.842 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:30:52.842 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:52.846 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:30:52.846 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:55.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:55.439+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.439681+0000) 2019-10-13T04:30:55.442 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:55.439+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.439975+0000) 2019-10-13T04:30:55.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:30:55.434+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.435837+0000) 2019-10-13T04:30:55.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:30:55.439+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.440263+0000) 2019-10-13T04:30:55.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:30:55.440+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.440690+0000) 2019-10-13T04:30:55.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:30:55.440+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.440822+0000) 2019-10-13T04:30:55.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:55.440+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.440991+0000) 2019-10-13T04:30:55.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:55.435+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.437562+0000) 2019-10-13T04:30:55.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:30:55.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57341 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:30:55.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:30:55.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:55.870023+0000) 2019-10-13T04:30:57.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:30:57.362+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:57.363237+0000) 2019-10-13T04:30:58.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:30:58.154+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:58.155090+0000) 2019-10-13T04:30:59.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:30:59.461+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:30:59.462039+0000) 2019-10-13T04:31:00.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:00.437+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.439612+0000) 2019-10-13T04:31:00.459 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:00.443+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.443594+0000) 2019-10-13T04:31:00.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:00.443+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.443922+0000) 2019-10-13T04:31:00.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:00.443+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.444178+0000) 2019-10-13T04:31:00.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:00.439+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.441554+0000) 2019-10-13T04:31:00.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:00.444+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.444567+0000) 2019-10-13T04:31:00.467 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:00.444+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.444795+0000) 2019-10-13T04:31:00.467 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:00.444+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:00.445089+0000) 2019-10-13T04:31:00.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:00.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57373 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:01.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:01.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:01.746626+0000) 2019-10-13T04:31:01.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:01.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:01.746609+0000) 2019-10-13T04:31:01.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:01.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:01.746608+0000) 2019-10-13T04:31:02.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:02.619+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:02.620703+0000) 2019-10-13T04:31:05.449 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:05.447+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.447539+0000) 2019-10-13T04:31:05.450 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:05.447+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.447677+0000) 2019-10-13T04:31:05.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:05.447+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.448066+0000) 2019-10-13T04:31:05.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:05.447+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.448227+0000) 2019-10-13T04:31:05.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:05.448+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.448954+0000) 2019-10-13T04:31:05.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:05.448+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.449210+0000) 2019-10-13T04:31:05.454 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:05.443+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.444042+0000) 2019-10-13T04:31:05.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:05.444+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.445234+0000) 2019-10-13T04:31:05.454 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:05.449+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:05.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:05.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:05.870171+0000) 2019-10-13T04:31:07.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:07.361+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:07.363339+0000) 2019-10-13T04:31:08.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:08.154+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:08.155286+0000) 2019-10-13T04:31:09.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:09.463+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:09.464611+0000) 2019-10-13T04:31:10.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:10.445+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.446675+0000) 2019-10-13T04:31:10.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:10.450+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.451609+0000) 2019-10-13T04:31:10.454 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:10.450+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.451917+0000) 2019-10-13T04:31:10.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:10.450+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.452097+0000) 2019-10-13T04:31:10.454 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:10.451+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.452389+0000) 2019-10-13T04:31:10.455 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:10.451+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.452564+0000) 2019-10-13T04:31:10.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:10.451+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.452812+0000) 2019-10-13T04:31:10.455 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:10.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57434 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:10.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:10.449+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:10.450091+0000) 2019-10-13T04:31:11.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:11.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:11.746826+0000) 2019-10-13T04:31:11.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:11.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:11.746828+0000) 2019-10-13T04:31:11.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:11.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:11.746827+0000) 2019-10-13T04:31:12.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:12.620+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:12.620864+0000) 2019-10-13T04:31:15.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:15.453+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.455340+0000) 2019-10-13T04:31:15.458 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:15.454+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.455622+0000) 2019-10-13T04:31:15.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:15.450+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.451716+0000) 2019-10-13T04:31:15.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:15.454+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.456046+0000) 2019-10-13T04:31:15.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:15.455+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.456668+0000) 2019-10-13T04:31:15.461 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:15.455+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.456821+0000) 2019-10-13T04:31:15.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:15.455+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.457053+0000) 2019-10-13T04:31:15.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:15.452+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.453181+0000) 2019-10-13T04:31:15.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:15.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:15.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:15.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:15.870330+0000) 2019-10-13T04:31:17.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:17.362+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:17.363559+0000) 2019-10-13T04:31:18.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:18.154+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:18.155479+0000) 2019-10-13T04:31:19.473 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:19.466+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:19.466935+0000) 2019-10-13T04:31:20.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:20.454+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.454891+0000) 2019-10-13T04:31:20.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:20.457+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.459265+0000) 2019-10-13T04:31:20.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:20.458+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.459538+0000) 2019-10-13T04:31:20.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:20.458+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.459994+0000) 2019-10-13T04:31:20.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:20.459+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.460505+0000) 2019-10-13T04:31:20.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:20.459+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.460693+0000) 2019-10-13T04:31:20.467 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:20.459+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.460939+0000) 2019-10-13T04:31:20.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:20.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:20.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:20.457+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:20.457780+0000) 2019-10-13T04:31:21.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:21.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:21.746994+0000) 2019-10-13T04:31:21.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:21.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:21.746987+0000) 2019-10-13T04:31:21.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:21.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:21.746993+0000) 2019-10-13T04:31:22.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:22.620+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:22.621017+0000) 2019-10-13T04:31:23.023 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:31:23.023 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:23.029 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:31:23.029 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:25.467 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:25.458+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.459286+0000) 2019-10-13T04:31:25.467 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:25.461+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.463313+0000) 2019-10-13T04:31:25.467 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:25.462+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.463846+0000) 2019-10-13T04:31:25.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:25.462+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.464194+0000) 2019-10-13T04:31:25.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:25.463+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.464590+0000) 2019-10-13T04:31:25.468 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:25.463+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.464870+0000) 2019-10-13T04:31:25.468 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:25.463+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.464984+0000) 2019-10-13T04:31:25.469 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:25.461+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.462160+0000) 2019-10-13T04:31:25.469 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:25.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:25.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:25.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:25.870495+0000) 2019-10-13T04:31:27.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:27.363+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:27.363720+0000) 2019-10-13T04:31:28.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:28.155+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:28.155653+0000) 2019-10-13T04:31:29.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:29.469+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:29.469325+0000) 2019-10-13T04:31:30.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:30.466+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.467812+0000) 2019-10-13T04:31:30.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:30.466+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.467962+0000) 2019-10-13T04:31:30.470 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:30.466+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.468034+0000) 2019-10-13T04:31:30.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:30.466+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.468229+0000) 2019-10-13T04:31:30.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:30.463+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.463929+0000) 2019-10-13T04:31:30.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:30.467+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.468872+0000) 2019-10-13T04:31:30.477 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:30.467+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.469035+0000) 2019-10-13T04:31:30.477 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:30.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:30.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:30.465+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:30.465881+0000) 2019-10-13T04:31:31.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:31.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:31.747210+0000) 2019-10-13T04:31:31.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:31.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:31.747206+0000) 2019-10-13T04:31:31.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:31.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:31.747207+0000) 2019-10-13T04:31:32.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:32.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:32.621151+0000) 2019-10-13T04:31:35.474 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:35.471+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.471871+0000) 2019-10-13T04:31:35.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:35.471+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.471966+0000) 2019-10-13T04:31:35.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:35.471+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.472209+0000) 2019-10-13T04:31:35.475 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:35.467+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.467853+0000) 2019-10-13T04:31:35.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:35.472+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.472993+0000) 2019-10-13T04:31:35.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:35.472+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.473354+0000) 2019-10-13T04:31:35.476 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:35.473+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.473535+0000) 2019-10-13T04:31:35.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:35.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:35.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:35.469+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.470579+0000) 2019-10-13T04:31:35.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:35.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:35.870660+0000) 2019-10-13T04:31:37.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:37.363+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:37.363895+0000) 2019-10-13T04:31:38.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:38.155+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:38.155858+0000) 2019-10-13T04:31:39.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:39.470+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:39.471401+0000) 2019-10-13T04:31:40.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:40.475+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.476325+0000) 2019-10-13T04:31:40.478 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:40.476+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.476652+0000) 2019-10-13T04:31:40.479 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:40.476+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.476816+0000) 2019-10-13T04:31:40.479 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:40.470+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.472334+0000) 2019-10-13T04:31:40.480 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:40.476+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.477224+0000) 2019-10-13T04:31:40.480 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:40.477+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.477426+0000) 2019-10-13T04:31:40.480 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:40.477+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.477928+0000) 2019-10-13T04:31:40.481 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:40.471+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:40.473418+0000) 2019-10-13T04:31:40.481 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:40.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:41.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:41.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:41.747338+0000) 2019-10-13T04:31:41.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:41.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:41.747392+0000) 2019-10-13T04:31:41.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:41.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:41.747426+0000) 2019-10-13T04:31:42.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:42.620+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:42.621287+0000) 2019-10-13T04:31:45.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:45.473+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.475490+0000) 2019-10-13T04:31:45.483 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:45.480+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.480407+0000) 2019-10-13T04:31:45.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:45.480+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.480667+0000) 2019-10-13T04:31:45.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:45.480+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.480843+0000) 2019-10-13T04:31:45.484 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:45.480+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.480997+0000) 2019-10-13T04:31:45.484 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:45.480+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.481242+0000) 2019-10-13T04:31:45.484 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:45.480+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.481328+0000) 2019-10-13T04:31:45.485 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:45.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:45.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:45.477+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.478889+0000) 2019-10-13T04:31:45.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:45.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:45.870870+0000) 2019-10-13T04:31:47.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:47.363+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:47.364064+0000) 2019-10-13T04:31:48.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:48.155+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:48.156058+0000) 2019-10-13T04:31:49.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:49.473+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:49.473957+0000) 2019-10-13T04:31:50.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:50.483+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.484304+0000) 2019-10-13T04:31:50.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:50.478+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.479988+0000) 2019-10-13T04:31:50.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:50.484+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.484845+0000) 2019-10-13T04:31:50.487 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:50.484+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.485051+0000) 2019-10-13T04:31:50.488 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:50.484+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.485280+0000) 2019-10-13T04:31:50.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:50.485+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.485420+0000) 2019-10-13T04:31:50.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:50.485+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.485753+0000) 2019-10-13T04:31:50.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:50.480+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:50.481990+0000) 2019-10-13T04:31:50.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:50.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:51.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:51.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:51.747549+0000) 2019-10-13T04:31:51.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:51.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:51.747532+0000) 2019-10-13T04:31:51.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:51.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:51.747595+0000) 2019-10-13T04:31:52.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:52.620+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:52.621441+0000) 2019-10-13T04:31:53.188 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:31:53.189 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:53.193 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:31:53.193 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:55.490 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:31:55.483+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.483765+0000) 2019-10-13T04:31:55.492 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:31:55.487+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.488728+0000) 2019-10-13T04:31:55.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:55.487+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.488890+0000) 2019-10-13T04:31:55.492 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:55.487+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.489127+0000) 2019-10-13T04:31:55.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:31:55.487+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.489194+0000) 2019-10-13T04:31:55.493 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:31:55.488+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.489554+0000) 2019-10-13T04:31:55.493 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:55.488+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.489653+0000) 2019-10-13T04:31:55.494 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:31:55.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:31:55.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:55.486+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.487651+0000) 2019-10-13T04:31:55.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:31:55.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:55.871036+0000) 2019-10-13T04:31:57.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:31:57.363+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:57.364278+0000) 2019-10-13T04:31:58.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:31:58.155+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:58.156249+0000) 2019-10-13T04:31:59.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:31:59.475+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:31:59.476261+0000) 2019-10-13T04:32:00.494 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:00.487+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.487885+0000) 2019-10-13T04:32:00.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:00.491+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.492800+0000) 2019-10-13T04:32:00.495 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:00.491+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.492962+0000) 2019-10-13T04:32:00.496 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:00.491+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.493207+0000) 2019-10-13T04:32:00.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:00.492+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.493560+0000) 2019-10-13T04:32:00.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:00.492+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.493707+0000) 2019-10-13T04:32:00.496 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:00.492+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.493908+0000) 2019-10-13T04:32:00.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:00.489+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:00.490550+0000) 2019-10-13T04:32:00.497 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:00.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:01.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:01.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:01.747726+0000) 2019-10-13T04:32:01.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:01.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:01.747762+0000) 2019-10-13T04:32:01.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:01.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:01.747803+0000) 2019-10-13T04:32:02.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:02.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:02.621596+0000) 2019-10-13T04:32:05.498 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:05.491+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.491787+0000) 2019-10-13T04:32:05.500 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:05.495+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.496776+0000) 2019-10-13T04:32:05.500 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:05.495+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.497180+0000) 2019-10-13T04:32:05.500 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:05.495+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.497305+0000) 2019-10-13T04:32:05.500 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:05.496+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.497576+0000) 2019-10-13T04:32:05.500 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:05.496+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.497591+0000) 2019-10-13T04:32:05.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:05.496+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.497753+0000) 2019-10-13T04:32:05.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:05.493+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.494215+0000) 2019-10-13T04:32:05.501 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:05.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:05.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:05.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:05.871180+0000) 2019-10-13T04:32:07.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:07.363+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:07.364466+0000) 2019-10-13T04:32:08.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:08.155+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:08.156409+0000) 2019-10-13T04:32:09.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:09.478+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:09.478572+0000) 2019-10-13T04:32:10.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:10.495+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.496119+0000) 2019-10-13T04:32:10.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:10.499+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.500986+0000) 2019-10-13T04:32:10.503 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:10.499+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.501122+0000) 2019-10-13T04:32:10.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:10.499+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.501334+0000) 2019-10-13T04:32:10.504 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:10.500+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.501873+0000) 2019-10-13T04:32:10.504 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:10.500+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.501993+0000) 2019-10-13T04:32:10.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:10.500+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.501636+0000) 2019-10-13T04:32:10.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:10.501+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:10.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:10.498+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:10.499588+0000) 2019-10-13T04:32:11.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:11.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:11.747930+0000) 2019-10-13T04:32:11.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:11.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:11.747968+0000) 2019-10-13T04:32:11.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:11.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:11.747930+0000) 2019-10-13T04:32:12.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:12.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:12.621759+0000) 2019-10-13T04:32:15.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:15.503+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.504715+0000) 2019-10-13T04:32:15.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:15.503+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.504981+0000) 2019-10-13T04:32:15.508 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:15.503+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.505300+0000) 2019-10-13T04:32:15.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:15.503+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.505205+0000) 2019-10-13T04:32:15.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:15.504+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.505618+0000) 2019-10-13T04:32:15.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:15.500+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.501150+0000) 2019-10-13T04:32:15.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:15.504+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.506195+0000) 2019-10-13T04:32:15.510 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:15.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57838 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:15.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:15.502+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.503375+0000) 2019-10-13T04:32:15.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:15.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:15.871350+0000) 2019-10-13T04:32:17.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:17.364+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:17.364650+0000) 2019-10-13T04:32:18.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:18.156+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:18.156571+0000) 2019-10-13T04:32:19.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:19.480+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:19.480979+0000) 2019-10-13T04:32:20.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:20.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.508790+0000) 2019-10-13T04:32:20.511 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:20.508+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.509107+0000) 2019-10-13T04:32:20.512 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:20.503+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.504225+0000) 2019-10-13T04:32:20.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:20.508+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.509282+0000) 2019-10-13T04:32:20.512 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:20.509+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.509928+0000) 2019-10-13T04:32:20.513 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:20.509+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.509805+0000) 2019-10-13T04:32:20.513 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:20.509+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.510125+0000) 2019-10-13T04:32:20.514 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:20.506+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:20.506917+0000) 2019-10-13T04:32:20.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:20.510+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:21.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:21.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:21.748153+0000) 2019-10-13T04:32:21.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:21.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:21.748157+0000) 2019-10-13T04:32:21.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:21.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:21.748152+0000) 2019-10-13T04:32:22.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:22.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:22.621945+0000) 2019-10-13T04:32:23.350 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:32:23.350 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:23.354 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:32:23.355 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:25.515 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:25.512+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.512680+0000) 2019-10-13T04:32:25.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:25.512+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.513154+0000) 2019-10-13T04:32:25.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:25.512+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.513352+0000) 2019-10-13T04:32:25.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:25.513+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.513546+0000) 2019-10-13T04:32:25.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:25.513+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.513787+0000) 2019-10-13T04:32:25.517 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:25.508+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.508986+0000) 2019-10-13T04:32:25.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:25.509+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.510375+0000) 2019-10-13T04:32:25.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:25.513+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.514117+0000) 2019-10-13T04:32:25.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:25.514+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57901 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:25.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:25.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:25.871535+0000) 2019-10-13T04:32:27.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:27.364+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:27.364840+0000) 2019-10-13T04:32:28.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:28.156+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:28.156768+0000) 2019-10-13T04:32:29.490 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:29.483+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:29.483273+0000) 2019-10-13T04:32:30.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:30.516+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.517000+0000) 2019-10-13T04:32:30.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:30.516+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.517227+0000) 2019-10-13T04:32:30.520 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:30.517+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.517539+0000) 2019-10-13T04:32:30.520 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:30.517+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.517683+0000) 2019-10-13T04:32:30.521 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:30.510+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.512684+0000) 2019-10-13T04:32:30.521 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:30.517+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.517892+0000) 2019-10-13T04:32:30.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:30.517+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.518150+0000) 2019-10-13T04:32:30.521 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:30.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:30.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:30.513+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:30.515183+0000) 2019-10-13T04:32:31.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:31.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:31.748338+0000) 2019-10-13T04:32:31.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:31.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:31.748335+0000) 2019-10-13T04:32:31.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:31.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:31.748336+0000) 2019-10-13T04:32:32.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:32.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:32.622125+0000) 2019-10-13T04:32:35.523 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:35.520+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.520903+0000) 2019-10-13T04:32:35.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:35.520+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.521273+0000) 2019-10-13T04:32:35.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:35.520+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.521379+0000) 2019-10-13T04:32:35.524 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:35.521+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.521508+0000) 2019-10-13T04:32:35.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:35.515+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.516957+0000) 2019-10-13T04:32:35.525 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:35.516+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.518169+0000) 2019-10-13T04:32:35.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:35.521+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.521888+0000) 2019-10-13T04:32:35.525 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:35.521+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.521775+0000) 2019-10-13T04:32:35.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:35.522+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:35.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:35.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:35.871668+0000) 2019-10-13T04:32:37.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:37.364+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:37.365019+0000) 2019-10-13T04:32:38.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:38.155+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:38.156966+0000) 2019-10-13T04:32:39.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:39.484+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:39.485658+0000) 2019-10-13T04:32:40.527 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:40.523+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.524779+0000) 2019-10-13T04:32:40.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:40.523+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.524997+0000) 2019-10-13T04:32:40.527 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:40.523+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.524928+0000) 2019-10-13T04:32:40.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:40.523+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.525254+0000) 2019-10-13T04:32:40.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:40.519+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.521272+0000) 2019-10-13T04:32:40.528 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:40.524+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.525611+0000) 2019-10-13T04:32:40.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:40.524+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.526125+0000) 2019-10-13T04:32:40.529 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:40.525+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 57993 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:40.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:40.521+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:40.523624+0000) 2019-10-13T04:32:41.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:41.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:41.748570+0000) 2019-10-13T04:32:41.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:41.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:41.748555+0000) 2019-10-13T04:32:41.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:41.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:41.748582+0000) 2019-10-13T04:32:42.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:42.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:42.622288+0000) 2019-10-13T04:32:45.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:45.526+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.528259+0000) 2019-10-13T04:32:45.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:45.527+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.528710+0000) 2019-10-13T04:32:45.532 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:45.523+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.524230+0000) 2019-10-13T04:32:45.532 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:45.527+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.529052+0000) 2019-10-13T04:32:45.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:45.527+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.529276+0000) 2019-10-13T04:32:45.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:45.528+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.529414+0000) 2019-10-13T04:32:45.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:45.528+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.529679+0000) 2019-10-13T04:32:45.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:45.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:45.533 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:45.525+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.526308+0000) 2019-10-13T04:32:45.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:45.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:45.871830+0000) 2019-10-13T04:32:47.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:47.364+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:47.365241+0000) 2019-10-13T04:32:48.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:48.156+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:48.157196+0000) 2019-10-13T04:32:49.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:49.487+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:49.488089+0000) 2019-10-13T04:32:50.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:50.530+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.531863+0000) 2019-10-13T04:32:50.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:50.530+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.532053+0000) 2019-10-13T04:32:50.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:50.530+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.532349+0000) 2019-10-13T04:32:50.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:50.527+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.527811+0000) 2019-10-13T04:32:50.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:50.531+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.532874+0000) 2019-10-13T04:32:50.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:50.531+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.533050+0000) 2019-10-13T04:32:50.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:50.531+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.533267+0000) 2019-10-13T04:32:50.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:50.529+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:50.530013+0000) 2019-10-13T04:32:50.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:50.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58054 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:51.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:51.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:51.748689+0000) 2019-10-13T04:32:51.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:51.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:51.748756+0000) 2019-10-13T04:32:51.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:51.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:51.748756+0000) 2019-10-13T04:32:52.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:52.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:52.622469+0000) 2019-10-13T04:32:53.553 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:32:53.553 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:53.558 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:32:53.558 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:55.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:32:55.530+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.531716+0000) 2019-10-13T04:32:55.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:55.534+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.536340+0000) 2019-10-13T04:32:55.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:55.535+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.536633+0000) 2019-10-13T04:32:55.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:32:55.535+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.536938+0000) 2019-10-13T04:32:55.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:32:55.535+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.537097+0000) 2019-10-13T04:32:55.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:55.535+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.536993+0000) 2019-10-13T04:32:55.543 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:55.533+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.534609+0000) 2019-10-13T04:32:55.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:32:55.536+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.537673+0000) 2019-10-13T04:32:55.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:32:55.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:32:55.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:32:55.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:55.872007+0000) 2019-10-13T04:32:57.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:32:57.364+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:57.365450+0000) 2019-10-13T04:32:58.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:32:58.156+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:58.157378+0000) 2019-10-13T04:32:59.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:32:59.489+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:32:59.490447+0000) 2019-10-13T04:33:00.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:00.539+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.540428+0000) 2019-10-13T04:33:00.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:00.539+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.540747+0000) 2019-10-13T04:33:00.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:00.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.540988+0000) 2019-10-13T04:33:00.544 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:00.536+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.536893+0000) 2019-10-13T04:33:00.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:00.539+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.541332+0000) 2019-10-13T04:33:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:00.540+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.541437+0000) 2019-10-13T04:33:00.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:00.540+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.541977+0000) 2019-10-13T04:33:00.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:00.537+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:00.538319+0000) 2019-10-13T04:33:00.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:00.541+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58117 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:01.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:01.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:01.748832+0000) 2019-10-13T04:33:01.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:01.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:01.748886+0000) 2019-10-13T04:33:01.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:01.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:01.748910+0000) 2019-10-13T04:33:02.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:02.621+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:02.622659+0000) 2019-10-13T04:33:05.546 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:05.539+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.540053+0000) 2019-10-13T04:33:05.548 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:05.544+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.544876+0000) 2019-10-13T04:33:05.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:05.544+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.545038+0000) 2019-10-13T04:33:05.548 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:05.544+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.545334+0000) 2019-10-13T04:33:05.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:05.545+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.545729+0000) 2019-10-13T04:33:05.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:05.545+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.545941+0000) 2019-10-13T04:33:05.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:05.545+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.546083+0000) 2019-10-13T04:33:05.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:05.541+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.541991+0000) 2019-10-13T04:33:05.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:05.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:05.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:05.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:05.872189+0000) 2019-10-13T04:33:07.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:07.365+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:07.365641+0000) 2019-10-13T04:33:08.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:08.157+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:08.157556+0000) 2019-10-13T04:33:09.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:09.492+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:09.492849+0000) 2019-10-13T04:33:10.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:10.543+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.544224+0000) 2019-10-13T04:33:10.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:10.548+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.549321+0000) 2019-10-13T04:33:10.553 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:10.549+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.549598+0000) 2019-10-13T04:33:10.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:10.549+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.549697+0000) 2019-10-13T04:33:10.553 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:10.549+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.549945+0000) 2019-10-13T04:33:10.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:10.549+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.550135+0000) 2019-10-13T04:33:10.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:10.550+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.550436+0000) 2019-10-13T04:33:10.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:10.546+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:10.546766+0000) 2019-10-13T04:33:10.554 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:10.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:11.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:11.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:11.749057+0000) 2019-10-13T04:33:11.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:11.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:11.749080+0000) 2019-10-13T04:33:11.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:11.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:11.749043+0000) 2019-10-13T04:33:12.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:12.622+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:12.622806+0000) 2019-10-13T04:33:15.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:15.552+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.553374+0000) 2019-10-13T04:33:15.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:15.552+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.553271+0000) 2019-10-13T04:33:15.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:15.553+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.553544+0000) 2019-10-13T04:33:15.556 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:15.553+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.553969+0000) 2019-10-13T04:33:15.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:15.548+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.549101+0000) 2019-10-13T04:33:15.557 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:15.553+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.554282+0000) 2019-10-13T04:33:15.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:15.554+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.554609+0000) 2019-10-13T04:33:15.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:15.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58210 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:15.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:15.551+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.551796+0000) 2019-10-13T04:33:15.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:15.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:15.872373+0000) 2019-10-13T04:33:17.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:17.365+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:17.365844+0000) 2019-10-13T04:33:18.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:18.157+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:18.157756+0000) 2019-10-13T04:33:19.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:19.495+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:19.495346+0000) 2019-10-13T04:33:20.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:20.556+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.556685+0000) 2019-10-13T04:33:20.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:20.556+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.557048+0000) 2019-10-13T04:33:20.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:20.556+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.557315+0000) 2019-10-13T04:33:20.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:20.552+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.553029+0000) 2019-10-13T04:33:20.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:20.557+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.558009+0000) 2019-10-13T04:33:20.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:20.557+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.558336+0000) 2019-10-13T04:33:20.562 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:20.558+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.558548+0000) 2019-10-13T04:33:20.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:20.558+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:20.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:20.554+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:20.555049+0000) 2019-10-13T04:33:21.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:21.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:21.749228+0000) 2019-10-13T04:33:21.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:21.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:21.749230+0000) 2019-10-13T04:33:21.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:21.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:21.749223+0000) 2019-10-13T04:33:22.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:22.622+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:22.623012+0000) 2019-10-13T04:33:23.664 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:33:23.664 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:23.668 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:33:23.669 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:25.562 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:25.559+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.560767+0000) 2019-10-13T04:33:25.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:25.559+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.560864+0000) 2019-10-13T04:33:25.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:25.559+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.561142+0000) 2019-10-13T04:33:25.564 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:25.555+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.557069+0000) 2019-10-13T04:33:25.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:25.560+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.561494+0000) 2019-10-13T04:33:25.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:25.560+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.561637+0000) 2019-10-13T04:33:25.564 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:25.560+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.562241+0000) 2019-10-13T04:33:25.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:25.561+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:25.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:25.558+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.560165+0000) 2019-10-13T04:33:25.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:25.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:25.872489+0000) 2019-10-13T04:33:27.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:27.365+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:27.366033+0000) 2019-10-13T04:33:28.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:28.157+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:28.157974+0000) 2019-10-13T04:33:29.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:29.497+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:29.497890+0000) 2019-10-13T04:33:30.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:30.563+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.565220+0000) 2019-10-13T04:33:30.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:30.563+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.565242+0000) 2019-10-13T04:33:30.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:30.564+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.565517+0000) 2019-10-13T04:33:30.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:30.564+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.565505+0000) 2019-10-13T04:33:30.568 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:30.564+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.565733+0000) 2019-10-13T04:33:30.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:30.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:30.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:30.564+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.565833+0000) 2019-10-13T04:33:30.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:30.559+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.561245+0000) 2019-10-13T04:33:30.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:30.562+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:30.563904+0000) 2019-10-13T04:33:31.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:31.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:31.749391+0000) 2019-10-13T04:33:31.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:31.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:31.749391+0000) 2019-10-13T04:33:31.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:31.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:31.749409+0000) 2019-10-13T04:33:32.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:32.622+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:32.623168+0000) 2019-10-13T04:33:35.570 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:35.562+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.562937+0000) 2019-10-13T04:33:35.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:35.565+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.567242+0000) 2019-10-13T04:33:35.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:35.566+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.567874+0000) 2019-10-13T04:33:35.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:35.566+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.568067+0000) 2019-10-13T04:33:35.571 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:35.566+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.568140+0000) 2019-10-13T04:33:35.571 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:35.566+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.568244+0000) 2019-10-13T04:33:35.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:35.567+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.568435+0000) 2019-10-13T04:33:35.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:35.563+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.564693+0000) 2019-10-13T04:33:35.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:35.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:35.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:35.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:35.872635+0000) 2019-10-13T04:33:37.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:37.365+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:37.366200+0000) 2019-10-13T04:33:38.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:38.157+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:38.158165+0000) 2019-10-13T04:33:39.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:39.499+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:39.500173+0000) 2019-10-13T04:33:40.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:40.570+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.571649+0000) 2019-10-13T04:33:40.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:40.569+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.571288+0000) 2019-10-13T04:33:40.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:40.570+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.571861+0000) 2019-10-13T04:33:40.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:40.570+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.572106+0000) 2019-10-13T04:33:40.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:40.570+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.572373+0000) 2019-10-13T04:33:40.576 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:40.566+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.567537+0000) 2019-10-13T04:33:40.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:40.568+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.569101+0000) 2019-10-13T04:33:40.578 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:40.571+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:40.572608+0000) 2019-10-13T04:33:40.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:40.572+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:41.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:41.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:41.749550+0000) 2019-10-13T04:33:41.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:41.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:41.749561+0000) 2019-10-13T04:33:41.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:41.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:41.749555+0000) 2019-10-13T04:33:42.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:42.622+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:42.623332+0000) 2019-10-13T04:33:45.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:45.573+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.575128+0000) 2019-10-13T04:33:45.577 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:45.574+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.575443+0000) 2019-10-13T04:33:45.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:45.574+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.575595+0000) 2019-10-13T04:33:45.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:45.569+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.570534+0000) 2019-10-13T04:33:45.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:45.574+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.576044+0000) 2019-10-13T04:33:45.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:45.574+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.576340+0000) 2019-10-13T04:33:45.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:45.575+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.576538+0000) 2019-10-13T04:33:45.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:45.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58397 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:45.580 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:45.573+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.574215+0000) 2019-10-13T04:33:45.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:45.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:45.872797+0000) 2019-10-13T04:33:47.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:47.365+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:47.366366+0000) 2019-10-13T04:33:48.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:48.157+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:48.158322+0000) 2019-10-13T04:33:49.509 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:49.502+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:49.502556+0000) 2019-10-13T04:33:50.581 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:50.573+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.574693+0000) 2019-10-13T04:33:50.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:50.579+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.579707+0000) 2019-10-13T04:33:50.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:50.579+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.579956+0000) 2019-10-13T04:33:50.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:50.579+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.580258+0000) 2019-10-13T04:33:50.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:50.580+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.580712+0000) 2019-10-13T04:33:50.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:50.580+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.581354+0000) 2019-10-13T04:33:50.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:50.581+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.581458+0000) 2019-10-13T04:33:50.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:50.581+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:50.584 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:50.577+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:50.578389+0000) 2019-10-13T04:33:51.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:51.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:51.749722+0000) 2019-10-13T04:33:51.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:51.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:51.749782+0000) 2019-10-13T04:33:51.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:51.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:51.749774+0000) 2019-10-13T04:33:52.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:52.623+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:52.623524+0000) 2019-10-13T04:33:53.850 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:33:53.850 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:53.857 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:33:53.857 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:55.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:33:55.578+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.578933+0000) 2019-10-13T04:33:55.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:33:55.583+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.583489+0000) 2019-10-13T04:33:55.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:33:55.583+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.584087+0000) 2019-10-13T04:33:55.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:55.583+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.584349+0000) 2019-10-13T04:33:55.587 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:55.584+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.584530+0000) 2019-10-13T04:33:55.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:55.584+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.584808+0000) 2019-10-13T04:33:55.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:33:55.584+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.585226+0000) 2019-10-13T04:33:55.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:33:55.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:33:55.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:55.581+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.582215+0000) 2019-10-13T04:33:55.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:33:55.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:55.872961+0000) 2019-10-13T04:33:57.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:33:57.366+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:57.366544+0000) 2019-10-13T04:33:58.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:33:58.158+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:58.158497+0000) 2019-10-13T04:33:59.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:33:59.504+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:33:59.504906+0000) 2019-10-13T04:34:00.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:00.587+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.588142+0000) 2019-10-13T04:34:00.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:00.582+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.583705+0000) 2019-10-13T04:34:00.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:00.588+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.588553+0000) 2019-10-13T04:34:00.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:00.588+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.588836+0000) 2019-10-13T04:34:00.591 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:00.588+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.588731+0000) 2019-10-13T04:34:00.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:00.588+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.588961+0000) 2019-10-13T04:34:00.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:00.584+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.585723+0000) 2019-10-13T04:34:00.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:00.589+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:00.589781+0000) 2019-10-13T04:34:00.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:00.589+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:01.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:01.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:01.749904+0000) 2019-10-13T04:34:01.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:01.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:01.749947+0000) 2019-10-13T04:34:01.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:01.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:01.749963+0000) 2019-10-13T04:34:02.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:02.623+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:02.623708+0000) 2019-10-13T04:34:05.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:05.591+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.591826+0000) 2019-10-13T04:34:05.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:05.591+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.592292+0000) 2019-10-13T04:34:05.595 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:05.592+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.592499+0000) 2019-10-13T04:34:05.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:05.592+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.592734+0000) 2019-10-13T04:34:05.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:05.592+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.592880+0000) 2019-10-13T04:34:05.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:05.592+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.593158+0000) 2019-10-13T04:34:05.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:05.587+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.588041+0000) 2019-10-13T04:34:05.597 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:05.589+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.590524+0000) 2019-10-13T04:34:05.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:05.593+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:05.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:05.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:05.873136+0000) 2019-10-13T04:34:07.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:07.366+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:07.366751+0000) 2019-10-13T04:34:08.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:08.157+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:08.158662+0000) 2019-10-13T04:34:09.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:09.507+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:09.507178+0000) 2019-10-13T04:34:10.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:10.594+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.596043+0000) 2019-10-13T04:34:10.598 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:10.594+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.596362+0000) 2019-10-13T04:34:10.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:10.591+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.591968+0000) 2019-10-13T04:34:10.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:10.595+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.596616+0000) 2019-10-13T04:34:10.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:10.595+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.597062+0000) 2019-10-13T04:34:10.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:10.595+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.597269+0000) 2019-10-13T04:34:10.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:10.596+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.597418+0000) 2019-10-13T04:34:10.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:10.592+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:10.593670+0000) 2019-10-13T04:34:10.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:10.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58550 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:11.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:11.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:11.750086+0000) 2019-10-13T04:34:11.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:11.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:11.750178+0000) 2019-10-13T04:34:11.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:11.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:11.750179+0000) 2019-10-13T04:34:12.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:12.622+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:12.623877+0000) 2019-10-13T04:34:15.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:15.598+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.600028+0000) 2019-10-13T04:34:15.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:15.598+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.600131+0000) 2019-10-13T04:34:15.602 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:15.598+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.600340+0000) 2019-10-13T04:34:15.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:15.599+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.600575+0000) 2019-10-13T04:34:15.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:15.594+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.596087+0000) 2019-10-13T04:34:15.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:15.599+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.600913+0000) 2019-10-13T04:34:15.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:15.600+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.601912+0000) 2019-10-13T04:34:15.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:15.596+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.597916+0000) 2019-10-13T04:34:15.605 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:15.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58582 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:15.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:15.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:15.873308+0000) 2019-10-13T04:34:17.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:17.365+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:17.366931+0000) 2019-10-13T04:34:18.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:18.158+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:18.158853+0000) 2019-10-13T04:34:19.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:19.508+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:19.509553+0000) 2019-10-13T04:34:20.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:20.597+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.599418+0000) 2019-10-13T04:34:20.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:20.603+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.604390+0000) 2019-10-13T04:34:20.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:20.603+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.604665+0000) 2019-10-13T04:34:20.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:20.603+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.605072+0000) 2019-10-13T04:34:20.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:20.603+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.605371+0000) 2019-10-13T04:34:20.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:20.604+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.605502+0000) 2019-10-13T04:34:20.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:20.604+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.606030+0000) 2019-10-13T04:34:20.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:20.605+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:20.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:20.600+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:20.602158+0000) 2019-10-13T04:34:21.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:21.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:21.750290+0000) 2019-10-13T04:34:21.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:21.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:21.750388+0000) 2019-10-13T04:34:21.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:21.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:21.750361+0000) 2019-10-13T04:34:22.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:22.622+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:22.623990+0000) 2019-10-13T04:34:24.005 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:34:24.005 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:24.009 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:34:24.009 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:25.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:25.606+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.608178+0000) 2019-10-13T04:34:25.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:25.603+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.604206+0000) 2019-10-13T04:34:25.612 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:25.607+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.608662+0000) 2019-10-13T04:34:25.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:25.607+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.608827+0000) 2019-10-13T04:34:25.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:25.607+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.609090+0000) 2019-10-13T04:34:25.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:25.607+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.609267+0000) 2019-10-13T04:34:25.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:25.607+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.609188+0000) 2019-10-13T04:34:25.614 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:25.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:25.614 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:25.605+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.606404+0000) 2019-10-13T04:34:25.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:25.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:25.873472+0000) 2019-10-13T04:34:27.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:27.366+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:27.367105+0000) 2019-10-13T04:34:28.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:28.158+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:28.159027+0000) 2019-10-13T04:34:29.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:29.511+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:29.511938+0000) 2019-10-13T04:34:30.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:30.609+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.611308+0000) 2019-10-13T04:34:30.614 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:30.610+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.611717+0000) 2019-10-13T04:34:30.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:30.606+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.606737+0000) 2019-10-13T04:34:30.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:30.610+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.612003+0000) 2019-10-13T04:34:30.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:30.610+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.612089+0000) 2019-10-13T04:34:30.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:30.610+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.612356+0000) 2019-10-13T04:34:30.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:30.611+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.612532+0000) 2019-10-13T04:34:30.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:30.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:30.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:30.608+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:30.608815+0000) 2019-10-13T04:34:31.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:31.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:31.750478+0000) 2019-10-13T04:34:31.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:31.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:31.750564+0000) 2019-10-13T04:34:31.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:31.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:31.750612+0000) 2019-10-13T04:34:32.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:32.623+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:32.624140+0000) 2019-10-13T04:34:35.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:35.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.610191+0000) 2019-10-13T04:34:35.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:35.611+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.612490+0000) 2019-10-13T04:34:35.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:35.614+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.614766+0000) 2019-10-13T04:34:35.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:35.614+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.615212+0000) 2019-10-13T04:34:35.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:35.615+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.615411+0000) 2019-10-13T04:34:35.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:35.615+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.615571+0000) 2019-10-13T04:34:35.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:35.615+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.615720+0000) 2019-10-13T04:34:35.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:35.615+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.615960+0000) 2019-10-13T04:34:35.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:35.616+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:35.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:35.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:35.873639+0000) 2019-10-13T04:34:37.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:37.366+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:37.367284+0000) 2019-10-13T04:34:38.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:38.158+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:38.159188+0000) 2019-10-13T04:34:39.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:39.514+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:39.514512+0000) 2019-10-13T04:34:40.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:40.617+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.618362+0000) 2019-10-13T04:34:40.621 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:40.618+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.618470+0000) 2019-10-13T04:34:40.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:40.618+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.618633+0000) 2019-10-13T04:34:40.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:40.618+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.618739+0000) 2019-10-13T04:34:40.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:40.618+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.618957+0000) 2019-10-13T04:34:40.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:40.613+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.614194+0000) 2019-10-13T04:34:40.622 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:40.614+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.615473+0000) 2019-10-13T04:34:40.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:40.619+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:40.619429+0000) 2019-10-13T04:34:40.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:40.619+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:41.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:41.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:41.750671+0000) 2019-10-13T04:34:41.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:41.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:41.750797+0000) 2019-10-13T04:34:41.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:41.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:41.750801+0000) 2019-10-13T04:34:42.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:42.623+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:42.624325+0000) 2019-10-13T04:34:45.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:45.621+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.622146+0000) 2019-10-13T04:34:45.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:45.621+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.622299+0000) 2019-10-13T04:34:45.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:45.622+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.622590+0000) 2019-10-13T04:34:45.625 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:45.622+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.622743+0000) 2019-10-13T04:34:45.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:45.617+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.618235+0000) 2019-10-13T04:34:45.626 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:45.618+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.618860+0000) 2019-10-13T04:34:45.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:45.622+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.623225+0000) 2019-10-13T04:34:45.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:45.622+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.623305+0000) 2019-10-13T04:34:45.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:45.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:45.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:45.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:45.873786+0000) 2019-10-13T04:34:47.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:47.367+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:47.367472+0000) 2019-10-13T04:34:48.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:48.159+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:48.159385+0000) 2019-10-13T04:34:49.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:49.516+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:49.516853+0000) 2019-10-13T04:34:50.628 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:50.625+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.626213+0000) 2019-10-13T04:34:50.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:50.625+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.626333+0000) 2019-10-13T04:34:50.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:50.626+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.626516+0000) 2019-10-13T04:34:50.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:50.626+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.626906+0000) 2019-10-13T04:34:50.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:50.621+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.622371+0000) 2019-10-13T04:34:50.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:50.626+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.627307+0000) 2019-10-13T04:34:50.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:50.627+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.627923+0000) 2019-10-13T04:34:50.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:50.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:50.631 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:50.623+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:50.624330+0000) 2019-10-13T04:34:51.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:51.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:51.750880+0000) 2019-10-13T04:34:51.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:51.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:51.750994+0000) 2019-10-13T04:34:51.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:51.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:51.751014+0000) 2019-10-13T04:34:52.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:52.624+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:52.624485+0000) 2019-10-13T04:34:54.102 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:34:54.102 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:54.106 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:34:54.106 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:55.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:55.628+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.629719+0000) 2019-10-13T04:34:55.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:55.628+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.629958+0000) 2019-10-13T04:34:55.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:34:55.628+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.629967+0000) 2019-10-13T04:34:55.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:34:55.629+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.630500+0000) 2019-10-13T04:34:55.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:34:55.625+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.625831+0000) 2019-10-13T04:34:55.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:55.629+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.630618+0000) 2019-10-13T04:34:55.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:34:55.629+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.630738+0000) 2019-10-13T04:34:55.634 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:34:55.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:34:55.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:55.627+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.627955+0000) 2019-10-13T04:34:55.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:34:55.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:55.873933+0000) 2019-10-13T04:34:57.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:34:57.366+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:57.367663+0000) 2019-10-13T04:34:58.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:34:58.158+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:58.159615+0000) 2019-10-13T04:34:59.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:34:59.519+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:34:59.519208+0000) 2019-10-13T04:35:00.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:00.631+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.633157+0000) 2019-10-13T04:35:00.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:00.631+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.633368+0000) 2019-10-13T04:35:00.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:00.632+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.633588+0000) 2019-10-13T04:35:00.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:00.628+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.628759+0000) 2019-10-13T04:35:00.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:00.632+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.633826+0000) 2019-10-13T04:35:00.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:00.632+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.633917+0000) 2019-10-13T04:35:00.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:00.632+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.634273+0000) 2019-10-13T04:35:00.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:00.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:00.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:00.630+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:00.631452+0000) 2019-10-13T04:35:01.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:01.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:01.751071+0000) 2019-10-13T04:35:01.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:01.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:01.751129+0000) 2019-10-13T04:35:01.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:01.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:01.751140+0000) 2019-10-13T04:35:02.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:02.623+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:02.624647+0000) 2019-10-13T04:35:05.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:05.635+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.636863+0000) 2019-10-13T04:35:05.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:05.635+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.637058+0000) 2019-10-13T04:35:05.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:05.636+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.637489+0000) 2019-10-13T04:35:05.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:05.631+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.632953+0000) 2019-10-13T04:35:05.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:05.636+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.637797+0000) 2019-10-13T04:35:05.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:05.636+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.638090+0000) 2019-10-13T04:35:05.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:05.636+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.638214+0000) 2019-10-13T04:35:05.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:05.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58893 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:05.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:05.633+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.635420+0000) 2019-10-13T04:35:05.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:05.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:05.874121+0000) 2019-10-13T04:35:07.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:07.367+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:07.367821+0000) 2019-10-13T04:35:08.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:08.159+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:08.159806+0000) 2019-10-13T04:35:09.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:09.520+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:09.521683+0000) 2019-10-13T04:35:10.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:10.639+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.641216+0000) 2019-10-13T04:35:10.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:10.640+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.641421+0000) 2019-10-13T04:35:10.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:10.640+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.641740+0000) 2019-10-13T04:35:10.644 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:10.640+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.641897+0000) 2019-10-13T04:35:10.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:10.640+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.641937+0000) 2019-10-13T04:35:10.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:10.640+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.642144+0000) 2019-10-13T04:35:10.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:10.635+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.637714+0000) 2019-10-13T04:35:10.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:10.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58922 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:10.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:10.638+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:10.639782+0000) 2019-10-13T04:35:11.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:11.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:11.751269+0000) 2019-10-13T04:35:11.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:11.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:11.751269+0000) 2019-10-13T04:35:11.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:11.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:11.751296+0000) 2019-10-13T04:35:12.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:12.623+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:12.624829+0000) 2019-10-13T04:35:15.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:15.643+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.645072+0000) 2019-10-13T04:35:15.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:15.643+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.645328+0000) 2019-10-13T04:35:15.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:15.644+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.645462+0000) 2019-10-13T04:35:15.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:15.640+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.640948+0000) 2019-10-13T04:35:15.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:15.644+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.645777+0000) 2019-10-13T04:35:15.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:15.644+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.645899+0000) 2019-10-13T04:35:15.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:15.644+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.646117+0000) 2019-10-13T04:35:15.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:15.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:15.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:15.642+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.644067+0000) 2019-10-13T04:35:15.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:15.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:15.874305+0000) 2019-10-13T04:35:17.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:17.367+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:17.367978+0000) 2019-10-13T04:35:18.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:18.159+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:18.159977+0000) 2019-10-13T04:35:19.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:19.523+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:19.523997+0000) 2019-10-13T04:35:20.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:20.648+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.649167+0000) 2019-10-13T04:35:20.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:20.649+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.649476+0000) 2019-10-13T04:35:20.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:20.649+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.649896+0000) 2019-10-13T04:35:20.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:20.649+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.650056+0000) 2019-10-13T04:35:20.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:20.649+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.650257+0000) 2019-10-13T04:35:20.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:20.645+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.646008+0000) 2019-10-13T04:35:20.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:20.650+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.650860+0000) 2019-10-13T04:35:20.654 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:20.651+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 58985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:20.654 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:20.647+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:20.648245+0000) 2019-10-13T04:35:21.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:21.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:21.751497+0000) 2019-10-13T04:35:21.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:21.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:21.751497+0000) 2019-10-13T04:35:21.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:21.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:21.751507+0000) 2019-10-13T04:35:22.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:22.624+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:22.625008+0000) 2019-10-13T04:35:24.280 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:35:24.280 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:24.285 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:35:24.285 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:25.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:25.653+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.653747+0000) 2019-10-13T04:35:25.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:25.653+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.653855+0000) 2019-10-13T04:35:25.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:25.653+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.654059+0000) 2019-10-13T04:35:25.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:25.649+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.649896+0000) 2019-10-13T04:35:25.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:25.654+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.654427+0000) 2019-10-13T04:35:25.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:25.654+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.655068+0000) 2019-10-13T04:35:25.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:25.654+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.655356+0000) 2019-10-13T04:35:25.658 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:25.650+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.651702+0000) 2019-10-13T04:35:25.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:25.655+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:25.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:25.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:25.874473+0000) 2019-10-13T04:35:27.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:27.367+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:27.368142+0000) 2019-10-13T04:35:28.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:28.159+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:28.160158+0000) 2019-10-13T04:35:29.533 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:29.525+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:29.526188+0000) 2019-10-13T04:35:30.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:30.652+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.652849+0000) 2019-10-13T04:35:30.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:30.657+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.657863+0000) 2019-10-13T04:35:30.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:30.657+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.658178+0000) 2019-10-13T04:35:30.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:30.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.658469+0000) 2019-10-13T04:35:30.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:30.658+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.658641+0000) 2019-10-13T04:35:30.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:30.658+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.658844+0000) 2019-10-13T04:35:30.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:30.658+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.659021+0000) 2019-10-13T04:35:30.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:30.654+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:30.655176+0000) 2019-10-13T04:35:30.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:30.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:31.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:31.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:31.751684+0000) 2019-10-13T04:35:31.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:31.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:31.751684+0000) 2019-10-13T04:35:31.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:31.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:31.751683+0000) 2019-10-13T04:35:32.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:32.624+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:32.625195+0000) 2019-10-13T04:35:35.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:35.661+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.661617+0000) 2019-10-13T04:35:35.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:35.656+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.657018+0000) 2019-10-13T04:35:35.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:35.661+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.662027+0000) 2019-10-13T04:35:35.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:35.662+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.662459+0000) 2019-10-13T04:35:35.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:35.662+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.662483+0000) 2019-10-13T04:35:35.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:35.662+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.662655+0000) 2019-10-13T04:35:35.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:35.662+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.662968+0000) 2019-10-13T04:35:35.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:35.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:35.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:35.659+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.659944+0000) 2019-10-13T04:35:35.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:35.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:35.874601+0000) 2019-10-13T04:35:37.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:37.368+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:37.368316+0000) 2019-10-13T04:35:38.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:38.160+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:38.160358+0000) 2019-10-13T04:35:39.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:39.528+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:39.528519+0000) 2019-10-13T04:35:40.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:40.664+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.665630+0000) 2019-10-13T04:35:40.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:40.664+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.666172+0000) 2019-10-13T04:35:40.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:40.660+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.661284+0000) 2019-10-13T04:35:40.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:40.664+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.666354+0000) 2019-10-13T04:35:40.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:40.665+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.666597+0000) 2019-10-13T04:35:40.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:40.665+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.666705+0000) 2019-10-13T04:35:40.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:40.665+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.666968+0000) 2019-10-13T04:35:40.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:40.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:40.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:40.663+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:40.664617+0000) 2019-10-13T04:35:41.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:41.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:41.751899+0000) 2019-10-13T04:35:41.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:41.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:41.751895+0000) 2019-10-13T04:35:41.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:41.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:41.751899+0000) 2019-10-13T04:35:42.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:42.625+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:42.625377+0000) 2019-10-13T04:35:45.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:45.668+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.669606+0000) 2019-10-13T04:35:45.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:45.668+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.669767+0000) 2019-10-13T04:35:45.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:45.665+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.665841+0000) 2019-10-13T04:35:45.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:45.668+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.670279+0000) 2019-10-13T04:35:45.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:45.669+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.670471+0000) 2019-10-13T04:35:45.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:45.669+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.670669+0000) 2019-10-13T04:35:45.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:45.669+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.670990+0000) 2019-10-13T04:35:45.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:45.666+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.667333+0000) 2019-10-13T04:35:45.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:45.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:45.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:45.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:45.874727+0000) 2019-10-13T04:35:47.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:47.367+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:47.368490+0000) 2019-10-13T04:35:48.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:48.159+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:48.160545+0000) 2019-10-13T04:35:49.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:49.530+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:49.530764+0000) 2019-10-13T04:35:50.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:50.672+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.674346+0000) 2019-10-13T04:35:50.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:50.669+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.670374+0000) 2019-10-13T04:35:50.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:50.673+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.674771+0000) 2019-10-13T04:35:50.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:50.673+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.674915+0000) 2019-10-13T04:35:50.678 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:50.674+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.675535+0000) 2019-10-13T04:35:50.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:50.674+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.675845+0000) 2019-10-13T04:35:50.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:50.674+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.676004+0000) 2019-10-13T04:35:50.679 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:50.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:50.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:50.672+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:50.672820+0000) 2019-10-13T04:35:51.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:51.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:51.752119+0000) 2019-10-13T04:35:51.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:51.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:51.752118+0000) 2019-10-13T04:35:51.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:51.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:51.752109+0000) 2019-10-13T04:35:52.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:52.625+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:52.625602+0000) 2019-10-13T04:35:54.430 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:35:54.430 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:54.435 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:35:54.435 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:55.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:35:55.671+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.673053+0000) 2019-10-13T04:35:55.680 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:55.677+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.678486+0000) 2019-10-13T04:35:55.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:55.677+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.678679+0000) 2019-10-13T04:35:55.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:55.677+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.678788+0000) 2019-10-13T04:35:55.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:35:55.677+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.679157+0000) 2019-10-13T04:35:55.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:35:55.677+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.679370+0000) 2019-10-13T04:35:55.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:35:55.677+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.678946+0000) 2019-10-13T04:35:55.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:55.673+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.675455+0000) 2019-10-13T04:35:55.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:35:55.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59202 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:35:55.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:35:55.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:55.874907+0000) 2019-10-13T04:35:57.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:35:57.367+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:57.368628+0000) 2019-10-13T04:35:58.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:35:58.160+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:58.160695+0000) 2019-10-13T04:35:59.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:35:59.532+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:35:59.533131+0000) 2019-10-13T04:36:00.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:00.680+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.682367+0000) 2019-10-13T04:36:00.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:00.681+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.682580+0000) 2019-10-13T04:36:00.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:00.681+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.682682+0000) 2019-10-13T04:36:00.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:00.681+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.683002+0000) 2019-10-13T04:36:00.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:00.676+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.678519+0000) 2019-10-13T04:36:00.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:00.677+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.679579+0000) 2019-10-13T04:36:00.687 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:00.681+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.683374+0000) 2019-10-13T04:36:00.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:00.682+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:00.683670+0000) 2019-10-13T04:36:00.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:00.683+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:01.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:01.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:01.752245+0000) 2019-10-13T04:36:01.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:01.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:01.752339+0000) 2019-10-13T04:36:01.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:01.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:01.752316+0000) 2019-10-13T04:36:02.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:02.624+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:02.625737+0000) 2019-10-13T04:36:05.688 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:05.685+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.686569+0000) 2019-10-13T04:36:05.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:05.685+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.687041+0000) 2019-10-13T04:36:05.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:05.685+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.687181+0000) 2019-10-13T04:36:05.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:05.685+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.687328+0000) 2019-10-13T04:36:05.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:05.686+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.687507+0000) 2019-10-13T04:36:05.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:05.686+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.687639+0000) 2019-10-13T04:36:05.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:05.681+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.683206+0000) 2019-10-13T04:36:05.691 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:05.682+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.684215+0000) 2019-10-13T04:36:05.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:05.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:05.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:05.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:05.875094+0000) 2019-10-13T04:36:07.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:07.368+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:07.368809+0000) 2019-10-13T04:36:08.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:08.160+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:08.160848+0000) 2019-10-13T04:36:09.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:09.534+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:09.535456+0000) 2019-10-13T04:36:10.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:10.690+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.690674+0000) 2019-10-13T04:36:10.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:10.690+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.690867+0000) 2019-10-13T04:36:10.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:10.690+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.691128+0000) 2019-10-13T04:36:10.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:10.686+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.687164+0000) 2019-10-13T04:36:10.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:10.690+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.691302+0000) 2019-10-13T04:36:10.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:10.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.691696+0000) 2019-10-13T04:36:10.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:10.691+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.692013+0000) 2019-10-13T04:36:10.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:10.687+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:10.688541+0000) 2019-10-13T04:36:10.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:10.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:11.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:11.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:11.752437+0000) 2019-10-13T04:36:11.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:11.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:11.752506+0000) 2019-10-13T04:36:11.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:11.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:11.752483+0000) 2019-10-13T04:36:12.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:12.625+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:12.625892+0000) 2019-10-13T04:36:15.697 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:15.694+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.694835+0000) 2019-10-13T04:36:15.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:15.694+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.694973+0000) 2019-10-13T04:36:15.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:15.690+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.690987+0000) 2019-10-13T04:36:15.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:15.694+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.695378+0000) 2019-10-13T04:36:15.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:15.695+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.695544+0000) 2019-10-13T04:36:15.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:15.695+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.696032+0000) 2019-10-13T04:36:15.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:15.695+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.696133+0000) 2019-10-13T04:36:15.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:15.691+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.692130+0000) 2019-10-13T04:36:15.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:15.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:15.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:15.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:15.875204+0000) 2019-10-13T04:36:17.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:17.368+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:17.369016+0000) 2019-10-13T04:36:18.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:18.160+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:18.161061+0000) 2019-10-13T04:36:19.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:19.537+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:19.537868+0000) 2019-10-13T04:36:20.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:20.698+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.698776+0000) 2019-10-13T04:36:20.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:20.698+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.698921+0000) 2019-10-13T04:36:20.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:20.693+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.694364+0000) 2019-10-13T04:36:20.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:20.699+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.699588+0000) 2019-10-13T04:36:20.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:20.699+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.699677+0000) 2019-10-13T04:36:20.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:20.699+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.699810+0000) 2019-10-13T04:36:20.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:20.699+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.700116+0000) 2019-10-13T04:36:20.703 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:20.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:20.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:20.697+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:20.697830+0000) 2019-10-13T04:36:21.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:21.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:21.752636+0000) 2019-10-13T04:36:21.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:21.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:21.752694+0000) 2019-10-13T04:36:21.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:21.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:21.752708+0000) 2019-10-13T04:36:22.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:22.625+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:22.626072+0000) 2019-10-13T04:36:24.611 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:36:24.611 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:24.615 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:36:24.616 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:25.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:25.701+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.702624+0000) 2019-10-13T04:36:25.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:25.701+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.702926+0000) 2019-10-13T04:36:25.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:25.697+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.698713+0000) 2019-10-13T04:36:25.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:25.701+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.703206+0000) 2019-10-13T04:36:25.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:25.702+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.703539+0000) 2019-10-13T04:36:25.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:25.702+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.703669+0000) 2019-10-13T04:36:25.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:25.702+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.703785+0000) 2019-10-13T04:36:25.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:25.702+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59389 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:25.707 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:25.699+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.700043+0000) 2019-10-13T04:36:25.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:25.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:25.875380+0000) 2019-10-13T04:36:27.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:27.368+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:27.369163+0000) 2019-10-13T04:36:28.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:28.159+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:28.161232+0000) 2019-10-13T04:36:29.547 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:29.539+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:29.540296+0000) 2019-10-13T04:36:30.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:30.704+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.706211+0000) 2019-10-13T04:36:30.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:30.704+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.706384+0000) 2019-10-13T04:36:30.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:30.705+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.706643+0000) 2019-10-13T04:36:30.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:30.705+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.706839+0000) 2019-10-13T04:36:30.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:30.701+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.701817+0000) 2019-10-13T04:36:30.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:30.702+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.703569+0000) 2019-10-13T04:36:30.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:30.705+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.707347+0000) 2019-10-13T04:36:30.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:30.706+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:30.707551+0000) 2019-10-13T04:36:30.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:30.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59418 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:31.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:31.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:31.752902+0000) 2019-10-13T04:36:31.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:31.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:31.752873+0000) 2019-10-13T04:36:31.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:31.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:31.752951+0000) 2019-10-13T04:36:32.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:32.626+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:32.626275+0000) 2019-10-13T04:36:35.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:35.709+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.710584+0000) 2019-10-13T04:36:35.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:35.709+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.710802+0000) 2019-10-13T04:36:35.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:35.709+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.710848+0000) 2019-10-13T04:36:35.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:35.709+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.711020+0000) 2019-10-13T04:36:35.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:35.709+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.711136+0000) 2019-10-13T04:36:35.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:35.709+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.711215+0000) 2019-10-13T04:36:35.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:35.704+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.705624+0000) 2019-10-13T04:36:35.715 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:35.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:35.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:35.707+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.707804+0000) 2019-10-13T04:36:35.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:35.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:35.875524+0000) 2019-10-13T04:36:37.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:37.368+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:37.369384+0000) 2019-10-13T04:36:38.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:38.160+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:38.161461+0000) 2019-10-13T04:36:39.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:39.542+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:39.542678+0000) 2019-10-13T04:36:40.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:40.707+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.708711+0000) 2019-10-13T04:36:40.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:40.712+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.713539+0000) 2019-10-13T04:36:40.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:40.712+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.713586+0000) 2019-10-13T04:36:40.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:40.712+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.713764+0000) 2019-10-13T04:36:40.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:40.712+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.714085+0000) 2019-10-13T04:36:40.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:40.712+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.714226+0000) 2019-10-13T04:36:40.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:40.713+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.714409+0000) 2019-10-13T04:36:40.718 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:40.713+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:40.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:40.711+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:40.711742+0000) 2019-10-13T04:36:41.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:41.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:41.753108+0000) 2019-10-13T04:36:41.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:41.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:41.753111+0000) 2019-10-13T04:36:41.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:41.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:41.753120+0000) 2019-10-13T04:36:42.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:42.626+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:42.626465+0000) 2019-10-13T04:36:45.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:45.710+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.712240+0000) 2019-10-13T04:36:45.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:45.715+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.717213+0000) 2019-10-13T04:36:45.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:45.715+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.717320+0000) 2019-10-13T04:36:45.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:45.716+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.717522+0000) 2019-10-13T04:36:45.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:45.716+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.717681+0000) 2019-10-13T04:36:45.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:45.716+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.717941+0000) 2019-10-13T04:36:45.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:45.717+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.718586+0000) 2019-10-13T04:36:45.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:45.718+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:45.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:45.713+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.715501+0000) 2019-10-13T04:36:45.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:45.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:45.875645+0000) 2019-10-13T04:36:47.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:47.369+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:47.369590+0000) 2019-10-13T04:36:48.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:48.161+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:48.161659+0000) 2019-10-13T04:36:49.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:49.543+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:49.544821+0000) 2019-10-13T04:36:50.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:50.719+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.721376+0000) 2019-10-13T04:36:50.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:50.720+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.721498+0000) 2019-10-13T04:36:50.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:50.720+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.721743+0000) 2019-10-13T04:36:50.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:50.720+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.721422+0000) 2019-10-13T04:36:50.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:50.720+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.722051+0000) 2019-10-13T04:36:50.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:50.715+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.717194+0000) 2019-10-13T04:36:50.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:50.720+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.722318+0000) 2019-10-13T04:36:50.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:50.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59542 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:50.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:50.718+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:50.719915+0000) 2019-10-13T04:36:51.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:51.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:51.753291+0000) 2019-10-13T04:36:51.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:51.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:51.753292+0000) 2019-10-13T04:36:51.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:51.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:51.753292+0000) 2019-10-13T04:36:52.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:52.625+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:52.626633+0000) 2019-10-13T04:36:54.757 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:36:54.758 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:54.762 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:36:54.762 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:55.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:55.725+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.725440+0000) 2019-10-13T04:36:55.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:36:55.725+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.725780+0000) 2019-10-13T04:36:55.728 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:55.725+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.725931+0000) 2019-10-13T04:36:55.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:36:55.719+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.721456+0000) 2019-10-13T04:36:55.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:55.720+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.722449+0000) 2019-10-13T04:36:55.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:36:55.725+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.726288+0000) 2019-10-13T04:36:55.730 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:36:55.726+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.726476+0000) 2019-10-13T04:36:55.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:55.726+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.726776+0000) 2019-10-13T04:36:55.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:36:55.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:36:55.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:36:55.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:55.875810+0000) 2019-10-13T04:36:57.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:36:57.369+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:57.369771+0000) 2019-10-13T04:36:58.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:36:58.161+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:58.161819+0000) 2019-10-13T04:36:59.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:36:59.546+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:36:59.547121+0000) 2019-10-13T04:37:00.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:00.728+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.729104+0000) 2019-10-13T04:37:00.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:00.728+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.729334+0000) 2019-10-13T04:37:00.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:00.729+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.729423+0000) 2019-10-13T04:37:00.732 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:00.729+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.729511+0000) 2019-10-13T04:37:00.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:00.724+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.725717+0000) 2019-10-13T04:37:00.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:00.729+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.729888+0000) 2019-10-13T04:37:00.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:00.729+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.730259+0000) 2019-10-13T04:37:00.733 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:00.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:00.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:00.727+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:00.727862+0000) 2019-10-13T04:37:01.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:01.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:01.753505+0000) 2019-10-13T04:37:01.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:01.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:01.753487+0000) 2019-10-13T04:37:01.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:01.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:01.753503+0000) 2019-10-13T04:37:02.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:02.626+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:02.626791+0000) 2019-10-13T04:37:05.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:05.727+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.728309+0000) 2019-10-13T04:37:05.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:05.733+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.733616+0000) 2019-10-13T04:37:05.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:05.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.733670+0000) 2019-10-13T04:37:05.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:05.733+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.733848+0000) 2019-10-13T04:37:05.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:05.733+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.734035+0000) 2019-10-13T04:37:05.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:05.733+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.734138+0000) 2019-10-13T04:37:05.737 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:05.733+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:05.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:05.733+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.734266+0000) 2019-10-13T04:37:05.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:05.729+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.730323+0000) 2019-10-13T04:37:05.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:05.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:05.875974+0000) 2019-10-13T04:37:07.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:07.369+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:07.369964+0000) 2019-10-13T04:37:08.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:08.161+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:08.162033+0000) 2019-10-13T04:37:09.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:09.548+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:09.549497+0000) 2019-10-13T04:37:10.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:10.734+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.735997+0000) 2019-10-13T04:37:10.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:10.734+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.736199+0000) 2019-10-13T04:37:10.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:10.735+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.736531+0000) 2019-10-13T04:37:10.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:10.735+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.736761+0000) 2019-10-13T04:37:10.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:10.735+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.737184+0000) 2019-10-13T04:37:10.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:10.731+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.732372+0000) 2019-10-13T04:37:10.740 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:10.736+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.737448+0000) 2019-10-13T04:37:10.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:10.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:10.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:10.733+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:10.734670+0000) 2019-10-13T04:37:11.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:11.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:11.753647+0000) 2019-10-13T04:37:11.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:11.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:11.753642+0000) 2019-10-13T04:37:11.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:11.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:11.753649+0000) 2019-10-13T04:37:12.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:12.626+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:12.626958+0000) 2019-10-13T04:37:15.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:15.738+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.740123+0000) 2019-10-13T04:37:15.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:15.739+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.740405+0000) 2019-10-13T04:37:15.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:15.739+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.740598+0000) 2019-10-13T04:37:15.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:15.739+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.740755+0000) 2019-10-13T04:37:15.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:15.739+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.740965+0000) 2019-10-13T04:37:15.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:15.739+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.741371+0000) 2019-10-13T04:37:15.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:15.735+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.736440+0000) 2019-10-13T04:37:15.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:15.736+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.737725+0000) 2019-10-13T04:37:15.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:15.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:15.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:15.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:15.876170+0000) 2019-10-13T04:37:17.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:17.368+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:17.370146+0000) 2019-10-13T04:37:18.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:18.161+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:18.162219+0000) 2019-10-13T04:37:19.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:19.551+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:19.551765+0000) 2019-10-13T04:37:20.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:20.742+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.743976+0000) 2019-10-13T04:37:20.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:20.738+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.739579+0000) 2019-10-13T04:37:20.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:20.743+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.744433+0000) 2019-10-13T04:37:20.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:20.743+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.744535+0000) 2019-10-13T04:37:20.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:20.743+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.744882+0000) 2019-10-13T04:37:20.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:20.743+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.745196+0000) 2019-10-13T04:37:20.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:20.743+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.745274+0000) 2019-10-13T04:37:20.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:20.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:20.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:20.741+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:20.742418+0000) 2019-10-13T04:37:21.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:21.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:21.753766+0000) 2019-10-13T04:37:21.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:21.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:21.753807+0000) 2019-10-13T04:37:21.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:21.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:21.753808+0000) 2019-10-13T04:37:22.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:22.626+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:22.627154+0000) 2019-10-13T04:37:24.880 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:37:24.880 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:24.885 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:37:24.885 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:25.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:25.746+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.748261+0000) 2019-10-13T04:37:25.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:25.746+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.748380+0000) 2019-10-13T04:37:25.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:25.747+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.749062+0000) 2019-10-13T04:37:25.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:25.747+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.749176+0000) 2019-10-13T04:37:25.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:25.743+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.744009+0000) 2019-10-13T04:37:25.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:25.747+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.749367+0000) 2019-10-13T04:37:25.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:25.748+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.749469+0000) 2019-10-13T04:37:25.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:25.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:25.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:25.745+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.745948+0000) 2019-10-13T04:37:25.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:25.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:25.876336+0000) 2019-10-13T04:37:27.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:27.369+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:27.370329+0000) 2019-10-13T04:37:28.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:28.161+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:28.162428+0000) 2019-10-13T04:37:29.561 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:29.554+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:29.554164+0000) 2019-10-13T04:37:30.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:30.747+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.747848+0000) 2019-10-13T04:37:30.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:30.751+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.752740+0000) 2019-10-13T04:37:30.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:30.751+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.752893+0000) 2019-10-13T04:37:30.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:30.751+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.753200+0000) 2019-10-13T04:37:30.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:30.752+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.753476+0000) 2019-10-13T04:37:30.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:30.752+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.753631+0000) 2019-10-13T04:37:30.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:30.752+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.754002+0000) 2019-10-13T04:37:30.757 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:30.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:30.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:30.750+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:30.751417+0000) 2019-10-13T04:37:31.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:31.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:31.753978+0000) 2019-10-13T04:37:31.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:31.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:31.753978+0000) 2019-10-13T04:37:31.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:31.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:31.753980+0000) 2019-10-13T04:37:32.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:32.627+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:32.627332+0000) 2019-10-13T04:37:35.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:35.750+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.751816+0000) 2019-10-13T04:37:35.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:35.755+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.756680+0000) 2019-10-13T04:37:35.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:35.755+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.756868+0000) 2019-10-13T04:37:35.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:35.755+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.757038+0000) 2019-10-13T04:37:35.760 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:35.755+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.757146+0000) 2019-10-13T04:37:35.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:35.756+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.757690+0000) 2019-10-13T04:37:35.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:35.756+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.758115+0000) 2019-10-13T04:37:35.761 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:35.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59822 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:35.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:35.753+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.754808+0000) 2019-10-13T04:37:35.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:35.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:35.876524+0000) 2019-10-13T04:37:37.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:37.370+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:37.370566+0000) 2019-10-13T04:37:38.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:38.162+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:38.162623+0000) 2019-10-13T04:37:39.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:39.556+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:39.557219+0000) 2019-10-13T04:37:40.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:40.760+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.761122+0000) 2019-10-13T04:37:40.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:40.760+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.761239+0000) 2019-10-13T04:37:40.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:40.761+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.761414+0000) 2019-10-13T04:37:40.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:40.761+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.761789+0000) 2019-10-13T04:37:40.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:40.761+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.762054+0000) 2019-10-13T04:37:40.771 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:40.761+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.762309+0000) 2019-10-13T04:37:40.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:40.763+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:40.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:40.754+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.756259+0000) 2019-10-13T04:37:40.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:40.758+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:40.760242+0000) 2019-10-13T04:37:41.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:41.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:41.754188+0000) 2019-10-13T04:37:41.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:41.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:41.754190+0000) 2019-10-13T04:37:41.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:41.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:41.754157+0000) 2019-10-13T04:37:42.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:42.626+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:42.627515+0000) 2019-10-13T04:37:45.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:45.764+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.764982+0000) 2019-10-13T04:37:45.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:45.764+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.765330+0000) 2019-10-13T04:37:45.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:45.759+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.761275+0000) 2019-10-13T04:37:45.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:45.765+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.765512+0000) 2019-10-13T04:37:45.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:45.765+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.765799+0000) 2019-10-13T04:37:45.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:45.765+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.766327+0000) 2019-10-13T04:37:45.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:45.766+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.766624+0000) 2019-10-13T04:37:45.769 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:45.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:45.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:45.761+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.763564+0000) 2019-10-13T04:37:45.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:45.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:45.876722+0000) 2019-10-13T04:37:47.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:47.370+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:47.370729+0000) 2019-10-13T04:37:48.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:48.162+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:48.162834+0000) 2019-10-13T04:37:49.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:49.558+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:49.559607+0000) 2019-10-13T04:37:50.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:50.768+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.769190+0000) 2019-10-13T04:37:50.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:50.764+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.765077+0000) 2019-10-13T04:37:50.772 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:50.769+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.769652+0000) 2019-10-13T04:37:50.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:50.769+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.769455+0000) 2019-10-13T04:37:50.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:50.769+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.769896+0000) 2019-10-13T04:37:50.773 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:50.770+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.770641+0000) 2019-10-13T04:37:50.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:50.770+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.770725+0000) 2019-10-13T04:37:50.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:50.766+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:50.767587+0000) 2019-10-13T04:37:50.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:50.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59914 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:51.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:51.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:51.754347+0000) 2019-10-13T04:37:51.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:51.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:51.754364+0000) 2019-10-13T04:37:51.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:51.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:51.754363+0000) 2019-10-13T04:37:52.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:52.627+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:52.627698+0000) 2019-10-13T04:37:55.077 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:37:55.077 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:55.082 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:37:55.082 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:55.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:55.772+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.774020+0000) 2019-10-13T04:37:55.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:37:55.772+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.774381+0000) 2019-10-13T04:37:55.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:37:55.773+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.774633+0000) 2019-10-13T04:37:55.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:55.773+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.774594+0000) 2019-10-13T04:37:55.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:55.773+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.774941+0000) 2019-10-13T04:37:55.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:37:55.769+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.770059+0000) 2019-10-13T04:37:55.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:37:55.773+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.775207+0000) 2019-10-13T04:37:55.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:37:55.774+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:37:55.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:55.771+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.772225+0000) 2019-10-13T04:37:55.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:37:55.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:55.876913+0000) 2019-10-13T04:37:57.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:37:57.369+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:57.370886+0000) 2019-10-13T04:37:58.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:37:58.162+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:58.163025+0000) 2019-10-13T04:37:59.568 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:37:59.561+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:37:59.561898+0000) 2019-10-13T04:38:00.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:00.776+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.777690+0000) 2019-10-13T04:38:00.780 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:00.776+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.777918+0000) 2019-10-13T04:38:00.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:00.776+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.778145+0000) 2019-10-13T04:38:00.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:00.776+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.778243+0000) 2019-10-13T04:38:00.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:00.772+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.773621+0000) 2019-10-13T04:38:00.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:00.777+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.778694+0000) 2019-10-13T04:38:00.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:00.777+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.778748+0000) 2019-10-13T04:38:00.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:00.778+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 59977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:00.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:00.774+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:00.775373+0000) 2019-10-13T04:38:01.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:01.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:01.754525+0000) 2019-10-13T04:38:01.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:01.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:01.754520+0000) 2019-10-13T04:38:01.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:01.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:01.754527+0000) 2019-10-13T04:38:02.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:02.627+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:02.627889+0000) 2019-10-13T04:38:05.785 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:05.780+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.781566+0000) 2019-10-13T04:38:05.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:05.780+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.781870+0000) 2019-10-13T04:38:05.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:05.780+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.782016+0000) 2019-10-13T04:38:05.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:05.780+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.782229+0000) 2019-10-13T04:38:05.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:05.781+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.782405+0000) 2019-10-13T04:38:05.786 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:05.781+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.782655+0000) 2019-10-13T04:38:05.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:05.777+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.778078+0000) 2019-10-13T04:38:05.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:05.778+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.779165+0000) 2019-10-13T04:38:05.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:05.782+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:05.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:05.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:05.877063+0000) 2019-10-13T04:38:07.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:07.370+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:07.371015+0000) 2019-10-13T04:38:08.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:08.162+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:08.163195+0000) 2019-10-13T04:38:09.571 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:09.564+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:09.564371+0000) 2019-10-13T04:38:10.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:10.784+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.785955+0000) 2019-10-13T04:38:10.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:10.784+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.786380+0000) 2019-10-13T04:38:10.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:10.785+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.786425+0000) 2019-10-13T04:38:10.789 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:10.785+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.786719+0000) 2019-10-13T04:38:10.789 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:10.785+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.786881+0000) 2019-10-13T04:38:10.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:10.781+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.782592+0000) 2019-10-13T04:38:10.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:10.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.787509+0000) 2019-10-13T04:38:10.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:10.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:10.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:10.783+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:10.784736+0000) 2019-10-13T04:38:11.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:11.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:11.754715+0000) 2019-10-13T04:38:11.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:11.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:11.754712+0000) 2019-10-13T04:38:11.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:11.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:11.754713+0000) 2019-10-13T04:38:12.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:12.627+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:12.628092+0000) 2019-10-13T04:38:15.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:15.788+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.790112+0000) 2019-10-13T04:38:15.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:15.788+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.790283+0000) 2019-10-13T04:38:15.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:15.789+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.790621+0000) 2019-10-13T04:38:15.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:15.785+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.785770+0000) 2019-10-13T04:38:15.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:15.789+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.791053+0000) 2019-10-13T04:38:15.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:15.789+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.791149+0000) 2019-10-13T04:38:15.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:15.790+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.791414+0000) 2019-10-13T04:38:15.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:15.791+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:15.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:15.788+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.788853+0000) 2019-10-13T04:38:15.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:15.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:15.877222+0000) 2019-10-13T04:38:17.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:17.370+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:17.371164+0000) 2019-10-13T04:38:18.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:18.162+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:18.163386+0000) 2019-10-13T04:38:19.573 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:19.566+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:19.566726+0000) 2019-10-13T04:38:20.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:20.793+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.794545+0000) 2019-10-13T04:38:20.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:20.793+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.794631+0000) 2019-10-13T04:38:20.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:20.793+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.794880+0000) 2019-10-13T04:38:20.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:20.793+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.795102+0000) 2019-10-13T04:38:20.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:20.794+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.795407+0000) 2019-10-13T04:38:20.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:20.794+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.795684+0000) 2019-10-13T04:38:20.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:20.788+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.789488+0000) 2019-10-13T04:38:20.800 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:20.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60101 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:20.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:20.791+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:20.792682+0000) 2019-10-13T04:38:21.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:21.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:21.754901+0000) 2019-10-13T04:38:21.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:21.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:21.754897+0000) 2019-10-13T04:38:21.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:21.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:21.754897+0000) 2019-10-13T04:38:22.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:22.628+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:22.628263+0000) 2019-10-13T04:38:25.251 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:38:25.251 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:25.255 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:38:25.255 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:25.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:25.792+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.793896+0000) 2019-10-13T04:38:25.802 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:25.798+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.798963+0000) 2019-10-13T04:38:25.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:25.799+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.799450+0000) 2019-10-13T04:38:25.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:25.799+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.799626+0000) 2019-10-13T04:38:25.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:25.799+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.799860+0000) 2019-10-13T04:38:25.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:25.799+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.800042+0000) 2019-10-13T04:38:25.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:25.794+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.796331+0000) 2019-10-13T04:38:25.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:25.800+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.800394+0000) 2019-10-13T04:38:25.804 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:25.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60133 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:25.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:25.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:25.877371+0000) 2019-10-13T04:38:27.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:27.371+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:27.371326+0000) 2019-10-13T04:38:28.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:28.163+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:28.163576+0000) 2019-10-13T04:38:29.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:29.568+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:29.569143+0000) 2019-10-13T04:38:30.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:30.797+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.798881+0000) 2019-10-13T04:38:30.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:30.802+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.803105+0000) 2019-10-13T04:38:30.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:30.802+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.803380+0000) 2019-10-13T04:38:30.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:30.802+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.803262+0000) 2019-10-13T04:38:30.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:30.803+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.803850+0000) 2019-10-13T04:38:30.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:30.803+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.804097+0000) 2019-10-13T04:38:30.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:30.803+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.804201+0000) 2019-10-13T04:38:30.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:30.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:30.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:30.800+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:30.801879+0000) 2019-10-13T04:38:31.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:31.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:31.755082+0000) 2019-10-13T04:38:31.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:31.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:31.755086+0000) 2019-10-13T04:38:31.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:31.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:31.755081+0000) 2019-10-13T04:38:32.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:32.627+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:32.628455+0000) 2019-10-13T04:38:35.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:35.800+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.802199+0000) 2019-10-13T04:38:35.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:35.806+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.807195+0000) 2019-10-13T04:38:35.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:35.807+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.807394+0000) 2019-10-13T04:38:35.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:35.807+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.807489+0000) 2019-10-13T04:38:35.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:35.807+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.807664+0000) 2019-10-13T04:38:35.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:35.807+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.807902+0000) 2019-10-13T04:38:35.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:35.808+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.808475+0000) 2019-10-13T04:38:35.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:35.809+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60194 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:35.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:35.803+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.805663+0000) 2019-10-13T04:38:35.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:35.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:35.877550+0000) 2019-10-13T04:38:37.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:37.371+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:37.371493+0000) 2019-10-13T04:38:38.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:38.163+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:38.163744+0000) 2019-10-13T04:38:39.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:39.570+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:39.571621+0000) 2019-10-13T04:38:40.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:40.810+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.811696+0000) 2019-10-13T04:38:40.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:40.810+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.811930+0000) 2019-10-13T04:38:40.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:40.807+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.808008+0000) 2019-10-13T04:38:40.815 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:40.810+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.812217+0000) 2019-10-13T04:38:40.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:40.810+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.812344+0000) 2019-10-13T04:38:40.815 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:40.811+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.812624+0000) 2019-10-13T04:38:40.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:40.811+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.812901+0000) 2019-10-13T04:38:40.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:40.808+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:40.809005+0000) 2019-10-13T04:38:40.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:40.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:41.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:41.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:41.755266+0000) 2019-10-13T04:38:41.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:41.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:41.755267+0000) 2019-10-13T04:38:41.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:41.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:41.755271+0000) 2019-10-13T04:38:42.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:42.627+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:42.628623+0000) 2019-10-13T04:38:45.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:45.814+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.815501+0000) 2019-10-13T04:38:45.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:45.814+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.815640+0000) 2019-10-13T04:38:45.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:45.814+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.815793+0000) 2019-10-13T04:38:45.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:45.814+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.816056+0000) 2019-10-13T04:38:45.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:45.810+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.811604+0000) 2019-10-13T04:38:45.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:45.814+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.816191+0000) 2019-10-13T04:38:45.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:45.815+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.816708+0000) 2019-10-13T04:38:45.820 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:45.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:45.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:45.812+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.813354+0000) 2019-10-13T04:38:45.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:45.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:45.877727+0000) 2019-10-13T04:38:47.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:47.370+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:47.371631+0000) 2019-10-13T04:38:48.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:48.162+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:48.163962+0000) 2019-10-13T04:38:49.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:49.573+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:49.574110+0000) 2019-10-13T04:38:50.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:50.817+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.819150+0000) 2019-10-13T04:38:50.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:50.817+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.819354+0000) 2019-10-13T04:38:50.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:50.814+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.815154+0000) 2019-10-13T04:38:50.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:50.818+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.819585+0000) 2019-10-13T04:38:50.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:50.818+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.820120+0000) 2019-10-13T04:38:50.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:50.818+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.820293+0000) 2019-10-13T04:38:50.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:50.819+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.820647+0000) 2019-10-13T04:38:50.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:50.820+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:50.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:50.816+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:50.817491+0000) 2019-10-13T04:38:51.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:51.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:51.755427+0000) 2019-10-13T04:38:51.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:51.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:51.755424+0000) 2019-10-13T04:38:51.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:51.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:51.755440+0000) 2019-10-13T04:38:52.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:52.628+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:52.628806+0000) 2019-10-13T04:38:55.415 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:38:55.415 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:55.427 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:38:55.427 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:55.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:38:55.818+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.819194+0000) 2019-10-13T04:38:55.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:38:55.822+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.823701+0000) 2019-10-13T04:38:55.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:55.822+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.824115+0000) 2019-10-13T04:38:55.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:38:55.822+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.824289+0000) 2019-10-13T04:38:55.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:38:55.823+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.824458+0000) 2019-10-13T04:38:55.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:55.823+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.824847+0000) 2019-10-13T04:38:55.829 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:55.822+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.823575+0000) 2019-10-13T04:38:55.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:38:55.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60318 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:38:55.830 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:55.819+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.820513+0000) 2019-10-13T04:38:55.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:38:55.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:55.877891+0000) 2019-10-13T04:38:57.374 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:38:57.371+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:57.371796+0000) 2019-10-13T04:38:58.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:38:58.163+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:58.164153+0000) 2019-10-13T04:38:59.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:38:59.575+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:38:59.576442+0000) 2019-10-13T04:39:00.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:00.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.822453+0000) 2019-10-13T04:39:00.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:00.825+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.827259+0000) 2019-10-13T04:39:00.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:00.826+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.827583+0000) 2019-10-13T04:39:00.830 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:00.826+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.827858+0000) 2019-10-13T04:39:00.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:00.826+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.827764+0000) 2019-10-13T04:39:00.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:00.826+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.827953+0000) 2019-10-13T04:39:00.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:00.826+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.828044+0000) 2019-10-13T04:39:00.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:00.824+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:00.824919+0000) 2019-10-13T04:39:00.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:00.827+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:01.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:01.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:01.755550+0000) 2019-10-13T04:39:01.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:01.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:01.755557+0000) 2019-10-13T04:39:01.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:01.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:01.755567+0000) 2019-10-13T04:39:02.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:02.628+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:02.628978+0000) 2019-10-13T04:39:05.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:05.825+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.825867+0000) 2019-10-13T04:39:05.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:05.829+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.830784+0000) 2019-10-13T04:39:05.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:05.829+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.830932+0000) 2019-10-13T04:39:05.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:05.830+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.831553+0000) 2019-10-13T04:39:05.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:05.830+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.831694+0000) 2019-10-13T04:39:05.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:05.830+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.831977+0000) 2019-10-13T04:39:05.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:05.830+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.832303+0000) 2019-10-13T04:39:05.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:05.827+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.828452+0000) 2019-10-13T04:39:05.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:05.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60381 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:05.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:05.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:05.878028+0000) 2019-10-13T04:39:07.374 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:07.371+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:07.371990+0000) 2019-10-13T04:39:08.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:08.163+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:08.164341+0000) 2019-10-13T04:39:09.585 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:09.578+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:09.578957+0000) 2019-10-13T04:39:10.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:10.834+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.834653+0000) 2019-10-13T04:39:10.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:10.834+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.835064+0000) 2019-10-13T04:39:10.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:10.834+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.835241+0000) 2019-10-13T04:39:10.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:10.829+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.830224+0000) 2019-10-13T04:39:10.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:10.835+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.835618+0000) 2019-10-13T04:39:10.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:10.835+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.835677+0000) 2019-10-13T04:39:10.844 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:10.835+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.836109+0000) 2019-10-13T04:39:10.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:10.831+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:10.831867+0000) 2019-10-13T04:39:10.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:10.837+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:11.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:11.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:11.755741+0000) 2019-10-13T04:39:11.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:11.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:11.755741+0000) 2019-10-13T04:39:11.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:11.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:11.755708+0000) 2019-10-13T04:39:12.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:12.629+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:12.629157+0000) 2019-10-13T04:39:15.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:15.839+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.839411+0000) 2019-10-13T04:39:15.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:15.839+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.839668+0000) 2019-10-13T04:39:15.842 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:15.839+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.839908+0000) 2019-10-13T04:39:15.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:15.833+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.835327+0000) 2019-10-13T04:39:15.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:15.839+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.840265+0000) 2019-10-13T04:39:15.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:15.840+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.840390+0000) 2019-10-13T04:39:15.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:15.840+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.840567+0000) 2019-10-13T04:39:15.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:15.835+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.837553+0000) 2019-10-13T04:39:15.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:15.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:15.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:15.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:15.878153+0000) 2019-10-13T04:39:17.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:17.371+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:17.372163+0000) 2019-10-13T04:39:18.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:18.164+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:18.164525+0000) 2019-10-13T04:39:19.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:19.580+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:19.581208+0000) 2019-10-13T04:39:20.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:20.844+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.844771+0000) 2019-10-13T04:39:20.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:20.844+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.844916+0000) 2019-10-13T04:39:20.848 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:20.844+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.845126+0000) 2019-10-13T04:39:20.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:20.844+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.845307+0000) 2019-10-13T04:39:20.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:20.845+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.845634+0000) 2019-10-13T04:39:20.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:20.839+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.840817+0000) 2019-10-13T04:39:20.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:20.839+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.841492+0000) 2019-10-13T04:39:20.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:20.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:20.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:20.845+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:20.846209+0000) 2019-10-13T04:39:21.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:21.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:21.755879+0000) 2019-10-13T04:39:21.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:21.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:21.755888+0000) 2019-10-13T04:39:21.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:21.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:21.755889+0000) 2019-10-13T04:39:22.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:22.628+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:22.629336+0000) 2019-10-13T04:39:25.567 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:39:25.567 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:25.571 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:39:25.571 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:25.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:25.846+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.847547+0000) 2019-10-13T04:39:25.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:25.846+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.847701+0000) 2019-10-13T04:39:25.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:25.842+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.843860+0000) 2019-10-13T04:39:25.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:25.846+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.847993+0000) 2019-10-13T04:39:25.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:25.847+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.848462+0000) 2019-10-13T04:39:25.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:25.844+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.846385+0000) 2019-10-13T04:39:25.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:25.847+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.848696+0000) 2019-10-13T04:39:25.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:25.847+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.848865+0000) 2019-10-13T04:39:25.866 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:25.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:25.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:25.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:25.878327+0000) 2019-10-13T04:39:27.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:27.370+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:27.372357+0000) 2019-10-13T04:39:28.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:28.163+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:28.164687+0000) 2019-10-13T04:39:29.591 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:29.583+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:29.583857+0000) 2019-10-13T04:39:30.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:30.850+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.851481+0000) 2019-10-13T04:39:30.855 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:30.850+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.851594+0000) 2019-10-13T04:39:30.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:30.850+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.851994+0000) 2019-10-13T04:39:30.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:30.850+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.852078+0000) 2019-10-13T04:39:30.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:30.850+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.852349+0000) 2019-10-13T04:39:30.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:30.851+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.852526+0000) 2019-10-13T04:39:30.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:30.846+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.847559+0000) 2019-10-13T04:39:30.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:30.848+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:30.849236+0000) 2019-10-13T04:39:30.857 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:30.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60534 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:31.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:31.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:31.756042+0000) 2019-10-13T04:39:31.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:31.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:31.756042+0000) 2019-10-13T04:39:31.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:31.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:31.756042+0000) 2019-10-13T04:39:32.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:32.628+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:32.629505+0000) 2019-10-13T04:39:35.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:35.853+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.855167+0000) 2019-10-13T04:39:35.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:35.853+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.855282+0000) 2019-10-13T04:39:35.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:35.850+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.851366+0000) 2019-10-13T04:39:35.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:35.854+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.855920+0000) 2019-10-13T04:39:35.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:35.854+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.855802+0000) 2019-10-13T04:39:35.859 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:35.854+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.856165+0000) 2019-10-13T04:39:35.859 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:35.854+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.856284+0000) 2019-10-13T04:39:35.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:35.852+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.852960+0000) 2019-10-13T04:39:35.860 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:35.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60565 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:35.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:35.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:35.878486+0000) 2019-10-13T04:39:37.374 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:37.371+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:37.372474+0000) 2019-10-13T04:39:38.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:38.164+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:38.164844+0000) 2019-10-13T04:39:39.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:39.585+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:39.586286+0000) 2019-10-13T04:39:40.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:40.855+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.855832+0000) 2019-10-13T04:39:40.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:40.858+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.859556+0000) 2019-10-13T04:39:40.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:40.858+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.859674+0000) 2019-10-13T04:39:40.864 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:40.858+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.859807+0000) 2019-10-13T04:39:40.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:40.858+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.860057+0000) 2019-10-13T04:39:40.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:40.858+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.860166+0000) 2019-10-13T04:39:40.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:40.859+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.861235+0000) 2019-10-13T04:39:40.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:40.857+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:40.858107+0000) 2019-10-13T04:39:40.865 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:40.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:41.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:41.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:41.756200+0000) 2019-10-13T04:39:41.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:41.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:41.756201+0000) 2019-10-13T04:39:41.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:41.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:41.756198+0000) 2019-10-13T04:39:42.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:42.629+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:42.629688+0000) 2019-10-13T04:39:45.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:45.862+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.863547+0000) 2019-10-13T04:39:45.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:45.862+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.863833+0000) 2019-10-13T04:39:45.866 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:45.862+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.863932+0000) 2019-10-13T04:39:45.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:45.858+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.859480+0000) 2019-10-13T04:39:45.867 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:45.863+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.864422+0000) 2019-10-13T04:39:45.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:45.863+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.864712+0000) 2019-10-13T04:39:45.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:45.863+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.864852+0000) 2019-10-13T04:39:45.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:45.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:45.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:45.861+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.862208+0000) 2019-10-13T04:39:45.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:45.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:45.878665+0000) 2019-10-13T04:39:47.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:47.372+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:47.372672+0000) 2019-10-13T04:39:48.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:48.164+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:48.165046+0000) 2019-10-13T04:39:49.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:49.587+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:49.588568+0000) 2019-10-13T04:39:50.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:50.861+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.862226+0000) 2019-10-13T04:39:50.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:50.865+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.867094+0000) 2019-10-13T04:39:50.870 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:50.865+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.867279+0000) 2019-10-13T04:39:50.870 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:50.866+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.867498+0000) 2019-10-13T04:39:50.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:50.866+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.867590+0000) 2019-10-13T04:39:50.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:50.866+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.867790+0000) 2019-10-13T04:39:50.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:50.866+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.868241+0000) 2019-10-13T04:39:50.871 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:50.867+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60654 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:50.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:50.864+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:50.865548+0000) 2019-10-13T04:39:51.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:51.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:51.756387+0000) 2019-10-13T04:39:51.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:51.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:51.756386+0000) 2019-10-13T04:39:51.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:51.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:51.756386+0000) 2019-10-13T04:39:52.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:52.629+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:52.629888+0000) 2019-10-13T04:39:55.703 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:39:55.703 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:55.707 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:39:55.707 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:55.873 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:55.870+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.870598+0000) 2019-10-13T04:39:55.873 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:39:55.870+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.870735+0000) 2019-10-13T04:39:55.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:55.870+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.871093+0000) 2019-10-13T04:39:55.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:39:55.865+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.866241+0000) 2019-10-13T04:39:55.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:39:55.870+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.871376+0000) 2019-10-13T04:39:55.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:55.871+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.871535+0000) 2019-10-13T04:39:55.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:39:55.871+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.871678+0000) 2019-10-13T04:39:55.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:55.866+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.867727+0000) 2019-10-13T04:39:55.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:39:55.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:39:55.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:39:55.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:55.878805+0000) 2019-10-13T04:39:57.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:39:57.372+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:57.372866+0000) 2019-10-13T04:39:58.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:39:58.164+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:58.165230+0000) 2019-10-13T04:39:59.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:39:59.590+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:39:59.591088+0000) 2019-10-13T04:40:00.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:00.868+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.869591+0000) 2019-10-13T04:40:00.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:00.874+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.874640+0000) 2019-10-13T04:40:00.877 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:00.874+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.874538+0000) 2019-10-13T04:40:00.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:00.874+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.874832+0000) 2019-10-13T04:40:00.878 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:00.875+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.875525+0000) 2019-10-13T04:40:00.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:00.875+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.875597+0000) 2019-10-13T04:40:00.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:00.875+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.875787+0000) 2019-10-13T04:40:00.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:00.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:00.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:00.873+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:00.874543+0000) 2019-10-13T04:40:01.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:01.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:01.756549+0000) 2019-10-13T04:40:01.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:01.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:01.756549+0000) 2019-10-13T04:40:01.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:01.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:01.756549+0000) 2019-10-13T04:40:02.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:02.629+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:02.630067+0000) 2019-10-13T04:40:05.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:05.878+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.878970+0000) 2019-10-13T04:40:05.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:05.878+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.878901+0000) 2019-10-13T04:40:05.881 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:05.879+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.879424+0000) 2019-10-13T04:40:05.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:05.872+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.874469+0000) 2019-10-13T04:40:05.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:05.879+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.879674+0000) 2019-10-13T04:40:05.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:05.879+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.879831+0000) 2019-10-13T04:40:05.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:05.874+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.876084+0000) 2019-10-13T04:40:05.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:05.880+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:05.880947+0000) 2019-10-13T04:40:05.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:05.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:07.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:07.372+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:07.373041+0000) 2019-10-13T04:40:08.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:08.165+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:08.165439+0000) 2019-10-13T04:40:09.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:09.591+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:09.593609+0000) 2019-10-13T04:40:10.885 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:10.881+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.883046+0000) 2019-10-13T04:40:10.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:10.881+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.883183+0000) 2019-10-13T04:40:10.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:10.877+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.879157+0000) 2019-10-13T04:40:10.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:10.882+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.883746+0000) 2019-10-13T04:40:10.887 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:10.883+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.884426+0000) 2019-10-13T04:40:10.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:10.883+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.884642+0000) 2019-10-13T04:40:10.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:10.883+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.884694+0000) 2019-10-13T04:40:10.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:10.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60772 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:10.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:10.879+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:10.881432+0000) 2019-10-13T04:40:11.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:11.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:11.756738+0000) 2019-10-13T04:40:11.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:11.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:11.756741+0000) 2019-10-13T04:40:11.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:11.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:11.756743+0000) 2019-10-13T04:40:12.631 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:12.628+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:12.629183+0000) 2019-10-13T04:40:12.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:12.629+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:12.630278+0000) 2019-10-13T04:40:15.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:15.881+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.883081+0000) 2019-10-13T04:40:15.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:15.886+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.888269+0000) 2019-10-13T04:40:15.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:15.887+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.888438+0000) 2019-10-13T04:40:15.894 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:15.887+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.888757+0000) 2019-10-13T04:40:15.894 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:15.887+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.888664+0000) 2019-10-13T04:40:15.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:15.887+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.888889+0000) 2019-10-13T04:40:15.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:15.887+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.888963+0000) 2019-10-13T04:40:15.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:15.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:15.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:15.883+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:15.885300+0000) 2019-10-13T04:40:17.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:17.372+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:17.373233+0000) 2019-10-13T04:40:18.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:18.164+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:18.165614+0000) 2019-10-13T04:40:19.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:19.595+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:19.596080+0000) 2019-10-13T04:40:20.894 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:20.890+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.891900+0000) 2019-10-13T04:40:20.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:20.890+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.892268+0000) 2019-10-13T04:40:20.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:20.891+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.892422+0000) 2019-10-13T04:40:20.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:20.886+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.887399+0000) 2019-10-13T04:40:20.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:20.891+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.892636+0000) 2019-10-13T04:40:20.896 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:20.891+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.892894+0000) 2019-10-13T04:40:20.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:20.891+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.893283+0000) 2019-10-13T04:40:20.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:20.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60832 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:20.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:20.890+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:20.890977+0000) 2019-10-13T04:40:21.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:21.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:21.756922+0000) 2019-10-13T04:40:21.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:21.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:21.756923+0000) 2019-10-13T04:40:21.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:21.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:21.756924+0000) 2019-10-13T04:40:22.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:22.629+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:22.630421+0000) 2019-10-13T04:40:22.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:22.628+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:22.629408+0000) 2019-10-13T04:40:25.839 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:40:25.840 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:25.844 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:40:25.844 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:25.899 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:25.894+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.896165+0000) 2019-10-13T04:40:25.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:25.895+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.896507+0000) 2019-10-13T04:40:25.899 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:25.895+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.896618+0000) 2019-10-13T04:40:25.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:25.895+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.896842+0000) 2019-10-13T04:40:25.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:25.890+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.891186+0000) 2019-10-13T04:40:25.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:25.895+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.897237+0000) 2019-10-13T04:40:25.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:25.895+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.897351+0000) 2019-10-13T04:40:25.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:25.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:25.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:25.892+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:25.893306+0000) 2019-10-13T04:40:27.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:27.372+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:27.373436+0000) 2019-10-13T04:40:28.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:28.164+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:28.165797+0000) 2019-10-13T04:40:29.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:29.598+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:29.599074+0000) 2019-10-13T04:40:30.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:30.894+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.895375+0000) 2019-10-13T04:40:30.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:30.898+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.900103+0000) 2019-10-13T04:40:30.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:30.898+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.900227+0000) 2019-10-13T04:40:30.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:30.899+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.900718+0000) 2019-10-13T04:40:30.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:30.899+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.900626+0000) 2019-10-13T04:40:30.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:30.899+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.901029+0000) 2019-10-13T04:40:30.904 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:30.899+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.901335+0000) 2019-10-13T04:40:30.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:30.897+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:30.897766+0000) 2019-10-13T04:40:30.905 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:30.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:31.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:31.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:31.757082+0000) 2019-10-13T04:40:31.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:31.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:31.757083+0000) 2019-10-13T04:40:31.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:31.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:31.757084+0000) 2019-10-13T04:40:32.631 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:32.629+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:32.629565+0000) 2019-10-13T04:40:32.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:32.630+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:32.630585+0000) 2019-10-13T04:40:35.907 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:35.902+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.904156+0000) 2019-10-13T04:40:35.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:35.902+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.904262+0000) 2019-10-13T04:40:35.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:35.903+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.904438+0000) 2019-10-13T04:40:35.907 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:35.903+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.904845+0000) 2019-10-13T04:40:35.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:35.903+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.905019+0000) 2019-10-13T04:40:35.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:35.899+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.899934+0000) 2019-10-13T04:40:35.908 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:35.900+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.901386+0000) 2019-10-13T04:40:35.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:35.904+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:35.905703+0000) 2019-10-13T04:40:35.909 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:35.905+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:37.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:37.373+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:37.373616+0000) 2019-10-13T04:40:38.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:38.165+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:38.165975+0000) 2019-10-13T04:40:39.608 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:39.600+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:39.601538+0000) 2019-10-13T04:40:40.910 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:40.907+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.908276+0000) 2019-10-13T04:40:40.911 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:40.908+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.908481+0000) 2019-10-13T04:40:40.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:40.903+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.904244+0000) 2019-10-13T04:40:40.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:40.908+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.908967+0000) 2019-10-13T04:40:40.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:40.909+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.909390+0000) 2019-10-13T04:40:40.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:40.909+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.909515+0000) 2019-10-13T04:40:40.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:40.909+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.909836+0000) 2019-10-13T04:40:40.913 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:40.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:40.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:40.905+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:40.906595+0000) 2019-10-13T04:40:41.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:41.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:41.757215+0000) 2019-10-13T04:40:41.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:41.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:41.757214+0000) 2019-10-13T04:40:41.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:41.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:41.757214+0000) 2019-10-13T04:40:42.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:42.629+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:42.629706+0000) 2019-10-13T04:40:42.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:42.630+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:42.630765+0000) 2019-10-13T04:40:45.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:45.912+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.912936+0000) 2019-10-13T04:40:45.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:45.912+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.913012+0000) 2019-10-13T04:40:45.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:45.912+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.913188+0000) 2019-10-13T04:40:45.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:45.912+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.913316+0000) 2019-10-13T04:40:45.916 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:45.913+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.913669+0000) 2019-10-13T04:40:45.916 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:45.913+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.913841+0000) 2019-10-13T04:40:45.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:45.908+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.908823+0000) 2019-10-13T04:40:45.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:45.909+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:45.910277+0000) 2019-10-13T04:40:45.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:45.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 60979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:47.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:47.373+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:47.373795+0000) 2019-10-13T04:40:48.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:48.165+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:48.166146+0000) 2019-10-13T04:40:49.610 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:49.603+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:49.603933+0000) 2019-10-13T04:40:50.919 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:50.916+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.916865+0000) 2019-10-13T04:40:50.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:50.916+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.917174+0000) 2019-10-13T04:40:50.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:50.916+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.917333+0000) 2019-10-13T04:40:50.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:50.912+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.912833+0000) 2019-10-13T04:40:50.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:50.917+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.917593+0000) 2019-10-13T04:40:50.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:50.917+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.917767+0000) 2019-10-13T04:40:50.921 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:50.917+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.918075+0000) 2019-10-13T04:40:50.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:50.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61008 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:50.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:50.915+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:50.916713+0000) 2019-10-13T04:40:51.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:51.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:51.757380+0000) 2019-10-13T04:40:51.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:51.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:51.757384+0000) 2019-10-13T04:40:51.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:51.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:51.757380+0000) 2019-10-13T04:40:52.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:52.629+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:52.629878+0000) 2019-10-13T04:40:52.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:52.630+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:52.630939+0000) 2019-10-13T04:40:55.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:40:55.920+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.921543+0000) 2019-10-13T04:40:55.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:40:55.920+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.921777+0000) 2019-10-13T04:40:55.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:40:55.920+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.921892+0000) 2019-10-13T04:40:55.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:40:55.920+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.922039+0000) 2019-10-13T04:40:55.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:55.920+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.922177+0000) 2019-10-13T04:40:55.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:40:55.917+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.917754+0000) 2019-10-13T04:40:55.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:55.921+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.922976+0000) 2019-10-13T04:40:55.926 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:40:55.922+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:40:55.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:55.919+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:55.920068+0000) 2019-10-13T04:40:56.028 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:40:56.029 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:56.033 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:40:56.033 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:57.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:40:57.372+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:57.373987+0000) 2019-10-13T04:40:58.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:40:58.165+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:58.166340+0000) 2019-10-13T04:40:59.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:40:59.605+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:40:59.606064+0000) 2019-10-13T04:41:00.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:00.924+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.925813+0000) 2019-10-13T04:41:00.928 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:00.924+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.926142+0000) 2019-10-13T04:41:00.928 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:00.924+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.926296+0000) 2019-10-13T04:41:00.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:00.920+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.922375+0000) 2019-10-13T04:41:00.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:00.925+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.926511+0000) 2019-10-13T04:41:00.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:00.926+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.927431+0000) 2019-10-13T04:41:00.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:00.926+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.927560+0000) 2019-10-13T04:41:00.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:00.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:00.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:00.922+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:00.924593+0000) 2019-10-13T04:41:01.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:01.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:01.757511+0000) 2019-10-13T04:41:01.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:01.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:01.757529+0000) 2019-10-13T04:41:01.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:01.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:01.757499+0000) 2019-10-13T04:41:02.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:02.629+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:02.630062+0000) 2019-10-13T04:41:02.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:02.630+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:02.631120+0000) 2019-10-13T04:41:05.932 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:05.928+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.929854+0000) 2019-10-13T04:41:05.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:05.928+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.930056+0000) 2019-10-13T04:41:05.933 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:05.929+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.930418+0000) 2019-10-13T04:41:05.933 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:05.929+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.930520+0000) 2019-10-13T04:41:05.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:05.924+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.926069+0000) 2019-10-13T04:41:05.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:05.929+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.931011+0000) 2019-10-13T04:41:05.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:05.929+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.931147+0000) 2019-10-13T04:41:05.935 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:05.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:05.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:05.927+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:05.929114+0000) 2019-10-13T04:41:07.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:07.373+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:07.374178+0000) 2019-10-13T04:41:08.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:08.165+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:08.166521+0000) 2019-10-13T04:41:09.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:09.607+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:09.608287+0000) 2019-10-13T04:41:10.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:10.927+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.929674+0000) 2019-10-13T04:41:10.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:10.932+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.934378+0000) 2019-10-13T04:41:10.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:10.933+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.934640+0000) 2019-10-13T04:41:10.940 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:10.933+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.934826+0000) 2019-10-13T04:41:10.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:10.933+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.935015+0000) 2019-10-13T04:41:10.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:10.933+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.935149+0000) 2019-10-13T04:41:10.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:10.933+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.935270+0000) 2019-10-13T04:41:10.941 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:10.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:10.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:10.931+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:10.933165+0000) 2019-10-13T04:41:11.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:11.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:11.757703+0000) 2019-10-13T04:41:11.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:11.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:11.757706+0000) 2019-10-13T04:41:11.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:11.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:11.757702+0000) 2019-10-13T04:41:12.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:12.629+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:12.630242+0000) 2019-10-13T04:41:12.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:12.630+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:12.631273+0000) 2019-10-13T04:41:15.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:15.936+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.937985+0000) 2019-10-13T04:41:15.941 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:15.936+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.938232+0000) 2019-10-13T04:41:15.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:15.932+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.933716+0000) 2019-10-13T04:41:15.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:15.937+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.938713+0000) 2019-10-13T04:41:15.943 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:15.937+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.939034+0000) 2019-10-13T04:41:15.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:15.937+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.939179+0000) 2019-10-13T04:41:15.944 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:15.937+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.939280+0000) 2019-10-13T04:41:15.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:15.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:15.944 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:15.935+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:15.935976+0000) 2019-10-13T04:41:17.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:17.372+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:17.374408+0000) 2019-10-13T04:41:18.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:18.166+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:18.166698+0000) 2019-10-13T04:41:19.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:19.610+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:19.610721+0000) 2019-10-13T04:41:20.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:20.936+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.937688+0000) 2019-10-13T04:41:20.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:20.940+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.942179+0000) 2019-10-13T04:41:20.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:20.941+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.942416+0000) 2019-10-13T04:41:20.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:20.941+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.942737+0000) 2019-10-13T04:41:20.946 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:20.941+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.942730+0000) 2019-10-13T04:41:20.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:20.941+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.942903+0000) 2019-10-13T04:41:20.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:20.941+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.943063+0000) 2019-10-13T04:41:20.946 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:20.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:20.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:20.939+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:20.940658+0000) 2019-10-13T04:41:21.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:21.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:21.757869+0000) 2019-10-13T04:41:21.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:21.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:21.757874+0000) 2019-10-13T04:41:21.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:21.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:21.757869+0000) 2019-10-13T04:41:22.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:22.630+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:22.630402+0000) 2019-10-13T04:41:22.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:22.630+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:22.631410+0000) 2019-10-13T04:41:25.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:25.945+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.945833+0000) 2019-10-13T04:41:25.948 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:25.945+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.945560+0000) 2019-10-13T04:41:25.949 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:25.945+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.946039+0000) 2019-10-13T04:41:25.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:25.945+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.946122+0000) 2019-10-13T04:41:25.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:25.941+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.942352+0000) 2019-10-13T04:41:25.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:25.946+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.947058+0000) 2019-10-13T04:41:25.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:25.946+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.947388+0000) 2019-10-13T04:41:25.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:25.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:25.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:25.943+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:25.944708+0000) 2019-10-13T04:41:26.201 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:41:26.202 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:26.206 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:41:26.206 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:27.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:27.374+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:27.374604+0000) 2019-10-13T04:41:28.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:28.166+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:28.166898+0000) 2019-10-13T04:41:29.619 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:29.612+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:29.612939+0000) 2019-10-13T04:41:30.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:30.949+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.950054+0000) 2019-10-13T04:41:30.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:30.949+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.950381+0000) 2019-10-13T04:41:30.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:30.950+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.950528+0000) 2019-10-13T04:41:30.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:30.945+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.946405+0000) 2019-10-13T04:41:30.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:30.950+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.950854+0000) 2019-10-13T04:41:30.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:30.950+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.951119+0000) 2019-10-13T04:41:30.955 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:30.951+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.951412+0000) 2019-10-13T04:41:30.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:30.952+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:30.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:30.947+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:30.948723+0000) 2019-10-13T04:41:31.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:31.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:31.758087+0000) 2019-10-13T04:41:31.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:31.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:31.758087+0000) 2019-10-13T04:41:31.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:31.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:31.758086+0000) 2019-10-13T04:41:32.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:32.630+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:32.630576+0000) 2019-10-13T04:41:32.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:32.631+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:32.631552+0000) 2019-10-13T04:41:35.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:35.953+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.954274+0000) 2019-10-13T04:41:35.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:35.954+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.954585+0000) 2019-10-13T04:41:35.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:35.954+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.954761+0000) 2019-10-13T04:41:35.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:35.954+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.954938+0000) 2019-10-13T04:41:35.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:35.954+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.955269+0000) 2019-10-13T04:41:35.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:35.949+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.949859+0000) 2019-10-13T04:41:35.960 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:35.955+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.955792+0000) 2019-10-13T04:41:35.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:35.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:35.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:35.952+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:35.953370+0000) 2019-10-13T04:41:37.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:37.374+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:37.374797+0000) 2019-10-13T04:41:38.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:38.167+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:38.167110+0000) 2019-10-13T04:41:39.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:39.614+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:39.615522+0000) 2019-10-13T04:41:40.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:40.957+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.958702+0000) 2019-10-13T04:41:40.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:40.957+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.958894+0000) 2019-10-13T04:41:40.961 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:40.957+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.959175+0000) 2019-10-13T04:41:40.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:40.954+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.955264+0000) 2019-10-13T04:41:40.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:40.958+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.959437+0000) 2019-10-13T04:41:40.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:40.958+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.959547+0000) 2019-10-13T04:41:40.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:40.958+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.960221+0000) 2019-10-13T04:41:40.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:40.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:40.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:40.956+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:40.957413+0000) 2019-10-13T04:41:41.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:41.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:41.758245+0000) 2019-10-13T04:41:41.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:41.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:41.758248+0000) 2019-10-13T04:41:41.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:41.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:41.758243+0000) 2019-10-13T04:41:42.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:42.629+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:42.630744+0000) 2019-10-13T04:41:42.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:42.631+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:42.631718+0000) 2019-10-13T04:41:45.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:45.961+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.963095+0000) 2019-10-13T04:41:45.965 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:45.961+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.963256+0000) 2019-10-13T04:41:45.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:45.962+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.963456+0000) 2019-10-13T04:41:45.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:45.958+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.959422+0000) 2019-10-13T04:41:45.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:45.962+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.963646+0000) 2019-10-13T04:41:45.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:45.962+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.964251+0000) 2019-10-13T04:41:45.967 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:45.963+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.964515+0000) 2019-10-13T04:41:45.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:45.959+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:45.960466+0000) 2019-10-13T04:41:45.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:45.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:47.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:47.374+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:47.374986+0000) 2019-10-13T04:41:48.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:48.166+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:48.167334+0000) 2019-10-13T04:41:49.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:49.617+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:49.617903+0000) 2019-10-13T04:41:50.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:50.965+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.967050+0000) 2019-10-13T04:41:50.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:50.965+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.967363+0000) 2019-10-13T04:41:50.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:50.962+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.962923+0000) 2019-10-13T04:41:50.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:50.966+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.967835+0000) 2019-10-13T04:41:50.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:50.966+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.967933+0000) 2019-10-13T04:41:50.972 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:50.966+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.968328+0000) 2019-10-13T04:41:50.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:50.966+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.968363+0000) 2019-10-13T04:41:50.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:50.965+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:50.965758+0000) 2019-10-13T04:41:50.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:50.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:51.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:51.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:51.758445+0000) 2019-10-13T04:41:51.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:51.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:51.758446+0000) 2019-10-13T04:41:51.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:51.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:51.758446+0000) 2019-10-13T04:41:52.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:52.630+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:52.630856+0000) 2019-10-13T04:41:52.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:52.631+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:52.631921+0000) 2019-10-13T04:41:55.973 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:55.969+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.971105+0000) 2019-10-13T04:41:55.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:41:55.964+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.966623+0000) 2019-10-13T04:41:55.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:41:55.970+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.972344+0000) 2019-10-13T04:41:55.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:41:55.971+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.972883+0000) 2019-10-13T04:41:55.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:41:55.971+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.972750+0000) 2019-10-13T04:41:55.976 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:55.971+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.973188+0000) 2019-10-13T04:41:55.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:41:55.971+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.973354+0000) 2019-10-13T04:41:55.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:55.967+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:55.968855+0000) 2019-10-13T04:41:55.977 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:41:55.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:41:56.339 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:41:56.340 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:56.344 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:41:56.344 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:57.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:41:57.374+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:57.375199+0000) 2019-10-13T04:41:58.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:41:58.166+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:58.167518+0000) 2019-10-13T04:41:59.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:41:59.619+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:41:59.620876+0000) 2019-10-13T04:42:00.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:00.974+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.975534+0000) 2019-10-13T04:42:00.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:00.969+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.971667+0000) 2019-10-13T04:42:00.979 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:00.975+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.976391+0000) 2019-10-13T04:42:00.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:00.975+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.976717+0000) 2019-10-13T04:42:00.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:00.975+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.976903+0000) 2019-10-13T04:42:00.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:00.975+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.977022+0000) 2019-10-13T04:42:00.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:00.975+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.977252+0000) 2019-10-13T04:42:00.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:00.971+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:00.973632+0000) 2019-10-13T04:42:00.981 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:00.976+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:01.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:01.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:01.758658+0000) 2019-10-13T04:42:01.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:01.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:01.758658+0000) 2019-10-13T04:42:01.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:01.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:01.758658+0000) 2019-10-13T04:42:02.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:02.630+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:02.631016+0000) 2019-10-13T04:42:02.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:02.631+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:02.632121+0000) 2019-10-13T04:42:05.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:05.978+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.979644+0000) 2019-10-13T04:42:05.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:05.974+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.975739+0000) 2019-10-13T04:42:05.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:05.978+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.979990+0000) 2019-10-13T04:42:05.983 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:05.978+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.980240+0000) 2019-10-13T04:42:05.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:05.979+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.980414+0000) 2019-10-13T04:42:05.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:05.979+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.980555+0000) 2019-10-13T04:42:05.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:05.979+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.980692+0000) 2019-10-13T04:42:05.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:05.976+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:05.978169+0000) 2019-10-13T04:42:05.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:05.980+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61451 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:07.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:07.374+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:07.375393+0000) 2019-10-13T04:42:08.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:08.167+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:08.167700+0000) 2019-10-13T04:42:09.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:09.622+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:09.623307+0000) 2019-10-13T04:42:10.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:10.982+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.983282+0000) 2019-10-13T04:42:10.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:10.983+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.983452+0000) 2019-10-13T04:42:10.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:10.983+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.983596+0000) 2019-10-13T04:42:10.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:10.983+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.983980+0000) 2019-10-13T04:42:10.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:10.983+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.984005+0000) 2019-10-13T04:42:10.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:10.978+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.979573+0000) 2019-10-13T04:42:10.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:10.984+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.984721+0000) 2019-10-13T04:42:10.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:10.985+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:10.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:10.981+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:10.981804+0000) 2019-10-13T04:42:11.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:11.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:11.758848+0000) 2019-10-13T04:42:11.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:11.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:11.758848+0000) 2019-10-13T04:42:11.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:11.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:11.758849+0000) 2019-10-13T04:42:12.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:12.630+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:12.631182+0000) 2019-10-13T04:42:12.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:12.631+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:12.632246+0000) 2019-10-13T04:42:15.989 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:15.987+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.987456+0000) 2019-10-13T04:42:15.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:15.982+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.983097+0000) 2019-10-13T04:42:15.991 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:15.988+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.988652+0000) 2019-10-13T04:42:15.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:15.988+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.988801+0000) 2019-10-13T04:42:15.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:15.988+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.988946+0000) 2019-10-13T04:42:15.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:15.988+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.989078+0000) 2019-10-13T04:42:15.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:15.988+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.989224+0000) 2019-10-13T04:42:15.992 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:15.989+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:15.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:15.985+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:15.986034+0000) 2019-10-13T04:42:17.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:17.375+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:17.375594+0000) 2019-10-13T04:42:18.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:18.167+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:18.167893+0000) 2019-10-13T04:42:19.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:19.625+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:19.625753+0000) 2019-10-13T04:42:20.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:20.990+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.991311+0000) 2019-10-13T04:42:20.994 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:20.991+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.991730+0000) 2019-10-13T04:42:20.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:20.991+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.991806+0000) 2019-10-13T04:42:20.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:20.991+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.991970+0000) 2019-10-13T04:42:20.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:20.986+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.987603+0000) 2019-10-13T04:42:20.995 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:20.991+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.992220+0000) 2019-10-13T04:42:20.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:20.992+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.992684+0000) 2019-10-13T04:42:20.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:20.988+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:20.989213+0000) 2019-10-13T04:42:20.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:20.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61542 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:21.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:21.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:21.759035+0000) 2019-10-13T04:42:21.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:21.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:21.759035+0000) 2019-10-13T04:42:21.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:21.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:21.759037+0000) 2019-10-13T04:42:22.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:22.631+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:22.631363+0000) 2019-10-13T04:42:22.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:22.631+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:22.632411+0000) 2019-10-13T04:42:25.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:25.994+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.995523+0000) 2019-10-13T04:42:25.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:25.994+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.995407+0000) 2019-10-13T04:42:25.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:25.994+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.996025+0000) 2019-10-13T04:42:25.999 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:25.994+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.996209+0000) 2019-10-13T04:42:25.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:25.991+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.992233+0000) 2019-10-13T04:42:26.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:25.995+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.996447+0000) 2019-10-13T04:42:26.000 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:25.994+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.996256+0000) 2019-10-13T04:42:26.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:25.996+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61571 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:26.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:25.993+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:25.994264+0000) 2019-10-13T04:42:26.504 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:42:26.504 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:26.509 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:42:26.509 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:27.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:27.374+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:27.375784+0000) 2019-10-13T04:42:28.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:28.167+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:28.168088+0000) 2019-10-13T04:42:29.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:29.627+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:29.628161+0000) 2019-10-13T04:42:31.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:30.997+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:30.999340+0000) 2019-10-13T04:42:31.002 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:30.998+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:30.999766+0000) 2019-10-13T04:42:31.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:30.998+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:31.000086+0000) 2019-10-13T04:42:31.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:30.998+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:31.000376+0000) 2019-10-13T04:42:31.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:30.994+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:30.995691+0000) 2019-10-13T04:42:31.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:30.999+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:31.000573+0000) 2019-10-13T04:42:31.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:30.999+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:31.000690+0000) 2019-10-13T04:42:31.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:31.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61601 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:31.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:30.997+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:30.998242+0000) 2019-10-13T04:42:31.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:31.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:31.759195+0000) 2019-10-13T04:42:31.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:31.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:31.759193+0000) 2019-10-13T04:42:31.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:31.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:31.759194+0000) 2019-10-13T04:42:32.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:32.630+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:32.631502+0000) 2019-10-13T04:42:32.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:32.632+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:32.632578+0000) 2019-10-13T04:42:36.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:36.002+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.003866+0000) 2019-10-13T04:42:36.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:36.002+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.004097+0000) 2019-10-13T04:42:36.007 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:36.003+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.004461+0000) 2019-10-13T04:42:36.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:36.003+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.004545+0000) 2019-10-13T04:42:36.007 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:36.003+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.004791+0000) 2019-10-13T04:42:36.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:35.999+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.000461+0000) 2019-10-13T04:42:36.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:36.003+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.005301+0000) 2019-10-13T04:42:36.008 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:36.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61630 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:36.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:36.002+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:36.002806+0000) 2019-10-13T04:42:37.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:37.374+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:37.375919+0000) 2019-10-13T04:42:38.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:38.167+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:38.168224+0000) 2019-10-13T04:42:39.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:39.630+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:39.630431+0000) 2019-10-13T04:42:41.012 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:41.007+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.008405+0000) 2019-10-13T04:42:41.013 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:41.007+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.008655+0000) 2019-10-13T04:42:41.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:41.007+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.008772+0000) 2019-10-13T04:42:41.013 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:41.007+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.008956+0000) 2019-10-13T04:42:41.013 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:41.007+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.009139+0000) 2019-10-13T04:42:41.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:41.008+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.009515+0000) 2019-10-13T04:42:41.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:41.002+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.003672+0000) 2019-10-13T04:42:41.014 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:41.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61662 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:41.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:41.006+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.007302+0000) 2019-10-13T04:42:41.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:41.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.759368+0000) 2019-10-13T04:42:41.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:41.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.759369+0000) 2019-10-13T04:42:41.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:41.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:41.759368+0000) 2019-10-13T04:42:42.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:42.630+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:42.631620+0000) 2019-10-13T04:42:42.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:42.632+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:42.632790+0000) 2019-10-13T04:42:46.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:46.011+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.012446+0000) 2019-10-13T04:42:46.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:46.011+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.012824+0000) 2019-10-13T04:42:46.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:46.008+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.008956+0000) 2019-10-13T04:42:46.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:46.011+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.013196+0000) 2019-10-13T04:42:46.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:46.011+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.013295+0000) 2019-10-13T04:42:46.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:46.012+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.013689+0000) 2019-10-13T04:42:46.017 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:46.012+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.013844+0000) 2019-10-13T04:42:46.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:46.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:46.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:46.010+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:46.011575+0000) 2019-10-13T04:42:47.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:47.375+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:47.376080+0000) 2019-10-13T04:42:48.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:48.167+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:48.168410+0000) 2019-10-13T04:42:49.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:49.632+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:49.633402+0000) 2019-10-13T04:42:51.019 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:51.015+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.016961+0000) 2019-10-13T04:42:51.020 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:51.016+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.017562+0000) 2019-10-13T04:42:51.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:51.016+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.017760+0000) 2019-10-13T04:42:51.020 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:51.016+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.017858+0000) 2019-10-13T04:42:51.020 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:51.016+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.018025+0000) 2019-10-13T04:42:51.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:51.011+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.013697+0000) 2019-10-13T04:42:51.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:51.017+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.018437+0000) 2019-10-13T04:42:51.021 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:51.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:51.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:51.013+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.014973+0000) 2019-10-13T04:42:51.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:51.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.759585+0000) 2019-10-13T04:42:51.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:51.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.759576+0000) 2019-10-13T04:42:51.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:51.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:51.759566+0000) 2019-10-13T04:42:52.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:52.631+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:52.631752+0000) 2019-10-13T04:42:52.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:52.632+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:52.632966+0000) 2019-10-13T04:42:56.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:56.020+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.021339+0000) 2019-10-13T04:42:56.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:42:56.021+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.021537+0000) 2019-10-13T04:42:56.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:56.021+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.021824+0000) 2019-10-13T04:42:56.025 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:42:56.021+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.022025+0000) 2019-10-13T04:42:56.025 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:42:56.021+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.022157+0000) 2019-10-13T04:42:56.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:42:56.022+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.022408+0000) 2019-10-13T04:42:56.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:42:56.016+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.018253+0000) 2019-10-13T04:42:56.026 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:42:56.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61750 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:42:56.026 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:56.018+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:56.019971+0000) 2019-10-13T04:42:56.619 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:42:56.619 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:56.634 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:42:56.634 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:57.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:42:57.375+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:57.376272+0000) 2019-10-13T04:42:58.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:42:58.168+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:58.168624+0000) 2019-10-13T04:42:59.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:42:59.635+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:42:59.635991+0000) 2019-10-13T04:43:01.027 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:01.024+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.025235+0000) 2019-10-13T04:43:01.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:01.020+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.021026+0000) 2019-10-13T04:43:01.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:01.025+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.025793+0000) 2019-10-13T04:43:01.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:01.025+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.026264+0000) 2019-10-13T04:43:01.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:01.026+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.026457+0000) 2019-10-13T04:43:01.029 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:01.026+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.026624+0000) 2019-10-13T04:43:01.029 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:01.026+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.027097+0000) 2019-10-13T04:43:01.030 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:01.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:01.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:01.023+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.024856+0000) 2019-10-13T04:43:01.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:01.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.759763+0000) 2019-10-13T04:43:01.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:01.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.759765+0000) 2019-10-13T04:43:01.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:01.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:01.759760+0000) 2019-10-13T04:43:02.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:02.631+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:02.631950+0000) 2019-10-13T04:43:02.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:02.632+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:02.633145+0000) 2019-10-13T04:43:06.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:06.029+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.030253+0000) 2019-10-13T04:43:06.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:06.030+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.030673+0000) 2019-10-13T04:43:06.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:06.025+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.026702+0000) 2019-10-13T04:43:06.034 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:06.030+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.030756+0000) 2019-10-13T04:43:06.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:06.030+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.030913+0000) 2019-10-13T04:43:06.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:06.030+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.031354+0000) 2019-10-13T04:43:06.034 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:06.031+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.031688+0000) 2019-10-13T04:43:06.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:06.027+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:06.028108+0000) 2019-10-13T04:43:06.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:06.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:07.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:07.376+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:07.376472+0000) 2019-10-13T04:43:08.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:08.168+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:08.168816+0000) 2019-10-13T04:43:09.645 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:09.637+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:09.638293+0000) 2019-10-13T04:43:11.036 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:11.032+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.033918+0000) 2019-10-13T04:43:11.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:11.032+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.034070+0000) 2019-10-13T04:43:11.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:11.032+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.034209+0000) 2019-10-13T04:43:11.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:11.032+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.034367+0000) 2019-10-13T04:43:11.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:11.033+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.034537+0000) 2019-10-13T04:43:11.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:11.029+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.029838+0000) 2019-10-13T04:43:11.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:11.033+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.034857+0000) 2019-10-13T04:43:11.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:11.034+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:11.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:11.031+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.032580+0000) 2019-10-13T04:43:11.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:11.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.759939+0000) 2019-10-13T04:43:11.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:11.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.759942+0000) 2019-10-13T04:43:11.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:11.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:11.759958+0000) 2019-10-13T04:43:12.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:12.631+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:12.632170+0000) 2019-10-13T04:43:12.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:12.632+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:12.633326+0000) 2019-10-13T04:43:16.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:16.033+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.034054+0000) 2019-10-13T04:43:16.048 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:16.034+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.035317+0000) 2019-10-13T04:43:16.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:16.036+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.037799+0000) 2019-10-13T04:43:16.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:16.036+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.037942+0000) 2019-10-13T04:43:16.049 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:16.036+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.038176+0000) 2019-10-13T04:43:16.049 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:16.036+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.038066+0000) 2019-10-13T04:43:16.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:16.037+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.038651+0000) 2019-10-13T04:43:16.050 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:16.037+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:16.038867+0000) 2019-10-13T04:43:16.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:16.038+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61870 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:17.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:17.375+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:17.376660+0000) 2019-10-13T04:43:18.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:18.167+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:18.169002+0000) 2019-10-13T04:43:19.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:19.640+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:19.641403+0000) 2019-10-13T04:43:21.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:21.040+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.041606+0000) 2019-10-13T04:43:21.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:21.040+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.041731+0000) 2019-10-13T04:43:21.048 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:21.040+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.042060+0000) 2019-10-13T04:43:21.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:21.040+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.042298+0000) 2019-10-13T04:43:21.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:21.041+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.042457+0000) 2019-10-13T04:43:21.048 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:21.041+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.042498+0000) 2019-10-13T04:43:21.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:21.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61902 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:21.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:21.036+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.037652+0000) 2019-10-13T04:43:21.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:21.038+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.039336+0000) 2019-10-13T04:43:21.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:21.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.760113+0000) 2019-10-13T04:43:21.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:21.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.760113+0000) 2019-10-13T04:43:21.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:21.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:21.760112+0000) 2019-10-13T04:43:22.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:22.631+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:22.632307+0000) 2019-10-13T04:43:22.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:22.633+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:22.633465+0000) 2019-10-13T04:43:26.048 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:26.044+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.045465+0000) 2019-10-13T04:43:26.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:26.044+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.045669+0000) 2019-10-13T04:43:26.048 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:26.044+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.045919+0000) 2019-10-13T04:43:26.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:26.044+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.046076+0000) 2019-10-13T04:43:26.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:26.041+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.041975+0000) 2019-10-13T04:43:26.050 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:26.045+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.046566+0000) 2019-10-13T04:43:26.050 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:26.045+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.046698+0000) 2019-10-13T04:43:26.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:26.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:26.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:26.042+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:26.043435+0000) 2019-10-13T04:43:26.731 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:43:26.731 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:26.736 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:43:26.736 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:27.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:27.376+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:27.376822+0000) 2019-10-13T04:43:28.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:28.168+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:28.169154+0000) 2019-10-13T04:43:29.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:29.644+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:29.644395+0000) 2019-10-13T04:43:31.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:31.044+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.044830+0000) 2019-10-13T04:43:31.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:31.048+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.049682+0000) 2019-10-13T04:43:31.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:31.048+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.049582+0000) 2019-10-13T04:43:31.053 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:31.048+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.049969+0000) 2019-10-13T04:43:31.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:31.048+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.050323+0000) 2019-10-13T04:43:31.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:31.049+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.050563+0000) 2019-10-13T04:43:31.054 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:31.049+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.050738+0000) 2019-10-13T04:43:31.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:31.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:31.055 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:31.047+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.048290+0000) 2019-10-13T04:43:31.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:31.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.760280+0000) 2019-10-13T04:43:31.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:31.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.760280+0000) 2019-10-13T04:43:31.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:31.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:31.760279+0000) 2019-10-13T04:43:32.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:32.631+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:32.632502+0000) 2019-10-13T04:43:32.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:32.632+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:32.633664+0000) 2019-10-13T04:43:36.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:36.048+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.049110+0000) 2019-10-13T04:43:36.057 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:36.052+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.053818+0000) 2019-10-13T04:43:36.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:36.052+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.053877+0000) 2019-10-13T04:43:36.058 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:36.052+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.054027+0000) 2019-10-13T04:43:36.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:36.052+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.054190+0000) 2019-10-13T04:43:36.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:36.052+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.054377+0000) 2019-10-13T04:43:36.060 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:36.054+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.055402+0000) 2019-10-13T04:43:36.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:36.052+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:36.053022+0000) 2019-10-13T04:43:36.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:36.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 61990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:37.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:37.376+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:37.377004+0000) 2019-10-13T04:43:38.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:38.168+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:38.169370+0000) 2019-10-13T04:43:39.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:39.646+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:39.646820+0000) 2019-10-13T04:43:41.061 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:41.057+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.057969+0000) 2019-10-13T04:43:41.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:41.057+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.058232+0000) 2019-10-13T04:43:41.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:41.058+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.058404+0000) 2019-10-13T04:43:41.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:41.058+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.058492+0000) 2019-10-13T04:43:41.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:41.058+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.058929+0000) 2019-10-13T04:43:41.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:41.058+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.058878+0000) 2019-10-13T04:43:41.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:41.052+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.054713+0000) 2019-10-13T04:43:41.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:41.054+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.056322+0000) 2019-10-13T04:43:41.063 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:41.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:41.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:41.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.760435+0000) 2019-10-13T04:43:41.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:41.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.760435+0000) 2019-10-13T04:43:41.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:41.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:41.760449+0000) 2019-10-13T04:43:42.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:42.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:42.632685+0000) 2019-10-13T04:43:42.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:42.632+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:42.633821+0000) 2019-10-13T04:43:46.064 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:46.061+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.061976+0000) 2019-10-13T04:43:46.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:46.061+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.062274+0000) 2019-10-13T04:43:46.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:46.056+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.058291+0000) 2019-10-13T04:43:46.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:46.062+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.062971+0000) 2019-10-13T04:43:46.066 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:46.062+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.063269+0000) 2019-10-13T04:43:46.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:46.062+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.063372+0000) 2019-10-13T04:43:46.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:46.063+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.063540+0000) 2019-10-13T04:43:46.067 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:46.058+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:46.060058+0000) 2019-10-13T04:43:46.067 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:46.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:47.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:47.376+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:47.377194+0000) 2019-10-13T04:43:48.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:48.169+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:48.169523+0000) 2019-10-13T04:43:49.656 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:49.648+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:49.649118+0000) 2019-10-13T04:43:51.069 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:51.066+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.066636+0000) 2019-10-13T04:43:51.070 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:51.066+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.066750+0000) 2019-10-13T04:43:51.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:51.066+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.067171+0000) 2019-10-13T04:43:51.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:51.061+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.063006+0000) 2019-10-13T04:43:51.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:51.067+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.067686+0000) 2019-10-13T04:43:51.071 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:51.067+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.067984+0000) 2019-10-13T04:43:51.071 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:51.067+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.068272+0000) 2019-10-13T04:43:51.072 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:51.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:51.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:51.063+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.065671+0000) 2019-10-13T04:43:51.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:51.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.760634+0000) 2019-10-13T04:43:51.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:51.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.760619+0000) 2019-10-13T04:43:51.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:51.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:51.760642+0000) 2019-10-13T04:43:52.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:52.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:52.632883+0000) 2019-10-13T04:43:52.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:52.633+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:52.633932+0000) 2019-10-13T04:43:56.074 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:43:56.069+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.070797+0000) 2019-10-13T04:43:56.074 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:43:56.069+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.070908+0000) 2019-10-13T04:43:56.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:56.069+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.071181+0000) 2019-10-13T04:43:56.075 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:43:56.070+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.071438+0000) 2019-10-13T04:43:56.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:43:56.066+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.067319+0000) 2019-10-13T04:43:56.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:56.068+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.068960+0000) 2019-10-13T04:43:56.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:43:56.070+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.071582+0000) 2019-10-13T04:43:56.077 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:56.070+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:56.071842+0000) 2019-10-13T04:43:56.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:43:56.071+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:43:56.887 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:43:56.887 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:56.891 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:43:56.891 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:57.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:43:57.375+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:57.377333+0000) 2019-10-13T04:43:58.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:43:58.168+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:58.169740+0000) 2019-10-13T04:43:59.658 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:43:59.650+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:43:59.651496+0000) 2019-10-13T04:44:01.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:01.073+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.074508+0000) 2019-10-13T04:44:01.079 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:01.073+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.075295+0000) 2019-10-13T04:44:01.079 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:01.073+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.075171+0000) 2019-10-13T04:44:01.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:01.074+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.075524+0000) 2019-10-13T04:44:01.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:01.074+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.075951+0000) 2019-10-13T04:44:01.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:01.074+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.076004+0000) 2019-10-13T04:44:01.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:01.070+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.071269+0000) 2019-10-13T04:44:01.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:01.071+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.072478+0000) 2019-10-13T04:44:01.084 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:01.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62142 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:01.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:01.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.760785+0000) 2019-10-13T04:44:01.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:01.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.760787+0000) 2019-10-13T04:44:01.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:01.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:01.760793+0000) 2019-10-13T04:44:02.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:02.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:02.633048+0000) 2019-10-13T04:44:02.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:02.633+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:02.634042+0000) 2019-10-13T04:44:06.081 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:06.077+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.078614+0000) 2019-10-13T04:44:06.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:06.077+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.078828+0000) 2019-10-13T04:44:06.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:06.073+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.074308+0000) 2019-10-13T04:44:06.083 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:06.077+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.079195+0000) 2019-10-13T04:44:06.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:06.077+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.079388+0000) 2019-10-13T04:44:06.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:06.078+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.079559+0000) 2019-10-13T04:44:06.083 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:06.078+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.079839+0000) 2019-10-13T04:44:06.083 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:06.079+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:06.084 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:06.076+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:06.077590+0000) 2019-10-13T04:44:07.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:07.376+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:07.377503+0000) 2019-10-13T04:44:08.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:08.169+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:08.169975+0000) 2019-10-13T04:44:09.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:09.653+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:09.654297+0000) 2019-10-13T04:44:11.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:11.077+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.078436+0000) 2019-10-13T04:44:11.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:11.081+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.082970+0000) 2019-10-13T04:44:11.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:11.082+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.083390+0000) 2019-10-13T04:44:11.086 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:11.082+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.083728+0000) 2019-10-13T04:44:11.086 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:11.082+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.083922+0000) 2019-10-13T04:44:11.087 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:11.082+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.084190+0000) 2019-10-13T04:44:11.087 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:11.083+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.084470+0000) 2019-10-13T04:44:11.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:11.080+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.080937+0000) 2019-10-13T04:44:11.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:11.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:11.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:11.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.760936+0000) 2019-10-13T04:44:11.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:11.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.760942+0000) 2019-10-13T04:44:11.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:11.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:11.761028+0000) 2019-10-13T04:44:12.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:12.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:12.633169+0000) 2019-10-13T04:44:12.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:12.633+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:12.634234+0000) 2019-10-13T04:44:16.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:16.082+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.082883+0000) 2019-10-13T04:44:16.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:16.086+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.087530+0000) 2019-10-13T04:44:16.090 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:16.086+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.087841+0000) 2019-10-13T04:44:16.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:16.086+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.088378+0000) 2019-10-13T04:44:16.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:16.087+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.088464+0000) 2019-10-13T04:44:16.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:16.087+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.088611+0000) 2019-10-13T04:44:16.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:16.087+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.088882+0000) 2019-10-13T04:44:16.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:16.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:16.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:16.086+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:16.086846+0000) 2019-10-13T04:44:17.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:17.377+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:17.377717+0000) 2019-10-13T04:44:18.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:18.169+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:18.170180+0000) 2019-10-13T04:44:19.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:19.656+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:19.656628+0000) 2019-10-13T04:44:21.094 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:21.090+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.091650+0000) 2019-10-13T04:44:21.094 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:21.090+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.091769+0000) 2019-10-13T04:44:21.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:21.087+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.087899+0000) 2019-10-13T04:44:21.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:21.090+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.092249+0000) 2019-10-13T04:44:21.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:21.091+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.092644+0000) 2019-10-13T04:44:21.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:21.091+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.092802+0000) 2019-10-13T04:44:21.096 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:21.091+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.093167+0000) 2019-10-13T04:44:21.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:21.089+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.090184+0000) 2019-10-13T04:44:21.097 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:21.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:21.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:21.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.761145+0000) 2019-10-13T04:44:21.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:21.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.761146+0000) 2019-10-13T04:44:21.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:21.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:21.761156+0000) 2019-10-13T04:44:22.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:22.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:22.633350+0000) 2019-10-13T04:44:22.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:22.633+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:22.634433+0000) 2019-10-13T04:44:26.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:26.095+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.095874+0000) 2019-10-13T04:44:26.100 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:26.095+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.096134+0000) 2019-10-13T04:44:26.100 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:26.095+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.096361+0000) 2019-10-13T04:44:26.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:26.096+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.096517+0000) 2019-10-13T04:44:26.100 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:26.096+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.096794+0000) 2019-10-13T04:44:26.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:26.096+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.097014+0000) 2019-10-13T04:44:26.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:26.092+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.092882+0000) 2019-10-13T04:44:26.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:26.094+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:26.095052+0000) 2019-10-13T04:44:26.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:26.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:27.034 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:44:27.034 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:27.038 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:44:27.039 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:27.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:27.377+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:27.377921+0000) 2019-10-13T04:44:28.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:28.170+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:28.170410+0000) 2019-10-13T04:44:29.665 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:29.658+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:29.658786+0000) 2019-10-13T04:44:31.102 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:31.099+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.100234+0000) 2019-10-13T04:44:31.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:31.095+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.096180+0000) 2019-10-13T04:44:31.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:31.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.100888+0000) 2019-10-13T04:44:31.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:31.100+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.101367+0000) 2019-10-13T04:44:31.104 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:31.101+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.101559+0000) 2019-10-13T04:44:31.104 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:31.101+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.101859+0000) 2019-10-13T04:44:31.104 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:31.100+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.101047+0000) 2019-10-13T04:44:31.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:31.097+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.098338+0000) 2019-10-13T04:44:31.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:31.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:31.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:31.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.761297+0000) 2019-10-13T04:44:31.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:31.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.761304+0000) 2019-10-13T04:44:31.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:31.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:31.761302+0000) 2019-10-13T04:44:32.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:32.633+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:32.633555+0000) 2019-10-13T04:44:32.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:32.634+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:32.634639+0000) 2019-10-13T04:44:36.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:36.098+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.099902+0000) 2019-10-13T04:44:36.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:36.104+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.104755+0000) 2019-10-13T04:44:36.107 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:36.104+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.104959+0000) 2019-10-13T04:44:36.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:36.104+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.105167+0000) 2019-10-13T04:44:36.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:36.105+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.105650+0000) 2019-10-13T04:44:36.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:36.105+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.105760+0000) 2019-10-13T04:44:36.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:36.105+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.105941+0000) 2019-10-13T04:44:36.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:36.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62350 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:36.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:36.101+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:36.103158+0000) 2019-10-13T04:44:37.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:37.377+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:37.378092+0000) 2019-10-13T04:44:38.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:38.170+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:38.170606+0000) 2019-10-13T04:44:39.668 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:39.660+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:39.661158+0000) 2019-10-13T04:44:41.112 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:41.108+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.109073+0000) 2019-10-13T04:44:41.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:41.108+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.109294+0000) 2019-10-13T04:44:41.112 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:41.109+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.109765+0000) 2019-10-13T04:44:41.113 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:41.109+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.109894+0000) 2019-10-13T04:44:41.113 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:41.109+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.110109+0000) 2019-10-13T04:44:41.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:41.109+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.110299+0000) 2019-10-13T04:44:41.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:41.103+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.105101+0000) 2019-10-13T04:44:41.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:41.104+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.106586+0000) 2019-10-13T04:44:41.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:41.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:41.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:41.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.761462+0000) 2019-10-13T04:44:41.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:41.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.761460+0000) 2019-10-13T04:44:41.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:41.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:41.761465+0000) 2019-10-13T04:44:42.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:42.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:42.633709+0000) 2019-10-13T04:44:42.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:42.633+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:42.634802+0000) 2019-10-13T04:44:46.115 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:46.111+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.112858+0000) 2019-10-13T04:44:46.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:46.111+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.112992+0000) 2019-10-13T04:44:46.116 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:46.111+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.113152+0000) 2019-10-13T04:44:46.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:46.111+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.113298+0000) 2019-10-13T04:44:46.116 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:46.112+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.113458+0000) 2019-10-13T04:44:46.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:46.107+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.109189+0000) 2019-10-13T04:44:46.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:46.108+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.110305+0000) 2019-10-13T04:44:46.117 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:46.112+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:46.113992+0000) 2019-10-13T04:44:46.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:46.113+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:47.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:47.376+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:47.378272+0000) 2019-10-13T04:44:48.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:48.169+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:48.170812+0000) 2019-10-13T04:44:49.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:49.662+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:49.663716+0000) 2019-10-13T04:44:51.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:51.115+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.117246+0000) 2019-10-13T04:44:51.124 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:51.116+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.117532+0000) 2019-10-13T04:44:51.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:51.116+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.117677+0000) 2019-10-13T04:44:51.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:51.116+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.117852+0000) 2019-10-13T04:44:51.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:51.116+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.117977+0000) 2019-10-13T04:44:51.125 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:51.116+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.118140+0000) 2019-10-13T04:44:51.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:51.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:51.125 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:51.111+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.112400+0000) 2019-10-13T04:44:51.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:51.115+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.116039+0000) 2019-10-13T04:44:51.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:51.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.761618+0000) 2019-10-13T04:44:51.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:51.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.761621+0000) 2019-10-13T04:44:51.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:51.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:51.761619+0000) 2019-10-13T04:44:52.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:52.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:52.633839+0000) 2019-10-13T04:44:52.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:52.634+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:52.634947+0000) 2019-10-13T04:44:56.123 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:44:56.119+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.120933+0000) 2019-10-13T04:44:56.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:56.120+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.121406+0000) 2019-10-13T04:44:56.125 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:56.120+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.121687+0000) 2019-10-13T04:44:56.125 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:44:56.120+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.122121+0000) 2019-10-13T04:44:56.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:44:56.120+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.122255+0000) 2019-10-13T04:44:56.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:44:56.120+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.122349+0000) 2019-10-13T04:44:56.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:44:56.116+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.117638+0000) 2019-10-13T04:44:56.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:56.118+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:56.118958+0000) 2019-10-13T04:44:56.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:44:56.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62470 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:44:57.265 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:44:57.265 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:57.270 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:44:57.270 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:57.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:44:57.377+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:57.378403+0000) 2019-10-13T04:44:58.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:44:58.170+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:58.171001+0000) 2019-10-13T04:44:59.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:44:59.665+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:44:59.666070+0000) 2019-10-13T04:45:01.128 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:01.123+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.125365+0000) 2019-10-13T04:45:01.128 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:01.124+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.125497+0000) 2019-10-13T04:45:01.128 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:01.124+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.125729+0000) 2019-10-13T04:45:01.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:01.124+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.125779+0000) 2019-10-13T04:45:01.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:01.121+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.122064+0000) 2019-10-13T04:45:01.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:01.122+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.122810+0000) 2019-10-13T04:45:01.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:01.124+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.126193+0000) 2019-10-13T04:45:01.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:01.124+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.126293+0000) 2019-10-13T04:45:01.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:01.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:01.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:01.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.761802+0000) 2019-10-13T04:45:01.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:01.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.761797+0000) 2019-10-13T04:45:01.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:01.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:01.761800+0000) 2019-10-13T04:45:02.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:02.632+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:02.633973+0000) 2019-10-13T04:45:02.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:02.634+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:02.635115+0000) 2019-10-13T04:45:06.133 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:06.128+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.129900+0000) 2019-10-13T04:45:06.133 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:06.128+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.130018+0000) 2019-10-13T04:45:06.134 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:06.128+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.130378+0000) 2019-10-13T04:45:06.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:06.129+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.130483+0000) 2019-10-13T04:45:06.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:06.129+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.130688+0000) 2019-10-13T04:45:06.135 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:06.129+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.130889+0000) 2019-10-13T04:45:06.135 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:06.126+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.126739+0000) 2019-10-13T04:45:06.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:06.127+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:06.128142+0000) 2019-10-13T04:45:06.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:06.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:07.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:07.378+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:07.378576+0000) 2019-10-13T04:45:08.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:08.170+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:08.171219+0000) 2019-10-13T04:45:09.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:09.668+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:09.668400+0000) 2019-10-13T04:45:11.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:11.129+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.130114+0000) 2019-10-13T04:45:11.137 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:11.133+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.134161+0000) 2019-10-13T04:45:11.138 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:11.133+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.134318+0000) 2019-10-13T04:45:11.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:11.134+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.134605+0000) 2019-10-13T04:45:11.138 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:11.134+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.134977+0000) 2019-10-13T04:45:11.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:11.134+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.135092+0000) 2019-10-13T04:45:11.139 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:11.134+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.135307+0000) 2019-10-13T04:45:11.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:11.131+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.132630+0000) 2019-10-13T04:45:11.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:11.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62561 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:11.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:11.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.761991+0000) 2019-10-13T04:45:11.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:11.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.761991+0000) 2019-10-13T04:45:11.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:11.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:11.761996+0000) 2019-10-13T04:45:12.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:12.633+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:12.634118+0000) 2019-10-13T04:45:12.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:12.634+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:12.635303+0000) 2019-10-13T04:45:16.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:16.137+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.137755+0000) 2019-10-13T04:45:16.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:16.133+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.133863+0000) 2019-10-13T04:45:16.141 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:16.137+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.138359+0000) 2019-10-13T04:45:16.141 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:16.138+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.138494+0000) 2019-10-13T04:45:16.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:16.138+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.138626+0000) 2019-10-13T04:45:16.142 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:16.138+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.139019+0000) 2019-10-13T04:45:16.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:16.134+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.135697+0000) 2019-10-13T04:45:16.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:16.138+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:16.139178+0000) 2019-10-13T04:45:16.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:16.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62590 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:17.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:17.378+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:17.378750+0000) 2019-10-13T04:45:18.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:18.171+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:18.171414+0000) 2019-10-13T04:45:19.677 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:19.670+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:19.670783+0000) 2019-10-13T04:45:21.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:21.141+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.142208+0000) 2019-10-13T04:45:21.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:21.137+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.138178+0000) 2019-10-13T04:45:21.146 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:21.142+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.142958+0000) 2019-10-13T04:45:21.146 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:21.142+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.143114+0000) 2019-10-13T04:45:21.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:21.142+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.143248+0000) 2019-10-13T04:45:21.148 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:21.143+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.143472+0000) 2019-10-13T04:45:21.148 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:21.143+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.143565+0000) 2019-10-13T04:45:21.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:21.139+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.140623+0000) 2019-10-13T04:45:21.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:21.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62622 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:21.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:21.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.762164+0000) 2019-10-13T04:45:21.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:21.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.762159+0000) 2019-10-13T04:45:21.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:21.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:21.762165+0000) 2019-10-13T04:45:22.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:22.634+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:22.634286+0000) 2019-10-13T04:45:22.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:22.634+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:22.635462+0000) 2019-10-13T04:45:26.149 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:26.141+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.142300+0000) 2019-10-13T04:45:26.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:26.146+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.146942+0000) 2019-10-13T04:45:26.150 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:26.146+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.147084+0000) 2019-10-13T04:45:26.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:26.146+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.147356+0000) 2019-10-13T04:45:26.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:26.147+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.147747+0000) 2019-10-13T04:45:26.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:26.147+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.147882+0000) 2019-10-13T04:45:26.151 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:26.147+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.148041+0000) 2019-10-13T04:45:26.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:26.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:26.153 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:26.145+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:26.145847+0000) 2019-10-13T04:45:27.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:27.377+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:27.378932+0000) 2019-10-13T04:45:27.469 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:45:27.469 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:27.474 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:45:27.474 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:28.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:28.170+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:28.171573+0000) 2019-10-13T04:45:29.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:29.672+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:29.673061+0000) 2019-10-13T04:45:31.154 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:31.144+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.146722+0000) 2019-10-13T04:45:31.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:31.149+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.151360+0000) 2019-10-13T04:45:31.156 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:31.150+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.151499+0000) 2019-10-13T04:45:31.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:31.150+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.151804+0000) 2019-10-13T04:45:31.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:31.150+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.152001+0000) 2019-10-13T04:45:31.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:31.150+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.152280+0000) 2019-10-13T04:45:31.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:31.151+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.152557+0000) 2019-10-13T04:45:31.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:31.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62681 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:31.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:31.147+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.148924+0000) 2019-10-13T04:45:31.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:31.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.762322+0000) 2019-10-13T04:45:31.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:31.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.762320+0000) 2019-10-13T04:45:31.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:31.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:31.762324+0000) 2019-10-13T04:45:32.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:32.633+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:32.634490+0000) 2019-10-13T04:45:32.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:32.633+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:32.635664+0000) 2019-10-13T04:45:36.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:36.153+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.154770+0000) 2019-10-13T04:45:36.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:36.153+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.155021+0000) 2019-10-13T04:45:36.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:36.153+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.155249+0000) 2019-10-13T04:45:36.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:36.154+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.155797+0000) 2019-10-13T04:45:36.159 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:36.154+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.155964+0000) 2019-10-13T04:45:36.160 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:36.154+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.156143+0000) 2019-10-13T04:45:36.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:36.149+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.151025+0000) 2019-10-13T04:45:36.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:36.151+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:36.153575+0000) 2019-10-13T04:45:36.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:36.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62710 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:37.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:37.378+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:37.379075+0000) 2019-10-13T04:45:38.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:38.170+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:38.171758+0000) 2019-10-13T04:45:39.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:39.674+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:39.675329+0000) 2019-10-13T04:45:41.161 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:41.152+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.154433+0000) 2019-10-13T04:45:41.162 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:41.157+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.158915+0000) 2019-10-13T04:45:41.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:41.157+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.158980+0000) 2019-10-13T04:45:41.163 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:41.157+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.159378+0000) 2019-10-13T04:45:41.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:41.158+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.159627+0000) 2019-10-13T04:45:41.164 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:41.158+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.159823+0000) 2019-10-13T04:45:41.164 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:41.158+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.159976+0000) 2019-10-13T04:45:41.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:41.156+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.158332+0000) 2019-10-13T04:45:41.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:41.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:41.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:41.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.762514+0000) 2019-10-13T04:45:41.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:41.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.762516+0000) 2019-10-13T04:45:41.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:41.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:41.762521+0000) 2019-10-13T04:45:42.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:42.633+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:42.634676+0000) 2019-10-13T04:45:42.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:42.635+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:42.635819+0000) 2019-10-13T04:45:46.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:46.161+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.162923+0000) 2019-10-13T04:45:46.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:46.161+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.163128+0000) 2019-10-13T04:45:46.166 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:46.161+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.163321+0000) 2019-10-13T04:45:46.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:46.162+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.163534+0000) 2019-10-13T04:45:46.167 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:46.158+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.159482+0000) 2019-10-13T04:45:46.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:46.162+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.164164+0000) 2019-10-13T04:45:46.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:46.162+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.164317+0000) 2019-10-13T04:45:46.167 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:46.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:46.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:46.161+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:46.161807+0000) 2019-10-13T04:45:47.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:47.378+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:47.379281+0000) 2019-10-13T04:45:48.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:48.171+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:48.171933+0000) 2019-10-13T04:45:49.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:49.677+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:49.677686+0000) 2019-10-13T04:45:51.178 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:51.163+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.163995+0000) 2019-10-13T04:45:51.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:51.164+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.164955+0000) 2019-10-13T04:45:51.178 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:51.165+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.166882+0000) 2019-10-13T04:45:51.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:51.165+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.167259+0000) 2019-10-13T04:45:51.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:51.166+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.167457+0000) 2019-10-13T04:45:51.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:51.166+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.167571+0000) 2019-10-13T04:45:51.179 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:51.166+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.167752+0000) 2019-10-13T04:45:51.179 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:51.166+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.168129+0000) 2019-10-13T04:45:51.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:51.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:51.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:51.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.762676+0000) 2019-10-13T04:45:51.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:51.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.762676+0000) 2019-10-13T04:45:51.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:51.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:51.762684+0000) 2019-10-13T04:45:52.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:52.634+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:52.634862+0000) 2019-10-13T04:45:52.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:52.635+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:52.635952+0000) 2019-10-13T04:45:56.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:45:56.170+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.171210+0000) 2019-10-13T04:45:56.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:45:56.170+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.171366+0000) 2019-10-13T04:45:56.178 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:45:56.171+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.171520+0000) 2019-10-13T04:45:56.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:56.171+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.171893+0000) 2019-10-13T04:45:56.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:56.171+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.171821+0000) 2019-10-13T04:45:56.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:45:56.171+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.172010+0000) 2019-10-13T04:45:56.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:45:56.167+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.168087+0000) 2019-10-13T04:45:56.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:56.169+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:56.170056+0000) 2019-10-13T04:45:56.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:45:56.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:45:57.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:45:57.379+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:57.379485+0000) 2019-10-13T04:45:57.568 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:45:57.568 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:57.573 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:45:57.573 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:58.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:45:58.171+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:58.172135+0000) 2019-10-13T04:45:59.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:45:59.679+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:45:59.680003+0000) 2019-10-13T04:46:01.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:01.174+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.175266+0000) 2019-10-13T04:46:01.179 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:01.170+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.171156+0000) 2019-10-13T04:46:01.179 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:01.175+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.175808+0000) 2019-10-13T04:46:01.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:01.175+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.176222+0000) 2019-10-13T04:46:01.180 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:01.175+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.176249+0000) 2019-10-13T04:46:01.180 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:01.176+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.176421+0000) 2019-10-13T04:46:01.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:01.176+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.176646+0000) 2019-10-13T04:46:01.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:01.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:01.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:01.173+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.174514+0000) 2019-10-13T04:46:01.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:01.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.762834+0000) 2019-10-13T04:46:01.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:01.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.762834+0000) 2019-10-13T04:46:01.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:01.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:01.762835+0000) 2019-10-13T04:46:02.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:02.634+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:02.635060+0000) 2019-10-13T04:46:02.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:02.635+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:02.636081+0000) 2019-10-13T04:46:06.186 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:06.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.175652+0000) 2019-10-13T04:46:06.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:06.177+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.178488+0000) 2019-10-13T04:46:06.192 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:06.179+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.179834+0000) 2019-10-13T04:46:06.193 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:06.179+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.180349+0000) 2019-10-13T04:46:06.193 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:06.180+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.180465+0000) 2019-10-13T04:46:06.194 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:06.180+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.180653+0000) 2019-10-13T04:46:06.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:06.180+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.180779+0000) 2019-10-13T04:46:06.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:06.180+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:06.181186+0000) 2019-10-13T04:46:06.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:06.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62891 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:07.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:07.379+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:07.379679+0000) 2019-10-13T04:46:08.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:08.172+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:08.172301+0000) 2019-10-13T04:46:09.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:09.682+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:09.683202+0000) 2019-10-13T04:46:11.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:11.183+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.183899+0000) 2019-10-13T04:46:11.187 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:11.183+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.184264+0000) 2019-10-13T04:46:11.188 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:11.179+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.179757+0000) 2019-10-13T04:46:11.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:11.184+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.184458+0000) 2019-10-13T04:46:11.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:11.184+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.184898+0000) 2019-10-13T04:46:11.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:11.184+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.185051+0000) 2019-10-13T04:46:11.189 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:11.184+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.185190+0000) 2019-10-13T04:46:11.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:11.186+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:11.190 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:11.181+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.182074+0000) 2019-10-13T04:46:11.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:11.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.762988+0000) 2019-10-13T04:46:11.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:11.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.762988+0000) 2019-10-13T04:46:11.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:11.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:11.762996+0000) 2019-10-13T04:46:12.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:12.633+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:12.635231+0000) 2019-10-13T04:46:12.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:12.636+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:12.636238+0000) 2019-10-13T04:46:16.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:16.186+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.188105+0000) 2019-10-13T04:46:16.191 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:16.183+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.184373+0000) 2019-10-13T04:46:16.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:16.187+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.188585+0000) 2019-10-13T04:46:16.191 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:16.187+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.189011+0000) 2019-10-13T04:46:16.192 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:16.187+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.189172+0000) 2019-10-13T04:46:16.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:16.187+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.189375+0000) 2019-10-13T04:46:16.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:16.188+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.189711+0000) 2019-10-13T04:46:16.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:16.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:16.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:16.186+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:16.187247+0000) 2019-10-13T04:46:17.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:17.378+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:17.379905+0000) 2019-10-13T04:46:18.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:18.171+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:18.172493+0000) 2019-10-13T04:46:19.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:19.685+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:19.685903+0000) 2019-10-13T04:46:21.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:21.190+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.192379+0000) 2019-10-13T04:46:21.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:21.191+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.192696+0000) 2019-10-13T04:46:21.195 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:21.191+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.192863+0000) 2019-10-13T04:46:21.196 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:21.187+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.188768+0000) 2019-10-13T04:46:21.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:21.191+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.193135+0000) 2019-10-13T04:46:21.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:21.192+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.193595+0000) 2019-10-13T04:46:21.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:21.192+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.194015+0000) 2019-10-13T04:46:21.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:21.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 62982 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:21.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:21.189+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.191235+0000) 2019-10-13T04:46:21.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:21.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.763166+0000) 2019-10-13T04:46:21.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:21.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.763163+0000) 2019-10-13T04:46:21.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:21.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:21.763166+0000) 2019-10-13T04:46:22.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:22.634+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:22.635388+0000) 2019-10-13T04:46:22.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:22.635+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:22.636421+0000) 2019-10-13T04:46:26.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:26.195+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.196883+0000) 2019-10-13T04:46:26.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:26.195+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.197333+0000) 2019-10-13T04:46:26.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:26.195+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.197323+0000) 2019-10-13T04:46:26.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:26.196+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.197477+0000) 2019-10-13T04:46:26.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:26.196+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.197790+0000) 2019-10-13T04:46:26.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:26.196+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.198116+0000) 2019-10-13T04:46:26.202 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:26.191+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.193720+0000) 2019-10-13T04:46:26.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:26.193+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:26.194798+0000) 2019-10-13T04:46:26.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:26.197+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63011 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:27.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:27.379+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:27.380079+0000) 2019-10-13T04:46:27.669 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:46:27.669 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:27.673 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:46:27.673 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:28.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:28.171+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:28.172653+0000) 2019-10-13T04:46:29.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:29.686+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:29.688554+0000) 2019-10-13T04:46:31.202 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:31.199+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.200472+0000) 2019-10-13T04:46:31.203 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:31.199+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.200803+0000) 2019-10-13T04:46:31.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:31.199+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.201146+0000) 2019-10-13T04:46:31.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:31.199+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.201170+0000) 2019-10-13T04:46:31.204 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:31.200+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.201557+0000) 2019-10-13T04:46:31.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:31.195+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.197043+0000) 2019-10-13T04:46:31.205 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:31.199+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.200979+0000) 2019-10-13T04:46:31.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:31.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:31.207 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:31.198+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.199858+0000) 2019-10-13T04:46:31.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:31.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.763321+0000) 2019-10-13T04:46:31.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:31.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.763318+0000) 2019-10-13T04:46:31.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:31.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:31.763319+0000) 2019-10-13T04:46:32.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:32.634+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:32.635570+0000) 2019-10-13T04:46:32.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:32.635+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:32.636607+0000) 2019-10-13T04:46:36.206 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:36.198+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.199595+0000) 2019-10-13T04:46:36.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:36.202+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.204111+0000) 2019-10-13T04:46:36.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:36.202+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.204240+0000) 2019-10-13T04:46:36.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:36.203+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.204412+0000) 2019-10-13T04:46:36.208 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:36.203+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.204622+0000) 2019-10-13T04:46:36.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:36.203+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.204798+0000) 2019-10-13T04:46:36.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:36.204+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.205724+0000) 2019-10-13T04:46:36.209 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:36.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:36.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:36.202+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:36.202838+0000) 2019-10-13T04:46:37.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:37.379+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:37.380278+0000) 2019-10-13T04:46:38.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:38.172+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:38.172834+0000) 2019-10-13T04:46:39.698 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:39.691+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:39.691769+0000) 2019-10-13T04:46:41.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:41.207+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.207785+0000) 2019-10-13T04:46:41.211 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:41.203+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.204062+0000) 2019-10-13T04:46:41.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:41.208+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.208545+0000) 2019-10-13T04:46:41.211 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:41.208+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.208794+0000) 2019-10-13T04:46:41.211 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:41.208+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.208960+0000) 2019-10-13T04:46:41.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:41.209+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.209560+0000) 2019-10-13T04:46:41.212 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:41.209+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.209946+0000) 2019-10-13T04:46:41.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:41.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63102 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:41.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:41.206+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.207560+0000) 2019-10-13T04:46:41.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:41.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.763497+0000) 2019-10-13T04:46:41.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:41.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.763504+0000) 2019-10-13T04:46:41.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:41.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:41.763499+0000) 2019-10-13T04:46:42.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:42.635+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:42.635751+0000) 2019-10-13T04:46:42.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:42.636+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:42.636812+0000) 2019-10-13T04:46:46.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:46.212+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.213010+0000) 2019-10-13T04:46:46.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:46.212+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.213259+0000) 2019-10-13T04:46:46.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:46.208+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.209565+0000) 2019-10-13T04:46:46.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:46.213+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.213681+0000) 2019-10-13T04:46:46.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:46.213+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.214007+0000) 2019-10-13T04:46:46.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:46.213+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.214275+0000) 2019-10-13T04:46:46.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:46.214+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.214502+0000) 2019-10-13T04:46:46.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:46.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:46.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:46.212+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:46.212862+0000) 2019-10-13T04:46:47.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:47.380+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:47.380466+0000) 2019-10-13T04:46:48.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:48.172+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:48.173024+0000) 2019-10-13T04:46:49.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:49.693+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:49.694062+0000) 2019-10-13T04:46:51.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:51.212+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.213263+0000) 2019-10-13T04:46:51.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:51.217+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.217943+0000) 2019-10-13T04:46:51.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:51.217+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.218179+0000) 2019-10-13T04:46:51.221 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:51.217+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.218310+0000) 2019-10-13T04:46:51.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:51.218+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.218455+0000) 2019-10-13T04:46:51.221 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:51.218+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.218530+0000) 2019-10-13T04:46:51.222 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:51.218+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.218702+0000) 2019-10-13T04:46:51.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:51.214+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.215518+0000) 2019-10-13T04:46:51.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:51.219+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:51.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:51.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.763688+0000) 2019-10-13T04:46:51.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:51.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.763690+0000) 2019-10-13T04:46:51.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:51.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:51.763689+0000) 2019-10-13T04:46:52.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:52.635+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:52.635959+0000) 2019-10-13T04:46:52.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:52.636+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:52.636996+0000) 2019-10-13T04:46:56.223 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:46:56.216+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.217136+0000) 2019-10-13T04:46:56.224 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:46:56.220+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.221325+0000) 2019-10-13T04:46:56.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:56.221+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.222044+0000) 2019-10-13T04:46:56.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:46:56.221+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.222113+0000) 2019-10-13T04:46:56.225 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:46:56.222+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.222522+0000) 2019-10-13T04:46:56.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:46:56.222+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.222650+0000) 2019-10-13T04:46:56.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:56.222+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.222806+0000) 2019-10-13T04:46:56.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:56.218+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:56.219293+0000) 2019-10-13T04:46:56.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:46:56.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63190 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:46:57.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:46:57.379+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:57.380608+0000) 2019-10-13T04:46:57.781 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:46:57.782 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:57.786 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:46:57.786 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:58.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:46:58.172+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:58.173188+0000) 2019-10-13T04:46:59.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:46:59.696+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:46:59.697051+0000) 2019-10-13T04:47:01.228 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:01.224+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.225763+0000) 2019-10-13T04:47:01.228 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:01.224+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.226219+0000) 2019-10-13T04:47:01.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:01.221+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.222577+0000) 2019-10-13T04:47:01.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:01.225+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.226499+0000) 2019-10-13T04:47:01.230 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:01.225+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.226414+0000) 2019-10-13T04:47:01.230 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:01.225+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.227040+0000) 2019-10-13T04:47:01.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:01.226+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.227407+0000) 2019-10-13T04:47:01.231 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:01.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:01.231 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:01.224+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.225249+0000) 2019-10-13T04:47:01.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:01.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.763860+0000) 2019-10-13T04:47:01.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:01.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.763859+0000) 2019-10-13T04:47:01.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:01.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:01.763860+0000) 2019-10-13T04:47:02.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:02.635+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:02.636147+0000) 2019-10-13T04:47:02.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:02.636+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:02.637190+0000) 2019-10-13T04:47:06.232 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:06.228+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.230149+0000) 2019-10-13T04:47:06.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:06.229+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.230665+0000) 2019-10-13T04:47:06.233 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:06.229+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.231039+0000) 2019-10-13T04:47:06.234 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:06.226+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.226969+0000) 2019-10-13T04:47:06.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:06.230+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.231534+0000) 2019-10-13T04:47:06.234 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:06.229+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.231316+0000) 2019-10-13T04:47:06.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:06.230+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.231993+0000) 2019-10-13T04:47:06.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:06.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:06.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:06.229+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:06.229936+0000) 2019-10-13T04:47:07.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:07.379+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:07.380760+0000) 2019-10-13T04:47:08.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:08.172+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:08.173377+0000) 2019-10-13T04:47:09.706 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:09.699+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:09.699441+0000) 2019-10-13T04:47:11.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:11.233+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.234600+0000) 2019-10-13T04:47:11.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:11.233+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.234955+0000) 2019-10-13T04:47:11.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:11.234+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.235426+0000) 2019-10-13T04:47:11.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:11.234+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.235634+0000) 2019-10-13T04:47:11.239 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:11.234+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.236106+0000) 2019-10-13T04:47:11.240 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:11.235+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.236469+0000) 2019-10-13T04:47:11.241 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:11.231+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.231746+0000) 2019-10-13T04:47:11.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:11.231+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.232668+0000) 2019-10-13T04:47:11.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:11.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:11.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:11.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.764067+0000) 2019-10-13T04:47:11.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:11.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.764064+0000) 2019-10-13T04:47:11.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:11.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:11.764063+0000) 2019-10-13T04:47:12.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:12.634+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:12.636356+0000) 2019-10-13T04:47:12.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:12.636+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:12.637390+0000) 2019-10-13T04:47:16.241 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:16.237+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.239237+0000) 2019-10-13T04:47:16.242 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:16.238+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.239673+0000) 2019-10-13T04:47:16.243 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:16.233+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.235297+0000) 2019-10-13T04:47:16.243 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:16.238+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.239905+0000) 2019-10-13T04:47:16.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:16.238+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.240234+0000) 2019-10-13T04:47:16.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:16.239+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.240478+0000) 2019-10-13T04:47:16.243 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:16.239+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.240653+0000) 2019-10-13T04:47:16.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:16.235+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:16.237212+0000) 2019-10-13T04:47:16.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:16.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:17.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:17.380+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:17.380916+0000) 2019-10-13T04:47:18.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:18.172+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:18.173543+0000) 2019-10-13T04:47:19.708 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:19.701+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:19.702002+0000) 2019-10-13T04:47:21.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:21.242+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.243748+0000) 2019-10-13T04:47:21.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:21.242+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.244114+0000) 2019-10-13T04:47:21.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:21.237+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.239643+0000) 2019-10-13T04:47:21.247 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:21.243+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.244528+0000) 2019-10-13T04:47:21.247 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:21.243+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.244607+0000) 2019-10-13T04:47:21.248 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:21.243+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.244907+0000) 2019-10-13T04:47:21.248 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:21.243+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.245014+0000) 2019-10-13T04:47:21.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:21.240+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.242132+0000) 2019-10-13T04:47:21.249 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:21.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:21.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:21.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.764217+0000) 2019-10-13T04:47:21.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:21.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.764210+0000) 2019-10-13T04:47:21.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:21.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:21.764220+0000) 2019-10-13T04:47:22.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:22.636+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:22.636581+0000) 2019-10-13T04:47:22.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:22.636+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:22.637587+0000) 2019-10-13T04:47:26.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:26.247+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.247993+0000) 2019-10-13T04:47:26.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:26.248+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.249103+0000) 2019-10-13T04:47:26.252 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:26.248+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.249158+0000) 2019-10-13T04:47:26.252 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:26.248+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.248990+0000) 2019-10-13T04:47:26.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:26.249+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.249394+0000) 2019-10-13T04:47:26.252 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:26.248+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.249355+0000) 2019-10-13T04:47:26.253 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:26.242+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.244657+0000) 2019-10-13T04:47:26.253 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:26.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:26.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:26.246+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:26.248326+0000) 2019-10-13T04:47:27.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:27.380+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:27.381099+0000) 2019-10-13T04:47:27.886 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:47:27.886 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:27.890 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:47:27.890 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:28.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:28.173+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:28.173738+0000) 2019-10-13T04:47:29.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:29.703+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:29.704528+0000) 2019-10-13T04:47:31.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:31.247+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.248535+0000) 2019-10-13T04:47:31.256 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:31.252+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.252982+0000) 2019-10-13T04:47:31.256 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:31.252+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.253121+0000) 2019-10-13T04:47:31.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:31.252+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.253361+0000) 2019-10-13T04:47:31.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:31.253+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.253711+0000) 2019-10-13T04:47:31.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:31.253+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.254089+0000) 2019-10-13T04:47:31.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:31.253+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.254214+0000) 2019-10-13T04:47:31.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:31.250+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.251461+0000) 2019-10-13T04:47:31.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:31.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63402 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:31.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:31.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.764417+0000) 2019-10-13T04:47:31.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:31.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.764414+0000) 2019-10-13T04:47:31.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:31.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:31.764416+0000) 2019-10-13T04:47:32.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:32.636+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:32.636780+0000) 2019-10-13T04:47:32.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:32.637+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:32.637771+0000) 2019-10-13T04:47:36.260 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:36.252+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.253290+0000) 2019-10-13T04:47:36.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:36.257+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.257686+0000) 2019-10-13T04:47:36.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:36.257+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.257819+0000) 2019-10-13T04:47:36.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:36.257+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.258035+0000) 2019-10-13T04:47:36.261 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:36.257+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.258300+0000) 2019-10-13T04:47:36.261 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:36.258+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.258626+0000) 2019-10-13T04:47:36.262 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:36.258+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.258841+0000) 2019-10-13T04:47:36.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:36.255+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:36.255968+0000) 2019-10-13T04:47:36.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:36.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:37.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:37.381+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:37.381259+0000) 2019-10-13T04:47:38.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:38.173+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:38.173943+0000) 2019-10-13T04:47:39.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:39.706+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:39.707478+0000) 2019-10-13T04:47:41.264 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:41.256+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.257507+0000) 2019-10-13T04:47:41.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:41.261+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.261698+0000) 2019-10-13T04:47:41.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:41.261+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.261921+0000) 2019-10-13T04:47:41.265 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:41.262+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.262477+0000) 2019-10-13T04:47:41.265 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:41.261+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.262214+0000) 2019-10-13T04:47:41.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:41.261+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.262380+0000) 2019-10-13T04:47:41.266 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:41.262+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.262942+0000) 2019-10-13T04:47:41.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:41.258+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.259160+0000) 2019-10-13T04:47:41.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:41.263+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:41.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:41.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.764584+0000) 2019-10-13T04:47:41.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:41.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.764565+0000) 2019-10-13T04:47:41.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:41.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:41.764580+0000) 2019-10-13T04:47:42.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:42.635+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:42.636974+0000) 2019-10-13T04:47:42.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:42.637+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:42.637955+0000) 2019-10-13T04:47:46.270 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:46.261+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.262092+0000) 2019-10-13T04:47:46.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:46.264+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.266065+0000) 2019-10-13T04:47:46.270 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:46.265+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.266476+0000) 2019-10-13T04:47:46.270 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:46.265+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.266627+0000) 2019-10-13T04:47:46.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:46.265+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.266922+0000) 2019-10-13T04:47:46.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:46.265+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.266785+0000) 2019-10-13T04:47:46.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:46.266+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.267459+0000) 2019-10-13T04:47:46.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:46.264+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:46.265200+0000) 2019-10-13T04:47:46.272 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:46.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:47.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:47.380+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:47.381420+0000) 2019-10-13T04:47:48.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:48.173+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:48.174134+0000) 2019-10-13T04:47:49.717 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:49.710+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:49.710687+0000) 2019-10-13T04:47:51.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:51.268+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.270130+0000) 2019-10-13T04:47:51.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:51.265+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.266213+0000) 2019-10-13T04:47:51.274 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:51.269+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.270808+0000) 2019-10-13T04:47:51.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:51.269+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.271060+0000) 2019-10-13T04:47:51.274 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:51.269+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.271248+0000) 2019-10-13T04:47:51.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:51.270+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.271555+0000) 2019-10-13T04:47:51.275 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:51.270+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.271525+0000) 2019-10-13T04:47:51.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:51.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:51.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:51.268+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.269606+0000) 2019-10-13T04:47:51.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:51.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.764742+0000) 2019-10-13T04:47:51.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:51.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.764736+0000) 2019-10-13T04:47:51.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:51.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:51.764740+0000) 2019-10-13T04:47:52.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:52.635+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:52.637178+0000) 2019-10-13T04:47:52.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:52.637+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:52.638131+0000) 2019-10-13T04:47:56.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:47:56.272+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.274351+0000) 2019-10-13T04:47:56.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:47:56.269+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.270546+0000) 2019-10-13T04:47:56.278 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:47:56.273+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.274991+0000) 2019-10-13T04:47:56.278 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:56.273+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.275254+0000) 2019-10-13T04:47:56.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:56.273+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.275285+0000) 2019-10-13T04:47:56.278 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:47:56.274+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.275517+0000) 2019-10-13T04:47:56.279 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:47:56.274+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.275650+0000) 2019-10-13T04:47:56.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:56.271+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:56.272210+0000) 2019-10-13T04:47:56.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:47:56.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:47:57.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:47:57.380+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:57.381565+0000) 2019-10-13T04:47:58.008 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:47:58.009 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:58.013 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:47:58.013 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:58.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:47:58.172+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:58.174315+0000) 2019-10-13T04:47:59.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:47:59.712+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:47:59.712959+0000) 2019-10-13T04:48:01.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:01.277+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.279105+0000) 2019-10-13T04:48:01.282 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:01.277+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.279214+0000) 2019-10-13T04:48:01.282 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:01.277+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.279368+0000) 2019-10-13T04:48:01.282 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:01.278+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.279676+0000) 2019-10-13T04:48:01.283 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:01.274+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.275352+0000) 2019-10-13T04:48:01.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:01.278+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.279951+0000) 2019-10-13T04:48:01.283 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:01.278+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.280244+0000) 2019-10-13T04:48:01.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:01.279+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:01.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:01.277+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.277840+0000) 2019-10-13T04:48:01.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:01.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.764948+0000) 2019-10-13T04:48:01.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:01.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.764948+0000) 2019-10-13T04:48:01.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:01.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:01.764948+0000) 2019-10-13T04:48:02.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:02.636+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:02.637349+0000) 2019-10-13T04:48:02.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:02.638+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:02.638305+0000) 2019-10-13T04:48:06.285 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:06.281+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.283164+0000) 2019-10-13T04:48:06.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:06.281+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.283337+0000) 2019-10-13T04:48:06.286 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:06.282+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.283669+0000) 2019-10-13T04:48:06.287 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:06.277+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.279290+0000) 2019-10-13T04:48:06.287 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:06.282+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.283952+0000) 2019-10-13T04:48:06.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:06.282+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.284069+0000) 2019-10-13T04:48:06.288 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:06.283+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.284679+0000) 2019-10-13T04:48:06.288 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:06.279+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:06.281645+0000) 2019-10-13T04:48:06.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:06.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:07.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:07.381+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:07.381745+0000) 2019-10-13T04:48:08.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:08.174+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:08.174540+0000) 2019-10-13T04:48:09.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:09.714+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:09.715384+0000) 2019-10-13T04:48:11.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:11.287+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.287532+0000) 2019-10-13T04:48:11.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:11.281+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.283693+0000) 2019-10-13T04:48:11.291 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:11.287+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.288005+0000) 2019-10-13T04:48:11.291 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:11.288+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.288394+0000) 2019-10-13T04:48:11.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:11.288+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.288543+0000) 2019-10-13T04:48:11.292 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:11.288+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.288674+0000) 2019-10-13T04:48:11.292 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:11.288+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.288919+0000) 2019-10-13T04:48:11.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:11.284+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.286175+0000) 2019-10-13T04:48:11.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:11.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63642 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:11.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:11.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.765128+0000) 2019-10-13T04:48:11.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:11.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.765130+0000) 2019-10-13T04:48:11.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:11.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:11.765128+0000) 2019-10-13T04:48:12.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:12.637+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:12.637522+0000) 2019-10-13T04:48:12.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:12.636+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:12.638511+0000) 2019-10-13T04:48:16.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:16.290+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.291209+0000) 2019-10-13T04:48:16.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:16.291+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.291410+0000) 2019-10-13T04:48:16.294 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:16.291+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.291553+0000) 2019-10-13T04:48:16.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:16.291+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.291751+0000) 2019-10-13T04:48:16.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:16.291+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.291833+0000) 2019-10-13T04:48:16.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:16.286+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.287727+0000) 2019-10-13T04:48:16.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:16.291+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.292114+0000) 2019-10-13T04:48:16.296 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:16.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:16.296 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:16.288+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:16.289973+0000) 2019-10-13T04:48:17.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:17.381+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:17.381994+0000) 2019-10-13T04:48:18.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:18.174+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:18.174734+0000) 2019-10-13T04:48:19.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:19.717+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:19.718509+0000) 2019-10-13T04:48:21.297 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:21.294+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.295105+0000) 2019-10-13T04:48:21.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:21.294+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.295296+0000) 2019-10-13T04:48:21.298 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:21.290+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.291146+0000) 2019-10-13T04:48:21.299 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:21.295+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.295703+0000) 2019-10-13T04:48:21.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:21.295+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.295877+0000) 2019-10-13T04:48:21.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:21.295+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.296051+0000) 2019-10-13T04:48:21.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:21.295+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.296162+0000) 2019-10-13T04:48:21.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:21.292+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.293223+0000) 2019-10-13T04:48:21.300 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:21.297+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:21.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:21.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.765310+0000) 2019-10-13T04:48:21.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:21.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.765309+0000) 2019-10-13T04:48:21.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:21.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:21.765315+0000) 2019-10-13T04:48:22.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:22.637+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:22.637652+0000) 2019-10-13T04:48:22.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:22.637+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:22.638661+0000) 2019-10-13T04:48:26.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:26.299+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.299408+0000) 2019-10-13T04:48:26.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:26.299+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.299926+0000) 2019-10-13T04:48:26.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:26.299+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.300142+0000) 2019-10-13T04:48:26.303 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:26.299+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.300329+0000) 2019-10-13T04:48:26.303 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:26.295+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.296043+0000) 2019-10-13T04:48:26.303 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:26.300+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.300432+0000) 2019-10-13T04:48:26.304 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:26.300+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.300851+0000) 2019-10-13T04:48:26.304 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:26.296+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:26.297439+0000) 2019-10-13T04:48:26.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:26.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:27.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:27.382+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:27.382142+0000) 2019-10-13T04:48:28.115 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:48:28.115 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:28.119 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:48:28.120 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:28.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:28.173+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:28.174875+0000) 2019-10-13T04:48:29.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:29.720+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:29.721397+0000) 2019-10-13T04:48:31.305 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:31.298+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.299095+0000) 2019-10-13T04:48:31.307 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:31.302+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.303608+0000) 2019-10-13T04:48:31.307 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:31.302+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.303780+0000) 2019-10-13T04:48:31.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:31.302+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.304115+0000) 2019-10-13T04:48:31.307 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:31.302+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.304349+0000) 2019-10-13T04:48:31.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:31.301+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.301884+0000) 2019-10-13T04:48:31.309 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:31.303+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.304434+0000) 2019-10-13T04:48:31.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:31.303+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.304603+0000) 2019-10-13T04:48:31.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:31.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:31.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:31.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.765505+0000) 2019-10-13T04:48:31.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:31.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.765503+0000) 2019-10-13T04:48:31.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:31.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:31.765508+0000) 2019-10-13T04:48:32.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:32.636+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:32.637789+0000) 2019-10-13T04:48:32.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:32.638+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:32.638767+0000) 2019-10-13T04:48:36.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:36.302+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.303374+0000) 2019-10-13T04:48:36.310 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:36.306+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.307962+0000) 2019-10-13T04:48:36.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:36.306+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.308172+0000) 2019-10-13T04:48:36.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:36.307+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.308437+0000) 2019-10-13T04:48:36.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:36.307+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.308768+0000) 2019-10-13T04:48:36.312 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:36.307+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.308874+0000) 2019-10-13T04:48:36.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:36.307+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.309034+0000) 2019-10-13T04:48:36.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:36.304+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:36.305608+0000) 2019-10-13T04:48:36.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:36.309+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:37.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:37.381+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:37.382332+0000) 2019-10-13T04:48:38.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:38.174+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:38.175065+0000) 2019-10-13T04:48:39.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:39.722+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:39.723707+0000) 2019-10-13T04:48:41.314 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:41.307+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.307933+0000) 2019-10-13T04:48:41.315 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:41.310+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.312371+0000) 2019-10-13T04:48:41.315 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:41.311+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.312854+0000) 2019-10-13T04:48:41.315 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:41.311+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.313071+0000) 2019-10-13T04:48:41.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:41.311+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.313303+0000) 2019-10-13T04:48:41.316 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:41.311+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.313282+0000) 2019-10-13T04:48:41.316 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:41.312+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.313467+0000) 2019-10-13T04:48:41.317 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:41.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63823 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:41.317 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:41.310+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.311151+0000) 2019-10-13T04:48:41.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:41.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.765691+0000) 2019-10-13T04:48:41.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:41.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.765690+0000) 2019-10-13T04:48:41.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:41.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:41.765695+0000) 2019-10-13T04:48:42.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:42.637+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:42.637943+0000) 2019-10-13T04:48:42.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:42.638+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:42.638931+0000) 2019-10-13T04:48:46.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:46.311+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.312708+0000) 2019-10-13T04:48:46.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:46.315+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.317186+0000) 2019-10-13T04:48:46.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:46.315+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.317357+0000) 2019-10-13T04:48:46.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:46.316+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.317555+0000) 2019-10-13T04:48:46.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:46.316+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.317740+0000) 2019-10-13T04:48:46.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:46.316+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.318039+0000) 2019-10-13T04:48:46.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:46.316+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.318191+0000) 2019-10-13T04:48:46.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:46.315+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:46.315797+0000) 2019-10-13T04:48:46.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:46.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:47.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:47.381+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:47.382478+0000) 2019-10-13T04:48:48.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:48.174+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:48.175257+0000) 2019-10-13T04:48:49.732 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:49.725+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:49.725845+0000) 2019-10-13T04:48:51.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:51.316+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.317299+0000) 2019-10-13T04:48:51.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:51.320+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.321706+0000) 2019-10-13T04:48:51.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:51.320+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.321951+0000) 2019-10-13T04:48:51.325 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:51.320+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.322130+0000) 2019-10-13T04:48:51.325 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:51.320+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.322336+0000) 2019-10-13T04:48:51.326 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:51.321+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.322465+0000) 2019-10-13T04:48:51.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:51.321+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.322735+0000) 2019-10-13T04:48:51.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:51.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:51.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:51.320+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.321289+0000) 2019-10-13T04:48:51.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:51.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.765877+0000) 2019-10-13T04:48:51.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:51.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.765875+0000) 2019-10-13T04:48:51.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:51.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:51.765874+0000) 2019-10-13T04:48:52.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:52.636+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:52.638147+0000) 2019-10-13T04:48:52.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:52.638+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:52.639117+0000) 2019-10-13T04:48:56.328 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:48:56.321+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.321752+0000) 2019-10-13T04:48:56.329 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:48:56.325+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.326078+0000) 2019-10-13T04:48:56.329 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:48:56.325+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.326201+0000) 2019-10-13T04:48:56.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:56.326+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.326464+0000) 2019-10-13T04:48:56.330 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:48:56.326+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.326536+0000) 2019-10-13T04:48:56.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:56.326+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.326752+0000) 2019-10-13T04:48:56.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:48:56.326+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.326997+0000) 2019-10-13T04:48:56.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:56.322+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:56.323698+0000) 2019-10-13T04:48:56.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:48:56.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:48:57.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:48:57.382+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:57.382626+0000) 2019-10-13T04:48:58.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:48:58.175+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:58.175426+0000) 2019-10-13T04:48:58.208 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:48:58.208 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:58.212 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:48:58.213 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:59.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:48:59.727+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:48:59.728178+0000) 2019-10-13T04:49:01.331 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:01.323+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.325169+0000) 2019-10-13T04:49:01.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:01.329+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.329588+0000) 2019-10-13T04:49:01.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:01.329+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.329986+0000) 2019-10-13T04:49:01.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:01.329+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.330204+0000) 2019-10-13T04:49:01.333 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:01.329+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.330362+0000) 2019-10-13T04:49:01.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:01.330+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.330628+0000) 2019-10-13T04:49:01.334 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:01.330+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.330869+0000) 2019-10-13T04:49:01.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:01.325+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.327655+0000) 2019-10-13T04:49:01.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:01.331+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:01.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:01.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.766064+0000) 2019-10-13T04:49:01.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:01.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.766065+0000) 2019-10-13T04:49:01.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:01.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:01.766063+0000) 2019-10-13T04:49:02.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:02.638+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:02.638355+0000) 2019-10-13T04:49:02.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:02.638+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:02.639293+0000) 2019-10-13T04:49:06.337 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:06.333+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.334010+0000) 2019-10-13T04:49:06.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:06.333+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.334264+0000) 2019-10-13T04:49:06.338 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:06.334+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.334469+0000) 2019-10-13T04:49:06.338 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:06.334+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.334786+0000) 2019-10-13T04:49:06.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:06.334+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.335027+0000) 2019-10-13T04:49:06.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:06.329+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.330936+0000) 2019-10-13T04:49:06.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:06.330+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.331887+0000) 2019-10-13T04:49:06.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:06.334+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:06.335293+0000) 2019-10-13T04:49:06.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:06.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 63971 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:07.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:07.382+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:07.382771+0000) 2019-10-13T04:49:08.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:08.175+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:08.175612+0000) 2019-10-13T04:49:09.737 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:09.729+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:09.730471+0000) 2019-10-13T04:49:11.340 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:11.337+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.337425+0000) 2019-10-13T04:49:11.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:11.337+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.337601+0000) 2019-10-13T04:49:11.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:11.332+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.334433+0000) 2019-10-13T04:49:11.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:11.337+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.338151+0000) 2019-10-13T04:49:11.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:11.337+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.338221+0000) 2019-10-13T04:49:11.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:11.337+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.338377+0000) 2019-10-13T04:49:11.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:11.338+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.338704+0000) 2019-10-13T04:49:11.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:11.334+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.335895+0000) 2019-10-13T04:49:11.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:11.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:11.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:11.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.766218+0000) 2019-10-13T04:49:11.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:11.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.766218+0000) 2019-10-13T04:49:11.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:11.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:11.766215+0000) 2019-10-13T04:49:12.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:12.638+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:12.638509+0000) 2019-10-13T04:49:12.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:12.638+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:12.639460+0000) 2019-10-13T04:49:16.350 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:16.336+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.337475+0000) 2019-10-13T04:49:16.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:16.339+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.340119+0000) 2019-10-13T04:49:16.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:16.339+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.341297+0000) 2019-10-13T04:49:16.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:16.340+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.341476+0000) 2019-10-13T04:49:16.351 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:16.340+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.341561+0000) 2019-10-13T04:49:16.351 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:16.340+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.341799+0000) 2019-10-13T04:49:16.351 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:16.340+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.341904+0000) 2019-10-13T04:49:16.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:16.340+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:16.342342+0000) 2019-10-13T04:49:16.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:16.341+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64032 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:17.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:17.382+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:17.382951+0000) 2019-10-13T04:49:18.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:18.174+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:18.175808+0000) 2019-10-13T04:49:19.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:19.732+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:19.732921+0000) 2019-10-13T04:49:21.346 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:21.342+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.344348+0000) 2019-10-13T04:49:21.347 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:21.340+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.340919+0000) 2019-10-13T04:49:21.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:21.343+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.344701+0000) 2019-10-13T04:49:21.348 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:21.343+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.344861+0000) 2019-10-13T04:49:21.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:21.343+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.345114+0000) 2019-10-13T04:49:21.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:21.344+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.345528+0000) 2019-10-13T04:49:21.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:21.344+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.345729+0000) 2019-10-13T04:49:21.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:21.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64062 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:21.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:21.342+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.343368+0000) 2019-10-13T04:49:21.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:21.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.766386+0000) 2019-10-13T04:49:21.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:21.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.766385+0000) 2019-10-13T04:49:21.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:21.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:21.766384+0000) 2019-10-13T04:49:22.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:22.637+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:22.638696+0000) 2019-10-13T04:49:22.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:22.639+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:22.639638+0000) 2019-10-13T04:49:26.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:26.343+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.344008+0000) 2019-10-13T04:49:26.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:26.347+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.348480+0000) 2019-10-13T04:49:26.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:26.347+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.348646+0000) 2019-10-13T04:49:26.353 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:26.347+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.348907+0000) 2019-10-13T04:49:26.353 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:26.347+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.349127+0000) 2019-10-13T04:49:26.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:26.347+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.349311+0000) 2019-10-13T04:49:26.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:26.348+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.349442+0000) 2019-10-13T04:49:26.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:26.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64090 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:26.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:26.345+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:26.346606+0000) 2019-10-13T04:49:27.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:27.382+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:27.383139+0000) 2019-10-13T04:49:28.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:28.175+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:28.175986+0000) 2019-10-13T04:49:28.310 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:49:28.311 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:28.319 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:49:28.320 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:29.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:29.734+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:29.735290+0000) 2019-10-13T04:49:31.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:31.347+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.348024+0000) 2019-10-13T04:49:31.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:31.351+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.352536+0000) 2019-10-13T04:49:31.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:31.351+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.352888+0000) 2019-10-13T04:49:31.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:31.351+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.353034+0000) 2019-10-13T04:49:31.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:31.351+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.353384+0000) 2019-10-13T04:49:31.356 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:31.352+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.353554+0000) 2019-10-13T04:49:31.356 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:31.352+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.353721+0000) 2019-10-13T04:49:31.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:31.349+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.350570+0000) 2019-10-13T04:49:31.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:31.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64121 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:31.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:31.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.766537+0000) 2019-10-13T04:49:31.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:31.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.766537+0000) 2019-10-13T04:49:31.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:31.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:31.766537+0000) 2019-10-13T04:49:32.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:32.638+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:32.638863+0000) 2019-10-13T04:49:32.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:32.639+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:32.639800+0000) 2019-10-13T04:49:36.358 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:36.355+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.356474+0000) 2019-10-13T04:49:36.360 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:36.352+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.352907+0000) 2019-10-13T04:49:36.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:36.355+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.356967+0000) 2019-10-13T04:49:36.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:36.355+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.356915+0000) 2019-10-13T04:49:36.361 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:36.355+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.357234+0000) 2019-10-13T04:49:36.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:36.355+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.357378+0000) 2019-10-13T04:49:36.361 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:36.356+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.357522+0000) 2019-10-13T04:49:36.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:36.357+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64151 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:36.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:36.355+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:36.356478+0000) 2019-10-13T04:49:37.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:37.382+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:37.383354+0000) 2019-10-13T04:49:38.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:38.175+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:38.176186+0000) 2019-10-13T04:49:39.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:39.737+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:39.737649+0000) 2019-10-13T04:49:41.363 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:41.360+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.360643+0000) 2019-10-13T04:49:41.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:41.356+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.356789+0000) 2019-10-13T04:49:41.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:41.360+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.361149+0000) 2019-10-13T04:49:41.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:41.360+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.361263+0000) 2019-10-13T04:49:41.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:41.361+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.361606+0000) 2019-10-13T04:49:41.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:41.361+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.362080+0000) 2019-10-13T04:49:41.365 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:41.362+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.362403+0000) 2019-10-13T04:49:41.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:41.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64182 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:41.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:41.359+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.359747+0000) 2019-10-13T04:49:41.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:41.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.766719+0000) 2019-10-13T04:49:41.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:41.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.766728+0000) 2019-10-13T04:49:41.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:41.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:41.766719+0000) 2019-10-13T04:49:42.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:42.638+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:42.639033+0000) 2019-10-13T04:49:42.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:42.639+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:42.639998+0000) 2019-10-13T04:49:46.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:46.364+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.364940+0000) 2019-10-13T04:49:46.369 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:46.360+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.361168+0000) 2019-10-13T04:49:46.369 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:46.365+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.365644+0000) 2019-10-13T04:49:46.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:46.365+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.365700+0000) 2019-10-13T04:49:46.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:46.365+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.365982+0000) 2019-10-13T04:49:46.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:46.365+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.366262+0000) 2019-10-13T04:49:46.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:46.365+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.366372+0000) 2019-10-13T04:49:46.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:46.363+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:46.363962+0000) 2019-10-13T04:49:46.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:46.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64210 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:47.386 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:47.383+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:47.383566+0000) 2019-10-13T04:49:48.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:48.176+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:48.176370+0000) 2019-10-13T04:49:49.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:49.738+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:49.739973+0000) 2019-10-13T04:49:51.371 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:51.363+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.365110+0000) 2019-10-13T04:49:51.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:51.369+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.369667+0000) 2019-10-13T04:49:51.372 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:51.369+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.369848+0000) 2019-10-13T04:49:51.373 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:51.369+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.370324+0000) 2019-10-13T04:49:51.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:51.369+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.370334+0000) 2019-10-13T04:49:51.373 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:51.370+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.370962+0000) 2019-10-13T04:49:51.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:51.370+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.371195+0000) 2019-10-13T04:49:51.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:51.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:51.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:51.366+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.368450+0000) 2019-10-13T04:49:51.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:51.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.766925+0000) 2019-10-13T04:49:51.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:51.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.766928+0000) 2019-10-13T04:49:51.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:51.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:51.766928+0000) 2019-10-13T04:49:52.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:52.639+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:52.639197+0000) 2019-10-13T04:49:52.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:52.639+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:52.640179+0000) 2019-10-13T04:49:56.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:49:56.373+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.373981+0000) 2019-10-13T04:49:56.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:49:56.373+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.374200+0000) 2019-10-13T04:49:56.377 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:49:56.373+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.374305+0000) 2019-10-13T04:49:56.378 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:49:56.369+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.370921+0000) 2019-10-13T04:49:56.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:56.374+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.374604+0000) 2019-10-13T04:49:56.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:49:56.374+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.374770+0000) 2019-10-13T04:49:56.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:56.374+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.374878+0000) 2019-10-13T04:49:56.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:49:56.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64271 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:49:56.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:56.371+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:56.373123+0000) 2019-10-13T04:49:57.386 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:49:57.383+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:57.383735+0000) 2019-10-13T04:49:58.183 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:49:58.176+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:58.176564+0000) 2019-10-13T04:49:58.408 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:49:58.408 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:58.412 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:49:58.413 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:59.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:49:59.741+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:49:59.742339+0000) 2019-10-13T04:50:01.380 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:01.376+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.378133+0000) 2019-10-13T04:50:01.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:01.377+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.378468+0000) 2019-10-13T04:50:01.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:01.377+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.378893+0000) 2019-10-13T04:50:01.382 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:01.377+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.379063+0000) 2019-10-13T04:50:01.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:01.377+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.379222+0000) 2019-10-13T04:50:01.382 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:01.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.379300+0000) 2019-10-13T04:50:01.383 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:01.373+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.375247+0000) 2019-10-13T04:50:01.383 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:01.375+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.376723+0000) 2019-10-13T04:50:01.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:01.379+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:01.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:01.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.767127+0000) 2019-10-13T04:50:01.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:01.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.767127+0000) 2019-10-13T04:50:01.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:01.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:01.767127+0000) 2019-10-13T04:50:02.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:02.638+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:02.639362+0000) 2019-10-13T04:50:02.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:02.639+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:02.640387+0000) 2019-10-13T04:50:06.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:06.380+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.382336+0000) 2019-10-13T04:50:06.385 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:06.381+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.382627+0000) 2019-10-13T04:50:06.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:06.381+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.382944+0000) 2019-10-13T04:50:06.386 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:06.382+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.383510+0000) 2019-10-13T04:50:06.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:06.382+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.383641+0000) 2019-10-13T04:50:06.386 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:06.382+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.383833+0000) 2019-10-13T04:50:06.387 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:06.379+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.379788+0000) 2019-10-13T04:50:06.387 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:06.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64330 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:06.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:06.381+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:06.381825+0000) 2019-10-13T04:50:07.386 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:07.383+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:07.383942+0000) 2019-10-13T04:50:08.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:08.175+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:08.176776+0000) 2019-10-13T04:50:09.751 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:09.743+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:09.744681+0000) 2019-10-13T04:50:11.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:11.385+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.387248+0000) 2019-10-13T04:50:11.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:11.386+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.387505+0000) 2019-10-13T04:50:11.390 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:11.386+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.387687+0000) 2019-10-13T04:50:11.392 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:11.386+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.387742+0000) 2019-10-13T04:50:11.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:11.383+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.384269+0000) 2019-10-13T04:50:11.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:11.385+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.386373+0000) 2019-10-13T04:50:11.394 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:11.386+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.387910+0000) 2019-10-13T04:50:11.394 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:11.387+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.388728+0000) 2019-10-13T04:50:11.395 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:11.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:11.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:11.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.767259+0000) 2019-10-13T04:50:11.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:11.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.767274+0000) 2019-10-13T04:50:11.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:11.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:11.767271+0000) 2019-10-13T04:50:12.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:12.638+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:12.639570+0000) 2019-10-13T04:50:12.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:12.640+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:12.640583+0000) 2019-10-13T04:50:16.394 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:16.389+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.391211+0000) 2019-10-13T04:50:16.394 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:16.389+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.391350+0000) 2019-10-13T04:50:16.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:16.390+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.391487+0000) 2019-10-13T04:50:16.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:16.390+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.391712+0000) 2019-10-13T04:50:16.395 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:16.390+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.391788+0000) 2019-10-13T04:50:16.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:16.386+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.387686+0000) 2019-10-13T04:50:16.396 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:16.390+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.392176+0000) 2019-10-13T04:50:16.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:16.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:16.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:16.389+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:16.390326+0000) 2019-10-13T04:50:17.386 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:17.383+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:17.384138+0000) 2019-10-13T04:50:18.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:18.176+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:18.176965+0000) 2019-10-13T04:50:19.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:19.746+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:19.747010+0000) 2019-10-13T04:50:21.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:21.393+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.394915+0000) 2019-10-13T04:50:21.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:21.393+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.395050+0000) 2019-10-13T04:50:21.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:21.394+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.395422+0000) 2019-10-13T04:50:21.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:21.393+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.395214+0000) 2019-10-13T04:50:21.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:21.394+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.395525+0000) 2019-10-13T04:50:21.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:21.394+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.395925+0000) 2019-10-13T04:50:21.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:21.391+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.392390+0000) 2019-10-13T04:50:21.399 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:21.396+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:21.400 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:21.393+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.393926+0000) 2019-10-13T04:50:21.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:21.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.767429+0000) 2019-10-13T04:50:21.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:21.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.767430+0000) 2019-10-13T04:50:21.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:21.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:21.767431+0000) 2019-10-13T04:50:22.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:22.639+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:22.639801+0000) 2019-10-13T04:50:22.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:22.640+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:22.640771+0000) 2019-10-13T04:50:26.402 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:26.399+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.399498+0000) 2019-10-13T04:50:26.402 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:26.399+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.399739+0000) 2019-10-13T04:50:26.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:26.399+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.400014+0000) 2019-10-13T04:50:26.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:26.395+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.396508+0000) 2019-10-13T04:50:26.404 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:26.399+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.400123+0000) 2019-10-13T04:50:26.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:26.399+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.400287+0000) 2019-10-13T04:50:26.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:26.400+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.400472+0000) 2019-10-13T04:50:26.405 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:26.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:26.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:26.398+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:26.398914+0000) 2019-10-13T04:50:27.386 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:27.383+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:27.384325+0000) 2019-10-13T04:50:28.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:28.176+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:28.177191+0000) 2019-10-13T04:50:28.500 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:50:28.500 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:28.506 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:50:28.506 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:29.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:29.749+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:29.749384+0000) 2019-10-13T04:50:31.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:31.403+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.403641+0000) 2019-10-13T04:50:31.407 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:31.399+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.399815+0000) 2019-10-13T04:50:31.408 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:31.403+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.404159+0000) 2019-10-13T04:50:31.408 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:31.403+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.404338+0000) 2019-10-13T04:50:31.408 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:31.404+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.404548+0000) 2019-10-13T04:50:31.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:31.404+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.404697+0000) 2019-10-13T04:50:31.408 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:31.404+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.405094+0000) 2019-10-13T04:50:31.409 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:31.401+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.401790+0000) 2019-10-13T04:50:31.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:31.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:31.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:31.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.767551+0000) 2019-10-13T04:50:31.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:31.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.767607+0000) 2019-10-13T04:50:31.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:31.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:31.767614+0000) 2019-10-13T04:50:32.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:32.639+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:32.639982+0000) 2019-10-13T04:50:32.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:32.640+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:32.640938+0000) 2019-10-13T04:50:36.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:36.407+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.408289+0000) 2019-10-13T04:50:36.412 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:36.403+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.404306+0000) 2019-10-13T04:50:36.413 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:36.408+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.408792+0000) 2019-10-13T04:50:36.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:36.408+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.408914+0000) 2019-10-13T04:50:36.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:36.408+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.409143+0000) 2019-10-13T04:50:36.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:36.409+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.409443+0000) 2019-10-13T04:50:36.413 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:36.409+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.409644+0000) 2019-10-13T04:50:36.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:36.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64511 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:36.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:36.405+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:36.406390+0000) 2019-10-13T04:50:37.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:37.384+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:37.384457+0000) 2019-10-13T04:50:38.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:38.177+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:38.177387+0000) 2019-10-13T04:50:39.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:39.750+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:39.751689+0000) 2019-10-13T04:50:41.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:41.411+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.411865+0000) 2019-10-13T04:50:41.415 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:41.406+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.407824+0000) 2019-10-13T04:50:41.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:41.411+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.412305+0000) 2019-10-13T04:50:41.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:41.412+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.412399+0000) 2019-10-13T04:50:41.416 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:41.412+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.412545+0000) 2019-10-13T04:50:41.416 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:41.412+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.412800+0000) 2019-10-13T04:50:41.416 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:41.412+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.413003+0000) 2019-10-13T04:50:41.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:41.408+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.410144+0000) 2019-10-13T04:50:41.417 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:41.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64542 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:41.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:41.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.767695+0000) 2019-10-13T04:50:41.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:41.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.767794+0000) 2019-10-13T04:50:41.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:41.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:41.767797+0000) 2019-10-13T04:50:42.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:42.640+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:42.640166+0000) 2019-10-13T04:50:42.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:42.640+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:42.641080+0000) 2019-10-13T04:50:46.418 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:46.414+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.415527+0000) 2019-10-13T04:50:46.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:46.414+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.415828+0000) 2019-10-13T04:50:46.419 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:46.409+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.411628+0000) 2019-10-13T04:50:46.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:46.414+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.416063+0000) 2019-10-13T04:50:46.419 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:46.414+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.416203+0000) 2019-10-13T04:50:46.420 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:46.415+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.416524+0000) 2019-10-13T04:50:46.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:46.415+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.416636+0000) 2019-10-13T04:50:46.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:46.411+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:46.413231+0000) 2019-10-13T04:50:46.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:46.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64570 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:47.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:47.383+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:47.384625+0000) 2019-10-13T04:50:48.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:48.176+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:48.177630+0000) 2019-10-13T04:50:49.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:49.753+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:49.754012+0000) 2019-10-13T04:50:51.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:51.418+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.419408+0000) 2019-10-13T04:50:51.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:51.413+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.415346+0000) 2019-10-13T04:50:51.423 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:51.418+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.419776+0000) 2019-10-13T04:50:51.423 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:51.418+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.420033+0000) 2019-10-13T04:50:51.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:51.418+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.420256+0000) 2019-10-13T04:50:51.423 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:51.419+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.420464+0000) 2019-10-13T04:50:51.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:51.419+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.420658+0000) 2019-10-13T04:50:51.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:51.415+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.417713+0000) 2019-10-13T04:50:51.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:51.420+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64601 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:51.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:51.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.767907+0000) 2019-10-13T04:50:51.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:51.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.767864+0000) 2019-10-13T04:50:51.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:51.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:51.767932+0000) 2019-10-13T04:50:52.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:52.638+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:52.640372+0000) 2019-10-13T04:50:52.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:52.640+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:52.641226+0000) 2019-10-13T04:50:56.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:50:56.418+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.419620+0000) 2019-10-13T04:50:56.427 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:50:56.422+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.423970+0000) 2019-10-13T04:50:56.427 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:50:56.422+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.424073+0000) 2019-10-13T04:50:56.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:56.422+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.424343+0000) 2019-10-13T04:50:56.427 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:50:56.423+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.424675+0000) 2019-10-13T04:50:56.428 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:50:56.423+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.424780+0000) 2019-10-13T04:50:56.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:56.423+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.424926+0000) 2019-10-13T04:50:56.428 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:50:56.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:50:56.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:56.422+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:56.423081+0000) 2019-10-13T04:50:57.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:50:57.384+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:57.384797+0000) 2019-10-13T04:50:58.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:50:58.177+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:58.177823+0000) 2019-10-13T04:50:58.604 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:50:58.604 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:58.608 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:50:58.608 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:59.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:50:59.755+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:50:59.756570+0000) 2019-10-13T04:51:01.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:01.426+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.428052+0000) 2019-10-13T04:51:01.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:01.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.428176+0000) 2019-10-13T04:51:01.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:01.427+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.428410+0000) 2019-10-13T04:51:01.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:01.427+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.428550+0000) 2019-10-13T04:51:01.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:01.423+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.424458+0000) 2019-10-13T04:51:01.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:01.427+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.428908+0000) 2019-10-13T04:51:01.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:01.428+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.429514+0000) 2019-10-13T04:51:01.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:01.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64662 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:01.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:01.426+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.427100+0000) 2019-10-13T04:51:01.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:01.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.768069+0000) 2019-10-13T04:51:01.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:01.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.768093+0000) 2019-10-13T04:51:01.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:01.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:01.768059+0000) 2019-10-13T04:51:02.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:02.639+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:02.640575+0000) 2019-10-13T04:51:02.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:02.640+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:02.641407+0000) 2019-10-13T04:51:06.442 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:06.428+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.429421+0000) 2019-10-13T04:51:06.442 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:06.430+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.431256+0000) 2019-10-13T04:51:06.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:06.431+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.432569+0000) 2019-10-13T04:51:06.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:06.431+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.432998+0000) 2019-10-13T04:51:06.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:06.432+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.433545+0000) 2019-10-13T04:51:06.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:06.432+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.433817+0000) 2019-10-13T04:51:06.443 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:06.432+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.434010+0000) 2019-10-13T04:51:06.443 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:06.432+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:06.434190+0000) 2019-10-13T04:51:06.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:06.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:07.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:07.383+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:07.385023+0000) 2019-10-13T04:51:08.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:08.177+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:08.178017+0000) 2019-10-13T04:51:09.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:09.758+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:09.758951+0000) 2019-10-13T04:51:11.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:11.436+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.436576+0000) 2019-10-13T04:51:11.439 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:11.436+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.436924+0000) 2019-10-13T04:51:11.440 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:11.437+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.437800+0000) 2019-10-13T04:51:11.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:11.437+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.437658+0000) 2019-10-13T04:51:11.440 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:11.437+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.438106+0000) 2019-10-13T04:51:11.441 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:11.433+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.433755+0000) 2019-10-13T04:51:11.441 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:11.437+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.438258+0000) 2019-10-13T04:51:11.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:11.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:11.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:11.435+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.436241+0000) 2019-10-13T04:51:11.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:11.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.768258+0000) 2019-10-13T04:51:11.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:11.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.768258+0000) 2019-10-13T04:51:11.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:11.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:11.768260+0000) 2019-10-13T04:51:12.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:12.640+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:12.640738+0000) 2019-10-13T04:51:12.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:12.640+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:12.641606+0000) 2019-10-13T04:51:16.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:16.440+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.440944+0000) 2019-10-13T04:51:16.444 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:16.440+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.441138+0000) 2019-10-13T04:51:16.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:16.440+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.441299+0000) 2019-10-13T04:51:16.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:16.441+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.441487+0000) 2019-10-13T04:51:16.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:16.436+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.437446+0000) 2019-10-13T04:51:16.445 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:16.441+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.441739+0000) 2019-10-13T04:51:16.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:16.441+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.442225+0000) 2019-10-13T04:51:16.446 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:16.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64751 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:16.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:16.439+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:16.439896+0000) 2019-10-13T04:51:17.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:17.384+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:17.385192+0000) 2019-10-13T04:51:18.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:18.177+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:18.178238+0000) 2019-10-13T04:51:19.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:19.761+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:19.761289+0000) 2019-10-13T04:51:21.447 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:21.444+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.445278+0000) 2019-10-13T04:51:21.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:21.445+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.446057+0000) 2019-10-13T04:51:21.449 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:21.441+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.442052+0000) 2019-10-13T04:51:21.449 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:21.446+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.446476+0000) 2019-10-13T04:51:21.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:21.446+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.446663+0000) 2019-10-13T04:51:21.449 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:21.446+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.446823+0000) 2019-10-13T04:51:21.450 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:21.446+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.447046+0000) 2019-10-13T04:51:21.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:21.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:21.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:21.443+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.444509+0000) 2019-10-13T04:51:21.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:21.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.768427+0000) 2019-10-13T04:51:21.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:21.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.768434+0000) 2019-10-13T04:51:21.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:21.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:21.768454+0000) 2019-10-13T04:51:22.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:22.640+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:22.640909+0000) 2019-10-13T04:51:22.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:22.641+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:22.641812+0000) 2019-10-13T04:51:26.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:26.449+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.449424+0000) 2019-10-13T04:51:26.452 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:26.449+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.449583+0000) 2019-10-13T04:51:26.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:26.449+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.449741+0000) 2019-10-13T04:51:26.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:26.449+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.449960+0000) 2019-10-13T04:51:26.453 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:26.449+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.450240+0000) 2019-10-13T04:51:26.454 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:26.445+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.446424+0000) 2019-10-13T04:51:26.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:26.450+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.450885+0000) 2019-10-13T04:51:26.454 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:26.451+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:26.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:26.447+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:26.448718+0000) 2019-10-13T04:51:27.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:27.384+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:27.385378+0000) 2019-10-13T04:51:28.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:28.178+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:28.178438+0000) 2019-10-13T04:51:28.746 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:51:28.746 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:28.750 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:51:28.750 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:29.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:29.762+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:29.763713+0000) 2019-10-13T04:51:31.455 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:31.451+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.453008+0000) 2019-10-13T04:51:31.455 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:31.451+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.453172+0000) 2019-10-13T04:51:31.456 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:31.452+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.453539+0000) 2019-10-13T04:51:31.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:31.453+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.454620+0000) 2019-10-13T04:51:31.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:31.448+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.450508+0000) 2019-10-13T04:51:31.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:31.453+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.454863+0000) 2019-10-13T04:51:31.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:31.453+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.455087+0000) 2019-10-13T04:51:31.458 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:31.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:31.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:31.451+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.453274+0000) 2019-10-13T04:51:31.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:31.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.768609+0000) 2019-10-13T04:51:31.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:31.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.768608+0000) 2019-10-13T04:51:31.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:31.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:31.768678+0000) 2019-10-13T04:51:32.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:32.640+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:32.641111+0000) 2019-10-13T04:51:32.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:32.641+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:32.642002+0000) 2019-10-13T04:51:36.460 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:36.456+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.457988+0000) 2019-10-13T04:51:36.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:36.456+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.458276+0000) 2019-10-13T04:51:36.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:36.457+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.458463+0000) 2019-10-13T04:51:36.461 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:36.457+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.458951+0000) 2019-10-13T04:51:36.462 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:36.457+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.459146+0000) 2019-10-13T04:51:36.462 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:36.457+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.459291+0000) 2019-10-13T04:51:36.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:36.452+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.454658+0000) 2019-10-13T04:51:36.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:36.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:36.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:36.455+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:36.457667+0000) 2019-10-13T04:51:37.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:37.384+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:37.385574+0000) 2019-10-13T04:51:38.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:38.177+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:38.178628+0000) 2019-10-13T04:51:39.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:39.765+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:39.766082+0000) 2019-10-13T04:51:41.465 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:41.461+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.462555+0000) 2019-10-13T04:51:41.465 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:41.461+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.462748+0000) 2019-10-13T04:51:41.465 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:41.461+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.463042+0000) 2019-10-13T04:51:41.466 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:41.457+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.459618+0000) 2019-10-13T04:51:41.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:41.461+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.463295+0000) 2019-10-13T04:51:41.467 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:41.461+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.463364+0000) 2019-10-13T04:51:41.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:41.462+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.463587+0000) 2019-10-13T04:51:41.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:41.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64902 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:41.468 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:41.460+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.461932+0000) 2019-10-13T04:51:41.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:41.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.768827+0000) 2019-10-13T04:51:41.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:41.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.768820+0000) 2019-10-13T04:51:41.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:41.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:41.768818+0000) 2019-10-13T04:51:42.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:42.640+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:42.641285+0000) 2019-10-13T04:51:42.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:42.641+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:42.642130+0000) 2019-10-13T04:51:46.469 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:46.461+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.462472+0000) 2019-10-13T04:51:46.469 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:46.465+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.467141+0000) 2019-10-13T04:51:46.470 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:46.465+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.467331+0000) 2019-10-13T04:51:46.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:46.466+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.467597+0000) 2019-10-13T04:51:46.470 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:46.466+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.467838+0000) 2019-10-13T04:51:46.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:46.466+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.467819+0000) 2019-10-13T04:51:46.471 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:46.466+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.468062+0000) 2019-10-13T04:51:46.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:46.463+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:46.464361+0000) 2019-10-13T04:51:46.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:46.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:47.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:47.385+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:47.385751+0000) 2019-10-13T04:51:48.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:48.178+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:48.178796+0000) 2019-10-13T04:51:49.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:49.767+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:49.768533+0000) 2019-10-13T04:51:51.473 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:51.469+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.470858+0000) 2019-10-13T04:51:51.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:51.469+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.471225+0000) 2019-10-13T04:51:51.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:51.466+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.466829+0000) 2019-10-13T04:51:51.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:51.470+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.471492+0000) 2019-10-13T04:51:51.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:51.470+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.471985+0000) 2019-10-13T04:51:51.475 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:51.471+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.472439+0000) 2019-10-13T04:51:51.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:51.471+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.472511+0000) 2019-10-13T04:51:51.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:51.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:51.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:51.468+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.469708+0000) 2019-10-13T04:51:51.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:51.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.768984+0000) 2019-10-13T04:51:51.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:51.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.768986+0000) 2019-10-13T04:51:51.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:51.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:51.768986+0000) 2019-10-13T04:51:52.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:52.640+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:52.641495+0000) 2019-10-13T04:51:52.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:52.641+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:52.642299+0000) 2019-10-13T04:51:56.477 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:51:56.470+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.471149+0000) 2019-10-13T04:51:56.478 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:51:56.475+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.475647+0000) 2019-10-13T04:51:56.479 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:51:56.475+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.475764+0000) 2019-10-13T04:51:56.479 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:51:56.475+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.475884+0000) 2019-10-13T04:51:56.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:56.475+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.476092+0000) 2019-10-13T04:51:56.479 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:56.476+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.476475+0000) 2019-10-13T04:51:56.480 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:51:56.476+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.476583+0000) 2019-10-13T04:51:56.480 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:51:56.477+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 64991 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:51:56.481 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:56.474+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:56.475218+0000) 2019-10-13T04:51:57.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:51:57.385+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:57.385923+0000) 2019-10-13T04:51:58.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:51:58.178+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:58.178938+0000) 2019-10-13T04:51:58.951 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:51:58.952 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:58.957 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:51:58.957 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:59.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:51:59.770+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:51:59.770828+0000) 2019-10-13T04:52:01.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:01.479+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.480362+0000) 2019-10-13T04:52:01.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:01.480+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.480629+0000) 2019-10-13T04:52:01.485 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:01.480+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.480452+0000) 2019-10-13T04:52:01.485 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:01.480+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.480887+0000) 2019-10-13T04:52:01.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:01.480+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.480967+0000) 2019-10-13T04:52:01.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:01.480+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.481188+0000) 2019-10-13T04:52:01.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:01.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:01.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:01.475+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.476527+0000) 2019-10-13T04:52:01.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:01.477+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.478642+0000) 2019-10-13T04:52:01.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:01.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.769126+0000) 2019-10-13T04:52:01.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:01.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.769164+0000) 2019-10-13T04:52:01.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:01.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:01.769164+0000) 2019-10-13T04:52:02.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:02.641+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:02.641679+0000) 2019-10-13T04:52:02.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:02.642+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:02.642502+0000) 2019-10-13T04:52:06.486 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:06.479+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.480039+0000) 2019-10-13T04:52:06.488 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:06.484+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.484456+0000) 2019-10-13T04:52:06.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:06.484+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.484642+0000) 2019-10-13T04:52:06.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:06.484+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.484879+0000) 2019-10-13T04:52:06.488 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:06.484+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.485008+0000) 2019-10-13T04:52:06.489 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:06.484+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.485218+0000) 2019-10-13T04:52:06.489 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:06.485+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.485593+0000) 2019-10-13T04:52:06.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:06.481+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:06.482614+0000) 2019-10-13T04:52:06.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:06.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:07.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:07.385+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:07.386096+0000) 2019-10-13T04:52:08.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:08.178+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:08.179113+0000) 2019-10-13T04:52:09.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:09.772+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:09.773138+0000) 2019-10-13T04:52:11.491 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:11.488+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.488661+0000) 2019-10-13T04:52:11.491 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:11.488+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.488898+0000) 2019-10-13T04:52:11.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:11.488+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.489145+0000) 2019-10-13T04:52:11.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:11.484+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.485087+0000) 2019-10-13T04:52:11.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:11.488+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.489388+0000) 2019-10-13T04:52:11.493 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:11.489+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.489532+0000) 2019-10-13T04:52:11.493 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:11.489+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.489752+0000) 2019-10-13T04:52:11.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:11.486+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.486878+0000) 2019-10-13T04:52:11.496 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:11.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:11.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:11.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.769303+0000) 2019-10-13T04:52:11.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:11.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.769312+0000) 2019-10-13T04:52:11.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:11.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:11.769312+0000) 2019-10-13T04:52:12.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:12.641+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:12.641860+0000) 2019-10-13T04:52:12.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:12.642+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:12.642706+0000) 2019-10-13T04:52:16.495 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:16.491+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.492899+0000) 2019-10-13T04:52:16.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:16.488+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.489152+0000) 2019-10-13T04:52:16.496 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:16.492+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.493635+0000) 2019-10-13T04:52:16.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:16.492+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.493826+0000) 2019-10-13T04:52:16.497 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:16.492+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.493986+0000) 2019-10-13T04:52:16.497 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:16.492+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.494217+0000) 2019-10-13T04:52:16.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:16.493+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.494640+0000) 2019-10-13T04:52:16.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:16.494+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:16.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:16.491+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:16.491953+0000) 2019-10-13T04:52:17.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:17.385+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:17.386230+0000) 2019-10-13T04:52:18.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:18.178+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:18.179246+0000) 2019-10-13T04:52:19.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:19.773+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:19.775660+0000) 2019-10-13T04:52:21.499 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:21.495+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.497169+0000) 2019-10-13T04:52:21.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:21.496+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.497479+0000) 2019-10-13T04:52:21.501 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:21.492+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.493927+0000) 2019-10-13T04:52:21.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:21.496+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.497748+0000) 2019-10-13T04:52:21.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:21.496+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.498279+0000) 2019-10-13T04:52:21.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:21.497+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.498438+0000) 2019-10-13T04:52:21.502 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:21.497+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.498924+0000) 2019-10-13T04:52:21.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:21.494+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.496476+0000) 2019-10-13T04:52:21.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:21.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:21.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:21.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.769501+0000) 2019-10-13T04:52:21.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:21.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.769507+0000) 2019-10-13T04:52:21.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:21.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:21.769507+0000) 2019-10-13T04:52:22.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:22.641+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:22.642040+0000) 2019-10-13T04:52:22.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:22.641+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:22.642922+0000) 2019-10-13T04:52:26.504 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:26.495+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.497390+0000) 2019-10-13T04:52:26.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:26.500+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.501876+0000) 2019-10-13T04:52:26.505 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:26.500+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.502063+0000) 2019-10-13T04:52:26.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:26.500+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.502282+0000) 2019-10-13T04:52:26.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:26.501+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.502505+0000) 2019-10-13T04:52:26.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:26.501+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.502654+0000) 2019-10-13T04:52:26.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:26.501+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.503013+0000) 2019-10-13T04:52:26.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:26.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65169 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:26.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:26.498+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:26.500428+0000) 2019-10-13T04:52:27.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:27.385+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:27.386348+0000) 2019-10-13T04:52:28.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:28.178+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:28.179439+0000) 2019-10-13T04:52:29.049 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:52:29.049 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:29.053 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:52:29.053 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:29.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:29.777+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:29.778035+0000) 2019-10-13T04:52:31.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:31.504+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.506058+0000) 2019-10-13T04:52:31.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:31.500+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.502124+0000) 2019-10-13T04:52:31.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:31.505+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.506577+0000) 2019-10-13T04:52:31.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:31.505+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.506705+0000) 2019-10-13T04:52:31.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:31.505+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.506923+0000) 2019-10-13T04:52:31.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:31.505+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.507215+0000) 2019-10-13T04:52:31.511 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:31.506+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.507461+0000) 2019-10-13T04:52:31.512 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:31.507+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:31.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:31.504+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.506426+0000) 2019-10-13T04:52:31.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:31.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.769690+0000) 2019-10-13T04:52:31.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:31.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.769690+0000) 2019-10-13T04:52:31.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:31.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:31.769690+0000) 2019-10-13T04:52:32.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:32.641+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:32.642221+0000) 2019-10-13T04:52:32.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:32.642+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:32.643101+0000) 2019-10-13T04:52:36.518 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:36.509+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.511326+0000) 2019-10-13T04:52:36.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:36.509+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.511142+0000) 2019-10-13T04:52:36.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:36.510+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.511532+0000) 2019-10-13T04:52:36.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:36.510+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.512121+0000) 2019-10-13T04:52:36.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:36.510+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.512248+0000) 2019-10-13T04:52:36.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:36.511+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.512606+0000) 2019-10-13T04:52:36.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:36.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65229 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:36.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:36.506+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.506773+0000) 2019-10-13T04:52:36.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:36.509+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:36.510397+0000) 2019-10-13T04:52:37.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:37.385+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:37.386521+0000) 2019-10-13T04:52:38.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:38.179+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:38.179641+0000) 2019-10-13T04:52:39.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:39.780+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:39.781100+0000) 2019-10-13T04:52:41.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:41.514+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.515316+0000) 2019-10-13T04:52:41.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:41.515+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.515522+0000) 2019-10-13T04:52:41.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:41.515+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.515615+0000) 2019-10-13T04:52:41.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:41.515+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.515754+0000) 2019-10-13T04:52:41.520 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:41.515+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.515849+0000) 2019-10-13T04:52:41.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:41.515+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.516150+0000) 2019-10-13T04:52:41.521 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:41.510+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.511201+0000) 2019-10-13T04:52:41.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:41.513+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.513848+0000) 2019-10-13T04:52:41.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:41.516+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65259 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:41.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:41.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.769853+0000) 2019-10-13T04:52:41.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:41.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.769872+0000) 2019-10-13T04:52:41.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:41.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:41.769872+0000) 2019-10-13T04:52:42.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:42.642+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:42.642415+0000) 2019-10-13T04:52:42.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:42.642+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:42.643284+0000) 2019-10-13T04:52:46.521 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:46.514+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.514951+0000) 2019-10-13T04:52:46.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:46.519+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.519460+0000) 2019-10-13T04:52:46.523 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:46.518+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.519347+0000) 2019-10-13T04:52:46.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:46.519+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.519699+0000) 2019-10-13T04:52:46.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:46.519+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.520180+0000) 2019-10-13T04:52:46.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:46.520+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.520792+0000) 2019-10-13T04:52:46.524 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:46.520+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.521025+0000) 2019-10-13T04:52:46.525 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:46.517+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:46.518288+0000) 2019-10-13T04:52:46.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:46.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65287 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:47.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:47.386+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:47.386690+0000) 2019-10-13T04:52:48.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:48.179+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:48.179814+0000) 2019-10-13T04:52:49.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:49.783+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:49.783443+0000) 2019-10-13T04:52:51.527 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:51.523+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.524239+0000) 2019-10-13T04:52:51.527 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:51.523+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.524350+0000) 2019-10-13T04:52:51.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:51.524+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.524512+0000) 2019-10-13T04:52:51.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:51.519+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.520477+0000) 2019-10-13T04:52:51.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:51.524+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.525048+0000) 2019-10-13T04:52:51.528 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:51.524+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.525235+0000) 2019-10-13T04:52:51.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:51.525+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.525513+0000) 2019-10-13T04:52:51.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:51.521+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.522674+0000) 2019-10-13T04:52:51.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:51.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:51.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:51.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.770022+0000) 2019-10-13T04:52:51.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:51.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.770029+0000) 2019-10-13T04:52:51.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:51.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:51.770030+0000) 2019-10-13T04:52:52.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:52.642+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:52.642611+0000) 2019-10-13T04:52:52.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:52.643+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:52.643466+0000) 2019-10-13T04:52:56.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:52:56.527+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.528149+0000) 2019-10-13T04:52:56.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:56.528+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.528403+0000) 2019-10-13T04:52:56.532 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:52:56.523+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.524271+0000) 2019-10-13T04:52:56.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:52:56.528+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.528658+0000) 2019-10-13T04:52:56.532 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:52:56.528+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.528864+0000) 2019-10-13T04:52:56.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:52:56.528+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.529053+0000) 2019-10-13T04:52:56.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:56.529+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.529482+0000) 2019-10-13T04:52:56.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:52:56.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:52:56.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:56.526+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:56.527610+0000) 2019-10-13T04:52:57.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:52:57.386+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:57.386859+0000) 2019-10-13T04:52:58.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:52:58.179+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:58.179990+0000) 2019-10-13T04:52:59.222 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:52:59.222 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:59.226 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:52:59.226 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:59.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:52:59.785+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:52:59.785785+0000) 2019-10-13T04:53:01.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:01.531+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.532613+0000) 2019-10-13T04:53:01.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:01.531+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.532756+0000) 2019-10-13T04:53:01.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:01.531+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.532891+0000) 2019-10-13T04:53:01.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:01.532+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.533798+0000) 2019-10-13T04:53:01.538 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:01.532+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.534044+0000) 2019-10-13T04:53:01.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:01.532+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.534179+0000) 2019-10-13T04:53:01.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:01.528+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.529659+0000) 2019-10-13T04:53:01.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:01.530+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.531373+0000) 2019-10-13T04:53:01.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:01.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65377 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:01.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:01.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.770208+0000) 2019-10-13T04:53:01.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:01.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.770222+0000) 2019-10-13T04:53:01.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:01.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:01.770222+0000) 2019-10-13T04:53:02.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:02.641+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:02.642770+0000) 2019-10-13T04:53:02.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:02.643+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:02.643645+0000) 2019-10-13T04:53:06.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:06.535+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.536782+0000) 2019-10-13T04:53:06.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:06.535+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.536966+0000) 2019-10-13T04:53:06.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:06.535+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.537168+0000) 2019-10-13T04:53:06.540 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:06.535+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.537336+0000) 2019-10-13T04:53:06.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:06.536+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.537483+0000) 2019-10-13T04:53:06.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:06.532+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.533403+0000) 2019-10-13T04:53:06.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:06.533+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.534341+0000) 2019-10-13T04:53:06.541 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:06.536+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:06.537753+0000) 2019-10-13T04:53:06.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:06.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65405 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:07.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:07.386+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:07.387035+0000) 2019-10-13T04:53:08.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:08.179+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:08.180198+0000) 2019-10-13T04:53:09.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:09.788+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:09.788137+0000) 2019-10-13T04:53:11.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:11.539+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.541027+0000) 2019-10-13T04:53:11.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:11.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.541195+0000) 2019-10-13T04:53:11.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:11.540+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.541400+0000) 2019-10-13T04:53:11.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:11.540+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.541520+0000) 2019-10-13T04:53:11.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:11.535+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.537541+0000) 2019-10-13T04:53:11.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:11.540+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.542013+0000) 2019-10-13T04:53:11.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:11.540+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.542174+0000) 2019-10-13T04:53:11.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:11.537+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.539163+0000) 2019-10-13T04:53:11.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:11.542+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:11.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:11.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.770411+0000) 2019-10-13T04:53:11.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:11.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.770420+0000) 2019-10-13T04:53:11.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:11.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:11.770420+0000) 2019-10-13T04:53:12.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:12.641+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:12.642970+0000) 2019-10-13T04:53:12.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:12.642+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:12.643825+0000) 2019-10-13T04:53:16.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:16.544+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.546099+0000) 2019-10-13T04:53:16.549 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:16.544+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.546260+0000) 2019-10-13T04:53:16.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:16.541+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.543048+0000) 2019-10-13T04:53:16.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:16.545+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.546680+0000) 2019-10-13T04:53:16.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:16.545+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.546777+0000) 2019-10-13T04:53:16.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:16.545+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.547167+0000) 2019-10-13T04:53:16.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:16.546+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.547572+0000) 2019-10-13T04:53:16.551 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:16.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:16.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:16.543+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:16.545711+0000) 2019-10-13T04:53:17.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:17.386+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:17.387203+0000) 2019-10-13T04:53:18.183 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:18.179+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:18.180388+0000) 2019-10-13T04:53:19.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:19.789+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:19.790533+0000) 2019-10-13T04:53:21.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:21.544+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.545840+0000) 2019-10-13T04:53:21.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:21.548+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.550228+0000) 2019-10-13T04:53:21.561 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:21.549+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.550780+0000) 2019-10-13T04:53:21.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:21.549+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.550964+0000) 2019-10-13T04:53:21.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:21.549+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.551142+0000) 2019-10-13T04:53:21.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:21.547+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.549349+0000) 2019-10-13T04:53:21.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:21.549+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.551384+0000) 2019-10-13T04:53:21.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:21.550+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.551862+0000) 2019-10-13T04:53:21.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:21.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65495 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:21.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:21.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.770598+0000) 2019-10-13T04:53:21.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:21.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.770606+0000) 2019-10-13T04:53:21.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:21.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:21.770606+0000) 2019-10-13T04:53:22.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:22.642+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:22.643129+0000) 2019-10-13T04:53:22.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:22.643+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:22.643995+0000) 2019-10-13T04:53:26.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:26.553+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.553896+0000) 2019-10-13T04:53:26.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:26.553+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.554029+0000) 2019-10-13T04:53:26.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:26.553+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.554374+0000) 2019-10-13T04:53:26.558 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:26.549+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.550377+0000) 2019-10-13T04:53:26.558 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:26.554+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.554726+0000) 2019-10-13T04:53:26.559 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:26.554+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.554852+0000) 2019-10-13T04:53:26.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:26.554+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.555107+0000) 2019-10-13T04:53:26.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:26.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65523 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:26.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:26.552+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:26.553244+0000) 2019-10-13T04:53:27.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:27.386+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:27.387373+0000) 2019-10-13T04:53:28.183 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:28.180+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:28.180591+0000) 2019-10-13T04:53:29.552 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:53:29.553 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:29.558 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:53:29.558 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:29.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:29.792+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:29.792878+0000) 2019-10-13T04:53:31.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:31.557+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.557723+0000) 2019-10-13T04:53:31.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:31.557+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.557889+0000) 2019-10-13T04:53:31.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:31.557+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.558259+0000) 2019-10-13T04:53:31.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:31.558+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.558548+0000) 2019-10-13T04:53:31.563 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:31.553+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.554138+0000) 2019-10-13T04:53:31.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:31.555+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.556100+0000) 2019-10-13T04:53:31.563 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:31.558+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.558860+0000) 2019-10-13T04:53:31.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:31.558+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.559048+0000) 2019-10-13T04:53:31.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:31.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:31.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:31.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.770789+0000) 2019-10-13T04:53:31.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:31.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.770790+0000) 2019-10-13T04:53:31.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:31.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:31.770791+0000) 2019-10-13T04:53:32.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:32.643+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:32.643305+0000) 2019-10-13T04:53:32.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:32.643+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:32.644138+0000) 2019-10-13T04:53:36.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:36.561+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.562023+0000) 2019-10-13T04:53:36.565 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:36.561+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.562168+0000) 2019-10-13T04:53:36.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:36.562+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.562450+0000) 2019-10-13T04:53:36.566 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:36.558+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.558849+0000) 2019-10-13T04:53:36.566 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:36.562+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.562691+0000) 2019-10-13T04:53:36.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:36.562+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.562959+0000) 2019-10-13T04:53:36.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:36.562+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.563161+0000) 2019-10-13T04:53:36.567 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:36.563+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:36.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:36.559+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:36.560186+0000) 2019-10-13T04:53:37.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:37.387+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:37.387583+0000) 2019-10-13T04:53:38.183 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:38.180+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:38.180788+0000) 2019-10-13T04:53:39.801 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:39.794+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:39.795098+0000) 2019-10-13T04:53:41.568 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:41.565+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.565562+0000) 2019-10-13T04:53:41.568 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:41.565+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.565762+0000) 2019-10-13T04:53:41.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:41.565+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.566033+0000) 2019-10-13T04:53:41.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:41.565+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.566165+0000) 2019-10-13T04:53:41.570 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:41.561+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.562286+0000) 2019-10-13T04:53:41.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:41.566+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.566643+0000) 2019-10-13T04:53:41.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:41.566+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.566770+0000) 2019-10-13T04:53:41.570 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:41.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:41.571 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:41.563+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.564285+0000) 2019-10-13T04:53:41.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:41.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.770950+0000) 2019-10-13T04:53:41.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:41.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.770971+0000) 2019-10-13T04:53:41.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:41.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:41.770971+0000) 2019-10-13T04:53:42.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:42.643+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:42.643444+0000) 2019-10-13T04:53:42.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:42.643+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:42.644263+0000) 2019-10-13T04:53:46.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:46.568+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.570038+0000) 2019-10-13T04:53:46.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:46.565+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.566321+0000) 2019-10-13T04:53:46.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:46.568+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.570387+0000) 2019-10-13T04:53:46.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:46.569+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.570716+0000) 2019-10-13T04:53:46.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:46.569+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.570900+0000) 2019-10-13T04:53:46.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:46.569+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.571272+0000) 2019-10-13T04:53:46.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:46.569+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.571378+0000) 2019-10-13T04:53:46.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:46.567+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:46.568050+0000) 2019-10-13T04:53:46.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:46.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:47.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:47.386+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:47.387753+0000) 2019-10-13T04:53:48.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:48.179+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:48.180948+0000) 2019-10-13T04:53:49.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:49.796+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:49.797532+0000) 2019-10-13T04:53:51.578 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:51.572+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.574067+0000) 2019-10-13T04:53:51.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:51.572+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.574280+0000) 2019-10-13T04:53:51.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:51.573+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.574436+0000) 2019-10-13T04:53:51.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:51.573+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.574913+0000) 2019-10-13T04:53:51.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:51.573+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.574807+0000) 2019-10-13T04:53:51.579 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:51.573+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.575381+0000) 2019-10-13T04:53:51.580 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:51.570+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.570870+0000) 2019-10-13T04:53:51.580 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:51.572+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.573079+0000) 2019-10-13T04:53:51.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:51.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:51.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:51.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.771158+0000) 2019-10-13T04:53:51.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:51.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.771184+0000) 2019-10-13T04:53:51.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:51.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:51.771184+0000) 2019-10-13T04:53:52.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:52.642+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:52.643566+0000) 2019-10-13T04:53:52.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:52.644+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:52.644423+0000) 2019-10-13T04:53:56.580 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:53:56.573+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.573792+0000) 2019-10-13T04:53:56.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:53:56.576+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.578177+0000) 2019-10-13T04:53:56.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:56.577+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.578580+0000) 2019-10-13T04:53:56.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:53:56.577+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.578669+0000) 2019-10-13T04:53:56.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:53:56.577+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.578854+0000) 2019-10-13T04:53:56.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:56.577+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.579176+0000) 2019-10-13T04:53:56.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:53:56.578+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.579734+0000) 2019-10-13T04:53:56.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:56.575+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:56.576232+0000) 2019-10-13T04:53:56.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:53:56.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:53:57.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:53:57.387+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:57.387932+0000) 2019-10-13T04:53:58.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:53:58.180+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:58.181183+0000) 2019-10-13T04:53:59.682 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:53:59.682 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:59.687 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:53:59.687 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:59.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:53:59.800+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:53:59.800152+0000) 2019-10-13T04:54:01.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:01.581+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.582751+0000) 2019-10-13T04:54:01.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:01.576+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.578716+0000) 2019-10-13T04:54:01.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:01.581+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.583191+0000) 2019-10-13T04:54:01.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:01.582+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.583461+0000) 2019-10-13T04:54:01.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:01.582+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.583869+0000) 2019-10-13T04:54:01.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:01.582+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.583966+0000) 2019-10-13T04:54:01.587 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:01.582+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.584161+0000) 2019-10-13T04:54:01.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:01.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:01.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:01.580+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.582451+0000) 2019-10-13T04:54:01.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:01.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.771333+0000) 2019-10-13T04:54:01.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:01.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.771334+0000) 2019-10-13T04:54:01.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:01.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:01.771336+0000) 2019-10-13T04:54:02.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:02.643+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:02.643732+0000) 2019-10-13T04:54:02.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:02.643+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:02.644572+0000) 2019-10-13T04:54:06.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:06.582+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.584053+0000) 2019-10-13T04:54:06.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:06.583+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.585527+0000) 2019-10-13T04:54:06.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:06.586+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.587627+0000) 2019-10-13T04:54:06.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:06.586+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.587804+0000) 2019-10-13T04:54:06.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:06.586+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.588143+0000) 2019-10-13T04:54:06.599 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:06.587+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.588547+0000) 2019-10-13T04:54:06.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:06.587+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.588673+0000) 2019-10-13T04:54:06.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:06.587+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:06.588788+0000) 2019-10-13T04:54:06.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:06.589+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65759 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:07.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:07.387+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:07.388081+0000) 2019-10-13T04:54:08.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:08.180+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:08.181364+0000) 2019-10-13T04:54:09.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:09.801+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:09.802512+0000) 2019-10-13T04:54:11.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:11.591+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.591916+0000) 2019-10-13T04:54:11.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:11.591+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.592220+0000) 2019-10-13T04:54:11.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:11.592+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.592479+0000) 2019-10-13T04:54:11.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:11.592+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.592606+0000) 2019-10-13T04:54:11.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:11.586+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.588141+0000) 2019-10-13T04:54:11.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:11.588+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.590612+0000) 2019-10-13T04:54:11.617 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:11.592+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.593088+0000) 2019-10-13T04:54:11.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:11.592+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.593248+0000) 2019-10-13T04:54:11.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:11.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65789 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:11.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:11.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.771522+0000) 2019-10-13T04:54:11.790 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:11.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.771525+0000) 2019-10-13T04:54:11.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:11.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:11.771525+0000) 2019-10-13T04:54:12.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:12.643+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:12.643919+0000) 2019-10-13T04:54:12.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:12.643+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:12.644762+0000) 2019-10-13T04:54:16.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:16.595+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.595854+0000) 2019-10-13T04:54:16.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:16.595+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.596135+0000) 2019-10-13T04:54:16.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:16.595+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.596295+0000) 2019-10-13T04:54:16.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:16.591+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.592212+0000) 2019-10-13T04:54:16.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:16.596+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.596557+0000) 2019-10-13T04:54:16.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:16.596+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.596797+0000) 2019-10-13T04:54:16.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:16.596+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.597124+0000) 2019-10-13T04:54:16.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:16.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65819 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:16.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:16.594+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:16.594909+0000) 2019-10-13T04:54:17.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:17.387+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:17.388245+0000) 2019-10-13T04:54:18.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:18.181+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:18.181568+0000) 2019-10-13T04:54:19.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:19.804+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:19.804823+0000) 2019-10-13T04:54:21.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:21.595+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.595951+0000) 2019-10-13T04:54:21.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:21.599+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.600024+0000) 2019-10-13T04:54:21.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:21.600+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.600401+0000) 2019-10-13T04:54:21.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:21.600+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.600576+0000) 2019-10-13T04:54:21.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:21.600+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.600712+0000) 2019-10-13T04:54:21.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:21.600+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.600834+0000) 2019-10-13T04:54:21.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:21.600+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.601059+0000) 2019-10-13T04:54:21.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:21.597+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.597821+0000) 2019-10-13T04:54:21.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:21.602+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65849 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:21.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:21.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.771694+0000) 2019-10-13T04:54:21.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:21.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.771704+0000) 2019-10-13T04:54:21.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:21.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:21.771705+0000) 2019-10-13T04:54:22.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:22.643+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:22.644106+0000) 2019-10-13T04:54:22.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:22.644+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:22.644918+0000) 2019-10-13T04:54:26.607 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:26.603+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.604075+0000) 2019-10-13T04:54:26.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:26.603+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.604267+0000) 2019-10-13T04:54:26.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:26.604+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.604512+0000) 2019-10-13T04:54:26.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:26.600+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.600753+0000) 2019-10-13T04:54:26.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:26.604+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.604808+0000) 2019-10-13T04:54:26.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:26.605+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.605421+0000) 2019-10-13T04:54:26.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:26.605+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.605592+0000) 2019-10-13T04:54:26.609 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:26.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65877 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:26.610 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:26.602+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:26.603443+0000) 2019-10-13T04:54:27.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:27.388+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:27.388415+0000) 2019-10-13T04:54:28.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:28.181+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:28.181761+0000) 2019-10-13T04:54:29.816 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:54:29.816 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:54:29.817 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:29.806+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:29.807320+0000) 2019-10-13T04:54:29.823 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:54:29.823 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:54:31.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:31.606+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.607949+0000) 2019-10-13T04:54:31.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:31.607+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.609319+0000) 2019-10-13T04:54:31.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:31.608+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.609616+0000) 2019-10-13T04:54:31.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:31.608+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.610018+0000) 2019-10-13T04:54:31.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:31.608+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.610169+0000) 2019-10-13T04:54:31.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:31.608+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.610344+0000) 2019-10-13T04:54:31.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:31.604+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.605565+0000) 2019-10-13T04:54:31.614 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:31.609+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:31.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:31.607+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.608587+0000) 2019-10-13T04:54:31.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:31.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.771873+0000) 2019-10-13T04:54:31.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:31.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.771885+0000) 2019-10-13T04:54:31.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:31.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:31.771885+0000) 2019-10-13T04:54:32.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:32.643+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:32.644271+0000) 2019-10-13T04:54:32.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:32.644+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:32.645102+0000) 2019-10-13T04:54:36.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:36.611+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.612919+0000) 2019-10-13T04:54:36.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:36.611+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.613122+0000) 2019-10-13T04:54:36.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:36.611+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.613370+0000) 2019-10-13T04:54:36.617 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:36.612+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.613637+0000) 2019-10-13T04:54:36.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:36.612+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.613786+0000) 2019-10-13T04:54:36.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:36.612+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.614050+0000) 2019-10-13T04:54:36.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:36.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.609728+0000) 2019-10-13T04:54:36.619 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:36.609+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:36.610614+0000) 2019-10-13T04:54:36.619 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:36.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:37.391 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:37.387+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:37.388585+0000) 2019-10-13T04:54:38.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:38.181+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:38.181953+0000) 2019-10-13T04:54:39.817 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:39.809+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:39.809643+0000) 2019-10-13T04:54:41.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:41.615+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.616574+0000) 2019-10-13T04:54:41.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:41.615+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.616749+0000) 2019-10-13T04:54:41.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:41.612+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.612749+0000) 2019-10-13T04:54:41.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:41.616+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.617473+0000) 2019-10-13T04:54:41.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:41.616+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.617650+0000) 2019-10-13T04:54:41.621 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:41.616+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.617984+0000) 2019-10-13T04:54:41.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:41.616+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.618223+0000) 2019-10-13T04:54:41.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:41.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:41.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:41.616+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.617070+0000) 2019-10-13T04:54:41.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:41.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.772079+0000) 2019-10-13T04:54:41.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:41.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.772075+0000) 2019-10-13T04:54:41.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:41.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:41.772075+0000) 2019-10-13T04:54:42.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:42.643+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:42.644465+0000) 2019-10-13T04:54:42.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:42.645+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:42.645250+0000) 2019-10-13T04:54:46.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:46.619+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.621349+0000) 2019-10-13T04:54:46.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:46.620+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.621572+0000) 2019-10-13T04:54:46.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:46.620+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.621731+0000) 2019-10-13T04:54:46.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:46.620+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.621894+0000) 2019-10-13T04:54:46.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:46.617+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.617824+0000) 2019-10-13T04:54:46.626 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:46.618+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.619035+0000) 2019-10-13T04:54:46.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:46.620+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.622186+0000) 2019-10-13T04:54:46.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:46.620+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:46.622264+0000) 2019-10-13T04:54:46.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:46.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 65995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:47.391 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:47.388+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:47.388748+0000) 2019-10-13T04:54:48.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:48.181+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:48.182178+0000) 2019-10-13T04:54:49.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:49.812+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:49.812120+0000) 2019-10-13T04:54:51.627 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:51.623+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.624895+0000) 2019-10-13T04:54:51.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:51.619+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.620967+0000) 2019-10-13T04:54:51.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:51.623+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.625312+0000) 2019-10-13T04:54:51.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:51.624+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.625668+0000) 2019-10-13T04:54:51.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:51.624+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.625558+0000) 2019-10-13T04:54:51.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:51.624+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.625801+0000) 2019-10-13T04:54:51.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:51.624+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.625943+0000) 2019-10-13T04:54:51.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:51.621+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.622909+0000) 2019-10-13T04:54:51.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:51.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:51.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:51.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.772256+0000) 2019-10-13T04:54:51.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:51.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.772263+0000) 2019-10-13T04:54:51.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:51.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:51.772264+0000) 2019-10-13T04:54:52.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:52.644+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:52.644651+0000) 2019-10-13T04:54:52.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:52.644+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:52.645424+0000) 2019-10-13T04:54:56.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:54:56.627+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.629275+0000) 2019-10-13T04:54:56.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:56.628+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.629426+0000) 2019-10-13T04:54:56.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:54:56.628+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.629792+0000) 2019-10-13T04:54:56.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:56.628+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.629749+0000) 2019-10-13T04:54:56.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:54:56.624+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.625751+0000) 2019-10-13T04:54:56.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:54:56.628+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.630112+0000) 2019-10-13T04:54:56.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:54:56.628+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.630315+0000) 2019-10-13T04:54:56.634 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:54:56.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:54:56.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:56.626+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:56.628229+0000) 2019-10-13T04:54:57.391 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:54:57.388+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:57.388920+0000) 2019-10-13T04:54:58.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:54:58.181+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:58.182349+0000) 2019-10-13T04:54:59.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:54:59.813+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:54:59.814474+0000) 2019-10-13T04:55:00.112 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:55:00.112 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:00.116 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:55:00.116 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:01.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:01.633+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.633582+0000) 2019-10-13T04:55:01.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:01.633+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.633854+0000) 2019-10-13T04:55:01.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:01.633+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.634016+0000) 2019-10-13T04:55:01.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:01.633+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.634148+0000) 2019-10-13T04:55:01.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:01.634+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.634410+0000) 2019-10-13T04:55:01.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:01.628+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.630441+0000) 2019-10-13T04:55:01.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:01.630+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.632411+0000) 2019-10-13T04:55:01.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:01.634+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.635035+0000) 2019-10-13T04:55:01.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:01.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:01.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:01.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.772445+0000) 2019-10-13T04:55:01.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:01.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.772453+0000) 2019-10-13T04:55:01.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:01.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:01.772454+0000) 2019-10-13T04:55:02.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:02.644+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:02.644841+0000) 2019-10-13T04:55:02.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:02.644+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:02.645573+0000) 2019-10-13T04:55:06.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:06.637+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.637821+0000) 2019-10-13T04:55:06.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:06.638+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.638389+0000) 2019-10-13T04:55:06.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:06.638+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.638496+0000) 2019-10-13T04:55:06.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:06.638+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.638721+0000) 2019-10-13T04:55:06.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:06.633+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.634609+0000) 2019-10-13T04:55:06.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:06.638+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.639008+0000) 2019-10-13T04:55:06.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:06.638+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.639372+0000) 2019-10-13T04:55:06.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:06.640+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:06.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:06.636+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:06.636867+0000) 2019-10-13T04:55:07.391 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:07.388+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:07.389087+0000) 2019-10-13T04:55:08.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:08.182+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:08.182500+0000) 2019-10-13T04:55:09.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:09.816+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:09.816765+0000) 2019-10-13T04:55:11.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:11.637+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.638410+0000) 2019-10-13T04:55:11.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:11.642+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.642787+0000) 2019-10-13T04:55:11.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:11.642+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.643195+0000) 2019-10-13T04:55:11.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:11.643+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.643669+0000) 2019-10-13T04:55:11.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:11.643+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.643548+0000) 2019-10-13T04:55:11.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:11.643+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.643787+0000) 2019-10-13T04:55:11.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:11.643+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.643899+0000) 2019-10-13T04:55:11.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:11.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:11.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:11.640+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.641185+0000) 2019-10-13T04:55:11.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:11.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.772584+0000) 2019-10-13T04:55:11.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:11.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.772579+0000) 2019-10-13T04:55:11.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:11.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:11.772579+0000) 2019-10-13T04:55:12.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:12.644+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:12.644965+0000) 2019-10-13T04:55:12.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:12.645+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:12.645787+0000) 2019-10-13T04:55:16.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:16.642+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.642881+0000) 2019-10-13T04:55:16.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:16.646+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.647495+0000) 2019-10-13T04:55:16.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:16.646+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.647590+0000) 2019-10-13T04:55:16.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:16.646+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.647763+0000) 2019-10-13T04:55:16.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:16.646+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.648053+0000) 2019-10-13T04:55:16.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:16.647+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.648404+0000) 2019-10-13T04:55:16.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:16.647+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.648554+0000) 2019-10-13T04:55:16.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:16.644+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:16.645727+0000) 2019-10-13T04:55:16.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:16.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66173 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:17.391 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:17.388+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:17.389249+0000) 2019-10-13T04:55:18.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:18.181+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:18.182688+0000) 2019-10-13T04:55:19.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:19.818+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:19.819128+0000) 2019-10-13T04:55:21.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:21.650+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.651651+0000) 2019-10-13T04:55:21.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:21.650+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.651798+0000) 2019-10-13T04:55:21.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:21.650+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.652035+0000) 2019-10-13T04:55:21.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:21.650+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.652168+0000) 2019-10-13T04:55:21.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:21.646+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.647693+0000) 2019-10-13T04:55:21.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:21.650+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.652384+0000) 2019-10-13T04:55:21.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:21.651+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.653129+0000) 2019-10-13T04:55:21.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:21.649+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.650317+0000) 2019-10-13T04:55:21.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:21.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:21.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:21.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.772733+0000) 2019-10-13T04:55:21.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:21.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.772733+0000) 2019-10-13T04:55:21.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:21.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:21.772736+0000) 2019-10-13T04:55:22.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:22.644+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:22.645122+0000) 2019-10-13T04:55:22.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:22.645+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:22.645993+0000) 2019-10-13T04:55:26.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:26.651+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.651936+0000) 2019-10-13T04:55:26.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:26.654+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.655940+0000) 2019-10-13T04:55:26.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:26.654+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.656283+0000) 2019-10-13T04:55:26.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:26.655+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.656813+0000) 2019-10-13T04:55:26.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:26.655+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.656983+0000) 2019-10-13T04:55:26.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:26.655+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.657266+0000) 2019-10-13T04:55:26.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:26.656+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.657410+0000) 2019-10-13T04:55:26.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:26.654+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:26.654892+0000) 2019-10-13T04:55:26.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:26.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:27.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:27.388+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:27.389416+0000) 2019-10-13T04:55:28.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:28.181+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:28.182883+0000) 2019-10-13T04:55:29.828 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:29.821+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:29.821529+0000) 2019-10-13T04:55:30.251 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:55:30.251 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:30.255 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:55:30.256 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:31.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:31.659+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.660674+0000) 2019-10-13T04:55:31.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:31.659+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.660991+0000) 2019-10-13T04:55:31.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:31.656+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.656930+0000) 2019-10-13T04:55:31.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:31.659+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.661351+0000) 2019-10-13T04:55:31.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:31.660+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.661559+0000) 2019-10-13T04:55:31.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:31.660+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.661748+0000) 2019-10-13T04:55:31.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:31.660+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.662089+0000) 2019-10-13T04:55:31.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:31.658+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.658965+0000) 2019-10-13T04:55:31.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:31.662+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66261 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:31.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:31.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.772904+0000) 2019-10-13T04:55:31.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:31.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.772906+0000) 2019-10-13T04:55:31.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:31.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:31.772906+0000) 2019-10-13T04:55:32.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:32.644+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:32.645232+0000) 2019-10-13T04:55:32.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:32.645+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:32.646146+0000) 2019-10-13T04:55:36.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:36.660+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.660942+0000) 2019-10-13T04:55:36.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:36.663+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.665376+0000) 2019-10-13T04:55:36.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:36.664+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.665649+0000) 2019-10-13T04:55:36.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:36.664+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.666080+0000) 2019-10-13T04:55:36.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:36.665+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.666443+0000) 2019-10-13T04:55:36.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:36.665+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.666589+0000) 2019-10-13T04:55:36.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:36.665+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.666737+0000) 2019-10-13T04:55:36.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:36.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:36.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:36.663+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:36.664147+0000) 2019-10-13T04:55:37.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:37.389+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:37.389604+0000) 2019-10-13T04:55:38.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:38.182+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:38.183080+0000) 2019-10-13T04:55:39.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:39.823+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:39.823922+0000) 2019-10-13T04:55:41.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:41.663+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.665477+0000) 2019-10-13T04:55:41.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:41.668+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.669766+0000) 2019-10-13T04:55:41.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:41.668+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.669934+0000) 2019-10-13T04:55:41.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:41.668+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.670150+0000) 2019-10-13T04:55:41.674 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:41.668+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.670177+0000) 2019-10-13T04:55:41.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:41.669+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.670511+0000) 2019-10-13T04:55:41.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:41.669+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.670856+0000) 2019-10-13T04:55:41.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:41.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:41.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:41.667+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.669156+0000) 2019-10-13T04:55:41.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:41.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.773004+0000) 2019-10-13T04:55:41.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:41.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.773052+0000) 2019-10-13T04:55:41.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:41.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:41.773077+0000) 2019-10-13T04:55:42.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:42.644+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:42.645379+0000) 2019-10-13T04:55:42.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:42.645+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:42.646289+0000) 2019-10-13T04:55:46.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:46.673+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.674062+0000) 2019-10-13T04:55:46.678 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:46.673+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.674238+0000) 2019-10-13T04:55:46.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:46.674+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.674424+0000) 2019-10-13T04:55:46.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:46.674+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.674689+0000) 2019-10-13T04:55:46.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:46.674+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.675101+0000) 2019-10-13T04:55:46.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:46.674+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.675335+0000) 2019-10-13T04:55:46.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:46.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.670325+0000) 2019-10-13T04:55:46.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:46.670+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:46.672667+0000) 2019-10-13T04:55:46.682 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:46.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:47.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:47.389+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:47.389785+0000) 2019-10-13T04:55:48.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:48.182+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:48.183240+0000) 2019-10-13T04:55:49.833 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:49.825+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:49.826323+0000) 2019-10-13T04:55:51.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:51.672+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.674437+0000) 2019-10-13T04:55:51.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:51.674+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.676302+0000) 2019-10-13T04:55:51.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:51.677+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.678151+0000) 2019-10-13T04:55:51.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:51.678+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.678802+0000) 2019-10-13T04:55:51.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:51.678+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.678835+0000) 2019-10-13T04:55:51.687 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:51.678+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.678997+0000) 2019-10-13T04:55:51.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:51.678+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.679298+0000) 2019-10-13T04:55:51.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:51.679+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.679404+0000) 2019-10-13T04:55:51.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:51.680+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66379 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:51.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:51.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.773118+0000) 2019-10-13T04:55:51.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:51.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.773168+0000) 2019-10-13T04:55:51.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:51.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:51.773139+0000) 2019-10-13T04:55:52.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:52.645+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:52.645555+0000) 2019-10-13T04:55:52.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:52.645+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:52.646433+0000) 2019-10-13T04:55:56.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:55:56.682+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.682510+0000) 2019-10-13T04:55:56.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:56.682+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.682774+0000) 2019-10-13T04:55:56.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:55:56.682+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.682917+0000) 2019-10-13T04:55:56.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:55:56.682+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.683185+0000) 2019-10-13T04:55:56.686 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:56.683+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.683450+0000) 2019-10-13T04:55:56.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:55:56.683+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.683597+0000) 2019-10-13T04:55:56.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:55:56.678+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.679190+0000) 2019-10-13T04:55:56.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:55:56.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66409 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:55:56.688 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:56.680+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:56.681525+0000) 2019-10-13T04:55:57.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:55:57.389+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:57.389964+0000) 2019-10-13T04:55:58.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:55:58.183+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:58.183472+0000) 2019-10-13T04:55:59.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:55:59.828+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:55:59.828674+0000) 2019-10-13T04:56:00.456 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:56:00.457 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:00.461 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:56:00.461 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:01.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:01.685+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.686767+0000) 2019-10-13T04:56:01.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:01.685+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.686910+0000) 2019-10-13T04:56:01.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:01.685+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.687083+0000) 2019-10-13T04:56:01.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:01.685+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.687169+0000) 2019-10-13T04:56:01.690 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:01.685+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.687259+0000) 2019-10-13T04:56:01.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:01.686+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.687617+0000) 2019-10-13T04:56:01.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:01.683+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.683808+0000) 2019-10-13T04:56:01.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:01.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66439 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:01.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:01.684+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.685705+0000) 2019-10-13T04:56:01.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:01.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.773241+0000) 2019-10-13T04:56:01.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:01.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.773276+0000) 2019-10-13T04:56:01.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:01.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:01.773292+0000) 2019-10-13T04:56:02.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:02.644+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:02.645736+0000) 2019-10-13T04:56:02.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:02.645+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:02.646594+0000) 2019-10-13T04:56:06.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:06.689+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.690725+0000) 2019-10-13T04:56:06.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:06.686+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.687069+0000) 2019-10-13T04:56:06.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:06.689+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.691010+0000) 2019-10-13T04:56:06.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:06.689+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.691236+0000) 2019-10-13T04:56:06.695 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:06.689+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.691025+0000) 2019-10-13T04:56:06.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:06.690+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.691698+0000) 2019-10-13T04:56:06.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:06.690+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.691994+0000) 2019-10-13T04:56:06.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:06.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:06.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:06.688+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:06.689298+0000) 2019-10-13T04:56:07.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:07.389+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:07.390133+0000) 2019-10-13T04:56:08.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:08.182+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:08.183658+0000) 2019-10-13T04:56:09.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:09.830+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:09.830839+0000) 2019-10-13T04:56:11.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:11.690+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.691122+0000) 2019-10-13T04:56:11.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:11.694+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.695511+0000) 2019-10-13T04:56:11.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:11.694+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.695830+0000) 2019-10-13T04:56:11.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:11.694+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.696010+0000) 2019-10-13T04:56:11.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:11.694+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.696162+0000) 2019-10-13T04:56:11.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:11.695+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.696455+0000) 2019-10-13T04:56:11.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:11.695+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.696615+0000) 2019-10-13T04:56:11.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:11.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:11.701 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:11.694+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.694761+0000) 2019-10-13T04:56:11.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:11.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.773413+0000) 2019-10-13T04:56:11.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:11.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.773436+0000) 2019-10-13T04:56:11.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:11.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:11.773437+0000) 2019-10-13T04:56:12.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:12.645+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:12.645902+0000) 2019-10-13T04:56:12.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:12.646+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:12.646772+0000) 2019-10-13T04:56:16.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:16.698+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.699926+0000) 2019-10-13T04:56:16.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:16.695+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.696205+0000) 2019-10-13T04:56:16.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:16.699+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.700638+0000) 2019-10-13T04:56:16.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:16.699+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.700767+0000) 2019-10-13T04:56:16.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:16.699+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.700938+0000) 2019-10-13T04:56:16.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:16.697+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.698045+0000) 2019-10-13T04:56:16.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:16.699+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.701298+0000) 2019-10-13T04:56:16.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:16.700+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:16.701540+0000) 2019-10-13T04:56:16.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:16.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66527 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:17.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:17.389+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:17.390300+0000) 2019-10-13T04:56:18.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:18.183+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:18.183842+0000) 2019-10-13T04:56:19.840 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:19.832+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:19.833232+0000) 2019-10-13T04:56:21.707 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:21.702+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.704390+0000) 2019-10-13T04:56:21.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:21.703+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.704819+0000) 2019-10-13T04:56:21.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:21.703+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.704971+0000) 2019-10-13T04:56:21.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:21.703+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.705072+0000) 2019-10-13T04:56:21.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:21.703+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.705271+0000) 2019-10-13T04:56:21.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:21.700+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.701229+0000) 2019-10-13T04:56:21.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:21.703+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.704695+0000) 2019-10-13T04:56:21.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:21.704+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66557 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:21.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:21.702+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.703303+0000) 2019-10-13T04:56:21.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:21.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.773572+0000) 2019-10-13T04:56:21.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:21.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.773600+0000) 2019-10-13T04:56:21.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:21.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:21.773582+0000) 2019-10-13T04:56:22.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:22.645+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:22.646040+0000) 2019-10-13T04:56:22.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:22.646+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:22.646935+0000) 2019-10-13T04:56:26.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:26.706+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.708146+0000) 2019-10-13T04:56:26.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:26.706+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.708383+0000) 2019-10-13T04:56:26.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:26.707+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.708745+0000) 2019-10-13T04:56:26.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:26.707+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.709183+0000) 2019-10-13T04:56:26.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:26.707+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.709046+0000) 2019-10-13T04:56:26.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:26.708+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.709599+0000) 2019-10-13T04:56:26.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:26.704+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.705521+0000) 2019-10-13T04:56:26.713 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:26.705+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:26.706473+0000) 2019-10-13T04:56:26.713 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:26.709+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:27.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:27.390+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:27.390465+0000) 2019-10-13T04:56:28.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:28.183+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:28.184051+0000) 2019-10-13T04:56:29.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:29.835+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:29.835680+0000) 2019-10-13T04:56:30.596 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:56:30.596 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:30.600 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:56:30.600 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:31.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:31.712+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.712703+0000) 2019-10-13T04:56:31.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:31.712+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.713182+0000) 2019-10-13T04:56:31.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:31.712+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.713316+0000) 2019-10-13T04:56:31.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:31.713+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.713450+0000) 2019-10-13T04:56:31.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:31.706+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.708312+0000) 2019-10-13T04:56:31.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:31.713+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.713813+0000) 2019-10-13T04:56:31.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:31.713+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.714099+0000) 2019-10-13T04:56:31.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:31.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:31.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:31.709+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.711567+0000) 2019-10-13T04:56:31.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:31.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.773731+0000) 2019-10-13T04:56:31.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:31.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.773748+0000) 2019-10-13T04:56:31.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:31.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:31.773741+0000) 2019-10-13T04:56:32.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:32.645+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:32.646173+0000) 2019-10-13T04:56:32.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:32.646+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:32.647119+0000) 2019-10-13T04:56:36.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:36.716+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.717321+0000) 2019-10-13T04:56:36.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:36.717+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.717491+0000) 2019-10-13T04:56:36.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:36.717+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.717747+0000) 2019-10-13T04:56:36.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:36.712+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.713889+0000) 2019-10-13T04:56:36.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:36.717+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.717938+0000) 2019-10-13T04:56:36.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:36.717+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.718381+0000) 2019-10-13T04:56:36.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:36.718+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.719031+0000) 2019-10-13T04:56:36.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:36.713+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:36.715052+0000) 2019-10-13T04:56:36.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:36.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:37.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:37.390+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:37.390656+0000) 2019-10-13T04:56:38.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:38.184+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:38.184244+0000) 2019-10-13T04:56:39.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:39.837+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:39.838157+0000) 2019-10-13T04:56:41.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:41.715+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.717226+0000) 2019-10-13T04:56:41.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:41.721+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.721907+0000) 2019-10-13T04:56:41.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:41.721+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.722196+0000) 2019-10-13T04:56:41.726 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:41.722+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.722408+0000) 2019-10-13T04:56:41.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:41.722+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.722577+0000) 2019-10-13T04:56:41.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:41.722+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.722912+0000) 2019-10-13T04:56:41.727 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:41.722+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.723322+0000) 2019-10-13T04:56:41.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:41.719+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.720818+0000) 2019-10-13T04:56:41.728 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:41.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:41.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:41.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.773916+0000) 2019-10-13T04:56:41.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:41.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.773940+0000) 2019-10-13T04:56:41.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:41.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:41.773931+0000) 2019-10-13T04:56:42.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:42.645+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:42.646343+0000) 2019-10-13T04:56:42.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:42.646+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:42.647262+0000) 2019-10-13T04:56:46.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:46.724+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.726121+0000) 2019-10-13T04:56:46.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:46.724+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.726286+0000) 2019-10-13T04:56:46.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:46.725+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.726717+0000) 2019-10-13T04:56:46.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:46.725+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.726775+0000) 2019-10-13T04:56:46.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:46.725+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.726949+0000) 2019-10-13T04:56:46.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:46.722+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.722888+0000) 2019-10-13T04:56:46.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:46.723+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.723766+0000) 2019-10-13T04:56:46.731 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:46.725+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:46.727386+0000) 2019-10-13T04:56:46.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:46.726+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:47.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:47.389+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:47.390829+0000) 2019-10-13T04:56:48.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:48.183+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:48.184435+0000) 2019-10-13T04:56:49.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:49.839+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:49.840501+0000) 2019-10-13T04:56:51.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:51.725+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.725839+0000) 2019-10-13T04:56:51.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:51.728+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.730330+0000) 2019-10-13T04:56:51.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:51.729+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.730916+0000) 2019-10-13T04:56:51.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:51.729+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.731123+0000) 2019-10-13T04:56:51.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:51.729+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.731144+0000) 2019-10-13T04:56:51.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:51.729+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.731276+0000) 2019-10-13T04:56:51.735 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:51.730+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.731436+0000) 2019-10-13T04:56:51.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:51.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:51.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:51.728+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.729719+0000) 2019-10-13T04:56:51.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:51.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.774026+0000) 2019-10-13T04:56:51.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:51.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.774087+0000) 2019-10-13T04:56:51.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:51.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:51.774064+0000) 2019-10-13T04:56:52.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:52.645+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:52.646515+0000) 2019-10-13T04:56:52.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:52.646+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:52.647417+0000) 2019-10-13T04:56:56.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:56:56.733+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.734991+0000) 2019-10-13T04:56:56.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:56:56.733+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.735370+0000) 2019-10-13T04:56:56.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:56:56.730+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.731011+0000) 2019-10-13T04:56:56.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:56.734+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.735615+0000) 2019-10-13T04:56:56.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:56:56.734+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.735765+0000) 2019-10-13T04:56:56.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:56:56.734+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.736126+0000) 2019-10-13T04:56:56.739 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:56.734+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.736329+0000) 2019-10-13T04:56:56.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:56.732+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:56.733091+0000) 2019-10-13T04:56:56.740 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:56:56.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:56:57.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:56:57.390+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:57.390976+0000) 2019-10-13T04:56:58.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:56:58.183+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:58.184660+0000) 2019-10-13T04:56:59.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:56:59.842+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:56:59.842928+0000) 2019-10-13T04:57:00.817 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:57:00.817 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:00.821 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:57:00.822 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:01.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:01.737+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.739244+0000) 2019-10-13T04:57:01.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:01.738+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.739439+0000) 2019-10-13T04:57:01.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:01.738+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.739636+0000) 2019-10-13T04:57:01.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:01.738+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.739843+0000) 2019-10-13T04:57:01.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:01.735+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.735944+0000) 2019-10-13T04:57:01.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:01.738+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.740010+0000) 2019-10-13T04:57:01.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:01.739+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.740544+0000) 2019-10-13T04:57:01.744 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:01.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:01.745 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:01.737+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.738687+0000) 2019-10-13T04:57:01.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:01.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.774199+0000) 2019-10-13T04:57:01.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:01.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.774206+0000) 2019-10-13T04:57:01.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:01.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:01.774228+0000) 2019-10-13T04:57:02.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:02.646+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:02.646668+0000) 2019-10-13T04:57:02.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:02.647+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:02.647573+0000) 2019-10-13T04:57:06.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:06.742+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.743530+0000) 2019-10-13T04:57:06.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:06.742+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.743675+0000) 2019-10-13T04:57:06.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:06.742+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.744144+0000) 2019-10-13T04:57:06.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:06.742+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.744306+0000) 2019-10-13T04:57:06.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:06.739+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.740215+0000) 2019-10-13T04:57:06.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:06.740+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.741751+0000) 2019-10-13T04:57:06.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:06.743+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.744903+0000) 2019-10-13T04:57:06.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:06.743+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:06.745342+0000) 2019-10-13T04:57:06.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:06.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:07.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:07.390+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:07.391189+0000) 2019-10-13T04:57:08.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:08.183+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:08.184845+0000) 2019-10-13T04:57:09.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:09.845+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:09.845484+0000) 2019-10-13T04:57:11.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:11.746+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.748014+0000) 2019-10-13T04:57:11.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:11.743+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.744021+0000) 2019-10-13T04:57:11.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:11.747+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.748663+0000) 2019-10-13T04:57:11.752 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:11.747+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.748549+0000) 2019-10-13T04:57:11.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:11.747+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.749003+0000) 2019-10-13T04:57:11.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:11.747+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.749175+0000) 2019-10-13T04:57:11.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:11.748+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.749589+0000) 2019-10-13T04:57:11.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:11.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:11.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:11.746+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.747241+0000) 2019-10-13T04:57:11.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:11.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.774322+0000) 2019-10-13T04:57:11.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:11.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.774358+0000) 2019-10-13T04:57:11.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:11.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:11.774392+0000) 2019-10-13T04:57:12.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:12.646+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:12.646784+0000) 2019-10-13T04:57:12.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:12.647+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:12.647747+0000) 2019-10-13T04:57:16.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:16.752+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.752719+0000) 2019-10-13T04:57:16.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:16.752+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.753099+0000) 2019-10-13T04:57:16.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:16.752+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.753218+0000) 2019-10-13T04:57:16.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:16.753+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.753390+0000) 2019-10-13T04:57:16.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:16.753+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.753637+0000) 2019-10-13T04:57:16.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:16.748+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.749298+0000) 2019-10-13T04:57:16.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:16.753+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.754026+0000) 2019-10-13T04:57:16.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:16.750+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:16.750968+0000) 2019-10-13T04:57:16.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:16.755+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66881 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:17.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:17.391+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:17.391330+0000) 2019-10-13T04:57:18.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:18.184+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:18.185019+0000) 2019-10-13T04:57:19.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:19.847+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:19.847860+0000) 2019-10-13T04:57:21.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:21.756+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.757127+0000) 2019-10-13T04:57:21.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:21.756+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.757359+0000) 2019-10-13T04:57:21.761 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:21.757+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.757528+0000) 2019-10-13T04:57:21.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:21.751+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.753410+0000) 2019-10-13T04:57:21.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:21.757+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.757942+0000) 2019-10-13T04:57:21.762 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:21.757+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.758242+0000) 2019-10-13T04:57:21.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:21.758+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.758621+0000) 2019-10-13T04:57:21.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:21.759+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:21.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:21.755+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.756831+0000) 2019-10-13T04:57:21.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:21.774+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.774522+0000) 2019-10-13T04:57:21.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:21.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.774519+0000) 2019-10-13T04:57:21.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:21.774+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:21.774565+0000) 2019-10-13T04:57:22.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:22.646+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:22.646982+0000) 2019-10-13T04:57:22.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:22.646+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:22.647909+0000) 2019-10-13T04:57:26.765 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:26.761+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.762124+0000) 2019-10-13T04:57:26.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:26.761+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.762211+0000) 2019-10-13T04:57:26.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:26.762+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.762467+0000) 2019-10-13T04:57:26.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:26.756+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.758385+0000) 2019-10-13T04:57:26.767 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:26.762+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.762857+0000) 2019-10-13T04:57:26.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:26.762+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.762926+0000) 2019-10-13T04:57:26.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:26.763+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.763533+0000) 2019-10-13T04:57:26.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:26.758+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:26.759839+0000) 2019-10-13T04:57:26.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:26.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:27.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:27.391+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:27.391500+0000) 2019-10-13T04:57:28.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:28.184+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:28.185206+0000) 2019-10-13T04:57:29.857 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:29.849+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:29.850223+0000) 2019-10-13T04:57:30.999 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:57:30.999 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:31.003 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:57:31.003 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:31.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:31.760+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.762095+0000) 2019-10-13T04:57:31.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:31.765+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.766873+0000) 2019-10-13T04:57:31.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:31.765+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.766993+0000) 2019-10-13T04:57:31.770 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:31.765+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.767236+0000) 2019-10-13T04:57:31.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:31.765+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.767118+0000) 2019-10-13T04:57:31.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:31.766+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.767698+0000) 2019-10-13T04:57:31.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:31.766+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.768007+0000) 2019-10-13T04:57:31.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:31.762+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.764634+0000) 2019-10-13T04:57:31.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:31.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:31.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:31.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.774630+0000) 2019-10-13T04:57:31.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:31.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.774627+0000) 2019-10-13T04:57:31.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:31.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:31.774676+0000) 2019-10-13T04:57:32.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:32.646+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:32.647159+0000) 2019-10-13T04:57:32.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:32.647+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:32.648073+0000) 2019-10-13T04:57:36.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:36.768+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.770361+0000) 2019-10-13T04:57:36.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:36.769+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.770685+0000) 2019-10-13T04:57:36.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:36.769+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.770906+0000) 2019-10-13T04:57:36.774 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:36.769+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.771082+0000) 2019-10-13T04:57:36.774 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:36.769+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.771164+0000) 2019-10-13T04:57:36.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:36.770+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.771468+0000) 2019-10-13T04:57:36.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:36.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.767454+0000) 2019-10-13T04:57:36.775 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:36.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 66995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:36.776 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:36.768+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:36.769682+0000) 2019-10-13T04:57:37.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:37.390+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:37.391667+0000) 2019-10-13T04:57:38.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:38.183+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:38.185401+0000) 2019-10-13T04:57:39.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:39.851+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:39.852662+0000) 2019-10-13T04:57:41.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:41.769+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.770456+0000) 2019-10-13T04:57:41.781 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:41.771+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.772675+0000) 2019-10-13T04:57:41.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:41.772+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.774351+0000) 2019-10-13T04:57:41.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:41.773+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.774479+0000) 2019-10-13T04:57:41.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:41.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.774743+0000) 2019-10-13T04:57:41.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:41.773+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.775208+0000) 2019-10-13T04:57:41.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:41.774+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.775974+0000) 2019-10-13T04:57:41.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:41.774+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.776223+0000) 2019-10-13T04:57:41.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:41.775+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:41.776592+0000) 2019-10-13T04:57:41.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:41.775+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:42.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:42.646+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:42.647322+0000) 2019-10-13T04:57:42.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:42.647+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:42.648264+0000) 2019-10-13T04:57:46.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:46.778+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.779500+0000) 2019-10-13T04:57:46.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:46.778+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.779642+0000) 2019-10-13T04:57:46.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:46.778+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.779889+0000) 2019-10-13T04:57:46.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:46.778+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.780161+0000) 2019-10-13T04:57:46.784 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:46.778+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.780300+0000) 2019-10-13T04:57:46.785 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:46.779+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.780717+0000) 2019-10-13T04:57:46.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:46.775+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.775769+0000) 2019-10-13T04:57:46.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:46.776+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:46.777049+0000) 2019-10-13T04:57:46.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:46.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:47.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:47.391+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:47.391842+0000) 2019-10-13T04:57:48.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:48.184+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:48.185581+0000) 2019-10-13T04:57:48.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:48.524+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:48.524976+0000) 2019-10-13T04:57:48.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:48.524+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:48.525211+0000) 2019-10-13T04:57:49.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:49.854+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:49.855031+0000) 2019-10-13T04:57:51.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:51.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.774895+0000) 2019-10-13T04:57:51.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:51.782+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.783650+0000) 2019-10-13T04:57:51.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:51.782+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.783815+0000) 2019-10-13T04:57:51.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:51.782+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.783953+0000) 2019-10-13T04:57:51.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:51.783+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.784566+0000) 2019-10-13T04:57:51.788 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:51.783+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.784811+0000) 2019-10-13T04:57:51.788 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:51.783+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.784947+0000) 2019-10-13T04:57:51.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:51.779+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.779899+0000) 2019-10-13T04:57:51.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:51.781+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:51.782068+0000) 2019-10-13T04:57:51.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:51.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:52.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:52.646+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:52.647489+0000) 2019-10-13T04:57:52.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:52.648+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:52.648422+0000) 2019-10-13T04:57:56.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:57:56.783+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.784038+0000) 2019-10-13T04:57:56.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:57:56.786+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.787929+0000) 2019-10-13T04:57:56.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:56.786+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.788147+0000) 2019-10-13T04:57:56.793 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:56.787+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.788422+0000) 2019-10-13T04:57:56.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:56.787+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.788593+0000) 2019-10-13T04:57:56.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:56.787+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.789300+0000) 2019-10-13T04:57:56.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:57:56.787+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.788961+0000) 2019-10-13T04:57:56.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:57:56.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:57:56.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:56.785+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:56.786524+0000) 2019-10-13T04:57:57.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:57:57.391+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:57.392019+0000) 2019-10-13T04:57:58.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:57:58.185+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:58.185767+0000) 2019-10-13T04:57:58.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:57:58.524+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:58.525137+0000) 2019-10-13T04:57:58.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:57:58.524+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:58.525313+0000) 2019-10-13T04:57:59.864 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:57:59.857+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:57:59.857581+0000) 2019-10-13T04:58:01.168 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:58:01.168 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:01.172 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:58:01.173 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:01.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:01.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.775059+0000) 2019-10-13T04:58:01.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:01.791+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.792211+0000) 2019-10-13T04:58:01.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:01.792+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.792568+0000) 2019-10-13T04:58:01.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:01.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.788636+0000) 2019-10-13T04:58:01.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:01.792+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.792758+0000) 2019-10-13T04:58:01.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:01.792+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.793175+0000) 2019-10-13T04:58:01.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:01.792+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.793323+0000) 2019-10-13T04:58:01.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:01.790+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.790875+0000) 2019-10-13T04:58:01.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:01.793+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:01.793539+0000) 2019-10-13T04:58:01.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:01.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:02.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:02.647+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:02.647623+0000) 2019-10-13T04:58:02.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:02.647+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:02.648581+0000) 2019-10-13T04:58:06.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:06.796+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.796696+0000) 2019-10-13T04:58:06.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:06.791+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.792655+0000) 2019-10-13T04:58:06.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:06.796+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.797341+0000) 2019-10-13T04:58:06.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:06.797+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.797444+0000) 2019-10-13T04:58:06.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:06.797+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.797856+0000) 2019-10-13T04:58:06.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:06.797+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.798067+0000) 2019-10-13T04:58:06.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:06.797+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.798189+0000) 2019-10-13T04:58:06.802 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:06.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:06.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:06.794+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:06.795613+0000) 2019-10-13T04:58:07.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:07.391+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:07.392183+0000) 2019-10-13T04:58:08.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:08.185+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:08.185998+0000) 2019-10-13T04:58:08.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:08.525+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:08.525303+0000) 2019-10-13T04:58:08.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:08.525+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:08.525413+0000) 2019-10-13T04:58:09.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:09.860+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:09.860078+0000) 2019-10-13T04:58:11.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:11.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.775242+0000) 2019-10-13T04:58:11.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:11.800+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.801240+0000) 2019-10-13T04:58:11.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:11.801+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.801486+0000) 2019-10-13T04:58:11.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:11.796+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.797978+0000) 2019-10-13T04:58:11.805 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:11.801+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.801750+0000) 2019-10-13T04:58:11.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:11.801+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.801961+0000) 2019-10-13T04:58:11.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:11.801+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.802250+0000) 2019-10-13T04:58:11.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:11.802+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.802565+0000) 2019-10-13T04:58:11.806 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:11.803+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:11.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:11.798+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:11.800239+0000) 2019-10-13T04:58:12.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:12.647+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:12.647792+0000) 2019-10-13T04:58:12.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:12.647+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:12.648755+0000) 2019-10-13T04:58:16.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:16.804+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.805932+0000) 2019-10-13T04:58:16.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:16.804+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.806155+0000) 2019-10-13T04:58:16.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:16.804+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.806320+0000) 2019-10-13T04:58:16.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:16.800+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.802243+0000) 2019-10-13T04:58:16.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:16.805+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.806886+0000) 2019-10-13T04:58:16.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:16.805+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.807027+0000) 2019-10-13T04:58:16.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:16.805+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.807179+0000) 2019-10-13T04:58:16.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:16.802+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:16.804554+0000) 2019-10-13T04:58:16.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:16.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:17.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:17.391+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:17.392354+0000) 2019-10-13T04:58:18.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:18.184+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:18.186208+0000) 2019-10-13T04:58:18.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:18.524+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:18.525500+0000) 2019-10-13T04:58:18.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:18.524+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:18.525565+0000) 2019-10-13T04:58:19.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:19.861+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:19.862593+0000) 2019-10-13T04:58:21.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:21.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.775438+0000) 2019-10-13T04:58:21.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:21.808+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.809966+0000) 2019-10-13T04:58:21.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:21.808+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.810089+0000) 2019-10-13T04:58:21.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:21.804+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.806660+0000) 2019-10-13T04:58:21.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:21.809+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.810596+0000) 2019-10-13T04:58:21.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:21.809+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.810748+0000) 2019-10-13T04:58:21.814 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:21.809+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.811062+0000) 2019-10-13T04:58:21.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:21.809+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.811240+0000) 2019-10-13T04:58:21.815 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:21.811+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:21.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:21.807+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:21.809390+0000) 2019-10-13T04:58:22.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:22.646+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:22.647977+0000) 2019-10-13T04:58:22.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:22.648+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:22.648918+0000) 2019-10-13T04:58:26.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:26.812+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.813954+0000) 2019-10-13T04:58:26.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:26.813+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.814706+0000) 2019-10-13T04:58:26.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:26.810+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.810834+0000) 2019-10-13T04:58:26.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:26.813+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.814990+0000) 2019-10-13T04:58:26.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:26.813+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.815301+0000) 2019-10-13T04:58:26.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:26.813+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.815325+0000) 2019-10-13T04:58:26.819 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:26.814+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.815427+0000) 2019-10-13T04:58:26.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:26.811+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:26.812692+0000) 2019-10-13T04:58:26.820 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:26.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67279 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:27.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:27.391+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:27.392420+0000) 2019-10-13T04:58:28.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:28.185+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:28.186427+0000) 2019-10-13T04:58:28.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:28.525+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:28.525751+0000) 2019-10-13T04:58:28.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:28.525+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:28.525760+0000) 2019-10-13T04:58:29.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:29.864+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:29.864981+0000) 2019-10-13T04:58:31.327 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:58:31.327 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:31.331 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:58:31.331 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:31.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:31.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.775616+0000) 2019-10-13T04:58:31.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:31.813+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.814417+0000) 2019-10-13T04:58:31.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:31.817+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.818876+0000) 2019-10-13T04:58:31.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:31.817+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.819221+0000) 2019-10-13T04:58:31.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:31.818+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.819459+0000) 2019-10-13T04:58:31.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:31.818+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.819664+0000) 2019-10-13T04:58:31.823 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:31.818+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.819889+0000) 2019-10-13T04:58:31.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:31.818+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.820107+0000) 2019-10-13T04:58:31.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:31.816+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:31.816963+0000) 2019-10-13T04:58:31.824 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:31.819+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:32.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:32.647+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:32.648154+0000) 2019-10-13T04:58:32.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:32.648+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:32.649076+0000) 2019-10-13T04:58:36.825 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:36.821+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.822968+0000) 2019-10-13T04:58:36.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:36.821+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.823336+0000) 2019-10-13T04:58:36.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:36.818+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.818893+0000) 2019-10-13T04:58:36.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:36.822+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.823742+0000) 2019-10-13T04:58:36.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:36.822+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.823627+0000) 2019-10-13T04:58:36.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:36.822+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.823849+0000) 2019-10-13T04:58:36.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:36.823+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.824607+0000) 2019-10-13T04:58:36.828 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:36.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:36.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:36.822+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:36.822819+0000) 2019-10-13T04:58:37.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:37.392+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:37.392575+0000) 2019-10-13T04:58:38.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:38.185+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:38.186608+0000) 2019-10-13T04:58:38.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:38.525+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:38.525947+0000) 2019-10-13T04:58:38.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:38.525+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:38.525949+0000) 2019-10-13T04:58:39.874 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:39.866+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:39.867343+0000) 2019-10-13T04:58:41.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:41.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.775811+0000) 2019-10-13T04:58:41.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:41.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.827363+0000) 2019-10-13T04:58:41.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:41.826+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.827764+0000) 2019-10-13T04:58:41.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:41.822+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.823341+0000) 2019-10-13T04:58:41.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:41.826+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.827862+0000) 2019-10-13T04:58:41.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:41.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.828053+0000) 2019-10-13T04:58:41.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:41.827+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.828520+0000) 2019-10-13T04:58:41.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:41.827+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.828566+0000) 2019-10-13T04:58:41.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:41.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:41.833 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:41.825+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:41.826726+0000) 2019-10-13T04:58:42.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:42.647+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:42.648315+0000) 2019-10-13T04:58:42.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:42.648+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:42.649229+0000) 2019-10-13T04:58:46.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:46.831+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.831845+0000) 2019-10-13T04:58:46.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:46.831+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.832028+0000) 2019-10-13T04:58:46.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:46.827+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.827882+0000) 2019-10-13T04:58:46.836 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:46.832+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.832541+0000) 2019-10-13T04:58:46.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:46.832+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.832647+0000) 2019-10-13T04:58:46.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:46.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.832789+0000) 2019-10-13T04:58:46.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:46.832+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.833042+0000) 2019-10-13T04:58:46.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:46.829+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:46.830608+0000) 2019-10-13T04:58:46.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:46.833+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:47.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:47.392+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:47.392770+0000) 2019-10-13T04:58:48.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:48.186+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:48.186762+0000) 2019-10-13T04:58:48.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:48.525+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:48.526127+0000) 2019-10-13T04:58:48.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:48.525+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:48.526130+0000) 2019-10-13T04:58:49.876 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:49.869+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:49.869830+0000) 2019-10-13T04:58:51.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:51.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.776011+0000) 2019-10-13T04:58:51.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:51.835+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.835825+0000) 2019-10-13T04:58:51.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:51.831+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.831766+0000) 2019-10-13T04:58:51.839 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:51.835+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.836241+0000) 2019-10-13T04:58:51.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:51.836+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.836508+0000) 2019-10-13T04:58:51.839 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:51.836+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.836671+0000) 2019-10-13T04:58:51.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:51.836+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.837043+0000) 2019-10-13T04:58:51.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:51.836+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.837306+0000) 2019-10-13T04:58:51.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:51.833+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:51.834161+0000) 2019-10-13T04:58:51.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:51.838+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:52.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:52.648+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:52.648506+0000) 2019-10-13T04:58:52.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:52.649+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:52.649384+0000) 2019-10-13T04:58:56.842 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:56.839+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.840210+0000) 2019-10-13T04:58:56.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:58:56.835+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.836572+0000) 2019-10-13T04:58:56.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:58:56.840+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.840689+0000) 2019-10-13T04:58:56.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:56.840+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.841173+0000) 2019-10-13T04:58:56.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:56.840+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.841389+0000) 2019-10-13T04:58:56.844 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:56.841+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.841433+0000) 2019-10-13T04:58:56.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:56.837+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.837940+0000) 2019-10-13T04:58:56.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:58:56.841+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:56.841687+0000) 2019-10-13T04:58:56.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:58:56.843+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67451 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:58:57.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:58:57.392+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:57.392954+0000) 2019-10-13T04:58:58.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:58:58.186+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:58.186936+0000) 2019-10-13T04:58:58.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:58:58.526+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:58.526330+0000) 2019-10-13T04:58:58.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:58:58.526+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:58.526336+0000) 2019-10-13T04:58:59.879 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:58:59.872+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:58:59.872189+0000) 2019-10-13T04:59:01.549 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:59:01.549 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:01.554 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:59:01.554 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:01.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:01.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.776219+0000) 2019-10-13T04:59:01.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:01.843+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.845342+0000) 2019-10-13T04:59:01.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:01.844+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.845599+0000) 2019-10-13T04:59:01.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:01.844+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.845704+0000) 2019-10-13T04:59:01.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:01.840+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.841741+0000) 2019-10-13T04:59:01.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:01.844+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.845921+0000) 2019-10-13T04:59:01.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:01.845+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.846451+0000) 2019-10-13T04:59:01.850 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:01.845+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.846656+0000) 2019-10-13T04:59:01.850 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:01.841+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:01.843147+0000) 2019-10-13T04:59:01.850 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:01.846+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:02.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:02.647+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:02.648681+0000) 2019-10-13T04:59:02.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:02.648+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:02.649586+0000) 2019-10-13T04:59:06.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:06.843+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.845478+0000) 2019-10-13T04:59:06.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:06.848+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.850198+0000) 2019-10-13T04:59:06.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:06.849+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.850687+0000) 2019-10-13T04:59:06.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:06.849+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.850521+0000) 2019-10-13T04:59:06.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:06.849+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.850460+0000) 2019-10-13T04:59:06.854 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:06.850+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.851493+0000) 2019-10-13T04:59:06.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:06.850+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.851672+0000) 2019-10-13T04:59:06.855 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:06.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:06.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:06.847+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:06.849184+0000) 2019-10-13T04:59:07.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:07.392+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:07.393118+0000) 2019-10-13T04:59:08.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:08.185+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:08.187147+0000) 2019-10-13T04:59:08.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:08.525+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:08.526489+0000) 2019-10-13T04:59:08.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:08.525+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:08.526542+0000) 2019-10-13T04:59:09.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:09.873+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:09.874533+0000) 2019-10-13T04:59:11.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:11.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.776435+0000) 2019-10-13T04:59:11.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:11.848+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.850100+0000) 2019-10-13T04:59:11.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:11.853+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.854503+0000) 2019-10-13T04:59:11.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:11.853+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.854943+0000) 2019-10-13T04:59:11.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:11.853+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.855144+0000) 2019-10-13T04:59:11.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:11.854+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.855567+0000) 2019-10-13T04:59:11.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:11.854+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.855722+0000) 2019-10-13T04:59:11.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:11.850+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.852465+0000) 2019-10-13T04:59:11.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:11.854+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:11.855926+0000) 2019-10-13T04:59:11.860 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:11.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:12.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:12.648+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:12.648860+0000) 2019-10-13T04:59:12.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:12.648+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:12.649767+0000) 2019-10-13T04:59:16.861 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:16.857+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.858513+0000) 2019-10-13T04:59:16.861 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:16.857+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.858606+0000) 2019-10-13T04:59:16.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:16.854+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.855564+0000) 2019-10-13T04:59:16.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:16.858+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.859965+0000) 2019-10-13T04:59:16.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:16.858+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.860062+0000) 2019-10-13T04:59:16.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:16.858+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.860370+0000) 2019-10-13T04:59:16.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:16.859+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.860612+0000) 2019-10-13T04:59:16.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:16.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67565 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:16.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:16.856+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:16.857426+0000) 2019-10-13T04:59:17.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:17.392+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:17.393293+0000) 2019-10-13T04:59:18.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:18.185+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:18.187335+0000) 2019-10-13T04:59:18.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:18.526+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:18.526687+0000) 2019-10-13T04:59:18.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:18.526+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:18.526725+0000) 2019-10-13T04:59:19.884 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:19.876+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:19.877024+0000) 2019-10-13T04:59:21.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:21.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.776625+0000) 2019-10-13T04:59:21.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:21.858+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.859099+0000) 2019-10-13T04:59:21.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:21.862+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.863844+0000) 2019-10-13T04:59:21.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:21.862+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.863979+0000) 2019-10-13T04:59:21.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:21.862+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.864229+0000) 2019-10-13T04:59:21.873 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:21.863+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.864427+0000) 2019-10-13T04:59:21.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:21.863+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.864633+0000) 2019-10-13T04:59:21.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:21.863+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.864800+0000) 2019-10-13T04:59:21.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:21.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:21.874 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:21.860+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:21.861513+0000) 2019-10-13T04:59:22.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:22.648+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:22.649017+0000) 2019-10-13T04:59:22.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:22.649+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:22.649909+0000) 2019-10-13T04:59:26.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:26.866+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.867827+0000) 2019-10-13T04:59:26.871 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:26.866+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.867616+0000) 2019-10-13T04:59:26.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:26.866+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.867992+0000) 2019-10-13T04:59:26.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:26.867+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.868408+0000) 2019-10-13T04:59:26.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:26.867+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.868512+0000) 2019-10-13T04:59:26.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:26.863+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.864467+0000) 2019-10-13T04:59:26.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:26.867+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.868909+0000) 2019-10-13T04:59:26.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:26.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:26.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:26.866+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:26.866942+0000) 2019-10-13T04:59:27.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:27.393+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:27.393463+0000) 2019-10-13T04:59:28.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:28.187+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:28.187537+0000) 2019-10-13T04:59:28.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:28.526+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:28.526934+0000) 2019-10-13T04:59:28.530 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:28.526+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:28.526884+0000) 2019-10-13T04:59:29.886 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:29.878+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:29.879616+0000) 2019-10-13T04:59:31.747 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T04:59:31.747 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:31.751 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T04:59:31.751 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:31.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:31.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.776754+0000) 2019-10-13T04:59:31.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:31.871+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.871933+0000) 2019-10-13T04:59:31.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:31.871+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.872033+0000) 2019-10-13T04:59:31.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:31.871+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.872350+0000) 2019-10-13T04:59:31.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:31.871+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.872189+0000) 2019-10-13T04:59:31.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:31.872+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.872463+0000) 2019-10-13T04:59:31.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:31.867+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.868372+0000) 2019-10-13T04:59:31.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:31.868+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.869437+0000) 2019-10-13T04:59:31.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:31.872+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:31.872788+0000) 2019-10-13T04:59:31.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:31.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:32.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:32.649+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:32.650037+0000) 2019-10-13T04:59:32.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:32.648+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:32.649162+0000) 2019-10-13T04:59:36.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:36.875+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.875727+0000) 2019-10-13T04:59:36.879 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:36.875+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.875979+0000) 2019-10-13T04:59:36.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:36.875+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.876114+0000) 2019-10-13T04:59:36.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:36.871+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.872005+0000) 2019-10-13T04:59:36.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:36.876+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.876502+0000) 2019-10-13T04:59:36.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:36.876+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.876904+0000) 2019-10-13T04:59:36.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:36.876+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.877081+0000) 2019-10-13T04:59:36.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:36.872+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:36.873631+0000) 2019-10-13T04:59:36.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:36.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67679 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:37.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:37.393+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:37.393639+0000) 2019-10-13T04:59:38.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:38.187+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:38.187733+0000) 2019-10-13T04:59:38.530 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:38.526+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:38.527096+0000) 2019-10-13T04:59:38.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:38.526+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:38.527117+0000) 2019-10-13T04:59:39.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:39.881+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:39.882054+0000) 2019-10-13T04:59:41.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:41.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.776941+0000) 2019-10-13T04:59:41.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:41.874+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.875660+0000) 2019-10-13T04:59:41.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:41.879+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.880206+0000) 2019-10-13T04:59:41.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:41.880+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.880492+0000) 2019-10-13T04:59:41.883 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:41.880+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.880717+0000) 2019-10-13T04:59:41.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:41.880+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.880882+0000) 2019-10-13T04:59:41.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:41.880+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.881145+0000) 2019-10-13T04:59:41.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:41.880+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.881284+0000) 2019-10-13T04:59:41.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:41.877+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:41.878122+0000) 2019-10-13T04:59:41.885 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:41.882+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67709 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:42.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:42.649+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:42.649353+0000) 2019-10-13T04:59:42.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:42.649+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:42.650194+0000) 2019-10-13T04:59:46.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:46.879+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.879845+0000) 2019-10-13T04:59:46.887 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:46.883+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.884429+0000) 2019-10-13T04:59:46.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:46.883+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.884858+0000) 2019-10-13T04:59:46.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:46.883+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.885025+0000) 2019-10-13T04:59:46.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:46.883+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.885196+0000) 2019-10-13T04:59:46.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:46.883+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.885369+0000) 2019-10-13T04:59:46.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:46.884+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.885605+0000) 2019-10-13T04:59:46.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:46.881+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:46.882512+0000) 2019-10-13T04:59:46.889 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:46.885+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:47.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:47.392+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:47.393816+0000) 2019-10-13T04:59:48.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:48.186+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:48.187919+0000) 2019-10-13T04:59:48.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:48.525+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:48.527321+0000) 2019-10-13T04:59:48.530 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:48.525+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:48.527315+0000) 2019-10-13T04:59:49.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:49.884+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:49.884546+0000) 2019-10-13T04:59:51.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:51.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.777104+0000) 2019-10-13T04:59:51.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:51.882+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.884364+0000) 2019-10-13T04:59:51.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:51.887+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.889383+0000) 2019-10-13T04:59:51.892 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:51.888+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.889597+0000) 2019-10-13T04:59:51.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:51.888+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.889843+0000) 2019-10-13T04:59:51.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:51.888+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.890058+0000) 2019-10-13T04:59:51.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:51.889+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.890437+0000) 2019-10-13T04:59:51.894 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:51.889+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.890679+0000) 2019-10-13T04:59:51.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:51.885+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:51.887449+0000) 2019-10-13T04:59:51.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:51.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67768 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:52.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:52.648+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:52.649502+0000) 2019-10-13T04:59:52.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:52.649+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:52.650335+0000) 2019-10-13T04:59:56.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T04:59:56.888+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.889925+0000) 2019-10-13T04:59:56.898 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:56.892+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.894056+0000) 2019-10-13T04:59:56.899 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:56.893+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.894517+0000) 2019-10-13T04:59:56.899 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:56.893+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.894852+0000) 2019-10-13T04:59:56.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T04:59:56.893+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.894873+0000) 2019-10-13T04:59:56.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:56.893+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.895044+0000) 2019-10-13T04:59:56.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T04:59:56.893+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.895291+0000) 2019-10-13T04:59:56.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:56.890+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:56.892387+0000) 2019-10-13T04:59:56.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T04:59:56.895+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T04:59:57.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T04:59:57.393+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:57.393966+0000) 2019-10-13T04:59:58.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T04:59:58.187+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:58.188100+0000) 2019-10-13T04:59:58.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T04:59:58.526+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:58.527504+0000) 2019-10-13T04:59:58.530 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T04:59:58.526+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:58.527484+0000) 2019-10-13T04:59:59.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T04:59:59.886+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T03:59:59.887037+0000) 2019-10-13T05:00:01.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:01.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.777309+0000) 2019-10-13T05:00:01.894 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:00:01.895 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:01.899 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:00:01.899 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:01.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:01.896+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.898325+0000) 2019-10-13T05:00:01.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:01.897+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.898431+0000) 2019-10-13T05:00:01.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:01.892+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.894290+0000) 2019-10-13T05:00:01.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:01.897+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.898651+0000) 2019-10-13T05:00:01.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:01.897+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.898786+0000) 2019-10-13T05:00:01.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:01.897+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.899086+0000) 2019-10-13T05:00:01.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:01.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:01.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:01.898+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.899543+0000) 2019-10-13T05:00:01.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:01.895+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:01.897173+0000) 2019-10-13T05:00:02.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:02.648+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:02.649673+0000) 2019-10-13T05:00:02.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:02.649+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:02.650495+0000) 2019-10-13T05:00:06.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:06.900+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.901785+0000) 2019-10-13T05:00:06.905 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:06.900+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.901972+0000) 2019-10-13T05:00:06.905 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:06.900+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.902332+0000) 2019-10-13T05:00:06.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:06.897+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.897843+0000) 2019-10-13T05:00:06.906 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:06.901+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.902490+0000) 2019-10-13T05:00:06.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:06.901+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.902554+0000) 2019-10-13T05:00:06.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:06.901+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.902701+0000) 2019-10-13T05:00:06.907 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:06.902+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67855 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:06.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:06.899+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:06.900100+0000) 2019-10-13T05:00:07.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:07.393+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:07.394103+0000) 2019-10-13T05:00:08.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:08.187+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:08.188248+0000) 2019-10-13T05:00:08.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:08.527+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:08.527679+0000) 2019-10-13T05:00:08.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:08.527+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:08.527705+0000) 2019-10-13T05:00:09.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:09.888+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:09.889493+0000) 2019-10-13T05:00:11.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:11.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.777504+0000) 2019-10-13T05:00:11.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:11.900+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.900942+0000) 2019-10-13T05:00:11.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:11.904+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.905453+0000) 2019-10-13T05:00:11.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:11.904+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.905714+0000) 2019-10-13T05:00:11.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:11.904+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.905900+0000) 2019-10-13T05:00:11.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:11.904+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.906251+0000) 2019-10-13T05:00:11.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:11.904+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.906361+0000) 2019-10-13T05:00:11.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:11.904+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.906251+0000) 2019-10-13T05:00:11.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:11.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:11.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:11.903+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:11.904557+0000) 2019-10-13T05:00:12.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:12.649+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:12.649867+0000) 2019-10-13T05:00:12.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:12.650+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:12.650638+0000) 2019-10-13T05:00:16.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:16.904+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.904723+0000) 2019-10-13T05:00:16.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:16.908+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.909207+0000) 2019-10-13T05:00:16.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:16.908+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.909355+0000) 2019-10-13T05:00:16.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:16.909+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.909428+0000) 2019-10-13T05:00:16.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:16.909+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.910033+0000) 2019-10-13T05:00:16.913 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:16.909+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.910251+0000) 2019-10-13T05:00:16.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:16.910+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.910409+0000) 2019-10-13T05:00:16.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:16.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:16.914 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:16.906+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:16.907521+0000) 2019-10-13T05:00:17.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:17.393+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:17.394248+0000) 2019-10-13T05:00:18.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:18.188+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:18.188440+0000) 2019-10-13T05:00:18.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:18.527+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:18.527884+0000) 2019-10-13T05:00:18.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:18.527+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:18.527846+0000) 2019-10-13T05:00:19.898 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:19.891+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:19.891835+0000) 2019-10-13T05:00:21.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:21.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.777641+0000) 2019-10-13T05:00:21.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:21.907+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.908695+0000) 2019-10-13T05:00:21.917 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:21.912+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.913193+0000) 2019-10-13T05:00:21.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:21.913+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.913457+0000) 2019-10-13T05:00:21.917 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:21.913+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.913565+0000) 2019-10-13T05:00:21.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:21.913+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.913798+0000) 2019-10-13T05:00:21.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:21.913+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.913992+0000) 2019-10-13T05:00:21.918 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:21.913+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.914140+0000) 2019-10-13T05:00:21.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:21.910+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:21.911535+0000) 2019-10-13T05:00:21.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:21.915+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:22.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:22.649+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:22.650049+0000) 2019-10-13T05:00:22.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:22.650+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:22.650807+0000) 2019-10-13T05:00:26.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:26.917+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.917747+0000) 2019-10-13T05:00:26.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:26.917+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.918098+0000) 2019-10-13T05:00:26.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:26.917+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.917995+0000) 2019-10-13T05:00:26.923 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:26.918+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.918443+0000) 2019-10-13T05:00:26.924 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:26.918+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.918964+0000) 2019-10-13T05:00:26.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:26.918+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.919264+0000) 2019-10-13T05:00:26.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:26.913+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.913988+0000) 2019-10-13T05:00:26.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:26.915+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:26.916448+0000) 2019-10-13T05:00:26.927 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:26.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 67975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:27.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:27.394+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:27.394444+0000) 2019-10-13T05:00:28.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:28.188+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:28.188611+0000) 2019-10-13T05:00:28.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:28.527+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:28.528022+0000) 2019-10-13T05:00:28.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:28.527+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:28.528007+0000) 2019-10-13T05:00:29.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:29.893+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:29.894104+0000) 2019-10-13T05:00:31.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:31.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.777822+0000) 2019-10-13T05:00:31.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:31.921+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.922651+0000) 2019-10-13T05:00:31.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:31.918+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.919307+0000) 2019-10-13T05:00:31.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:31.919+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.920573+0000) 2019-10-13T05:00:31.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:31.921+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.923131+0000) 2019-10-13T05:00:31.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:31.921+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.923233+0000) 2019-10-13T05:00:31.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:31.922+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.923404+0000) 2019-10-13T05:00:31.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:31.922+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.923786+0000) 2019-10-13T05:00:31.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:31.922+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:31.924014+0000) 2019-10-13T05:00:31.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:31.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68008 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:32.017 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:00:32.017 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:32.021 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:00:32.021 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:32.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:32.649+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:32.650224+0000) 2019-10-13T05:00:32.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:32.650+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:32.650975+0000) 2019-10-13T05:00:36.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:36.922+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.922882+0000) 2019-10-13T05:00:36.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:36.925+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.927345+0000) 2019-10-13T05:00:36.931 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:36.926+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.927764+0000) 2019-10-13T05:00:36.931 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:36.926+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.927890+0000) 2019-10-13T05:00:36.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:36.926+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.928040+0000) 2019-10-13T05:00:36.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:36.926+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.928158+0000) 2019-10-13T05:00:36.932 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:36.927+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.928821+0000) 2019-10-13T05:00:36.932 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:36.924+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:36.925339+0000) 2019-10-13T05:00:36.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:36.928+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:37.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:37.393+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:37.394633+0000) 2019-10-13T05:00:38.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:38.187+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:38.188789+0000) 2019-10-13T05:00:38.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:38.527+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:38.528181+0000) 2019-10-13T05:00:38.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:38.527+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:38.528173+0000) 2019-10-13T05:00:39.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:39.896+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:39.896439+0000) 2019-10-13T05:00:41.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:41.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.778030+0000) 2019-10-13T05:00:41.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:41.925+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.927395+0000) 2019-10-13T05:00:41.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:41.930+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.932140+0000) 2019-10-13T05:00:41.935 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:41.931+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.932455+0000) 2019-10-13T05:00:41.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:41.931+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.932497+0000) 2019-10-13T05:00:41.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:41.931+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.932649+0000) 2019-10-13T05:00:41.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:41.931+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.932826+0000) 2019-10-13T05:00:41.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:41.931+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.932942+0000) 2019-10-13T05:00:41.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:41.927+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:41.929706+0000) 2019-10-13T05:00:41.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:41.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:42.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:42.649+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:42.650391+0000) 2019-10-13T05:00:42.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:42.650+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:42.651146+0000) 2019-10-13T05:00:46.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:46.934+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.936375+0000) 2019-10-13T05:00:46.940 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:46.935+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.936562+0000) 2019-10-13T05:00:46.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:46.935+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.936936+0000) 2019-10-13T05:00:46.940 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:46.935+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.936866+0000) 2019-10-13T05:00:46.940 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:46.935+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.937058+0000) 2019-10-13T05:00:46.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:46.930+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.932452+0000) 2019-10-13T05:00:46.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:46.936+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.937606+0000) 2019-10-13T05:00:46.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:46.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:46.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:46.932+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:46.934653+0000) 2019-10-13T05:00:47.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:47.394+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:47.394826+0000) 2019-10-13T05:00:48.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:48.188+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:48.189015+0000) 2019-10-13T05:00:48.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:48.527+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:48.528419+0000) 2019-10-13T05:00:48.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:48.527+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:48.528409+0000) 2019-10-13T05:00:49.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:49.898+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:49.898940+0000) 2019-10-13T05:00:51.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:51.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.778212+0000) 2019-10-13T05:00:51.943 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:51.938+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.940188+0000) 2019-10-13T05:00:51.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:51.939+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.940557+0000) 2019-10-13T05:00:51.944 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:51.939+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.940717+0000) 2019-10-13T05:00:51.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:51.939+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.940970+0000) 2019-10-13T05:00:51.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:51.934+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.936115+0000) 2019-10-13T05:00:51.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:51.939+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.941270+0000) 2019-10-13T05:00:51.945 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:51.940+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.941489+0000) 2019-10-13T05:00:51.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:51.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68128 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:51.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:51.937+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:51.939558+0000) 2019-10-13T05:00:52.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:52.650+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:52.650587+0000) 2019-10-13T05:00:52.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:52.650+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:52.651306+0000) 2019-10-13T05:00:56.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:00:56.939+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.940258+0000) 2019-10-13T05:00:56.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:56.943+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.944836+0000) 2019-10-13T05:00:56.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:56.943+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.945005+0000) 2019-10-13T05:00:56.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:00:56.943+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.945030+0000) 2019-10-13T05:00:56.954 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:56.944+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.945437+0000) 2019-10-13T05:00:56.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:56.944+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.945618+0000) 2019-10-13T05:00:56.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:00:56.944+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.945819+0000) 2019-10-13T05:00:56.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:00:56.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:00:56.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:56.942+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:56.943008+0000) 2019-10-13T05:00:57.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:00:57.394+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:57.394992+0000) 2019-10-13T05:00:58.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:00:58.188+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:58.189219+0000) 2019-10-13T05:00:58.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:00:58.528+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:58.528630+0000) 2019-10-13T05:00:58.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:00:58.528+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:58.528610+0000) 2019-10-13T05:00:59.908 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:00:59.900+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:00:59.900970+0000) 2019-10-13T05:01:01.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:01.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.778435+0000) 2019-10-13T05:01:01.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:01.944+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.945024+0000) 2019-10-13T05:01:01.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:01.948+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.949120+0000) 2019-10-13T05:01:01.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:01.949+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.949622+0000) 2019-10-13T05:01:01.954 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:01.949+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.949815+0000) 2019-10-13T05:01:01.955 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:01.949+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.949942+0000) 2019-10-13T05:01:01.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:01.949+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.950150+0000) 2019-10-13T05:01:01.955 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:01.950+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.950608+0000) 2019-10-13T05:01:01.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:01.946+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:01.947405+0000) 2019-10-13T05:01:01.958 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:01.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68190 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:02.161 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:01:02.161 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:02.166 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:01:02.166 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:02.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:02.650+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:02.650774+0000) 2019-10-13T05:01:02.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:02.650+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:02.651464+0000) 2019-10-13T05:01:06.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:06.948+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.949308+0000) 2019-10-13T05:01:06.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:06.953+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.954076+0000) 2019-10-13T05:01:06.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:06.953+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.954222+0000) 2019-10-13T05:01:06.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:06.954+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.954481+0000) 2019-10-13T05:01:06.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:06.954+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.954915+0000) 2019-10-13T05:01:06.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:06.954+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.955127+0000) 2019-10-13T05:01:06.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:06.954+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.955322+0000) 2019-10-13T05:01:06.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:06.951+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:06.952615+0000) 2019-10-13T05:01:06.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:06.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:07.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:07.394+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:07.395151+0000) 2019-10-13T05:01:08.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:08.189+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:08.189392+0000) 2019-10-13T05:01:08.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:08.528+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:08.528764+0000) 2019-10-13T05:01:08.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:08.528+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:08.528812+0000) 2019-10-13T05:01:09.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:09.902+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:09.903288+0000) 2019-10-13T05:01:11.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:11.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.778608+0000) 2019-10-13T05:01:11.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:11.953+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.954199+0000) 2019-10-13T05:01:11.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:11.958+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.958678+0000) 2019-10-13T05:01:11.962 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:11.958+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.959330+0000) 2019-10-13T05:01:11.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:11.959+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.959492+0000) 2019-10-13T05:01:11.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:11.959+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.959628+0000) 2019-10-13T05:01:11.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:11.959+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.959847+0000) 2019-10-13T05:01:11.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:11.959+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.960124+0000) 2019-10-13T05:01:11.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:11.956+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:11.956839+0000) 2019-10-13T05:01:11.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:11.961+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:12.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:12.650+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:12.650943+0000) 2019-10-13T05:01:12.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:12.651+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:12.651615+0000) 2019-10-13T05:01:16.966 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:16.961+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.963261+0000) 2019-10-13T05:01:16.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:16.962+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.963495+0000) 2019-10-13T05:01:16.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:16.962+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.963646+0000) 2019-10-13T05:01:16.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:16.958+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.959537+0000) 2019-10-13T05:01:16.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:16.961+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.961817+0000) 2019-10-13T05:01:16.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:16.962+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.964275+0000) 2019-10-13T05:01:16.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:16.963+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.964558+0000) 2019-10-13T05:01:16.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:16.963+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:16.964895+0000) 2019-10-13T05:01:16.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:16.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:17.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:17.394+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:17.395296+0000) 2019-10-13T05:01:18.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:18.188+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:18.189573+0000) 2019-10-13T05:01:18.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:18.527+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:18.528947+0000) 2019-10-13T05:01:18.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:18.527+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:18.528965+0000) 2019-10-13T05:01:19.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:19.905+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:19.905687+0000) 2019-10-13T05:01:21.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:21.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.778809+0000) 2019-10-13T05:01:21.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:21.966+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.967793+0000) 2019-10-13T05:01:21.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:21.966+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.968064+0000) 2019-10-13T05:01:21.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:21.966+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.967958+0000) 2019-10-13T05:01:21.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:21.963+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.963946+0000) 2019-10-13T05:01:21.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:21.967+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.968593+0000) 2019-10-13T05:01:21.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:21.967+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.968967+0000) 2019-10-13T05:01:21.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:21.967+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.969029+0000) 2019-10-13T05:01:21.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:21.964+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:21.965535+0000) 2019-10-13T05:01:21.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:21.969+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:22.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:22.650+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:22.651111+0000) 2019-10-13T05:01:22.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:22.651+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:22.651767+0000) 2019-10-13T05:01:26.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:26.971+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.972497+0000) 2019-10-13T05:01:26.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:26.967+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.968363+0000) 2019-10-13T05:01:26.976 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:26.971+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.973078+0000) 2019-10-13T05:01:26.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:26.971+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.973287+0000) 2019-10-13T05:01:26.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:26.972+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.973515+0000) 2019-10-13T05:01:26.978 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:26.972+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.973606+0000) 2019-10-13T05:01:26.978 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:26.972+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.973844+0000) 2019-10-13T05:01:26.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:26.971+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:26.971830+0000) 2019-10-13T05:01:26.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:26.974+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:27.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:27.394+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:27.395468+0000) 2019-10-13T05:01:28.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:28.188+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:28.189700+0000) 2019-10-13T05:01:28.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:28.528+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:28.529124+0000) 2019-10-13T05:01:28.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:28.528+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:28.529159+0000) 2019-10-13T05:01:29.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:29.908+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:29.908110+0000) 2019-10-13T05:01:31.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:31.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.779032+0000) 2019-10-13T05:01:31.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:31.971+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.972812+0000) 2019-10-13T05:01:31.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:31.975+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.977389+0000) 2019-10-13T05:01:31.981 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:31.976+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.977654+0000) 2019-10-13T05:01:31.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:31.976+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.978103+0000) 2019-10-13T05:01:31.981 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:31.976+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.978371+0000) 2019-10-13T05:01:31.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:31.977+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.978538+0000) 2019-10-13T05:01:31.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:31.977+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.978743+0000) 2019-10-13T05:01:31.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:31.974+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:31.975821+0000) 2019-10-13T05:01:31.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:31.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:32.335 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:01:32.335 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:32.339 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:01:32.339 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:32.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:32.650+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:32.651266+0000) 2019-10-13T05:01:32.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:32.650+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:32.651895+0000) 2019-10-13T05:01:36.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:36.975+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.977315+0000) 2019-10-13T05:01:36.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:36.980+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.982152+0000) 2019-10-13T05:01:36.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:36.981+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.982565+0000) 2019-10-13T05:01:36.985 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:36.981+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.982415+0000) 2019-10-13T05:01:36.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:36.981+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.982978+0000) 2019-10-13T05:01:36.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:36.981+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.983084+0000) 2019-10-13T05:01:36.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:36.981+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.983092+0000) 2019-10-13T05:01:36.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:36.978+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:36.979811+0000) 2019-10-13T05:01:36.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:36.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:37.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:37.395+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:37.395634+0000) 2019-10-13T05:01:38.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:38.189+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:38.189880+0000) 2019-10-13T05:01:38.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:38.528+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:38.529320+0000) 2019-10-13T05:01:38.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:38.528+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:38.529324+0000) 2019-10-13T05:01:39.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:39.909+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:39.910593+0000) 2019-10-13T05:01:41.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:41.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.779223+0000) 2019-10-13T05:01:41.989 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:41.984+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.986383+0000) 2019-10-13T05:01:41.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:41.980+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.982569+0000) 2019-10-13T05:01:41.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:41.985+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.986949+0000) 2019-10-13T05:01:41.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:41.985+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.987146+0000) 2019-10-13T05:01:41.990 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:41.986+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.987517+0000) 2019-10-13T05:01:41.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:41.986+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.987625+0000) 2019-10-13T05:01:41.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:41.986+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.988000+0000) 2019-10-13T05:01:41.992 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:41.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:41.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:41.984+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:41.986445+0000) 2019-10-13T05:01:42.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:42.651+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:42.651472+0000) 2019-10-13T05:01:42.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:42.651+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:42.652061+0000) 2019-10-13T05:01:46.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:46.991+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.991410+0000) 2019-10-13T05:01:46.995 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:46.991+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.991771+0000) 2019-10-13T05:01:46.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:46.991+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.992053+0000) 2019-10-13T05:01:46.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:46.991+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.991943+0000) 2019-10-13T05:01:46.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:46.991+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.992343+0000) 2019-10-13T05:01:46.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:46.986+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.986972+0000) 2019-10-13T05:01:46.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:46.992+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.992663+0000) 2019-10-13T05:01:46.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:46.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68455 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:46.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:46.989+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:46.990190+0000) 2019-10-13T05:01:47.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:47.395+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:47.395804+0000) 2019-10-13T05:01:48.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:48.189+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:48.190111+0000) 2019-10-13T05:01:48.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:48.529+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:48.529495+0000) 2019-10-13T05:01:48.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:48.529+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:48.529503+0000) 2019-10-13T05:01:49.920 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:49.912+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:49.912882+0000) 2019-10-13T05:01:51.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:51.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.779356+0000) 2019-10-13T05:01:51.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:51.990+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.991486+0000) 2019-10-13T05:01:52.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:51.995+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.995746+0000) 2019-10-13T05:01:52.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:51.995+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.996048+0000) 2019-10-13T05:01:52.000 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:51.996+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.996518+0000) 2019-10-13T05:01:52.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:51.995+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.996289+0000) 2019-10-13T05:01:52.001 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:51.996+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.996634+0000) 2019-10-13T05:01:52.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:51.996+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.996969+0000) 2019-10-13T05:01:52.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:51.993+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:51.994444+0000) 2019-10-13T05:01:52.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:51.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:52.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:52.651+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:52.651592+0000) 2019-10-13T05:01:52.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:52.651+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:52.652180+0000) 2019-10-13T05:01:57.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:01:56.994+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:56.995511+0000) 2019-10-13T05:01:57.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:56.999+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:57.000050+0000) 2019-10-13T05:01:57.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:01:57.000+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:57.000420+0000) 2019-10-13T05:01:57.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:01:57.000+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:57.000826+0000) 2019-10-13T05:01:57.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:57.000+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:57.000957+0000) 2019-10-13T05:01:57.004 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:57.000+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:57.001249+0000) 2019-10-13T05:01:57.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:57.001+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:57.001470+0000) 2019-10-13T05:01:57.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:56.997+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:56.998344+0000) 2019-10-13T05:01:57.005 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:01:57.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:01:57.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:01:57.395+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:57.396002+0000) 2019-10-13T05:01:58.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:01:58.190+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:58.190293+0000) 2019-10-13T05:01:58.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:01:58.529+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:58.529649+0000) 2019-10-13T05:01:58.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:01:58.529+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:58.529687+0000) 2019-10-13T05:01:59.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:01:59.914+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:01:59.915218+0000) 2019-10-13T05:02:01.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:01.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:01.779504+0000) 2019-10-13T05:02:02.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:02.003+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.004716+0000) 2019-10-13T05:02:02.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:02.003+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.005093+0000) 2019-10-13T05:02:02.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:02.000+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.001171+0000) 2019-10-13T05:02:02.009 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:02.003+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.005366+0000) 2019-10-13T05:02:02.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:02.004+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.005727+0000) 2019-10-13T05:02:02.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:02.004+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.005870+0000) 2019-10-13T05:02:02.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:02.004+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.006245+0000) 2019-10-13T05:02:02.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:02.002+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.003035+0000) 2019-10-13T05:02:02.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:02.006+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68550 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:02.481 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:02:02.481 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:02.488 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:02:02.488 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:02.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:02.650+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.651748+0000) 2019-10-13T05:02:02.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:02.651+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:02.652362+0000) 2019-10-13T05:02:07.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:07.005+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.005799+0000) 2019-10-13T05:02:07.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:07.007+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.009260+0000) 2019-10-13T05:02:07.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:07.007+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.009381+0000) 2019-10-13T05:02:07.014 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:07.008+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.009715+0000) 2019-10-13T05:02:07.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:07.008+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.009887+0000) 2019-10-13T05:02:07.014 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:07.008+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.010281+0000) 2019-10-13T05:02:07.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:07.009+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.010425+0000) 2019-10-13T05:02:07.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:07.007+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.008299+0000) 2019-10-13T05:02:07.015 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:07.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68575 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:07.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:07.395+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:07.396163+0000) 2019-10-13T05:02:08.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:08.189+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:08.190467+0000) 2019-10-13T05:02:08.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:08.528+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:08.529823+0000) 2019-10-13T05:02:08.533 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:08.528+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:08.529857+0000) 2019-10-13T05:02:09.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:09.917+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:09.917551+0000) 2019-10-13T05:02:11.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:11.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:11.779620+0000) 2019-10-13T05:02:12.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:12.012+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.013588+0000) 2019-10-13T05:02:12.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:12.012+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.013849+0000) 2019-10-13T05:02:12.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:12.012+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.013933+0000) 2019-10-13T05:02:12.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:12.012+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.014192+0000) 2019-10-13T05:02:12.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:12.009+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.010201+0000) 2019-10-13T05:02:12.018 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:12.013+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.014504+0000) 2019-10-13T05:02:12.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:12.013+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.014754+0000) 2019-10-13T05:02:12.018 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:12.014+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:12.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:12.011+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.012305+0000) 2019-10-13T05:02:12.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:12.650+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.651948+0000) 2019-10-13T05:02:12.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:12.651+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:12.652524+0000) 2019-10-13T05:02:17.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:17.015+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.017364+0000) 2019-10-13T05:02:17.021 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:17.016+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.017558+0000) 2019-10-13T05:02:17.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:17.016+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.017809+0000) 2019-10-13T05:02:17.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:17.016+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.018010+0000) 2019-10-13T05:02:17.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:17.016+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.018212+0000) 2019-10-13T05:02:17.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:17.017+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.018500+0000) 2019-10-13T05:02:17.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:17.012+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.013522+0000) 2019-10-13T05:02:17.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:17.015+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.016171+0000) 2019-10-13T05:02:17.023 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:17.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:17.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:17.395+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:17.396333+0000) 2019-10-13T05:02:18.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:18.189+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:18.190644+0000) 2019-10-13T05:02:18.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:18.529+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:18.530041+0000) 2019-10-13T05:02:18.533 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:18.529+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:18.530079+0000) 2019-10-13T05:02:19.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:19.920+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:19.920137+0000) 2019-10-13T05:02:21.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:21.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:21.779805+0000) 2019-10-13T05:02:22.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:22.015+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.017591+0000) 2019-10-13T05:02:22.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:22.018+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.019804+0000) 2019-10-13T05:02:22.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:22.020+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.021736+0000) 2019-10-13T05:02:22.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:22.020+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.022168+0000) 2019-10-13T05:02:22.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:22.021+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.022581+0000) 2019-10-13T05:02:22.035 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:22.021+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.022701+0000) 2019-10-13T05:02:22.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:22.021+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.022872+0000) 2019-10-13T05:02:22.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:22.021+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.023330+0000) 2019-10-13T05:02:22.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:22.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:22.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:22.651+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.652123+0000) 2019-10-13T05:02:22.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:22.651+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:22.652698+0000) 2019-10-13T05:02:27.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:27.025+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.026998+0000) 2019-10-13T05:02:27.030 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:27.025+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.027205+0000) 2019-10-13T05:02:27.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:27.025+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.027362+0000) 2019-10-13T05:02:27.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:27.021+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.023471+0000) 2019-10-13T05:02:27.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:27.026+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.027626+0000) 2019-10-13T05:02:27.031 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:27.026+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.027885+0000) 2019-10-13T05:02:27.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:27.027+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.028431+0000) 2019-10-13T05:02:27.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:27.023+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.025359+0000) 2019-10-13T05:02:27.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:27.028+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:27.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:27.396+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:27.396523+0000) 2019-10-13T05:02:28.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:28.190+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:28.190810+0000) 2019-10-13T05:02:28.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:28.529+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:28.530243+0000) 2019-10-13T05:02:28.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:28.529+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:28.530304+0000) 2019-10-13T05:02:29.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:29.921+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:29.922471+0000) 2019-10-13T05:02:31.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:31.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:31.780022+0000) 2019-10-13T05:02:32.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:32.025+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.026918+0000) 2019-10-13T05:02:32.034 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:32.031+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.031420+0000) 2019-10-13T05:02:32.034 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:32.031+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.031589+0000) 2019-10-13T05:02:32.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:32.031+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.032030+0000) 2019-10-13T05:02:32.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:32.031+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.032377+0000) 2019-10-13T05:02:32.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:32.032+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.032677+0000) 2019-10-13T05:02:32.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:32.032+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.033128+0000) 2019-10-13T05:02:32.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:32.027+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.029441+0000) 2019-10-13T05:02:32.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:32.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:32.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:32.651+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.652292+0000) 2019-10-13T05:02:32.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:32.652+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:32.652850+0000) 2019-10-13T05:02:32.720 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:02:32.720 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:32.724 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:02:32.724 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:37.038 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:37.035+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.035677+0000) 2019-10-13T05:02:37.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:37.035+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.035937+0000) 2019-10-13T05:02:37.039 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:37.035+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.036229+0000) 2019-10-13T05:02:37.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:37.031+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.032710+0000) 2019-10-13T05:02:37.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:37.036+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.036504+0000) 2019-10-13T05:02:37.040 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:37.036+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.036899+0000) 2019-10-13T05:02:37.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:37.036+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.037215+0000) 2019-10-13T05:02:37.041 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:37.033+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.034293+0000) 2019-10-13T05:02:37.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:37.038+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:37.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:37.396+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:37.396693+0000) 2019-10-13T05:02:38.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:38.190+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:38.190986+0000) 2019-10-13T05:02:38.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:38.530+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:38.530409+0000) 2019-10-13T05:02:38.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:38.530+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:38.530477+0000) 2019-10-13T05:02:39.932 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:39.924+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:39.924873+0000) 2019-10-13T05:02:41.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:41.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:41.780233+0000) 2019-10-13T05:02:42.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:42.039+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.040100+0000) 2019-10-13T05:02:42.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:42.039+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.040250+0000) 2019-10-13T05:02:42.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:42.040+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.040442+0000) 2019-10-13T05:02:42.044 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:42.040+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.040545+0000) 2019-10-13T05:02:42.044 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:42.040+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.041032+0000) 2019-10-13T05:02:42.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:42.036+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.036834+0000) 2019-10-13T05:02:42.045 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:42.041+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.041427+0000) 2019-10-13T05:02:42.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:42.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:42.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:42.038+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.039143+0000) 2019-10-13T05:02:42.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:42.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.652461+0000) 2019-10-13T05:02:42.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:42.652+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:42.652986+0000) 2019-10-13T05:02:47.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:47.043+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.044630+0000) 2019-10-13T05:02:47.048 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:47.043+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.044744+0000) 2019-10-13T05:02:47.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:47.043+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.045217+0000) 2019-10-13T05:02:47.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:47.040+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.041225+0000) 2019-10-13T05:02:47.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:47.044+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.045404+0000) 2019-10-13T05:02:47.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:47.044+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.045907+0000) 2019-10-13T05:02:47.050 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:47.044+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.046197+0000) 2019-10-13T05:02:47.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:47.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:47.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:47.043+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.044243+0000) 2019-10-13T05:02:47.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:47.395+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:47.396878+0000) 2019-10-13T05:02:48.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:48.189+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:48.191221+0000) 2019-10-13T05:02:48.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:48.529+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:48.530596+0000) 2019-10-13T05:02:48.533 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:48.529+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:48.530623+0000) 2019-10-13T05:02:49.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:49.926+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:49.927331+0000) 2019-10-13T05:02:51.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:51.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:51.780434+0000) 2019-10-13T05:02:52.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:52.047+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.049313+0000) 2019-10-13T05:02:52.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:52.045+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.045839+0000) 2019-10-13T05:02:52.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:52.048+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.049933+0000) 2019-10-13T05:02:52.053 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:52.048+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.049980+0000) 2019-10-13T05:02:52.053 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:52.048+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.050237+0000) 2019-10-13T05:02:52.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:52.046+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.047361+0000) 2019-10-13T05:02:52.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:52.049+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.050414+0000) 2019-10-13T05:02:52.061 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:52.049+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.050610+0000) 2019-10-13T05:02:52.061 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:52.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68849 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:52.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:52.651+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.652620+0000) 2019-10-13T05:02:52.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:52.652+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:52.653138+0000) 2019-10-13T05:02:57.056 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:02:57.052+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.054142+0000) 2019-10-13T05:02:57.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:02:57.049+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.050238+0000) 2019-10-13T05:02:57.058 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:57.053+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.054706+0000) 2019-10-13T05:02:57.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:02:57.053+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.054924+0000) 2019-10-13T05:02:57.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:57.053+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.055097+0000) 2019-10-13T05:02:57.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:57.053+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.055245+0000) 2019-10-13T05:02:57.059 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:57.054+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.055419+0000) 2019-10-13T05:02:57.059 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:02:57.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68876 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:02:57.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:57.053+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.054148+0000) 2019-10-13T05:02:57.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:02:57.396+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:57.397050+0000) 2019-10-13T05:02:58.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:02:58.190+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:58.191423+0000) 2019-10-13T05:02:58.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:02:58.530+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:58.530814+0000) 2019-10-13T05:02:58.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:02:58.530+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:58.530816+0000) 2019-10-13T05:02:59.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:02:59.929+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:02:59.929676+0000) 2019-10-13T05:03:01.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:01.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:01.780615+0000) 2019-10-13T05:03:02.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:02.057+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.058857+0000) 2019-10-13T05:03:02.062 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:02.057+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.059054+0000) 2019-10-13T05:03:02.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:02.054+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.054907+0000) 2019-10-13T05:03:02.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:02.058+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.059416+0000) 2019-10-13T05:03:02.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:02.058+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.059772+0000) 2019-10-13T05:03:02.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:02.058+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.059572+0000) 2019-10-13T05:03:02.063 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:02.058+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.060138+0000) 2019-10-13T05:03:02.064 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:02.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68910 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:02.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:02.056+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.057378+0000) 2019-10-13T05:03:02.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:02.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.652760+0000) 2019-10-13T05:03:02.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:02.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:02.653239+0000) 2019-10-13T05:03:02.898 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:03:02.898 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:02.903 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:03:02.904 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:07.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:07.061+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.063205+0000) 2019-10-13T05:03:07.066 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:07.062+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.063424+0000) 2019-10-13T05:03:07.067 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:07.062+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.063908+0000) 2019-10-13T05:03:07.067 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:07.062+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.064097+0000) 2019-10-13T05:03:07.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:07.063+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.064517+0000) 2019-10-13T05:03:07.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:07.059+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.060357+0000) 2019-10-13T05:03:07.068 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:07.063+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.065172+0000) 2019-10-13T05:03:07.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:07.065+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:07.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:07.062+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.063383+0000) 2019-10-13T05:03:07.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:07.396+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:07.397219+0000) 2019-10-13T05:03:08.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:08.191+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:08.191636+0000) 2019-10-13T05:03:08.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:08.530+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:08.530992+0000) 2019-10-13T05:03:08.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:08.530+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:08.530994+0000) 2019-10-13T05:03:09.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:09.932+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:09.932117+0000) 2019-10-13T05:03:11.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:11.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:11.780783+0000) 2019-10-13T05:03:12.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:12.062+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.063932+0000) 2019-10-13T05:03:12.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:12.067+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.068496+0000) 2019-10-13T05:03:12.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:12.067+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.068703+0000) 2019-10-13T05:03:12.072 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:12.067+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.069121+0000) 2019-10-13T05:03:12.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:12.068+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.069611+0000) 2019-10-13T05:03:12.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:12.067+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.069224+0000) 2019-10-13T05:03:12.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:12.068+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.069654+0000) 2019-10-13T05:03:12.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:12.064+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.066500+0000) 2019-10-13T05:03:12.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:12.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:12.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:12.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.652936+0000) 2019-10-13T05:03:12.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:12.652+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:12.653398+0000) 2019-10-13T05:03:17.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:17.072+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.072972+0000) 2019-10-13T05:03:17.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:17.072+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.073333+0000) 2019-10-13T05:03:17.076 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:17.072+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.073368+0000) 2019-10-13T05:03:17.076 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:17.073+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.073612+0000) 2019-10-13T05:03:17.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:17.067+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.069160+0000) 2019-10-13T05:03:17.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:17.068+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.070543+0000) 2019-10-13T05:03:17.078 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:17.073+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.073977+0000) 2019-10-13T05:03:17.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:17.074+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.074450+0000) 2019-10-13T05:03:17.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:17.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 68996 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:17.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:17.397+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:17.397419+0000) 2019-10-13T05:03:18.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:18.191+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:18.191803+0000) 2019-10-13T05:03:18.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:18.530+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:18.531194+0000) 2019-10-13T05:03:18.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:18.530+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:18.531195+0000) 2019-10-13T05:03:19.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:19.933+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:19.934425+0000) 2019-10-13T05:03:21.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:21.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:21.780945+0000) 2019-10-13T05:03:22.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:22.078+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.078435+0000) 2019-10-13T05:03:22.082 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:22.078+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.078703+0000) 2019-10-13T05:03:22.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:22.078+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.078922+0000) 2019-10-13T05:03:22.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:22.078+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.079100+0000) 2019-10-13T05:03:22.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:22.078+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.079197+0000) 2019-10-13T05:03:22.082 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:22.079+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.079522+0000) 2019-10-13T05:03:22.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:22.073+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.075085+0000) 2019-10-13T05:03:22.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:22.074+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.076148+0000) 2019-10-13T05:03:22.084 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:22.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69030 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:22.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:22.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.653086+0000) 2019-10-13T05:03:22.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:22.652+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:22.653554+0000) 2019-10-13T05:03:27.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:27.082+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.082808+0000) 2019-10-13T05:03:27.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:27.078+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.079030+0000) 2019-10-13T05:03:27.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:27.082+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.083069+0000) 2019-10-13T05:03:27.086 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:27.082+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.083372+0000) 2019-10-13T05:03:27.087 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:27.083+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.083817+0000) 2019-10-13T05:03:27.087 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:27.083+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.084090+0000) 2019-10-13T05:03:27.087 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:27.084+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.084509+0000) 2019-10-13T05:03:27.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:27.085+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:27.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:27.081+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.082479+0000) 2019-10-13T05:03:27.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:27.397+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:27.397593+0000) 2019-10-13T05:03:28.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:28.191+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:28.191973+0000) 2019-10-13T05:03:28.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:28.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:28.531357+0000) 2019-10-13T05:03:28.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:28.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:28.531380+0000) 2019-10-13T05:03:29.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:29.936+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:29.936774+0000) 2019-10-13T05:03:31.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:31.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:31.781078+0000) 2019-10-13T05:03:32.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:32.086+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.087828+0000) 2019-10-13T05:03:32.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:32.086+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.088037+0000) 2019-10-13T05:03:32.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:32.086+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.088125+0000) 2019-10-13T05:03:32.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:32.087+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.088494+0000) 2019-10-13T05:03:32.092 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:32.087+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.088962+0000) 2019-10-13T05:03:32.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:32.084+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.084885+0000) 2019-10-13T05:03:32.092 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:32.087+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.089363+0000) 2019-10-13T05:03:32.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:32.089+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69089 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:32.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:32.085+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.086486+0000) 2019-10-13T05:03:32.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:32.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.653257+0000) 2019-10-13T05:03:32.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:32.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:32.653749+0000) 2019-10-13T05:03:33.010 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:03:33.011 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:33.014 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:03:33.015 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:37.095 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:37.090+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.092174+0000) 2019-10-13T05:03:37.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:37.090+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.092384+0000) 2019-10-13T05:03:37.095 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:37.091+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.092695+0000) 2019-10-13T05:03:37.096 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:37.091+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.092811+0000) 2019-10-13T05:03:37.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:37.088+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.089002+0000) 2019-10-13T05:03:37.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:37.091+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.092982+0000) 2019-10-13T05:03:37.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:37.092+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.093649+0000) 2019-10-13T05:03:37.097 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:37.093+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:37.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:37.090+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.091199+0000) 2019-10-13T05:03:37.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:37.396+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:37.397734+0000) 2019-10-13T05:03:38.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:38.191+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:38.192133+0000) 2019-10-13T05:03:38.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:38.530+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:38.531483+0000) 2019-10-13T05:03:38.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:38.530+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:38.531538+0000) 2019-10-13T05:03:39.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:39.938+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:39.939333+0000) 2019-10-13T05:03:41.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:41.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:41.781203+0000) 2019-10-13T05:03:42.100 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:42.095+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.097012+0000) 2019-10-13T05:03:42.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:42.095+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.097229+0000) 2019-10-13T05:03:42.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:42.095+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.097348+0000) 2019-10-13T05:03:42.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:42.092+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.093371+0000) 2019-10-13T05:03:42.101 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:42.096+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.097927+0000) 2019-10-13T05:03:42.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:42.096+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.097973+0000) 2019-10-13T05:03:42.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:42.096+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.098220+0000) 2019-10-13T05:03:42.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:42.094+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.095259+0000) 2019-10-13T05:03:42.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:42.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:42.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:42.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.653384+0000) 2019-10-13T05:03:42.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:42.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:42.653891+0000) 2019-10-13T05:03:47.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:47.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.101540+0000) 2019-10-13T05:03:47.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:47.100+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.101829+0000) 2019-10-13T05:03:47.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:47.097+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.097883+0000) 2019-10-13T05:03:47.106 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:47.100+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.102115+0000) 2019-10-13T05:03:47.106 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:47.100+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.102345+0000) 2019-10-13T05:03:47.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:47.101+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.102490+0000) 2019-10-13T05:03:47.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:47.101+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.102918+0000) 2019-10-13T05:03:47.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:47.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69175 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:47.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:47.100+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.101094+0000) 2019-10-13T05:03:47.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:47.396+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:47.397876+0000) 2019-10-13T05:03:48.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:48.191+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:48.192308+0000) 2019-10-13T05:03:48.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:48.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:48.531676+0000) 2019-10-13T05:03:48.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:48.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:48.531769+0000) 2019-10-13T05:03:49.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:49.940+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:49.941616+0000) 2019-10-13T05:03:51.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:51.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:51.781414+0000) 2019-10-13T05:03:52.109 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:52.105+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.106646+0000) 2019-10-13T05:03:52.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:52.102+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.102843+0000) 2019-10-13T05:03:52.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:52.105+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.107373+0000) 2019-10-13T05:03:52.110 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:52.106+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.107522+0000) 2019-10-13T05:03:52.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:52.106+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.107879+0000) 2019-10-13T05:03:52.111 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:52.106+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.108089+0000) 2019-10-13T05:03:52.111 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:52.106+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.108256+0000) 2019-10-13T05:03:52.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:52.104+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.104755+0000) 2019-10-13T05:03:52.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:52.107+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:52.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:52.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.653559+0000) 2019-10-13T05:03:52.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:52.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:52.654033+0000) 2019-10-13T05:03:57.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:57.109+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.111105+0000) 2019-10-13T05:03:57.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:57.109+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.111303+0000) 2019-10-13T05:03:57.115 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:03:57.110+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.111516+0000) 2019-10-13T05:03:57.115 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:03:57.110+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.111795+0000) 2019-10-13T05:03:57.115 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:57.110+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.112010+0000) 2019-10-13T05:03:57.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:57.110+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.111927+0000) 2019-10-13T05:03:57.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:03:57.107+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.108274+0000) 2019-10-13T05:03:57.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:03:57.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69236 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:03:57.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:57.110+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.110841+0000) 2019-10-13T05:03:57.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:03:57.397+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:57.398043+0000) 2019-10-13T05:03:58.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:03:58.192+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:58.192509+0000) 2019-10-13T05:03:58.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:03:58.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:58.531861+0000) 2019-10-13T05:03:58.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:03:58.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:58.531976+0000) 2019-10-13T05:03:59.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:03:59.943+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:03:59.943980+0000) 2019-10-13T05:04:01.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:01.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:01.781570+0000) 2019-10-13T05:04:02.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:02.115+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.116170+0000) 2019-10-13T05:04:02.120 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:02.116+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.116552+0000) 2019-10-13T05:04:02.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:02.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.116760+0000) 2019-10-13T05:04:02.121 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:02.116+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.116865+0000) 2019-10-13T05:04:02.121 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:02.116+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.116992+0000) 2019-10-13T05:04:02.122 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:02.116+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.117191+0000) 2019-10-13T05:04:02.123 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:02.111+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.113420+0000) 2019-10-13T05:04:02.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:02.113+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.115148+0000) 2019-10-13T05:04:02.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:02.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:02.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:02.653+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.653702+0000) 2019-10-13T05:04:02.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:02.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:02.654189+0000) 2019-10-13T05:04:03.140 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:04:03.140 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:03.144 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:04:03.144 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:07.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:07.119+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.120252+0000) 2019-10-13T05:04:07.123 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:07.120+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.120437+0000) 2019-10-13T05:04:07.124 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:07.114+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.116192+0000) 2019-10-13T05:04:07.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:07.120+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.120846+0000) 2019-10-13T05:04:07.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:07.120+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.121070+0000) 2019-10-13T05:04:07.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:07.120+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.121247+0000) 2019-10-13T05:04:07.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:07.121+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.121651+0000) 2019-10-13T05:04:07.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:07.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69295 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:07.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:07.117+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.119358+0000) 2019-10-13T05:04:07.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:07.397+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:07.398240+0000) 2019-10-13T05:04:08.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:08.192+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:08.192660+0000) 2019-10-13T05:04:08.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:08.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:08.532068+0000) 2019-10-13T05:04:08.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:08.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:08.532188+0000) 2019-10-13T05:04:09.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:09.945+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:09.946373+0000) 2019-10-13T05:04:11.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:11.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:11.781780+0000) 2019-10-13T05:04:12.127 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:12.124+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.124420+0000) 2019-10-13T05:04:12.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:12.124+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.124681+0000) 2019-10-13T05:04:12.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:12.119+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.121051+0000) 2019-10-13T05:04:12.129 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:12.124+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.125008+0000) 2019-10-13T05:04:12.129 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:12.124+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.125148+0000) 2019-10-13T05:04:12.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:12.125+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.125493+0000) 2019-10-13T05:04:12.129 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:12.125+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.125720+0000) 2019-10-13T05:04:12.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:12.121+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.123322+0000) 2019-10-13T05:04:12.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:12.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:12.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:12.653+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.653866+0000) 2019-10-13T05:04:12.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:12.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:12.654348+0000) 2019-10-13T05:04:17.132 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:17.128+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.128942+0000) 2019-10-13T05:04:17.132 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:17.128+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.129059+0000) 2019-10-13T05:04:17.132 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:17.128+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.129173+0000) 2019-10-13T05:04:17.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:17.129+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.129391+0000) 2019-10-13T05:04:17.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:17.124+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.125478+0000) 2019-10-13T05:04:17.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:17.125+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.126505+0000) 2019-10-13T05:04:17.134 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:17.129+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.129789+0000) 2019-10-13T05:04:17.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:17.129+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.130038+0000) 2019-10-13T05:04:17.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:17.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69356 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:17.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:17.397+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:17.398375+0000) 2019-10-13T05:04:18.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:18.191+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:18.192847+0000) 2019-10-13T05:04:18.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:18.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:18.532230+0000) 2019-10-13T05:04:18.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:18.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:18.532361+0000) 2019-10-13T05:04:19.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:19.948+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:19.948762+0000) 2019-10-13T05:04:21.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:21.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:21.781958+0000) 2019-10-13T05:04:22.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:22.132+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.133433+0000) 2019-10-13T05:04:22.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:22.132+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.133787+0000) 2019-10-13T05:04:22.137 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:22.132+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.133954+0000) 2019-10-13T05:04:22.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:22.129+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.130186+0000) 2019-10-13T05:04:22.138 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:22.132+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.134139+0000) 2019-10-13T05:04:22.138 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:22.132+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.134315+0000) 2019-10-13T05:04:22.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:22.133+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.134784+0000) 2019-10-13T05:04:22.138 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:22.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:22.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:22.132+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.132899+0000) 2019-10-13T05:04:22.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:22.652+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.654009+0000) 2019-10-13T05:04:22.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:22.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:22.654500+0000) 2019-10-13T05:04:27.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:27.136+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.137591+0000) 2019-10-13T05:04:27.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:27.136+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.137795+0000) 2019-10-13T05:04:27.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:27.133+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.133857+0000) 2019-10-13T05:04:27.142 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:27.136+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.138300+0000) 2019-10-13T05:04:27.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:27.137+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.138741+0000) 2019-10-13T05:04:27.142 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:27.137+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.138614+0000) 2019-10-13T05:04:27.142 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:27.137+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.138990+0000) 2019-10-13T05:04:27.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:27.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:27.143 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:27.135+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.136258+0000) 2019-10-13T05:04:27.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:27.397+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:27.398525+0000) 2019-10-13T05:04:28.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:28.192+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:28.193040+0000) 2019-10-13T05:04:28.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:28.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:28.532376+0000) 2019-10-13T05:04:28.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:28.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:28.532474+0000) 2019-10-13T05:04:29.958 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:29.950+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:29.951060+0000) 2019-10-13T05:04:31.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:31.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:31.782162+0000) 2019-10-13T05:04:32.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:32.140+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.142092+0000) 2019-10-13T05:04:32.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:32.141+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.142520+0000) 2019-10-13T05:04:32.146 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:32.137+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.138487+0000) 2019-10-13T05:04:32.146 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:32.141+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.142964+0000) 2019-10-13T05:04:32.147 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:32.141+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.143096+0000) 2019-10-13T05:04:32.147 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:32.142+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.143409+0000) 2019-10-13T05:04:32.147 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:32.142+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.143589+0000) 2019-10-13T05:04:32.148 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:32.139+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.140454+0000) 2019-10-13T05:04:32.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:32.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69449 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:32.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:32.653+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.654189+0000) 2019-10-13T05:04:32.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:32.654+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:32.654651+0000) 2019-10-13T05:04:33.286 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:04:33.286 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:33.290 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:04:33.291 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:37.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:37.145+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.146422+0000) 2019-10-13T05:04:37.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:37.145+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.146617+0000) 2019-10-13T05:04:37.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:37.145+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.146809+0000) 2019-10-13T05:04:37.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:37.142+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.142917+0000) 2019-10-13T05:04:37.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:37.146+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.147536+0000) 2019-10-13T05:04:37.151 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:37.146+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.148045+0000) 2019-10-13T05:04:37.151 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:37.146+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.148316+0000) 2019-10-13T05:04:37.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:37.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69476 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:37.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:37.144+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.145564+0000) 2019-10-13T05:04:37.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:37.398+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:37.398665+0000) 2019-10-13T05:04:38.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:38.192+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:38.193213+0000) 2019-10-13T05:04:38.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:38.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:38.532560+0000) 2019-10-13T05:04:38.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:38.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:38.532639+0000) 2019-10-13T05:04:39.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:39.953+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:39.954039+0000) 2019-10-13T05:04:41.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:41.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:41.782350+0000) 2019-10-13T05:04:42.156 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:42.150+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.151623+0000) 2019-10-13T05:04:42.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:42.150+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.151783+0000) 2019-10-13T05:04:42.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:42.150+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.152005+0000) 2019-10-13T05:04:42.157 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:42.150+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.152184+0000) 2019-10-13T05:04:42.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:42.151+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.152581+0000) 2019-10-13T05:04:42.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:42.151+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.152912+0000) 2019-10-13T05:04:42.157 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:42.145+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.146672+0000) 2019-10-13T05:04:42.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:42.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:42.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:42.149+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.150129+0000) 2019-10-13T05:04:42.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:42.653+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.654305+0000) 2019-10-13T05:04:42.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:42.654+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:42.654785+0000) 2019-10-13T05:04:47.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:47.151+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.152180+0000) 2019-10-13T05:04:47.160 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:47.156+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.156812+0000) 2019-10-13T05:04:47.162 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:47.156+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.156947+0000) 2019-10-13T05:04:47.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:47.156+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.157264+0000) 2019-10-13T05:04:47.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:47.157+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.157678+0000) 2019-10-13T05:04:47.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:47.157+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.157803+0000) 2019-10-13T05:04:47.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:47.154+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.154964+0000) 2019-10-13T05:04:47.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:47.157+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.158008+0000) 2019-10-13T05:04:47.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:47.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:47.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:47.398+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:47.398811+0000) 2019-10-13T05:04:48.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:48.193+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:48.193330+0000) 2019-10-13T05:04:48.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:48.532+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:48.532745+0000) 2019-10-13T05:04:48.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:48.532+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:48.532856+0000) 2019-10-13T05:04:49.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:49.956+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:49.956375+0000) 2019-10-13T05:04:51.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:51.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:51.782488+0000) 2019-10-13T05:04:52.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:52.160+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.160888+0000) 2019-10-13T05:04:52.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:52.160+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.161178+0000) 2019-10-13T05:04:52.164 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:52.160+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.161386+0000) 2019-10-13T05:04:52.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:52.154+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.156654+0000) 2019-10-13T05:04:52.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:52.161+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.161602+0000) 2019-10-13T05:04:52.165 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:52.161+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.161893+0000) 2019-10-13T05:04:52.165 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:52.161+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.162268+0000) 2019-10-13T05:04:52.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:52.156+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.158357+0000) 2019-10-13T05:04:52.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:52.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69569 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:52.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:52.654+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.654410+0000) 2019-10-13T05:04:52.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:52.653+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:52.654897+0000) 2019-10-13T05:04:57.168 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:04:57.164+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.165275+0000) 2019-10-13T05:04:57.169 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:04:57.159+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.161443+0000) 2019-10-13T05:04:57.169 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:57.165+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.165839+0000) 2019-10-13T05:04:57.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:57.165+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.166345+0000) 2019-10-13T05:04:57.169 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:04:57.166+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.166440+0000) 2019-10-13T05:04:57.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:57.166+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.166615+0000) 2019-10-13T05:04:57.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:57.166+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.166846+0000) 2019-10-13T05:04:57.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:04:57.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:04:57.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:57.161+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.163515+0000) 2019-10-13T05:04:57.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:04:57.398+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:57.398985+0000) 2019-10-13T05:04:58.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:04:58.193+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:58.193506+0000) 2019-10-13T05:04:58.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:04:58.532+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:58.532935+0000) 2019-10-13T05:04:58.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:04:58.532+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:58.533043+0000) 2019-10-13T05:04:59.966 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:04:59.957+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:04:59.958806+0000) 2019-10-13T05:05:01.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:01.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:01.782663+0000) 2019-10-13T05:05:02.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:02.169+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.169740+0000) 2019-10-13T05:05:02.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:02.169+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.170106+0000) 2019-10-13T05:05:02.173 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:02.169+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.170253+0000) 2019-10-13T05:05:02.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:02.164+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.166317+0000) 2019-10-13T05:05:02.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:02.170+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.170432+0000) 2019-10-13T05:05:02.174 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:02.170+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.170918+0000) 2019-10-13T05:05:02.174 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:02.170+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.171218+0000) 2019-10-13T05:05:02.175 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:02.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69630 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:02.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:02.166+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.168563+0000) 2019-10-13T05:05:02.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:02.653+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.654563+0000) 2019-10-13T05:05:02.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:02.654+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:02.655079+0000) 2019-10-13T05:05:03.432 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:05:03.432 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:03.436 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:05:03.436 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:07.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:07.172+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.174170+0000) 2019-10-13T05:05:07.178 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:07.173+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.174520+0000) 2019-10-13T05:05:07.178 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:07.173+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.174651+0000) 2019-10-13T05:05:07.178 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:07.173+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.174858+0000) 2019-10-13T05:05:07.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:07.173+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.175040+0000) 2019-10-13T05:05:07.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:07.173+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.175099+0000) 2019-10-13T05:05:07.179 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:07.170+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.170964+0000) 2019-10-13T05:05:07.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:07.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69655 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:07.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:07.172+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.172836+0000) 2019-10-13T05:05:07.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:07.397+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:07.399168+0000) 2019-10-13T05:05:08.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:08.192+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:08.193719+0000) 2019-10-13T05:05:08.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:08.532+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:08.533121+0000) 2019-10-13T05:05:08.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:08.532+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:08.533206+0000) 2019-10-13T05:05:09.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:09.960+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:09.961147+0000) 2019-10-13T05:05:11.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:11.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:11.782846+0000) 2019-10-13T05:05:12.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:12.177+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.178838+0000) 2019-10-13T05:05:12.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:12.177+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.179157+0000) 2019-10-13T05:05:12.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:12.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.175739+0000) 2019-10-13T05:05:12.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:12.178+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.179577+0000) 2019-10-13T05:05:12.183 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:12.178+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.179658+0000) 2019-10-13T05:05:12.183 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:12.178+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.179873+0000) 2019-10-13T05:05:12.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:12.178+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.180176+0000) 2019-10-13T05:05:12.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:12.175+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.176449+0000) 2019-10-13T05:05:12.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:12.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69689 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:12.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:12.653+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.654700+0000) 2019-10-13T05:05:12.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:12.654+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:12.655219+0000) 2019-10-13T05:05:17.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:17.181+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.183185+0000) 2019-10-13T05:05:17.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:17.182+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.183498+0000) 2019-10-13T05:05:17.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:17.182+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.183841+0000) 2019-10-13T05:05:17.187 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:17.182+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.183762+0000) 2019-10-13T05:05:17.188 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:17.182+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.184197+0000) 2019-10-13T05:05:17.189 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:17.179+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.180293+0000) 2019-10-13T05:05:17.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:17.181+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.182020+0000) 2019-10-13T05:05:17.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:17.183+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.184447+0000) 2019-10-13T05:05:17.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:17.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69716 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:17.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:17.398+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:17.399315+0000) 2019-10-13T05:05:18.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:18.193+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:18.193889+0000) 2019-10-13T05:05:18.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:18.531+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:18.533315+0000) 2019-10-13T05:05:18.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:18.531+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:18.533352+0000) 2019-10-13T05:05:19.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:19.963+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:19.963452+0000) 2019-10-13T05:05:21.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:21.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:21.783028+0000) 2019-10-13T05:05:22.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:22.185+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.187250+0000) 2019-10-13T05:05:22.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:22.186+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.187609+0000) 2019-10-13T05:05:22.192 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:22.186+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.187749+0000) 2019-10-13T05:05:22.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:22.186+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.188045+0000) 2019-10-13T05:05:22.192 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:22.186+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.188170+0000) 2019-10-13T05:05:22.193 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:22.187+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.188635+0000) 2019-10-13T05:05:22.193 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:22.183+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.183967+0000) 2019-10-13T05:05:22.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:22.184+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.184802+0000) 2019-10-13T05:05:22.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:22.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69750 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:22.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:22.654+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.654871+0000) 2019-10-13T05:05:22.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:22.654+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:22.655356+0000) 2019-10-13T05:05:27.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:27.186+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.187168+0000) 2019-10-13T05:05:27.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:27.190+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.191721+0000) 2019-10-13T05:05:27.195 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:27.190+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.191920+0000) 2019-10-13T05:05:27.196 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:27.190+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.192219+0000) 2019-10-13T05:05:27.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:27.191+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.192416+0000) 2019-10-13T05:05:27.196 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:27.191+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.192555+0000) 2019-10-13T05:05:27.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:27.191+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.192707+0000) 2019-10-13T05:05:27.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:27.192+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69775 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:27.197 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:27.189+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.190595+0000) 2019-10-13T05:05:27.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:27.398+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:27.399490+0000) 2019-10-13T05:05:28.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:28.193+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:28.194080+0000) 2019-10-13T05:05:28.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:28.533+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:28.533497+0000) 2019-10-13T05:05:28.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:28.533+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:28.533521+0000) 2019-10-13T05:05:29.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:29.965+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:29.965800+0000) 2019-10-13T05:05:31.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:31.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:31.783200+0000) 2019-10-13T05:05:32.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:32.195+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.196070+0000) 2019-10-13T05:05:32.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:32.195+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.196385+0000) 2019-10-13T05:05:32.201 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:32.191+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.192395+0000) 2019-10-13T05:05:32.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:32.196+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.196921+0000) 2019-10-13T05:05:32.201 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:32.196+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.197078+0000) 2019-10-13T05:05:32.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:32.197+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.197410+0000) 2019-10-13T05:05:32.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:32.197+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.197718+0000) 2019-10-13T05:05:32.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:32.194+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.194769+0000) 2019-10-13T05:05:32.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:32.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:32.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:32.654+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.655033+0000) 2019-10-13T05:05:32.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:32.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:32.655486+0000) 2019-10-13T05:05:33.555 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:05:33.556 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:33.561 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:05:33.561 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:37.206 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:37.200+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.200797+0000) 2019-10-13T05:05:37.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:37.200+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.200948+0000) 2019-10-13T05:05:37.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:37.200+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.201260+0000) 2019-10-13T05:05:37.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:37.201+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.201439+0000) 2019-10-13T05:05:37.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:37.201+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.201623+0000) 2019-10-13T05:05:37.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:37.201+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.201956+0000) 2019-10-13T05:05:37.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:37.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:37.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:37.195+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.196222+0000) 2019-10-13T05:05:37.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:37.197+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.198096+0000) 2019-10-13T05:05:37.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:37.399+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:37.399654+0000) 2019-10-13T05:05:38.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:38.193+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:38.194294+0000) 2019-10-13T05:05:38.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:38.533+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:38.533682+0000) 2019-10-13T05:05:38.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:38.533+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:38.533702+0000) 2019-10-13T05:05:39.975 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:39.968+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:39.968167+0000) 2019-10-13T05:05:41.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:41.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:41.783387+0000) 2019-10-13T05:05:42.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:42.198+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.200601+0000) 2019-10-13T05:05:42.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:42.204+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.205349+0000) 2019-10-13T05:05:42.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:42.205+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.205586+0000) 2019-10-13T05:05:42.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:42.205+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.205812+0000) 2019-10-13T05:05:42.209 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:42.205+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.206370+0000) 2019-10-13T05:05:42.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:42.206+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.206797+0000) 2019-10-13T05:05:42.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:42.206+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.206952+0000) 2019-10-13T05:05:42.210 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:42.207+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69870 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:42.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:42.202+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.203822+0000) 2019-10-13T05:05:42.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:42.654+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.655173+0000) 2019-10-13T05:05:42.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:42.654+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:42.655638+0000) 2019-10-13T05:05:47.213 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:47.209+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.209604+0000) 2019-10-13T05:05:47.213 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:47.209+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.209797+0000) 2019-10-13T05:05:47.214 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:47.204+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.205991+0000) 2019-10-13T05:05:47.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:47.210+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.210648+0000) 2019-10-13T05:05:47.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:47.210+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.211128+0000) 2019-10-13T05:05:47.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:47.210+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.210989+0000) 2019-10-13T05:05:47.215 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:47.210+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.210788+0000) 2019-10-13T05:05:47.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:47.206+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.208265+0000) 2019-10-13T05:05:47.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:47.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:47.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:47.399+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:47.399852+0000) 2019-10-13T05:05:48.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:48.193+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:48.194496+0000) 2019-10-13T05:05:48.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:48.532+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:48.533892+0000) 2019-10-13T05:05:48.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:48.532+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:48.533893+0000) 2019-10-13T05:05:49.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:49.969+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:49.970467+0000) 2019-10-13T05:05:51.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:51.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:51.783610+0000) 2019-10-13T05:05:52.218 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:52.214+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.215402+0000) 2019-10-13T05:05:52.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:52.209+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.211158+0000) 2019-10-13T05:05:52.220 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:52.214+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.216015+0000) 2019-10-13T05:05:52.220 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:52.214+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.216168+0000) 2019-10-13T05:05:52.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:52.215+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.216403+0000) 2019-10-13T05:05:52.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:52.215+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.216770+0000) 2019-10-13T05:05:52.221 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:52.215+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.216864+0000) 2019-10-13T05:05:52.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:52.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69929 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:52.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:52.211+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.213628+0000) 2019-10-13T05:05:52.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:52.654+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.655366+0000) 2019-10-13T05:05:52.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:52.654+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:52.655789+0000) 2019-10-13T05:05:57.225 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:05:57.214+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.215649+0000) 2019-10-13T05:05:57.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:57.216+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.217003+0000) 2019-10-13T05:05:57.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:57.217+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.218925+0000) 2019-10-13T05:05:57.226 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:05:57.218+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.219697+0000) 2019-10-13T05:05:57.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:57.218+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.219924+0000) 2019-10-13T05:05:57.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:57.218+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.220189+0000) 2019-10-13T05:05:57.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:57.218+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.220341+0000) 2019-10-13T05:05:57.227 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:05:57.219+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.220521+0000) 2019-10-13T05:05:57.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:05:57.220+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:05:57.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:05:57.398+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:57.400020+0000) 2019-10-13T05:05:58.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:05:58.193+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:58.194633+0000) 2019-10-13T05:05:58.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:05:58.533+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:58.534042+0000) 2019-10-13T05:05:58.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:05:58.533+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:58.534052+0000) 2019-10-13T05:05:59.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:05:59.972+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:05:59.972800+0000) 2019-10-13T05:06:01.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:01.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:01.783800+0000) 2019-10-13T05:06:02.228 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:02.218+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.219155+0000) 2019-10-13T05:06:02.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:02.221+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.223233+0000) 2019-10-13T05:06:02.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:02.222+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.223972+0000) 2019-10-13T05:06:02.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:02.222+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.224142+0000) 2019-10-13T05:06:02.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:02.223+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.224579+0000) 2019-10-13T05:06:02.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:02.223+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.224570+0000) 2019-10-13T05:06:02.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:02.223+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.224789+0000) 2019-10-13T05:06:02.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:02.222+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.222858+0000) 2019-10-13T05:06:02.230 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:02.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 69990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:02.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:02.654+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.655531+0000) 2019-10-13T05:06:02.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:02.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:02.655947+0000) 2019-10-13T05:06:03.699 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:06:03.700 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:03.710 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:06:03.710 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:07.231 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:07.223+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.223989+0000) 2019-10-13T05:06:07.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:07.227+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.228513+0000) 2019-10-13T05:06:07.232 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:07.227+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.228694+0000) 2019-10-13T05:06:07.232 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:07.227+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.228965+0000) 2019-10-13T05:06:07.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:07.227+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.229187+0000) 2019-10-13T05:06:07.233 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:07.228+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.229486+0000) 2019-10-13T05:06:07.233 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:07.228+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.229599+0000) 2019-10-13T05:06:07.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:07.226+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.227049+0000) 2019-10-13T05:06:07.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:07.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:07.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:07.398+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:07.400204+0000) 2019-10-13T05:06:08.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:08.194+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:08.194794+0000) 2019-10-13T05:06:08.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:08.533+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:08.534209+0000) 2019-10-13T05:06:08.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:08.533+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:08.534175+0000) 2019-10-13T05:06:09.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:09.974+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:09.975067+0000) 2019-10-13T05:06:11.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:11.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:11.784039+0000) 2019-10-13T05:06:12.236 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:12.231+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.232921+0000) 2019-10-13T05:06:12.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:12.231+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.233106+0000) 2019-10-13T05:06:12.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:12.231+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.233302+0000) 2019-10-13T05:06:12.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:12.232+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.233544+0000) 2019-10-13T05:06:12.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:12.228+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.229414+0000) 2019-10-13T05:06:12.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:12.232+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.234043+0000) 2019-10-13T05:06:12.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:12.232+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.234347+0000) 2019-10-13T05:06:12.238 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:12.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:12.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:12.231+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.231933+0000) 2019-10-13T05:06:12.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:12.655+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.655698+0000) 2019-10-13T05:06:12.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:12.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:12.656104+0000) 2019-10-13T05:06:17.240 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:17.236+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.237369+0000) 2019-10-13T05:06:17.240 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:17.237+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.237647+0000) 2019-10-13T05:06:17.241 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:17.232+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.233564+0000) 2019-10-13T05:06:17.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:17.237+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.238059+0000) 2019-10-13T05:06:17.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:17.237+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.238311+0000) 2019-10-13T05:06:17.242 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:17.238+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.238510+0000) 2019-10-13T05:06:17.242 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:17.238+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.238758+0000) 2019-10-13T05:06:17.242 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:17.234+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.235405+0000) 2019-10-13T05:06:17.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:17.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:17.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:17.399+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:17.400356+0000) 2019-10-13T05:06:18.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:18.194+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:18.194963+0000) 2019-10-13T05:06:18.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:18.534+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:18.534367+0000) 2019-10-13T05:06:18.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:18.534+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:18.534366+0000) 2019-10-13T05:06:19.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:19.977+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:19.977422+0000) 2019-10-13T05:06:21.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:21.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:21.784234+0000) 2019-10-13T05:06:22.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:22.241+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.242158+0000) 2019-10-13T05:06:22.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:22.241+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.242327+0000) 2019-10-13T05:06:22.245 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:22.242+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.242513+0000) 2019-10-13T05:06:22.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:22.242+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.243054+0000) 2019-10-13T05:06:22.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:22.242+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.243228+0000) 2019-10-13T05:06:22.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:22.238+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.239165+0000) 2019-10-13T05:06:22.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:22.243+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.243897+0000) 2019-10-13T05:06:22.247 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:22.244+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:22.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:22.240+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.240876+0000) 2019-10-13T05:06:22.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:22.655+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.655883+0000) 2019-10-13T05:06:22.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:22.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:22.656280+0000) 2019-10-13T05:06:27.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:27.246+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.246454+0000) 2019-10-13T05:06:27.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:27.246+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.246665+0000) 2019-10-13T05:06:27.251 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:27.246+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.247216+0000) 2019-10-13T05:06:27.251 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:27.246+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.247357+0000) 2019-10-13T05:06:27.251 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:27.247+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.247562+0000) 2019-10-13T05:06:27.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:27.247+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.247785+0000) 2019-10-13T05:06:27.252 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:27.242+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.243500+0000) 2019-10-13T05:06:27.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:27.243+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.244548+0000) 2019-10-13T05:06:27.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:27.248+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:27.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:27.400+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:27.400516+0000) 2019-10-13T05:06:28.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:28.194+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:28.195075+0000) 2019-10-13T05:06:28.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:28.534+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:28.534583+0000) 2019-10-13T05:06:28.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:28.534+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:28.534579+0000) 2019-10-13T05:06:29.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:29.979+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:29.979793+0000) 2019-10-13T05:06:31.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:31.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:31.784426+0000) 2019-10-13T05:06:32.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:32.251+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.251592+0000) 2019-10-13T05:06:32.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:32.251+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.251800+0000) 2019-10-13T05:06:32.260 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:32.251+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.251924+0000) 2019-10-13T05:06:32.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:32.251+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.252108+0000) 2019-10-13T05:06:32.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:32.251+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.252245+0000) 2019-10-13T05:06:32.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:32.251+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.252376+0000) 2019-10-13T05:06:32.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:32.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70169 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:32.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:32.245+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.246726+0000) 2019-10-13T05:06:32.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:32.248+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.249899+0000) 2019-10-13T05:06:32.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:32.656+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.656046+0000) 2019-10-13T05:06:32.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:32.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:32.656430+0000) 2019-10-13T05:06:33.845 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:06:33.845 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:33.849 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:06:33.849 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:37.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:37.249+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.251422+0000) 2019-10-13T05:06:37.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:37.254+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.256221+0000) 2019-10-13T05:06:37.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:37.255+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.256717+0000) 2019-10-13T05:06:37.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:37.255+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.256689+0000) 2019-10-13T05:06:37.260 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:37.255+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.256962+0000) 2019-10-13T05:06:37.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:37.255+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.257117+0000) 2019-10-13T05:06:37.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:37.255+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.257268+0000) 2019-10-13T05:06:37.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:37.252+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.253899+0000) 2019-10-13T05:06:37.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:37.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:37.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:37.399+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:37.400682+0000) 2019-10-13T05:06:38.198 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:38.194+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:38.195268+0000) 2019-10-13T05:06:38.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:38.533+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:38.534770+0000) 2019-10-13T05:06:38.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:38.533+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:38.534776+0000) 2019-10-13T05:06:39.989 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:39.981+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:39.982257+0000) 2019-10-13T05:06:41.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:41.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:41.784612+0000) 2019-10-13T05:06:42.263 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:42.259+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.260496+0000) 2019-10-13T05:06:42.263 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:42.259+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.260898+0000) 2019-10-13T05:06:42.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:42.255+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.256869+0000) 2019-10-13T05:06:42.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:42.259+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.261201+0000) 2019-10-13T05:06:42.265 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:42.260+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.261439+0000) 2019-10-13T05:06:42.265 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:42.260+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.261780+0000) 2019-10-13T05:06:42.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:42.260+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.262026+0000) 2019-10-13T05:06:42.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:42.257+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.259105+0000) 2019-10-13T05:06:42.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:42.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:42.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:42.654+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.656208+0000) 2019-10-13T05:06:42.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:42.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:42.656569+0000) 2019-10-13T05:06:47.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:47.264+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.265474+0000) 2019-10-13T05:06:47.270 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:47.264+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.265897+0000) 2019-10-13T05:06:47.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:47.264+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.266085+0000) 2019-10-13T05:06:47.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:47.264+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.266204+0000) 2019-10-13T05:06:47.272 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:47.264+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.266342+0000) 2019-10-13T05:06:47.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:47.265+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.266528+0000) 2019-10-13T05:06:47.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:47.261+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.262419+0000) 2019-10-13T05:06:47.273 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:47.263+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.264442+0000) 2019-10-13T05:06:47.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:47.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:47.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:47.400+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:47.400792+0000) 2019-10-13T05:06:48.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:48.194+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:48.195459+0000) 2019-10-13T05:06:48.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:48.534+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:48.534992+0000) 2019-10-13T05:06:48.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:48.534+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:48.534990+0000) 2019-10-13T05:06:49.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:49.984+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:49.984708+0000) 2019-10-13T05:06:51.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:51.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:51.784782+0000) 2019-10-13T05:06:52.273 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:52.268+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.270241+0000) 2019-10-13T05:06:52.273 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:52.269+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.270431+0000) 2019-10-13T05:06:52.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:52.269+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.270670+0000) 2019-10-13T05:06:52.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:52.266+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.266884+0000) 2019-10-13T05:06:52.275 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:52.269+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.271023+0000) 2019-10-13T05:06:52.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:52.269+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.271056+0000) 2019-10-13T05:06:52.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:52.270+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.271549+0000) 2019-10-13T05:06:52.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:52.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70289 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:52.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:52.268+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.268983+0000) 2019-10-13T05:06:52.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:52.655+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.656370+0000) 2019-10-13T05:06:52.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:52.656+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:52.656761+0000) 2019-10-13T05:06:57.277 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:06:57.269+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.270049+0000) 2019-10-13T05:06:57.278 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:57.273+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.274431+0000) 2019-10-13T05:06:57.278 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:57.273+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.274577+0000) 2019-10-13T05:06:57.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:06:57.273+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.274799+0000) 2019-10-13T05:06:57.280 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:06:57.273+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.274890+0000) 2019-10-13T05:06:57.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:57.273+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.275067+0000) 2019-10-13T05:06:57.281 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:57.273+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.275167+0000) 2019-10-13T05:06:57.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:57.272+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.273146+0000) 2019-10-13T05:06:57.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:06:57.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:06:57.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:06:57.400+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:57.400956+0000) 2019-10-13T05:06:58.198 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:06:58.195+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:58.195651+0000) 2019-10-13T05:06:58.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:06:58.533+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:58.535213+0000) 2019-10-13T05:06:58.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:06:58.533+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:58.535216+0000) 2019-10-13T05:06:59.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:06:59.986+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:06:59.987050+0000) 2019-10-13T05:07:01.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:01.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:01.784958+0000) 2019-10-13T05:07:02.280 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:02.272+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.273475+0000) 2019-10-13T05:07:02.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:02.277+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.278154+0000) 2019-10-13T05:07:02.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:02.277+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.278328+0000) 2019-10-13T05:07:02.282 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:02.278+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.278595+0000) 2019-10-13T05:07:02.282 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:02.278+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.278729+0000) 2019-10-13T05:07:02.282 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:02.278+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.279057+0000) 2019-10-13T05:07:02.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:02.278+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.278961+0000) 2019-10-13T05:07:02.283 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:02.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70350 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:02.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:02.276+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.277282+0000) 2019-10-13T05:07:02.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:02.656+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.656551+0000) 2019-10-13T05:07:02.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:02.656+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:02.656896+0000) 2019-10-13T05:07:04.277 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:07:04.278 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:04.281 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:07:04.282 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:07.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:07.282+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.283162+0000) 2019-10-13T05:07:07.289 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:07.283+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.284117+0000) 2019-10-13T05:07:07.289 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:07.283+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.284376+0000) 2019-10-13T05:07:07.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:07.283+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.284259+0000) 2019-10-13T05:07:07.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:07.277+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.278626+0000) 2019-10-13T05:07:07.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:07.280+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.281595+0000) 2019-10-13T05:07:07.291 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:07.284+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.284528+0000) 2019-10-13T05:07:07.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:07.284+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.284803+0000) 2019-10-13T05:07:07.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:07.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70375 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:07.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:07.400+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:07.401138+0000) 2019-10-13T05:07:08.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:08.195+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:08.195836+0000) 2019-10-13T05:07:08.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:08.535+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:08.535428+0000) 2019-10-13T05:07:08.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:08.535+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:08.535425+0000) 2019-10-13T05:07:09.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:09.989+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:09.989489+0000) 2019-10-13T05:07:11.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:11.785+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:11.785178+0000) 2019-10-13T05:07:12.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:12.287+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.287626+0000) 2019-10-13T05:07:12.291 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:12.287+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.287768+0000) 2019-10-13T05:07:12.291 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:12.287+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.287994+0000) 2019-10-13T05:07:12.292 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:12.283+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.283873+0000) 2019-10-13T05:07:12.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:12.287+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.288363+0000) 2019-10-13T05:07:12.292 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:12.288+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.288591+0000) 2019-10-13T05:07:12.292 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:12.288+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.288790+0000) 2019-10-13T05:07:12.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:12.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70409 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:12.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:12.285+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.286497+0000) 2019-10-13T05:07:12.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:12.656+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.656743+0000) 2019-10-13T05:07:12.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:12.656+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:12.657034+0000) 2019-10-13T05:07:17.296 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:17.287+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.288217+0000) 2019-10-13T05:07:17.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:17.291+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.292013+0000) 2019-10-13T05:07:17.296 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:17.292+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.292440+0000) 2019-10-13T05:07:17.297 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:17.292+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.292723+0000) 2019-10-13T05:07:17.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:17.292+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.292940+0000) 2019-10-13T05:07:17.297 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:17.292+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.293072+0000) 2019-10-13T05:07:17.297 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:17.293+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.293440+0000) 2019-10-13T05:07:17.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:17.290+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.290945+0000) 2019-10-13T05:07:17.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:17.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:17.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:17.401+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:17.401312+0000) 2019-10-13T05:07:18.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:18.194+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:18.196007+0000) 2019-10-13T05:07:18.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:18.534+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:18.535625+0000) 2019-10-13T05:07:18.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:18.534+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:18.535643+0000) 2019-10-13T05:07:19.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:19.991+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:19.991847+0000) 2019-10-13T05:07:21.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:21.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:21.785372+0000) 2019-10-13T05:07:22.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:22.290+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.292353+0000) 2019-10-13T05:07:22.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:22.294+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.296217+0000) 2019-10-13T05:07:22.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:22.295+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.296488+0000) 2019-10-13T05:07:22.301 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:22.295+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.297022+0000) 2019-10-13T05:07:22.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:22.295+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.297279+0000) 2019-10-13T05:07:22.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:22.295+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.297380+0000) 2019-10-13T05:07:22.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:22.296+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.297749+0000) 2019-10-13T05:07:22.302 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:22.297+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70470 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:22.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:22.293+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.295398+0000) 2019-10-13T05:07:22.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:22.655+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.656849+0000) 2019-10-13T05:07:22.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:22.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:22.657189+0000) 2019-10-13T05:07:27.304 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:27.295+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.296783+0000) 2019-10-13T05:07:27.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:27.300+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.301439+0000) 2019-10-13T05:07:27.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:27.300+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.301606+0000) 2019-10-13T05:07:27.305 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:27.300+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.301801+0000) 2019-10-13T05:07:27.305 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:27.300+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.301954+0000) 2019-10-13T05:07:27.305 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:27.300+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.302325+0000) 2019-10-13T05:07:27.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:27.301+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.302535+0000) 2019-10-13T05:07:27.306 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:27.302+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70495 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:27.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:27.298+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.299852+0000) 2019-10-13T05:07:27.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:27.400+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:27.401476+0000) 2019-10-13T05:07:28.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:28.195+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:28.196177+0000) 2019-10-13T05:07:28.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:28.535+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:28.535849+0000) 2019-10-13T05:07:28.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:28.535+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:28.535856+0000) 2019-10-13T05:07:30.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:29.993+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:29.994219+0000) 2019-10-13T05:07:31.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:31.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:31.785546+0000) 2019-10-13T05:07:32.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:32.304+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.306076+0000) 2019-10-13T05:07:32.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:32.304+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.306207+0000) 2019-10-13T05:07:32.309 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:32.304+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.306352+0000) 2019-10-13T05:07:32.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:32.300+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.302174+0000) 2019-10-13T05:07:32.310 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:32.305+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.306864+0000) 2019-10-13T05:07:32.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:32.305+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.307139+0000) 2019-10-13T05:07:32.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:32.306+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.307475+0000) 2019-10-13T05:07:32.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:32.301+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.303636+0000) 2019-10-13T05:07:32.311 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:32.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70529 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:32.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:32.656+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.657020+0000) 2019-10-13T05:07:32.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:32.655+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:32.657311+0000) 2019-10-13T05:07:34.431 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:07:34.431 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:34.436 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:07:34.437 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:37.313 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:37.308+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.310323+0000) 2019-10-13T05:07:37.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:37.309+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.310442+0000) 2019-10-13T05:07:37.314 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:37.309+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.310531+0000) 2019-10-13T05:07:37.314 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:37.309+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.310961+0000) 2019-10-13T05:07:37.315 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:37.305+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.306336+0000) 2019-10-13T05:07:37.315 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:37.309+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.311370+0000) 2019-10-13T05:07:37.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:37.310+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.311614+0000) 2019-10-13T05:07:37.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:37.311+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:37.316 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:37.307+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.308641+0000) 2019-10-13T05:07:37.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:37.400+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:37.401579+0000) 2019-10-13T05:07:38.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:38.195+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:38.196332+0000) 2019-10-13T05:07:38.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:38.535+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:38.536047+0000) 2019-10-13T05:07:38.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:38.535+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:38.536045+0000) 2019-10-13T05:07:40.004 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:39.996+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:39.996773+0000) 2019-10-13T05:07:41.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:41.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:41.785721+0000) 2019-10-13T05:07:42.319 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:42.313+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.314646+0000) 2019-10-13T05:07:42.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:42.313+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.314889+0000) 2019-10-13T05:07:42.320 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:42.313+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.315002+0000) 2019-10-13T05:07:42.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:42.313+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.315330+0000) 2019-10-13T05:07:42.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:42.314+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.315446+0000) 2019-10-13T05:07:42.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:42.314+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.315769+0000) 2019-10-13T05:07:42.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:42.309+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.309810+0000) 2019-10-13T05:07:42.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:42.312+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.312847+0000) 2019-10-13T05:07:42.321 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:42.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:42.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:42.656+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.657155+0000) 2019-10-13T05:07:42.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:42.656+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:42.657470+0000) 2019-10-13T05:07:47.325 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:47.314+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.315066+0000) 2019-10-13T05:07:47.325 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:47.318+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.319238+0000) 2019-10-13T05:07:47.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:47.319+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.319860+0000) 2019-10-13T05:07:47.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:47.319+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.320014+0000) 2019-10-13T05:07:47.325 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:47.319+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.320242+0000) 2019-10-13T05:07:47.326 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:47.320+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.320470+0000) 2019-10-13T05:07:47.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:47.320+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.320631+0000) 2019-10-13T05:07:47.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:47.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:47.326 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:47.317+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.318054+0000) 2019-10-13T05:07:47.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:47.401+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:47.401768+0000) 2019-10-13T05:07:48.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:48.196+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:48.196484+0000) 2019-10-13T05:07:48.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:48.535+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:48.536203+0000) 2019-10-13T05:07:48.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:48.535+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:48.536205+0000) 2019-10-13T05:07:50.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:49.998+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:49.999289+0000) 2019-10-13T05:07:51.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:51.785+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:51.785905+0000) 2019-10-13T05:07:52.327 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:52.323+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.324208+0000) 2019-10-13T05:07:52.328 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:52.323+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.324335+0000) 2019-10-13T05:07:52.328 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:52.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.324573+0000) 2019-10-13T05:07:52.328 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:52.324+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.324787+0000) 2019-10-13T05:07:52.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:52.324+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.324875+0000) 2019-10-13T05:07:52.328 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:52.324+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.325028+0000) 2019-10-13T05:07:52.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:52.318+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.319482+0000) 2019-10-13T05:07:52.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:52.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70648 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:52.329 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:52.321+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.321817+0000) 2019-10-13T05:07:52.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:52.657+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.657322+0000) 2019-10-13T05:07:52.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:52.657+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:52.657630+0000) 2019-10-13T05:07:57.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:57.328+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.328418+0000) 2019-10-13T05:07:57.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:07:57.328+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.328836+0000) 2019-10-13T05:07:57.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:57.328+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.328984+0000) 2019-10-13T05:07:57.334 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:07:57.324+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.324836+0000) 2019-10-13T05:07:57.335 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:57.329+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.329493+0000) 2019-10-13T05:07:57.335 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:07:57.329+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.329692+0000) 2019-10-13T05:07:57.335 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:57.329+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.329855+0000) 2019-10-13T05:07:57.335 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:07:57.330+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:07:57.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:07:57.327+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.327971+0000) 2019-10-13T05:07:57.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:07:57.401+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:57.401953+0000) 2019-10-13T05:07:58.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:07:58.196+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:58.196615+0000) 2019-10-13T05:07:58.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:07:58.536+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:58.536311+0000) 2019-10-13T05:07:58.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:07:58.536+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:07:58.536360+0000) 2019-10-13T05:08:00.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:00.001+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:00.001727+0000) 2019-10-13T05:08:01.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:01.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:01.786099+0000) 2019-10-13T05:08:02.336 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:02.332+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.333275+0000) 2019-10-13T05:08:02.336 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:02.333+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.333453+0000) 2019-10-13T05:08:02.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:02.329+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.329841+0000) 2019-10-13T05:08:02.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:02.333+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.334006+0000) 2019-10-13T05:08:02.337 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:02.333+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.334071+0000) 2019-10-13T05:08:02.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:02.334+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.334485+0000) 2019-10-13T05:08:02.338 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:02.334+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.334626+0000) 2019-10-13T05:08:02.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:02.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:02.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:02.332+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.333338+0000) 2019-10-13T05:08:02.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:02.657+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.657538+0000) 2019-10-13T05:08:02.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:02.657+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:02.657803+0000) 2019-10-13T05:08:04.644 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:08:04.645 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:04.649 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:08:04.649 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:07.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:07.336+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.337412+0000) 2019-10-13T05:08:07.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:07.332+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.333359+0000) 2019-10-13T05:08:07.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:07.336+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.338145+0000) 2019-10-13T05:08:07.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:07.336+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.338308+0000) 2019-10-13T05:08:07.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:07.337+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.338511+0000) 2019-10-13T05:08:07.342 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:07.337+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.338617+0000) 2019-10-13T05:08:07.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:07.337+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.338787+0000) 2019-10-13T05:08:07.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:07.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:07.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:07.335+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.336637+0000) 2019-10-13T05:08:07.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:07.401+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:07.402086+0000) 2019-10-13T05:08:08.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:08.195+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:08.196788+0000) 2019-10-13T05:08:08.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:08.535+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:08.536456+0000) 2019-10-13T05:08:08.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:08.535+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:08.536516+0000) 2019-10-13T05:08:10.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:10.004+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:10.004155+0000) 2019-10-13T05:08:11.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:11.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:11.786306+0000) 2019-10-13T05:08:12.344 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:12.340+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.342073+0000) 2019-10-13T05:08:12.345 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:12.336+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.337861+0000) 2019-10-13T05:08:12.346 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:12.341+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.342405+0000) 2019-10-13T05:08:12.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:12.341+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.342584+0000) 2019-10-13T05:08:12.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:12.341+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.342897+0000) 2019-10-13T05:08:12.346 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:12.341+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.342946+0000) 2019-10-13T05:08:12.347 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:12.342+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.343601+0000) 2019-10-13T05:08:12.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:12.343+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70767 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:12.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:12.340+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.341793+0000) 2019-10-13T05:08:12.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:12.656+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.657668+0000) 2019-10-13T05:08:12.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:12.656+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:12.657954+0000) 2019-10-13T05:08:17.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:17.340+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.342390+0000) 2019-10-13T05:08:17.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:17.345+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.347195+0000) 2019-10-13T05:08:17.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:17.345+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.347382+0000) 2019-10-13T05:08:17.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:17.346+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.347579+0000) 2019-10-13T05:08:17.351 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:17.346+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.347769+0000) 2019-10-13T05:08:17.351 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:17.346+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.347970+0000) 2019-10-13T05:08:17.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:17.342+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.344589+0000) 2019-10-13T05:08:17.352 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:17.346+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.348157+0000) 2019-10-13T05:08:17.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:17.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:17.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:17.401+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:17.402216+0000) 2019-10-13T05:08:18.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:18.196+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:18.196924+0000) 2019-10-13T05:08:18.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:18.535+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:18.536636+0000) 2019-10-13T05:08:18.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:18.535+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:18.536678+0000) 2019-10-13T05:08:20.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:20.004+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:20.006433+0000) 2019-10-13T05:08:21.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:21.785+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:21.786519+0000) 2019-10-13T05:08:22.355 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:22.350+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.351533+0000) 2019-10-13T05:08:22.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:22.350+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.351714+0000) 2019-10-13T05:08:22.356 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:22.345+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.347675+0000) 2019-10-13T05:08:22.356 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:22.350+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.352132+0000) 2019-10-13T05:08:22.356 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:22.350+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.352369+0000) 2019-10-13T05:08:22.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:22.351+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.352891+0000) 2019-10-13T05:08:22.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:22.351+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.353000+0000) 2019-10-13T05:08:22.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:22.348+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.350231+0000) 2019-10-13T05:08:22.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:22.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:22.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:22.657+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.657818+0000) 2019-10-13T05:08:22.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:22.657+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:22.658081+0000) 2019-10-13T05:08:27.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:27.351+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.351725+0000) 2019-10-13T05:08:27.359 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:27.354+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.356230+0000) 2019-10-13T05:08:27.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:27.355+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.356725+0000) 2019-10-13T05:08:27.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:27.355+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.356885+0000) 2019-10-13T05:08:27.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:27.355+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.357211+0000) 2019-10-13T05:08:27.360 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:27.356+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.357420+0000) 2019-10-13T05:08:27.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:27.356+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.357560+0000) 2019-10-13T05:08:27.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:27.353+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.353956+0000) 2019-10-13T05:08:27.363 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:27.357+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:27.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:27.400+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:27.402372+0000) 2019-10-13T05:08:28.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:28.195+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:28.197102+0000) 2019-10-13T05:08:28.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:28.536+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:28.536818+0000) 2019-10-13T05:08:28.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:28.536+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:28.536841+0000) 2019-10-13T05:08:30.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:30.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:30.008884+0000) 2019-10-13T05:08:31.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:31.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:31.786727+0000) 2019-10-13T05:08:32.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:32.360+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.361070+0000) 2019-10-13T05:08:32.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:32.360+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.361384+0000) 2019-10-13T05:08:32.365 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:32.361+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.361500+0000) 2019-10-13T05:08:32.366 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:32.361+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.361644+0000) 2019-10-13T05:08:32.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:32.361+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.361840+0000) 2019-10-13T05:08:32.366 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:32.361+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.362093+0000) 2019-10-13T05:08:32.366 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:32.355+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.356575+0000) 2019-10-13T05:08:32.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:32.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:32.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:32.357+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.358593+0000) 2019-10-13T05:08:32.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:32.657+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.658254+0000) 2019-10-13T05:08:32.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:32.657+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:32.657990+0000) 2019-10-13T05:08:34.826 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:08:34.826 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:34.830 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:08:34.831 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:37.368 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:37.364+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.364817+0000) 2019-10-13T05:08:37.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:37.359+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.360637+0000) 2019-10-13T05:08:37.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:37.364+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.365383+0000) 2019-10-13T05:08:37.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:37.365+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.365792+0000) 2019-10-13T05:08:37.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:37.365+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.366093+0000) 2019-10-13T05:08:37.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:37.365+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.366309+0000) 2019-10-13T05:08:37.370 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:37.366+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.366393+0000) 2019-10-13T05:08:37.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:37.362+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.362974+0000) 2019-10-13T05:08:37.370 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:37.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:37.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:37.402+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:37.402515+0000) 2019-10-13T05:08:38.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:38.196+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:38.197293+0000) 2019-10-13T05:08:38.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:38.536+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:38.537033+0000) 2019-10-13T05:08:38.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:38.536+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:38.537032+0000) 2019-10-13T05:08:40.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:40.011+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:40.012142+0000) 2019-10-13T05:08:41.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:41.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:41.786920+0000) 2019-10-13T05:08:42.374 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:42.369+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.369524+0000) 2019-10-13T05:08:42.375 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:42.369+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.369708+0000) 2019-10-13T05:08:42.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:42.369+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.370138+0000) 2019-10-13T05:08:42.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:42.369+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.370215+0000) 2019-10-13T05:08:42.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:42.370+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.370922+0000) 2019-10-13T05:08:42.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:42.370+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.371319+0000) 2019-10-13T05:08:42.376 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:42.365+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.366290+0000) 2019-10-13T05:08:42.376 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:42.366+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.367629+0000) 2019-10-13T05:08:42.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:42.372+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:42.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:42.657+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.658161+0000) 2019-10-13T05:08:42.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:42.658+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:42.658415+0000) 2019-10-13T05:08:47.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:47.374+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.374392+0000) 2019-10-13T05:08:47.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:47.369+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.370532+0000) 2019-10-13T05:08:47.379 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:47.374+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.375182+0000) 2019-10-13T05:08:47.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:47.375+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.375503+0000) 2019-10-13T05:08:47.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:47.375+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.375790+0000) 2019-10-13T05:08:47.380 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:47.375+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.376101+0000) 2019-10-13T05:08:47.380 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:47.375+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.376201+0000) 2019-10-13T05:08:47.381 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:47.372+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.372771+0000) 2019-10-13T05:08:47.381 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:47.377+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 70969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:47.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:47.402+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:47.402646+0000) 2019-10-13T05:08:48.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:48.197+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:48.197437+0000) 2019-10-13T05:08:48.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:48.537+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:48.537204+0000) 2019-10-13T05:08:48.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:48.537+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:48.537181+0000) 2019-10-13T05:08:50.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:50.014+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:50.014769+0000) 2019-10-13T05:08:51.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:51.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:51.787108+0000) 2019-10-13T05:08:52.382 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:52.378+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.379639+0000) 2019-10-13T05:08:52.383 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:52.378+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.380029+0000) 2019-10-13T05:08:52.383 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:52.375+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.375954+0000) 2019-10-13T05:08:52.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:52.378+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.380259+0000) 2019-10-13T05:08:52.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:52.379+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.380523+0000) 2019-10-13T05:08:52.384 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:52.379+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.380785+0000) 2019-10-13T05:08:52.384 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:52.379+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.381160+0000) 2019-10-13T05:08:52.385 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:52.376+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.377516+0000) 2019-10-13T05:08:52.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:52.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71003 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:52.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:52.657+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.658324+0000) 2019-10-13T05:08:52.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:52.658+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:52.658572+0000) 2019-10-13T05:08:57.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:57.383+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.384496+0000) 2019-10-13T05:08:57.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:08:57.383+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.384744+0000) 2019-10-13T05:08:57.388 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:08:57.383+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.384896+0000) 2019-10-13T05:08:57.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:57.383+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.385085+0000) 2019-10-13T05:08:57.388 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:57.383+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.385218+0000) 2019-10-13T05:08:57.389 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:57.384+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.385393+0000) 2019-10-13T05:08:57.389 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:08:57.380+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.381287+0000) 2019-10-13T05:08:57.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:08:57.380+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.381695+0000) 2019-10-13T05:08:57.390 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:08:57.386+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:08:57.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:08:57.401+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:57.402771+0000) 2019-10-13T05:08:58.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:08:58.196+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:58.197584+0000) 2019-10-13T05:08:58.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:08:58.536+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:58.537297+0000) 2019-10-13T05:08:58.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:08:58.536+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:08:58.537349+0000) 2019-10-13T05:09:00.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:00.017+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:00.017286+0000) 2019-10-13T05:09:01.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:01.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:01.787296+0000) 2019-10-13T05:09:02.391 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:02.387+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.389014+0000) 2019-10-13T05:09:02.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:02.383+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.385120+0000) 2019-10-13T05:09:02.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:02.388+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.389601+0000) 2019-10-13T05:09:02.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:02.388+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.389744+0000) 2019-10-13T05:09:02.393 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:02.388+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.389921+0000) 2019-10-13T05:09:02.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:02.388+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.390083+0000) 2019-10-13T05:09:02.394 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:02.389+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.390467+0000) 2019-10-13T05:09:02.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:02.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71061 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:02.395 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:02.386+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.387928+0000) 2019-10-13T05:09:02.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:02.657+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.658488+0000) 2019-10-13T05:09:02.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:02.657+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:02.658669+0000) 2019-10-13T05:09:04.948 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:09:04.948 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:04.953 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:09:04.953 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:07.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:07.391+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.393373+0000) 2019-10-13T05:09:07.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:07.392+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.393692+0000) 2019-10-13T05:09:07.397 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:07.387+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.389486+0000) 2019-10-13T05:09:07.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:07.392+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.394360+0000) 2019-10-13T05:09:07.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:07.392+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.394038+0000) 2019-10-13T05:09:07.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:07.393+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.394781+0000) 2019-10-13T05:09:07.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:07.393+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.395135+0000) 2019-10-13T05:09:07.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:07.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71087 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:07.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:07.390+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.392201+0000) 2019-10-13T05:09:07.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:07.401+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:07.402898+0000) 2019-10-13T05:09:08.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:08.197+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:08.197717+0000) 2019-10-13T05:09:08.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:08.536+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:08.537411+0000) 2019-10-13T05:09:08.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:08.536+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:08.537516+0000) 2019-10-13T05:09:10.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:10.018+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:10.020410+0000) 2019-10-13T05:09:11.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:11.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:11.787503+0000) 2019-10-13T05:09:12.401 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:12.396+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.398154+0000) 2019-10-13T05:09:12.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:12.396+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.398376+0000) 2019-10-13T05:09:12.402 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:12.392+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.394619+0000) 2019-10-13T05:09:12.402 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:12.397+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.398745+0000) 2019-10-13T05:09:12.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:12.397+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.398985+0000) 2019-10-13T05:09:12.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:12.395+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.397076+0000) 2019-10-13T05:09:12.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:12.397+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.399113+0000) 2019-10-13T05:09:12.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:12.398+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.399564+0000) 2019-10-13T05:09:12.406 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:12.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71121 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:12.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:12.658+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.658616+0000) 2019-10-13T05:09:12.666 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:12.657+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:12.658823+0000) 2019-10-13T05:09:17.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:17.402+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.402816+0000) 2019-10-13T05:09:17.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:17.402+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.403053+0000) 2019-10-13T05:09:17.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:17.402+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.403162+0000) 2019-10-13T05:09:17.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:17.403+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.403413+0000) 2019-10-13T05:09:17.407 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:17.399+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.399741+0000) 2019-10-13T05:09:17.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:17.403+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.403894+0000) 2019-10-13T05:09:17.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:17.404+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.404632+0000) 2019-10-13T05:09:17.408 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:17.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:17.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:17.400+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:17.401418+0000) 2019-10-13T05:09:18.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:18.197+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:18.197862+0000) 2019-10-13T05:09:18.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:18.537+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:18.537578+0000) 2019-10-13T05:09:18.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:18.537+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:18.537651+0000) 2019-10-13T05:09:20.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:20.022+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:20.023440+0000) 2019-10-13T05:09:21.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:21.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:21.787686+0000) 2019-10-13T05:09:22.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:22.407+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.407426+0000) 2019-10-13T05:09:22.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:22.407+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.407703+0000) 2019-10-13T05:09:22.411 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:22.407+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.407910+0000) 2019-10-13T05:09:22.411 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:22.407+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.408294+0000) 2019-10-13T05:09:22.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:22.407+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.408363+0000) 2019-10-13T05:09:22.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:22.408+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.408537+0000) 2019-10-13T05:09:22.412 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:22.403+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.404541+0000) 2019-10-13T05:09:22.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:22.405+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.406468+0000) 2019-10-13T05:09:22.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:22.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:22.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:22.658+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.658760+0000) 2019-10-13T05:09:22.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:22.658+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:22.658944+0000) 2019-10-13T05:09:24.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:24.153+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:24.153241+0000) 2019-10-13T05:09:27.416 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:27.412+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.412611+0000) 2019-10-13T05:09:27.416 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:27.412+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.412852+0000) 2019-10-13T05:09:27.417 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:27.412+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.413247+0000) 2019-10-13T05:09:27.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:27.413+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.413441+0000) 2019-10-13T05:09:27.417 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:27.413+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.413758+0000) 2019-10-13T05:09:27.418 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:27.408+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.408805+0000) 2019-10-13T05:09:27.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:27.413+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.414365+0000) 2019-10-13T05:09:27.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:27.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:27.419 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:27.410+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:27.411468+0000) 2019-10-13T05:09:28.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:28.197+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:28.198029+0000) 2019-10-13T05:09:28.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:28.537+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:28.537771+0000) 2019-10-13T05:09:28.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:28.537+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:28.537861+0000) 2019-10-13T05:09:30.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:30.025+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:30.025718+0000) 2019-10-13T05:09:31.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:31.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:31.787895+0000) 2019-10-13T05:09:32.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:32.412+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.412977+0000) 2019-10-13T05:09:32.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:32.417+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.417617+0000) 2019-10-13T05:09:32.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:32.417+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.417871+0000) 2019-10-13T05:09:32.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:32.417+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.418173+0000) 2019-10-13T05:09:32.422 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:32.417+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.418378+0000) 2019-10-13T05:09:32.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:32.418+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.418468+0000) 2019-10-13T05:09:32.422 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:32.418+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.418627+0000) 2019-10-13T05:09:32.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:32.414+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.414966+0000) 2019-10-13T05:09:32.423 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:32.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:32.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:32.658+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.658905+0000) 2019-10-13T05:09:32.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:32.658+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:32.659101+0000) 2019-10-13T05:09:34.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:34.152+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:34.153394+0000) 2019-10-13T05:09:35.129 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:09:35.129 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:35.133 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:09:35.134 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:37.424 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:37.416+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.417568+0000) 2019-10-13T05:09:37.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:37.420+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.422227+0000) 2019-10-13T05:09:37.426 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:37.421+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.422515+0000) 2019-10-13T05:09:37.426 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:37.421+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.422512+0000) 2019-10-13T05:09:37.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:37.421+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.422797+0000) 2019-10-13T05:09:37.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:37.421+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.422983+0000) 2019-10-13T05:09:37.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:37.421+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.423062+0000) 2019-10-13T05:09:37.428 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:37.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:37.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:37.420+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:37.421014+0000) 2019-10-13T05:09:38.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:38.197+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:38.198199+0000) 2019-10-13T05:09:38.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:38.537+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:38.537951+0000) 2019-10-13T05:09:38.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:38.537+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:38.538018+0000) 2019-10-13T05:09:40.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:40.027+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:40.028103+0000) 2019-10-13T05:09:41.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:41.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:41.788093+0000) 2019-10-13T05:09:42.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:42.422+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.423676+0000) 2019-10-13T05:09:42.432 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:42.425+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.426814+0000) 2019-10-13T05:09:42.433 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:42.425+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.427082+0000) 2019-10-13T05:09:42.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:42.425+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.427355+0000) 2019-10-13T05:09:42.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:42.426+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.427482+0000) 2019-10-13T05:09:42.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:42.424+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.424939+0000) 2019-10-13T05:09:42.441 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:42.426+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.427680+0000) 2019-10-13T05:09:42.441 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:42.426+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.427916+0000) 2019-10-13T05:09:42.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:42.428+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:42.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:42.658+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.659053+0000) 2019-10-13T05:09:42.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:42.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:42.659276+0000) 2019-10-13T05:09:44.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:44.152+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:44.153597+0000) 2019-10-13T05:09:47.434 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:47.429+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.431286+0000) 2019-10-13T05:09:47.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:47.426+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.427442+0000) 2019-10-13T05:09:47.435 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:47.430+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.431729+0000) 2019-10-13T05:09:47.435 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:47.430+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.432031+0000) 2019-10-13T05:09:47.435 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:47.430+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.432270+0000) 2019-10-13T05:09:47.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:47.431+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.432430+0000) 2019-10-13T05:09:47.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:47.431+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.432581+0000) 2019-10-13T05:09:47.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:47.429+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:47.430483+0000) 2019-10-13T05:09:47.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:47.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71323 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:48.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:48.197+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:48.198385+0000) 2019-10-13T05:09:48.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:48.537+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:48.538121+0000) 2019-10-13T05:09:48.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:48.537+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:48.538213+0000) 2019-10-13T05:09:50.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:50.030+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:50.030397+0000) 2019-10-13T05:09:51.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:51.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:51.788280+0000) 2019-10-13T05:09:52.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:52.434+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.435979+0000) 2019-10-13T05:09:52.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:52.435+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.436455+0000) 2019-10-13T05:09:52.440 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:52.435+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.436658+0000) 2019-10-13T05:09:52.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:52.435+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.436716+0000) 2019-10-13T05:09:52.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:52.435+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.437025+0000) 2019-10-13T05:09:52.441 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:52.431+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.432836+0000) 2019-10-13T05:09:52.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:52.432+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.433826+0000) 2019-10-13T05:09:52.441 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:52.436+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.437456+0000) 2019-10-13T05:09:52.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:52.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:52.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:52.658+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.659229+0000) 2019-10-13T05:09:52.666 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:52.658+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:52.659423+0000) 2019-10-13T05:09:54.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:54.153+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:54.153788+0000) 2019-10-13T05:09:57.443 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:09:57.439+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.440448+0000) 2019-10-13T05:09:57.444 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:09:57.434+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.436238+0000) 2019-10-13T05:09:57.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:09:57.439+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.441121+0000) 2019-10-13T05:09:57.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:57.439+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.441163+0000) 2019-10-13T05:09:57.444 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:57.439+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.441115+0000) 2019-10-13T05:09:57.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:09:57.440+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.441670+0000) 2019-10-13T05:09:57.445 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:57.440+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.441884+0000) 2019-10-13T05:09:57.446 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:09:57.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71383 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:09:57.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:09:57.438+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:57.439863+0000) 2019-10-13T05:09:58.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:09:58.198+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:58.198570+0000) 2019-10-13T05:09:58.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:09:58.537+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:58.538275+0000) 2019-10-13T05:09:58.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:09:58.537+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:09:58.538353+0000) 2019-10-13T05:10:00.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:00.032+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:00.032995+0000) 2019-10-13T05:10:01.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:01.788+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:01.788491+0000) 2019-10-13T05:10:02.448 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:02.444+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.445050+0000) 2019-10-13T05:10:02.449 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:02.444+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.445308+0000) 2019-10-13T05:10:02.449 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:02.445+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.445591+0000) 2019-10-13T05:10:02.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:02.445+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.445864+0000) 2019-10-13T05:10:02.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:02.445+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.445961+0000) 2019-10-13T05:10:02.450 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:02.438+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.440512+0000) 2019-10-13T05:10:02.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:02.441+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.442823+0000) 2019-10-13T05:10:02.450 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:02.445+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.446177+0000) 2019-10-13T05:10:02.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:02.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:02.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:02.658+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.659380+0000) 2019-10-13T05:10:02.666 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:02.658+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:02.659583+0000) 2019-10-13T05:10:04.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:04.153+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:04.153981+0000) 2019-10-13T05:10:05.210 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:10:05.210 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:05.215 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:10:05.215 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:07.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:07.444+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.445147+0000) 2019-10-13T05:10:07.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:07.447+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.447970+0000) 2019-10-13T05:10:07.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:07.448+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.449172+0000) 2019-10-13T05:10:07.458 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:07.449+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.449608+0000) 2019-10-13T05:10:07.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:07.449+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.449825+0000) 2019-10-13T05:10:07.459 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:07.449+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.450113+0000) 2019-10-13T05:10:07.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:07.450+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.450451+0000) 2019-10-13T05:10:07.460 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:07.450+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:07.450483+0000) 2019-10-13T05:10:07.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:07.451+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:08.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:08.198+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:08.198730+0000) 2019-10-13T05:10:08.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:08.538+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:08.538373+0000) 2019-10-13T05:10:08.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:08.538+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:08.538450+0000) 2019-10-13T05:10:10.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:10.034+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:10.035443+0000) 2019-10-13T05:10:11.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:11.788+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:11.788708+0000) 2019-10-13T05:10:12.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:12.453+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.453858+0000) 2019-10-13T05:10:12.457 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:12.453+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.454258+0000) 2019-10-13T05:10:12.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:12.454+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.454441+0000) 2019-10-13T05:10:12.458 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:12.453+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.454382+0000) 2019-10-13T05:10:12.458 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:12.454+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.454703+0000) 2019-10-13T05:10:12.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:12.449+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.450616+0000) 2019-10-13T05:10:12.459 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:12.454+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.455070+0000) 2019-10-13T05:10:12.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:12.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71475 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:12.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:12.451+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.452704+0000) 2019-10-13T05:10:12.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:12.659+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.659471+0000) 2019-10-13T05:10:12.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:12.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:12.659761+0000) 2019-10-13T05:10:14.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:14.153+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:14.154180+0000) 2019-10-13T05:10:17.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:17.457+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.458291+0000) 2019-10-13T05:10:17.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:17.458+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.458456+0000) 2019-10-13T05:10:17.461 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:17.458+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.458651+0000) 2019-10-13T05:10:17.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:17.453+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.454372+0000) 2019-10-13T05:10:17.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:17.458+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.458885+0000) 2019-10-13T05:10:17.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:17.459+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.459522+0000) 2019-10-13T05:10:17.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:17.459+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.459691+0000) 2019-10-13T05:10:17.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:17.456+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:17.456906+0000) 2019-10-13T05:10:17.464 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:17.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71501 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:18.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:18.198+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:18.198905+0000) 2019-10-13T05:10:18.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:18.538+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:18.538605+0000) 2019-10-13T05:10:18.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:18.538+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:18.538576+0000) 2019-10-13T05:10:20.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:20.037+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:20.037741+0000) 2019-10-13T05:10:21.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:21.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:21.788904+0000) 2019-10-13T05:10:22.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:22.457+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.458167+0000) 2019-10-13T05:10:22.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:22.461+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.462733+0000) 2019-10-13T05:10:22.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:22.461+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.463160+0000) 2019-10-13T05:10:22.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:22.462+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.463536+0000) 2019-10-13T05:10:22.467 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:22.462+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.463417+0000) 2019-10-13T05:10:22.467 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:22.462+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.463656+0000) 2019-10-13T05:10:22.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:22.462+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.463802+0000) 2019-10-13T05:10:22.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:22.459+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.460718+0000) 2019-10-13T05:10:22.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:22.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71533 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:22.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:22.658+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.659614+0000) 2019-10-13T05:10:22.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:22.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:22.659871+0000) 2019-10-13T05:10:24.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:24.153+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:24.154333+0000) 2019-10-13T05:10:27.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:27.465+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.467096+0000) 2019-10-13T05:10:27.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:27.466+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.467452+0000) 2019-10-13T05:10:27.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:27.463+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.463793+0000) 2019-10-13T05:10:27.471 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:27.466+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.467665+0000) 2019-10-13T05:10:27.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:27.467+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.468403+0000) 2019-10-13T05:10:27.472 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:27.467+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.468493+0000) 2019-10-13T05:10:27.472 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:27.467+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.468658+0000) 2019-10-13T05:10:27.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:27.464+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:27.465160+0000) 2019-10-13T05:10:27.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:27.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71559 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:28.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:28.198+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:28.199071+0000) 2019-10-13T05:10:28.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:28.537+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:28.538789+0000) 2019-10-13T05:10:28.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:28.537+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:28.538780+0000) 2019-10-13T05:10:30.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:30.039+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:30.039974+0000) 2019-10-13T05:10:31.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:31.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:31.789096+0000) 2019-10-13T05:10:32.476 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:32.468+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.468928+0000) 2019-10-13T05:10:32.477 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:32.472+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.473488+0000) 2019-10-13T05:10:32.479 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:32.472+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.473675+0000) 2019-10-13T05:10:32.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:32.472+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.473841+0000) 2019-10-13T05:10:32.480 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:32.472+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.474239+0000) 2019-10-13T05:10:32.480 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:32.473+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.474442+0000) 2019-10-13T05:10:32.480 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:32.473+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.474720+0000) 2019-10-13T05:10:32.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:32.471+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.472341+0000) 2019-10-13T05:10:32.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:32.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71593 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:32.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:32.659+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.659755+0000) 2019-10-13T05:10:32.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:32.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:32.659966+0000) 2019-10-13T05:10:34.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:34.153+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:34.154493+0000) 2019-10-13T05:10:35.450 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:10:35.450 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:35.458 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:10:35.458 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:37.480 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:37.471+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.472717+0000) 2019-10-13T05:10:37.480 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:37.476+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.477617+0000) 2019-10-13T05:10:37.481 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:37.476+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.477668+0000) 2019-10-13T05:10:37.481 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:37.476+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.477836+0000) 2019-10-13T05:10:37.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:37.476+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.478003+0000) 2019-10-13T05:10:37.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:37.476+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.478370+0000) 2019-10-13T05:10:37.482 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:37.477+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.478505+0000) 2019-10-13T05:10:37.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:37.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:37.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:37.475+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:37.475993+0000) 2019-10-13T05:10:38.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:38.197+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:38.199204+0000) 2019-10-13T05:10:38.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:38.538+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:38.538903+0000) 2019-10-13T05:10:38.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:38.538+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:38.538945+0000) 2019-10-13T05:10:40.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:40.042+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:40.042297+0000) 2019-10-13T05:10:41.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:41.788+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:41.789251+0000) 2019-10-13T05:10:42.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:42.480+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.482019+0000) 2019-10-13T05:10:42.486 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:42.480+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.482216+0000) 2019-10-13T05:10:42.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:42.481+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.482637+0000) 2019-10-13T05:10:42.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:42.481+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.482500+0000) 2019-10-13T05:10:42.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:42.481+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.483012+0000) 2019-10-13T05:10:42.488 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:42.475+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.477520+0000) 2019-10-13T05:10:42.488 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:42.482+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.483428+0000) 2019-10-13T05:10:42.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:42.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:42.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:42.478+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.480708+0000) 2019-10-13T05:10:42.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:42.659+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.659912+0000) 2019-10-13T05:10:42.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:42.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:42.660102+0000) 2019-10-13T05:10:44.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:44.154+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:44.154679+0000) 2019-10-13T05:10:47.489 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:47.485+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.486163+0000) 2019-10-13T05:10:47.489 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:47.480+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.481907+0000) 2019-10-13T05:10:47.490 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:47.486+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.486665+0000) 2019-10-13T05:10:47.490 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:47.486+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.486869+0000) 2019-10-13T05:10:47.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:47.486+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.487198+0000) 2019-10-13T05:10:47.491 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:47.486+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.487151+0000) 2019-10-13T05:10:47.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:47.487+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.487437+0000) 2019-10-13T05:10:47.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:47.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71677 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:47.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:47.483+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:47.484777+0000) 2019-10-13T05:10:48.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:48.198+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:48.199320+0000) 2019-10-13T05:10:48.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:48.538+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:48.539097+0000) 2019-10-13T05:10:48.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:48.538+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:48.539184+0000) 2019-10-13T05:10:50.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:50.043+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:50.044666+0000) 2019-10-13T05:10:51.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:51.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:51.789429+0000) 2019-10-13T05:10:52.493 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:52.490+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.490898+0000) 2019-10-13T05:10:52.495 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:52.485+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.486892+0000) 2019-10-13T05:10:52.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:52.491+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.491479+0000) 2019-10-13T05:10:52.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:52.491+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.491654+0000) 2019-10-13T05:10:52.495 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:52.491+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.491857+0000) 2019-10-13T05:10:52.495 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:52.491+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.492043+0000) 2019-10-13T05:10:52.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:52.491+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.492264+0000) 2019-10-13T05:10:52.496 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:52.486+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.488298+0000) 2019-10-13T05:10:52.499 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:52.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71711 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:52.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:52.659+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.660096+0000) 2019-10-13T05:10:52.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:52.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:52.660279+0000) 2019-10-13T05:10:54.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:54.154+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:54.154861+0000) 2019-10-13T05:10:57.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:10:57.495+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.496048+0000) 2019-10-13T05:10:57.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:10:57.495+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.496133+0000) 2019-10-13T05:10:57.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:57.495+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.496373+0000) 2019-10-13T05:10:57.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:57.495+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.496292+0000) 2019-10-13T05:10:57.500 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:10:57.491+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.492463+0000) 2019-10-13T05:10:57.501 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:57.496+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.496916+0000) 2019-10-13T05:10:57.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:10:57.496+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.497079+0000) 2019-10-13T05:10:57.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:10:57.493+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:57.494460+0000) 2019-10-13T05:10:57.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:10:57.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:10:58.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:10:58.199+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:58.199493+0000) 2019-10-13T05:10:58.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:10:58.539+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:58.539293+0000) 2019-10-13T05:10:58.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:10:58.539+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:10:58.539377+0000) 2019-10-13T05:11:00.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:00.046+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:00.046946+0000) 2019-10-13T05:11:01.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:01.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:01.789632+0000) 2019-10-13T05:11:02.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:02.500+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.501175+0000) 2019-10-13T05:11:02.504 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:02.501+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.501516+0000) 2019-10-13T05:11:02.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:02.501+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.501759+0000) 2019-10-13T05:11:02.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:02.501+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.501951+0000) 2019-10-13T05:11:02.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:02.497+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.497815+0000) 2019-10-13T05:11:02.507 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:02.501+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.502288+0000) 2019-10-13T05:11:02.507 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:02.501+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.502372+0000) 2019-10-13T05:11:02.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:02.498+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.499443+0000) 2019-10-13T05:11:02.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:02.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:02.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:02.659+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.660288+0000) 2019-10-13T05:11:02.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:02.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:02.660434+0000) 2019-10-13T05:11:04.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:04.155+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:04.155052+0000) 2019-10-13T05:11:05.638 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:11:05.639 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:05.643 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:11:05.643 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:07.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:07.505+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.506465+0000) 2019-10-13T05:11:07.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:07.505+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.506582+0000) 2019-10-13T05:11:07.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:07.501+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.502315+0000) 2019-10-13T05:11:07.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:07.505+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.507134+0000) 2019-10-13T05:11:07.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:07.506+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.507481+0000) 2019-10-13T05:11:07.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:07.503+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.504007+0000) 2019-10-13T05:11:07.512 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:07.506+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.507534+0000) 2019-10-13T05:11:07.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:07.506+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:07.507953+0000) 2019-10-13T05:11:07.512 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:07.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:08.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:08.198+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:08.199640+0000) 2019-10-13T05:11:08.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:08.538+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:08.539465+0000) 2019-10-13T05:11:08.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:08.538+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:08.539537+0000) 2019-10-13T05:11:10.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:10.048+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:10.049478+0000) 2019-10-13T05:11:11.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:11.788+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:11.789819+0000) 2019-10-13T05:11:12.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:12.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.509619+0000) 2019-10-13T05:11:12.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:12.508+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.509863+0000) 2019-10-13T05:11:12.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:12.508+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.510014+0000) 2019-10-13T05:11:12.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:12.508+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.510217+0000) 2019-10-13T05:11:12.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:12.504+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.505335+0000) 2019-10-13T05:11:12.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:12.508+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.508818+0000) 2019-10-13T05:11:12.524 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:12.509+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.510453+0000) 2019-10-13T05:11:12.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:12.509+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.510667+0000) 2019-10-13T05:11:12.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:12.510+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:12.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:12.659+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.660449+0000) 2019-10-13T05:11:12.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:12.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:12.660588+0000) 2019-10-13T05:11:14.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:14.154+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:14.155237+0000) 2019-10-13T05:11:17.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:17.512+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.513823+0000) 2019-10-13T05:11:17.518 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:17.512+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.514071+0000) 2019-10-13T05:11:17.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:17.512+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.514290+0000) 2019-10-13T05:11:17.518 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:17.513+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.514407+0000) 2019-10-13T05:11:17.518 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:17.513+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.514920+0000) 2019-10-13T05:11:17.519 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:17.509+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.510287+0000) 2019-10-13T05:11:17.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:17.511+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.512259+0000) 2019-10-13T05:11:17.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:17.513+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:17.515171+0000) 2019-10-13T05:11:17.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:17.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71855 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:18.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:18.199+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:18.199791+0000) 2019-10-13T05:11:18.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:18.538+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:18.539594+0000) 2019-10-13T05:11:18.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:18.538+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:18.539659+0000) 2019-10-13T05:11:20.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:20.051+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:20.051931+0000) 2019-10-13T05:11:21.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:21.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:21.789974+0000) 2019-10-13T05:11:22.520 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:22.516+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.517921+0000) 2019-10-13T05:11:22.521 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:22.513+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.513976+0000) 2019-10-13T05:11:22.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:22.518+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.519503+0000) 2019-10-13T05:11:22.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:22.518+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.519696+0000) 2019-10-13T05:11:22.523 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:22.517+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.519298+0000) 2019-10-13T05:11:22.523 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:22.518+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.519864+0000) 2019-10-13T05:11:22.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:22.518+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.520120+0000) 2019-10-13T05:11:22.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:22.515+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.515755+0000) 2019-10-13T05:11:22.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:22.519+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:22.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:22.659+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.660607+0000) 2019-10-13T05:11:22.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:22.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:22.660765+0000) 2019-10-13T05:11:24.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:24.154+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:24.155437+0000) 2019-10-13T05:11:27.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:27.521+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.522753+0000) 2019-10-13T05:11:27.526 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:27.517+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.518684+0000) 2019-10-13T05:11:27.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:27.521+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.523251+0000) 2019-10-13T05:11:27.527 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:27.522+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.523437+0000) 2019-10-13T05:11:27.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:27.522+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.523761+0000) 2019-10-13T05:11:27.527 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:27.522+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.524045+0000) 2019-10-13T05:11:27.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:27.523+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.524568+0000) 2019-10-13T05:11:27.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:27.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71914 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:27.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:27.521+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:27.522183+0000) 2019-10-13T05:11:28.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:28.199+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:28.199991+0000) 2019-10-13T05:11:28.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:28.539+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:28.539807+0000) 2019-10-13T05:11:28.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:28.539+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:28.539802+0000) 2019-10-13T05:11:30.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:30.054+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:30.054373+0000) 2019-10-13T05:11:31.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:31.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:31.790129+0000) 2019-10-13T05:11:32.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:32.526+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.527376+0000) 2019-10-13T05:11:32.540 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:32.527+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.527472+0000) 2019-10-13T05:11:32.540 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:32.527+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.527586+0000) 2019-10-13T05:11:32.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:32.527+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.527686+0000) 2019-10-13T05:11:32.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:32.521+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.523509+0000) 2019-10-13T05:11:32.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:32.524+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.525797+0000) 2019-10-13T05:11:32.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:32.527+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.528185+0000) 2019-10-13T05:11:32.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:32.527+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.528256+0000) 2019-10-13T05:11:32.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:32.529+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71948 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:32.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:32.660+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.660742+0000) 2019-10-13T05:11:32.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:32.659+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:32.660872+0000) 2019-10-13T05:11:34.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:34.155+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:34.155605+0000) 2019-10-13T05:11:35.820 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:11:35.820 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:35.824 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:11:35.824 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:37.534 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:37.525+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.526873+0000) 2019-10-13T05:11:37.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:37.531+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.531580+0000) 2019-10-13T05:11:37.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:37.531+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.531769+0000) 2019-10-13T05:11:37.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:37.531+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.531965+0000) 2019-10-13T05:11:37.535 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:37.531+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.532223+0000) 2019-10-13T05:11:37.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:37.531+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.532320+0000) 2019-10-13T05:11:37.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:37.532+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.532601+0000) 2019-10-13T05:11:37.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:37.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 71975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:37.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:37.528+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:37.530607+0000) 2019-10-13T05:11:38.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:38.199+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:38.200157+0000) 2019-10-13T05:11:38.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:38.539+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:38.540002+0000) 2019-10-13T05:11:38.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:38.539+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:38.540003+0000) 2019-10-13T05:11:40.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:40.055+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:40.056745+0000) 2019-10-13T05:11:41.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:41.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:41.790293+0000) 2019-10-13T05:11:42.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:42.535+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.536235+0000) 2019-10-13T05:11:42.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:42.536+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.536527+0000) 2019-10-13T05:11:42.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:42.530+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.531972+0000) 2019-10-13T05:11:42.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:42.536+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.536675+0000) 2019-10-13T05:11:42.540 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:42.536+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.536766+0000) 2019-10-13T05:11:42.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:42.536+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.537073+0000) 2019-10-13T05:11:42.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:42.536+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.537383+0000) 2019-10-13T05:11:42.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:42.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72007 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:42.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:42.532+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.534565+0000) 2019-10-13T05:11:42.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:42.660+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.660842+0000) 2019-10-13T05:11:42.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:42.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:42.660993+0000) 2019-10-13T05:11:44.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:44.155+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:44.155779+0000) 2019-10-13T05:11:47.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:47.534+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.535294+0000) 2019-10-13T05:11:47.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:47.539+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.540124+0000) 2019-10-13T05:11:47.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:47.539+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.540243+0000) 2019-10-13T05:11:47.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:47.539+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.540354+0000) 2019-10-13T05:11:47.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:47.540+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.540600+0000) 2019-10-13T05:11:47.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:47.540+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.540960+0000) 2019-10-13T05:11:47.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:47.540+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.541138+0000) 2019-10-13T05:11:47.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:47.536+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:47.537466+0000) 2019-10-13T05:11:47.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:47.542+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:48.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:48.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:48.200324+0000) 2019-10-13T05:11:48.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:48.540+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:48.540188+0000) 2019-10-13T05:11:48.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:48.540+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:48.540186+0000) 2019-10-13T05:11:50.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:50.058+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:50.059101+0000) 2019-10-13T05:11:51.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:51.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:51.790444+0000) 2019-10-13T05:11:52.547 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:52.539+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.539893+0000) 2019-10-13T05:11:52.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:52.543+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.544738+0000) 2019-10-13T05:11:52.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:52.543+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.544829+0000) 2019-10-13T05:11:52.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:52.543+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.545039+0000) 2019-10-13T05:11:52.549 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:52.543+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.545174+0000) 2019-10-13T05:11:52.549 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:52.544+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.545478+0000) 2019-10-13T05:11:52.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:52.544+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.545549+0000) 2019-10-13T05:11:52.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:52.542+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.542865+0000) 2019-10-13T05:11:52.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:52.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:52.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:52.660+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.661006+0000) 2019-10-13T05:11:52.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:52.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:52.661113+0000) 2019-10-13T05:11:54.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:54.154+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:54.155940+0000) 2019-10-13T05:11:57.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:11:57.544+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.544797+0000) 2019-10-13T05:11:57.553 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:11:57.548+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.549589+0000) 2019-10-13T05:11:57.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:11:57.548+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.549867+0000) 2019-10-13T05:11:57.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:57.548+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.550111+0000) 2019-10-13T05:11:57.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:57.548+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.550253+0000) 2019-10-13T05:11:57.553 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:11:57.549+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.550411+0000) 2019-10-13T05:11:57.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:57.549+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.550586+0000) 2019-10-13T05:11:57.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:11:57.546+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:57.547507+0000) 2019-10-13T05:11:57.555 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:11:57.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72096 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:11:58.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:11:58.199+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:58.200488+0000) 2019-10-13T05:11:58.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:11:58.539+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:58.540355+0000) 2019-10-13T05:11:58.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:11:58.539+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:11:58.540381+0000) 2019-10-13T05:12:00.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:00.061+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:00.062258+0000) 2019-10-13T05:12:01.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:01.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:01.790599+0000) 2019-10-13T05:12:02.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:02.548+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.549287+0000) 2019-10-13T05:12:02.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:02.552+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.553894+0000) 2019-10-13T05:12:02.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:02.552+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.554226+0000) 2019-10-13T05:12:02.561 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:02.552+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.554353+0000) 2019-10-13T05:12:02.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:02.552+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.554390+0000) 2019-10-13T05:12:02.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:02.553+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.554595+0000) 2019-10-13T05:12:02.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:02.553+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.554738+0000) 2019-10-13T05:12:02.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:02.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:02.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:02.551+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.551809+0000) 2019-10-13T05:12:02.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:02.660+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.661139+0000) 2019-10-13T05:12:02.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:02.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:02.661230+0000) 2019-10-13T05:12:04.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:04.155+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:04.156109+0000) 2019-10-13T05:12:06.000 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:12:06.000 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:06.005 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:12:06.005 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:07.560 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:07.552+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.553357+0000) 2019-10-13T05:12:07.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:07.556+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.558210+0000) 2019-10-13T05:12:07.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:07.557+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.558463+0000) 2019-10-13T05:12:07.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:07.557+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.558873+0000) 2019-10-13T05:12:07.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:07.557+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.558938+0000) 2019-10-13T05:12:07.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:07.557+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.559082+0000) 2019-10-13T05:12:07.563 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:07.556+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.558101+0000) 2019-10-13T05:12:07.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:07.554+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:07.555556+0000) 2019-10-13T05:12:07.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:07.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:08.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:08.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:08.200647+0000) 2019-10-13T05:12:08.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:08.539+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:08.540547+0000) 2019-10-13T05:12:08.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:08.539+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:08.540560+0000) 2019-10-13T05:12:10.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:10.064+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:10.065225+0000) 2019-10-13T05:12:11.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:11.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:11.790814+0000) 2019-10-13T05:12:12.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:12.560+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.562332+0000) 2019-10-13T05:12:12.565 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:12.561+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.562482+0000) 2019-10-13T05:12:12.566 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:12.557+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.558311+0000) 2019-10-13T05:12:12.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:12.561+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.563077+0000) 2019-10-13T05:12:12.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:12.561+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.563111+0000) 2019-10-13T05:12:12.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:12.562+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.563416+0000) 2019-10-13T05:12:12.567 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:12.562+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.563581+0000) 2019-10-13T05:12:12.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:12.559+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.560584+0000) 2019-10-13T05:12:12.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:12.563+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:12.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:12.660+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.661297+0000) 2019-10-13T05:12:12.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:12.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:12.661325+0000) 2019-10-13T05:12:14.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:14.155+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:14.156283+0000) 2019-10-13T05:12:17.570 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:17.561+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.562508+0000) 2019-10-13T05:12:17.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:17.566+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.567065+0000) 2019-10-13T05:12:17.572 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:17.566+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.567284+0000) 2019-10-13T05:12:17.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:17.567+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.567560+0000) 2019-10-13T05:12:17.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:17.567+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.567681+0000) 2019-10-13T05:12:17.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:17.567+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.567971+0000) 2019-10-13T05:12:17.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:17.567+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.568164+0000) 2019-10-13T05:12:17.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:17.564+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:17.565663+0000) 2019-10-13T05:12:17.575 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:17.569+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72216 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:18.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:18.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:18.200794+0000) 2019-10-13T05:12:18.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:18.540+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:18.540657+0000) 2019-10-13T05:12:18.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:18.540+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:18.540651+0000) 2019-10-13T05:12:20.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:20.067+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:20.067132+0000) 2019-10-13T05:12:21.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:21.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:21.791021+0000) 2019-10-13T05:12:22.576 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:22.566+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.567785+0000) 2019-10-13T05:12:22.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:22.571+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.571956+0000) 2019-10-13T05:12:22.577 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:22.571+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.572163+0000) 2019-10-13T05:12:22.577 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:22.572+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.572413+0000) 2019-10-13T05:12:22.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:22.572+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.572680+0000) 2019-10-13T05:12:22.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:22.572+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.572970+0000) 2019-10-13T05:12:22.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:22.572+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.572840+0000) 2019-10-13T05:12:22.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:22.569+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.570836+0000) 2019-10-13T05:12:22.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:22.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:22.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:22.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.661449+0000) 2019-10-13T05:12:22.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:22.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:22.661445+0000) 2019-10-13T05:12:24.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:24.156+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:24.156474+0000) 2019-10-13T05:12:27.580 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:27.570+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.572092+0000) 2019-10-13T05:12:27.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:27.575+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.575901+0000) 2019-10-13T05:12:27.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:27.575+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.576020+0000) 2019-10-13T05:12:27.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:27.575+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.576313+0000) 2019-10-13T05:12:27.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:27.576+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.576678+0000) 2019-10-13T05:12:27.581 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:27.576+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.576961+0000) 2019-10-13T05:12:27.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:27.576+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.577211+0000) 2019-10-13T05:12:27.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:27.572+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:27.574487+0000) 2019-10-13T05:12:27.582 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:27.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:28.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:28.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:28.200955+0000) 2019-10-13T05:12:28.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:28.540+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:28.540779+0000) 2019-10-13T05:12:28.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:28.540+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:28.540821+0000) 2019-10-13T05:12:30.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:30.069+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:30.070247+0000) 2019-10-13T05:12:31.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:31.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:31.791225+0000) 2019-10-13T05:12:32.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:32.579+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.579705+0000) 2019-10-13T05:12:32.583 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:32.579+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.580057+0000) 2019-10-13T05:12:32.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:32.573+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.575454+0000) 2019-10-13T05:12:32.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:32.579+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.580239+0000) 2019-10-13T05:12:32.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:32.580+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.580502+0000) 2019-10-13T05:12:32.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:32.580+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.580760+0000) 2019-10-13T05:12:32.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:32.580+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.580909+0000) 2019-10-13T05:12:32.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:32.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:32.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:32.577+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.578763+0000) 2019-10-13T05:12:32.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:32.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.661534+0000) 2019-10-13T05:12:32.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:32.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:32.661604+0000) 2019-10-13T05:12:34.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:34.156+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:34.156687+0000) 2019-10-13T05:12:36.148 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:12:36.148 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:36.152 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:12:36.153 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:37.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:37.579+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.580131+0000) 2019-10-13T05:12:37.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:37.582+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.584221+0000) 2019-10-13T05:12:37.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:37.582+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.584370+0000) 2019-10-13T05:12:37.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:37.583+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.584984+0000) 2019-10-13T05:12:37.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:37.583+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.585200+0000) 2019-10-13T05:12:37.590 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:37.583+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.585352+0000) 2019-10-13T05:12:37.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:37.584+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.585550+0000) 2019-10-13T05:12:37.591 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:37.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:37.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:37.583+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:37.584345+0000) 2019-10-13T05:12:38.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:38.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:38.201121+0000) 2019-10-13T05:12:38.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:38.539+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:38.540955+0000) 2019-10-13T05:12:38.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:38.539+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:38.540947+0000) 2019-10-13T05:12:40.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:40.072+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:40.073383+0000) 2019-10-13T05:12:41.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:41.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:41.791436+0000) 2019-10-13T05:12:42.592 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:42.587+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.589131+0000) 2019-10-13T05:12:42.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:42.588+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.589458+0000) 2019-10-13T05:12:42.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:42.584+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.585466+0000) 2019-10-13T05:12:42.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:42.588+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.589579+0000) 2019-10-13T05:12:42.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:42.588+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.589679+0000) 2019-10-13T05:12:42.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:42.588+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.590192+0000) 2019-10-13T05:12:42.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:42.589+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.590467+0000) 2019-10-13T05:12:42.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:42.585+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.586678+0000) 2019-10-13T05:12:42.595 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:42.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:42.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:42.660+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.661694+0000) 2019-10-13T05:12:42.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:42.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:42.661760+0000) 2019-10-13T05:12:44.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:44.156+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:44.156849+0000) 2019-10-13T05:12:47.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:47.591+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.593002+0000) 2019-10-13T05:12:47.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:47.591+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.593270+0000) 2019-10-13T05:12:47.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:47.591+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.593114+0000) 2019-10-13T05:12:47.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:47.589+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.589858+0000) 2019-10-13T05:12:47.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:47.592+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.593948+0000) 2019-10-13T05:12:47.597 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:47.592+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.594364+0000) 2019-10-13T05:12:47.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:47.593+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.594688+0000) 2019-10-13T05:12:47.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:47.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:47.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:47.591+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:47.592191+0000) 2019-10-13T05:12:48.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:48.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:48.201312+0000) 2019-10-13T05:12:48.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:48.540+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:48.541111+0000) 2019-10-13T05:12:48.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:48.540+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:48.541103+0000) 2019-10-13T05:12:50.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:50.074+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:50.075739+0000) 2019-10-13T05:12:51.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:51.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:51.791642+0000) 2019-10-13T05:12:52.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:52.596+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.597579+0000) 2019-10-13T05:12:52.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:52.592+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.593266+0000) 2019-10-13T05:12:52.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:52.596+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.597989+0000) 2019-10-13T05:12:52.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:52.596+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.598197+0000) 2019-10-13T05:12:52.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:52.596+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.598263+0000) 2019-10-13T05:12:52.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:52.597+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.598785+0000) 2019-10-13T05:12:52.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:52.597+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.598750+0000) 2019-10-13T05:12:52.603 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:52.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:52.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:52.595+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.596424+0000) 2019-10-13T05:12:52.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:52.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.661866+0000) 2019-10-13T05:12:52.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:52.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:52.661891+0000) 2019-10-13T05:12:54.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:54.156+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:54.157015+0000) 2019-10-13T05:12:57.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:57.600+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.601572+0000) 2019-10-13T05:12:57.605 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:12:57.596+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.597425+0000) 2019-10-13T05:12:57.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:12:57.600+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.602275+0000) 2019-10-13T05:12:57.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:57.601+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.602415+0000) 2019-10-13T05:12:57.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:57.601+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.602554+0000) 2019-10-13T05:12:57.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:12:57.601+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.602768+0000) 2019-10-13T05:12:57.606 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:12:57.601+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.602830+0000) 2019-10-13T05:12:57.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:12:57.599+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:57.600680+0000) 2019-10-13T05:12:57.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:12:57.603+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:12:58.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:12:58.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:58.201423+0000) 2019-10-13T05:12:58.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:12:58.540+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:58.541294+0000) 2019-10-13T05:12:58.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:12:58.540+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:12:58.541291+0000) 2019-10-13T05:13:00.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:00.077+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:00.078079+0000) 2019-10-13T05:13:01.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:01.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:01.791881+0000) 2019-10-13T05:13:02.610 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:02.606+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.606785+0000) 2019-10-13T05:13:02.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:02.606+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.607015+0000) 2019-10-13T05:13:02.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:02.606+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.607190+0000) 2019-10-13T05:13:02.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:02.606+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.607336+0000) 2019-10-13T05:13:02.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:02.602+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.603364+0000) 2019-10-13T05:13:02.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:02.607+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.607687+0000) 2019-10-13T05:13:02.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:02.607+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.607789+0000) 2019-10-13T05:13:02.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:02.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:02.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:02.605+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.605877+0000) 2019-10-13T05:13:02.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:02.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.662012+0000) 2019-10-13T05:13:02.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:02.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:02.661972+0000) 2019-10-13T05:13:04.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:04.156+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:04.157188+0000) 2019-10-13T05:13:06.337 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:13:06.337 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:06.341 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:13:06.341 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:07.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:07.606+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.606858+0000) 2019-10-13T05:13:07.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:07.610+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.611123+0000) 2019-10-13T05:13:07.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:07.611+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.611549+0000) 2019-10-13T05:13:07.615 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:07.611+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.611723+0000) 2019-10-13T05:13:07.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:07.611+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.611836+0000) 2019-10-13T05:13:07.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:07.611+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.611981+0000) 2019-10-13T05:13:07.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:07.611+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.612305+0000) 2019-10-13T05:13:07.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:07.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:07.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:07.609+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:07.609988+0000) 2019-10-13T05:13:08.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:08.201+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:08.201566+0000) 2019-10-13T05:13:08.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:08.541+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:08.541490+0000) 2019-10-13T05:13:08.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:08.541+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:08.541469+0000) 2019-10-13T05:13:10.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:10.080+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:10.080402+0000) 2019-10-13T05:13:11.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:11.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:11.792046+0000) 2019-10-13T05:13:12.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:12.614+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.615313+0000) 2019-10-13T05:13:12.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:12.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.611163+0000) 2019-10-13T05:13:12.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:12.615+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.615684+0000) 2019-10-13T05:13:12.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:12.615+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.616028+0000) 2019-10-13T05:13:12.620 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:12.615+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.616197+0000) 2019-10-13T05:13:12.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:12.615+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.616387+0000) 2019-10-13T05:13:12.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:12.616+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.616612+0000) 2019-10-13T05:13:12.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:12.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:12.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:12.613+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.614766+0000) 2019-10-13T05:13:12.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:12.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.662168+0000) 2019-10-13T05:13:12.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:12.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:12.662135+0000) 2019-10-13T05:13:14.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:14.157+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:14.157382+0000) 2019-10-13T05:13:17.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:17.619+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.619764+0000) 2019-10-13T05:13:17.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:17.619+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.620217+0000) 2019-10-13T05:13:17.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:17.619+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.620377+0000) 2019-10-13T05:13:17.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:17.620+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.620529+0000) 2019-10-13T05:13:17.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:17.614+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.616265+0000) 2019-10-13T05:13:17.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:17.620+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.620863+0000) 2019-10-13T05:13:17.631 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:17.615+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.617440+0000) 2019-10-13T05:13:17.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:17.620+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:17.620997+0000) 2019-10-13T05:13:17.635 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:17.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72576 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:18.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:18.201+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:18.201680+0000) 2019-10-13T05:13:18.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:18.541+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:18.541642+0000) 2019-10-13T05:13:18.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:18.541+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:18.541643+0000) 2019-10-13T05:13:20.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:20.081+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:20.082687+0000) 2019-10-13T05:13:21.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:21.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:21.792209+0000) 2019-10-13T05:13:22.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:22.623+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.624508+0000) 2019-10-13T05:13:22.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:22.618+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.620699+0000) 2019-10-13T05:13:22.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:22.623+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.624821+0000) 2019-10-13T05:13:22.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:22.623+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.625307+0000) 2019-10-13T05:13:22.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:22.624+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.625460+0000) 2019-10-13T05:13:22.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:22.624+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.625679+0000) 2019-10-13T05:13:22.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:22.624+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.625574+0000) 2019-10-13T05:13:22.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:22.620+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.622393+0000) 2019-10-13T05:13:22.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:22.625+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:22.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:22.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.662269+0000) 2019-10-13T05:13:22.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:22.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:22.662262+0000) 2019-10-13T05:13:24.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:24.156+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:24.157593+0000) 2019-10-13T05:13:27.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:27.627+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.629371+0000) 2019-10-13T05:13:27.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:27.628+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.629414+0000) 2019-10-13T05:13:27.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:27.628+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.629635+0000) 2019-10-13T05:13:27.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:27.628+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.629798+0000) 2019-10-13T05:13:27.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:27.628+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.630139+0000) 2019-10-13T05:13:27.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:27.625+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.625749+0000) 2019-10-13T05:13:27.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:27.626+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.626809+0000) 2019-10-13T05:13:27.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:27.629+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:27.630613+0000) 2019-10-13T05:13:27.636 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:27.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:28.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:28.200+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:28.201848+0000) 2019-10-13T05:13:28.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:28.540+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:28.541765+0000) 2019-10-13T05:13:28.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:28.540+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:28.541807+0000) 2019-10-13T05:13:30.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:30.084+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:30.085030+0000) 2019-10-13T05:13:31.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:31.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:31.792439+0000) 2019-10-13T05:13:32.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:32.629+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.629884+0000) 2019-10-13T05:13:32.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:32.632+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.634148+0000) 2019-10-13T05:13:32.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:32.633+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.634469+0000) 2019-10-13T05:13:32.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:32.633+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.635008+0000) 2019-10-13T05:13:32.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:32.633+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.635048+0000) 2019-10-13T05:13:32.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:32.634+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.635466+0000) 2019-10-13T05:13:32.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:32.634+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.635756+0000) 2019-10-13T05:13:32.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:32.631+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.632594+0000) 2019-10-13T05:13:32.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:32.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:32.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:32.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.662413+0000) 2019-10-13T05:13:32.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:32.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:32.662421+0000) 2019-10-13T05:13:34.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:34.157+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:34.157777+0000) 2019-10-13T05:13:36.544 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:13:36.544 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:36.549 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:13:36.550 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:37.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:37.637+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.639373+0000) 2019-10-13T05:13:37.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:37.638+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.639496+0000) 2019-10-13T05:13:37.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:37.638+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.639802+0000) 2019-10-13T05:13:37.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:37.635+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.635989+0000) 2019-10-13T05:13:37.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:37.638+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.640133+0000) 2019-10-13T05:13:37.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:37.638+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.640239+0000) 2019-10-13T05:13:37.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:37.639+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.640623+0000) 2019-10-13T05:13:37.645 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:37.637+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:37.637789+0000) 2019-10-13T05:13:37.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:37.640+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:38.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:38.201+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:38.202016+0000) 2019-10-13T05:13:38.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:38.541+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:38.541931+0000) 2019-10-13T05:13:38.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:38.541+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:38.541976+0000) 2019-10-13T05:13:40.096 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:40.086+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:40.087491+0000) 2019-10-13T05:13:41.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:41.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:41.792628+0000) 2019-10-13T05:13:42.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:42.641+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.643091+0000) 2019-10-13T05:13:42.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:42.641+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.643365+0000) 2019-10-13T05:13:42.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:42.642+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.643598+0000) 2019-10-13T05:13:42.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:42.642+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.643832+0000) 2019-10-13T05:13:42.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:42.642+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.644057+0000) 2019-10-13T05:13:42.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:42.638+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.639532+0000) 2019-10-13T05:13:42.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:42.640+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.640931+0000) 2019-10-13T05:13:42.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:42.642+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.644169+0000) 2019-10-13T05:13:42.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:42.643+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:42.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:42.662+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.662575+0000) 2019-10-13T05:13:42.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:42.662+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:42.662545+0000) 2019-10-13T05:13:44.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:44.157+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:44.157971+0000) 2019-10-13T05:13:47.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:47.641+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.642695+0000) 2019-10-13T05:13:47.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:47.645+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.646930+0000) 2019-10-13T05:13:47.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:47.645+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.647254+0000) 2019-10-13T05:13:47.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:47.646+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.647408+0000) 2019-10-13T05:13:47.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:47.646+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.647500+0000) 2019-10-13T05:13:47.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:47.646+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.647675+0000) 2019-10-13T05:13:47.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:47.646+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.647842+0000) 2019-10-13T05:13:47.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:47.644+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:47.645239+0000) 2019-10-13T05:13:47.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:47.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:48.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:48.201+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:48.202115+0000) 2019-10-13T05:13:48.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:48.541+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:48.542100+0000) 2019-10-13T05:13:48.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:48.541+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:48.542154+0000) 2019-10-13T05:13:50.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:50.089+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:50.089977+0000) 2019-10-13T05:13:51.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:51.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:51.792819+0000) 2019-10-13T05:13:52.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:52.650+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.650900+0000) 2019-10-13T05:13:52.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:52.650+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.651134+0000) 2019-10-13T05:13:52.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:52.646+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.647002+0000) 2019-10-13T05:13:52.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:52.651+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.651740+0000) 2019-10-13T05:13:52.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:52.651+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.651857+0000) 2019-10-13T05:13:52.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:52.651+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.652076+0000) 2019-10-13T05:13:52.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:52.651+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.652273+0000) 2019-10-13T05:13:52.656 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:52.648+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.649225+0000) 2019-10-13T05:13:52.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:52.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72787 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:52.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:52.662+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.662687+0000) 2019-10-13T05:13:52.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:52.662+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:52.662642+0000) 2019-10-13T05:13:54.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:54.157+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:54.158165+0000) 2019-10-13T05:13:57.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:13:57.655+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.655456+0000) 2019-10-13T05:13:57.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:57.655+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.655731+0000) 2019-10-13T05:13:57.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:57.655+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.655617+0000) 2019-10-13T05:13:57.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:13:57.655+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.656304+0000) 2019-10-13T05:13:57.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:57.656+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.656545+0000) 2019-10-13T05:13:57.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:13:57.656+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.656818+0000) 2019-10-13T05:13:57.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:13:57.651+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.651747+0000) 2019-10-13T05:13:57.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:13:57.652+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:57.652887+0000) 2019-10-13T05:13:57.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:13:57.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:13:58.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:13:58.202+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:58.202279+0000) 2019-10-13T05:13:58.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:13:58.542+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:58.542284+0000) 2019-10-13T05:13:58.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:13:58.542+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:13:58.542313+0000) 2019-10-13T05:14:00.099 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:00.092+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:00.092363+0000) 2019-10-13T05:14:01.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:01.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:01.792977+0000) 2019-10-13T05:14:02.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:02.659+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.659861+0000) 2019-10-13T05:14:02.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:02.659+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.659819+0000) 2019-10-13T05:14:02.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:02.659+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.660159+0000) 2019-10-13T05:14:02.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:02.653+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.655600+0000) 2019-10-13T05:14:02.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:02.660+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.660537+0000) 2019-10-13T05:14:02.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:02.660+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.660711+0000) 2019-10-13T05:14:02.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:02.660+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.660912+0000) 2019-10-13T05:14:02.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:02.662+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.662789+0000) 2019-10-13T05:14:02.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:02.662+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:02.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:02.657+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.658910+0000) 2019-10-13T05:14:02.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:02.660+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:02.662722+0000) 2019-10-13T05:14:04.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:04.158+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:04.158375+0000) 2019-10-13T05:14:06.744 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:14:06.744 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:06.748 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:14:06.748 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:07.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:07.663+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.664516+0000) 2019-10-13T05:14:07.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:07.663+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.664731+0000) 2019-10-13T05:14:07.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:07.663+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.664938+0000) 2019-10-13T05:14:07.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:07.659+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.661295+0000) 2019-10-13T05:14:07.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:07.664+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.665538+0000) 2019-10-13T05:14:07.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:07.664+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.665952+0000) 2019-10-13T05:14:07.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:07.664+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.666052+0000) 2019-10-13T05:14:07.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:07.661+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:07.662862+0000) 2019-10-13T05:14:07.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:07.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:08.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:08.201+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:08.202395+0000) 2019-10-13T05:14:08.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:08.541+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:08.542449+0000) 2019-10-13T05:14:08.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:08.541+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:08.542491+0000) 2019-10-13T05:14:10.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:10.094+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:10.095530+0000) 2019-10-13T05:14:11.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:11.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:11.793204+0000) 2019-10-13T05:14:12.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:12.661+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.662951+0000) 2019-10-13T05:14:12.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:12.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.662864+0000) 2019-10-13T05:14:12.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:12.668+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.669976+0000) 2019-10-13T05:14:12.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:12.668+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.670126+0000) 2019-10-13T05:14:12.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:12.664+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.666259+0000) 2019-10-13T05:14:12.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:12.668+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.670367+0000) 2019-10-13T05:14:12.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:12.669+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.670426+0000) 2019-10-13T05:14:12.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:12.669+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.670549+0000) 2019-10-13T05:14:12.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:12.669+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.670897+0000) 2019-10-13T05:14:12.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:12.669+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:12.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:12.667+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:12.668901+0000) 2019-10-13T05:14:14.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:14.157+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:14.158590+0000) 2019-10-13T05:14:17.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:17.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.668936+0000) 2019-10-13T05:14:17.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:17.672+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.673626+0000) 2019-10-13T05:14:17.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:17.672+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.673855+0000) 2019-10-13T05:14:17.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:17.672+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.673997+0000) 2019-10-13T05:14:17.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:17.672+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.674130+0000) 2019-10-13T05:14:17.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:17.673+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.674719+0000) 2019-10-13T05:14:17.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:17.673+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.674891+0000) 2019-10-13T05:14:17.681 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:17.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:17.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:17.671+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:17.672262+0000) 2019-10-13T05:14:18.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:18.201+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:18.202558+0000) 2019-10-13T05:14:18.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:18.541+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:18.542613+0000) 2019-10-13T05:14:18.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:18.541+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:18.542670+0000) 2019-10-13T05:14:20.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:20.097+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:20.097809+0000) 2019-10-13T05:14:21.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:21.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:21.793417+0000) 2019-10-13T05:14:22.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:22.662+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.663095+0000) 2019-10-13T05:14:22.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:22.662+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.663053+0000) 2019-10-13T05:14:22.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:22.676+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.678167+0000) 2019-10-13T05:14:22.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:22.677+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.678433+0000) 2019-10-13T05:14:22.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:22.677+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.678578+0000) 2019-10-13T05:14:22.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:22.673+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.674640+0000) 2019-10-13T05:14:22.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:22.677+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.678966+0000) 2019-10-13T05:14:22.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:22.677+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.679219+0000) 2019-10-13T05:14:22.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:22.678+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.679464+0000) 2019-10-13T05:14:22.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:22.676+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:22.677620+0000) 2019-10-13T05:14:22.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:22.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72960 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:24.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:24.158+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:24.158718+0000) 2019-10-13T05:14:27.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:27.681+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.682869+0000) 2019-10-13T05:14:27.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:27.681+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.683005+0000) 2019-10-13T05:14:27.686 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:27.681+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.683217+0000) 2019-10-13T05:14:27.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:27.679+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.679769+0000) 2019-10-13T05:14:27.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:27.682+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.683697+0000) 2019-10-13T05:14:27.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:27.682+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.683849+0000) 2019-10-13T05:14:27.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:27.682+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.683978+0000) 2019-10-13T05:14:27.688 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:27.680+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:27.681084+0000) 2019-10-13T05:14:27.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:27.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 72988 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:28.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:28.202+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:28.202728+0000) 2019-10-13T05:14:28.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:28.542+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:28.542776+0000) 2019-10-13T05:14:28.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:28.542+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:28.542816+0000) 2019-10-13T05:14:30.107 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:30.099+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:30.099922+0000) 2019-10-13T05:14:31.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:31.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:31.793577+0000) 2019-10-13T05:14:32.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:32.662+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.663256+0000) 2019-10-13T05:14:32.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:32.662+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.663252+0000) 2019-10-13T05:14:32.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:32.686+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.687471+0000) 2019-10-13T05:14:32.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:32.686+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.687768+0000) 2019-10-13T05:14:32.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:32.686+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.687890+0000) 2019-10-13T05:14:32.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:32.683+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.683839+0000) 2019-10-13T05:14:32.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:32.686+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.688183+0000) 2019-10-13T05:14:32.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:32.687+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.688477+0000) 2019-10-13T05:14:32.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:32.687+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.688589+0000) 2019-10-13T05:14:32.693 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:32.688+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:32.694 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:32.685+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:32.686595+0000) 2019-10-13T05:14:34.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:34.158+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:34.158864+0000) 2019-10-13T05:14:36.931 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:14:36.932 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:36.936 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:14:36.936 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:37.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:37.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.691706+0000) 2019-10-13T05:14:37.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:37.691+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.692103+0000) 2019-10-13T05:14:37.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:37.686+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.687537+0000) 2019-10-13T05:14:37.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:37.692+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.692649+0000) 2019-10-13T05:14:37.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:37.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.692810+0000) 2019-10-13T05:14:37.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:37.692+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.693165+0000) 2019-10-13T05:14:37.698 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:37.693+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.693411+0000) 2019-10-13T05:14:37.698 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:37.690+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:37.690886+0000) 2019-10-13T05:14:37.698 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:37.694+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:38.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:38.202+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:38.202933+0000) 2019-10-13T05:14:38.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:38.542+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:38.542943+0000) 2019-10-13T05:14:38.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:38.542+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:38.542979+0000) 2019-10-13T05:14:40.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:40.101+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:40.102181+0000) 2019-10-13T05:14:41.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:41.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:41.793749+0000) 2019-10-13T05:14:42.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:42.663+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.663440+0000) 2019-10-13T05:14:42.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:42.663+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.663434+0000) 2019-10-13T05:14:42.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:42.696+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.696458+0000) 2019-10-13T05:14:42.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:42.696+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.696620+0000) 2019-10-13T05:14:42.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:42.696+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.696821+0000) 2019-10-13T05:14:42.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:42.696+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.696962+0000) 2019-10-13T05:14:42.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:42.692+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.693008+0000) 2019-10-13T05:14:42.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:42.696+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.697260+0000) 2019-10-13T05:14:42.701 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:42.697+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.697791+0000) 2019-10-13T05:14:42.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:42.694+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:42.695340+0000) 2019-10-13T05:14:42.703 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:42.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:44.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:44.158+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:44.159008+0000) 2019-10-13T05:14:47.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:47.700+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.701227+0000) 2019-10-13T05:14:47.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:47.701+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.701404+0000) 2019-10-13T05:14:47.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:47.700+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.701127+0000) 2019-10-13T05:14:47.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:47.701+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.701579+0000) 2019-10-13T05:14:47.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:47.701+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.701650+0000) 2019-10-13T05:14:47.706 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:47.701+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.701923+0000) 2019-10-13T05:14:47.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:47.697+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.697876+0000) 2019-10-13T05:14:47.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:47.703+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:47.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:47.699+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:47.700282+0000) 2019-10-13T05:14:48.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:48.202+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:48.203079+0000) 2019-10-13T05:14:48.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:48.542+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:48.543068+0000) 2019-10-13T05:14:48.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:48.542+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:48.543128+0000) 2019-10-13T05:14:50.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:50.105+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:50.105199+0000) 2019-10-13T05:14:51.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:51.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:51.793924+0000) 2019-10-13T05:14:52.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:52.662+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.663631+0000) 2019-10-13T05:14:52.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:52.661+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.663635+0000) 2019-10-13T05:14:52.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:52.704+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.705433+0000) 2019-10-13T05:14:52.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:52.704+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.705610+0000) 2019-10-13T05:14:52.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:52.704+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.705746+0000) 2019-10-13T05:14:52.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:52.700+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.702332+0000) 2019-10-13T05:14:52.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:52.704+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.706023+0000) 2019-10-13T05:14:52.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:52.704+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.706147+0000) 2019-10-13T05:14:52.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:52.704+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.706272+0000) 2019-10-13T05:14:52.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:52.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:52.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:52.702+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:52.704612+0000) 2019-10-13T05:14:54.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:54.157+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:54.159182+0000) 2019-10-13T05:14:57.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:14:57.704+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.706020+0000) 2019-10-13T05:14:57.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:14:57.709+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.710622+0000) 2019-10-13T05:14:57.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:57.710+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.711458+0000) 2019-10-13T05:14:57.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:57.709+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.711377+0000) 2019-10-13T05:14:57.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:57.710+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.711578+0000) 2019-10-13T05:14:57.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:14:57.710+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.711822+0000) 2019-10-13T05:14:57.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:14:57.710+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.711965+0000) 2019-10-13T05:14:57.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:14:57.706+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:57.708178+0000) 2019-10-13T05:14:57.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:14:57.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:14:58.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:14:58.201+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:58.203250+0000) 2019-10-13T05:14:58.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:14:58.542+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:58.543180+0000) 2019-10-13T05:14:58.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:14:58.542+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:14:58.543268+0000) 2019-10-13T05:15:00.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:00.106+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:00.107495+0000) 2019-10-13T05:15:01.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:01.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:01.794141+0000) 2019-10-13T05:15:02.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:02.663+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.663795+0000) 2019-10-13T05:15:02.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:02.662+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.663816+0000) 2019-10-13T05:15:02.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:02.714+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.715553+0000) 2019-10-13T05:15:02.719 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:02.714+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.715720+0000) 2019-10-13T05:15:02.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:02.714+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.715941+0000) 2019-10-13T05:15:02.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:02.714+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.716065+0000) 2019-10-13T05:15:02.720 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:02.710+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.711825+0000) 2019-10-13T05:15:02.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:02.715+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.716512+0000) 2019-10-13T05:15:02.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:02.715+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.716743+0000) 2019-10-13T05:15:02.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:02.717+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:02.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:02.712+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:02.714253+0000) 2019-10-13T05:15:04.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:04.158+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:04.159411+0000) 2019-10-13T05:15:07.060 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:15:07.060 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:07.065 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:15:07.065 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:07.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:07.718+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.720331+0000) 2019-10-13T05:15:07.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:07.715+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.716672+0000) 2019-10-13T05:15:07.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:07.719+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.720794+0000) 2019-10-13T05:15:07.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:07.719+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.720919+0000) 2019-10-13T05:15:07.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:07.719+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.721227+0000) 2019-10-13T05:15:07.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:07.720+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.721426+0000) 2019-10-13T05:15:07.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:07.720+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.721566+0000) 2019-10-13T05:15:07.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:07.717+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:07.718128+0000) 2019-10-13T05:15:07.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:07.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73220 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:08.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:08.202+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:08.203406+0000) 2019-10-13T05:15:08.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:08.542+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:08.543343+0000) 2019-10-13T05:15:08.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:08.542+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:08.543437+0000) 2019-10-13T05:15:10.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:10.109+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:10.109863+0000) 2019-10-13T05:15:11.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:11.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:11.794329+0000) 2019-10-13T05:15:12.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:12.663+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.663938+0000) 2019-10-13T05:15:12.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:12.663+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.663989+0000) 2019-10-13T05:15:12.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:12.723+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.724674+0000) 2019-10-13T05:15:12.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:12.719+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.720475+0000) 2019-10-13T05:15:12.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:12.723+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.725387+0000) 2019-10-13T05:15:12.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:12.724+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.725537+0000) 2019-10-13T05:15:12.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:12.724+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.725684+0000) 2019-10-13T05:15:12.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:12.724+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.725870+0000) 2019-10-13T05:15:12.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:12.724+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.725895+0000) 2019-10-13T05:15:12.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:12.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:12.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:12.723+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:12.723833+0000) 2019-10-13T05:15:14.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:14.159+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:14.159533+0000) 2019-10-13T05:15:17.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:17.727+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.728390+0000) 2019-10-13T05:15:17.731 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:17.727+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.728502+0000) 2019-10-13T05:15:17.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:17.724+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.724922+0000) 2019-10-13T05:15:17.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:17.727+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.728882+0000) 2019-10-13T05:15:17.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:17.727+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.729163+0000) 2019-10-13T05:15:17.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:17.728+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.729793+0000) 2019-10-13T05:15:17.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:17.728+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.729671+0000) 2019-10-13T05:15:17.733 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:17.725+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:17.726333+0000) 2019-10-13T05:15:17.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:17.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73279 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:18.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:18.203+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:18.203544+0000) 2019-10-13T05:15:18.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:18.543+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:18.543484+0000) 2019-10-13T05:15:18.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:18.543+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:18.543575+0000) 2019-10-13T05:15:20.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:20.111+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:20.112187+0000) 2019-10-13T05:15:21.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:21.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:21.794535+0000) 2019-10-13T05:15:22.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:22.663+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.664101+0000) 2019-10-13T05:15:22.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:22.663+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.664176+0000) 2019-10-13T05:15:22.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:22.733+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.733429+0000) 2019-10-13T05:15:22.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:22.733+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.733559+0000) 2019-10-13T05:15:22.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:22.729+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.730137+0000) 2019-10-13T05:15:22.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:22.733+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.734065+0000) 2019-10-13T05:15:22.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:22.733+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.734300+0000) 2019-10-13T05:15:22.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:22.734+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.734418+0000) 2019-10-13T05:15:22.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:22.734+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.734626+0000) 2019-10-13T05:15:22.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:22.730+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:22.731454+0000) 2019-10-13T05:15:22.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:22.735+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:24.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:24.159+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:24.159724+0000) 2019-10-13T05:15:27.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:27.737+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.738031+0000) 2019-10-13T05:15:27.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:27.737+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.738150+0000) 2019-10-13T05:15:27.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:27.737+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.738341+0000) 2019-10-13T05:15:27.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:27.734+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.734727+0000) 2019-10-13T05:15:27.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:27.738+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.739002+0000) 2019-10-13T05:15:27.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:27.738+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.739365+0000) 2019-10-13T05:15:27.743 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:27.739+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.739525+0000) 2019-10-13T05:15:27.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:27.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:27.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:27.736+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:27.737142+0000) 2019-10-13T05:15:28.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:28.203+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:28.203722+0000) 2019-10-13T05:15:28.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:28.543+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:28.543646+0000) 2019-10-13T05:15:28.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:28.543+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:28.543712+0000) 2019-10-13T05:15:30.122 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:30.114+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:30.114522+0000) 2019-10-13T05:15:31.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:31.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:31.794684+0000) 2019-10-13T05:15:32.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:32.664+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.664309+0000) 2019-10-13T05:15:32.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:32.664+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.664355+0000) 2019-10-13T05:15:32.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:32.742+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.742426+0000) 2019-10-13T05:15:32.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:32.742+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.742870+0000) 2019-10-13T05:15:32.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:32.742+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.743078+0000) 2019-10-13T05:15:32.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:32.742+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.743179+0000) 2019-10-13T05:15:32.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:32.743+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.743404+0000) 2019-10-13T05:15:32.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:32.743+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.743633+0000) 2019-10-13T05:15:32.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:32.738+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.739659+0000) 2019-10-13T05:15:32.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:32.740+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:32.741338+0000) 2019-10-13T05:15:32.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:32.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:34.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:34.159+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:34.159916+0000) 2019-10-13T05:15:37.208 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:15:37.208 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:37.212 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:15:37.212 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:37.750 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:37.746+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.747507+0000) 2019-10-13T05:15:37.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:37.746+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.747656+0000) 2019-10-13T05:15:37.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:37.746+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.747898+0000) 2019-10-13T05:15:37.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:37.746+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.748149+0000) 2019-10-13T05:15:37.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:37.747+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.748409+0000) 2019-10-13T05:15:37.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:37.747+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.748605+0000) 2019-10-13T05:15:37.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:37.743+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.744581+0000) 2019-10-13T05:15:37.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:37.745+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:37.745807+0000) 2019-10-13T05:15:37.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:37.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:38.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:38.202+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:38.203873+0000) 2019-10-13T05:15:38.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:38.542+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:38.543809+0000) 2019-10-13T05:15:38.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:38.542+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:38.543903+0000) 2019-10-13T05:15:40.124 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:40.116+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:40.116911+0000) 2019-10-13T05:15:41.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:41.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:41.794898+0000) 2019-10-13T05:15:42.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:42.663+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.664496+0000) 2019-10-13T05:15:42.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:42.663+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.664534+0000) 2019-10-13T05:15:42.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:42.751+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.752562+0000) 2019-10-13T05:15:42.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:42.746+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.748378+0000) 2019-10-13T05:15:42.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:42.751+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.753015+0000) 2019-10-13T05:15:42.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:42.751+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.753120+0000) 2019-10-13T05:15:42.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:42.752+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.753633+0000) 2019-10-13T05:15:42.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:42.752+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.753963+0000) 2019-10-13T05:15:42.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:42.752+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.754230+0000) 2019-10-13T05:15:42.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:42.749+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:42.750989+0000) 2019-10-13T05:15:42.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:42.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:44.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:44.158+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:44.160120+0000) 2019-10-13T05:15:47.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:47.751+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.752977+0000) 2019-10-13T05:15:47.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:47.756+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.757746+0000) 2019-10-13T05:15:47.762 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:47.756+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.757904+0000) 2019-10-13T05:15:47.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:47.756+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.758091+0000) 2019-10-13T05:15:47.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:47.756+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.758279+0000) 2019-10-13T05:15:47.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:47.757+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.758498+0000) 2019-10-13T05:15:47.762 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:47.757+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.758650+0000) 2019-10-13T05:15:47.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:47.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:47.764 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:47.754+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:47.756329+0000) 2019-10-13T05:15:48.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:48.203+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:48.204009+0000) 2019-10-13T05:15:48.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:48.543+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:48.543998+0000) 2019-10-13T05:15:48.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:48.543+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:48.544064+0000) 2019-10-13T05:15:50.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:50.118+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:50.119240+0000) 2019-10-13T05:15:51.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:51.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:51.795109+0000) 2019-10-13T05:15:52.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:52.663+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.664649+0000) 2019-10-13T05:15:52.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:52.663+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.664716+0000) 2019-10-13T05:15:52.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:52.760+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.761945+0000) 2019-10-13T05:15:52.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:52.760+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.762173+0000) 2019-10-13T05:15:52.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:52.761+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.762512+0000) 2019-10-13T05:15:52.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:52.761+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.762427+0000) 2019-10-13T05:15:52.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:52.761+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.762790+0000) 2019-10-13T05:15:52.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:52.761+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.763127+0000) 2019-10-13T05:15:52.769 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:52.763+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:52.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:52.756+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.758242+0000) 2019-10-13T05:15:52.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:52.759+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:52.761149+0000) 2019-10-13T05:15:54.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:54.159+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:54.160266+0000) 2019-10-13T05:15:57.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:15:57.761+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.762275+0000) 2019-10-13T05:15:57.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:57.765+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.766893+0000) 2019-10-13T05:15:57.770 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:15:57.766+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.767557+0000) 2019-10-13T05:15:57.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:57.766+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.767652+0000) 2019-10-13T05:15:57.771 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:57.766+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.767764+0000) 2019-10-13T05:15:57.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:15:57.766+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.768074+0000) 2019-10-13T05:15:57.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:15:57.766+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.768302+0000) 2019-10-13T05:15:57.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:15:57.763+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:57.764662+0000) 2019-10-13T05:15:57.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:15:57.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73511 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:15:58.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:15:58.203+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:58.204175+0000) 2019-10-13T05:15:58.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:15:58.543+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:58.544142+0000) 2019-10-13T05:15:58.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:15:58.543+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:15:58.544212+0000) 2019-10-13T05:16:00.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:00.120+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:00.121243+0000) 2019-10-13T05:16:01.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:01.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:01.795298+0000) 2019-10-13T05:16:02.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:02.664+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.664776+0000) 2019-10-13T05:16:02.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:02.664+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.664920+0000) 2019-10-13T05:16:02.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:02.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.767421+0000) 2019-10-13T05:16:02.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:02.770+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.771730+0000) 2019-10-13T05:16:02.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:02.770+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.772225+0000) 2019-10-13T05:16:02.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:02.771+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.772478+0000) 2019-10-13T05:16:02.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:02.771+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.772777+0000) 2019-10-13T05:16:02.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:02.771+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.772965+0000) 2019-10-13T05:16:02.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:02.771+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.773096+0000) 2019-10-13T05:16:02.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:02.770+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:02.770909+0000) 2019-10-13T05:16:02.779 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:02.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:04.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:04.160+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:04.160446+0000) 2019-10-13T05:16:07.346 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:16:07.346 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:07.351 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:16:07.351 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:07.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:07.776+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.776769+0000) 2019-10-13T05:16:07.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:07.776+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.776918+0000) 2019-10-13T05:16:07.780 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:07.776+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.777077+0000) 2019-10-13T05:16:07.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:07.772+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.772972+0000) 2019-10-13T05:16:07.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:07.777+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.777652+0000) 2019-10-13T05:16:07.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:07.777+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.777802+0000) 2019-10-13T05:16:07.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:07.777+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.778032+0000) 2019-10-13T05:16:07.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:07.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73568 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:07.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:07.776+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:07.776828+0000) 2019-10-13T05:16:08.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:08.204+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:08.204344+0000) 2019-10-13T05:16:08.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:08.543+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:08.544312+0000) 2019-10-13T05:16:08.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:08.543+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:08.544352+0000) 2019-10-13T05:16:10.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:10.122+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:10.123439+0000) 2019-10-13T05:16:11.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:11.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:11.795491+0000) 2019-10-13T05:16:12.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:12.664+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.664955+0000) 2019-10-13T05:16:12.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:12.664+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.665098+0000) 2019-10-13T05:16:12.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:12.781+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.781578+0000) 2019-10-13T05:16:12.785 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:12.781+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.781919+0000) 2019-10-13T05:16:12.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:12.781+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.782064+0000) 2019-10-13T05:16:12.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:12.777+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.777860+0000) 2019-10-13T05:16:12.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:12.782+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.782743+0000) 2019-10-13T05:16:12.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:12.782+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.783052+0000) 2019-10-13T05:16:12.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:12.783+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.783470+0000) 2019-10-13T05:16:12.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:12.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73601 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:12.788 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:12.780+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:12.780945+0000) 2019-10-13T05:16:14.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:14.160+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:14.160631+0000) 2019-10-13T05:16:17.789 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:17.786+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.786482+0000) 2019-10-13T05:16:17.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:17.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.786734+0000) 2019-10-13T05:16:17.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:17.786+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.786908+0000) 2019-10-13T05:16:17.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:17.786+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.787022+0000) 2019-10-13T05:16:17.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:17.782+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.782725+0000) 2019-10-13T05:16:17.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:17.786+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.787362+0000) 2019-10-13T05:16:17.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:17.787+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.787592+0000) 2019-10-13T05:16:17.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:17.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73629 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:17.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:17.784+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:17.785539+0000) 2019-10-13T05:16:18.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:18.204+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:18.204512+0000) 2019-10-13T05:16:18.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:18.544+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:18.544475+0000) 2019-10-13T05:16:18.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:18.544+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:18.544536+0000) 2019-10-13T05:16:20.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:20.125+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:20.125951+0000) 2019-10-13T05:16:21.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:21.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:21.795695+0000) 2019-10-13T05:16:22.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:22.663+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.665156+0000) 2019-10-13T05:16:22.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:22.665+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.665279+0000) 2019-10-13T05:16:22.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:22.789+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.791201+0000) 2019-10-13T05:16:22.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:22.786+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.786922+0000) 2019-10-13T05:16:22.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:22.790+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.791885+0000) 2019-10-13T05:16:22.796 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:22.790+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.792160+0000) 2019-10-13T05:16:22.796 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:22.790+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.792025+0000) 2019-10-13T05:16:22.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:22.791+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.792450+0000) 2019-10-13T05:16:22.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:22.791+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.792601+0000) 2019-10-13T05:16:22.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:22.789+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:22.790235+0000) 2019-10-13T05:16:22.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:22.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73660 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:24.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:24.159+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:24.160823+0000) 2019-10-13T05:16:27.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:27.794+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.795840+0000) 2019-10-13T05:16:27.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:27.794+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.796069+0000) 2019-10-13T05:16:27.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:27.794+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.796360+0000) 2019-10-13T05:16:27.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:27.791+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.792371+0000) 2019-10-13T05:16:27.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:27.795+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.796622+0000) 2019-10-13T05:16:27.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:27.795+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.797000+0000) 2019-10-13T05:16:27.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:27.795+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.797141+0000) 2019-10-13T05:16:27.802 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:27.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:27.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:27.794+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:27.795065+0000) 2019-10-13T05:16:28.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:28.203+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:28.204684+0000) 2019-10-13T05:16:28.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:28.543+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:28.544647+0000) 2019-10-13T05:16:28.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:28.543+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:28.544680+0000) 2019-10-13T05:16:30.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:30.128+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:30.128272+0000) 2019-10-13T05:16:31.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:31.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:31.795883+0000) 2019-10-13T05:16:32.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:32.664+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.665393+0000) 2019-10-13T05:16:32.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:32.664+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.665444+0000) 2019-10-13T05:16:32.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:32.799+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.800999+0000) 2019-10-13T05:16:32.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:32.795+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.796852+0000) 2019-10-13T05:16:32.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:32.797+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.798840+0000) 2019-10-13T05:16:32.809 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:32.800+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.801512+0000) 2019-10-13T05:16:32.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:32.800+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.801735+0000) 2019-10-13T05:16:32.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:32.800+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.801944+0000) 2019-10-13T05:16:32.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:32.800+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.802026+0000) 2019-10-13T05:16:32.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:32.801+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:32.802524+0000) 2019-10-13T05:16:32.810 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:32.802+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:34.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:34.160+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:34.161030+0000) 2019-10-13T05:16:37.477 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:16:37.477 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:37.484 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:16:37.484 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:37.809 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:37.804+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.805904+0000) 2019-10-13T05:16:37.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:37.804+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.806057+0000) 2019-10-13T05:16:37.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:37.804+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.806212+0000) 2019-10-13T05:16:37.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:37.805+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.806534+0000) 2019-10-13T05:16:37.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:37.805+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.806859+0000) 2019-10-13T05:16:37.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:37.805+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.807059+0000) 2019-10-13T05:16:37.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:37.800+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.802197+0000) 2019-10-13T05:16:37.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:37.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73749 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:37.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:37.802+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:37.804677+0000) 2019-10-13T05:16:38.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:38.203+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:38.204836+0000) 2019-10-13T05:16:38.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:38.543+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:38.544804+0000) 2019-10-13T05:16:38.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:38.543+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:38.544824+0000) 2019-10-13T05:16:40.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:40.129+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:40.130653+0000) 2019-10-13T05:16:41.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:41.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:41.796035+0000) 2019-10-13T05:16:42.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:42.664+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.665560+0000) 2019-10-13T05:16:42.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:42.664+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.665613+0000) 2019-10-13T05:16:42.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:42.804+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.805776+0000) 2019-10-13T05:16:42.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:42.808+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.810343+0000) 2019-10-13T05:16:42.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:42.809+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.810565+0000) 2019-10-13T05:16:42.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:42.809+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.810754+0000) 2019-10-13T05:16:42.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:42.809+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.810864+0000) 2019-10-13T05:16:42.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:42.809+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.811189+0000) 2019-10-13T05:16:42.815 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:42.810+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.811598+0000) 2019-10-13T05:16:42.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:42.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:42.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:42.807+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:42.809507+0000) 2019-10-13T05:16:44.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:44.159+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:44.161222+0000) 2019-10-13T05:16:47.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:47.813+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.814964+0000) 2019-10-13T05:16:47.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:47.813+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.815305+0000) 2019-10-13T05:16:47.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:47.810+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.811473+0000) 2019-10-13T05:16:47.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:47.814+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.815547+0000) 2019-10-13T05:16:47.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:47.814+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.815804+0000) 2019-10-13T05:16:47.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:47.814+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.816179+0000) 2019-10-13T05:16:47.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:47.814+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.816333+0000) 2019-10-13T05:16:47.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:47.812+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:47.813369+0000) 2019-10-13T05:16:47.821 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:47.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73808 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:48.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:48.204+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:48.204980+0000) 2019-10-13T05:16:48.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:48.544+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:48.544991+0000) 2019-10-13T05:16:48.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:48.544+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:48.545012+0000) 2019-10-13T05:16:50.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:50.132+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:50.133195+0000) 2019-10-13T05:16:51.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:51.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:51.796210+0000) 2019-10-13T05:16:52.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:52.665+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.665760+0000) 2019-10-13T05:16:52.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:52.665+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.665797+0000) 2019-10-13T05:16:52.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:52.818+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.819370+0000) 2019-10-13T05:16:52.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:52.819+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.819520+0000) 2019-10-13T05:16:52.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:52.819+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.819837+0000) 2019-10-13T05:16:52.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:52.814+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.815523+0000) 2019-10-13T05:16:52.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:52.819+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.820231+0000) 2019-10-13T05:16:52.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:52.820+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.820666+0000) 2019-10-13T05:16:52.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:52.820+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.820806+0000) 2019-10-13T05:16:52.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:52.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:52.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:52.817+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:52.818180+0000) 2019-10-13T05:16:54.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:54.161+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:54.161410+0000) 2019-10-13T05:16:57.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:16:57.819+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.819768+0000) 2019-10-13T05:16:57.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:57.823+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.824157+0000) 2019-10-13T05:16:57.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:16:57.824+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.824527+0000) 2019-10-13T05:16:57.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:57.824+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.824710+0000) 2019-10-13T05:16:57.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:16:57.824+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.824909+0000) 2019-10-13T05:16:57.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:57.824+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.825374+0000) 2019-10-13T05:16:57.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:16:57.825+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.825624+0000) 2019-10-13T05:16:57.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:16:57.821+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:57.822693+0000) 2019-10-13T05:16:57.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:16:57.826+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:16:58.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:16:58.204+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:58.205166+0000) 2019-10-13T05:16:58.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:16:58.544+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:58.545114+0000) 2019-10-13T05:16:58.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:16:58.544+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:16:58.545160+0000) 2019-10-13T05:17:00.143 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:00.135+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:00.135538+0000) 2019-10-13T05:17:01.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:01.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:01.796394+0000) 2019-10-13T05:17:02.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:02.665+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.665942+0000) 2019-10-13T05:17:02.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:02.665+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.665972+0000) 2019-10-13T05:17:02.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:02.828+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.829314+0000) 2019-10-13T05:17:02.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:02.824+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.825028+0000) 2019-10-13T05:17:02.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:02.829+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.829760+0000) 2019-10-13T05:17:02.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:02.829+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.829888+0000) 2019-10-13T05:17:02.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:02.829+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.829989+0000) 2019-10-13T05:17:02.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:02.829+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.830245+0000) 2019-10-13T05:17:02.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:02.829+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.830383+0000) 2019-10-13T05:17:02.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:02.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:02.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:02.827+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:02.827771+0000) 2019-10-13T05:17:04.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:04.161+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:04.161585+0000) 2019-10-13T05:17:07.586 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:17:07.586 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:07.591 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:17:07.592 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:07.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:07.828+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.829447+0000) 2019-10-13T05:17:07.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:07.832+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.834103+0000) 2019-10-13T05:17:07.838 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:07.833+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.834524+0000) 2019-10-13T05:17:07.838 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:07.833+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.834682+0000) 2019-10-13T05:17:07.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:07.833+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.834876+0000) 2019-10-13T05:17:07.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:07.833+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.834979+0000) 2019-10-13T05:17:07.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:07.833+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.835339+0000) 2019-10-13T05:17:07.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:07.831+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:07.831949+0000) 2019-10-13T05:17:07.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:07.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:08.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:08.204+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:08.205337+0000) 2019-10-13T05:17:08.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:08.544+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:08.545291+0000) 2019-10-13T05:17:08.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:08.544+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:08.545313+0000) 2019-10-13T05:17:10.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:10.137+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:10.138046+0000) 2019-10-13T05:17:11.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:11.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:11.796586+0000) 2019-10-13T05:17:12.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:12.665+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.666148+0000) 2019-10-13T05:17:12.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:12.665+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.666108+0000) 2019-10-13T05:17:12.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:12.833+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.834277+0000) 2019-10-13T05:17:12.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:12.837+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.838967+0000) 2019-10-13T05:17:12.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:12.837+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.839130+0000) 2019-10-13T05:17:12.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:12.837+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.839289+0000) 2019-10-13T05:17:12.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:12.838+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.839676+0000) 2019-10-13T05:17:12.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:12.838+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.839808+0000) 2019-10-13T05:17:12.843 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:12.838+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.840130+0000) 2019-10-13T05:17:12.844 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:12.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:12.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:12.837+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:12.837973+0000) 2019-10-13T05:17:14.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:14.160+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:14.161746+0000) 2019-10-13T05:17:17.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:17.842+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.844287+0000) 2019-10-13T05:17:17.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:17.843+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.844427+0000) 2019-10-13T05:17:17.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:17.843+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.844753+0000) 2019-10-13T05:17:17.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:17.839+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.840159+0000) 2019-10-13T05:17:17.849 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:17.843+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.844829+0000) 2019-10-13T05:17:17.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:17.843+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.845007+0000) 2019-10-13T05:17:17.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:17.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.845600+0000) 2019-10-13T05:17:17.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:17.841+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:17.841887+0000) 2019-10-13T05:17:17.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:17.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 73989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:18.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:18.204+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:18.205539+0000) 2019-10-13T05:17:18.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:18.544+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:18.545463+0000) 2019-10-13T05:17:18.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:18.544+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:18.545488+0000) 2019-10-13T05:17:20.148 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:20.140+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:20.140409+0000) 2019-10-13T05:17:21.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:21.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:21.796798+0000) 2019-10-13T05:17:22.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:22.665+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.666288+0000) 2019-10-13T05:17:22.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:22.665+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.666297+0000) 2019-10-13T05:17:22.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:22.847+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.848632+0000) 2019-10-13T05:17:22.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:22.847+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.848906+0000) 2019-10-13T05:17:22.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:22.847+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.849141+0000) 2019-10-13T05:17:22.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:22.843+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.844849+0000) 2019-10-13T05:17:22.854 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:22.848+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.849483+0000) 2019-10-13T05:17:22.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:22.848+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.849849+0000) 2019-10-13T05:17:22.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:22.848+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.850098+0000) 2019-10-13T05:17:22.855 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:22.844+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:22.846719+0000) 2019-10-13T05:17:22.855 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:22.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74020 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:24.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:24.160+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:24.161929+0000) 2019-10-13T05:17:27.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:27.847+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.848807+0000) 2019-10-13T05:17:27.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:27.852+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.853563+0000) 2019-10-13T05:17:27.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:27.852+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.853816+0000) 2019-10-13T05:17:27.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:27.852+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.853952+0000) 2019-10-13T05:17:27.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:27.852+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.854080+0000) 2019-10-13T05:17:27.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:27.852+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.854379+0000) 2019-10-13T05:17:27.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:27.853+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.854595+0000) 2019-10-13T05:17:27.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:27.850+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:27.852109+0000) 2019-10-13T05:17:27.859 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:27.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:28.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:28.205+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:28.205707+0000) 2019-10-13T05:17:28.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:28.544+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:28.545594+0000) 2019-10-13T05:17:28.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:28.544+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:28.545657+0000) 2019-10-13T05:17:30.150 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:30.142+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:30.142915+0000) 2019-10-13T05:17:31.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:31.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:31.796983+0000) 2019-10-13T05:17:32.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:32.666+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.666473+0000) 2019-10-13T05:17:32.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:32.665+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.666483+0000) 2019-10-13T05:17:32.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:32.852+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.854442+0000) 2019-10-13T05:17:32.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:32.857+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.858826+0000) 2019-10-13T05:17:32.863 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:32.857+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.859185+0000) 2019-10-13T05:17:32.863 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:32.857+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.859346+0000) 2019-10-13T05:17:32.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:32.858+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.859605+0000) 2019-10-13T05:17:32.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:32.858+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.859849+0000) 2019-10-13T05:17:32.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:32.857+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.859074+0000) 2019-10-13T05:17:32.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:32.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:32.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:32.856+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:32.857798+0000) 2019-10-13T05:17:34.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:34.161+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:34.162127+0000) 2019-10-13T05:17:37.732 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:17:37.733 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:37.749 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:17:37.749 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:37.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:37.862+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.863136+0000) 2019-10-13T05:17:37.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:37.862+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.863336+0000) 2019-10-13T05:17:37.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:37.863+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.863534+0000) 2019-10-13T05:17:37.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:37.859+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.859965+0000) 2019-10-13T05:17:37.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:37.863+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.863910+0000) 2019-10-13T05:17:37.868 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:37.863+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.864082+0000) 2019-10-13T05:17:37.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:37.864+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.864760+0000) 2019-10-13T05:17:37.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:37.860+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:37.861174+0000) 2019-10-13T05:17:37.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:37.865+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:38.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:38.205+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:38.205891+0000) 2019-10-13T05:17:38.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:38.545+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:38.545713+0000) 2019-10-13T05:17:38.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:38.545+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:38.545782+0000) 2019-10-13T05:17:40.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:40.144+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:40.145153+0000) 2019-10-13T05:17:41.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:41.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:41.797166+0000) 2019-10-13T05:17:42.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:42.666+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.666683+0000) 2019-10-13T05:17:42.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:42.666+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.666664+0000) 2019-10-13T05:17:42.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:42.867+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.867670+0000) 2019-10-13T05:17:42.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:42.867+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.867929+0000) 2019-10-13T05:17:42.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:42.867+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.868130+0000) 2019-10-13T05:17:42.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:42.862+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.863050+0000) 2019-10-13T05:17:42.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:42.867+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.868350+0000) 2019-10-13T05:17:42.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:42.868+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.868543+0000) 2019-10-13T05:17:42.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:42.868+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.868635+0000) 2019-10-13T05:17:42.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:42.870+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:42.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:42.865+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:42.866500+0000) 2019-10-13T05:17:44.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:44.161+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:44.162375+0000) 2019-10-13T05:17:47.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:47.867+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.868025+0000) 2019-10-13T05:17:47.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:47.872+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.872648+0000) 2019-10-13T05:17:47.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:47.872+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.873077+0000) 2019-10-13T05:17:47.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:47.873+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.873473+0000) 2019-10-13T05:17:47.877 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:47.873+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.873533+0000) 2019-10-13T05:17:47.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:47.873+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.873697+0000) 2019-10-13T05:17:47.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:47.873+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.873826+0000) 2019-10-13T05:17:47.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:47.869+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:47.869910+0000) 2019-10-13T05:17:47.879 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:47.875+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:48.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:48.205+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:48.206090+0000) 2019-10-13T05:17:48.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:48.545+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:48.545884+0000) 2019-10-13T05:17:48.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:48.545+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:48.545917+0000) 2019-10-13T05:17:50.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:50.146+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:50.147378+0000) 2019-10-13T05:17:51.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:51.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:51.797379+0000) 2019-10-13T05:17:52.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:52.665+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.666824+0000) 2019-10-13T05:17:52.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:52.666+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.666843+0000) 2019-10-13T05:17:52.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:52.876+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.877694+0000) 2019-10-13T05:17:52.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:52.876+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.877617+0000) 2019-10-13T05:17:52.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:52.876+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.878007+0000) 2019-10-13T05:17:52.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:52.876+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.878364+0000) 2019-10-13T05:17:52.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:52.876+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.878367+0000) 2019-10-13T05:17:52.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:52.877+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.878719+0000) 2019-10-13T05:17:52.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:52.872+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.873690+0000) 2019-10-13T05:17:52.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:52.875+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:52.876163+0000) 2019-10-13T05:17:52.887 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:52.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:54.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:54.161+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:54.162600+0000) 2019-10-13T05:17:57.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:57.880+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.882357+0000) 2019-10-13T05:17:57.887 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:17:57.881+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.882678+0000) 2019-10-13T05:17:57.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:17:57.877+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.878310+0000) 2019-10-13T05:17:57.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:17:57.879+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.880681+0000) 2019-10-13T05:17:57.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:57.881+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.883289+0000) 2019-10-13T05:17:57.890 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:57.882+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.883547+0000) 2019-10-13T05:17:57.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:17:57.882+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.883723+0000) 2019-10-13T05:17:57.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:17:57.882+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:57.883919+0000) 2019-10-13T05:17:57.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:17:57.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74229 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:17:58.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:17:58.205+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:58.206268+0000) 2019-10-13T05:17:58.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:17:58.545+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:58.546047+0000) 2019-10-13T05:17:58.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:17:58.545+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:17:58.546068+0000) 2019-10-13T05:18:00.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:00.149+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:00.149906+0000) 2019-10-13T05:18:01.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:01.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:01.797584+0000) 2019-10-13T05:18:02.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:02.666+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.667030+0000) 2019-10-13T05:18:02.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:02.666+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.667055+0000) 2019-10-13T05:18:02.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:02.886+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.887465+0000) 2019-10-13T05:18:02.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:02.882+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.883165+0000) 2019-10-13T05:18:02.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:02.886+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.887884+0000) 2019-10-13T05:18:02.892 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:02.886+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.888207+0000) 2019-10-13T05:18:02.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:02.887+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.888516+0000) 2019-10-13T05:18:02.892 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:02.886+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.888122+0000) 2019-10-13T05:18:02.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:02.887+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.888719+0000) 2019-10-13T05:18:02.893 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:02.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74260 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:02.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:02.885+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:02.886278+0000) 2019-10-13T05:18:04.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:04.162+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:04.162792+0000) 2019-10-13T05:18:07.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:07.886+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.887696+0000) 2019-10-13T05:18:07.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:07.890+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.892339+0000) 2019-10-13T05:18:07.896 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:07.891+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.892829+0000) 2019-10-13T05:18:07.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:07.891+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.892929+0000) 2019-10-13T05:18:07.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:07.891+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.893296+0000) 2019-10-13T05:18:07.897 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:07.892+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.893507+0000) 2019-10-13T05:18:07.897 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:07.892+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.893973+0000) 2019-10-13T05:18:07.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:07.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:07.899 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:07.890+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:07.891246+0000) 2019-10-13T05:18:07.952 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:18:07.953 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:07.957 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:18:07.957 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:08.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:08.205+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:08.206433+0000) 2019-10-13T05:18:08.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:08.545+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:08.546194+0000) 2019-10-13T05:18:08.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:08.545+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:08.546211+0000) 2019-10-13T05:18:10.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:10.152+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:10.152354+0000) 2019-10-13T05:18:11.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:11.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:11.797809+0000) 2019-10-13T05:18:12.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:12.666+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.667239+0000) 2019-10-13T05:18:12.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:12.666+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.667232+0000) 2019-10-13T05:18:12.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:12.895+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.897010+0000) 2019-10-13T05:18:12.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:12.896+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.897461+0000) 2019-10-13T05:18:12.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:12.896+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.897503+0000) 2019-10-13T05:18:12.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:12.896+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.897673+0000) 2019-10-13T05:18:12.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:12.896+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.897783+0000) 2019-10-13T05:18:12.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:12.892+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.893859+0000) 2019-10-13T05:18:12.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:12.897+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.898433+0000) 2019-10-13T05:18:12.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:12.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:12.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:12.894+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:12.895851+0000) 2019-10-13T05:18:14.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:14.162+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:14.162977+0000) 2019-10-13T05:18:17.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:17.900+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.901677+0000) 2019-10-13T05:18:17.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:17.900+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.901903+0000) 2019-10-13T05:18:17.906 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:17.900+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.902094+0000) 2019-10-13T05:18:17.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:17.896+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.897779+0000) 2019-10-13T05:18:17.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:17.901+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.902508+0000) 2019-10-13T05:18:17.908 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:17.901+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.902963+0000) 2019-10-13T05:18:17.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:17.901+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.903150+0000) 2019-10-13T05:18:17.908 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:17.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:17.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:17.898+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:17.900392+0000) 2019-10-13T05:18:18.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:18.206+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:18.206609+0000) 2019-10-13T05:18:18.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:18.545+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:18.546369+0000) 2019-10-13T05:18:18.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:18.545+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:18.546377+0000) 2019-10-13T05:18:20.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:20.153+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:20.154711+0000) 2019-10-13T05:18:21.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:21.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:21.797992+0000) 2019-10-13T05:18:22.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:22.667+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.667424+0000) 2019-10-13T05:18:22.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:22.666+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.667430+0000) 2019-10-13T05:18:22.909 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:22.905+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.906141+0000) 2019-10-13T05:18:22.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:22.905+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.906348+0000) 2019-10-13T05:18:22.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:22.900+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.902044+0000) 2019-10-13T05:18:22.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:22.906+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.906773+0000) 2019-10-13T05:18:22.911 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:22.906+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.906644+0000) 2019-10-13T05:18:22.911 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:22.906+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.907116+0000) 2019-10-13T05:18:22.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:22.906+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.907389+0000) 2019-10-13T05:18:22.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:22.903+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:22.904802+0000) 2019-10-13T05:18:22.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:22.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74380 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:24.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:24.162+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:24.163164+0000) 2019-10-13T05:18:27.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:27.910+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.910889+0000) 2019-10-13T05:18:27.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:27.910+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.911294+0000) 2019-10-13T05:18:27.915 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:27.911+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.911485+0000) 2019-10-13T05:18:27.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:27.911+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.911775+0000) 2019-10-13T05:18:27.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:27.906+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.907178+0000) 2019-10-13T05:18:27.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:27.908+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.908843+0000) 2019-10-13T05:18:27.918 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:27.911+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.911991+0000) 2019-10-13T05:18:27.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:27.912+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:27.912480+0000) 2019-10-13T05:18:27.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:27.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:28.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:28.206+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:28.206786+0000) 2019-10-13T05:18:28.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:28.546+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:28.546537+0000) 2019-10-13T05:18:28.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:28.546+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:28.546536+0000) 2019-10-13T05:18:30.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:30.156+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:30.157010+0000) 2019-10-13T05:18:31.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:31.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:31.798175+0000) 2019-10-13T05:18:32.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:32.667+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.667616+0000) 2019-10-13T05:18:32.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:32.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.667642+0000) 2019-10-13T05:18:32.919 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:32.915+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.916024+0000) 2019-10-13T05:18:32.919 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:32.915+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.916290+0000) 2019-10-13T05:18:32.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:32.915+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.916336+0000) 2019-10-13T05:18:32.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:32.911+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.912041+0000) 2019-10-13T05:18:32.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:32.916+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.916813+0000) 2019-10-13T05:18:32.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:32.916+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.917147+0000) 2019-10-13T05:18:32.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:32.916+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.917367+0000) 2019-10-13T05:18:32.922 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:32.913+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:32.914324+0000) 2019-10-13T05:18:32.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:32.918+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:34.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:34.163+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:34.163357+0000) 2019-10-13T05:18:37.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:37.919+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.920703+0000) 2019-10-13T05:18:37.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:37.919+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.920941+0000) 2019-10-13T05:18:37.924 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:37.919+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.921117+0000) 2019-10-13T05:18:37.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:37.916+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.916812+0000) 2019-10-13T05:18:37.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:37.920+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.921538+0000) 2019-10-13T05:18:37.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:37.920+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.921637+0000) 2019-10-13T05:18:37.926 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:37.920+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.921841+0000) 2019-10-13T05:18:37.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:37.918+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:37.919053+0000) 2019-10-13T05:18:37.926 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:37.922+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74469 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:38.041 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:18:38.041 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:38.046 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:18:38.046 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:38.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:38.205+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:38.206948+0000) 2019-10-13T05:18:38.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:38.545+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:38.546643+0000) 2019-10-13T05:18:38.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:38.545+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:38.546663+0000) 2019-10-13T05:18:40.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:40.158+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:40.159362+0000) 2019-10-13T05:18:41.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:41.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:41.798370+0000) 2019-10-13T05:18:42.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:42.666+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.667802+0000) 2019-10-13T05:18:42.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:42.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.667790+0000) 2019-10-13T05:18:42.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:42.923+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.925115+0000) 2019-10-13T05:18:42.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:42.923+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.925035+0000) 2019-10-13T05:18:42.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:42.924+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.925421+0000) 2019-10-13T05:18:42.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:42.924+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.925784+0000) 2019-10-13T05:18:42.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:42.924+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.926165+0000) 2019-10-13T05:18:42.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:42.921+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.922146+0000) 2019-10-13T05:18:42.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:42.925+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.926482+0000) 2019-10-13T05:18:42.932 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:42.922+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:42.923334+0000) 2019-10-13T05:18:42.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:42.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:44.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:44.162+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:44.163579+0000) 2019-10-13T05:18:47.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:47.928+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.929677+0000) 2019-10-13T05:18:47.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:47.928+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.929865+0000) 2019-10-13T05:18:47.933 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:47.928+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.930222+0000) 2019-10-13T05:18:47.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:47.925+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.925958+0000) 2019-10-13T05:18:47.934 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:47.929+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.930769+0000) 2019-10-13T05:18:47.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:47.929+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.930967+0000) 2019-10-13T05:18:47.935 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:47.929+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.931277+0000) 2019-10-13T05:18:47.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:47.927+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:47.928438+0000) 2019-10-13T05:18:47.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:47.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:48.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:48.206+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:48.207079+0000) 2019-10-13T05:18:48.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:48.546+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:48.546812+0000) 2019-10-13T05:18:48.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:48.546+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:48.546834+0000) 2019-10-13T05:18:50.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:50.161+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:50.161771+0000) 2019-10-13T05:18:51.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:51.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:51.798591+0000) 2019-10-13T05:18:52.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:52.667+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.668008+0000) 2019-10-13T05:18:52.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:52.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.667966+0000) 2019-10-13T05:18:52.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:52.933+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.934756+0000) 2019-10-13T05:18:52.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:52.933+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.934939+0000) 2019-10-13T05:18:52.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:52.933+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.935125+0000) 2019-10-13T05:18:52.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:52.930+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.930786+0000) 2019-10-13T05:18:52.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:52.934+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.935454+0000) 2019-10-13T05:18:52.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:52.934+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.935723+0000) 2019-10-13T05:18:52.940 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:52.934+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.935997+0000) 2019-10-13T05:18:52.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:52.936+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74561 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:52.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:52.933+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:52.934019+0000) 2019-10-13T05:18:54.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:54.163+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:54.163770+0000) 2019-10-13T05:18:57.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:18:57.934+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.935039+0000) 2019-10-13T05:18:57.943 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:57.938+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.939408+0000) 2019-10-13T05:18:57.943 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:57.938+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.939757+0000) 2019-10-13T05:18:57.943 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:18:57.938+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.940015+0000) 2019-10-13T05:18:57.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:18:57.939+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.940481+0000) 2019-10-13T05:18:57.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:57.939+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.940634+0000) 2019-10-13T05:18:57.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:18:57.939+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.940766+0000) 2019-10-13T05:18:57.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:18:57.936+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:57.937699+0000) 2019-10-13T05:18:57.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:18:57.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:18:58.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:18:58.206+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:58.207221+0000) 2019-10-13T05:18:58.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:18:58.546+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:58.546996+0000) 2019-10-13T05:18:58.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:18:58.546+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:18:58.547001+0000) 2019-10-13T05:19:00.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:00.164+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:00.164173+0000) 2019-10-13T05:19:01.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:01.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:01.798784+0000) 2019-10-13T05:19:02.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:02.667+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.668192+0000) 2019-10-13T05:19:02.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:02.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.668149+0000) 2019-10-13T05:19:02.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:02.942+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.943950+0000) 2019-10-13T05:19:02.948 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:02.942+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.944098+0000) 2019-10-13T05:19:02.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:02.943+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.944849+0000) 2019-10-13T05:19:02.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:02.938+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.940128+0000) 2019-10-13T05:19:02.949 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:02.943+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.945090+0000) 2019-10-13T05:19:02.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:02.943+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.945242+0000) 2019-10-13T05:19:02.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:02.944+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.945825+0000) 2019-10-13T05:19:02.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:02.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:02.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:02.941+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:02.943002+0000) 2019-10-13T05:19:04.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:04.163+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:04.163990+0000) 2019-10-13T05:19:07.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:07.943+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.944849+0000) 2019-10-13T05:19:07.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:07.949+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.949586+0000) 2019-10-13T05:19:07.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:07.949+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.949714+0000) 2019-10-13T05:19:07.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:07.949+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.949841+0000) 2019-10-13T05:19:07.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:07.949+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.950008+0000) 2019-10-13T05:19:07.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:07.950+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.950419+0000) 2019-10-13T05:19:07.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:07.950+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.950612+0000) 2019-10-13T05:19:07.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:07.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74648 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:07.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:07.946+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:07.947921+0000) 2019-10-13T05:19:08.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:08.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:08.207367+0000) 2019-10-13T05:19:08.264 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:19:08.264 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:08.269 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:19:08.269 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:08.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:08.546+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:08.547170+0000) 2019-10-13T05:19:08.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:08.546+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:08.547177+0000) 2019-10-13T05:19:10.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:10.165+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:10.166684+0000) 2019-10-13T05:19:11.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:11.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:11.799002+0000) 2019-10-13T05:19:12.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:12.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.668363+0000) 2019-10-13T05:19:12.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:12.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.668306+0000) 2019-10-13T05:19:12.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:12.946+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.948374+0000) 2019-10-13T05:19:12.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:12.952+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.953012+0000) 2019-10-13T05:19:12.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:12.952+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.953184+0000) 2019-10-13T05:19:12.957 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:12.953+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.953732+0000) 2019-10-13T05:19:12.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:12.953+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.953980+0000) 2019-10-13T05:19:12.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:12.953+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.954272+0000) 2019-10-13T05:19:12.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:12.954+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.954402+0000) 2019-10-13T05:19:12.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:12.955+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74681 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:12.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:12.950+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:12.951832+0000) 2019-10-13T05:19:14.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:14.163+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:14.164189+0000) 2019-10-13T05:19:17.960 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:17.957+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.957404+0000) 2019-10-13T05:19:17.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:17.957+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.957657+0000) 2019-10-13T05:19:17.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:17.952+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.953547+0000) 2019-10-13T05:19:17.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:17.958+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.958417+0000) 2019-10-13T05:19:17.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:17.958+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.958750+0000) 2019-10-13T05:19:17.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:17.958+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.958900+0000) 2019-10-13T05:19:17.962 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:17.958+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.959104+0000) 2019-10-13T05:19:17.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:17.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74709 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:17.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:17.955+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:17.955862+0000) 2019-10-13T05:19:18.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:18.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:18.207517+0000) 2019-10-13T05:19:18.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:18.547+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:18.547346+0000) 2019-10-13T05:19:18.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:18.547+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:18.547353+0000) 2019-10-13T05:19:20.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:20.168+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:20.169076+0000) 2019-10-13T05:19:21.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:21.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:21.799193+0000) 2019-10-13T05:19:22.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:22.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.668507+0000) 2019-10-13T05:19:22.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:22.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.668476+0000) 2019-10-13T05:19:22.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:22.960+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.961939+0000) 2019-10-13T05:19:22.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:22.960+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.962104+0000) 2019-10-13T05:19:22.965 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:22.960+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.962339+0000) 2019-10-13T05:19:22.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:22.957+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.958093+0000) 2019-10-13T05:19:22.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:22.961+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.962872+0000) 2019-10-13T05:19:22.967 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:22.961+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.962760+0000) 2019-10-13T05:19:22.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:22.961+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.963047+0000) 2019-10-13T05:19:22.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:22.963+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74740 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:22.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:22.959+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:22.960754+0000) 2019-10-13T05:19:24.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:24.163+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:24.164383+0000) 2019-10-13T05:19:27.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:27.961+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.962043+0000) 2019-10-13T05:19:27.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:27.965+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.966914+0000) 2019-10-13T05:19:27.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:27.965+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.967062+0000) 2019-10-13T05:19:27.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:27.965+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.967259+0000) 2019-10-13T05:19:27.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:27.966+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.967474+0000) 2019-10-13T05:19:27.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:27.966+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.967691+0000) 2019-10-13T05:19:27.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:27.966+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.967738+0000) 2019-10-13T05:19:27.972 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:27.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74768 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:27.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:27.964+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:27.965574+0000) 2019-10-13T05:19:28.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:28.206+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:28.207680+0000) 2019-10-13T05:19:28.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:28.546+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:28.547541+0000) 2019-10-13T05:19:28.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:28.546+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:28.547537+0000) 2019-10-13T05:19:30.179 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:30.170+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:30.171434+0000) 2019-10-13T05:19:31.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:31.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:31.799387+0000) 2019-10-13T05:19:32.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:32.667+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.668684+0000) 2019-10-13T05:19:32.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:32.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.668654+0000) 2019-10-13T05:19:32.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:32.966+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.967335+0000) 2019-10-13T05:19:32.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:32.969+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.971296+0000) 2019-10-13T05:19:32.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:32.970+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.971549+0000) 2019-10-13T05:19:32.977 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:32.970+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.971654+0000) 2019-10-13T05:19:32.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:32.970+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.972023+0000) 2019-10-13T05:19:32.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:32.970+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.972241+0000) 2019-10-13T05:19:32.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:32.971+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.972493+0000) 2019-10-13T05:19:32.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:32.969+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:32.970073+0000) 2019-10-13T05:19:32.978 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:32.973+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:34.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:34.163+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:34.164558+0000) 2019-10-13T05:19:37.979 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:37.974+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.976171+0000) 2019-10-13T05:19:37.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:37.971+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.972122+0000) 2019-10-13T05:19:37.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:37.975+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.976809+0000) 2019-10-13T05:19:37.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:37.975+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.976919+0000) 2019-10-13T05:19:37.980 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:37.976+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.977446+0000) 2019-10-13T05:19:37.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:37.975+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.977253+0000) 2019-10-13T05:19:37.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:37.976+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.977753+0000) 2019-10-13T05:19:37.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:37.973+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:37.973979+0000) 2019-10-13T05:19:37.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:37.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:38.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:38.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:38.207865+0000) 2019-10-13T05:19:38.469 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:19:38.469 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:38.473 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:19:38.473 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:38.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:38.546+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:38.547710+0000) 2019-10-13T05:19:38.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:38.546+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:38.547706+0000) 2019-10-13T05:19:40.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:40.173+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:40.173857+0000) 2019-10-13T05:19:41.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:41.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:41.799600+0000) 2019-10-13T05:19:42.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:42.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.668847+0000) 2019-10-13T05:19:42.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:42.668+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.668807+0000) 2019-10-13T05:19:42.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:42.976+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.977255+0000) 2019-10-13T05:19:42.985 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:42.980+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.981979+0000) 2019-10-13T05:19:42.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:42.980+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.982070+0000) 2019-10-13T05:19:42.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:42.980+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.982351+0000) 2019-10-13T05:19:42.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:42.981+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.982591+0000) 2019-10-13T05:19:42.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:42.981+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.982749+0000) 2019-10-13T05:19:42.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:42.981+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.982910+0000) 2019-10-13T05:19:42.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:42.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74860 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:42.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:42.980+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:42.980846+0000) 2019-10-13T05:19:44.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:44.164+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:44.164757+0000) 2019-10-13T05:19:47.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:47.981+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.981940+0000) 2019-10-13T05:19:47.991 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:47.985+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.986618+0000) 2019-10-13T05:19:47.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:47.985+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.986894+0000) 2019-10-13T05:19:47.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:47.985+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.987097+0000) 2019-10-13T05:19:47.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:47.986+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.987429+0000) 2019-10-13T05:19:47.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:47.986+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.987535+0000) 2019-10-13T05:19:47.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:47.986+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.987841+0000) 2019-10-13T05:19:47.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:47.984+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:47.984763+0000) 2019-10-13T05:19:47.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:47.987+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:48.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:48.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:48.208040+0000) 2019-10-13T05:19:48.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:48.547+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:48.547897+0000) 2019-10-13T05:19:48.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:48.547+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:48.547899+0000) 2019-10-13T05:19:50.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:50.176+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:50.176444+0000) 2019-10-13T05:19:51.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:51.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:51.799799+0000) 2019-10-13T05:19:52.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:52.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.669045+0000) 2019-10-13T05:19:52.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:52.667+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.669020+0000) 2019-10-13T05:19:52.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:52.984+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.986582+0000) 2019-10-13T05:19:52.995 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:52.991+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.991513+0000) 2019-10-13T05:19:52.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:52.991+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.991600+0000) 2019-10-13T05:19:52.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:52.991+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.991774+0000) 2019-10-13T05:19:52.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:52.991+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.991879+0000) 2019-10-13T05:19:52.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:52.991+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.992035+0000) 2019-10-13T05:19:52.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:52.992+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.992584+0000) 2019-10-13T05:19:52.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:52.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:52.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:52.988+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:52.989947+0000) 2019-10-13T05:19:54.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:54.164+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:54.164959+0000) 2019-10-13T05:19:57.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:19:57.995+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.995815+0000) 2019-10-13T05:19:57.999 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:57.995+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.996218+0000) 2019-10-13T05:19:58.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:19:57.990+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.991881+0000) 2019-10-13T05:19:58.000 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:57.996+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.996553+0000) 2019-10-13T05:19:58.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:19:57.996+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.996707+0000) 2019-10-13T05:19:58.001 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:19:57.996+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.997113+0000) 2019-10-13T05:19:58.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:57.996+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.997325+0000) 2019-10-13T05:19:58.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:19:57.992+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:57.994429+0000) 2019-10-13T05:19:58.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:19:57.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:19:58.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:19:58.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:58.208183+0000) 2019-10-13T05:19:58.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:19:58.547+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:58.548068+0000) 2019-10-13T05:19:58.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:19:58.547+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:19:58.548067+0000) 2019-10-13T05:20:00.186 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:00.177+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:00.178826+0000) 2019-10-13T05:20:01.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:01.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:01.799986+0000) 2019-10-13T05:20:02.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:02.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:02.669211+0000) 2019-10-13T05:20:02.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:02.668+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:02.669169+0000) 2019-10-13T05:20:03.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:03.000+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:03.001132+0000) 2019-10-13T05:20:03.005 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:03.000+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:03.001017+0000) 2019-10-13T05:20:03.005 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:03.001+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:03.001431+0000) 2019-10-13T05:20:03.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:03.001+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:03.001610+0000) 2019-10-13T05:20:03.005 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:03.001+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:03.001813+0000) 2019-10-13T05:20:03.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:02.995+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:02.997441+0000) 2019-10-13T05:20:03.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:03.001+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:03.002358+0000) 2019-10-13T05:20:03.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:02.997+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:02.998965+0000) 2019-10-13T05:20:03.007 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:03.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 74980 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:04.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:04.164+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:04.165171+0000) 2019-10-13T05:20:08.009 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:08.005+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.005393+0000) 2019-10-13T05:20:08.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:08.005+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.005584+0000) 2019-10-13T05:20:08.009 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:08.005+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.005963+0000) 2019-10-13T05:20:08.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:08.000+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.001626+0000) 2019-10-13T05:20:08.010 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:08.006+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.006440+0000) 2019-10-13T05:20:08.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:08.006+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.006781+0000) 2019-10-13T05:20:08.011 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:08.006+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.006836+0000) 2019-10-13T05:20:08.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:08.003+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.004219+0000) 2019-10-13T05:20:08.012 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:08.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75008 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:08.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:08.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.208331+0000) 2019-10-13T05:20:08.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:08.547+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.548242+0000) 2019-10-13T05:20:08.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:08.547+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:08.548251+0000) 2019-10-13T05:20:08.574 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:20:08.574 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:08.579 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:20:08.579 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:10.188 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:10.180+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:10.181130+0000) 2019-10-13T05:20:11.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:11.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:11.800191+0000) 2019-10-13T05:20:12.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:12.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:12.669384+0000) 2019-10-13T05:20:12.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:12.668+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:12.669323+0000) 2019-10-13T05:20:13.013 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:13.008+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.010046+0000) 2019-10-13T05:20:13.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:13.008+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.010165+0000) 2019-10-13T05:20:13.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:13.008+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.010346+0000) 2019-10-13T05:20:13.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:13.005+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.006433+0000) 2019-10-13T05:20:13.014 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:13.009+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.010678+0000) 2019-10-13T05:20:13.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:13.009+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.011071+0000) 2019-10-13T05:20:13.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:13.009+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.011237+0000) 2019-10-13T05:20:13.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:13.007+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:13.008279+0000) 2019-10-13T05:20:13.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:13.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:14.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:14.164+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:14.165373+0000) 2019-10-13T05:20:18.018 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:18.013+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.014547+0000) 2019-10-13T05:20:18.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:18.013+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.014750+0000) 2019-10-13T05:20:18.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:18.009+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.010126+0000) 2019-10-13T05:20:18.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:18.013+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.015175+0000) 2019-10-13T05:20:18.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:18.013+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.015322+0000) 2019-10-13T05:20:18.019 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:18.014+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.015513+0000) 2019-10-13T05:20:18.019 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:18.014+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.015971+0000) 2019-10-13T05:20:18.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:18.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:18.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:18.013+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.013734+0000) 2019-10-13T05:20:18.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:18.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.208517+0000) 2019-10-13T05:20:18.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:18.547+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.548433+0000) 2019-10-13T05:20:18.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:18.547+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:18.548442+0000) 2019-10-13T05:20:20.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:20.183+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:20.183492+0000) 2019-10-13T05:20:21.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:21.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:21.800371+0000) 2019-10-13T05:20:22.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:22.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:22.669595+0000) 2019-10-13T05:20:22.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:22.669+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:22.669494+0000) 2019-10-13T05:20:23.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:23.018+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.019412+0000) 2019-10-13T05:20:23.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:23.018+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.019693+0000) 2019-10-13T05:20:23.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:23.018+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.020029+0000) 2019-10-13T05:20:23.023 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:23.018+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.020274+0000) 2019-10-13T05:20:23.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:23.015+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.015909+0000) 2019-10-13T05:20:23.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:23.019+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.020630+0000) 2019-10-13T05:20:23.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:23.019+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.020784+0000) 2019-10-13T05:20:23.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:23.017+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:23.018204+0000) 2019-10-13T05:20:23.026 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:23.021+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75100 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:24.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:24.164+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:24.165560+0000) 2019-10-13T05:20:28.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:28.019+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.019896+0000) 2019-10-13T05:20:28.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:28.023+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.024568+0000) 2019-10-13T05:20:28.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:28.023+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.024897+0000) 2019-10-13T05:20:28.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:28.023+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.024998+0000) 2019-10-13T05:20:28.030 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:28.023+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.025123+0000) 2019-10-13T05:20:28.030 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:28.023+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.025257+0000) 2019-10-13T05:20:28.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:28.024+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.025410+0000) 2019-10-13T05:20:28.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:28.021+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.021871+0000) 2019-10-13T05:20:28.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:28.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75128 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:28.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:28.208+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.208699+0000) 2019-10-13T05:20:28.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:28.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.548591+0000) 2019-10-13T05:20:28.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:28.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:28.548598+0000) 2019-10-13T05:20:30.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:30.186+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:30.186623+0000) 2019-10-13T05:20:31.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:31.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:31.800569+0000) 2019-10-13T05:20:32.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:32.669+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:32.669798+0000) 2019-10-13T05:20:32.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:32.668+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:32.669635+0000) 2019-10-13T05:20:33.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:33.027+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.029329+0000) 2019-10-13T05:20:33.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:33.024+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.025425+0000) 2019-10-13T05:20:33.033 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:33.028+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.030099+0000) 2019-10-13T05:20:33.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:33.028+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.030278+0000) 2019-10-13T05:20:33.034 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:33.026+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.027108+0000) 2019-10-13T05:20:33.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:33.029+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.030804+0000) 2019-10-13T05:20:33.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:33.029+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.030909+0000) 2019-10-13T05:20:33.035 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:33.029+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:33.031178+0000) 2019-10-13T05:20:33.036 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:33.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75161 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:34.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:34.165+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:34.165738+0000) 2019-10-13T05:20:38.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:38.034+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.034758+0000) 2019-10-13T05:20:38.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:38.029+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.030590+0000) 2019-10-13T05:20:38.039 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:38.035+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.035441+0000) 2019-10-13T05:20:38.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:38.034+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.035320+0000) 2019-10-13T05:20:38.040 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:38.035+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.035654+0000) 2019-10-13T05:20:38.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:38.035+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.035817+0000) 2019-10-13T05:20:38.040 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:38.035+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.036083+0000) 2019-10-13T05:20:38.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:38.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75189 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:38.041 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:38.033+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.034063+0000) 2019-10-13T05:20:38.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:38.208+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.208810+0000) 2019-10-13T05:20:38.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:38.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.548749+0000) 2019-10-13T05:20:38.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:38.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:38.548755+0000) 2019-10-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:20:38.780 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:20:38.784 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:40.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:40.189+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:40.189142+0000) 2019-10-13T05:20:41.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:41.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:41.800721+0000) 2019-10-13T05:20:42.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:42.669+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:42.669954+0000) 2019-10-13T05:20:42.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:42.668+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:42.669815+0000) 2019-10-13T05:20:43.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:43.039+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.039823+0000) 2019-10-13T05:20:43.043 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:43.039+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.040126+0000) 2019-10-13T05:20:43.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:43.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.036135+0000) 2019-10-13T05:20:43.044 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:43.040+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.040482+0000) 2019-10-13T05:20:43.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:43.040+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.040772+0000) 2019-10-13T05:20:43.045 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:43.040+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.040907+0000) 2019-10-13T05:20:43.045 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:43.040+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.041010+0000) 2019-10-13T05:20:43.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:43.035+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:43.037249+0000) 2019-10-13T05:20:43.046 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:43.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75220 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:44.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:44.165+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:44.165919+0000) 2019-10-13T05:20:48.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:48.044+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.044459+0000) 2019-10-13T05:20:48.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:48.044+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.044665+0000) 2019-10-13T05:20:48.048 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:48.044+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.045008+0000) 2019-10-13T05:20:48.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:48.039+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.040906+0000) 2019-10-13T05:20:48.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:48.044+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.045245+0000) 2019-10-13T05:20:48.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:48.045+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.045902+0000) 2019-10-13T05:20:48.049 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:48.045+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.046077+0000) 2019-10-13T05:20:48.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:48.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75248 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:48.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:48.042+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.043778+0000) 2019-10-13T05:20:48.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:48.208+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.208958+0000) 2019-10-13T05:20:48.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:48.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.548922+0000) 2019-10-13T05:20:48.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:48.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:48.548932+0000) 2019-10-13T05:20:50.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:50.189+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:50.191644+0000) 2019-10-13T05:20:51.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:51.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:51.800913+0000) 2019-10-13T05:20:52.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:52.670+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:52.670142+0000) 2019-10-13T05:20:52.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:52.669+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:52.669965+0000) 2019-10-13T05:20:53.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:53.047+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.048131+0000) 2019-10-13T05:20:53.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:53.048+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.048728+0000) 2019-10-13T05:20:53.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:53.048+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.048795+0000) 2019-10-13T05:20:53.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:53.042+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.044562+0000) 2019-10-13T05:20:53.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:53.048+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.049200+0000) 2019-10-13T05:20:53.053 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:53.048+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.048882+0000) 2019-10-13T05:20:53.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:53.049+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.049482+0000) 2019-10-13T05:20:53.053 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:53.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:53.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:53.045+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:53.046926+0000) 2019-10-13T05:20:54.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:54.165+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:54.166108+0000) 2019-10-13T05:20:58.055 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:58.050+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.052214+0000) 2019-10-13T05:20:58.055 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:20:58.051+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.052393+0000) 2019-10-13T05:20:58.056 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:58.051+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.052502+0000) 2019-10-13T05:20:58.056 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:20:58.051+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.052734+0000) 2019-10-13T05:20:58.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:20:58.048+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.049080+0000) 2019-10-13T05:20:58.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:58.051+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.053140+0000) 2019-10-13T05:20:58.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:20:58.052+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.053429+0000) 2019-10-13T05:20:58.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:20:58.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:20:58.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:20:58.050+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.050911+0000) 2019-10-13T05:20:58.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:20:58.207+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.209128+0000) 2019-10-13T05:20:58.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:20:58.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.549109+0000) 2019-10-13T05:20:58.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:20:58.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:20:58.549115+0000) 2019-10-13T05:21:00.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:00.193+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:00.194134+0000) 2019-10-13T05:21:01.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:01.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:01.801102+0000) 2019-10-13T05:21:02.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:02.668+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:02.670338+0000) 2019-10-13T05:21:02.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:02.669+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:02.670143+0000) 2019-10-13T05:21:03.059 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:03.055+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.056544+0000) 2019-10-13T05:21:03.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:03.055+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.056742+0000) 2019-10-13T05:21:03.060 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:03.055+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.057159+0000) 2019-10-13T05:21:03.061 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:03.055+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.057370+0000) 2019-10-13T05:21:03.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:03.056+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.057496+0000) 2019-10-13T05:21:03.061 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:03.056+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.057998+0000) 2019-10-13T05:21:03.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:03.053+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.054004+0000) 2019-10-13T05:21:03.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:03.054+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:03.055393+0000) 2019-10-13T05:21:03.063 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:03.058+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:04.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:04.165+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:04.166298+0000) 2019-10-13T05:21:08.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:08.061+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.062441+0000) 2019-10-13T05:21:08.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:08.061+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.062604+0000) 2019-10-13T05:21:08.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:08.061+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.062799+0000) 2019-10-13T05:21:08.067 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:08.061+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.062962+0000) 2019-10-13T05:21:08.067 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:08.061+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.063057+0000) 2019-10-13T05:21:08.067 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:08.062+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.063414+0000) 2019-10-13T05:21:08.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:08.058+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.058744+0000) 2019-10-13T05:21:08.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:08.059+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.059834+0000) 2019-10-13T05:21:08.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:08.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:08.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:08.208+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.209316+0000) 2019-10-13T05:21:08.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:08.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.549285+0000) 2019-10-13T05:21:08.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:08.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:08.549287+0000) 2019-10-13T05:21:08.978 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:21:08.978 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:08.985 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:21:08.985 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:10.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:10.195+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:10.196259+0000) 2019-10-13T05:21:11.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:11.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:11.801314+0000) 2019-10-13T05:21:12.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:12.669+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:12.670528+0000) 2019-10-13T05:21:12.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:12.669+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:12.670320+0000) 2019-10-13T05:21:13.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:13.062+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.063237+0000) 2019-10-13T05:21:13.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:13.065+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.067103+0000) 2019-10-13T05:21:13.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:13.065+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.066979+0000) 2019-10-13T05:21:13.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:13.065+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.067376+0000) 2019-10-13T05:21:13.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:13.066+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.067704+0000) 2019-10-13T05:21:13.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:13.066+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.067903+0000) 2019-10-13T05:21:13.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:13.066+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.068022+0000) 2019-10-13T05:21:13.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:13.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75401 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:13.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:13.063+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:13.064556+0000) 2019-10-13T05:21:14.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:14.165+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:14.166489+0000) 2019-10-13T05:21:18.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:18.069+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.070843+0000) 2019-10-13T05:21:18.074 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:18.069+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.070939+0000) 2019-10-13T05:21:18.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:18.069+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.071086+0000) 2019-10-13T05:21:18.075 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:18.069+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.071305+0000) 2019-10-13T05:21:18.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:18.070+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.071479+0000) 2019-10-13T05:21:18.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:18.066+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.067075+0000) 2019-10-13T05:21:18.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:18.070+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.071777+0000) 2019-10-13T05:21:18.076 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:18.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:18.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:18.068+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.069670+0000) 2019-10-13T05:21:18.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:18.208+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.209501+0000) 2019-10-13T05:21:18.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:18.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.549452+0000) 2019-10-13T05:21:18.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:18.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:18.549453+0000) 2019-10-13T05:21:20.207 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:20.199+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:20.199299+0000) 2019-10-13T05:21:21.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:21.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:21.801543+0000) 2019-10-13T05:21:22.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:22.670+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:22.670733+0000) 2019-10-13T05:21:22.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:22.670+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:22.670463+0000) 2019-10-13T05:21:23.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:23.075+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.076241+0000) 2019-10-13T05:21:23.080 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:23.076+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.076639+0000) 2019-10-13T05:21:23.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:23.076+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.076494+0000) 2019-10-13T05:21:23.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:23.076+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.076818+0000) 2019-10-13T05:21:23.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:23.076+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.077242+0000) 2019-10-13T05:21:23.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:23.076+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.076952+0000) 2019-10-13T05:21:23.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:23.072+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.072914+0000) 2019-10-13T05:21:23.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:23.078+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:23.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:23.074+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:23.075464+0000) 2019-10-13T05:21:24.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:24.166+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:24.166681+0000) 2019-10-13T05:21:28.084 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:28.080+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.081026+0000) 2019-10-13T05:21:28.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:28.080+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.081191+0000) 2019-10-13T05:21:28.085 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:28.081+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.081488+0000) 2019-10-13T05:21:28.085 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:28.081+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.081623+0000) 2019-10-13T05:21:28.086 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:28.081+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.081806+0000) 2019-10-13T05:21:28.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:28.076+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.077454+0000) 2019-10-13T05:21:28.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:28.078+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.079323+0000) 2019-10-13T05:21:28.087 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:28.081+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.082175+0000) 2019-10-13T05:21:28.087 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:28.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75488 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:28.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:28.209+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.209668+0000) 2019-10-13T05:21:28.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:28.549+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.549619+0000) 2019-10-13T05:21:28.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:28.549+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:28.549626+0000) 2019-10-13T05:21:30.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:30.201+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:30.201782+0000) 2019-10-13T05:21:31.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:31.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:31.801732+0000) 2019-10-13T05:21:32.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:32.670+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:32.670924+0000) 2019-10-13T05:21:32.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:32.669+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:32.670613+0000) 2019-10-13T05:21:33.087 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:33.084+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.084666+0000) 2019-10-13T05:21:33.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:33.084+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.085086+0000) 2019-10-13T05:21:33.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:33.084+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.085215+0000) 2019-10-13T05:21:33.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:33.079+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.081459+0000) 2019-10-13T05:21:33.089 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:33.085+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.085765+0000) 2019-10-13T05:21:33.089 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:33.085+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.086128+0000) 2019-10-13T05:21:33.090 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:33.085+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.086184+0000) 2019-10-13T05:21:33.090 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:33.087+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:33.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:33.081+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:33.083158+0000) 2019-10-13T05:21:34.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:34.166+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:34.166895+0000) 2019-10-13T05:21:38.092 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:38.088+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.089039+0000) 2019-10-13T05:21:38.092 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:38.088+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.089213+0000) 2019-10-13T05:21:38.093 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:38.089+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.089541+0000) 2019-10-13T05:21:38.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:38.083+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.085506+0000) 2019-10-13T05:21:38.094 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:38.089+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.089786+0000) 2019-10-13T05:21:38.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:38.089+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.090358+0000) 2019-10-13T05:21:38.094 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:38.090+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.090415+0000) 2019-10-13T05:21:38.094 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:38.085+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.087492+0000) 2019-10-13T05:21:38.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:38.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:38.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:38.209+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.209813+0000) 2019-10-13T05:21:38.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:38.549+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.549795+0000) 2019-10-13T05:21:38.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:38.549+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:38.549798+0000) 2019-10-13T05:21:39.083 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:21:39.084 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:39.088 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:21:39.088 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:40.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:40.203+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:40.204228+0000) 2019-10-13T05:21:41.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:41.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:41.801947+0000) 2019-10-13T05:21:42.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:42.670+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:42.671106+0000) 2019-10-13T05:21:42.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:42.669+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:42.670761+0000) 2019-10-13T05:21:43.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:43.088+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.090432+0000) 2019-10-13T05:21:43.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:43.092+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.094059+0000) 2019-10-13T05:21:43.100 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:43.092+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.094375+0000) 2019-10-13T05:21:43.100 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:43.093+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.094837+0000) 2019-10-13T05:21:43.100 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:43.093+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.095199+0000) 2019-10-13T05:21:43.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:43.093+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.095330+0000) 2019-10-13T05:21:43.101 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:43.094+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.095626+0000) 2019-10-13T05:21:43.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:43.091+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:43.093043+0000) 2019-10-13T05:21:43.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:43.096+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:44.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:44.166+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:44.167086+0000) 2019-10-13T05:21:48.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:48.097+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.099110+0000) 2019-10-13T05:21:48.102 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:48.098+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.099419+0000) 2019-10-13T05:21:48.102 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:48.098+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.099510+0000) 2019-10-13T05:21:48.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:48.094+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.095106+0000) 2019-10-13T05:21:48.104 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:48.098+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.099961+0000) 2019-10-13T05:21:48.104 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:48.098+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.100317+0000) 2019-10-13T05:21:48.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:48.099+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.100464+0000) 2019-10-13T05:21:48.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:48.100+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:48.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:48.096+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.097442+0000) 2019-10-13T05:21:48.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:48.208+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.209950+0000) 2019-10-13T05:21:48.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:48.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.549967+0000) 2019-10-13T05:21:48.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:48.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:48.549981+0000) 2019-10-13T05:21:50.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:50.205+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:50.206586+0000) 2019-10-13T05:21:51.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:51.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:51.802135+0000) 2019-10-13T05:21:52.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:52.669+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:52.671314+0000) 2019-10-13T05:21:52.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:52.670+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:52.670934+0000) 2019-10-13T05:21:53.106 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:53.101+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.103379+0000) 2019-10-13T05:21:53.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:53.102+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.103575+0000) 2019-10-13T05:21:53.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:53.098+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.099515+0000) 2019-10-13T05:21:53.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:53.102+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.104010+0000) 2019-10-13T05:21:53.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:53.102+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.104204+0000) 2019-10-13T05:21:53.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:53.103+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.104460+0000) 2019-10-13T05:21:53.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:53.103+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.104605+0000) 2019-10-13T05:21:53.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:53.104+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:53.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:53.101+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:53.102319+0000) 2019-10-13T05:21:54.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:54.166+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:54.167278+0000) 2019-10-13T05:21:58.110 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:58.105+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.107343+0000) 2019-10-13T05:21:58.111 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:21:58.106+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.107659+0000) 2019-10-13T05:21:58.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:21:58.106+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.107768+0000) 2019-10-13T05:21:58.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:21:58.102+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.103455+0000) 2019-10-13T05:21:58.112 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:58.106+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.108145+0000) 2019-10-13T05:21:58.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:21:58.107+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.108526+0000) 2019-10-13T05:21:58.112 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:58.107+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.108775+0000) 2019-10-13T05:21:58.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:21:58.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:21:58.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:21:58.105+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.106534+0000) 2019-10-13T05:21:58.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:21:58.209+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.210116+0000) 2019-10-13T05:21:58.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:21:58.548+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.550174+0000) 2019-10-13T05:21:58.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:21:58.548+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:21:58.550152+0000) 2019-10-13T05:22:00.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:00.208+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:00.209031+0000) 2019-10-13T05:22:01.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:01.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:01.802304+0000) 2019-10-13T05:22:02.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:02.670+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:02.671493+0000) 2019-10-13T05:22:02.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:02.670+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:02.671118+0000) 2019-10-13T05:22:03.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:03.111+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.112799+0000) 2019-10-13T05:22:03.117 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:03.111+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.112904+0000) 2019-10-13T05:22:03.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:03.111+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.113192+0000) 2019-10-13T05:22:03.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:03.112+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.113573+0000) 2019-10-13T05:22:03.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:03.112+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.113716+0000) 2019-10-13T05:22:03.117 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:03.112+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.113864+0000) 2019-10-13T05:22:03.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:03.108+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.108902+0000) 2019-10-13T05:22:03.118 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:03.109+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:03.110270+0000) 2019-10-13T05:22:03.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:03.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:04.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:04.166+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:04.167461+0000) 2019-10-13T05:22:08.120 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:08.117+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.117644+0000) 2019-10-13T05:22:08.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:08.112+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.113472+0000) 2019-10-13T05:22:08.122 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:08.117+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.118195+0000) 2019-10-13T05:22:08.122 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:08.118+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.118419+0000) 2019-10-13T05:22:08.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:08.118+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.118634+0000) 2019-10-13T05:22:08.122 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:08.118+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.118440+0000) 2019-10-13T05:22:08.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:08.118+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.118797+0000) 2019-10-13T05:22:08.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:08.115+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.116161+0000) 2019-10-13T05:22:08.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:08.120+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:08.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:08.209+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.210286+0000) 2019-10-13T05:22:08.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:08.549+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.550316+0000) 2019-10-13T05:22:08.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:08.549+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:08.550368+0000) 2019-10-13T05:22:09.288 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:22:09.289 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:09.294 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:22:09.294 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:10.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:10.211+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:10.211547+0000) 2019-10-13T05:22:11.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:11.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:11.802469+0000) 2019-10-13T05:22:12.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:12.671+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:12.671697+0000) 2019-10-13T05:22:12.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:12.671+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:12.671321+0000) 2019-10-13T05:22:13.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:13.122+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.122648+0000) 2019-10-13T05:22:13.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:13.122+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.122967+0000) 2019-10-13T05:22:13.127 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:13.117+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.118248+0000) 2019-10-13T05:22:13.127 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:13.122+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.123085+0000) 2019-10-13T05:22:13.127 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:13.122+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.123354+0000) 2019-10-13T05:22:13.127 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:13.123+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.123536+0000) 2019-10-13T05:22:13.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:13.123+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.123697+0000) 2019-10-13T05:22:13.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:13.119+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:13.119985+0000) 2019-10-13T05:22:13.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:13.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:14.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:14.167+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:14.167636+0000) 2019-10-13T05:22:18.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:18.122+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.123092+0000) 2019-10-13T05:22:18.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:18.126+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.126855+0000) 2019-10-13T05:22:18.132 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:18.126+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.127064+0000) 2019-10-13T05:22:18.132 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:18.127+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.127424+0000) 2019-10-13T05:22:18.132 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:18.127+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.127523+0000) 2019-10-13T05:22:18.133 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:18.127+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.127777+0000) 2019-10-13T05:22:18.133 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:18.127+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.128027+0000) 2019-10-13T05:22:18.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:18.124+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.125689+0000) 2019-10-13T05:22:18.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:18.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75789 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:18.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:18.210+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.210451+0000) 2019-10-13T05:22:18.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:18.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.550495+0000) 2019-10-13T05:22:18.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:18.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:18.550581+0000) 2019-10-13T05:22:20.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:20.213+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:20.213804+0000) 2019-10-13T05:22:21.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:21.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:21.802680+0000) 2019-10-13T05:22:22.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:22.671+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:22.671903+0000) 2019-10-13T05:22:22.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:22.670+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:22.671530+0000) 2019-10-13T05:22:23.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:23.124+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.126541+0000) 2019-10-13T05:22:23.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:23.130+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.131311+0000) 2019-10-13T05:22:23.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:23.131+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.131558+0000) 2019-10-13T05:22:23.135 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:23.131+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.131788+0000) 2019-10-13T05:22:23.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:23.131+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.132067+0000) 2019-10-13T05:22:23.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:23.131+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.132253+0000) 2019-10-13T05:22:23.136 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:23.132+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.132534+0000) 2019-10-13T05:22:23.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:23.127+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:23.129209+0000) 2019-10-13T05:22:23.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:23.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:24.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:24.167+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:24.167809+0000) 2019-10-13T05:22:28.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:28.134+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.136042+0000) 2019-10-13T05:22:28.141 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:28.136+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.138354+0000) 2019-10-13T05:22:28.141 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:28.137+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.138647+0000) 2019-10-13T05:22:28.142 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:28.137+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.138725+0000) 2019-10-13T05:22:28.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:28.137+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.138870+0000) 2019-10-13T05:22:28.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:28.132+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.134506+0000) 2019-10-13T05:22:28.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:28.138+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.139777+0000) 2019-10-13T05:22:28.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:28.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75848 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:28.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:28.135+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.136775+0000) 2019-10-13T05:22:28.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:28.209+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.210614+0000) 2019-10-13T05:22:28.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:28.549+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.550662+0000) 2019-10-13T05:22:28.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:28.549+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:28.550731+0000) 2019-10-13T05:22:30.224 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:30.215+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:30.216303+0000) 2019-10-13T05:22:31.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:31.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:31.802907+0000) 2019-10-13T05:22:32.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:32.670+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:32.672089+0000) 2019-10-13T05:22:32.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:32.670+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:32.671678+0000) 2019-10-13T05:22:33.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:33.139+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.141217+0000) 2019-10-13T05:22:33.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:33.140+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.142292+0000) 2019-10-13T05:22:33.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:33.141+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.142492+0000) 2019-10-13T05:22:33.146 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:33.141+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.142869+0000) 2019-10-13T05:22:33.146 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:33.141+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.143040+0000) 2019-10-13T05:22:33.146 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:33.136+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.138673+0000) 2019-10-13T05:22:33.147 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:33.142+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.143673+0000) 2019-10-13T05:22:33.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:33.138+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:33.140315+0000) 2019-10-13T05:22:33.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:33.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75881 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:34.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:34.167+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:34.167981+0000) 2019-10-13T05:22:38.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:38.145+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.146806+0000) 2019-10-13T05:22:38.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:38.145+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.146610+0000) 2019-10-13T05:22:38.150 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:38.145+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.147255+0000) 2019-10-13T05:22:38.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:38.141+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.142555+0000) 2019-10-13T05:22:38.151 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:38.146+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.147580+0000) 2019-10-13T05:22:38.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:38.146+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.147710+0000) 2019-10-13T05:22:38.152 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:38.146+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.147874+0000) 2019-10-13T05:22:38.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:38.143+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.144512+0000) 2019-10-13T05:22:38.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:38.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:38.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:38.210+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.210724+0000) 2019-10-13T05:22:38.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:38.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.550806+0000) 2019-10-13T05:22:38.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:38.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:38.550873+0000) 2019-10-13T05:22:39.411 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:22:39.411 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:39.415 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:22:39.415 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:40.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:40.218+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:40.218685+0000) 2019-10-13T05:22:41.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:41.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:41.803098+0000) 2019-10-13T05:22:42.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:42.670+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:42.672269+0000) 2019-10-13T05:22:42.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:42.671+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:42.671857+0000) 2019-10-13T05:22:43.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:43.149+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.151273+0000) 2019-10-13T05:22:43.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:43.150+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.151954+0000) 2019-10-13T05:22:43.155 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:43.146+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.147586+0000) 2019-10-13T05:22:43.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:43.150+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.152340+0000) 2019-10-13T05:22:43.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:43.151+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.152411+0000) 2019-10-13T05:22:43.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:43.151+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.152531+0000) 2019-10-13T05:22:43.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:43.151+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.152754+0000) 2019-10-13T05:22:43.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:43.148+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:43.149072+0000) 2019-10-13T05:22:43.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:43.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:44.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:44.167+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:44.168169+0000) 2019-10-13T05:22:48.160 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:48.155+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.156824+0000) 2019-10-13T05:22:48.160 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:48.155+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.156992+0000) 2019-10-13T05:22:48.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:48.155+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.157089+0000) 2019-10-13T05:22:48.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:48.155+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.157270+0000) 2019-10-13T05:22:48.161 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:48.156+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.157560+0000) 2019-10-13T05:22:48.162 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:48.152+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.153459+0000) 2019-10-13T05:22:48.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:48.153+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.154213+0000) 2019-10-13T05:22:48.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:48.156+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.157802+0000) 2019-10-13T05:22:48.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:48.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 75968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:48.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:48.209+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.210835+0000) 2019-10-13T05:22:48.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:48.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.550995+0000) 2019-10-13T05:22:48.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:48.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:48.551008+0000) 2019-10-13T05:22:50.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:50.221+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:50.221830+0000) 2019-10-13T05:22:51.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:51.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:51.803285+0000) 2019-10-13T05:22:52.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:52.672+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:52.672424+0000) 2019-10-13T05:22:52.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:52.671+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:52.672057+0000) 2019-10-13T05:22:53.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:53.160+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.160912+0000) 2019-10-13T05:22:53.164 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:53.160+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.161080+0000) 2019-10-13T05:22:53.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:53.156+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.157002+0000) 2019-10-13T05:22:53.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:53.161+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.161409+0000) 2019-10-13T05:22:53.166 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:53.161+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.161653+0000) 2019-10-13T05:22:53.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:53.161+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.161941+0000) 2019-10-13T05:22:53.167 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:53.161+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.162074+0000) 2019-10-13T05:22:53.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:53.158+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:53.159463+0000) 2019-10-13T05:22:53.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:53.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:54.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:54.167+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:54.168352+0000) 2019-10-13T05:22:58.168 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:22:58.164+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.164616+0000) 2019-10-13T05:22:58.168 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:58.164+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.165013+0000) 2019-10-13T05:22:58.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:58.164+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.165102+0000) 2019-10-13T05:22:58.169 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:22:58.164+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.165297+0000) 2019-10-13T05:22:58.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:22:58.165+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.165430+0000) 2019-10-13T05:22:58.170 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:22:58.160+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.160996+0000) 2019-10-13T05:22:58.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:58.165+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.165647+0000) 2019-10-13T05:22:58.171 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:22:58.166+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:22:58.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:22:58.162+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.163217+0000) 2019-10-13T05:22:58.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:22:58.210+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.210991+0000) 2019-10-13T05:22:58.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:22:58.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.551118+0000) 2019-10-13T05:22:58.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:22:58.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:22:58.551149+0000) 2019-10-13T05:23:00.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:00.223+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:00.224172+0000) 2019-10-13T05:23:01.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:01.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:01.803480+0000) 2019-10-13T05:23:02.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:02.672+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:02.672574+0000) 2019-10-13T05:23:02.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:02.672+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:02.672243+0000) 2019-10-13T05:23:03.171 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:03.167+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.168364+0000) 2019-10-13T05:23:03.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:03.168+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.168647+0000) 2019-10-13T05:23:03.172 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:03.168+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.168903+0000) 2019-10-13T05:23:03.173 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:03.163+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.164218+0000) 2019-10-13T05:23:03.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:03.168+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.169214+0000) 2019-10-13T05:23:03.173 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:03.169+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.169403+0000) 2019-10-13T05:23:03.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:03.169+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.169481+0000) 2019-10-13T05:23:03.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:03.165+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:03.166307+0000) 2019-10-13T05:23:03.175 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:03.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:04.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:04.168+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:04.168557+0000) 2019-10-13T05:23:08.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:08.172+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.173081+0000) 2019-10-13T05:23:08.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:08.172+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.173363+0000) 2019-10-13T05:23:08.177 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:08.168+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.169589+0000) 2019-10-13T05:23:08.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:08.173+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.173718+0000) 2019-10-13T05:23:08.178 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:08.173+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.173849+0000) 2019-10-13T05:23:08.178 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:08.173+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.173932+0000) 2019-10-13T05:23:08.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:08.173+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.174370+0000) 2019-10-13T05:23:08.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:08.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:08.179 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:08.171+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.172012+0000) 2019-10-13T05:23:08.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:08.211+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.211175+0000) 2019-10-13T05:23:08.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:08.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.551287+0000) 2019-10-13T05:23:08.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:08.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:08.551307+0000) 2019-10-13T05:23:09.633 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:23:09.633 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:09.637 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:23:09.637 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:10.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:10.226+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:10.226610+0000) 2019-10-13T05:23:11.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:11.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:11.803678+0000) 2019-10-13T05:23:12.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:12.671+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:12.672750+0000) 2019-10-13T05:23:12.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:12.671+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:12.672427+0000) 2019-10-13T05:23:13.181 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:13.171+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.173459+0000) 2019-10-13T05:23:13.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:13.176+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.178329+0000) 2019-10-13T05:23:13.182 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:13.177+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.178442+0000) 2019-10-13T05:23:13.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:13.177+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.178879+0000) 2019-10-13T05:23:13.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:13.177+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.179092+0000) 2019-10-13T05:23:13.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:13.177+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.179359+0000) 2019-10-13T05:23:13.183 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:13.178+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.179603+0000) 2019-10-13T05:23:13.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:13.174+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:13.176101+0000) 2019-10-13T05:23:13.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:13.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76121 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:14.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:14.167+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:14.168702+0000) 2019-10-13T05:23:18.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:18.176+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.178112+0000) 2019-10-13T05:23:18.186 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:18.181+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.183126+0000) 2019-10-13T05:23:18.186 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:18.181+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.183373+0000) 2019-10-13T05:23:18.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:18.182+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.183530+0000) 2019-10-13T05:23:18.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:18.182+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.183754+0000) 2019-10-13T05:23:18.187 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:18.182+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.183820+0000) 2019-10-13T05:23:18.188 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:18.182+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.184156+0000) 2019-10-13T05:23:18.188 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:18.178+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.180364+0000) 2019-10-13T05:23:18.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:18.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:18.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:18.210+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.211339+0000) 2019-10-13T05:23:18.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:18.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.551395+0000) 2019-10-13T05:23:18.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:18.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:18.551468+0000) 2019-10-13T05:23:20.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:20.228+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:20.228939+0000) 2019-10-13T05:23:21.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:21.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:21.803881+0000) 2019-10-13T05:23:22.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:22.672+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:22.672931+0000) 2019-10-13T05:23:22.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:22.671+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:22.672591+0000) 2019-10-13T05:23:23.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:23.181+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.183224+0000) 2019-10-13T05:23:23.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:23.186+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.187912+0000) 2019-10-13T05:23:23.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:23.187+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.188686+0000) 2019-10-13T05:23:23.192 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:23.187+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.188915+0000) 2019-10-13T05:23:23.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:23.187+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.189046+0000) 2019-10-13T05:23:23.192 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:23.187+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.189323+0000) 2019-10-13T05:23:23.193 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:23.188+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.189686+0000) 2019-10-13T05:23:23.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:23.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76180 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:23.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:23.184+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:23.186727+0000) 2019-10-13T05:23:24.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:24.168+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:24.168824+0000) 2019-10-13T05:23:28.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:28.191+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.192893+0000) 2019-10-13T05:23:28.196 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:28.191+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.193119+0000) 2019-10-13T05:23:28.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:28.192+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.193465+0000) 2019-10-13T05:23:28.197 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:28.188+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.189347+0000) 2019-10-13T05:23:28.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:28.190+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.191464+0000) 2019-10-13T05:23:28.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:28.192+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.194037+0000) 2019-10-13T05:23:28.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:28.192+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.194199+0000) 2019-10-13T05:23:28.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:28.193+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.194391+0000) 2019-10-13T05:23:28.201 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:28.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:28.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:28.210+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.211455+0000) 2019-10-13T05:23:28.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:28.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.551557+0000) 2019-10-13T05:23:28.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:28.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:28.551649+0000) 2019-10-13T05:23:30.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:30.231+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:30.232008+0000) 2019-10-13T05:23:31.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:31.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:31.804074+0000) 2019-10-13T05:23:32.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:32.672+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:32.673058+0000) 2019-10-13T05:23:32.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:32.671+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:32.672714+0000) 2019-10-13T05:23:33.201 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:33.192+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.193581+0000) 2019-10-13T05:23:33.202 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:33.196+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.198358+0000) 2019-10-13T05:23:33.202 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:33.197+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.198534+0000) 2019-10-13T05:23:33.202 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:33.197+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.198702+0000) 2019-10-13T05:23:33.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:33.197+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.198903+0000) 2019-10-13T05:23:33.203 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:33.197+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.199054+0000) 2019-10-13T05:23:33.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:33.198+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.199445+0000) 2019-10-13T05:23:33.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:33.199+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:33.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:33.196+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:33.196777+0000) 2019-10-13T05:23:34.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:34.168+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:34.168963+0000) 2019-10-13T05:23:38.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:38.202+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.203173+0000) 2019-10-13T05:23:38.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:38.203+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.203507+0000) 2019-10-13T05:23:38.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:38.203+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.203670+0000) 2019-10-13T05:23:38.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:38.203+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.203830+0000) 2019-10-13T05:23:38.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:38.203+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.204190+0000) 2019-10-13T05:23:38.208 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:38.203+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.204324+0000) 2019-10-13T05:23:38.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:38.198+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.199475+0000) 2019-10-13T05:23:38.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:38.199+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.200708+0000) 2019-10-13T05:23:38.209 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:38.206+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:38.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:38.211+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.211566+0000) 2019-10-13T05:23:38.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:38.551+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.551715+0000) 2019-10-13T05:23:38.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:38.551+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:38.551798+0000) 2019-10-13T05:23:39.814 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:23:39.814 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:39.818 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:23:39.818 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:40.242 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:40.233+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:40.234328+0000) 2019-10-13T05:23:41.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:41.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:41.804267+0000) 2019-10-13T05:23:42.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:42.672+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:42.673266+0000) 2019-10-13T05:23:42.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:42.672+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:42.672909+0000) 2019-10-13T05:23:43.212 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:43.207+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.208124+0000) 2019-10-13T05:23:43.212 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:43.207+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.208329+0000) 2019-10-13T05:23:43.212 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:43.208+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.208580+0000) 2019-10-13T05:23:43.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:43.208+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.208652+0000) 2019-10-13T05:23:43.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:43.208+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.208966+0000) 2019-10-13T05:23:43.213 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:43.203+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.204722+0000) 2019-10-13T05:23:43.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:43.209+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.209527+0000) 2019-10-13T05:23:43.214 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:43.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:43.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:43.206+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:43.207336+0000) 2019-10-13T05:23:44.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:44.168+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:44.169081+0000) 2019-10-13T05:23:48.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:48.211+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.211730+0000) 2019-10-13T05:23:48.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:48.207+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.208662+0000) 2019-10-13T05:23:48.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:48.213+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.213477+0000) 2019-10-13T05:23:48.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:48.213+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.213858+0000) 2019-10-13T05:23:48.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:48.213+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.214246+0000) 2019-10-13T05:23:48.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:48.214+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.214429+0000) 2019-10-13T05:23:48.218 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:48.214+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.214663+0000) 2019-10-13T05:23:48.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:48.214+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.214776+0000) 2019-10-13T05:23:48.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:48.211+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.211888+0000) 2019-10-13T05:23:48.219 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:48.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:48.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:48.551+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.551903+0000) 2019-10-13T05:23:48.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:48.551+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:48.551929+0000) 2019-10-13T05:23:50.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:50.236+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:50.236612+0000) 2019-10-13T05:23:51.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:51.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:51.804438+0000) 2019-10-13T05:23:52.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:52.673+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:52.673434+0000) 2019-10-13T05:23:52.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:52.672+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:52.673098+0000) 2019-10-13T05:23:53.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:53.217+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.218114+0000) 2019-10-13T05:23:53.221 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:53.217+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.218298+0000) 2019-10-13T05:23:53.222 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:53.218+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.218458+0000) 2019-10-13T05:23:53.222 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:53.213+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.213971+0000) 2019-10-13T05:23:53.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:53.218+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.218985+0000) 2019-10-13T05:23:53.223 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:53.218+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.219038+0000) 2019-10-13T05:23:53.223 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:53.219+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.219558+0000) 2019-10-13T05:23:53.224 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:53.215+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:53.216720+0000) 2019-10-13T05:23:53.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:53.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:54.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:54.169+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:54.169266+0000) 2019-10-13T05:23:58.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:58.210+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.211893+0000) 2019-10-13T05:23:58.226 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:58.221+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.223220+0000) 2019-10-13T05:23:58.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:58.222+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.223403+0000) 2019-10-13T05:23:58.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:23:58.222+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.223599+0000) 2019-10-13T05:23:58.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:23:58.218+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.219143+0000) 2019-10-13T05:23:58.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:23:58.222+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.223911+0000) 2019-10-13T05:23:58.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:23:58.222+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.224025+0000) 2019-10-13T05:23:58.228 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:23:58.223+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.224512+0000) 2019-10-13T05:23:58.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:23:58.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76389 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:23:58.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:23:58.222+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.222995+0000) 2019-10-13T05:23:58.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:23:58.551+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.552097+0000) 2019-10-13T05:23:58.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:23:58.551+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:23:58.552102+0000) 2019-10-13T05:24:00.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:00.239+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:00.239613+0000) 2019-10-13T05:24:01.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:01.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:01.804598+0000) 2019-10-13T05:24:02.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:02.672+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:02.673625+0000) 2019-10-13T05:24:02.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:02.672+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:02.673315+0000) 2019-10-13T05:24:03.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:03.226+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.228252+0000) 2019-10-13T05:24:03.233 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:03.226+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.228364+0000) 2019-10-13T05:24:03.233 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:03.227+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.228847+0000) 2019-10-13T05:24:03.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:03.227+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.229064+0000) 2019-10-13T05:24:03.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:03.227+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.229373+0000) 2019-10-13T05:24:03.234 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:03.228+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.229533+0000) 2019-10-13T05:24:03.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:03.222+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.224113+0000) 2019-10-13T05:24:03.235 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:03.224+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:03.225983+0000) 2019-10-13T05:24:03.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:03.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76420 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:04.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:04.168+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:04.169441+0000) 2019-10-13T05:24:08.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:08.211+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.212046+0000) 2019-10-13T05:24:08.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:08.226+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.228710+0000) 2019-10-13T05:24:08.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:08.231+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.233233+0000) 2019-10-13T05:24:08.237 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:08.232+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.233473+0000) 2019-10-13T05:24:08.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:08.232+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.233773+0000) 2019-10-13T05:24:08.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:08.232+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.233968+0000) 2019-10-13T05:24:08.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:08.232+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.234263+0000) 2019-10-13T05:24:08.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:08.233+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.234455+0000) 2019-10-13T05:24:08.239 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:08.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:08.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:08.230+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.232243+0000) 2019-10-13T05:24:08.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:08.550+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.552300+0000) 2019-10-13T05:24:08.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:08.550+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:08.552299+0000) 2019-10-13T05:24:09.927 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:24:09.927 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:09.931 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:24:09.932 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:10.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:10.241+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:10.242763+0000) 2019-10-13T05:24:11.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:11.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:11.804807+0000) 2019-10-13T05:24:12.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:12.673+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:12.673810+0000) 2019-10-13T05:24:12.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:12.672+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:12.673500+0000) 2019-10-13T05:24:13.241 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:13.236+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.237977+0000) 2019-10-13T05:24:13.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:13.236+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.238186+0000) 2019-10-13T05:24:13.241 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:13.236+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.238379+0000) 2019-10-13T05:24:13.242 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:13.232+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.234677+0000) 2019-10-13T05:24:13.242 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:13.237+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.238512+0000) 2019-10-13T05:24:13.243 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:13.237+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.238779+0000) 2019-10-13T05:24:13.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:13.238+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.239673+0000) 2019-10-13T05:24:13.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:13.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:13.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:13.235+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:13.236775+0000) 2019-10-13T05:24:14.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:14.168+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:14.169581+0000) 2019-10-13T05:24:18.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:18.211+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.212242+0000) 2019-10-13T05:24:18.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:18.241+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.242898+0000) 2019-10-13T05:24:18.246 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:18.237+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.238504+0000) 2019-10-13T05:24:18.247 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:18.242+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.243447+0000) 2019-10-13T05:24:18.247 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:18.242+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.243610+0000) 2019-10-13T05:24:18.247 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:18.242+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.243810+0000) 2019-10-13T05:24:18.247 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:18.242+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.243938+0000) 2019-10-13T05:24:18.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:18.242+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.244190+0000) 2019-10-13T05:24:18.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:18.244+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:18.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:18.241+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.242120+0000) 2019-10-13T05:24:18.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:18.551+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.552474+0000) 2019-10-13T05:24:18.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:18.551+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:18.552472+0000) 2019-10-13T05:24:20.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:20.244+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:20.245082+0000) 2019-10-13T05:24:21.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:21.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:21.804982+0000) 2019-10-13T05:24:22.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:22.673+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:22.673997+0000) 2019-10-13T05:24:22.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:22.673+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:22.673661+0000) 2019-10-13T05:24:23.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:23.246+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.247386+0000) 2019-10-13T05:24:23.251 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:23.242+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.243012+0000) 2019-10-13T05:24:23.252 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:23.247+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.247811+0000) 2019-10-13T05:24:23.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:23.247+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.248068+0000) 2019-10-13T05:24:23.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:23.247+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.248255+0000) 2019-10-13T05:24:23.252 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:23.248+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.248390+0000) 2019-10-13T05:24:23.252 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:23.248+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.248747+0000) 2019-10-13T05:24:23.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:23.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:23.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:23.245+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:23.246102+0000) 2019-10-13T05:24:24.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:24.169+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:24.169734+0000) 2019-10-13T05:24:28.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:28.212+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.212434+0000) 2019-10-13T05:24:28.255 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:28.252+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.252622+0000) 2019-10-13T05:24:28.256 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:28.247+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.248413+0000) 2019-10-13T05:24:28.256 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:28.252+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.253176+0000) 2019-10-13T05:24:28.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:28.253+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.253583+0000) 2019-10-13T05:24:28.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:28.253+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.253652+0000) 2019-10-13T05:24:28.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:28.253+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.253867+0000) 2019-10-13T05:24:28.258 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:28.253+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.254133+0000) 2019-10-13T05:24:28.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:28.249+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.250062+0000) 2019-10-13T05:24:28.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:28.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76568 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:28.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:28.552+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.552662+0000) 2019-10-13T05:24:28.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:28.552+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:28.552653+0000) 2019-10-13T05:24:30.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:30.247+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:30.248127+0000) 2019-10-13T05:24:31.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:31.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:31.805187+0000) 2019-10-13T05:24:32.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:32.673+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:32.674124+0000) 2019-10-13T05:24:32.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:32.673+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:32.673810+0000) 2019-10-13T05:24:33.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:33.256+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.257277+0000) 2019-10-13T05:24:33.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:33.257+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.257449+0000) 2019-10-13T05:24:33.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:33.252+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.253163+0000) 2019-10-13T05:24:33.262 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:33.257+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.258183+0000) 2019-10-13T05:24:33.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:33.257+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.258330+0000) 2019-10-13T05:24:33.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:33.258+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.258569+0000) 2019-10-13T05:24:33.263 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:33.258+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.258732+0000) 2019-10-13T05:24:33.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:33.254+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:33.255445+0000) 2019-10-13T05:24:33.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:33.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76601 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:34.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:34.169+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:34.169901+0000) 2019-10-13T05:24:38.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:38.212+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.212622+0000) 2019-10-13T05:24:38.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:38.261+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.262295+0000) 2019-10-13T05:24:38.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:38.262+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.262503+0000) 2019-10-13T05:24:38.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:38.257+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.257988+0000) 2019-10-13T05:24:38.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:38.262+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.262789+0000) 2019-10-13T05:24:38.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:38.262+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.263381+0000) 2019-10-13T05:24:38.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:38.263+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.263781+0000) 2019-10-13T05:24:38.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:38.263+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.263928+0000) 2019-10-13T05:24:38.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:38.260+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.260774+0000) 2019-10-13T05:24:38.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:38.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76629 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:38.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:38.552+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.552837+0000) 2019-10-13T05:24:38.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:38.552+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:38.552837+0000) 2019-10-13T05:24:40.007 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:24:40.007 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:40.011 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:24:40.011 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:40.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:40.250+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:40.250559+0000) 2019-10-13T05:24:41.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:41.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:41.805378+0000) 2019-10-13T05:24:42.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:42.673+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:42.674266+0000) 2019-10-13T05:24:42.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:42.673+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:42.673973+0000) 2019-10-13T05:24:43.271 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:43.266+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.268262+0000) 2019-10-13T05:24:43.272 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:43.267+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.268476+0000) 2019-10-13T05:24:43.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:43.267+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.268818+0000) 2019-10-13T05:24:43.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:43.263+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.263988+0000) 2019-10-13T05:24:43.273 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:43.267+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.268991+0000) 2019-10-13T05:24:43.273 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:43.267+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.269158+0000) 2019-10-13T05:24:43.273 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:43.268+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.269677+0000) 2019-10-13T05:24:43.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:43.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76660 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:43.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:43.266+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:43.267198+0000) 2019-10-13T05:24:44.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:44.169+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:44.170073+0000) 2019-10-13T05:24:48.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:48.212+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.212867+0000) 2019-10-13T05:24:48.275 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:48.266+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.267512+0000) 2019-10-13T05:24:48.276 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:48.271+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.272419+0000) 2019-10-13T05:24:48.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:48.271+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.272496+0000) 2019-10-13T05:24:48.276 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:48.271+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.272995+0000) 2019-10-13T05:24:48.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:48.271+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.273081+0000) 2019-10-13T05:24:48.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:48.271+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.273350+0000) 2019-10-13T05:24:48.277 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:48.272+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.273842+0000) 2019-10-13T05:24:48.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:48.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:48.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:48.270+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.271125+0000) 2019-10-13T05:24:48.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:48.552+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.553002+0000) 2019-10-13T05:24:48.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:48.552+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:48.553002+0000) 2019-10-13T05:24:50.260 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:50.252+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:50.252780+0000) 2019-10-13T05:24:51.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:51.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:51.805593+0000) 2019-10-13T05:24:52.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:52.673+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:52.674427+0000) 2019-10-13T05:24:52.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:52.672+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:52.674084+0000) 2019-10-13T05:24:53.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:53.275+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.277191+0000) 2019-10-13T05:24:53.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:53.276+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.277551+0000) 2019-10-13T05:24:53.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:53.276+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.277736+0000) 2019-10-13T05:24:53.281 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:53.276+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.277888+0000) 2019-10-13T05:24:53.282 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:53.276+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.278093+0000) 2019-10-13T05:24:53.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:53.270+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.272722+0000) 2019-10-13T05:24:53.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:53.276+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.278374+0000) 2019-10-13T05:24:53.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:53.273+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:53.275146+0000) 2019-10-13T05:24:53.285 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:53.278+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:54.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:54.169+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:54.170245+0000) 2019-10-13T05:24:58.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:58.212+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.213053+0000) 2019-10-13T05:24:58.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:24:58.280+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.282086+0000) 2019-10-13T05:24:58.286 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:24:58.280+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.282229+0000) 2019-10-13T05:24:58.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:24:58.280+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.282379+0000) 2019-10-13T05:24:58.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:24:58.281+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.282724+0000) 2019-10-13T05:24:58.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:58.281+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.282960+0000) 2019-10-13T05:24:58.287 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:24:58.276+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.278396+0000) 2019-10-13T05:24:58.287 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:58.281+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.283227+0000) 2019-10-13T05:24:58.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:24:58.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76749 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:24:58.288 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:24:58.279+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.281081+0000) 2019-10-13T05:24:58.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:24:58.552+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.553169+0000) 2019-10-13T05:24:58.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:24:58.552+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:24:58.553168+0000) 2019-10-13T05:25:00.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:00.254+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:00.255176+0000) 2019-10-13T05:25:01.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:01.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:01.805782+0000) 2019-10-13T05:25:02.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:02.674+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:02.674623+0000) 2019-10-13T05:25:02.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:02.673+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:02.674211+0000) 2019-10-13T05:25:03.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:03.284+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.286036+0000) 2019-10-13T05:25:03.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:03.284+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.285931+0000) 2019-10-13T05:25:03.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:03.284+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.286263+0000) 2019-10-13T05:25:03.290 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:03.285+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.286548+0000) 2019-10-13T05:25:03.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:03.281+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.282804+0000) 2019-10-13T05:25:03.291 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:03.285+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.286925+0000) 2019-10-13T05:25:03.291 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:03.285+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.287043+0000) 2019-10-13T05:25:03.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:03.282+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:03.284210+0000) 2019-10-13T05:25:03.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:03.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:04.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:04.169+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:04.170426+0000) 2019-10-13T05:25:08.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:08.212+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.213248+0000) 2019-10-13T05:25:08.295 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:08.291+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.291560+0000) 2019-10-13T05:25:08.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:08.291+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.291422+0000) 2019-10-13T05:25:08.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:08.291+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.291804+0000) 2019-10-13T05:25:08.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:08.291+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.291906+0000) 2019-10-13T05:25:08.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:08.291+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.292229+0000) 2019-10-13T05:25:08.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:08.286+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.287458+0000) 2019-10-13T05:25:08.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:08.289+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.289839+0000) 2019-10-13T05:25:08.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:08.292+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.292538+0000) 2019-10-13T05:25:08.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:08.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76807 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:08.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:08.552+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.553339+0000) 2019-10-13T05:25:08.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:08.552+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:08.553353+0000) 2019-10-13T05:25:10.105 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:25:10.105 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:10.110 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:25:10.110 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:10.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:10.256+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:10.257619+0000) 2019-10-13T05:25:11.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:11.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:11.805991+0000) 2019-10-13T05:25:12.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:12.674+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:12.674824+0000) 2019-10-13T05:25:12.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:12.673+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:12.674360+0000) 2019-10-13T05:25:13.299 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:13.295+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.296168+0000) 2019-10-13T05:25:13.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:13.291+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.292015+0000) 2019-10-13T05:25:13.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:13.296+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.296826+0000) 2019-10-13T05:25:13.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:13.296+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.297010+0000) 2019-10-13T05:25:13.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:13.297+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.297515+0000) 2019-10-13T05:25:13.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:13.297+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.297515+0000) 2019-10-13T05:25:13.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:13.297+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.297735+0000) 2019-10-13T05:25:13.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:13.293+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:13.294697+0000) 2019-10-13T05:25:13.302 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:13.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76840 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:14.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:14.170+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:14.170592+0000) 2019-10-13T05:25:18.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:18.213+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.213461+0000) 2019-10-13T05:25:18.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:18.301+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.301928+0000) 2019-10-13T05:25:18.306 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:18.296+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.297539+0000) 2019-10-13T05:25:18.306 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:18.301+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.302263+0000) 2019-10-13T05:25:18.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:18.302+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.302614+0000) 2019-10-13T05:25:18.306 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:18.302+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.302638+0000) 2019-10-13T05:25:18.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:18.302+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.302866+0000) 2019-10-13T05:25:18.307 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:18.302+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.303100+0000) 2019-10-13T05:25:18.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:18.299+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.299777+0000) 2019-10-13T05:25:18.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:18.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:18.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:18.553+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.553545+0000) 2019-10-13T05:25:18.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:18.553+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:18.553559+0000) 2019-10-13T05:25:20.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:20.259+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:20.259882+0000) 2019-10-13T05:25:21.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:21.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:21.806189+0000) 2019-10-13T05:25:22.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:22.674+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:22.675005+0000) 2019-10-13T05:25:22.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:22.674+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:22.674566+0000) 2019-10-13T05:25:23.310 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:23.306+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.306821+0000) 2019-10-13T05:25:23.311 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:23.306+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.307180+0000) 2019-10-13T05:25:23.311 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:23.306+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.307206+0000) 2019-10-13T05:25:23.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:23.306+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.307291+0000) 2019-10-13T05:25:23.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:23.307+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.307522+0000) 2019-10-13T05:25:23.312 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:23.301+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.302357+0000) 2019-10-13T05:25:23.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:23.307+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.308231+0000) 2019-10-13T05:25:23.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:23.304+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:23.305680+0000) 2019-10-13T05:25:23.314 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:23.309+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:24.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:24.170+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:24.170762+0000) 2019-10-13T05:25:28.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:28.212+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.213621+0000) 2019-10-13T05:25:28.315 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:28.310+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.312313+0000) 2019-10-13T05:25:28.315 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:28.311+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.312679+0000) 2019-10-13T05:25:28.316 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:28.307+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.307858+0000) 2019-10-13T05:25:28.317 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:28.311+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.312934+0000) 2019-10-13T05:25:28.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:28.311+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.313177+0000) 2019-10-13T05:25:28.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:28.311+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.313345+0000) 2019-10-13T05:25:28.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:28.312+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.313687+0000) 2019-10-13T05:25:28.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:28.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76925 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:28.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:28.310+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.311010+0000) 2019-10-13T05:25:28.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:28.552+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.553715+0000) 2019-10-13T05:25:28.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:28.552+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:28.553716+0000) 2019-10-13T05:25:30.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:30.261+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:30.262209+0000) 2019-10-13T05:25:31.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:31.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:31.806378+0000) 2019-10-13T05:25:32.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:32.674+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:32.675198+0000) 2019-10-13T05:25:32.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:32.674+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:32.674775+0000) 2019-10-13T05:25:33.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:33.315+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.316785+0000) 2019-10-13T05:25:33.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:33.315+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.316933+0000) 2019-10-13T05:25:33.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:33.311+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.312491+0000) 2019-10-13T05:25:33.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:33.316+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.317547+0000) 2019-10-13T05:25:33.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:33.316+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.317701+0000) 2019-10-13T05:25:33.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:33.316+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.317829+0000) 2019-10-13T05:25:33.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:33.316+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.318018+0000) 2019-10-13T05:25:33.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:33.314+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:33.315067+0000) 2019-10-13T05:25:33.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:33.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:34.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:34.170+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:34.170895+0000) 2019-10-13T05:25:38.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:38.213+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.213782+0000) 2019-10-13T05:25:38.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:38.320+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.321866+0000) 2019-10-13T05:25:38.325 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:38.320+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.322051+0000) 2019-10-13T05:25:38.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:38.320+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.322233+0000) 2019-10-13T05:25:38.326 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:38.317+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.318456+0000) 2019-10-13T05:25:38.326 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:38.321+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.322505+0000) 2019-10-13T05:25:38.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:38.321+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.322767+0000) 2019-10-13T05:25:38.327 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:38.321+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.322966+0000) 2019-10-13T05:25:38.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:38.319+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.319889+0000) 2019-10-13T05:25:38.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:38.323+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 76985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:38.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:38.552+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.553852+0000) 2019-10-13T05:25:38.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:38.552+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:38.553889+0000) 2019-10-13T05:25:40.227 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:25:40.227 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:40.231 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:25:40.231 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:40.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:40.263+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:40.264455+0000) 2019-10-13T05:25:41.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:41.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:41.806563+0000) 2019-10-13T05:25:42.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:42.674+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:42.675380+0000) 2019-10-13T05:25:42.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:42.673+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:42.674957+0000) 2019-10-13T05:25:43.329 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:43.324+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.326094+0000) 2019-10-13T05:25:43.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:43.324+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.326292+0000) 2019-10-13T05:25:43.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:43.320+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.322257+0000) 2019-10-13T05:25:43.330 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:43.325+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.326630+0000) 2019-10-13T05:25:43.331 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:43.326+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.327886+0000) 2019-10-13T05:25:43.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:43.326+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.328135+0000) 2019-10-13T05:25:43.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:43.326+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.328047+0000) 2019-10-13T05:25:43.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:43.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77016 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:43.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:43.324+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:43.325938+0000) 2019-10-13T05:25:44.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:44.170+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:44.171064+0000) 2019-10-13T05:25:48.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:48.213+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.213964+0000) 2019-10-13T05:25:48.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:48.329+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.331334+0000) 2019-10-13T05:25:48.335 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:48.325+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.326908+0000) 2019-10-13T05:25:48.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:48.330+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.331823+0000) 2019-10-13T05:25:48.336 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:48.330+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.332039+0000) 2019-10-13T05:25:48.336 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:48.330+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.332194+0000) 2019-10-13T05:25:48.336 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:48.330+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.332301+0000) 2019-10-13T05:25:48.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:48.331+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.332571+0000) 2019-10-13T05:25:48.337 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:48.327+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.329471+0000) 2019-10-13T05:25:48.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:48.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77043 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:48.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:48.553+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.554021+0000) 2019-10-13T05:25:48.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:48.553+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:48.554055+0000) 2019-10-13T05:25:50.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:50.264+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:50.266717+0000) 2019-10-13T05:25:51.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:51.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:51.806739+0000) 2019-10-13T05:25:52.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:52.675+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:52.675613+0000) 2019-10-13T05:25:52.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:52.674+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:52.675164+0000) 2019-10-13T05:25:53.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:53.335+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.336107+0000) 2019-10-13T05:25:53.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:53.335+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.336244+0000) 2019-10-13T05:25:53.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:53.335+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.336367+0000) 2019-10-13T05:25:53.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:53.330+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.331956+0000) 2019-10-13T05:25:53.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:53.336+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.336786+0000) 2019-10-13T05:25:53.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:53.336+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.337113+0000) 2019-10-13T05:25:53.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:53.336+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.337292+0000) 2019-10-13T05:25:53.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:53.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:53.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:53.333+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:53.335069+0000) 2019-10-13T05:25:54.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:54.170+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:54.171272+0000) 2019-10-13T05:25:58.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:58.213+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.214153+0000) 2019-10-13T05:25:58.343 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:25:58.340+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.340528+0000) 2019-10-13T05:25:58.344 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:25:58.335+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.336080+0000) 2019-10-13T05:25:58.345 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:58.340+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.341053+0000) 2019-10-13T05:25:58.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:25:58.340+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.341293+0000) 2019-10-13T05:25:58.345 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:25:58.341+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.341459+0000) 2019-10-13T05:25:58.345 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:58.341+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.341461+0000) 2019-10-13T05:25:58.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:25:58.341+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.341617+0000) 2019-10-13T05:25:58.346 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:25:58.337+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.338352+0000) 2019-10-13T05:25:58.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:25:58.343+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:25:58.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:25:58.553+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.554181+0000) 2019-10-13T05:25:58.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:25:58.553+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:25:58.554211+0000) 2019-10-13T05:26:00.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:00.268+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:00.269213+0000) 2019-10-13T05:26:01.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:01.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:01.806927+0000) 2019-10-13T05:26:02.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:02.675+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:02.675782+0000) 2019-10-13T05:26:02.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:02.674+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:02.675339+0000) 2019-10-13T05:26:03.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:03.339+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.340606+0000) 2019-10-13T05:26:03.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:03.344+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.345005+0000) 2019-10-13T05:26:03.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:03.345+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.345456+0000) 2019-10-13T05:26:03.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:03.345+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.345734+0000) 2019-10-13T05:26:03.350 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:03.345+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.345967+0000) 2019-10-13T05:26:03.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:03.345+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.346161+0000) 2019-10-13T05:26:03.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:03.345+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.346100+0000) 2019-10-13T05:26:03.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:03.342+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:03.343253+0000) 2019-10-13T05:26:03.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:03.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:04.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:04.171+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:04.171492+0000) 2019-10-13T05:26:08.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:08.214+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.214328+0000) 2019-10-13T05:26:08.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:08.349+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.349826+0000) 2019-10-13T05:26:08.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:08.349+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.350066+0000) 2019-10-13T05:26:08.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:08.345+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.345969+0000) 2019-10-13T05:26:08.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:08.349+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.350351+0000) 2019-10-13T05:26:08.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:08.350+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.350517+0000) 2019-10-13T05:26:08.355 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:08.350+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.350764+0000) 2019-10-13T05:26:08.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:08.350+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.351015+0000) 2019-10-13T05:26:08.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:08.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:08.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:08.347+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.348345+0000) 2019-10-13T05:26:08.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:08.553+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.554376+0000) 2019-10-13T05:26:08.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:08.554+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:08.554385+0000) 2019-10-13T05:26:10.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:10.270+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:10.271547+0000) 2019-10-13T05:26:10.341 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:26:10.341 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:10.345 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:26:10.345 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:11.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:11.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:11.807119+0000) 2019-10-13T05:26:12.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:12.674+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:12.675910+0000) 2019-10-13T05:26:12.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:12.674+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:12.675491+0000) 2019-10-13T05:26:13.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:13.353+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.354914+0000) 2019-10-13T05:26:13.358 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:13.353+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.355027+0000) 2019-10-13T05:26:13.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:13.353+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.355226+0000) 2019-10-13T05:26:13.359 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:13.354+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.355446+0000) 2019-10-13T05:26:13.359 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:13.350+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.351339+0000) 2019-10-13T05:26:13.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:13.354+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.356194+0000) 2019-10-13T05:26:13.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:13.354+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.356329+0000) 2019-10-13T05:26:13.360 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:13.352+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:13.353172+0000) 2019-10-13T05:26:13.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:13.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:14.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:14.170+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:14.171652+0000) 2019-10-13T05:26:18.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:18.213+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.214496+0000) 2019-10-13T05:26:18.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:18.354+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.354875+0000) 2019-10-13T05:26:18.363 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:18.358+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.359701+0000) 2019-10-13T05:26:18.363 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:18.358+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.359878+0000) 2019-10-13T05:26:18.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:18.358+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.359979+0000) 2019-10-13T05:26:18.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:18.358+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.360210+0000) 2019-10-13T05:26:18.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:18.359+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.360410+0000) 2019-10-13T05:26:18.364 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:18.359+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.360843+0000) 2019-10-13T05:26:18.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:18.360+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:18.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:18.357+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.358666+0000) 2019-10-13T05:26:18.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:18.553+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.554579+0000) 2019-10-13T05:26:18.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:18.553+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:18.554580+0000) 2019-10-13T05:26:20.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:20.273+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:20.273883+0000) 2019-10-13T05:26:21.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:21.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:21.807310+0000) 2019-10-13T05:26:22.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:22.675+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:22.676097+0000) 2019-10-13T05:26:22.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:22.675+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:22.675644+0000) 2019-10-13T05:26:23.367 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:23.363+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.364406+0000) 2019-10-13T05:26:23.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:23.359+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.360467+0000) 2019-10-13T05:26:23.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:23.363+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.364759+0000) 2019-10-13T05:26:23.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:23.363+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.364930+0000) 2019-10-13T05:26:23.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:23.363+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.365245+0000) 2019-10-13T05:26:23.369 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:23.363+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.365357+0000) 2019-10-13T05:26:23.370 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:23.364+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.365941+0000) 2019-10-13T05:26:23.370 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:23.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:23.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:23.361+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:23.362570+0000) 2019-10-13T05:26:24.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:24.171+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:24.171795+0000) 2019-10-13T05:26:28.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:28.213+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.214633+0000) 2019-10-13T05:26:28.371 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:28.367+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.368460+0000) 2019-10-13T05:26:28.372 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:28.367+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.368709+0000) 2019-10-13T05:26:28.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:28.367+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.369241+0000) 2019-10-13T05:26:28.372 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:28.368+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.369570+0000) 2019-10-13T05:26:28.373 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:28.364+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.364836+0000) 2019-10-13T05:26:28.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:28.368+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.369926+0000) 2019-10-13T05:26:28.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:28.368+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.369929+0000) 2019-10-13T05:26:28.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:28.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:28.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:28.366+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.367526+0000) 2019-10-13T05:26:28.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:28.553+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.554703+0000) 2019-10-13T05:26:28.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:28.553+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:28.554754+0000) 2019-10-13T05:26:30.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:30.276+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:30.276210+0000) 2019-10-13T05:26:31.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:31.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:31.807501+0000) 2019-10-13T05:26:32.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:32.675+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:32.676277+0000) 2019-10-13T05:26:32.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:32.674+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:32.675813+0000) 2019-10-13T05:26:33.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:33.367+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.369481+0000) 2019-10-13T05:26:33.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:33.372+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.373453+0000) 2019-10-13T05:26:33.379 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:33.372+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.373648+0000) 2019-10-13T05:26:33.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:33.372+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.373972+0000) 2019-10-13T05:26:33.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:33.372+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.374270+0000) 2019-10-13T05:26:33.380 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:33.373+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.374599+0000) 2019-10-13T05:26:33.380 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:33.373+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.374889+0000) 2019-10-13T05:26:33.381 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:33.370+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:33.372244+0000) 2019-10-13T05:26:33.381 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:33.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:34.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:34.171+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:34.171964+0000) 2019-10-13T05:26:38.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:38.214+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.214820+0000) 2019-10-13T05:26:38.382 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:38.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.378384+0000) 2019-10-13T05:26:38.382 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:38.378+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.378558+0000) 2019-10-13T05:26:38.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:38.378+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.378659+0000) 2019-10-13T05:26:38.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:38.378+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.378865+0000) 2019-10-13T05:26:38.383 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:38.378+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.379044+0000) 2019-10-13T05:26:38.383 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:38.378+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.379171+0000) 2019-10-13T05:26:38.383 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:38.373+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.374801+0000) 2019-10-13T05:26:38.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:38.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:38.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:38.374+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.376479+0000) 2019-10-13T05:26:38.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:38.554+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.554885+0000) 2019-10-13T05:26:38.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:38.554+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:38.554852+0000) 2019-10-13T05:26:40.286 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:40.277+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:40.278536+0000) 2019-10-13T05:26:40.571 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:26:40.571 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:40.577 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:26:40.577 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:41.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:41.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:41.807688+0000) 2019-10-13T05:26:42.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:42.676+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:42.676463+0000) 2019-10-13T05:26:42.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:42.675+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:42.675996+0000) 2019-10-13T05:26:43.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:43.376+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.377853+0000) 2019-10-13T05:26:43.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:43.379+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.381272+0000) 2019-10-13T05:26:43.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:43.381+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.382219+0000) 2019-10-13T05:26:43.392 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:43.382+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.382686+0000) 2019-10-13T05:26:43.392 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:43.382+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.382843+0000) 2019-10-13T05:26:43.392 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:43.382+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.383153+0000) 2019-10-13T05:26:43.392 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:43.383+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.383554+0000) 2019-10-13T05:26:43.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:43.383+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:43.383426+0000) 2019-10-13T05:26:43.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:43.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77375 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:44.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:44.171+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:44.172132+0000) 2019-10-13T05:26:48.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:48.214+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.215005+0000) 2019-10-13T05:26:48.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:48.381+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.382536+0000) 2019-10-13T05:26:48.391 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:48.386+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.386866+0000) 2019-10-13T05:26:48.392 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:48.386+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.387061+0000) 2019-10-13T05:26:48.392 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:48.386+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.387366+0000) 2019-10-13T05:26:48.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:48.387+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.387428+0000) 2019-10-13T05:26:48.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:48.387+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.387799+0000) 2019-10-13T05:26:48.393 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:48.387+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.387904+0000) 2019-10-13T05:26:48.393 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:48.384+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.385076+0000) 2019-10-13T05:26:48.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:48.389+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:48.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:48.554+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.555014+0000) 2019-10-13T05:26:48.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:48.554+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:48.555017+0000) 2019-10-13T05:26:50.288 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:50.280+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:50.280765+0000) 2019-10-13T05:26:51.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:51.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:51.807913+0000) 2019-10-13T05:26:52.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:52.676+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:52.676645+0000) 2019-10-13T05:26:52.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:52.675+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:52.676183+0000) 2019-10-13T05:26:53.398 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:53.387+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.388263+0000) 2019-10-13T05:26:53.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:53.391+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.392195+0000) 2019-10-13T05:26:53.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:53.391+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.392380+0000) 2019-10-13T05:26:53.398 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:53.392+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.392551+0000) 2019-10-13T05:26:53.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:53.392+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.392714+0000) 2019-10-13T05:26:53.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:53.392+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.393065+0000) 2019-10-13T05:26:53.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:53.392+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.393259+0000) 2019-10-13T05:26:53.399 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:53.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:53.400 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:53.390+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:53.391197+0000) 2019-10-13T05:26:54.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:54.172+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:54.172304+0000) 2019-10-13T05:26:58.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:58.214+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.215221+0000) 2019-10-13T05:26:58.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:26:58.390+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.391106+0000) 2019-10-13T05:26:58.400 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:26:58.394+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.396070+0000) 2019-10-13T05:26:58.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:26:58.394+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.396231+0000) 2019-10-13T05:26:58.400 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:58.394+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.396371+0000) 2019-10-13T05:26:58.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:26:58.395+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.396768+0000) 2019-10-13T05:26:58.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:26:58.395+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.396772+0000) 2019-10-13T05:26:58.401 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:58.395+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.396965+0000) 2019-10-13T05:26:58.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:26:58.392+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.393300+0000) 2019-10-13T05:26:58.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:26:58.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:26:58.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:26:58.553+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.555190+0000) 2019-10-13T05:26:58.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:26:58.553+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:26:58.555190+0000) 2019-10-13T05:27:00.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:00.282+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:00.283121+0000) 2019-10-13T05:27:01.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:01.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:01.808102+0000) 2019-10-13T05:27:02.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:02.675+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:02.676816+0000) 2019-10-13T05:27:02.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:02.675+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:02.676363+0000) 2019-10-13T05:27:03.405 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:03.400+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.401561+0000) 2019-10-13T05:27:03.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:03.400+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.401933+0000) 2019-10-13T05:27:03.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:03.396+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.397053+0000) 2019-10-13T05:27:03.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:03.400+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.402218+0000) 2019-10-13T05:27:03.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:03.400+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.402259+0000) 2019-10-13T05:27:03.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:03.401+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.402425+0000) 2019-10-13T05:27:03.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:03.401+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.402684+0000) 2019-10-13T05:27:03.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:03.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77495 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:03.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:03.399+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:03.400357+0000) 2019-10-13T05:27:04.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:04.171+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:04.172455+0000) 2019-10-13T05:27:08.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:08.214+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.215428+0000) 2019-10-13T05:27:08.409 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:08.400+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.401384+0000) 2019-10-13T05:27:08.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:08.405+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.406516+0000) 2019-10-13T05:27:08.410 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:08.405+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.406646+0000) 2019-10-13T05:27:08.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:08.405+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.407104+0000) 2019-10-13T05:27:08.411 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:08.405+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.407203+0000) 2019-10-13T05:27:08.411 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:08.406+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.407572+0000) 2019-10-13T05:27:08.411 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:08.406+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.407540+0000) 2019-10-13T05:27:08.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:08.403+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.404578+0000) 2019-10-13T05:27:08.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:08.408+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:08.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:08.554+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.555342+0000) 2019-10-13T05:27:08.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:08.554+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:08.555340+0000) 2019-10-13T05:27:10.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:10.285+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:10.285431+0000) 2019-10-13T05:27:10.776 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:27:10.777 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:10.781 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:27:10.781 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:11.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:11.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:11.808299+0000) 2019-10-13T05:27:12.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:12.675+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:12.677040+0000) 2019-10-13T05:27:12.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:12.676+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:12.676536+0000) 2019-10-13T05:27:13.414 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:13.405+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.406714+0000) 2019-10-13T05:27:13.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:13.410+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.411553+0000) 2019-10-13T05:27:13.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:13.410+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.411722+0000) 2019-10-13T05:27:13.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:13.410+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.411928+0000) 2019-10-13T05:27:13.416 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:13.410+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.412244+0000) 2019-10-13T05:27:13.416 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:13.410+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.412351+0000) 2019-10-13T05:27:13.416 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:13.411+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.412577+0000) 2019-10-13T05:27:13.419 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:13.409+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:13.410252+0000) 2019-10-13T05:27:13.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:13.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:14.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:14.171+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:14.172625+0000) 2019-10-13T05:27:18.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:18.215+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.215622+0000) 2019-10-13T05:27:18.419 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:18.410+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.411605+0000) 2019-10-13T05:27:18.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:18.415+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.416484+0000) 2019-10-13T05:27:18.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:18.415+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.416797+0000) 2019-10-13T05:27:18.420 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:18.415+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.417036+0000) 2019-10-13T05:27:18.421 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:18.415+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.417133+0000) 2019-10-13T05:27:18.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:18.415+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.417367+0000) 2019-10-13T05:27:18.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:18.413+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.413983+0000) 2019-10-13T05:27:18.422 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:18.416+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.417781+0000) 2019-10-13T05:27:18.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:18.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:18.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:18.554+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.555510+0000) 2019-10-13T05:27:18.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:18.554+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:18.555510+0000) 2019-10-13T05:27:20.296 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:20.286+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:20.288397+0000) 2019-10-13T05:27:21.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:21.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:21.808526+0000) 2019-10-13T05:27:22.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:22.676+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:22.677232+0000) 2019-10-13T05:27:22.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:22.675+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:22.676697+0000) 2019-10-13T05:27:23.425 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:23.419+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.421275+0000) 2019-10-13T05:27:23.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:23.419+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.421175+0000) 2019-10-13T05:27:23.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:23.420+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.421478+0000) 2019-10-13T05:27:23.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:23.420+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.421588+0000) 2019-10-13T05:27:23.426 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:23.420+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.421971+0000) 2019-10-13T05:27:23.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:23.420+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.422255+0000) 2019-10-13T05:27:23.427 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:23.415+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.417447+0000) 2019-10-13T05:27:23.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:23.417+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:23.419013+0000) 2019-10-13T05:27:23.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:23.421+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:24.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:24.172+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:24.172787+0000) 2019-10-13T05:27:28.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:28.215+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.215763+0000) 2019-10-13T05:27:28.428 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:28.424+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.424996+0000) 2019-10-13T05:27:28.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:28.424+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.425274+0000) 2019-10-13T05:27:28.429 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:28.425+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.425720+0000) 2019-10-13T05:27:28.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:28.425+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.425756+0000) 2019-10-13T05:27:28.429 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:28.424+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.425100+0000) 2019-10-13T05:27:28.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:28.425+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.425887+0000) 2019-10-13T05:27:28.430 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:28.420+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.421946+0000) 2019-10-13T05:27:28.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:28.427+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:28.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:28.421+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.423643+0000) 2019-10-13T05:27:28.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:28.555+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.555692+0000) 2019-10-13T05:27:28.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:28.555+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:28.555689+0000) 2019-10-13T05:27:30.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:30.289+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:30.290792+0000) 2019-10-13T05:27:31.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:31.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:31.808727+0000) 2019-10-13T05:27:32.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:32.677+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:32.677388+0000) 2019-10-13T05:27:32.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:32.676+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:32.676863+0000) 2019-10-13T05:27:33.433 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:33.429+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.429960+0000) 2019-10-13T05:27:33.433 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:33.429+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.430112+0000) 2019-10-13T05:27:33.434 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:33.430+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.430743+0000) 2019-10-13T05:27:33.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:33.430+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.430969+0000) 2019-10-13T05:27:33.434 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:33.430+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.431041+0000) 2019-10-13T05:27:33.435 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:33.430+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.431217+0000) 2019-10-13T05:27:33.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:33.424+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.426682+0000) 2019-10-13T05:27:33.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:33.425+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:33.427648+0000) 2019-10-13T05:27:33.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:33.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:34.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:34.172+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:34.172929+0000) 2019-10-13T05:27:38.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:38.215+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.215953+0000) 2019-10-13T05:27:38.437 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:38.433+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.433612+0000) 2019-10-13T05:27:38.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:38.433+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.433413+0000) 2019-10-13T05:27:38.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:38.433+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.434010+0000) 2019-10-13T05:27:38.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:38.433+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.434258+0000) 2019-10-13T05:27:38.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:38.433+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.433778+0000) 2019-10-13T05:27:38.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:38.434+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.434429+0000) 2019-10-13T05:27:38.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:38.429+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.429863+0000) 2019-10-13T05:27:38.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:38.430+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.431516+0000) 2019-10-13T05:27:38.439 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:38.436+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:38.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:38.555+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.555817+0000) 2019-10-13T05:27:38.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:38.555+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:38.555857+0000) 2019-10-13T05:27:40.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:40.292+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:40.293283+0000) 2019-10-13T05:27:40.991 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:27:40.991 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:40.995 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:27:40.995 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:41.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:41.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:41.808897+0000) 2019-10-13T05:27:42.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:42.676+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:42.677518+0000) 2019-10-13T05:27:42.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:42.676+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:42.677050+0000) 2019-10-13T05:27:43.441 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:43.437+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.438458+0000) 2019-10-13T05:27:43.442 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:43.433+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.434042+0000) 2019-10-13T05:27:43.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:43.437+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.438922+0000) 2019-10-13T05:27:43.443 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:43.437+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.439181+0000) 2019-10-13T05:27:43.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:43.437+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.439283+0000) 2019-10-13T05:27:43.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:43.438+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.439431+0000) 2019-10-13T05:27:43.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:43.438+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.439811+0000) 2019-10-13T05:27:43.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:43.435+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:43.435881+0000) 2019-10-13T05:27:43.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:43.440+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:44.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:44.172+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:44.173098+0000) 2019-10-13T05:27:48.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:48.215+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.216148+0000) 2019-10-13T05:27:48.446 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:48.438+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.438813+0000) 2019-10-13T05:27:48.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:48.443+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.444467+0000) 2019-10-13T05:27:48.448 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:48.443+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.444645+0000) 2019-10-13T05:27:48.448 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:48.443+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.445110+0000) 2019-10-13T05:27:48.448 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:48.443+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.445303+0000) 2019-10-13T05:27:48.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:48.444+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.445416+0000) 2019-10-13T05:27:48.449 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:48.444+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.445797+0000) 2019-10-13T05:27:48.449 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:48.440+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.441184+0000) 2019-10-13T05:27:48.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:48.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:48.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:48.555+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.556035+0000) 2019-10-13T05:27:48.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:48.555+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:48.556034+0000) 2019-10-13T05:27:50.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:50.295+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:50.295624+0000) 2019-10-13T05:27:51.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:51.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:51.809109+0000) 2019-10-13T05:27:52.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:52.676+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:52.677712+0000) 2019-10-13T05:27:52.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:52.676+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:52.677219+0000) 2019-10-13T05:27:53.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:53.447+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.448871+0000) 2019-10-13T05:27:53.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:53.447+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.448945+0000) 2019-10-13T05:27:53.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:53.443+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.444582+0000) 2019-10-13T05:27:53.454 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:53.448+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.449485+0000) 2019-10-13T05:27:53.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:53.448+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.449663+0000) 2019-10-13T05:27:53.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:53.448+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.449830+0000) 2019-10-13T05:27:53.454 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:53.448+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.450008+0000) 2019-10-13T05:27:53.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:53.445+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:53.446477+0000) 2019-10-13T05:27:53.455 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:53.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:54.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:54.172+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:54.173260+0000) 2019-10-13T05:27:58.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:58.215+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.216342+0000) 2019-10-13T05:27:58.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:27:58.448+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.449078+0000) 2019-10-13T05:27:58.458 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:58.452+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.454099+0000) 2019-10-13T05:27:58.459 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:27:58.453+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.454393+0000) 2019-10-13T05:27:58.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:27:58.452+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.454291+0000) 2019-10-13T05:27:58.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:58.453+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.454757+0000) 2019-10-13T05:27:58.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:27:58.453+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.454823+0000) 2019-10-13T05:27:58.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:27:58.453+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.455141+0000) 2019-10-13T05:27:58.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:27:58.450+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.451248+0000) 2019-10-13T05:27:58.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:27:58.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:27:58.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:27:58.555+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.556190+0000) 2019-10-13T05:27:58.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:27:58.555+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:27:58.556181+0000) 2019-10-13T05:28:00.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:00.297+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:00.298096+0000) 2019-10-13T05:28:01.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:01.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:01.809285+0000) 2019-10-13T05:28:02.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:02.677+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:02.677889+0000) 2019-10-13T05:28:02.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:02.676+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:02.677391+0000) 2019-10-13T05:28:03.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:03.453+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.454445+0000) 2019-10-13T05:28:03.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:03.458+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.459392+0000) 2019-10-13T05:28:03.463 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:03.457+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.459304+0000) 2019-10-13T05:28:03.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:03.458+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.459688+0000) 2019-10-13T05:28:03.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:03.458+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.459588+0000) 2019-10-13T05:28:03.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:03.458+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.459862+0000) 2019-10-13T05:28:03.464 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:03.458+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.459966+0000) 2019-10-13T05:28:03.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:03.456+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:03.457558+0000) 2019-10-13T05:28:03.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:03.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77855 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:04.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:04.172+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:04.173464+0000) 2019-10-13T05:28:08.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:08.216+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.216570+0000) 2019-10-13T05:28:08.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:08.462+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.463942+0000) 2019-10-13T05:28:08.468 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:08.462+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.464164+0000) 2019-10-13T05:28:08.468 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:08.463+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.464473+0000) 2019-10-13T05:28:08.468 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:08.463+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.464668+0000) 2019-10-13T05:28:08.469 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:08.463+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.464874+0000) 2019-10-13T05:28:08.470 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:08.458+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.459611+0000) 2019-10-13T05:28:08.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:08.464+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.465555+0000) 2019-10-13T05:28:08.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:08.465+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:08.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:08.461+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.462593+0000) 2019-10-13T05:28:08.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:08.555+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.556366+0000) 2019-10-13T05:28:08.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:08.555+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:08.556367+0000) 2019-10-13T05:28:10.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:10.299+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:10.300541+0000) 2019-10-13T05:28:11.096 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:28:11.096 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:11.100 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:28:11.100 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:11.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:11.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:11.809511+0000) 2019-10-13T05:28:12.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:12.677+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:12.678094+0000) 2019-10-13T05:28:12.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:12.676+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:12.677573+0000) 2019-10-13T05:28:13.471 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:13.468+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.468432+0000) 2019-10-13T05:28:13.472 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:13.468+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.468754+0000) 2019-10-13T05:28:13.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:13.468+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.469088+0000) 2019-10-13T05:28:13.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:13.468+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.469192+0000) 2019-10-13T05:28:13.473 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:13.469+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.469528+0000) 2019-10-13T05:28:13.473 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:13.468+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.469073+0000) 2019-10-13T05:28:13.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:13.462+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.464667+0000) 2019-10-13T05:28:13.474 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:13.464+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:13.465767+0000) 2019-10-13T05:28:13.474 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:13.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:14.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:14.173+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:14.173613+0000) 2019-10-13T05:28:18.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:18.216+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.216755+0000) 2019-10-13T05:28:18.476 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:18.466+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.468288+0000) 2019-10-13T05:28:18.477 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:18.473+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.473547+0000) 2019-10-13T05:28:18.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:18.473+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.473690+0000) 2019-10-13T05:28:18.477 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:18.473+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.473974+0000) 2019-10-13T05:28:18.477 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:18.473+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.474132+0000) 2019-10-13T05:28:18.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:18.473+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.474307+0000) 2019-10-13T05:28:18.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:18.474+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.474530+0000) 2019-10-13T05:28:18.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:18.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:18.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:18.470+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.472428+0000) 2019-10-13T05:28:18.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:18.556+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.556534+0000) 2019-10-13T05:28:18.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:18.556+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:18.556536+0000) 2019-10-13T05:28:20.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:20.302+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:20.303071+0000) 2019-10-13T05:28:21.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:21.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:21.809663+0000) 2019-10-13T05:28:22.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:22.678+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:22.678258+0000) 2019-10-13T05:28:22.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:22.677+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:22.677761+0000) 2019-10-13T05:28:23.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:23.477+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.478221+0000) 2019-10-13T05:28:23.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:23.477+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.478387+0000) 2019-10-13T05:28:23.482 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:23.478+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.478674+0000) 2019-10-13T05:28:23.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:23.478+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.478793+0000) 2019-10-13T05:28:23.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:23.473+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.474717+0000) 2019-10-13T05:28:23.484 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:23.478+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.479099+0000) 2019-10-13T05:28:23.484 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:23.478+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.479188+0000) 2019-10-13T05:28:23.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:23.480+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 77975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:23.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:23.475+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:23.475927+0000) 2019-10-13T05:28:24.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:24.173+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:24.173782+0000) 2019-10-13T05:28:28.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:28.215+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.216933+0000) 2019-10-13T05:28:28.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:28.480+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.482294+0000) 2019-10-13T05:28:28.486 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:28.481+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.482614+0000) 2019-10-13T05:28:28.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:28.481+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.482904+0000) 2019-10-13T05:28:28.486 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:28.481+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.483000+0000) 2019-10-13T05:28:28.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:28.477+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.478557+0000) 2019-10-13T05:28:28.487 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:28.482+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.483612+0000) 2019-10-13T05:28:28.488 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:28.482+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.483880+0000) 2019-10-13T05:28:28.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:28.479+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.480002+0000) 2019-10-13T05:28:28.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:28.484+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:28.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:28.555+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.556709+0000) 2019-10-13T05:28:28.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:28.555+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:28.556710+0000) 2019-10-13T05:28:30.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:30.304+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:30.305518+0000) 2019-10-13T05:28:31.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:31.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:31.809898+0000) 2019-10-13T05:28:32.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:32.677+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:32.678440+0000) 2019-10-13T05:28:32.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:32.677+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:32.677937+0000) 2019-10-13T05:28:33.491 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:33.486+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.487892+0000) 2019-10-13T05:28:33.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:33.483+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.484375+0000) 2019-10-13T05:28:33.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:33.486+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.488238+0000) 2019-10-13T05:28:33.493 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:33.487+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.488732+0000) 2019-10-13T05:28:33.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:33.487+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.488563+0000) 2019-10-13T05:28:33.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:33.487+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.488924+0000) 2019-10-13T05:28:33.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:33.487+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.488840+0000) 2019-10-13T05:28:33.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:33.484+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:33.485442+0000) 2019-10-13T05:28:33.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:33.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:34.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:34.173+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:34.173965+0000) 2019-10-13T05:28:38.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:38.216+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.217125+0000) 2019-10-13T05:28:38.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:38.490+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.492367+0000) 2019-10-13T05:28:38.496 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:38.491+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.492556+0000) 2019-10-13T05:28:38.496 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:38.491+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.492664+0000) 2019-10-13T05:28:38.497 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:38.488+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.488877+0000) 2019-10-13T05:28:38.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:38.491+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.492807+0000) 2019-10-13T05:28:38.497 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:38.491+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.493191+0000) 2019-10-13T05:28:38.497 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:38.492+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.493571+0000) 2019-10-13T05:28:38.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:38.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:38.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:38.489+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.490488+0000) 2019-10-13T05:28:38.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:38.556+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.556896+0000) 2019-10-13T05:28:38.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:38.556+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:38.556897+0000) 2019-10-13T05:28:40.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:40.307+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:40.307545+0000) 2019-10-13T05:28:41.294 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:28:41.294 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:41.298 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:28:41.299 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:41.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:41.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:41.810123+0000) 2019-10-13T05:28:42.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:42.677+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:42.678624+0000) 2019-10-13T05:28:42.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:42.677+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:42.678109+0000) 2019-10-13T05:28:43.500 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:43.495+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.497028+0000) 2019-10-13T05:28:43.500 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:43.495+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.497384+0000) 2019-10-13T05:28:43.501 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:43.492+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.493025+0000) 2019-10-13T05:28:43.501 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:43.496+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.497752+0000) 2019-10-13T05:28:43.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:43.496+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.497885+0000) 2019-10-13T05:28:43.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:43.496+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.498131+0000) 2019-10-13T05:28:43.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:43.497+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.498516+0000) 2019-10-13T05:28:43.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:43.494+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:43.495426+0000) 2019-10-13T05:28:43.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:43.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:44.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:44.173+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:44.174147+0000) 2019-10-13T05:28:48.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:48.216+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.217302+0000) 2019-10-13T05:28:48.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:48.500+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.501688+0000) 2019-10-13T05:28:48.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:48.500+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.501833+0000) 2019-10-13T05:28:48.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:48.496+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.497639+0000) 2019-10-13T05:28:48.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:48.501+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.502439+0000) 2019-10-13T05:28:48.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:48.501+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.502731+0000) 2019-10-13T05:28:48.507 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:48.501+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.503140+0000) 2019-10-13T05:28:48.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:48.501+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.503075+0000) 2019-10-13T05:28:48.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:48.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:48.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:48.500+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.501004+0000) 2019-10-13T05:28:48.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:48.556+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.557079+0000) 2019-10-13T05:28:48.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:48.556+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:48.557077+0000) 2019-10-13T05:28:50.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:50.309+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:50.309900+0000) 2019-10-13T05:28:51.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:51.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:51.810370+0000) 2019-10-13T05:28:52.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:52.678+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:52.678810+0000) 2019-10-13T05:28:52.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:52.678+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:52.678330+0000) 2019-10-13T05:28:53.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:53.501+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.502152+0000) 2019-10-13T05:28:53.510 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:53.505+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.507280+0000) 2019-10-13T05:28:53.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:53.506+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.507675+0000) 2019-10-13T05:28:53.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:53.506+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.507924+0000) 2019-10-13T05:28:53.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:53.506+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.508101+0000) 2019-10-13T05:28:53.512 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:53.506+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.508224+0000) 2019-10-13T05:28:53.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:53.506+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.508334+0000) 2019-10-13T05:28:53.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:53.503+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:53.504332+0000) 2019-10-13T05:28:53.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:53.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:54.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:54.173+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:54.174350+0000) 2019-10-13T05:28:58.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:58.217+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.217508+0000) 2019-10-13T05:28:58.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:28:58.511+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.511702+0000) 2019-10-13T05:28:58.515 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:58.511+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.511886+0000) 2019-10-13T05:28:58.515 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:28:58.511+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.512134+0000) 2019-10-13T05:28:58.516 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:28:58.506+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.507593+0000) 2019-10-13T05:28:58.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:28:58.511+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.512351+0000) 2019-10-13T05:28:58.517 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:58.512+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.512543+0000) 2019-10-13T05:28:58.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:28:58.512+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.512984+0000) 2019-10-13T05:28:58.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:28:58.514+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:28:58.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:28:58.509+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.510334+0000) 2019-10-13T05:28:58.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:28:58.556+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.557263+0000) 2019-10-13T05:28:58.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:28:58.556+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:28:58.557263+0000) 2019-10-13T05:29:00.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:00.311+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:00.312333+0000) 2019-10-13T05:29:01.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:01.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:01.810587+0000) 2019-10-13T05:29:02.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:02.678+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:02.678969+0000) 2019-10-13T05:29:02.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:02.677+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:02.678468+0000) 2019-10-13T05:29:03.519 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:03.509+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.511701+0000) 2019-10-13T05:29:03.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:03.515+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.516309+0000) 2019-10-13T05:29:03.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:03.516+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.516821+0000) 2019-10-13T05:29:03.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:03.516+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.517199+0000) 2019-10-13T05:29:03.521 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:03.516+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.517276+0000) 2019-10-13T05:29:03.521 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:03.517+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.517684+0000) 2019-10-13T05:29:03.521 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:03.517+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.517767+0000) 2019-10-13T05:29:03.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:03.512+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:03.514548+0000) 2019-10-13T05:29:03.523 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:03.519+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:04.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:04.174+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:04.174538+0000) 2019-10-13T05:29:08.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:08.217+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.217701+0000) 2019-10-13T05:29:08.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:08.521+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.521681+0000) 2019-10-13T05:29:08.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:08.521+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.521843+0000) 2019-10-13T05:29:08.526 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:08.521+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.522386+0000) 2019-10-13T05:29:08.526 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:08.522+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.522522+0000) 2019-10-13T05:29:08.527 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:08.516+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.518505+0000) 2019-10-13T05:29:08.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:08.522+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.522855+0000) 2019-10-13T05:29:08.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:08.523+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.523394+0000) 2019-10-13T05:29:08.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:08.517+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.519403+0000) 2019-10-13T05:29:08.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:08.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78246 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:08.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:08.557+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.557422+0000) 2019-10-13T05:29:08.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:08.557+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:08.557426+0000) 2019-10-13T05:29:10.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:10.313+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:10.314394+0000) 2019-10-13T05:29:11.474 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:29:11.474 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:11.479 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:29:11.479 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:11.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:11.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:11.810779+0000) 2019-10-13T05:29:12.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:12.678+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:12.679116+0000) 2019-10-13T05:29:12.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:12.677+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:12.678608+0000) 2019-10-13T05:29:13.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:13.524+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.526249+0000) 2019-10-13T05:29:13.529 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:13.525+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.526467+0000) 2019-10-13T05:29:13.530 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:13.521+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.522562+0000) 2019-10-13T05:29:13.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:13.525+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.526824+0000) 2019-10-13T05:29:13.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:13.525+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.526948+0000) 2019-10-13T05:29:13.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:13.525+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.527363+0000) 2019-10-13T05:29:13.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:13.526+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.527775+0000) 2019-10-13T05:29:13.537 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:13.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:13.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:13.524+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:13.525104+0000) 2019-10-13T05:29:14.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:14.173+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:14.174681+0000) 2019-10-13T05:29:18.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:18.217+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.217914+0000) 2019-10-13T05:29:18.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:18.529+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.531272+0000) 2019-10-13T05:29:18.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:18.530+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.531449+0000) 2019-10-13T05:29:18.536 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:18.530+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.531918+0000) 2019-10-13T05:29:18.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:18.530+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.531849+0000) 2019-10-13T05:29:18.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:18.530+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.532071+0000) 2019-10-13T05:29:18.538 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:18.530+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.532173+0000) 2019-10-13T05:29:18.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:18.527+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.527927+0000) 2019-10-13T05:29:18.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:18.528+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.529158+0000) 2019-10-13T05:29:18.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:18.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:18.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:18.556+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.557521+0000) 2019-10-13T05:29:18.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:18.556+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:18.557570+0000) 2019-10-13T05:29:20.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:20.316+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:20.316977+0000) 2019-10-13T05:29:21.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:21.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:21.810960+0000) 2019-10-13T05:29:22.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:22.678+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:22.679325+0000) 2019-10-13T05:29:22.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:22.678+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:22.678779+0000) 2019-10-13T05:29:23.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:23.534+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.536160+0000) 2019-10-13T05:29:23.539 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:23.534+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.536304+0000) 2019-10-13T05:29:23.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:23.531+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.532384+0000) 2019-10-13T05:29:23.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:23.535+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.536629+0000) 2019-10-13T05:29:23.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:23.535+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.536859+0000) 2019-10-13T05:29:23.541 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:23.535+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.537006+0000) 2019-10-13T05:29:23.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:23.535+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.537243+0000) 2019-10-13T05:29:23.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:23.533+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:23.533865+0000) 2019-10-13T05:29:23.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:23.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:24.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:24.174+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:24.174831+0000) 2019-10-13T05:29:28.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:28.217+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.218107+0000) 2019-10-13T05:29:28.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:28.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.541101+0000) 2019-10-13T05:29:28.545 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:28.539+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.541217+0000) 2019-10-13T05:29:28.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:28.540+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.541599+0000) 2019-10-13T05:29:28.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:28.540+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.541788+0000) 2019-10-13T05:29:28.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:28.540+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.541890+0000) 2019-10-13T05:29:28.548 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:28.540+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.542170+0000) 2019-10-13T05:29:28.549 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:28.536+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.537382+0000) 2019-10-13T05:29:28.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:28.539+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.540054+0000) 2019-10-13T05:29:28.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:28.543+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:28.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:28.557+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.557698+0000) 2019-10-13T05:29:28.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:28.557+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:28.557696+0000) 2019-10-13T05:29:30.328 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:30.318+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:30.319369+0000) 2019-10-13T05:29:31.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:31.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:31.811137+0000) 2019-10-13T05:29:32.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:32.678+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:32.679531+0000) 2019-10-13T05:29:32.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:32.678+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:32.678987+0000) 2019-10-13T05:29:33.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:33.544+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.546178+0000) 2019-10-13T05:29:33.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:33.544+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.546342+0000) 2019-10-13T05:29:33.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:33.541+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.541851+0000) 2019-10-13T05:29:33.551 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:33.545+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.546947+0000) 2019-10-13T05:29:33.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:33.545+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.547133+0000) 2019-10-13T05:29:33.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:33.546+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.547478+0000) 2019-10-13T05:29:33.551 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:33.546+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.547768+0000) 2019-10-13T05:29:33.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:33.547+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:33.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:33.544+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:33.544947+0000) 2019-10-13T05:29:34.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:34.174+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:34.174983+0000) 2019-10-13T05:29:38.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:38.217+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.218303+0000) 2019-10-13T05:29:38.554 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:38.545+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.546239+0000) 2019-10-13T05:29:38.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:38.549+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.551373+0000) 2019-10-13T05:29:38.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:38.550+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.551655+0000) 2019-10-13T05:29:38.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:38.550+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.551711+0000) 2019-10-13T05:29:38.555 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:38.550+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.551920+0000) 2019-10-13T05:29:38.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:38.550+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.552100+0000) 2019-10-13T05:29:38.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:38.547+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.548572+0000) 2019-10-13T05:29:38.556 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:38.550+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.552310+0000) 2019-10-13T05:29:38.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:38.552+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:38.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:38.556+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.557809+0000) 2019-10-13T05:29:38.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:38.556+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:38.557808+0000) 2019-10-13T05:29:40.329 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:40.321+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:40.321601+0000) 2019-10-13T05:29:41.629 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:29:41.629 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:41.634 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:29:41.634 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:41.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:41.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:41.811331+0000) 2019-10-13T05:29:42.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:42.679+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:42.679719+0000) 2019-10-13T05:29:42.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:42.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:42.679170+0000) 2019-10-13T05:29:43.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:43.555+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.555929+0000) 2019-10-13T05:29:43.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:43.555+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.556078+0000) 2019-10-13T05:29:43.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:43.556+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.556440+0000) 2019-10-13T05:29:43.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:43.556+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.556629+0000) 2019-10-13T05:29:43.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:43.556+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.556912+0000) 2019-10-13T05:29:43.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:43.556+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.557119+0000) 2019-10-13T05:29:43.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:43.551+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.552003+0000) 2019-10-13T05:29:43.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:43.552+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:43.553392+0000) 2019-10-13T05:29:43.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:43.558+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78455 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:44.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:44.174+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:44.175109+0000) 2019-10-13T05:29:48.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:48.218+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.218466+0000) 2019-10-13T05:29:48.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:48.557+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.557936+0000) 2019-10-13T05:29:48.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:48.557+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.557934+0000) 2019-10-13T05:29:48.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:48.559+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.560144+0000) 2019-10-13T05:29:48.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:48.560+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.560467+0000) 2019-10-13T05:29:48.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:48.560+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.560548+0000) 2019-10-13T05:29:48.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:48.560+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.560744+0000) 2019-10-13T05:29:48.564 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:48.560+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.560826+0000) 2019-10-13T05:29:48.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:48.556+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.556803+0000) 2019-10-13T05:29:48.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:48.560+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.561130+0000) 2019-10-13T05:29:48.566 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:48.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:48.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:48.557+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:48.558652+0000) 2019-10-13T05:29:50.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:50.322+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:50.323962+0000) 2019-10-13T05:29:51.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:51.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:51.811517+0000) 2019-10-13T05:29:52.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:52.679+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:52.679907+0000) 2019-10-13T05:29:52.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:52.678+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:52.679337+0000) 2019-10-13T05:29:53.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:53.564+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.565134+0000) 2019-10-13T05:29:53.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:53.564+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.565271+0000) 2019-10-13T05:29:53.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:53.565+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.565427+0000) 2019-10-13T05:29:53.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:53.565+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.566032+0000) 2019-10-13T05:29:53.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:53.566+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.566464+0000) 2019-10-13T05:29:53.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:53.566+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.566618+0000) 2019-10-13T05:29:53.570 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:53.560+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.562098+0000) 2019-10-13T05:29:53.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:53.561+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:53.562978+0000) 2019-10-13T05:29:53.571 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:53.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:54.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:54.175+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:54.175305+0000) 2019-10-13T05:29:58.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:58.217+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.218663+0000) 2019-10-13T05:29:58.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:58.557+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.558108+0000) 2019-10-13T05:29:58.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:58.557+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.558106+0000) 2019-10-13T05:29:58.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:29:58.568+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.569922+0000) 2019-10-13T05:29:58.573 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:29:58.568+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.570029+0000) 2019-10-13T05:29:58.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:29:58.568+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.570154+0000) 2019-10-13T05:29:58.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:29:58.569+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.570401+0000) 2019-10-13T05:29:58.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:29:58.564+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.565878+0000) 2019-10-13T05:29:58.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:29:58.569+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.570899+0000) 2019-10-13T05:29:58.575 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:29:58.569+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.570792+0000) 2019-10-13T05:29:58.575 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:29:58.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:29:58.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:29:58.566+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:29:58.568401+0000) 2019-10-13T05:30:00.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:00.325+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:00.326453+0000) 2019-10-13T05:30:01.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:01.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:01.811651+0000) 2019-10-13T05:30:02.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:02.679+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:02.680091+0000) 2019-10-13T05:30:02.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:02.678+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:02.679486+0000) 2019-10-13T05:30:03.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:03.572+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.574130+0000) 2019-10-13T05:30:03.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:03.568+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.569660+0000) 2019-10-13T05:30:03.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:03.573+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.574609+0000) 2019-10-13T05:30:03.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:03.573+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.574768+0000) 2019-10-13T05:30:03.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:03.573+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.574948+0000) 2019-10-13T05:30:03.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:03.574+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.575569+0000) 2019-10-13T05:30:03.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:03.574+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.575681+0000) 2019-10-13T05:30:03.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:03.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78575 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:03.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:03.571+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:03.572531+0000) 2019-10-13T05:30:04.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:04.174+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:04.175473+0000) 2019-10-13T05:30:08.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:08.218+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.218866+0000) 2019-10-13T05:30:08.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:08.557+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.558313+0000) 2019-10-13T05:30:08.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:08.557+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.558313+0000) 2019-10-13T05:30:08.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:08.577+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.579337+0000) 2019-10-13T05:30:08.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:08.578+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.579464+0000) 2019-10-13T05:30:08.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:08.578+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.579703+0000) 2019-10-13T05:30:08.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:08.578+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.580006+0000) 2019-10-13T05:30:08.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:08.578+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.580037+0000) 2019-10-13T05:30:08.584 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:08.579+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.580858+0000) 2019-10-13T05:30:08.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:08.575+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.575860+0000) 2019-10-13T05:30:08.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:08.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:08.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:08.577+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:08.577881+0000) 2019-10-13T05:30:10.336 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:10.328+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:10.328815+0000) 2019-10-13T05:30:11.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:11.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:11.812409+0000) 2019-10-13T05:30:11.817 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:30:11.817 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:11.821 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:30:11.822 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:12.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:12.679+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:12.680241+0000) 2019-10-13T05:30:12.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:12.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:12.679639+0000) 2019-10-13T05:30:13.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:13.582+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.584174+0000) 2019-10-13T05:30:13.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:13.583+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.584401+0000) 2019-10-13T05:30:13.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:13.583+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.584541+0000) 2019-10-13T05:30:13.589 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:13.579+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.579990+0000) 2019-10-13T05:30:13.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:13.583+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.584815+0000) 2019-10-13T05:30:13.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:13.583+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.584968+0000) 2019-10-13T05:30:13.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:13.583+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.585290+0000) 2019-10-13T05:30:13.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:13.581+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:13.581766+0000) 2019-10-13T05:30:13.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:13.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:14.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:14.174+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:14.175643+0000) 2019-10-13T05:30:18.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:18.217+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.219104+0000) 2019-10-13T05:30:18.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:18.557+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.558500+0000) 2019-10-13T05:30:18.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:18.557+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.558534+0000) 2019-10-13T05:30:18.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:18.587+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.588573+0000) 2019-10-13T05:30:18.592 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:18.587+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.588875+0000) 2019-10-13T05:30:18.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:18.583+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.584016+0000) 2019-10-13T05:30:18.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:18.587+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.589170+0000) 2019-10-13T05:30:18.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:18.588+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.589481+0000) 2019-10-13T05:30:18.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:18.588+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.589887+0000) 2019-10-13T05:30:18.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:18.588+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.589947+0000) 2019-10-13T05:30:18.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:18.585+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:18.586656+0000) 2019-10-13T05:30:18.594 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:18.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:20.339 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:20.330+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:20.331118+0000) 2019-10-13T05:30:21.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:21.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:21.812581+0000) 2019-10-13T05:30:22.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:22.679+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:22.680388+0000) 2019-10-13T05:30:22.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:22.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:22.679786+0000) 2019-10-13T05:30:23.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:23.591+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.593194+0000) 2019-10-13T05:30:23.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:23.592+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.593487+0000) 2019-10-13T05:30:23.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:23.592+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.593706+0000) 2019-10-13T05:30:23.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:23.588+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.589330+0000) 2019-10-13T05:30:23.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:23.592+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.594346+0000) 2019-10-13T05:30:23.598 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:23.593+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.594489+0000) 2019-10-13T05:30:23.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:23.593+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.594706+0000) 2019-10-13T05:30:23.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:23.590+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:23.591527+0000) 2019-10-13T05:30:23.600 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:23.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:24.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:24.175+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:24.175824+0000) 2019-10-13T05:30:28.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:28.218+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.219310+0000) 2019-10-13T05:30:28.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:28.558+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.558688+0000) 2019-10-13T05:30:28.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:28.558+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.558727+0000) 2019-10-13T05:30:28.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:28.598+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.599008+0000) 2019-10-13T05:30:28.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:28.598+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.599232+0000) 2019-10-13T05:30:28.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:28.594+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.595125+0000) 2019-10-13T05:30:28.603 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:28.599+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.599562+0000) 2019-10-13T05:30:28.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:28.599+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.600019+0000) 2019-10-13T05:30:28.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:28.599+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.600179+0000) 2019-10-13T05:30:28.604 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:28.600+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.600489+0000) 2019-10-13T05:30:28.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:28.596+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:28.597300+0000) 2019-10-13T05:30:28.605 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:28.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78726 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:30.341 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:30.333+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:30.333471+0000) 2019-10-13T05:30:31.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:31.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:31.812750+0000) 2019-10-13T05:30:32.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:32.680+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:32.680574+0000) 2019-10-13T05:30:32.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:32.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:32.679964+0000) 2019-10-13T05:30:33.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:33.603+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.604200+0000) 2019-10-13T05:30:33.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:33.598+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.599633+0000) 2019-10-13T05:30:33.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:33.604+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.604787+0000) 2019-10-13T05:30:33.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:33.604+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.604883+0000) 2019-10-13T05:30:33.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:33.604+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.605223+0000) 2019-10-13T05:30:33.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:33.604+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.605108+0000) 2019-10-13T05:30:33.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:33.600+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.601521+0000) 2019-10-13T05:30:33.610 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:33.605+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:33.605650+0000) 2019-10-13T05:30:33.610 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:33.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:34.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:34.175+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:34.176006+0000) 2019-10-13T05:30:38.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:38.219+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.219501+0000) 2019-10-13T05:30:38.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:38.558+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.558883+0000) 2019-10-13T05:30:38.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:38.558+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.558950+0000) 2019-10-13T05:30:38.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:38.603+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.604091+0000) 2019-10-13T05:30:38.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:38.608+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.609116+0000) 2019-10-13T05:30:38.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:38.608+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.609270+0000) 2019-10-13T05:30:38.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:38.609+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.609750+0000) 2019-10-13T05:30:38.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:38.609+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.609884+0000) 2019-10-13T05:30:38.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:38.609+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.610293+0000) 2019-10-13T05:30:38.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:38.610+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.610629+0000) 2019-10-13T05:30:38.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:38.606+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:38.607586+0000) 2019-10-13T05:30:38.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:38.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:40.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:40.334+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:40.335810+0000) 2019-10-13T05:30:41.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:41.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:41.812930+0000) 2019-10-13T05:30:41.971 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:30:41.971 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:41.975 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:30:41.975 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:42.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:42.680+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:42.680784+0000) 2019-10-13T05:30:42.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:42.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:42.680142+0000) 2019-10-13T05:30:43.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:43.612+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.614266+0000) 2019-10-13T05:30:43.618 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:43.613+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.614754+0000) 2019-10-13T05:30:43.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:43.608+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.610329+0000) 2019-10-13T05:30:43.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:43.613+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.615195+0000) 2019-10-13T05:30:43.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:43.613+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.615299+0000) 2019-10-13T05:30:43.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:43.614+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.615462+0000) 2019-10-13T05:30:43.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:43.614+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.615697+0000) 2019-10-13T05:30:43.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:43.609+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:43.611660+0000) 2019-10-13T05:30:43.621 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:43.615+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:44.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:44.175+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:44.176171+0000) 2019-10-13T05:30:48.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:48.218+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.219667+0000) 2019-10-13T05:30:48.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:48.558+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.559071+0000) 2019-10-13T05:30:48.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:48.558+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.559141+0000) 2019-10-13T05:30:48.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:48.613+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.615326+0000) 2019-10-13T05:30:48.626 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:48.614+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.616376+0000) 2019-10-13T05:30:48.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:48.617+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.618803+0000) 2019-10-13T05:30:48.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:48.617+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.619302+0000) 2019-10-13T05:30:48.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:48.618+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.619484+0000) 2019-10-13T05:30:48.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:48.618+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.619940+0000) 2019-10-13T05:30:48.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:48.618+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.620189+0000) 2019-10-13T05:30:48.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:48.618+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:48.620384+0000) 2019-10-13T05:30:48.628 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:48.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:50.346 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:50.337+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:50.338151+0000) 2019-10-13T05:30:51.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:51.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:51.813164+0000) 2019-10-13T05:30:52.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:52.679+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:52.680938+0000) 2019-10-13T05:30:52.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:52.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:52.680337+0000) 2019-10-13T05:30:53.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:53.618+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.619575+0000) 2019-10-13T05:30:53.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:53.623+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.624600+0000) 2019-10-13T05:30:53.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:53.623+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.624973+0000) 2019-10-13T05:30:53.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:53.623+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.625148+0000) 2019-10-13T05:30:53.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:53.624+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.625435+0000) 2019-10-13T05:30:53.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:53.624+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.625697+0000) 2019-10-13T05:30:53.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:53.624+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.625781+0000) 2019-10-13T05:30:53.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:53.621+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:53.622639+0000) 2019-10-13T05:30:53.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:53.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78876 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:54.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:54.175+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:54.176354+0000) 2019-10-13T05:30:58.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:58.219+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.219858+0000) 2019-10-13T05:30:58.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:58.558+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.559286+0000) 2019-10-13T05:30:58.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:58.558+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.559229+0000) 2019-10-13T05:30:58.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:30:58.628+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.629517+0000) 2019-10-13T05:30:58.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:30:58.628+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.629686+0000) 2019-10-13T05:30:58.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:30:58.628+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.630007+0000) 2019-10-13T05:30:58.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:30:58.628+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.630090+0000) 2019-10-13T05:30:58.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:30:58.625+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.626360+0000) 2019-10-13T05:30:58.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:30:58.629+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.630426+0000) 2019-10-13T05:30:58.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:30:58.629+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.630957+0000) 2019-10-13T05:30:58.636 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:30:58.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:30:58.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:30:58.627+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:30:58.628551+0000) 2019-10-13T05:31:00.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:00.340+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:00.341169+0000) 2019-10-13T05:31:01.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:01.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:01.813381+0000) 2019-10-13T05:31:02.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:02.680+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:02.681096+0000) 2019-10-13T05:31:02.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:02.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:02.680535+0000) 2019-10-13T05:31:03.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:03.633+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.635067+0000) 2019-10-13T05:31:03.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:03.634+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.635492+0000) 2019-10-13T05:31:03.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:03.634+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.635534+0000) 2019-10-13T05:31:03.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:03.630+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.631168+0000) 2019-10-13T05:31:03.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:03.634+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.635759+0000) 2019-10-13T05:31:03.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:03.634+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.636123+0000) 2019-10-13T05:31:03.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:03.634+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.636261+0000) 2019-10-13T05:31:03.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:03.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:03.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:03.633+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:03.634197+0000) 2019-10-13T05:31:04.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:04.175+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:04.176522+0000) 2019-10-13T05:31:08.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:08.219+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.220052+0000) 2019-10-13T05:31:08.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:08.558+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.559421+0000) 2019-10-13T05:31:08.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:08.558+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.559443+0000) 2019-10-13T05:31:08.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:08.638+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.639532+0000) 2019-10-13T05:31:08.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:08.638+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.639867+0000) 2019-10-13T05:31:08.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:08.638+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.640013+0000) 2019-10-13T05:31:08.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:08.634+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.634894+0000) 2019-10-13T05:31:08.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:08.638+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.640355+0000) 2019-10-13T05:31:08.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:08.639+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.640491+0000) 2019-10-13T05:31:08.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:08.639+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.640621+0000) 2019-10-13T05:31:08.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:08.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:08.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:08.637+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:08.638093+0000) 2019-10-13T05:31:10.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:10.343+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:10.343548+0000) 2019-10-13T05:31:11.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:11.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:11.813585+0000) 2019-10-13T05:31:12.101 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:31:12.101 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:12.105 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:31:12.105 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:12.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:12.680+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:12.681282+0000) 2019-10-13T05:31:12.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:12.679+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:12.680690+0000) 2019-10-13T05:31:13.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:13.644+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.644729+0000) 2019-10-13T05:31:13.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:13.644+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.644980+0000) 2019-10-13T05:31:13.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:13.644+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.645080+0000) 2019-10-13T05:31:13.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:13.639+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.640451+0000) 2019-10-13T05:31:13.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:13.644+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.645385+0000) 2019-10-13T05:31:13.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:13.645+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.645677+0000) 2019-10-13T05:31:13.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:13.645+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.645863+0000) 2019-10-13T05:31:13.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:13.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 78997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:13.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:13.643+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:13.643760+0000) 2019-10-13T05:31:14.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:14.176+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:14.176665+0000) 2019-10-13T05:31:18.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:18.219+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.220226+0000) 2019-10-13T05:31:18.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:18.559+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.559626+0000) 2019-10-13T05:31:18.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:18.559+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.559619+0000) 2019-10-13T05:31:18.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:18.649+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.649758+0000) 2019-10-13T05:31:18.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:18.649+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.650061+0000) 2019-10-13T05:31:18.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:18.644+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.645043+0000) 2019-10-13T05:31:18.654 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:18.649+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.650325+0000) 2019-10-13T05:31:18.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:18.650+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.650884+0000) 2019-10-13T05:31:18.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:18.650+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.650985+0000) 2019-10-13T05:31:18.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:18.650+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.651138+0000) 2019-10-13T05:31:18.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:18.646+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:18.646882+0000) 2019-10-13T05:31:18.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:18.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:20.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:20.345+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:20.346010+0000) 2019-10-13T05:31:21.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:21.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:21.813772+0000) 2019-10-13T05:31:22.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:22.681+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:22.681461+0000) 2019-10-13T05:31:22.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:22.680+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:22.680903+0000) 2019-10-13T05:31:23.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:23.648+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.649326+0000) 2019-10-13T05:31:23.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:23.654+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.655251+0000) 2019-10-13T05:31:23.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:23.655+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.655430+0000) 2019-10-13T05:31:23.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:23.655+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.655566+0000) 2019-10-13T05:31:23.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:23.655+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.655913+0000) 2019-10-13T05:31:23.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:23.655+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.656193+0000) 2019-10-13T05:31:23.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:23.656+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.656425+0000) 2019-10-13T05:31:23.660 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:23.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79056 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:23.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:23.651+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:23.652728+0000) 2019-10-13T05:31:24.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:24.176+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:24.176805+0000) 2019-10-13T05:31:28.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:28.219+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.220445+0000) 2019-10-13T05:31:28.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:28.558+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.559784+0000) 2019-10-13T05:31:28.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:28.558+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.559784+0000) 2019-10-13T05:31:28.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:28.652+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.654106+0000) 2019-10-13T05:31:28.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:28.656+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.658201+0000) 2019-10-13T05:31:28.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:28.657+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.658816+0000) 2019-10-13T05:31:28.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:28.657+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.659111+0000) 2019-10-13T05:31:28.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:28.657+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.659358+0000) 2019-10-13T05:31:28.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:28.658+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.659457+0000) 2019-10-13T05:31:28.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:28.658+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.659961+0000) 2019-10-13T05:31:28.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:28.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:28.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:28.654+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:28.656304+0000) 2019-10-13T05:31:30.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:30.347+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:30.348162+0000) 2019-10-13T05:31:31.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:31.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:31.813965+0000) 2019-10-13T05:31:32.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:32.680+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:32.681637+0000) 2019-10-13T05:31:32.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:32.680+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:32.681077+0000) 2019-10-13T05:31:33.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:33.661+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.662788+0000) 2019-10-13T05:31:33.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:33.661+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.662988+0000) 2019-10-13T05:31:33.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:33.661+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.663176+0000) 2019-10-13T05:31:33.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:33.657+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.658953+0000) 2019-10-13T05:31:33.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:33.662+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.664024+0000) 2019-10-13T05:31:33.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:33.662+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.664247+0000) 2019-10-13T05:31:33.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:33.663+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.664493+0000) 2019-10-13T05:31:33.668 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:33.659+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:33.660938+0000) 2019-10-13T05:31:33.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:33.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79117 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:34.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:34.176+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:34.176922+0000) 2019-10-13T05:31:38.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:38.219+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.220685+0000) 2019-10-13T05:31:38.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:38.559+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.559957+0000) 2019-10-13T05:31:38.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:38.559+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.559956+0000) 2019-10-13T05:31:38.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:38.666+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.667901+0000) 2019-10-13T05:31:38.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:38.666+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.668178+0000) 2019-10-13T05:31:38.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:38.667+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.668506+0000) 2019-10-13T05:31:38.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:38.662+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.663967+0000) 2019-10-13T05:31:38.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:38.667+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.669086+0000) 2019-10-13T05:31:38.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:38.667+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.669195+0000) 2019-10-13T05:31:38.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:38.667+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.669347+0000) 2019-10-13T05:31:38.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:38.669+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:38.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:38.664+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:38.666142+0000) 2019-10-13T05:31:40.358 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:40.349+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:40.350626+0000) 2019-10-13T05:31:41.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:41.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:41.814160+0000) 2019-10-13T05:31:42.264 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:31:42.264 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:42.271 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:31:42.272 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:42.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:42.681+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:42.681843+0000) 2019-10-13T05:31:42.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:42.680+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:42.681296+0000) 2019-10-13T05:31:43.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:43.671+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.672803+0000) 2019-10-13T05:31:43.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:43.671+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.673322+0000) 2019-10-13T05:31:43.677 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:43.672+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.673395+0000) 2019-10-13T05:31:43.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:43.672+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.673582+0000) 2019-10-13T05:31:43.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:43.672+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.673798+0000) 2019-10-13T05:31:43.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:43.672+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.673952+0000) 2019-10-13T05:31:43.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:43.669+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.669752+0000) 2019-10-13T05:31:43.679 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:43.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79176 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:43.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:43.670+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:43.671696+0000) 2019-10-13T05:31:44.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:44.176+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:44.177092+0000) 2019-10-13T05:31:48.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:48.220+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.220860+0000) 2019-10-13T05:31:48.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:48.559+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.560145+0000) 2019-10-13T05:31:48.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:48.559+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.560147+0000) 2019-10-13T05:31:48.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:48.676+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.677664+0000) 2019-10-13T05:31:48.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:48.676+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.678016+0000) 2019-10-13T05:31:48.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:48.672+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.673397+0000) 2019-10-13T05:31:48.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:48.677+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.678533+0000) 2019-10-13T05:31:48.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:48.677+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.678698+0000) 2019-10-13T05:31:48.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:48.677+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.679264+0000) 2019-10-13T05:31:48.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:48.678+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.679402+0000) 2019-10-13T05:31:48.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:48.675+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:48.675846+0000) 2019-10-13T05:31:48.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:48.680+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:50.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:50.352+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:50.352924+0000) 2019-10-13T05:31:51.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:51.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:51.814351+0000) 2019-10-13T05:31:52.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:52.681+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:52.682046+0000) 2019-10-13T05:31:52.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:52.680+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:52.681497+0000) 2019-10-13T05:31:53.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:53.681+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.683032+0000) 2019-10-13T05:31:53.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:53.681+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.683326+0000) 2019-10-13T05:31:53.687 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:53.682+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.683513+0000) 2019-10-13T05:31:53.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:53.682+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.683660+0000) 2019-10-13T05:31:53.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:53.682+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.683806+0000) 2019-10-13T05:31:53.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:53.682+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.684179+0000) 2019-10-13T05:31:53.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:53.678+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.679345+0000) 2019-10-13T05:31:53.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:53.685+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:53.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:53.680+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:53.681287+0000) 2019-10-13T05:31:54.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:54.176+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:54.177240+0000) 2019-10-13T05:31:58.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:58.220+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.221024+0000) 2019-10-13T05:31:58.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:58.559+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.560344+0000) 2019-10-13T05:31:58.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:58.559+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.560344+0000) 2019-10-13T05:31:58.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:31:58.684+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.685055+0000) 2019-10-13T05:31:58.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:31:58.688+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.688654+0000) 2019-10-13T05:31:58.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:31:58.688+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.688975+0000) 2019-10-13T05:31:58.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:31:58.688+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.689163+0000) 2019-10-13T05:31:58.700 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:31:58.689+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.689492+0000) 2019-10-13T05:31:58.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:31:58.689+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.690064+0000) 2019-10-13T05:31:58.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:31:58.689+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.690140+0000) 2019-10-13T05:31:58.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:31:58.691+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:31:58.701 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:31:58.686+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:31:58.687094+0000) 2019-10-13T05:32:00.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:00.355+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:00.355507+0000) 2019-10-13T05:32:01.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:01.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:01.814549+0000) 2019-10-13T05:32:02.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:02.681+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:02.682233+0000) 2019-10-13T05:32:02.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:02.680+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:02.681705+0000) 2019-10-13T05:32:03.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:03.689+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.690495+0000) 2019-10-13T05:32:03.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:03.691+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.692442+0000) 2019-10-13T05:32:03.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:03.693+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.693854+0000) 2019-10-13T05:32:03.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:03.693+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.694167+0000) 2019-10-13T05:32:03.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:03.694+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.694486+0000) 2019-10-13T05:32:03.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:03.694+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.694669+0000) 2019-10-13T05:32:03.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:03.694+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.695003+0000) 2019-10-13T05:32:03.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:03.695+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:03.695470+0000) 2019-10-13T05:32:03.703 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:03.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:04.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:04.177+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:04.177432+0000) 2019-10-13T05:32:08.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:08.221+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.221216+0000) 2019-10-13T05:32:08.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:08.560+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.560505+0000) 2019-10-13T05:32:08.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:08.560+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.560543+0000) 2019-10-13T05:32:08.701 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:08.697+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.698008+0000) 2019-10-13T05:32:08.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:08.698+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.699221+0000) 2019-10-13T05:32:08.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:08.699+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.699444+0000) 2019-10-13T05:32:08.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:08.699+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.699611+0000) 2019-10-13T05:32:08.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:08.699+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.699750+0000) 2019-10-13T05:32:08.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:08.699+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.699958+0000) 2019-10-13T05:32:08.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:08.694+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.695460+0000) 2019-10-13T05:32:08.704 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:08.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:08.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:08.696+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:08.697385+0000) 2019-10-13T05:32:10.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:10.357+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:10.357932+0000) 2019-10-13T05:32:11.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:11.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:11.814709+0000) 2019-10-13T05:32:12.410 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:32:12.410 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:12.416 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:32:12.416 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:12.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:12.682+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:12.682433+0000) 2019-10-13T05:32:12.690 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:12.681+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:12.681902+0000) 2019-10-13T05:32:13.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:13.702+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.703405+0000) 2019-10-13T05:32:13.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:13.702+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.703506+0000) 2019-10-13T05:32:13.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:13.702+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.703808+0000) 2019-10-13T05:32:13.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:13.699+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.699865+0000) 2019-10-13T05:32:13.708 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:13.702+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.704028+0000) 2019-10-13T05:32:13.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:13.702+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.704264+0000) 2019-10-13T05:32:13.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:13.703+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.704433+0000) 2019-10-13T05:32:13.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:13.700+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:13.701320+0000) 2019-10-13T05:32:13.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:13.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:14.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:14.176+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:14.177607+0000) 2019-10-13T05:32:18.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:18.220+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.221433+0000) 2019-10-13T05:32:18.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:18.559+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.560705+0000) 2019-10-13T05:32:18.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:18.559+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.560698+0000) 2019-10-13T05:32:18.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:18.707+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.708447+0000) 2019-10-13T05:32:18.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:18.707+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.708686+0000) 2019-10-13T05:32:18.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:18.707+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.708871+0000) 2019-10-13T05:32:18.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:18.707+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.709044+0000) 2019-10-13T05:32:18.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:18.703+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.704950+0000) 2019-10-13T05:32:18.713 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:18.707+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.709148+0000) 2019-10-13T05:32:18.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:18.708+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.709444+0000) 2019-10-13T05:32:18.715 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:18.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:18.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:18.705+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:18.706954+0000) 2019-10-13T05:32:20.368 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:20.358+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:20.360369+0000) 2019-10-13T05:32:21.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:21.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:21.814861+0000) 2019-10-13T05:32:22.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:22.681+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:22.682591+0000) 2019-10-13T05:32:22.690 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:22.680+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:22.682047+0000) 2019-10-13T05:32:23.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:23.707+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.708970+0000) 2019-10-13T05:32:23.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:23.712+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.713646+0000) 2019-10-13T05:32:23.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:23.712+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.713936+0000) 2019-10-13T05:32:23.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:23.712+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.714219+0000) 2019-10-13T05:32:23.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:23.713+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.714759+0000) 2019-10-13T05:32:23.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:23.713+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.715032+0000) 2019-10-13T05:32:23.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:23.713+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.715302+0000) 2019-10-13T05:32:23.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:23.709+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:23.711624+0000) 2019-10-13T05:32:23.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:23.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:24.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:24.176+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:24.177779+0000) 2019-10-13T05:32:28.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:28.220+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.221621+0000) 2019-10-13T05:32:28.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:28.559+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.560916+0000) 2019-10-13T05:32:28.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:28.559+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.560919+0000) 2019-10-13T05:32:28.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:28.717+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.719238+0000) 2019-10-13T05:32:28.723 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:28.718+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.719418+0000) 2019-10-13T05:32:28.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:28.718+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.719608+0000) 2019-10-13T05:32:28.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:28.718+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.719911+0000) 2019-10-13T05:32:28.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:28.718+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.720124+0000) 2019-10-13T05:32:28.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:28.718+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.720182+0000) 2019-10-13T05:32:28.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:28.713+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.714916+0000) 2019-10-13T05:32:28.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:28.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:28.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:28.716+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:28.717840+0000) 2019-10-13T05:32:30.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:30.362+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:30.362912+0000) 2019-10-13T05:32:31.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:31.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:31.815073+0000) 2019-10-13T05:32:32.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:32.682+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:32.682738+0000) 2019-10-13T05:32:32.690 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:32.681+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:32.682229+0000) 2019-10-13T05:32:33.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:33.722+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.723586+0000) 2019-10-13T05:32:33.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:33.722+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.723763+0000) 2019-10-13T05:32:33.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:33.722+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.724005+0000) 2019-10-13T05:32:33.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:33.722+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.724364+0000) 2019-10-13T05:32:33.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:33.723+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.724467+0000) 2019-10-13T05:32:33.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:33.718+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.719386+0000) 2019-10-13T05:32:33.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:33.721+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.722121+0000) 2019-10-13T05:32:33.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:33.723+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:33.724800+0000) 2019-10-13T05:32:33.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:33.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:34.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:34.177+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:34.177934+0000) 2019-10-13T05:32:38.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:38.221+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.221782+0000) 2019-10-13T05:32:38.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:38.560+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.561155+0000) 2019-10-13T05:32:38.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:38.560+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.561153+0000) 2019-10-13T05:32:38.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:38.723+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.724221+0000) 2019-10-13T05:32:38.733 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:38.727+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.729235+0000) 2019-10-13T05:32:38.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:38.727+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.729355+0000) 2019-10-13T05:32:38.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:38.728+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.729626+0000) 2019-10-13T05:32:38.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:38.728+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.729714+0000) 2019-10-13T05:32:38.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:38.728+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.729981+0000) 2019-10-13T05:32:38.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:38.728+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.730241+0000) 2019-10-13T05:32:38.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:38.725+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:38.726437+0000) 2019-10-13T05:32:38.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:38.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:40.373 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:40.364+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:40.365291+0000) 2019-10-13T05:32:41.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:41.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:41.815269+0000) 2019-10-13T05:32:42.613 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:32:42.614 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:42.618 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:32:42.618 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:42.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:42.682+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:42.682934+0000) 2019-10-13T05:32:42.690 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:42.681+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:42.682410+0000) 2019-10-13T05:32:43.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:43.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.733775+0000) 2019-10-13T05:32:43.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:43.733+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.734066+0000) 2019-10-13T05:32:43.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:43.733+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.734240+0000) 2019-10-13T05:32:43.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:43.734+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.734476+0000) 2019-10-13T05:32:43.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:43.729+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.729761+0000) 2019-10-13T05:32:43.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:43.734+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.735311+0000) 2019-10-13T05:32:43.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:43.734+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.735375+0000) 2019-10-13T05:32:43.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:43.731+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:43.732170+0000) 2019-10-13T05:32:43.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:43.737+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:44.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:44.177+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:44.178079+0000) 2019-10-13T05:32:48.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:48.221+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.221943+0000) 2019-10-13T05:32:48.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:48.561+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.561340+0000) 2019-10-13T05:32:48.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:48.561+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.561339+0000) 2019-10-13T05:32:48.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:48.733+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.734181+0000) 2019-10-13T05:32:48.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:48.738+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.739182+0000) 2019-10-13T05:32:48.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:48.739+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.739499+0000) 2019-10-13T05:32:48.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:48.739+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.739730+0000) 2019-10-13T05:32:48.743 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:48.739+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.739903+0000) 2019-10-13T05:32:48.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:48.739+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.740133+0000) 2019-10-13T05:32:48.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:48.739+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.740289+0000) 2019-10-13T05:32:48.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:48.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:48.745 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:48.736+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:48.737398+0000) 2019-10-13T05:32:50.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:50.367+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:50.367549+0000) 2019-10-13T05:32:51.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:51.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:51.815455+0000) 2019-10-13T05:32:52.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:52.682+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:52.683152+0000) 2019-10-13T05:32:52.690 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:52.682+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:52.682576+0000) 2019-10-13T05:32:53.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:53.743+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.744098+0000) 2019-10-13T05:32:53.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:53.743+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.744232+0000) 2019-10-13T05:32:53.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:53.738+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.739557+0000) 2019-10-13T05:32:53.749 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:53.744+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.744613+0000) 2019-10-13T05:32:53.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:53.744+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.744766+0000) 2019-10-13T05:32:53.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:53.744+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.745038+0000) 2019-10-13T05:32:53.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:53.744+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.745220+0000) 2019-10-13T05:32:53.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:53.741+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:53.742271+0000) 2019-10-13T05:32:53.750 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:53.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:32:54.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:54.178+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:54.178252+0000) 2019-10-13T05:32:58.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:58.222+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.222120+0000) 2019-10-13T05:32:58.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:58.561+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.561552+0000) 2019-10-13T05:32:58.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:58.561+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.561554+0000) 2019-10-13T05:32:58.753 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:32:58.747+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.749238+0000) 2019-10-13T05:32:58.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:32:58.748+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.749412+0000) 2019-10-13T05:32:58.753 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:32:58.748+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.749642+0000) 2019-10-13T05:32:58.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:32:58.748+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.749721+0000) 2019-10-13T05:32:58.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:32:58.748+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.750041+0000) 2019-10-13T05:32:58.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:32:58.744+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.745044+0000) 2019-10-13T05:32:58.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:32:58.748+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.750264+0000) 2019-10-13T05:32:58.755 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:32:58.746+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:32:58.747208+0000) 2019-10-13T05:32:58.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:32:58.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:00.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:00.369+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:00.369785+0000) 2019-10-13T05:33:01.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:01.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:01.815632+0000) 2019-10-13T05:33:02.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:02.681+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:02.683382+0000) 2019-10-13T05:33:02.690 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:02.682+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:02.682785+0000) 2019-10-13T05:33:03.758 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:03.752+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.754300+0000) 2019-10-13T05:33:03.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:03.753+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.754461+0000) 2019-10-13T05:33:03.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:03.753+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.754677+0000) 2019-10-13T05:33:03.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:03.753+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.754824+0000) 2019-10-13T05:33:03.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:03.749+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.749739+0000) 2019-10-13T05:33:03.759 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:03.753+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.755066+0000) 2019-10-13T05:33:03.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:03.753+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.755190+0000) 2019-10-13T05:33:03.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:03.751+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:03.752601+0000) 2019-10-13T05:33:03.761 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:03.756+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:04.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:04.177+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:04.178449+0000) 2019-10-13T05:33:08.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:08.220+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.222277+0000) 2019-10-13T05:33:08.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:08.560+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.561693+0000) 2019-10-13T05:33:08.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:08.560+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.561694+0000) 2019-10-13T05:33:08.762 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:08.757+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.758804+0000) 2019-10-13T05:33:08.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:08.757+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.759092+0000) 2019-10-13T05:33:08.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:08.752+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.754386+0000) 2019-10-13T05:33:08.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:08.757+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.759284+0000) 2019-10-13T05:33:08.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:08.758+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.759418+0000) 2019-10-13T05:33:08.764 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:08.758+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.759512+0000) 2019-10-13T05:33:08.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:08.758+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.759791+0000) 2019-10-13T05:33:08.764 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:08.754+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:08.756409+0000) 2019-10-13T05:33:08.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:08.759+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:10.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:10.371+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:10.371980+0000) 2019-10-13T05:33:11.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:11.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:11.815803+0000) 2019-10-13T05:33:12.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:12.682+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:12.683560+0000) 2019-10-13T05:33:12.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:12.682+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:12.682993+0000) 2019-10-13T05:33:12.785 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:33:12.785 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:12.789 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:33:12.790 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:13.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:13.761+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.763300+0000) 2019-10-13T05:33:13.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:13.761+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.763380+0000) 2019-10-13T05:33:13.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:13.762+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.763490+0000) 2019-10-13T05:33:13.767 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:13.762+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.763746+0000) 2019-10-13T05:33:13.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:13.762+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.763890+0000) 2019-10-13T05:33:13.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:13.757+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.759208+0000) 2019-10-13T05:33:13.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:13.762+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.764151+0000) 2019-10-13T05:33:13.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:13.759+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:13.761199+0000) 2019-10-13T05:33:13.769 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:13.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:14.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:14.177+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:14.178617+0000) 2019-10-13T05:33:18.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:18.221+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.222429+0000) 2019-10-13T05:33:18.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:18.560+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.561847+0000) 2019-10-13T05:33:18.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:18.560+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.561854+0000) 2019-10-13T05:33:18.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:18.766+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.767656+0000) 2019-10-13T05:33:18.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:18.766+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.767775+0000) 2019-10-13T05:33:18.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:18.766+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.767923+0000) 2019-10-13T05:33:18.772 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:18.766+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.768117+0000) 2019-10-13T05:33:18.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:18.767+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.768409+0000) 2019-10-13T05:33:18.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:18.761+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.763257+0000) 2019-10-13T05:33:18.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:18.767+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.768613+0000) 2019-10-13T05:33:18.773 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:18.763+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:18.765600+0000) 2019-10-13T05:33:18.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:18.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:20.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:20.373+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:20.374319+0000) 2019-10-13T05:33:21.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:21.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:21.815980+0000) 2019-10-13T05:33:22.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:22.683+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:22.683743+0000) 2019-10-13T05:33:22.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:22.682+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:22.683147+0000) 2019-10-13T05:33:23.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:23.770+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.771707+0000) 2019-10-13T05:33:23.775 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:23.770+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.772163+0000) 2019-10-13T05:33:23.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:23.767+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.767774+0000) 2019-10-13T05:33:23.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:23.770+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.772352+0000) 2019-10-13T05:33:23.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:23.771+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.772511+0000) 2019-10-13T05:33:23.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:23.771+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.772770+0000) 2019-10-13T05:33:23.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:23.771+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.772870+0000) 2019-10-13T05:33:23.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:23.768+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:23.769245+0000) 2019-10-13T05:33:23.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:23.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:24.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:24.178+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:24.178820+0000) 2019-10-13T05:33:28.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:28.222+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.222590+0000) 2019-10-13T05:33:28.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:28.561+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.562027+0000) 2019-10-13T05:33:28.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:28.561+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.562025+0000) 2019-10-13T05:33:28.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:28.776+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.776491+0000) 2019-10-13T05:33:28.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:28.771+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.772039+0000) 2019-10-13T05:33:28.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:28.776+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.777376+0000) 2019-10-13T05:33:28.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:28.777+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.777537+0000) 2019-10-13T05:33:28.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:28.777+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.777781+0000) 2019-10-13T05:33:28.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:28.777+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.777877+0000) 2019-10-13T05:33:28.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:28.777+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.778239+0000) 2019-10-13T05:33:28.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:28.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:28.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:28.774+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:28.775105+0000) 2019-10-13T05:33:30.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:30.375+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:30.376729+0000) 2019-10-13T05:33:31.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:31.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:31.816177+0000) 2019-10-13T05:33:32.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:32.683+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:32.683927+0000) 2019-10-13T05:33:32.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:32.682+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:32.683267+0000) 2019-10-13T05:33:33.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:33.776+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.777009+0000) 2019-10-13T05:33:33.785 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:33.781+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.782187+0000) 2019-10-13T05:33:33.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:33.782+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.782725+0000) 2019-10-13T05:33:33.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:33.782+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.782640+0000) 2019-10-13T05:33:33.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:33.782+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.782955+0000) 2019-10-13T05:33:33.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:33.782+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.783001+0000) 2019-10-13T05:33:33.787 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:33.782+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.783188+0000) 2019-10-13T05:33:33.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:33.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:33.788 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:33.779+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:33.780564+0000) 2019-10-13T05:33:34.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:34.178+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:34.178993+0000) 2019-10-13T05:33:38.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:38.222+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.222786+0000) 2019-10-13T05:33:38.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:38.562+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.562244+0000) 2019-10-13T05:33:38.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:38.562+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.562243+0000) 2019-10-13T05:33:38.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:38.785+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.786290+0000) 2019-10-13T05:33:38.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:38.786+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.786453+0000) 2019-10-13T05:33:38.790 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:38.786+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.786764+0000) 2019-10-13T05:33:38.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:38.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.787018+0000) 2019-10-13T05:33:38.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:38.786+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.787310+0000) 2019-10-13T05:33:38.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:38.782+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.783022+0000) 2019-10-13T05:33:38.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:38.787+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.788184+0000) 2019-10-13T05:33:38.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:38.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79889 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:38.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:38.785+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:38.785820+0000) 2019-10-13T05:33:40.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:40.378+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:40.378963+0000) 2019-10-13T05:33:41.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:41.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:41.816412+0000) 2019-10-13T05:33:42.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:42.684+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:42.684089+0000) 2019-10-13T05:33:42.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:42.683+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:42.683438+0000) 2019-10-13T05:33:42.964 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:33:42.964 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:42.968 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:33:42.968 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:43.795 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:43.791+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.791655+0000) 2019-10-13T05:33:43.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:43.791+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.792306+0000) 2019-10-13T05:33:43.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:43.792+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.792498+0000) 2019-10-13T05:33:43.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:43.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.787882+0000) 2019-10-13T05:33:43.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:43.792+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.792793+0000) 2019-10-13T05:33:43.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:43.792+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.793065+0000) 2019-10-13T05:33:43.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:43.792+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.793155+0000) 2019-10-13T05:33:43.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:43.788+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:43.789533+0000) 2019-10-13T05:33:43.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:43.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:44.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:44.178+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:44.179167+0000) 2019-10-13T05:33:48.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:48.222+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.222985+0000) 2019-10-13T05:33:48.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:48.561+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.562450+0000) 2019-10-13T05:33:48.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:48.561+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.562450+0000) 2019-10-13T05:33:48.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:48.795+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.796523+0000) 2019-10-13T05:33:48.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:48.795+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.796742+0000) 2019-10-13T05:33:48.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:48.795+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.796797+0000) 2019-10-13T05:33:48.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:48.795+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.797047+0000) 2019-10-13T05:33:48.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:48.791+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.792612+0000) 2019-10-13T05:33:48.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:48.796+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.797586+0000) 2019-10-13T05:33:48.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:48.796+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.797738+0000) 2019-10-13T05:33:48.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:48.793+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:48.794235+0000) 2019-10-13T05:33:48.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:48.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:50.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:50.380+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:50.381417+0000) 2019-10-13T05:33:51.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:51.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:51.816626+0000) 2019-10-13T05:33:52.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:52.683+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:52.684276+0000) 2019-10-13T05:33:52.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:52.683+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:52.683622+0000) 2019-10-13T05:33:53.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:53.800+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.801851+0000) 2019-10-13T05:33:53.805 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:53.800+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.802029+0000) 2019-10-13T05:33:53.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:53.797+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.797851+0000) 2019-10-13T05:33:53.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:53.801+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.802412+0000) 2019-10-13T05:33:53.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:53.801+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.802703+0000) 2019-10-13T05:33:53.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:53.801+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.802872+0000) 2019-10-13T05:33:53.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:53.801+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.802936+0000) 2019-10-13T05:33:53.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:53.799+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:53.800375+0000) 2019-10-13T05:33:53.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:53.803+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 79982 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:33:54.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:54.178+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:54.179333+0000) 2019-10-13T05:33:58.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:58.222+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.223172+0000) 2019-10-13T05:33:58.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:58.561+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.562664+0000) 2019-10-13T05:33:58.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:58.561+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.562643+0000) 2019-10-13T05:33:58.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:33:58.799+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.801597+0000) 2019-10-13T05:33:58.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:33:58.805+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.806751+0000) 2019-10-13T05:33:58.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:33:58.805+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.807251+0000) 2019-10-13T05:33:58.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:33:58.806+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.807561+0000) 2019-10-13T05:33:58.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:33:58.806+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.808304+0000) 2019-10-13T05:33:58.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:33:58.807+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.808429+0000) 2019-10-13T05:33:58.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:33:58.807+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.808730+0000) 2019-10-13T05:33:58.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:33:58.803+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:33:58.804892+0000) 2019-10-13T05:33:58.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:33:58.808+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80012 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:00.392 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:00.382+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:00.383965+0000) 2019-10-13T05:34:01.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:01.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:01.816781+0000) 2019-10-13T05:34:02.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:02.683+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:02.684412+0000) 2019-10-13T05:34:02.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:02.682+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:02.683801+0000) 2019-10-13T05:34:03.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:03.805+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.806984+0000) 2019-10-13T05:34:03.816 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:03.810+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.812023+0000) 2019-10-13T05:34:03.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:03.810+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.812078+0000) 2019-10-13T05:34:03.816 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:03.810+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.812292+0000) 2019-10-13T05:34:03.816 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:03.811+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.812500+0000) 2019-10-13T05:34:03.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:03.811+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.812659+0000) 2019-10-13T05:34:03.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:03.811+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.813030+0000) 2019-10-13T05:34:03.817 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:03.807+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:03.808994+0000) 2019-10-13T05:34:03.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:03.814+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80043 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:04.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:04.178+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:04.179506+0000) 2019-10-13T05:34:08.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:08.222+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.223363+0000) 2019-10-13T05:34:08.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:08.562+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.562824+0000) 2019-10-13T05:34:08.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:08.562+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.562824+0000) 2019-10-13T05:34:08.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:08.816+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.818114+0000) 2019-10-13T05:34:08.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:08.816+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.818232+0000) 2019-10-13T05:34:08.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:08.812+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.813546+0000) 2019-10-13T05:34:08.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:08.817+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.818667+0000) 2019-10-13T05:34:08.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:08.817+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.818729+0000) 2019-10-13T05:34:08.824 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:08.817+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.819002+0000) 2019-10-13T05:34:08.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:08.817+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.819091+0000) 2019-10-13T05:34:08.824 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:08.818+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:08.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:08.815+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:08.817231+0000) 2019-10-13T05:34:10.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:10.385+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:10.386376+0000) 2019-10-13T05:34:11.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:11.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:11.816927+0000) 2019-10-13T05:34:12.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:12.684+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:12.684606+0000) 2019-10-13T05:34:12.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:12.683+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:12.683971+0000) 2019-10-13T05:34:13.127 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:34:13.127 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:13.132 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:34:13.132 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:13.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:13.816+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.817070+0000) 2019-10-13T05:34:13.826 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:13.821+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.822056+0000) 2019-10-13T05:34:13.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:13.822+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.822433+0000) 2019-10-13T05:34:13.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:13.822+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.822577+0000) 2019-10-13T05:34:13.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:13.822+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.822894+0000) 2019-10-13T05:34:13.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:13.822+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.823054+0000) 2019-10-13T05:34:13.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:13.822+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.823249+0000) 2019-10-13T05:34:13.828 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:13.819+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:13.820274+0000) 2019-10-13T05:34:13.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:13.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:14.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:14.179+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:14.179684+0000) 2019-10-13T05:34:18.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:18.223+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.223509+0000) 2019-10-13T05:34:18.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:18.562+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.563027+0000) 2019-10-13T05:34:18.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:18.562+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.563029+0000) 2019-10-13T05:34:18.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:18.827+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.827611+0000) 2019-10-13T05:34:18.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:18.827+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.827742+0000) 2019-10-13T05:34:18.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:18.827+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.827948+0000) 2019-10-13T05:34:18.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:18.827+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.828081+0000) 2019-10-13T05:34:18.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:18.827+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.828227+0000) 2019-10-13T05:34:18.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:18.828+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.828411+0000) 2019-10-13T05:34:18.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:18.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80136 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:18.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:18.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.822315+0000) 2019-10-13T05:34:18.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:18.823+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:18.824578+0000) 2019-10-13T05:34:20.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:20.388+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:20.388670+0000) 2019-10-13T05:34:21.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:21.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:21.817118+0000) 2019-10-13T05:34:22.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:22.684+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:22.684742+0000) 2019-10-13T05:34:22.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:22.683+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:22.684140+0000) 2019-10-13T05:34:23.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:23.827+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.828392+0000) 2019-10-13T05:34:23.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:23.829+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.830508+0000) 2019-10-13T05:34:23.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:23.831+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.832152+0000) 2019-10-13T05:34:23.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:23.831+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.832310+0000) 2019-10-13T05:34:23.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:23.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.832711+0000) 2019-10-13T05:34:23.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:23.832+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.833160+0000) 2019-10-13T05:34:23.848 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:23.832+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.832970+0000) 2019-10-13T05:34:23.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:23.833+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:23.833439+0000) 2019-10-13T05:34:23.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:23.834+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:24.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:24.179+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:24.179809+0000) 2019-10-13T05:34:28.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:28.223+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.223628+0000) 2019-10-13T05:34:28.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:28.563+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.563206+0000) 2019-10-13T05:34:28.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:28.563+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.563207+0000) 2019-10-13T05:34:28.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:28.836+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.837090+0000) 2019-10-13T05:34:28.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:28.836+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.837273+0000) 2019-10-13T05:34:28.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:28.831+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.832681+0000) 2019-10-13T05:34:28.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:28.837+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.837646+0000) 2019-10-13T05:34:28.842 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:28.837+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.837794+0000) 2019-10-13T05:34:28.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:28.837+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.838279+0000) 2019-10-13T05:34:28.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:28.837+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.838319+0000) 2019-10-13T05:34:28.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:28.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:28.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:28.835+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:28.836560+0000) 2019-10-13T05:34:30.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:30.390+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:30.391049+0000) 2019-10-13T05:34:31.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:31.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:31.817317+0000) 2019-10-13T05:34:32.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:32.683+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:32.684946+0000) 2019-10-13T05:34:32.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:32.683+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:32.684339+0000) 2019-10-13T05:34:33.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:33.840+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.842010+0000) 2019-10-13T05:34:33.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:33.836+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.837429+0000) 2019-10-13T05:34:33.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:33.841+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.842425+0000) 2019-10-13T05:34:33.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:33.841+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.842654+0000) 2019-10-13T05:34:33.847 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:33.841+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.842986+0000) 2019-10-13T05:34:33.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:33.841+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.843154+0000) 2019-10-13T05:34:33.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:33.841+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.843309+0000) 2019-10-13T05:34:33.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:33.838+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:33.839521+0000) 2019-10-13T05:34:33.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:33.843+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:34.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:34.178+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:34.179977+0000) 2019-10-13T05:34:38.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:38.222+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.223798+0000) 2019-10-13T05:34:38.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:38.562+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.563408+0000) 2019-10-13T05:34:38.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:38.562+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.563411+0000) 2019-10-13T05:34:38.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:38.845+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.846758+0000) 2019-10-13T05:34:38.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:38.845+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.847026+0000) 2019-10-13T05:34:38.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:38.842+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.842807+0000) 2019-10-13T05:34:38.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:38.846+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.847562+0000) 2019-10-13T05:34:38.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:38.846+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.847499+0000) 2019-10-13T05:34:38.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:38.846+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.847824+0000) 2019-10-13T05:34:38.852 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:38.846+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.848125+0000) 2019-10-13T05:34:38.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:38.843+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:38.844555+0000) 2019-10-13T05:34:38.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:38.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80260 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:40.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:40.393+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:40.393391+0000) 2019-10-13T05:34:41.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:41.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:41.817508+0000) 2019-10-13T05:34:42.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:42.684+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:42.685114+0000) 2019-10-13T05:34:42.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:42.683+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:42.684536+0000) 2019-10-13T05:34:43.273 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:34:43.273 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:43.277 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:34:43.278 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:43.855 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:43.850+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.851953+0000) 2019-10-13T05:34:43.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:43.850+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.852248+0000) 2019-10-13T05:34:43.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:43.851+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.852423+0000) 2019-10-13T05:34:43.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:43.847+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.847761+0000) 2019-10-13T05:34:43.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:43.851+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.852866+0000) 2019-10-13T05:34:43.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:43.851+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.853215+0000) 2019-10-13T05:34:43.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:43.852+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.853582+0000) 2019-10-13T05:34:43.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:43.849+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:43.850104+0000) 2019-10-13T05:34:43.858 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:43.854+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80292 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:44.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:44.179+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:44.180148+0000) 2019-10-13T05:34:48.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:48.223+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.223988+0000) 2019-10-13T05:34:48.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:48.562+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.563606+0000) 2019-10-13T05:34:48.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:48.562+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.563616+0000) 2019-10-13T05:34:48.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:48.856+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.858152+0000) 2019-10-13T05:34:48.862 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:48.856+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.858273+0000) 2019-10-13T05:34:48.862 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:48.857+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.858482+0000) 2019-10-13T05:34:48.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:48.857+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.858732+0000) 2019-10-13T05:34:48.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:48.852+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.854405+0000) 2019-10-13T05:34:48.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:48.857+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.858983+0000) 2019-10-13T05:34:48.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:48.857+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.859119+0000) 2019-10-13T05:34:48.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:48.859+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80323 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:48.864 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:48.854+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:48.856471+0000) 2019-10-13T05:34:50.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:50.394+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:50.395851+0000) 2019-10-13T05:34:51.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:51.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:51.817721+0000) 2019-10-13T05:34:52.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:52.684+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:52.685283+0000) 2019-10-13T05:34:52.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:52.683+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:52.684711+0000) 2019-10-13T05:34:53.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:53.855+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.857428+0000) 2019-10-13T05:34:53.867 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:53.861+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.862515+0000) 2019-10-13T05:34:53.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:53.861+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.862601+0000) 2019-10-13T05:34:53.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:53.861+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.862916+0000) 2019-10-13T05:34:53.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:53.861+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.862940+0000) 2019-10-13T05:34:53.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:53.861+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.863206+0000) 2019-10-13T05:34:53.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:53.861+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.863367+0000) 2019-10-13T05:34:53.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:53.859+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:53.861207+0000) 2019-10-13T05:34:53.871 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:53.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:34:54.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:54.179+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:54.180319+0000) 2019-10-13T05:34:58.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:58.223+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.224140+0000) 2019-10-13T05:34:58.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:58.563+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.563800+0000) 2019-10-13T05:34:58.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:58.563+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.563820+0000) 2019-10-13T05:34:58.871 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:34:58.867+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.868306+0000) 2019-10-13T05:34:58.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:34:58.868+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.868607+0000) 2019-10-13T05:34:58.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:34:58.868+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.868869+0000) 2019-10-13T05:34:58.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:34:58.868+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.868953+0000) 2019-10-13T05:34:58.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:34:58.862+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.864339+0000) 2019-10-13T05:34:58.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:34:58.863+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.865392+0000) 2019-10-13T05:34:58.874 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:34:58.868+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.869343+0000) 2019-10-13T05:34:58.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:34:58.869+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:34:58.869619+0000) 2019-10-13T05:34:58.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:34:58.870+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:00.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:00.397+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:00.398085+0000) 2019-10-13T05:35:01.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:01.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:01.817911+0000) 2019-10-13T05:35:02.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:02.685+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:02.685420+0000) 2019-10-13T05:35:02.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:02.684+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:02.684902+0000) 2019-10-13T05:35:03.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:03.866+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.867421+0000) 2019-10-13T05:35:03.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:03.872+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.872554+0000) 2019-10-13T05:35:03.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:03.872+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.872749+0000) 2019-10-13T05:35:03.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:03.872+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.872704+0000) 2019-10-13T05:35:03.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:03.872+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.872847+0000) 2019-10-13T05:35:03.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:03.873+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.873488+0000) 2019-10-13T05:35:03.877 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:03.873+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.873711+0000) 2019-10-13T05:35:03.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:03.869+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:03.870183+0000) 2019-10-13T05:35:03.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:03.875+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80416 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:04.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:04.180+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:04.180442+0000) 2019-10-13T05:35:08.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:08.224+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.224341+0000) 2019-10-13T05:35:08.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:08.563+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.564008+0000) 2019-10-13T05:35:08.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:08.563+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.564012+0000) 2019-10-13T05:35:08.880 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:08.876+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.877173+0000) 2019-10-13T05:35:08.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:08.872+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.872883+0000) 2019-10-13T05:35:08.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:08.877+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.877441+0000) 2019-10-13T05:35:08.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:08.877+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.877909+0000) 2019-10-13T05:35:08.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:08.877+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.878390+0000) 2019-10-13T05:35:08.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:08.878+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.878512+0000) 2019-10-13T05:35:08.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:08.878+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.878700+0000) 2019-10-13T05:35:08.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:08.874+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:08.875297+0000) 2019-10-13T05:35:08.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:08.880+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80447 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:10.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:10.399+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:10.400514+0000) 2019-10-13T05:35:11.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:11.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:11.818084+0000) 2019-10-13T05:35:12.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:12.685+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:12.685567+0000) 2019-10-13T05:35:12.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:12.684+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:12.685071+0000) 2019-10-13T05:35:13.420 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:35:13.420 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:13.424 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:35:13.424 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:13.885 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:13.881+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.881673+0000) 2019-10-13T05:35:13.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:13.881+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.881828+0000) 2019-10-13T05:35:13.886 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:13.881+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.882115+0000) 2019-10-13T05:35:13.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:13.881+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.882322+0000) 2019-10-13T05:35:13.888 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:13.882+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.882450+0000) 2019-10-13T05:35:13.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:13.877+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.877960+0000) 2019-10-13T05:35:13.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:13.879+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.879948+0000) 2019-10-13T05:35:13.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:13.882+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:13.883047+0000) 2019-10-13T05:35:13.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:13.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80479 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:14.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:14.180+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:14.180576+0000) 2019-10-13T05:35:18.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:18.223+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.224582+0000) 2019-10-13T05:35:18.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:18.563+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.564202+0000) 2019-10-13T05:35:18.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:18.563+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.564203+0000) 2019-10-13T05:35:18.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:18.880+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.881322+0000) 2019-10-13T05:35:18.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:18.885+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.886433+0000) 2019-10-13T05:35:18.890 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:18.885+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.886677+0000) 2019-10-13T05:35:18.891 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:18.885+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.886881+0000) 2019-10-13T05:35:18.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:18.885+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.887056+0000) 2019-10-13T05:35:18.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:18.885+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.887242+0000) 2019-10-13T05:35:18.891 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:18.886+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.887428+0000) 2019-10-13T05:35:18.892 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:18.882+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:18.883376+0000) 2019-10-13T05:35:18.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:18.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80508 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:20.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:20.402+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:20.402868+0000) 2019-10-13T05:35:21.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:21.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:21.818275+0000) 2019-10-13T05:35:22.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:22.684+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:22.685734+0000) 2019-10-13T05:35:22.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:22.684+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:22.685272+0000) 2019-10-13T05:35:23.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:23.889+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.891302+0000) 2019-10-13T05:35:23.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:23.890+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.891651+0000) 2019-10-13T05:35:23.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:23.886+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.887133+0000) 2019-10-13T05:35:23.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:23.890+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.891865+0000) 2019-10-13T05:35:23.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:23.890+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.892346+0000) 2019-10-13T05:35:23.896 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:23.891+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.892471+0000) 2019-10-13T05:35:23.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:23.891+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.892624+0000) 2019-10-13T05:35:23.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:23.888+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:23.889489+0000) 2019-10-13T05:35:23.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:23.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:24.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:24.179+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:24.180742+0000) 2019-10-13T05:35:28.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:28.224+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.224743+0000) 2019-10-13T05:35:28.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:28.563+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.564415+0000) 2019-10-13T05:35:28.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:28.563+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.564423+0000) 2019-10-13T05:35:28.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:28.895+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.897074+0000) 2019-10-13T05:35:28.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:28.891+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.892556+0000) 2019-10-13T05:35:28.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:28.895+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.897313+0000) 2019-10-13T05:35:28.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:28.896+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.897686+0000) 2019-10-13T05:35:28.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:28.896+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.897973+0000) 2019-10-13T05:35:28.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:28.896+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.898195+0000) 2019-10-13T05:35:28.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:28.897+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.898393+0000) 2019-10-13T05:35:28.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:28.893+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:28.893751+0000) 2019-10-13T05:35:28.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:28.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80571 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:30.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:30.405+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:30.405312+0000) 2019-10-13T05:35:31.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:31.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:31.818447+0000) 2019-10-13T05:35:32.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:32.685+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:32.685926+0000) 2019-10-13T05:35:32.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:32.684+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:32.685427+0000) 2019-10-13T05:35:33.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:33.900+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.901951+0000) 2019-10-13T05:35:33.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:33.896+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.897374+0000) 2019-10-13T05:35:33.906 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:33.901+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.902591+0000) 2019-10-13T05:35:33.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:33.901+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.902647+0000) 2019-10-13T05:35:33.906 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:33.901+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.903014+0000) 2019-10-13T05:35:33.907 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:33.901+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.903144+0000) 2019-10-13T05:35:33.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:33.901+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.903376+0000) 2019-10-13T05:35:33.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:33.899+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:33.900021+0000) 2019-10-13T05:35:33.908 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:33.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:34.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:34.180+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:34.180921+0000) 2019-10-13T05:35:38.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:38.223+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.224942+0000) 2019-10-13T05:35:38.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:38.564+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.564624+0000) 2019-10-13T05:35:38.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:38.564+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.564621+0000) 2019-10-13T05:35:38.910 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:38.905+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.906799+0000) 2019-10-13T05:35:38.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:38.905+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.906923+0000) 2019-10-13T05:35:38.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:38.905+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.907056+0000) 2019-10-13T05:35:38.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:38.901+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.903158+0000) 2019-10-13T05:35:38.911 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:38.906+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.907650+0000) 2019-10-13T05:35:38.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:38.906+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.907829+0000) 2019-10-13T05:35:38.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:38.906+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.908187+0000) 2019-10-13T05:35:38.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:38.902+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:38.904372+0000) 2019-10-13T05:35:38.913 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:38.908+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80632 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:40.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:40.406+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:40.407627+0000) 2019-10-13T05:35:41.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:41.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:41.818639+0000) 2019-10-13T05:35:42.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:42.685+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:42.686098+0000) 2019-10-13T05:35:42.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:42.684+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:42.685608+0000) 2019-10-13T05:35:43.591 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:35:43.592 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:43.596 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:35:43.596 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:43.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:43.911+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.911736+0000) 2019-10-13T05:35:43.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:43.911+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.912085+0000) 2019-10-13T05:35:43.916 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:43.911+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.912221+0000) 2019-10-13T05:35:43.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:43.905+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.907001+0000) 2019-10-13T05:35:43.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:43.912+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.912808+0000) 2019-10-13T05:35:43.917 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:43.912+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.913002+0000) 2019-10-13T05:35:43.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:43.912+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.913145+0000) 2019-10-13T05:35:43.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:43.907+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:43.909234+0000) 2019-10-13T05:35:43.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:43.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:44.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:44.180+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:44.181118+0000) 2019-10-13T05:35:48.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:48.224+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.225112+0000) 2019-10-13T05:35:48.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:48.564+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.564815+0000) 2019-10-13T05:35:48.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:48.564+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.564809+0000) 2019-10-13T05:35:48.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:48.910+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.912167+0000) 2019-10-13T05:35:48.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:48.916+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.917244+0000) 2019-10-13T05:35:48.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:48.917+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.917629+0000) 2019-10-13T05:35:48.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:48.917+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.917704+0000) 2019-10-13T05:35:48.923 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:48.917+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.917902+0000) 2019-10-13T05:35:48.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:48.917+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.917948+0000) 2019-10-13T05:35:48.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:48.917+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.918120+0000) 2019-10-13T05:35:48.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:48.914+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:48.914887+0000) 2019-10-13T05:35:48.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:48.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:50.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:50.409+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:50.410068+0000) 2019-10-13T05:35:51.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:51.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:51.818829+0000) 2019-10-13T05:35:52.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:52.686+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:52.686257+0000) 2019-10-13T05:35:52.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:52.685+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:52.685792+0000) 2019-10-13T05:35:53.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:53.916+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.917429+0000) 2019-10-13T05:35:53.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:53.922+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.922588+0000) 2019-10-13T05:35:53.927 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:53.922+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.922659+0000) 2019-10-13T05:35:53.927 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:53.922+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.922870+0000) 2019-10-13T05:35:53.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:53.922+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.923201+0000) 2019-10-13T05:35:53.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:53.922+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.923351+0000) 2019-10-13T05:35:53.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:53.923+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.923838+0000) 2019-10-13T05:35:53.928 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:53.918+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:53.919397+0000) 2019-10-13T05:35:53.929 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:53.925+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:35:54.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:54.181+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:54.181291+0000) 2019-10-13T05:35:58.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:58.225+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.225306+0000) 2019-10-13T05:35:58.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:58.564+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.564951+0000) 2019-10-13T05:35:58.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:58.564+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.564966+0000) 2019-10-13T05:35:58.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:35:58.926+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.927304+0000) 2019-10-13T05:35:58.931 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:35:58.927+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.928129+0000) 2019-10-13T05:35:58.931 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:35:58.927+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.928262+0000) 2019-10-13T05:35:58.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:35:58.922+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.923564+0000) 2019-10-13T05:35:58.933 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:35:58.928+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.928566+0000) 2019-10-13T05:35:58.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:35:58.928+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.929127+0000) 2019-10-13T05:35:58.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:35:58.924+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.925082+0000) 2019-10-13T05:35:58.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:35:58.929+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:35:58.929544+0000) 2019-10-13T05:35:58.935 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:35:58.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:00.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:00.411+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:00.412310+0000) 2019-10-13T05:36:01.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:01.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:01.818947+0000) 2019-10-13T05:36:02.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:02.685+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:02.686434+0000) 2019-10-13T05:36:02.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:02.685+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:02.685978+0000) 2019-10-13T05:36:03.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:03.931+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.932683+0000) 2019-10-13T05:36:03.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:03.931+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.932827+0000) 2019-10-13T05:36:03.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:03.927+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.928504+0000) 2019-10-13T05:36:03.937 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:03.931+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.933338+0000) 2019-10-13T05:36:03.937 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:03.932+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.933637+0000) 2019-10-13T05:36:03.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:03.932+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.933800+0000) 2019-10-13T05:36:03.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:03.932+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.934023+0000) 2019-10-13T05:36:03.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:03.929+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:03.930381+0000) 2019-10-13T05:36:03.939 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:03.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:04.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:04.180+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:04.181441+0000) 2019-10-13T05:36:08.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:08.224+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.225475+0000) 2019-10-13T05:36:08.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:08.563+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.565154+0000) 2019-10-13T05:36:08.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:08.563+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.565153+0000) 2019-10-13T05:36:08.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:08.936+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.937700+0000) 2019-10-13T05:36:08.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:08.936+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.937860+0000) 2019-10-13T05:36:08.942 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:08.937+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.938655+0000) 2019-10-13T05:36:08.942 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:08.937+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.938796+0000) 2019-10-13T05:36:08.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:08.937+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.938994+0000) 2019-10-13T05:36:08.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:08.933+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.934241+0000) 2019-10-13T05:36:08.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:08.937+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.939260+0000) 2019-10-13T05:36:08.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:08.934+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:08.935484+0000) 2019-10-13T05:36:08.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:08.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80819 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:10.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:10.414+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:10.414643+0000) 2019-10-13T05:36:11.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:11.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:11.819113+0000) 2019-10-13T05:36:12.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:12.685+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:12.686639+0000) 2019-10-13T05:36:12.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:12.685+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:12.686158+0000) 2019-10-13T05:36:13.736 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:36:13.737 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:13.746 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:36:13.746 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:13.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:13.941+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.942812+0000) 2019-10-13T05:36:13.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:13.937+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.938185+0000) 2019-10-13T05:36:13.947 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:13.941+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.943367+0000) 2019-10-13T05:36:13.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:13.942+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.943521+0000) 2019-10-13T05:36:13.947 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:13.942+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.943697+0000) 2019-10-13T05:36:13.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:13.942+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.943901+0000) 2019-10-13T05:36:13.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:13.939+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.940212+0000) 2019-10-13T05:36:13.948 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:13.942+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:13.944169+0000) 2019-10-13T05:36:13.949 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:13.944+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:14.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:14.180+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:14.181595+0000) 2019-10-13T05:36:18.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:18.225+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.225671+0000) 2019-10-13T05:36:18.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:18.564+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.565389+0000) 2019-10-13T05:36:18.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:18.564+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.565388+0000) 2019-10-13T05:36:18.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:18.945+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.947042+0000) 2019-10-13T05:36:18.950 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:18.945+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.947205+0000) 2019-10-13T05:36:18.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:18.941+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.942450+0000) 2019-10-13T05:36:18.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:18.946+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.947814+0000) 2019-10-13T05:36:18.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:18.946+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.947938+0000) 2019-10-13T05:36:18.952 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:18.947+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.948565+0000) 2019-10-13T05:36:18.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:18.944+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.944993+0000) 2019-10-13T05:36:18.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:18.947+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:18.948844+0000) 2019-10-13T05:36:18.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:18.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80880 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:20.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:20.416+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:20.416999+0000) 2019-10-13T05:36:21.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:21.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:21.819285+0000) 2019-10-13T05:36:22.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:22.686+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:22.686811+0000) 2019-10-13T05:36:22.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:22.685+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:22.686357+0000) 2019-10-13T05:36:23.956 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:23.951+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.952409+0000) 2019-10-13T05:36:23.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:23.951+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.952645+0000) 2019-10-13T05:36:23.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:23.951+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.952722+0000) 2019-10-13T05:36:23.956 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:23.951+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.952906+0000) 2019-10-13T05:36:23.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:23.947+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.948211+0000) 2019-10-13T05:36:23.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:23.951+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.953233+0000) 2019-10-13T05:36:23.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:23.952+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.953604+0000) 2019-10-13T05:36:23.958 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:23.953+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:23.958 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:23.949+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:23.950609+0000) 2019-10-13T05:36:24.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:24.181+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:24.181764+0000) 2019-10-13T05:36:28.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:28.225+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.225840+0000) 2019-10-13T05:36:28.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:28.565+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.565586+0000) 2019-10-13T05:36:28.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:28.565+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.565584+0000) 2019-10-13T05:36:28.973 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:28.956+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.956554+0000) 2019-10-13T05:36:28.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:28.956+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.956902+0000) 2019-10-13T05:36:28.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:28.956+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.957101+0000) 2019-10-13T05:36:28.975 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:28.956+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.957237+0000) 2019-10-13T05:36:28.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:28.956+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.957363+0000) 2019-10-13T05:36:28.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:28.957+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.957594+0000) 2019-10-13T05:36:28.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:28.950+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.952538+0000) 2019-10-13T05:36:28.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:28.951+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:28.953638+0000) 2019-10-13T05:36:28.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:28.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:30.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:30.418+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:30.419314+0000) 2019-10-13T05:36:31.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:31.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:31.819467+0000) 2019-10-13T05:36:32.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:32.686+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:32.686971+0000) 2019-10-13T05:36:32.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:32.685+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:32.686540+0000) 2019-10-13T05:36:33.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:33.960+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.961242+0000) 2019-10-13T05:36:33.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:33.961+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.961406+0000) 2019-10-13T05:36:33.965 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:33.961+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.961577+0000) 2019-10-13T05:36:33.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:33.955+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.957616+0000) 2019-10-13T05:36:33.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:33.961+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.962234+0000) 2019-10-13T05:36:33.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:33.962+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.962428+0000) 2019-10-13T05:36:33.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:33.962+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.962738+0000) 2019-10-13T05:36:33.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:33.963+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 80975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:33.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:33.958+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:33.960209+0000) 2019-10-13T05:36:34.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:34.181+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:34.181934+0000) 2019-10-13T05:36:38.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:38.225+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.226014+0000) 2019-10-13T05:36:38.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:38.565+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.565764+0000) 2019-10-13T05:36:38.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:38.565+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.565781+0000) 2019-10-13T05:36:38.969 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:38.965+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.965848+0000) 2019-10-13T05:36:38.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:38.965+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.966058+0000) 2019-10-13T05:36:38.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:38.965+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.966379+0000) 2019-10-13T05:36:38.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:38.965+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.966382+0000) 2019-10-13T05:36:38.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:38.960+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.961706+0000) 2019-10-13T05:36:38.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:38.966+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.966791+0000) 2019-10-13T05:36:38.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:38.966+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.966948+0000) 2019-10-13T05:36:38.972 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:38.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:38.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:38.964+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:38.964741+0000) 2019-10-13T05:36:40.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:40.421+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:40.421750+0000) 2019-10-13T05:36:41.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:41.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:41.819617+0000) 2019-10-13T05:36:42.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:42.687+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:42.687153+0000) 2019-10-13T05:36:42.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:42.686+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:42.686751+0000) 2019-10-13T05:36:43.878 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:36:43.878 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:43.882 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:36:43.882 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:43.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:43.965+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.965912+0000) 2019-10-13T05:36:43.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:43.970+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.970987+0000) 2019-10-13T05:36:43.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:43.970+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.971225+0000) 2019-10-13T05:36:43.975 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:43.970+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.971322+0000) 2019-10-13T05:36:43.976 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:43.971+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.971843+0000) 2019-10-13T05:36:43.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:43.971+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.971932+0000) 2019-10-13T05:36:43.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:43.971+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.972038+0000) 2019-10-13T05:36:43.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:43.967+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:43.968121+0000) 2019-10-13T05:36:43.977 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:43.973+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:44.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:44.182+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:44.182101+0000) 2019-10-13T05:36:48.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:48.225+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.226176+0000) 2019-10-13T05:36:48.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:48.565+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.565973+0000) 2019-10-13T05:36:48.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:48.565+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.565973+0000) 2019-10-13T05:36:48.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:48.974+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.975949+0000) 2019-10-13T05:36:48.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:48.974+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.976178+0000) 2019-10-13T05:36:48.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:48.975+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.976391+0000) 2019-10-13T05:36:48.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:48.971+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.971896+0000) 2019-10-13T05:36:48.981 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:48.975+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.976532+0000) 2019-10-13T05:36:48.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:48.975+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.976983+0000) 2019-10-13T05:36:48.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:48.975+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.977110+0000) 2019-10-13T05:36:48.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:48.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81067 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:48.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:48.974+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:48.974887+0000) 2019-10-13T05:36:50.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:50.423+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:50.424072+0000) 2019-10-13T05:36:51.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:51.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:51.819802+0000) 2019-10-13T05:36:52.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:52.686+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:52.687333+0000) 2019-10-13T05:36:52.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:52.686+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:52.686906+0000) 2019-10-13T05:36:53.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:53.979+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.980956+0000) 2019-10-13T05:36:53.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:53.979+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.981338+0000) 2019-10-13T05:36:53.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:53.975+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.976635+0000) 2019-10-13T05:36:53.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:53.980+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.981567+0000) 2019-10-13T05:36:53.987 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:53.980+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.981717+0000) 2019-10-13T05:36:53.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:53.980+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.981827+0000) 2019-10-13T05:36:53.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:53.981+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.982819+0000) 2019-10-13T05:36:53.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:53.978+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:53.978916+0000) 2019-10-13T05:36:53.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:53.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:54.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:54.180+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:54.182239+0000) 2019-10-13T05:36:58.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:58.225+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.226334+0000) 2019-10-13T05:36:58.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:58.565+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.566170+0000) 2019-10-13T05:36:58.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:58.565+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.566177+0000) 2019-10-13T05:36:58.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:36:58.981+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.981975+0000) 2019-10-13T05:36:58.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:36:58.985+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.987069+0000) 2019-10-13T05:36:58.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:36:58.985+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.987305+0000) 2019-10-13T05:36:58.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:36:58.986+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.987509+0000) 2019-10-13T05:36:58.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:36:58.986+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.987757+0000) 2019-10-13T05:36:58.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:36:58.986+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.987850+0000) 2019-10-13T05:36:58.992 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:36:58.986+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.988378+0000) 2019-10-13T05:36:58.992 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:36:58.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81128 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:36:58.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:36:58.985+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:36:58.986502+0000) 2019-10-13T05:37:00.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:00.425+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:00.426314+0000) 2019-10-13T05:37:01.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:01.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:01.820029+0000) 2019-10-13T05:37:02.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:02.686+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:02.687476+0000) 2019-10-13T05:37:02.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:02.686+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:02.687077+0000) 2019-10-13T05:37:03.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:03.986+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.986783+0000) 2019-10-13T05:37:03.996 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:03.990+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.991890+0000) 2019-10-13T05:37:03.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:03.990+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.992132+0000) 2019-10-13T05:37:03.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:03.991+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.992497+0000) 2019-10-13T05:37:03.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:03.990+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.992340+0000) 2019-10-13T05:37:03.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:03.991+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.992648+0000) 2019-10-13T05:37:03.997 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:03.991+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.993147+0000) 2019-10-13T05:37:03.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:03.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81160 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:03.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:03.990+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:03.991325+0000) 2019-10-13T05:37:04.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:04.181+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:04.182391+0000) 2019-10-13T05:37:08.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:08.226+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.226542+0000) 2019-10-13T05:37:08.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:08.565+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.566321+0000) 2019-10-13T05:37:08.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:08.565+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.566370+0000) 2019-10-13T05:37:08.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:08.990+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.991413+0000) 2019-10-13T05:37:09.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:08.995+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.996608+0000) 2019-10-13T05:37:09.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:08.995+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.996993+0000) 2019-10-13T05:37:09.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:08.995+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.996835+0000) 2019-10-13T05:37:09.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:08.996+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.997407+0000) 2019-10-13T05:37:09.001 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:08.996+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.997532+0000) 2019-10-13T05:37:09.001 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:08.996+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.997868+0000) 2019-10-13T05:37:09.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:08.994+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:08.994924+0000) 2019-10-13T05:37:09.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:08.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81191 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:10.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:10.428+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:10.428412+0000) 2019-10-13T05:37:11.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:11.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:11.820213+0000) 2019-10-13T05:37:12.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:12.687+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:12.687643+0000) 2019-10-13T05:37:12.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:12.687+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:12.687261+0000) 2019-10-13T05:37:14.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:14.001+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.001552+0000) 2019-10-13T05:37:14.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:13.996+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:13.996801+0000) 2019-10-13T05:37:14.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:14.001+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.002116+0000) 2019-10-13T05:37:14.006 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:14.002+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.002502+0000) 2019-10-13T05:37:14.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:14.001+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.002371+0000) 2019-10-13T05:37:14.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:14.002+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.003042+0000) 2019-10-13T05:37:14.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:14.002+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.003187+0000) 2019-10-13T05:37:14.007 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:14.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:14.008 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:13.999+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.000342+0000) 2019-10-13T05:37:14.045 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:37:14.046 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:14.049 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:37:14.049 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:14.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:14.182+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:14.182616+0000) 2019-10-13T05:37:18.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:18.226+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:18.226731+0000) 2019-10-13T05:37:18.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:18.566+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:18.566513+0000) 2019-10-13T05:37:18.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:18.566+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:18.566606+0000) 2019-10-13T05:37:19.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:18.999+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.001473+0000) 2019-10-13T05:37:19.010 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:19.006+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.006625+0000) 2019-10-13T05:37:19.010 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:19.006+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.006988+0000) 2019-10-13T05:37:19.011 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:19.006+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.007358+0000) 2019-10-13T05:37:19.011 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:19.007+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.007452+0000) 2019-10-13T05:37:19.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:19.007+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.007654+0000) 2019-10-13T05:37:19.011 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:19.007+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.008023+0000) 2019-10-13T05:37:19.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:19.002+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:19.003896+0000) 2019-10-13T05:37:19.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:19.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81252 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:20.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:20.429+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:20.430714+0000) 2019-10-13T05:37:21.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:21.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:21.820368+0000) 2019-10-13T05:37:22.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:22.687+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:22.687819+0000) 2019-10-13T05:37:22.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:22.686+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:22.687437+0000) 2019-10-13T05:37:24.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:24.011+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.012095+0000) 2019-10-13T05:37:24.016 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:24.011+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.012305+0000) 2019-10-13T05:37:24.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:24.012+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.012581+0000) 2019-10-13T05:37:24.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:24.012+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.012431+0000) 2019-10-13T05:37:24.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:24.012+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.012863+0000) 2019-10-13T05:37:24.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:24.012+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.013340+0000) 2019-10-13T05:37:24.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:24.006+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.008053+0000) 2019-10-13T05:37:24.018 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:24.014+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:24.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:24.008+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.010420+0000) 2019-10-13T05:37:24.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:24.182+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:24.182728+0000) 2019-10-13T05:37:28.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:28.226+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:28.226933+0000) 2019-10-13T05:37:28.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:28.566+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:28.566712+0000) 2019-10-13T05:37:28.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:28.566+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:28.566769+0000) 2019-10-13T05:37:29.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:29.010+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.011585+0000) 2019-10-13T05:37:29.023 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:29.015+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.016285+0000) 2019-10-13T05:37:29.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:29.016+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.016674+0000) 2019-10-13T05:37:29.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:29.017+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.017405+0000) 2019-10-13T05:37:29.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:29.017+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.017506+0000) 2019-10-13T05:37:29.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:29.017+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.017612+0000) 2019-10-13T05:37:29.024 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:29.017+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.017769+0000) 2019-10-13T05:37:29.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:29.019+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:29.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:29.013+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:29.014073+0000) 2019-10-13T05:37:30.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:30.432+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:30.433034+0000) 2019-10-13T05:37:31.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:31.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:31.820535+0000) 2019-10-13T05:37:32.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:32.686+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:32.687946+0000) 2019-10-13T05:37:32.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:32.686+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:32.687620+0000) 2019-10-13T05:37:34.025 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:34.019+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.021374+0000) 2019-10-13T05:37:34.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:34.020+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.021698+0000) 2019-10-13T05:37:34.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:34.015+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.016546+0000) 2019-10-13T05:37:34.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:34.020+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.022050+0000) 2019-10-13T05:37:34.027 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:34.020+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.022154+0000) 2019-10-13T05:37:34.027 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:34.020+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.022349+0000) 2019-10-13T05:37:34.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:34.021+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.022423+0000) 2019-10-13T05:37:34.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:34.018+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.019010+0000) 2019-10-13T05:37:34.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:34.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:34.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:34.181+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:34.182860+0000) 2019-10-13T05:37:38.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:38.226+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:38.227120+0000) 2019-10-13T05:37:38.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:38.566+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:38.566885+0000) 2019-10-13T05:37:38.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:38.566+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:38.566928+0000) 2019-10-13T05:37:39.029 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:39.024+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.026236+0000) 2019-10-13T05:37:39.030 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:39.025+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.026731+0000) 2019-10-13T05:37:39.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:39.025+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.026626+0000) 2019-10-13T05:37:39.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:39.025+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.026946+0000) 2019-10-13T05:37:39.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:39.025+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.027226+0000) 2019-10-13T05:37:39.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:39.021+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.022085+0000) 2019-10-13T05:37:39.031 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:39.026+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.027583+0000) 2019-10-13T05:37:39.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:39.023+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:39.024536+0000) 2019-10-13T05:37:39.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:39.028+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:40.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:40.434+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:40.435214+0000) 2019-10-13T05:37:41.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:41.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:41.820758+0000) 2019-10-13T05:37:42.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:42.687+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:42.688056+0000) 2019-10-13T05:37:42.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:42.687+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:42.687814+0000) 2019-10-13T05:37:44.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:44.030+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.031496+0000) 2019-10-13T05:37:44.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:44.030+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.031618+0000) 2019-10-13T05:37:44.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:44.030+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.031818+0000) 2019-10-13T05:37:44.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:44.030+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.032364+0000) 2019-10-13T05:37:44.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:44.031+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.032566+0000) 2019-10-13T05:37:44.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:44.031+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.032835+0000) 2019-10-13T05:37:44.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:44.026+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.027181+0000) 2019-10-13T05:37:44.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:44.029+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.029812+0000) 2019-10-13T05:37:44.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:44.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:44.130 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:37:44.130 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:44.134 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:37:44.134 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:44.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:44.181+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:44.183003+0000) 2019-10-13T05:37:48.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:48.225+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:48.227289+0000) 2019-10-13T05:37:48.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:48.566+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:48.567077+0000) 2019-10-13T05:37:48.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:48.566+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:48.567116+0000) 2019-10-13T05:37:49.040 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:49.035+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.036847+0000) 2019-10-13T05:37:49.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:49.031+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.032240+0000) 2019-10-13T05:37:49.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:49.035+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.037362+0000) 2019-10-13T05:37:49.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:49.036+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.037525+0000) 2019-10-13T05:37:49.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:49.036+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.037656+0000) 2019-10-13T05:37:49.042 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:49.036+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.038029+0000) 2019-10-13T05:37:49.042 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:49.036+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.038213+0000) 2019-10-13T05:37:49.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:49.034+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:49.035081+0000) 2019-10-13T05:37:49.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:49.038+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81439 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:50.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:50.437+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:50.437508+0000) 2019-10-13T05:37:51.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:51.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:51.821004+0000) 2019-10-13T05:37:52.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:52.686+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:52.688225+0000) 2019-10-13T05:37:52.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:52.687+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:52.687974+0000) 2019-10-13T05:37:54.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:54.040+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.042080+0000) 2019-10-13T05:37:54.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:54.040+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.042302+0000) 2019-10-13T05:37:54.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:54.041+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.042543+0000) 2019-10-13T05:37:54.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:54.037+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.037758+0000) 2019-10-13T05:37:54.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:54.041+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.042930+0000) 2019-10-13T05:37:54.047 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:54.041+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.043092+0000) 2019-10-13T05:37:54.048 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:54.041+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.043259+0000) 2019-10-13T05:37:54.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:54.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81471 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:37:54.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:54.040+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.040892+0000) 2019-10-13T05:37:54.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:54.181+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:54.183199+0000) 2019-10-13T05:37:58.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:58.227+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:58.227483+0000) 2019-10-13T05:37:58.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:58.566+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:58.567263+0000) 2019-10-13T05:37:58.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:58.566+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:58.567304+0000) 2019-10-13T05:37:59.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:37:59.046+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.047002+0000) 2019-10-13T05:37:59.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:37:59.041+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.042405+0000) 2019-10-13T05:37:59.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:37:59.047+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.047661+0000) 2019-10-13T05:37:59.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:37:59.047+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.047762+0000) 2019-10-13T05:37:59.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:37:59.047+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.047812+0000) 2019-10-13T05:37:59.052 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:37:59.047+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.048151+0000) 2019-10-13T05:37:59.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:37:59.047+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.048331+0000) 2019-10-13T05:37:59.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:37:59.044+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:37:59.045181+0000) 2019-10-13T05:37:59.053 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:37:59.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:00.448 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:00.439+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:00.439884+0000) 2019-10-13T05:38:01.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:01.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:01.821134+0000) 2019-10-13T05:38:02.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:02.688+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:02.688140+0000) 2019-10-13T05:38:02.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:02.688+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:02.688390+0000) 2019-10-13T05:38:04.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:04.045+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.047150+0000) 2019-10-13T05:38:04.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:04.051+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.052380+0000) 2019-10-13T05:38:04.057 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:04.052+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.052575+0000) 2019-10-13T05:38:04.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:04.052+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.052684+0000) 2019-10-13T05:38:04.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:04.052+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.053194+0000) 2019-10-13T05:38:04.057 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:04.052+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.053321+0000) 2019-10-13T05:38:04.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:04.053+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.053491+0000) 2019-10-13T05:38:04.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:04.048+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.049938+0000) 2019-10-13T05:38:04.059 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:04.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81532 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:04.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:04.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:04.183332+0000) 2019-10-13T05:38:08.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:08.227+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:08.227656+0000) 2019-10-13T05:38:08.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:08.567+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:08.567449+0000) 2019-10-13T05:38:08.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:08.567+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:08.567531+0000) 2019-10-13T05:38:09.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:09.050+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.052399+0000) 2019-10-13T05:38:09.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:09.057+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.057678+0000) 2019-10-13T05:38:09.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:09.057+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.057833+0000) 2019-10-13T05:38:09.061 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:09.057+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.057989+0000) 2019-10-13T05:38:09.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:09.057+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.058387+0000) 2019-10-13T05:38:09.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:09.058+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.058726+0000) 2019-10-13T05:38:09.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:09.058+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.059096+0000) 2019-10-13T05:38:09.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:09.054+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:09.055918+0000) 2019-10-13T05:38:09.064 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:09.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81563 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:10.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:10.441+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:10.442192+0000) 2019-10-13T05:38:11.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:11.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:11.821241+0000) 2019-10-13T05:38:12.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:12.688+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:12.688550+0000) 2019-10-13T05:38:12.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:12.687+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:12.688328+0000) 2019-10-13T05:38:14.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:14.055+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.057406+0000) 2019-10-13T05:38:14.067 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:14.062+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.063008+0000) 2019-10-13T05:38:14.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:14.062+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.063125+0000) 2019-10-13T05:38:14.067 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:14.062+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.063359+0000) 2019-10-13T05:38:14.067 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:14.063+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.063465+0000) 2019-10-13T05:38:14.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:14.063+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.063525+0000) 2019-10-13T05:38:14.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:14.063+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.063821+0000) 2019-10-13T05:38:14.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:14.058+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.060566+0000) 2019-10-13T05:38:14.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:14.065+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:14.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:14.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:14.183457+0000) 2019-10-13T05:38:14.353 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:38:14.353 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:14.357 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:38:14.358 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:18.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:18.226+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:18.227825+0000) 2019-10-13T05:38:18.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:18.566+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:18.567643+0000) 2019-10-13T05:38:18.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:18.566+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:18.567738+0000) 2019-10-13T05:38:19.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:19.066+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.067586+0000) 2019-10-13T05:38:19.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:19.066+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.067789+0000) 2019-10-13T05:38:19.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:19.062+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.063331+0000) 2019-10-13T05:38:19.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:19.066+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.068076+0000) 2019-10-13T05:38:19.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:19.067+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.068623+0000) 2019-10-13T05:38:19.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:19.067+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.068777+0000) 2019-10-13T05:38:19.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:19.067+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.069190+0000) 2019-10-13T05:38:19.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:19.064+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:19.065085+0000) 2019-10-13T05:38:19.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:19.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81624 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:20.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:20.444+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:20.444794+0000) 2019-10-13T05:38:21.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:21.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:21.821381+0000) 2019-10-13T05:38:22.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:22.687+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:22.688762+0000) 2019-10-13T05:38:22.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:22.687+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:22.688491+0000) 2019-10-13T05:38:24.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:24.067+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.067936+0000) 2019-10-13T05:38:24.077 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:24.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.073214+0000) 2019-10-13T05:38:24.078 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:24.072+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.073561+0000) 2019-10-13T05:38:24.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:24.072+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.073942+0000) 2019-10-13T05:38:24.078 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:24.072+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.074053+0000) 2019-10-13T05:38:24.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:24.072+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.074288+0000) 2019-10-13T05:38:24.078 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:24.073+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.074601+0000) 2019-10-13T05:38:24.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:24.070+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.071304+0000) 2019-10-13T05:38:24.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:24.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81656 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:24.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:24.182+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:24.183602+0000) 2019-10-13T05:38:28.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:28.227+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:28.228027+0000) 2019-10-13T05:38:28.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:28.567+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:28.567818+0000) 2019-10-13T05:38:28.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:28.567+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:28.567935+0000) 2019-10-13T05:38:29.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:29.077+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.078693+0000) 2019-10-13T05:38:29.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:29.073+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.074343+0000) 2019-10-13T05:38:29.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:29.077+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.079177+0000) 2019-10-13T05:38:29.083 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:29.078+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.079644+0000) 2019-10-13T05:38:29.083 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:29.078+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.079955+0000) 2019-10-13T05:38:29.084 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:29.078+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.080094+0000) 2019-10-13T05:38:29.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:29.079+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.080422+0000) 2019-10-13T05:38:29.084 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:29.075+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:29.076464+0000) 2019-10-13T05:38:29.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:29.081+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81687 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:30.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:30.446+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:30.447322+0000) 2019-10-13T05:38:31.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:31.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:31.821562+0000) 2019-10-13T05:38:32.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:32.688+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:32.688957+0000) 2019-10-13T05:38:32.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:32.688+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:32.688656+0000) 2019-10-13T05:38:34.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:34.083+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.084393+0000) 2019-10-13T05:38:34.088 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:34.083+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.084556+0000) 2019-10-13T05:38:34.088 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:34.083+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.084771+0000) 2019-10-13T05:38:34.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:34.083+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.084914+0000) 2019-10-13T05:38:34.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:34.079+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.080240+0000) 2019-10-13T05:38:34.089 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:34.084+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.085727+0000) 2019-10-13T05:38:34.090 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:34.084+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.085957+0000) 2019-10-13T05:38:34.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:34.081+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.082443+0000) 2019-10-13T05:38:34.090 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:34.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81719 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:34.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:34.182+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:34.183748+0000) 2019-10-13T05:38:38.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:38.227+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:38.228203+0000) 2019-10-13T05:38:38.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:38.567+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:38.568003+0000) 2019-10-13T05:38:38.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:38.567+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:38.568134+0000) 2019-10-13T05:38:39.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:39.083+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.084348+0000) 2019-10-13T05:38:39.093 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:39.088+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.089456+0000) 2019-10-13T05:38:39.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:39.088+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.089653+0000) 2019-10-13T05:38:39.094 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:39.088+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.090153+0000) 2019-10-13T05:38:39.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:39.088+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.090046+0000) 2019-10-13T05:38:39.094 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:39.088+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.090229+0000) 2019-10-13T05:38:39.094 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:39.089+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.090511+0000) 2019-10-13T05:38:39.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:39.086+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:39.087545+0000) 2019-10-13T05:38:39.096 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:39.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81748 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:40.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:40.450+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:40.450576+0000) 2019-10-13T05:38:41.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:41.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:41.821675+0000) 2019-10-13T05:38:42.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:42.688+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:42.689128+0000) 2019-10-13T05:38:42.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:42.688+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:42.688795+0000) 2019-10-13T05:38:44.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:44.093+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.094145+0000) 2019-10-13T05:38:44.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:44.088+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.089305+0000) 2019-10-13T05:38:44.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:44.094+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.094444+0000) 2019-10-13T05:38:44.098 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:44.094+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.094875+0000) 2019-10-13T05:38:44.098 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:44.094+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.095151+0000) 2019-10-13T05:38:44.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:44.095+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.095486+0000) 2019-10-13T05:38:44.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:44.095+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.095721+0000) 2019-10-13T05:38:44.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:44.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:44.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:44.091+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.092485+0000) 2019-10-13T05:38:44.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:44.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:44.183928+0000) 2019-10-13T05:38:44.458 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:38:44.458 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:44.462 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:38:44.462 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:48.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:48.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:48.228366+0000) 2019-10-13T05:38:48.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:48.567+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:48.568216+0000) 2019-10-13T05:38:48.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:48.567+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:48.568325+0000) 2019-10-13T05:38:49.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:49.094+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.094780+0000) 2019-10-13T05:38:49.105 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:49.099+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.099481+0000) 2019-10-13T05:38:49.105 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:49.099+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.099628+0000) 2019-10-13T05:38:49.105 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:49.099+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.100252+0000) 2019-10-13T05:38:49.105 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:49.100+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.100431+0000) 2019-10-13T05:38:49.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:49.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.100630+0000) 2019-10-13T05:38:49.106 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:49.100+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.100791+0000) 2019-10-13T05:38:49.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:49.096+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:49.097065+0000) 2019-10-13T05:38:49.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:49.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:50.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:50.452+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:50.452958+0000) 2019-10-13T05:38:51.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:51.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:51.821869+0000) 2019-10-13T05:38:52.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:52.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:52.689299+0000) 2019-10-13T05:38:52.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:52.688+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:52.688990+0000) 2019-10-13T05:38:54.109 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:54.104+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.104393+0000) 2019-10-13T05:38:54.110 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:54.104+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.104518+0000) 2019-10-13T05:38:54.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:54.104+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.104659+0000) 2019-10-13T05:38:54.110 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:54.104+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.104857+0000) 2019-10-13T05:38:54.110 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:54.104+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.104994+0000) 2019-10-13T05:38:54.110 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:54.104+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.105266+0000) 2019-10-13T05:38:54.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:54.099+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.100787+0000) 2019-10-13T05:38:54.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:54.100+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.102673+0000) 2019-10-13T05:38:54.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:54.107+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:38:54.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:54.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:54.184115+0000) 2019-10-13T05:38:58.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:58.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:58.228555+0000) 2019-10-13T05:38:58.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:58.568+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:58.568426+0000) 2019-10-13T05:38:58.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:58.568+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:58.568553+0000) 2019-10-13T05:38:59.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:38:59.108+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.109213+0000) 2019-10-13T05:38:59.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:38:59.108+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.109320+0000) 2019-10-13T05:38:59.113 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:38:59.109+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.109487+0000) 2019-10-13T05:38:59.113 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:38:59.109+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.109709+0000) 2019-10-13T05:38:59.114 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:38:59.109+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.110027+0000) 2019-10-13T05:38:59.114 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:38:59.109+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.110321+0000) 2019-10-13T05:38:59.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:38:59.103+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.105565+0000) 2019-10-13T05:38:59.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:38:59.105+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:38:59.107106+0000) 2019-10-13T05:38:59.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:38:59.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:00.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:00.453+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:00.455241+0000) 2019-10-13T05:39:01.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:01.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:01.822029+0000) 2019-10-13T05:39:02.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:02.688+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:02.689477+0000) 2019-10-13T05:39:02.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:02.688+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:02.689163+0000) 2019-10-13T05:39:04.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:04.107+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.109376+0000) 2019-10-13T05:39:04.118 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:04.113+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.114762+0000) 2019-10-13T05:39:04.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:04.113+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.114890+0000) 2019-10-13T05:39:04.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:04.113+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.115108+0000) 2019-10-13T05:39:04.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:04.114+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.115430+0000) 2019-10-13T05:39:04.119 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:04.114+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.115742+0000) 2019-10-13T05:39:04.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:04.114+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.115860+0000) 2019-10-13T05:39:04.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:04.110+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.112661+0000) 2019-10-13T05:39:04.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:04.116+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81904 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:04.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:04.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:04.184255+0000) 2019-10-13T05:39:08.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:08.227+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:08.228736+0000) 2019-10-13T05:39:08.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:08.567+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:08.568599+0000) 2019-10-13T05:39:08.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:08.567+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:08.568719+0000) 2019-10-13T05:39:09.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:09.118+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.119712+0000) 2019-10-13T05:39:09.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:09.118+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.119924+0000) 2019-10-13T05:39:09.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:09.118+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.120287+0000) 2019-10-13T05:39:09.124 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:09.114+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.115504+0000) 2019-10-13T05:39:09.125 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:09.119+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.120730+0000) 2019-10-13T05:39:09.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:09.119+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.120762+0000) 2019-10-13T05:39:09.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:09.119+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.120908+0000) 2019-10-13T05:39:09.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:09.115+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:09.116632+0000) 2019-10-13T05:39:09.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:09.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:10.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:10.456+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:10.457579+0000) 2019-10-13T05:39:11.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:11.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:11.822161+0000) 2019-10-13T05:39:12.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:12.688+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:12.689653+0000) 2019-10-13T05:39:12.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:12.688+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:12.689304+0000) 2019-10-13T05:39:14.128 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:14.123+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.124431+0000) 2019-10-13T05:39:14.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:14.123+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.124825+0000) 2019-10-13T05:39:14.128 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:14.123+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.124942+0000) 2019-10-13T05:39:14.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:14.123+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.125060+0000) 2019-10-13T05:39:14.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:14.120+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.120841+0000) 2019-10-13T05:39:14.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:14.124+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.125482+0000) 2019-10-13T05:39:14.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:14.125+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.126447+0000) 2019-10-13T05:39:14.131 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:14.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:14.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:14.122+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.123440+0000) 2019-10-13T05:39:14.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:14.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:14.184388+0000) 2019-10-13T05:39:14.629 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:39:14.629 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:14.633 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:39:14.633 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:18.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:18.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:18.228877+0000) 2019-10-13T05:39:18.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:18.568+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:18.568770+0000) 2019-10-13T05:39:18.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:18.568+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:18.568880+0000) 2019-10-13T05:39:19.133 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:19.128+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.129803+0000) 2019-10-13T05:39:19.133 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:19.128+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.129953+0000) 2019-10-13T05:39:19.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:19.124+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.125091+0000) 2019-10-13T05:39:19.134 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:19.128+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.130387+0000) 2019-10-13T05:39:19.135 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:19.129+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.130660+0000) 2019-10-13T05:39:19.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:19.129+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.130772+0000) 2019-10-13T05:39:19.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:19.129+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.130921+0000) 2019-10-13T05:39:19.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:19.126+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:19.127434+0000) 2019-10-13T05:39:19.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:19.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 81996 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:20.468 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:20.459+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:20.459841+0000) 2019-10-13T05:39:21.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:21.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:21.822322+0000) 2019-10-13T05:39:22.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:22.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:22.689832+0000) 2019-10-13T05:39:22.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:22.689+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:22.689476+0000) 2019-10-13T05:39:24.138 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:24.129+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.129909+0000) 2019-10-13T05:39:24.139 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:24.133+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.135027+0000) 2019-10-13T05:39:24.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:24.133+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.135193+0000) 2019-10-13T05:39:24.139 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:24.134+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.135529+0000) 2019-10-13T05:39:24.140 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:24.134+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.135730+0000) 2019-10-13T05:39:24.140 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:24.134+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.135872+0000) 2019-10-13T05:39:24.140 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:24.134+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.136121+0000) 2019-10-13T05:39:24.141 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:24.132+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.133146+0000) 2019-10-13T05:39:24.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:24.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:24.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:24.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:24.184515+0000) 2019-10-13T05:39:28.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:28.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:28.229045+0000) 2019-10-13T05:39:28.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:28.568+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:28.568973+0000) 2019-10-13T05:39:28.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:28.568+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:28.569047+0000) 2019-10-13T05:39:29.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:29.139+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.139902+0000) 2019-10-13T05:39:29.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:29.139+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.140168+0000) 2019-10-13T05:39:29.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:29.140+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.140593+0000) 2019-10-13T05:39:29.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:29.134+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.135718+0000) 2019-10-13T05:39:29.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:29.140+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.140835+0000) 2019-10-13T05:39:29.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:29.140+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.141293+0000) 2019-10-13T05:39:29.159 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:29.141+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.141565+0000) 2019-10-13T05:39:29.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:29.136+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:29.137471+0000) 2019-10-13T05:39:29.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:29.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82059 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:30.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:30.461+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:30.462213+0000) 2019-10-13T05:39:31.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:31.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:31.822512+0000) 2019-10-13T05:39:32.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:32.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:32.690021+0000) 2019-10-13T05:39:32.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:32.689+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:32.689644+0000) 2019-10-13T05:39:34.148 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:34.145+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.145559+0000) 2019-10-13T05:39:34.150 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:34.139+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.140650+0000) 2019-10-13T05:39:34.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:34.145+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.145873+0000) 2019-10-13T05:39:34.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:34.145+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.146253+0000) 2019-10-13T05:39:34.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:34.145+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.146302+0000) 2019-10-13T05:39:34.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:34.146+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.146437+0000) 2019-10-13T05:39:34.151 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:34.146+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.146764+0000) 2019-10-13T05:39:34.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:34.141+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.142489+0000) 2019-10-13T05:39:34.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:34.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:34.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:34.184+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:34.184602+0000) 2019-10-13T05:39:38.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:38.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:38.229206+0000) 2019-10-13T05:39:38.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:38.568+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:38.569162+0000) 2019-10-13T05:39:38.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:38.568+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:38.569228+0000) 2019-10-13T05:39:39.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:39.150+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.150544+0000) 2019-10-13T05:39:39.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:39.150+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.150841+0000) 2019-10-13T05:39:39.155 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:39.150+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.151053+0000) 2019-10-13T05:39:39.155 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:39.150+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.151135+0000) 2019-10-13T05:39:39.155 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:39.150+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.151268+0000) 2019-10-13T05:39:39.155 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:39.151+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.151411+0000) 2019-10-13T05:39:39.156 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:39.146+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.146801+0000) 2019-10-13T05:39:39.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:39.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82120 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:39.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:39.148+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:39.149049+0000) 2019-10-13T05:39:40.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:40.464+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:40.464282+0000) 2019-10-13T05:39:41.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:41.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:41.822734+0000) 2019-10-13T05:39:42.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:42.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:42.690204+0000) 2019-10-13T05:39:42.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:42.688+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:42.689879+0000) 2019-10-13T05:39:44.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:44.148+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.150494+0000) 2019-10-13T05:39:44.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:44.155+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.155782+0000) 2019-10-13T05:39:44.160 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:44.155+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.155970+0000) 2019-10-13T05:39:44.160 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:44.155+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.156052+0000) 2019-10-13T05:39:44.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:44.155+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.156382+0000) 2019-10-13T05:39:44.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:44.156+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.156561+0000) 2019-10-13T05:39:44.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:44.156+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.156994+0000) 2019-10-13T05:39:44.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:44.152+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.154076+0000) 2019-10-13T05:39:44.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:44.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:44.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:44.184+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:44.184710+0000) 2019-10-13T05:39:44.854 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:39:44.854 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:44.858 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:39:44.859 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:48.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:48.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:48.229417+0000) 2019-10-13T05:39:48.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:48.568+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:48.569325+0000) 2019-10-13T05:39:48.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:48.568+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:48.569419+0000) 2019-10-13T05:39:49.164 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:49.159+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.160952+0000) 2019-10-13T05:39:49.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:49.154+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.156119+0000) 2019-10-13T05:39:49.165 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:49.159+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.161307+0000) 2019-10-13T05:39:49.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:49.160+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.161665+0000) 2019-10-13T05:39:49.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:49.160+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.161918+0000) 2019-10-13T05:39:49.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:49.160+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.162197+0000) 2019-10-13T05:39:49.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:49.155+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.157511+0000) 2019-10-13T05:39:49.166 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:49.160+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:49.162362+0000) 2019-10-13T05:39:49.167 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:49.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82183 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:50.474 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:50.464+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:50.466370+0000) 2019-10-13T05:39:51.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:51.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:51.822966+0000) 2019-10-13T05:39:52.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:52.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:52.690353+0000) 2019-10-13T05:39:52.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:52.689+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:52.690036+0000) 2019-10-13T05:39:54.169 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:54.164+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.165983+0000) 2019-10-13T05:39:54.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:54.164+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.166131+0000) 2019-10-13T05:39:54.170 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:54.160+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.161830+0000) 2019-10-13T05:39:54.171 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:54.164+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.166308+0000) 2019-10-13T05:39:54.171 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:54.165+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.166665+0000) 2019-10-13T05:39:54.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:54.165+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.167125+0000) 2019-10-13T05:39:54.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:54.166+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.167612+0000) 2019-10-13T05:39:54.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:54.162+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.164237+0000) 2019-10-13T05:39:54.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:54.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:39:54.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:54.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:54.184846+0000) 2019-10-13T05:39:58.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:58.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:58.229574+0000) 2019-10-13T05:39:58.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:58.568+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:58.569497+0000) 2019-10-13T05:39:58.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:58.568+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:58.569584+0000) 2019-10-13T05:39:59.174 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:39:59.169+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.170969+0000) 2019-10-13T05:39:59.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:39:59.169+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.171055+0000) 2019-10-13T05:39:59.175 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:39:59.169+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.171291+0000) 2019-10-13T05:39:59.175 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:39:59.170+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.171484+0000) 2019-10-13T05:39:59.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:39:59.170+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.171761+0000) 2019-10-13T05:39:59.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:39:59.165+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.166193+0000) 2019-10-13T05:39:59.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:39:59.167+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.167878+0000) 2019-10-13T05:39:59.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:39:59.170+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:39:59.172001+0000) 2019-10-13T05:39:59.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:39:59.172+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:00.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:00.467+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:00.468650+0000) 2019-10-13T05:40:01.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:01.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:01.823131+0000) 2019-10-13T05:40:02.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:02.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:02.690513+0000) 2019-10-13T05:40:02.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:02.689+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:02.690206+0000) 2019-10-13T05:40:04.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:04.174+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.175544+0000) 2019-10-13T05:40:04.179 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:04.174+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.175850+0000) 2019-10-13T05:40:04.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:04.170+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.171112+0000) 2019-10-13T05:40:04.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:04.174+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.176274+0000) 2019-10-13T05:40:04.180 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:04.175+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.176406+0000) 2019-10-13T05:40:04.181 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:04.175+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.176708+0000) 2019-10-13T05:40:04.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:04.175+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.177034+0000) 2019-10-13T05:40:04.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:04.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:04.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:04.173+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.174511+0000) 2019-10-13T05:40:04.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:04.183+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:04.184972+0000) 2019-10-13T05:40:08.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:08.229+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:08.229740+0000) 2019-10-13T05:40:08.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:08.569+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:08.569683+0000) 2019-10-13T05:40:08.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:08.569+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:08.569766+0000) 2019-10-13T05:40:09.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:09.179+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.180410+0000) 2019-10-13T05:40:09.184 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:09.179+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.180650+0000) 2019-10-13T05:40:09.184 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:09.179+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.180980+0000) 2019-10-13T05:40:09.185 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:09.180+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.181428+0000) 2019-10-13T05:40:09.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:09.180+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.181538+0000) 2019-10-13T05:40:09.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:09.176+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.176768+0000) 2019-10-13T05:40:09.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:09.180+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.181962+0000) 2019-10-13T05:40:09.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:09.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82307 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:09.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:09.178+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:09.179225+0000) 2019-10-13T05:40:10.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:10.471+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:10.471815+0000) 2019-10-13T05:40:11.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:11.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:11.823336+0000) 2019-10-13T05:40:12.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:12.690+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:12.690683+0000) 2019-10-13T05:40:12.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:12.689+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:12.690388+0000) 2019-10-13T05:40:14.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:14.184+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.185015+0000) 2019-10-13T05:40:14.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:14.184+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.185082+0000) 2019-10-13T05:40:14.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:14.184+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.185027+0000) 2019-10-13T05:40:14.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:14.185+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.185797+0000) 2019-10-13T05:40:14.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:14.179+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.180615+0000) 2019-10-13T05:40:14.191 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:14.185+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.186014+0000) 2019-10-13T05:40:14.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:14.185+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.186143+0000) 2019-10-13T05:40:14.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:14.185+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.186332+0000) 2019-10-13T05:40:14.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:14.181+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:14.182478+0000) 2019-10-13T05:40:14.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:14.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82339 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:15.025 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:40:15.026 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:15.031 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:40:15.031 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:18.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:18.229+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:18.229865+0000) 2019-10-13T05:40:18.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:18.569+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:18.569858+0000) 2019-10-13T05:40:18.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:18.569+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:18.569925+0000) 2019-10-13T05:40:19.194 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:19.190+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.190532+0000) 2019-10-13T05:40:19.194 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:19.190+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.190782+0000) 2019-10-13T05:40:19.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:19.190+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.190930+0000) 2019-10-13T05:40:19.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:19.190+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.191103+0000) 2019-10-13T05:40:19.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:19.190+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.191157+0000) 2019-10-13T05:40:19.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:19.190+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.191354+0000) 2019-10-13T05:40:19.195 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:19.185+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.186502+0000) 2019-10-13T05:40:19.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:19.187+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:19.188185+0000) 2019-10-13T05:40:19.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:19.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:20.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:20.473+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:20.474227+0000) 2019-10-13T05:40:21.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:21.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:21.823513+0000) 2019-10-13T05:40:22.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:22.690+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:22.690817+0000) 2019-10-13T05:40:22.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:22.690+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:22.690589+0000) 2019-10-13T05:40:24.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:24.184+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.185228+0000) 2019-10-13T05:40:24.198 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:24.195+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.195438+0000) 2019-10-13T05:40:24.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:24.195+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.195961+0000) 2019-10-13T05:40:24.199 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:24.195+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.196045+0000) 2019-10-13T05:40:24.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:24.195+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.196284+0000) 2019-10-13T05:40:24.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:24.195+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.196322+0000) 2019-10-13T05:40:24.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:24.196+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.196662+0000) 2019-10-13T05:40:24.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:24.191+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.191786+0000) 2019-10-13T05:40:24.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:24.191+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:24.192710+0000) 2019-10-13T05:40:24.201 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:24.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:28.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:28.229+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:28.230039+0000) 2019-10-13T05:40:28.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:28.569+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:28.570044+0000) 2019-10-13T05:40:28.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:28.569+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:28.570111+0000) 2019-10-13T05:40:29.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:29.199+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.200365+0000) 2019-10-13T05:40:29.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:29.194+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.195530+0000) 2019-10-13T05:40:29.205 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:29.200+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.200738+0000) 2019-10-13T05:40:29.205 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:29.200+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.200842+0000) 2019-10-13T05:40:29.205 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:29.200+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.201229+0000) 2019-10-13T05:40:29.205 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:29.200+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.201185+0000) 2019-10-13T05:40:29.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:29.200+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.201358+0000) 2019-10-13T05:40:29.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:29.196+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:29.197300+0000) 2019-10-13T05:40:29.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:29.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:30.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:30.476+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:30.476677+0000) 2019-10-13T05:40:31.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:31.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:31.823670+0000) 2019-10-13T05:40:32.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:32.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:32.690967+0000) 2019-10-13T05:40:32.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:32.689+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:32.690780+0000) 2019-10-13T05:40:34.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:34.184+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.185442+0000) 2019-10-13T05:40:34.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:34.204+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.205698+0000) 2019-10-13T05:40:34.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:34.204+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.205568+0000) 2019-10-13T05:40:34.210 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:34.199+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.200747+0000) 2019-10-13T05:40:34.210 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:34.204+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.206151+0000) 2019-10-13T05:40:34.210 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:34.205+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.206516+0000) 2019-10-13T05:40:34.210 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:34.204+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.206389+0000) 2019-10-13T05:40:34.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:34.205+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.206666+0000) 2019-10-13T05:40:34.212 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:34.207+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:34.212 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:34.202+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:34.204106+0000) 2019-10-13T05:40:38.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:38.228+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:38.230220+0000) 2019-10-13T05:40:38.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:38.569+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:38.570178+0000) 2019-10-13T05:40:38.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:38.569+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:38.570237+0000) 2019-10-13T05:40:39.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:39.209+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.210493+0000) 2019-10-13T05:40:39.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:39.209+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.210718+0000) 2019-10-13T05:40:39.214 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:39.209+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.210821+0000) 2019-10-13T05:40:39.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:39.204+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.206534+0000) 2019-10-13T05:40:39.215 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:39.209+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.211108+0000) 2019-10-13T05:40:39.215 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:39.209+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.211207+0000) 2019-10-13T05:40:39.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:39.209+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.211387+0000) 2019-10-13T05:40:39.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:39.206+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:39.208186+0000) 2019-10-13T05:40:39.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:39.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:40.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:40.478+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:40.479089+0000) 2019-10-13T05:40:41.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:41.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:41.823885+0000) 2019-10-13T05:40:42.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:42.690+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:42.691139+0000) 2019-10-13T05:40:42.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:42.690+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:42.690995+0000) 2019-10-13T05:40:44.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:44.184+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.185588+0000) 2019-10-13T05:40:44.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:44.213+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.214653+0000) 2019-10-13T05:40:44.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:44.213+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.215148+0000) 2019-10-13T05:40:44.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:44.213+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.215285+0000) 2019-10-13T05:40:44.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:44.213+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.215284+0000) 2019-10-13T05:40:44.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:44.214+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.215469+0000) 2019-10-13T05:40:44.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:44.209+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.210744+0000) 2019-10-13T05:40:44.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:44.214+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.216143+0000) 2019-10-13T05:40:44.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:44.210+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:44.212669+0000) 2019-10-13T05:40:44.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:44.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:45.232 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:40:45.232 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:45.236 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:40:45.236 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:48.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:48.229+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:48.230463+0000) 2019-10-13T05:40:48.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:48.569+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:48.570343+0000) 2019-10-13T05:40:48.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:48.569+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:48.570394+0000) 2019-10-13T05:40:49.223 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:49.218+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.220157+0000) 2019-10-13T05:40:49.224 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:49.214+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.215298+0000) 2019-10-13T05:40:49.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:49.219+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.220669+0000) 2019-10-13T05:40:49.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:49.219+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.220929+0000) 2019-10-13T05:40:49.225 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:49.219+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.220777+0000) 2019-10-13T05:40:49.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:49.219+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.221164+0000) 2019-10-13T05:40:49.225 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:49.220+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.221561+0000) 2019-10-13T05:40:49.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:49.217+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:49.218435+0000) 2019-10-13T05:40:49.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:49.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82555 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:50.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:50.480+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:50.481456+0000) 2019-10-13T05:40:51.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:51.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:51.824082+0000) 2019-10-13T05:40:52.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:52.689+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:52.691309+0000) 2019-10-13T05:40:52.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:52.690+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:52.691173+0000) 2019-10-13T05:40:54.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:54.185+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.185785+0000) 2019-10-13T05:40:54.228 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:54.224+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.225489+0000) 2019-10-13T05:40:54.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:54.219+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.220532+0000) 2019-10-13T05:40:54.229 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:54.224+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.225871+0000) 2019-10-13T05:40:54.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:54.224+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.226129+0000) 2019-10-13T05:40:54.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:54.224+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.226379+0000) 2019-10-13T05:40:54.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:54.225+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.226588+0000) 2019-10-13T05:40:54.230 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:54.225+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.226796+0000) 2019-10-13T05:40:54.231 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:54.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:40:54.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:54.223+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:54.223800+0000) 2019-10-13T05:40:58.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:58.230+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:58.230655+0000) 2019-10-13T05:40:58.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:58.570+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:58.570545+0000) 2019-10-13T05:40:58.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:58.570+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:58.570620+0000) 2019-10-13T05:40:59.234 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:40:59.229+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.230701+0000) 2019-10-13T05:40:59.234 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:40:59.229+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.231015+0000) 2019-10-13T05:40:59.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:40:59.229+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.231144+0000) 2019-10-13T05:40:59.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:40:59.225+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.226307+0000) 2019-10-13T05:40:59.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:40:59.230+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.231472+0000) 2019-10-13T05:40:59.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:40:59.230+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.231667+0000) 2019-10-13T05:40:59.236 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:40:59.230+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.231699+0000) 2019-10-13T05:40:59.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:40:59.227+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:40:59.227899+0000) 2019-10-13T05:40:59.237 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:40:59.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82616 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:00.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:00.483+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:00.484375+0000) 2019-10-13T05:41:01.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:01.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:01.824270+0000) 2019-10-13T05:41:02.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:02.691+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:02.691500+0000) 2019-10-13T05:41:02.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:02.690+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:02.691377+0000) 2019-10-13T05:41:04.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:04.185+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.185988+0000) 2019-10-13T05:41:04.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:04.234+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.235254+0000) 2019-10-13T05:41:04.239 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:04.235+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.235465+0000) 2019-10-13T05:41:04.239 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:04.235+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.235697+0000) 2019-10-13T05:41:04.239 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:04.235+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.235875+0000) 2019-10-13T05:41:04.240 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:04.230+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.231296+0000) 2019-10-13T05:41:04.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:04.236+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.236499+0000) 2019-10-13T05:41:04.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:04.236+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.236769+0000) 2019-10-13T05:41:04.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:04.232+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:04.233226+0000) 2019-10-13T05:41:04.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:04.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82648 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:08.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:08.230+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:08.230815+0000) 2019-10-13T05:41:08.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:08.570+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:08.570740+0000) 2019-10-13T05:41:08.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:08.570+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:08.570779+0000) 2019-10-13T05:41:09.244 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:09.240+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.240783+0000) 2019-10-13T05:41:09.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:09.240+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.241001+0000) 2019-10-13T05:41:09.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:09.240+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.241178+0000) 2019-10-13T05:41:09.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:09.235+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.236267+0000) 2019-10-13T05:41:09.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:09.241+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.242052+0000) 2019-10-13T05:41:09.246 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:09.241+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.242311+0000) 2019-10-13T05:41:09.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:09.242+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.242660+0000) 2019-10-13T05:41:09.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:09.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82679 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:09.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:09.239+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:09.240720+0000) 2019-10-13T05:41:10.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:10.486+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:10.486871+0000) 2019-10-13T05:41:11.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:11.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:11.824477+0000) 2019-10-13T05:41:12.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:12.691+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:12.691676+0000) 2019-10-13T05:41:12.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:12.690+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:12.691566+0000) 2019-10-13T05:41:14.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:14.186+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.186135+0000) 2019-10-13T05:41:14.249 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:14.245+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.245519+0000) 2019-10-13T05:41:14.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:14.245+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.245688+0000) 2019-10-13T05:41:14.249 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:14.245+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.245983+0000) 2019-10-13T05:41:14.250 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:14.240+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.240815+0000) 2019-10-13T05:41:14.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:14.245+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.246253+0000) 2019-10-13T05:41:14.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:14.246+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.246418+0000) 2019-10-13T05:41:14.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:14.246+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.246774+0000) 2019-10-13T05:41:14.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:14.248+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82711 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:14.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:14.243+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:14.243781+0000) 2019-10-13T05:41:15.455 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:41:15.455 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:15.459 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:41:15.459 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:18.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:18.229+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:18.230978+0000) 2019-10-13T05:41:18.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:18.569+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:18.570932+0000) 2019-10-13T05:41:18.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:18.569+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:18.570989+0000) 2019-10-13T05:41:19.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:19.249+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.250409+0000) 2019-10-13T05:41:19.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:19.249+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.250850+0000) 2019-10-13T05:41:19.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:19.249+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.250810+0000) 2019-10-13T05:41:19.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:19.245+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.246336+0000) 2019-10-13T05:41:19.255 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:19.250+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.251586+0000) 2019-10-13T05:41:19.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:19.250+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.251520+0000) 2019-10-13T05:41:19.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:19.250+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.251730+0000) 2019-10-13T05:41:19.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:19.246+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:19.247342+0000) 2019-10-13T05:41:19.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:19.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82740 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:20.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:20.488+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:20.489049+0000) 2019-10-13T05:41:21.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:21.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:21.824667+0000) 2019-10-13T05:41:22.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:22.690+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:22.691839+0000) 2019-10-13T05:41:22.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:22.690+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:22.691798+0000) 2019-10-13T05:41:24.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:24.185+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.186290+0000) 2019-10-13T05:41:24.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:24.254+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.256306+0000) 2019-10-13T05:41:24.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:24.255+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.256605+0000) 2019-10-13T05:41:24.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:24.255+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.256486+0000) 2019-10-13T05:41:24.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:24.255+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.256879+0000) 2019-10-13T05:41:24.261 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:24.250+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.252149+0000) 2019-10-13T05:41:24.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:24.251+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.253200+0000) 2019-10-13T05:41:24.262 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:24.255+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.257346+0000) 2019-10-13T05:41:24.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:24.256+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:24.257764+0000) 2019-10-13T05:41:24.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:24.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82772 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:28.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:28.230+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:28.231109+0000) 2019-10-13T05:41:28.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:28.569+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:28.571144+0000) 2019-10-13T05:41:28.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:28.569+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:28.571154+0000) 2019-10-13T05:41:29.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:29.258+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.260263+0000) 2019-10-13T05:41:29.264 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:29.259+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.260413+0000) 2019-10-13T05:41:29.264 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:29.259+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.260703+0000) 2019-10-13T05:41:29.264 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:29.259+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.260614+0000) 2019-10-13T05:41:29.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:29.254+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.256268+0000) 2019-10-13T05:41:29.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:29.259+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.260968+0000) 2019-10-13T05:41:29.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:29.259+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.260924+0000) 2019-10-13T05:41:29.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:29.261+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82803 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:29.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:29.256+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:29.258470+0000) 2019-10-13T05:41:30.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:30.490+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:30.491289+0000) 2019-10-13T05:41:31.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:31.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:31.824827+0000) 2019-10-13T05:41:32.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:32.691+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:32.691982+0000) 2019-10-13T05:41:32.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:32.690+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:32.691972+0000) 2019-10-13T05:41:34.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:34.185+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.186448+0000) 2019-10-13T05:41:34.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:34.263+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.264945+0000) 2019-10-13T05:41:34.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:34.263+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.264838+0000) 2019-10-13T05:41:34.268 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:34.263+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.265268+0000) 2019-10-13T05:41:34.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:34.259+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.261186+0000) 2019-10-13T05:41:34.270 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:34.264+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.265630+0000) 2019-10-13T05:41:34.270 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:34.264+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.265706+0000) 2019-10-13T05:41:34.270 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:34.265+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.266418+0000) 2019-10-13T05:41:34.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:34.260+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:34.262489+0000) 2019-10-13T05:41:34.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:34.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82835 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:38.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:38.230+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:38.231305+0000) 2019-10-13T05:41:38.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:38.570+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:38.571329+0000) 2019-10-13T05:41:38.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:38.570+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:38.571329+0000) 2019-10-13T05:41:39.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:39.264+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.265130+0000) 2019-10-13T05:41:39.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:39.269+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.270505+0000) 2019-10-13T05:41:39.274 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:39.269+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.270507+0000) 2019-10-13T05:41:39.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:39.269+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.270710+0000) 2019-10-13T05:41:39.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:39.269+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.270809+0000) 2019-10-13T05:41:39.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:39.269+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.271116+0000) 2019-10-13T05:41:39.275 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:39.270+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.271511+0000) 2019-10-13T05:41:39.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:39.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:39.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:39.268+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:39.268954+0000) 2019-10-13T05:41:40.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:40.492+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:40.493570+0000) 2019-10-13T05:41:41.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:41.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:41.824958+0000) 2019-10-13T05:41:42.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:42.691+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:42.692146+0000) 2019-10-13T05:41:42.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:42.691+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:42.692103+0000) 2019-10-13T05:41:44.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:44.186+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.186637+0000) 2019-10-13T05:41:44.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:44.269+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.270278+0000) 2019-10-13T05:41:44.279 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:44.274+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.275634+0000) 2019-10-13T05:41:44.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:44.274+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.275692+0000) 2019-10-13T05:41:44.280 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:44.274+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.276080+0000) 2019-10-13T05:41:44.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:44.275+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.276407+0000) 2019-10-13T05:41:44.280 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:44.275+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.276563+0000) 2019-10-13T05:41:44.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:44.275+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.276738+0000) 2019-10-13T05:41:44.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:44.272+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:44.273015+0000) 2019-10-13T05:41:44.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:44.277+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82896 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:45.642 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:41:45.643 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:45.647 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:41:45.647 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:48.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:48.231+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:48.231481+0000) 2019-10-13T05:41:48.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:48.571+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:48.571504+0000) 2019-10-13T05:41:48.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:48.571+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:48.571508+0000) 2019-10-13T05:41:49.284 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:49.280+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.281316+0000) 2019-10-13T05:41:49.285 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:49.275+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.276679+0000) 2019-10-13T05:41:49.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:49.281+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.281858+0000) 2019-10-13T05:41:49.285 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:49.281+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.282100+0000) 2019-10-13T05:41:49.286 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:49.281+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.282228+0000) 2019-10-13T05:41:49.286 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:49.282+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.282613+0000) 2019-10-13T05:41:49.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:49.282+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.283014+0000) 2019-10-13T05:41:49.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:49.277+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:49.278628+0000) 2019-10-13T05:41:49.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:49.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:50.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:50.495+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:50.495968+0000) 2019-10-13T05:41:51.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:51.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:51.825148+0000) 2019-10-13T05:41:52.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:52.691+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:52.692350+0000) 2019-10-13T05:41:52.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:52.691+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:52.692255+0000) 2019-10-13T05:41:54.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:54.186+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.186822+0000) 2019-10-13T05:41:54.292 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:54.281+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.282077+0000) 2019-10-13T05:41:54.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:54.285+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.286341+0000) 2019-10-13T05:41:54.293 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:54.286+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.286550+0000) 2019-10-13T05:41:54.293 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:54.286+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.286675+0000) 2019-10-13T05:41:54.293 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:54.286+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.287253+0000) 2019-10-13T05:41:54.293 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:54.287+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.287511+0000) 2019-10-13T05:41:54.293 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:54.287+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.287751+0000) 2019-10-13T05:41:54.294 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:54.283+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:54.283983+0000) 2019-10-13T05:41:54.294 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:54.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:58.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:58.231+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:58.231641+0000) 2019-10-13T05:41:58.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:58.571+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:58.571704+0000) 2019-10-13T05:41:58.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:58.571+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:58.571727+0000) 2019-10-13T05:41:59.296 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:41:59.292+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.292447+0000) 2019-10-13T05:41:59.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:41:59.292+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.292606+0000) 2019-10-13T05:41:59.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:41:59.292+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.292868+0000) 2019-10-13T05:41:59.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:41:59.292+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.293053+0000) 2019-10-13T05:41:59.297 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:41:59.292+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.293221+0000) 2019-10-13T05:41:59.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:41:59.287+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.287747+0000) 2019-10-13T05:41:59.298 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:41:59.293+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.293522+0000) 2019-10-13T05:41:59.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:41:59.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 82988 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:41:59.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:41:59.289+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:41:59.290486+0000) 2019-10-13T05:42:00.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:00.498+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:00.498476+0000) 2019-10-13T05:42:01.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:01.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:01.825323+0000) 2019-10-13T05:42:02.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:02.692+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:02.692551+0000) 2019-10-13T05:42:02.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:02.691+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:02.692379+0000) 2019-10-13T05:42:04.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:04.185+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.187031+0000) 2019-10-13T05:42:04.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:04.295+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.296812+0000) 2019-10-13T05:42:04.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:04.295+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.296916+0000) 2019-10-13T05:42:04.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:04.291+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.292467+0000) 2019-10-13T05:42:04.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:04.296+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.297472+0000) 2019-10-13T05:42:04.302 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:04.296+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.297713+0000) 2019-10-13T05:42:04.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:04.296+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.297879+0000) 2019-10-13T05:42:04.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:04.296+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.298108+0000) 2019-10-13T05:42:04.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:04.293+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:04.294201+0000) 2019-10-13T05:42:04.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:04.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83020 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:08.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:08.230+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:08.231787+0000) 2019-10-13T05:42:08.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:08.570+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:08.571893+0000) 2019-10-13T05:42:08.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:08.570+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:08.571894+0000) 2019-10-13T05:42:09.305 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:09.300+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.301870+0000) 2019-10-13T05:42:09.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:09.300+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.301932+0000) 2019-10-13T05:42:09.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:09.301+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.302461+0000) 2019-10-13T05:42:09.306 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:09.301+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.302612+0000) 2019-10-13T05:42:09.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:09.301+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.302898+0000) 2019-10-13T05:42:09.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:09.301+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.302997+0000) 2019-10-13T05:42:09.307 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:09.297+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.298159+0000) 2019-10-13T05:42:09.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:09.298+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:09.299431+0000) 2019-10-13T05:42:09.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:09.303+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:10.509 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:10.500+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:10.500725+0000) 2019-10-13T05:42:11.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:11.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:11.825485+0000) 2019-10-13T05:42:12.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:12.691+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:12.692743+0000) 2019-10-13T05:42:12.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:12.691+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:12.692578+0000) 2019-10-13T05:42:14.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:14.186+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.187210+0000) 2019-10-13T05:42:14.309 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:14.305+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.306436+0000) 2019-10-13T05:42:14.310 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:14.305+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.306598+0000) 2019-10-13T05:42:14.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:14.305+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.306931+0000) 2019-10-13T05:42:14.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:14.305+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.307093+0000) 2019-10-13T05:42:14.311 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:14.301+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.302573+0000) 2019-10-13T05:42:14.311 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:14.306+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.307670+0000) 2019-10-13T05:42:14.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:14.306+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.307893+0000) 2019-10-13T05:42:14.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:14.303+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:14.304444+0000) 2019-10-13T05:42:14.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:14.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:15.832 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:42:15.832 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:15.837 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:42:15.837 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:18.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:18.230+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:18.231976+0000) 2019-10-13T05:42:18.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:18.571+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:18.572063+0000) 2019-10-13T05:42:18.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:18.571+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:18.572063+0000) 2019-10-13T05:42:19.315 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:19.310+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.312164+0000) 2019-10-13T05:42:19.316 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:19.311+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.312566+0000) 2019-10-13T05:42:19.317 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:19.306+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.308241+0000) 2019-10-13T05:42:19.317 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:19.311+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.312842+0000) 2019-10-13T05:42:19.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:19.311+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.312809+0000) 2019-10-13T05:42:19.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:19.311+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.313167+0000) 2019-10-13T05:42:19.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:19.312+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.313653+0000) 2019-10-13T05:42:19.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:19.308+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:19.310492+0000) 2019-10-13T05:42:19.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:19.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:20.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:20.502+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:20.502969+0000) 2019-10-13T05:42:21.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:21.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:21.825635+0000) 2019-10-13T05:42:22.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:22.691+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:22.692914+0000) 2019-10-13T05:42:22.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:22.691+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:22.692792+0000) 2019-10-13T05:42:24.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:24.186+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.187425+0000) 2019-10-13T05:42:24.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:24.310+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.312458+0000) 2019-10-13T05:42:24.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:24.316+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.317640+0000) 2019-10-13T05:42:24.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:24.316+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.318128+0000) 2019-10-13T05:42:24.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:24.316+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.318362+0000) 2019-10-13T05:42:24.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:24.317+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.318559+0000) 2019-10-13T05:42:24.323 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:24.317+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.318777+0000) 2019-10-13T05:42:24.323 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:24.317+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.318974+0000) 2019-10-13T05:42:24.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:24.313+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:24.315073+0000) 2019-10-13T05:42:24.324 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:24.319+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:28.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:28.230+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:28.232144+0000) 2019-10-13T05:42:28.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:28.571+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:28.572256+0000) 2019-10-13T05:42:28.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:28.571+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:28.572255+0000) 2019-10-13T05:42:29.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:29.321+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.322830+0000) 2019-10-13T05:42:29.326 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:29.317+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.318193+0000) 2019-10-13T05:42:29.327 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:29.321+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.323342+0000) 2019-10-13T05:42:29.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:29.322+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.323721+0000) 2019-10-13T05:42:29.327 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:29.322+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.324178+0000) 2019-10-13T05:42:29.328 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:29.322+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.324293+0000) 2019-10-13T05:42:29.328 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:29.323+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.324509+0000) 2019-10-13T05:42:29.329 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:29.320+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:29.321173+0000) 2019-10-13T05:42:29.330 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:29.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83176 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:30.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:30.504+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:30.505358+0000) 2019-10-13T05:42:31.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:31.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:31.825828+0000) 2019-10-13T05:42:32.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:32.692+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:32.693066+0000) 2019-10-13T05:42:32.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:32.692+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:32.692994+0000) 2019-10-13T05:42:34.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:34.187+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.187535+0000) 2019-10-13T05:42:34.332 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:34.322+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.323721+0000) 2019-10-13T05:42:34.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:34.328+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.328892+0000) 2019-10-13T05:42:34.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:34.328+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.329168+0000) 2019-10-13T05:42:34.333 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:34.329+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.329439+0000) 2019-10-13T05:42:34.333 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:34.329+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.329628+0000) 2019-10-13T05:42:34.333 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:34.329+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.329923+0000) 2019-10-13T05:42:34.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:34.329+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.330184+0000) 2019-10-13T05:42:34.335 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:34.331+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:34.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:34.326+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:34.327660+0000) 2019-10-13T05:42:38.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:38.232+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:38.232348+0000) 2019-10-13T05:42:38.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:38.572+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:38.572468+0000) 2019-10-13T05:42:38.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:38.572+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:38.572465+0000) 2019-10-13T05:42:39.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:39.333+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.334271+0000) 2019-10-13T05:42:39.338 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:39.334+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.334632+0000) 2019-10-13T05:42:39.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:39.334+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.334781+0000) 2019-10-13T05:42:39.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:39.329+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.330069+0000) 2019-10-13T05:42:39.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:39.334+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.334926+0000) 2019-10-13T05:42:39.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:39.334+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.335241+0000) 2019-10-13T05:42:39.340 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:39.335+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.335715+0000) 2019-10-13T05:42:39.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:39.337+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83238 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:39.341 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:39.332+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:39.333443+0000) 2019-10-13T05:42:40.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:40.507+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:40.507815+0000) 2019-10-13T05:42:41.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:41.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:41.826067+0000) 2019-10-13T05:42:42.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:42.693+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:42.693236+0000) 2019-10-13T05:42:42.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:42.692+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:42.693120+0000) 2019-10-13T05:42:44.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:44.187+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.187690+0000) 2019-10-13T05:42:44.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:44.333+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.334354+0000) 2019-10-13T05:42:44.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:44.338+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.339137+0000) 2019-10-13T05:42:44.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:44.339+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.339780+0000) 2019-10-13T05:42:44.344 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:44.339+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.340105+0000) 2019-10-13T05:42:44.344 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:44.339+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.340218+0000) 2019-10-13T05:42:44.344 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:44.339+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.340355+0000) 2019-10-13T05:42:44.344 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:44.340+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.340784+0000) 2019-10-13T05:42:44.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:44.335+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:44.336594+0000) 2019-10-13T05:42:44.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:44.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83271 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:46.029 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:42:46.029 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:46.033 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:42:46.034 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:48.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:48.232+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:48.232523+0000) 2019-10-13T05:42:48.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:48.571+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:48.572667+0000) 2019-10-13T05:42:48.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:48.571+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:48.572665+0000) 2019-10-13T05:42:49.347 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:49.343+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.344586+0000) 2019-10-13T05:42:49.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:49.339+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.340098+0000) 2019-10-13T05:42:49.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:49.343+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.345304+0000) 2019-10-13T05:42:49.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:49.344+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.345436+0000) 2019-10-13T05:42:49.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:49.344+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.345727+0000) 2019-10-13T05:42:49.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:49.344+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.345830+0000) 2019-10-13T05:42:49.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:49.344+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.346139+0000) 2019-10-13T05:42:49.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:49.341+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:49.342699+0000) 2019-10-13T05:42:49.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:49.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:50.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:50.509+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:50.510135+0000) 2019-10-13T05:42:51.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:51.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:51.826262+0000) 2019-10-13T05:42:52.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:52.692+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:52.693430+0000) 2019-10-13T05:42:52.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:52.692+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:52.693270+0000) 2019-10-13T05:42:54.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:54.186+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.187882+0000) 2019-10-13T05:42:54.353 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:54.348+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.349913+0000) 2019-10-13T05:42:54.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:54.348+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.350244+0000) 2019-10-13T05:42:54.354 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:54.348+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.350346+0000) 2019-10-13T05:42:54.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:54.344+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.345483+0000) 2019-10-13T05:42:54.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:54.349+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.350812+0000) 2019-10-13T05:42:54.355 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:54.349+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.351205+0000) 2019-10-13T05:42:54.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:54.350+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.351507+0000) 2019-10-13T05:42:54.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:54.347+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:54.348024+0000) 2019-10-13T05:42:54.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:54.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:42:58.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:58.231+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:58.232679+0000) 2019-10-13T05:42:58.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:58.572+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:58.572830+0000) 2019-10-13T05:42:58.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:58.572+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:58.572831+0000) 2019-10-13T05:42:59.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:42:59.354+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.355429+0000) 2019-10-13T05:42:59.359 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:42:59.354+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.355723+0000) 2019-10-13T05:42:59.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:42:59.350+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.351286+0000) 2019-10-13T05:42:59.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:42:59.355+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.356490+0000) 2019-10-13T05:42:59.361 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:42:59.355+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.356767+0000) 2019-10-13T05:42:59.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:42:59.355+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.356938+0000) 2019-10-13T05:42:59.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:42:59.355+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.357218+0000) 2019-10-13T05:42:59.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:42:59.352+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:42:59.352945+0000) 2019-10-13T05:42:59.363 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:42:59.357+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83364 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:00.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:00.511+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:00.512128+0000) 2019-10-13T05:43:01.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:01.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:01.826430+0000) 2019-10-13T05:43:02.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:02.692+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:02.693588+0000) 2019-10-13T05:43:02.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:02.693+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:02.693459+0000) 2019-10-13T05:43:04.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:04.187+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.188043+0000) 2019-10-13T05:43:04.364 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:04.359+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.361045+0000) 2019-10-13T05:43:04.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:04.359+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.361297+0000) 2019-10-13T05:43:04.365 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:04.355+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.356193+0000) 2019-10-13T05:43:04.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:04.360+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.361600+0000) 2019-10-13T05:43:04.366 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:04.360+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.361617+0000) 2019-10-13T05:43:04.366 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:04.360+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.362058+0000) 2019-10-13T05:43:04.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:04.360+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.362380+0000) 2019-10-13T05:43:04.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:04.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83397 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:04.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:04.358+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:04.359645+0000) 2019-10-13T05:43:08.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:08.232+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:08.232847+0000) 2019-10-13T05:43:08.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:08.572+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:08.572997+0000) 2019-10-13T05:43:08.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:08.572+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:08.573000+0000) 2019-10-13T05:43:09.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:09.357+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.359655+0000) 2019-10-13T05:43:09.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:09.363+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.364996+0000) 2019-10-13T05:43:09.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:09.363+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.365338+0000) 2019-10-13T05:43:09.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:09.363+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.365240+0000) 2019-10-13T05:43:09.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:09.364+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.365531+0000) 2019-10-13T05:43:09.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:09.364+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.365938+0000) 2019-10-13T05:43:09.370 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:09.364+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.366036+0000) 2019-10-13T05:43:09.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:09.361+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:09.363424+0000) 2019-10-13T05:43:09.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:09.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:10.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:10.512+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:10.514631+0000) 2019-10-13T05:43:11.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:11.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:11.826594+0000) 2019-10-13T05:43:12.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:12.693+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:12.693799+0000) 2019-10-13T05:43:12.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:12.691+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:12.693630+0000) 2019-10-13T05:43:14.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:14.187+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.188220+0000) 2019-10-13T05:43:14.373 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:14.363+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.365232+0000) 2019-10-13T05:43:14.374 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:14.368+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.370385+0000) 2019-10-13T05:43:14.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:14.369+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.370773+0000) 2019-10-13T05:43:14.374 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:14.369+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.371086+0000) 2019-10-13T05:43:14.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:14.369+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.371327+0000) 2019-10-13T05:43:14.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:14.370+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.371511+0000) 2019-10-13T05:43:14.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:14.365+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.367661+0000) 2019-10-13T05:43:14.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:14.370+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:14.371693+0000) 2019-10-13T05:43:14.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:14.372+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:16.237 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:43:16.238 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:16.242 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:43:16.242 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:18.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:18.232+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:18.232999+0000) 2019-10-13T05:43:18.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:18.572+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:18.573219+0000) 2019-10-13T05:43:18.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:18.572+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:18.573220+0000) 2019-10-13T05:43:19.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:19.369+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.370832+0000) 2019-10-13T05:43:19.380 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:19.375+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.376014+0000) 2019-10-13T05:43:19.380 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:19.375+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.376187+0000) 2019-10-13T05:43:19.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:19.375+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.376373+0000) 2019-10-13T05:43:19.380 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:19.376+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.376521+0000) 2019-10-13T05:43:19.380 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:19.376+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.376857+0000) 2019-10-13T05:43:19.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:19.376+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.377259+0000) 2019-10-13T05:43:19.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:19.372+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:19.374279+0000) 2019-10-13T05:43:19.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:19.379+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83490 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:20.525 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:20.516+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:20.516873+0000) 2019-10-13T05:43:21.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:21.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:21.826760+0000) 2019-10-13T05:43:22.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:22.693+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:22.694010+0000) 2019-10-13T05:43:22.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:22.693+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:22.693812+0000) 2019-10-13T05:43:24.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:24.188+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.188384+0000) 2019-10-13T05:43:24.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:24.380+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.380925+0000) 2019-10-13T05:43:24.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:24.380+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.381092+0000) 2019-10-13T05:43:24.385 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:24.380+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.381314+0000) 2019-10-13T05:43:24.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:24.381+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.381449+0000) 2019-10-13T05:43:24.385 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:24.381+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.381608+0000) 2019-10-13T05:43:24.386 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:24.376+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.376817+0000) 2019-10-13T05:43:24.386 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:24.381+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.381956+0000) 2019-10-13T05:43:24.386 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:24.377+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:24.378353+0000) 2019-10-13T05:43:24.387 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:24.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83523 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:28.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:28.233+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:28.233194+0000) 2019-10-13T05:43:28.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:28.573+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:28.573419+0000) 2019-10-13T05:43:28.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:28.573+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:28.573429+0000) 2019-10-13T05:43:29.389 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:29.384+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.385071+0000) 2019-10-13T05:43:29.390 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:29.385+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.385621+0000) 2019-10-13T05:43:29.390 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:29.385+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.385780+0000) 2019-10-13T05:43:29.390 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:29.385+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.386055+0000) 2019-10-13T05:43:29.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:29.385+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.386244+0000) 2019-10-13T05:43:29.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:29.380+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.381652+0000) 2019-10-13T05:43:29.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:29.382+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.383007+0000) 2019-10-13T05:43:29.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:29.386+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:29.386765+0000) 2019-10-13T05:43:29.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:29.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:30.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:30.518+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:30.519273+0000) 2019-10-13T05:43:31.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:31.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:31.826900+0000) 2019-10-13T05:43:32.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:32.694+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:32.694167+0000) 2019-10-13T05:43:32.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:32.693+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:32.693996+0000) 2019-10-13T05:43:34.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:34.188+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.188533+0000) 2019-10-13T05:43:34.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:34.389+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.390500+0000) 2019-10-13T05:43:34.394 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:34.389+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.390824+0000) 2019-10-13T05:43:34.395 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:34.385+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.386473+0000) 2019-10-13T05:43:34.395 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:34.389+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.390877+0000) 2019-10-13T05:43:34.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:34.390+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.391565+0000) 2019-10-13T05:43:34.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:34.390+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.391908+0000) 2019-10-13T05:43:34.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:34.390+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.392144+0000) 2019-10-13T05:43:34.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:34.388+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:34.388874+0000) 2019-10-13T05:43:34.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:34.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:38.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:38.232+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:38.233392+0000) 2019-10-13T05:43:38.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:38.572+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:38.573645+0000) 2019-10-13T05:43:38.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:38.572+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:38.573643+0000) 2019-10-13T05:43:39.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:39.394+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.395776+0000) 2019-10-13T05:43:39.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:39.394+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.395956+0000) 2019-10-13T05:43:39.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:39.394+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.396065+0000) 2019-10-13T05:43:39.400 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:39.390+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.391268+0000) 2019-10-13T05:43:39.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:39.395+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.396498+0000) 2019-10-13T05:43:39.401 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:39.395+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.397227+0000) 2019-10-13T05:43:39.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:39.396+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.397486+0000) 2019-10-13T05:43:39.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:39.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83616 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:39.403 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:39.393+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:39.394115+0000) 2019-10-13T05:43:40.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:40.521+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:40.521609+0000) 2019-10-13T05:43:41.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:41.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:41.827079+0000) 2019-10-13T05:43:42.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:42.692+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:42.694329+0000) 2019-10-13T05:43:42.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:42.693+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:42.694182+0000) 2019-10-13T05:43:44.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:44.187+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.188713+0000) 2019-10-13T05:43:44.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:44.400+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.401584+0000) 2019-10-13T05:43:44.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:44.400+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.401779+0000) 2019-10-13T05:43:44.406 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:44.396+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.397261+0000) 2019-10-13T05:43:44.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:44.400+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.402330+0000) 2019-10-13T05:43:44.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:44.401+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.402629+0000) 2019-10-13T05:43:44.407 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:44.401+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.402825+0000) 2019-10-13T05:43:44.407 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:44.401+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.402987+0000) 2019-10-13T05:43:44.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:44.398+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:44.399129+0000) 2019-10-13T05:43:44.408 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:44.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:46.457 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:43:46.457 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:46.462 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:43:46.462 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:48.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:48.232+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:48.233623+0000) 2019-10-13T05:43:48.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:48.573+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:48.573840+0000) 2019-10-13T05:43:48.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:48.573+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:48.573840+0000) 2019-10-13T05:43:49.410 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:49.401+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.402092+0000) 2019-10-13T05:43:49.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:49.406+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.407502+0000) 2019-10-13T05:43:49.411 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:49.406+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.407508+0000) 2019-10-13T05:43:49.411 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:49.406+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.407722+0000) 2019-10-13T05:43:49.411 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:49.406+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.407891+0000) 2019-10-13T05:43:49.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:49.406+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.407984+0000) 2019-10-13T05:43:49.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:49.406+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.408106+0000) 2019-10-13T05:43:49.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:49.403+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:49.404321+0000) 2019-10-13T05:43:49.413 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:49.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:50.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:50.523+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:50.523903+0000) 2019-10-13T05:43:51.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:51.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:51.827241+0000) 2019-10-13T05:43:52.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:52.693+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:52.694474+0000) 2019-10-13T05:43:52.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:52.693+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:52.694360+0000) 2019-10-13T05:43:54.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:54.188+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.188876+0000) 2019-10-13T05:43:54.413 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:54.408+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.410271+0000) 2019-10-13T05:43:54.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:54.409+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.410458+0000) 2019-10-13T05:43:54.414 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:54.409+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.411185+0000) 2019-10-13T05:43:54.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:54.409+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.411329+0000) 2019-10-13T05:43:54.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:54.410+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.411550+0000) 2019-10-13T05:43:54.415 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:54.406+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.406792+0000) 2019-10-13T05:43:54.415 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:54.410+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.411850+0000) 2019-10-13T05:43:54.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:54.407+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:54.408089+0000) 2019-10-13T05:43:54.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:54.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:43:58.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:58.233+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:58.233781+0000) 2019-10-13T05:43:58.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:58.573+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:58.574035+0000) 2019-10-13T05:43:58.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:58.573+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:58.574035+0000) 2019-10-13T05:43:59.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:43:59.414+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.416244+0000) 2019-10-13T05:43:59.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:43:59.415+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.416548+0000) 2019-10-13T05:43:59.421 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:43:59.411+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.412252+0000) 2019-10-13T05:43:59.421 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:43:59.415+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.416937+0000) 2019-10-13T05:43:59.421 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:43:59.415+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.417310+0000) 2019-10-13T05:43:59.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:43:59.416+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.417652+0000) 2019-10-13T05:43:59.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:43:59.416+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.417739+0000) 2019-10-13T05:43:59.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:43:59.413+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:43:59.413919+0000) 2019-10-13T05:43:59.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:43:59.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:00.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:00.526+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:00.526299+0000) 2019-10-13T05:44:01.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:01.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:01.827393+0000) 2019-10-13T05:44:02.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:02.694+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:02.694622+0000) 2019-10-13T05:44:02.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:02.693+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:02.694564+0000) 2019-10-13T05:44:04.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:04.188+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.189048+0000) 2019-10-13T05:44:04.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:04.420+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.421323+0000) 2019-10-13T05:44:04.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:04.421+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.421565+0000) 2019-10-13T05:44:04.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:04.421+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.421707+0000) 2019-10-13T05:44:04.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:04.415+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.416978+0000) 2019-10-13T05:44:04.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:04.421+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.422099+0000) 2019-10-13T05:44:04.426 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:04.422+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.422439+0000) 2019-10-13T05:44:04.426 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:04.422+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.422728+0000) 2019-10-13T05:44:04.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:04.418+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:04.419800+0000) 2019-10-13T05:44:04.428 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:04.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:08.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:08.233+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:08.233974+0000) 2019-10-13T05:44:08.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:08.573+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:08.574257+0000) 2019-10-13T05:44:08.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:08.573+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:08.574257+0000) 2019-10-13T05:44:09.429 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:09.425+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.426295+0000) 2019-10-13T05:44:09.430 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:09.426+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.426599+0000) 2019-10-13T05:44:09.430 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:09.426+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.426796+0000) 2019-10-13T05:44:09.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:09.420+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.422198+0000) 2019-10-13T05:44:09.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:09.426+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.427292+0000) 2019-10-13T05:44:09.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:09.426+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.427384+0000) 2019-10-13T05:44:09.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:09.427+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.427768+0000) 2019-10-13T05:44:09.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:09.422+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:09.424165+0000) 2019-10-13T05:44:09.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:09.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:10.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:10.526+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:10.528592+0000) 2019-10-13T05:44:11.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:11.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:11.827652+0000) 2019-10-13T05:44:12.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:12.694+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:12.694804+0000) 2019-10-13T05:44:12.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:12.693+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:12.694780+0000) 2019-10-13T05:44:14.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:14.188+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.189172+0000) 2019-10-13T05:44:14.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:14.425+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.426801+0000) 2019-10-13T05:44:14.436 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:14.431+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.431975+0000) 2019-10-13T05:44:14.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:14.431+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.432186+0000) 2019-10-13T05:44:14.436 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:14.432+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.432451+0000) 2019-10-13T05:44:14.436 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:14.432+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.432656+0000) 2019-10-13T05:44:14.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:14.432+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.432851+0000) 2019-10-13T05:44:14.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:14.432+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.433205+0000) 2019-10-13T05:44:14.438 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:14.428+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:14.430223+0000) 2019-10-13T05:44:14.438 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:14.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:16.663 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:44:16.663 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:16.667 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:44:16.667 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:18.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:18.234+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:18.234174+0000) 2019-10-13T05:44:18.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:18.574+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:18.574443+0000) 2019-10-13T05:44:18.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:18.574+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:18.574441+0000) 2019-10-13T05:44:19.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:19.436+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.436959+0000) 2019-10-13T05:44:19.440 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:19.436+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.437269+0000) 2019-10-13T05:44:19.442 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:19.431+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.432200+0000) 2019-10-13T05:44:19.442 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:19.437+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.437785+0000) 2019-10-13T05:44:19.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:19.437+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.437894+0000) 2019-10-13T05:44:19.442 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:19.437+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.438104+0000) 2019-10-13T05:44:19.442 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:19.438+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.438512+0000) 2019-10-13T05:44:19.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:19.433+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:19.434629+0000) 2019-10-13T05:44:19.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:19.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83868 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:20.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:20.529+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:20.530706+0000) 2019-10-13T05:44:21.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:21.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:21.827791+0000) 2019-10-13T05:44:22.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:22.693+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:22.694986+0000) 2019-10-13T05:44:22.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:22.694+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:22.694959+0000) 2019-10-13T05:44:24.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:24.188+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.189374+0000) 2019-10-13T05:44:24.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:24.440+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.442173+0000) 2019-10-13T05:44:24.446 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:24.437+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.438013+0000) 2019-10-13T05:44:24.447 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:24.441+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.442721+0000) 2019-10-13T05:44:24.447 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:24.441+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.443032+0000) 2019-10-13T05:44:24.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:24.441+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.443192+0000) 2019-10-13T05:44:24.447 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:24.442+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.443401+0000) 2019-10-13T05:44:24.447 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:24.442+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.443637+0000) 2019-10-13T05:44:24.448 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:24.438+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:24.439229+0000) 2019-10-13T05:44:24.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:24.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:28.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:28.233+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:28.234407+0000) 2019-10-13T05:44:28.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:28.573+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:28.574653+0000) 2019-10-13T05:44:28.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:28.573+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:28.574649+0000) 2019-10-13T05:44:29.451 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:29.445+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.447061+0000) 2019-10-13T05:44:29.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:29.445+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.447339+0000) 2019-10-13T05:44:29.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:29.446+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.447554+0000) 2019-10-13T05:44:29.451 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:29.446+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.447963+0000) 2019-10-13T05:44:29.452 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:29.442+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.443593+0000) 2019-10-13T05:44:29.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:29.447+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.448408+0000) 2019-10-13T05:44:29.453 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:29.446+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.448278+0000) 2019-10-13T05:44:29.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:29.444+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:29.445372+0000) 2019-10-13T05:44:29.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:29.449+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:30.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:30.533+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:30.533518+0000) 2019-10-13T05:44:31.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:31.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:31.827999+0000) 2019-10-13T05:44:32.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:32.694+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:32.695174+0000) 2019-10-13T05:44:32.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:32.694+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:32.695134+0000) 2019-10-13T05:44:34.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:34.188+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.189555+0000) 2019-10-13T05:44:34.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:34.451+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.452527+0000) 2019-10-13T05:44:34.457 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:34.451+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.452671+0000) 2019-10-13T05:44:34.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:34.451+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.452807+0000) 2019-10-13T05:44:34.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:34.451+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.452946+0000) 2019-10-13T05:44:34.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:34.451+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.453096+0000) 2019-10-13T05:44:34.458 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:34.452+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.453466+0000) 2019-10-13T05:44:34.458 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:34.447+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.448366+0000) 2019-10-13T05:44:34.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:34.448+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:34.449388+0000) 2019-10-13T05:44:34.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:34.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:38.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:38.233+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:38.234622+0000) 2019-10-13T05:44:38.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:38.574+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:38.574839+0000) 2019-10-13T05:44:38.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:38.574+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:38.574840+0000) 2019-10-13T05:44:39.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:39.455+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.457000+0000) 2019-10-13T05:44:39.461 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:39.455+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.457250+0000) 2019-10-13T05:44:39.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:39.456+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.457454+0000) 2019-10-13T05:44:39.462 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:39.456+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.457524+0000) 2019-10-13T05:44:39.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:39.451+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.452141+0000) 2019-10-13T05:44:39.463 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:39.456+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.458031+0000) 2019-10-13T05:44:39.463 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:39.457+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.458453+0000) 2019-10-13T05:44:39.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:39.454+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:39.455164+0000) 2019-10-13T05:44:39.464 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:39.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 83994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:40.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:40.535+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:40.535856+0000) 2019-10-13T05:44:41.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:41.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:41.828179+0000) 2019-10-13T05:44:42.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:42.694+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:42.695359+0000) 2019-10-13T05:44:42.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:42.694+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:42.695335+0000) 2019-10-13T05:44:44.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:44.189+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.189705+0000) 2019-10-13T05:44:44.465 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:44.460+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.462227+0000) 2019-10-13T05:44:44.466 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:44.456+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.457617+0000) 2019-10-13T05:44:44.466 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:44.461+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.462677+0000) 2019-10-13T05:44:44.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:44.461+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.462839+0000) 2019-10-13T05:44:44.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:44.461+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.463012+0000) 2019-10-13T05:44:44.467 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:44.461+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.463284+0000) 2019-10-13T05:44:44.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:44.458+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.459561+0000) 2019-10-13T05:44:44.468 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:44.462+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:44.463969+0000) 2019-10-13T05:44:44.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:44.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:46.868 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:44:46.869 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:46.873 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:44:46.873 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:48.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:48.234+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:48.234794+0000) 2019-10-13T05:44:48.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:48.574+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:48.575029+0000) 2019-10-13T05:44:48.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:48.574+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:48.575027+0000) 2019-10-13T05:44:49.472 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:49.467+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.467690+0000) 2019-10-13T05:44:49.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:49.467+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.467826+0000) 2019-10-13T05:44:49.472 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:49.467+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.468054+0000) 2019-10-13T05:44:49.473 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:49.467+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.468303+0000) 2019-10-13T05:44:49.473 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:49.468+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.468696+0000) 2019-10-13T05:44:49.473 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:49.468+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.468829+0000) 2019-10-13T05:44:49.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:49.462+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.463068+0000) 2019-10-13T05:44:49.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:49.465+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:49.465765+0000) 2019-10-13T05:44:49.475 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:49.470+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84058 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:50.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:50.538+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:50.538197+0000) 2019-10-13T05:44:51.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:51.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:51.828340+0000) 2019-10-13T05:44:52.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:52.695+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:52.695567+0000) 2019-10-13T05:44:52.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:52.695+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:52.695518+0000) 2019-10-13T05:44:54.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:54.189+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.189831+0000) 2019-10-13T05:44:54.475 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:54.465+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.467209+0000) 2019-10-13T05:44:54.475 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:54.472+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.472413+0000) 2019-10-13T05:44:54.476 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:54.472+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.473169+0000) 2019-10-13T05:44:54.476 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:54.472+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.472594+0000) 2019-10-13T05:44:54.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:54.472+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.473300+0000) 2019-10-13T05:44:54.477 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:54.473+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.473436+0000) 2019-10-13T05:44:54.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:54.473+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.473572+0000) 2019-10-13T05:44:54.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:54.475+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:44:54.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:54.469+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:54.471018+0000) 2019-10-13T05:44:58.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:58.234+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:58.234997+0000) 2019-10-13T05:44:58.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:58.575+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:58.575243+0000) 2019-10-13T05:44:58.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:58.575+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:58.575242+0000) 2019-10-13T05:44:59.481 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:44:59.477+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.477589+0000) 2019-10-13T05:44:59.481 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:44:59.477+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.477717+0000) 2019-10-13T05:44:59.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:44:59.471+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.473215+0000) 2019-10-13T05:44:59.482 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:44:59.477+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.478359+0000) 2019-10-13T05:44:59.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:44:59.478+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.478559+0000) 2019-10-13T05:44:59.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:44:59.478+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.478701+0000) 2019-10-13T05:44:59.483 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:44:59.478+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.479169+0000) 2019-10-13T05:44:59.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:44:59.473+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:44:59.475161+0000) 2019-10-13T05:44:59.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:44:59.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84120 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:00.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:00.539+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:00.540472+0000) 2019-10-13T05:45:01.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:01.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:01.828501+0000) 2019-10-13T05:45:02.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:02.695+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:02.695699+0000) 2019-10-13T05:45:02.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:02.694+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:02.695699+0000) 2019-10-13T05:45:04.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:04.189+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.190016+0000) 2019-10-13T05:45:04.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:04.482+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.483304+0000) 2019-10-13T05:45:04.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:04.483+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.483511+0000) 2019-10-13T05:45:04.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:04.483+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.483661+0000) 2019-10-13T05:45:04.488 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:04.477+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.479366+0000) 2019-10-13T05:45:04.489 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:04.483+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.484065+0000) 2019-10-13T05:45:04.489 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:04.483+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.484046+0000) 2019-10-13T05:45:04.489 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:04.484+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.484776+0000) 2019-10-13T05:45:04.490 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:04.480+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:04.481892+0000) 2019-10-13T05:45:04.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:04.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:08.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:08.234+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:08.235165+0000) 2019-10-13T05:45:08.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:08.574+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:08.575432+0000) 2019-10-13T05:45:08.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:08.574+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:08.575431+0000) 2019-10-13T05:45:09.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:09.483+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.483843+0000) 2019-10-13T05:45:09.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:09.487+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.488938+0000) 2019-10-13T05:45:09.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:09.487+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.489067+0000) 2019-10-13T05:45:09.494 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:09.487+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.489240+0000) 2019-10-13T05:45:09.494 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:09.488+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.489513+0000) 2019-10-13T05:45:09.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:09.488+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.489753+0000) 2019-10-13T05:45:09.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:09.488+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.490268+0000) 2019-10-13T05:45:09.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:09.486+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:09.487024+0000) 2019-10-13T05:45:09.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:09.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:10.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:10.542+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:10.542819+0000) 2019-10-13T05:45:11.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:11.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:11.828633+0000) 2019-10-13T05:45:12.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:12.694+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:12.695859+0000) 2019-10-13T05:45:12.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:12.695+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:12.695889+0000) 2019-10-13T05:45:14.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:14.189+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.190183+0000) 2019-10-13T05:45:14.497 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:14.492+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.494000+0000) 2019-10-13T05:45:14.498 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:14.492+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.494226+0000) 2019-10-13T05:45:14.498 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:14.493+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.494390+0000) 2019-10-13T05:45:14.498 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:14.493+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.494480+0000) 2019-10-13T05:45:14.499 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:14.489+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.489846+0000) 2019-10-13T05:45:14.499 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:14.493+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.494879+0000) 2019-10-13T05:45:14.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:14.493+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.495163+0000) 2019-10-13T05:45:14.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:14.491+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:14.491830+0000) 2019-10-13T05:45:14.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:14.495+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:16.973 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:45:16.974 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:16.978 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:45:16.978 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:18.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:18.234+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:18.235382+0000) 2019-10-13T05:45:18.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:18.574+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:18.575616+0000) 2019-10-13T05:45:18.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:18.574+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:18.575614+0000) 2019-10-13T05:45:19.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:19.493+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.493910+0000) 2019-10-13T05:45:19.503 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:19.497+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.499022+0000) 2019-10-13T05:45:19.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:19.497+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.499318+0000) 2019-10-13T05:45:19.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:19.497+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.499369+0000) 2019-10-13T05:45:19.503 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:19.498+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.499828+0000) 2019-10-13T05:45:19.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:19.498+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.500203+0000) 2019-10-13T05:45:19.504 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:19.498+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.500223+0000) 2019-10-13T05:45:19.505 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:19.496+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:19.496812+0000) 2019-10-13T05:45:19.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:19.500+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84246 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:20.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:20.544+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:20.545162+0000) 2019-10-13T05:45:21.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:21.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:21.828832+0000) 2019-10-13T05:45:22.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:22.695+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:22.696097+0000) 2019-10-13T05:45:22.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:22.695+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:22.696010+0000) 2019-10-13T05:45:24.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:24.189+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.190327+0000) 2019-10-13T05:45:24.507 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:24.498+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.498897+0000) 2019-10-13T05:45:24.507 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:24.503+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.504449+0000) 2019-10-13T05:45:24.508 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:24.503+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.504689+0000) 2019-10-13T05:45:24.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:24.503+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.504859+0000) 2019-10-13T05:45:24.508 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:24.503+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.504995+0000) 2019-10-13T05:45:24.509 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:24.500+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.501010+0000) 2019-10-13T05:45:24.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:24.503+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.505355+0000) 2019-10-13T05:45:24.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:24.504+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:24.505664+0000) 2019-10-13T05:45:24.510 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:24.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:28.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:28.235+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:28.235576+0000) 2019-10-13T05:45:28.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:28.575+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:28.575844+0000) 2019-10-13T05:45:28.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:28.575+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:28.575844+0000) 2019-10-13T05:45:29.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:29.504+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.505208+0000) 2019-10-13T05:45:29.514 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:29.508+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.509457+0000) 2019-10-13T05:45:29.515 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:29.508+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.509656+0000) 2019-10-13T05:45:29.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:29.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.510046+0000) 2019-10-13T05:45:29.515 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:29.508+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.510354+0000) 2019-10-13T05:45:29.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:29.509+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.510568+0000) 2019-10-13T05:45:29.515 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:29.509+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.510665+0000) 2019-10-13T05:45:29.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:29.506+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:29.507452+0000) 2019-10-13T05:45:29.516 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:29.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:30.555 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:30.546+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:30.547526+0000) 2019-10-13T05:45:31.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:31.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:31.829010+0000) 2019-10-13T05:45:32.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:32.695+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:32.696284+0000) 2019-10-13T05:45:32.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:32.695+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:32.696196+0000) 2019-10-13T05:45:34.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:34.190+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.190509+0000) 2019-10-13T05:45:34.523 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:34.509+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.510248+0000) 2019-10-13T05:45:34.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:34.514+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.515003+0000) 2019-10-13T05:45:34.523 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:34.514+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.515012+0000) 2019-10-13T05:45:34.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:34.514+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.515324+0000) 2019-10-13T05:45:34.523 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:34.515+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.515603+0000) 2019-10-13T05:45:34.524 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:34.515+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.515896+0000) 2019-10-13T05:45:34.524 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:34.516+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.516411+0000) 2019-10-13T05:45:34.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:34.517+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:34.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:34.513+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:34.513858+0000) 2019-10-13T05:45:38.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:38.235+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:38.235766+0000) 2019-10-13T05:45:38.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:38.575+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:38.576034+0000) 2019-10-13T05:45:38.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:38.575+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:38.576034+0000) 2019-10-13T05:45:39.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:39.519+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.519941+0000) 2019-10-13T05:45:39.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:39.519+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.520114+0000) 2019-10-13T05:45:39.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:39.519+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.520268+0000) 2019-10-13T05:45:39.524 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:39.520+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.520460+0000) 2019-10-13T05:45:39.524 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:39.520+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.520636+0000) 2019-10-13T05:45:39.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:39.515+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.515876+0000) 2019-10-13T05:45:39.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:39.520+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.520887+0000) 2019-10-13T05:45:39.526 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:39.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:39.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:39.517+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:39.518247+0000) 2019-10-13T05:45:40.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:40.549+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:40.549991+0000) 2019-10-13T05:45:41.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:41.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:41.829188+0000) 2019-10-13T05:45:42.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:42.696+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:42.696360+0000) 2019-10-13T05:45:42.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:42.696+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:42.696494+0000) 2019-10-13T05:45:44.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:44.190+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.190701+0000) 2019-10-13T05:45:44.528 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:44.524+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.524988+0000) 2019-10-13T05:45:44.529 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:44.519+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.520301+0000) 2019-10-13T05:45:44.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:44.525+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.525510+0000) 2019-10-13T05:45:44.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:44.525+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.525720+0000) 2019-10-13T05:45:44.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:44.525+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.525820+0000) 2019-10-13T05:45:44.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:44.525+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.525978+0000) 2019-10-13T05:45:44.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:44.525+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.526129+0000) 2019-10-13T05:45:44.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:44.521+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:44.522457+0000) 2019-10-13T05:45:44.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:44.527+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:47.136 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:45:47.137 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:47.141 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:45:47.141 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:48.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:48.235+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:48.235960+0000) 2019-10-13T05:45:48.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:48.576+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:48.576227+0000) 2019-10-13T05:45:48.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:48.576+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:48.576208+0000) 2019-10-13T05:45:49.532 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:49.522+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.524528+0000) 2019-10-13T05:45:49.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:49.529+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.529755+0000) 2019-10-13T05:45:49.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:49.529+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.529941+0000) 2019-10-13T05:45:49.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:49.529+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.530100+0000) 2019-10-13T05:45:49.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:49.530+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.530525+0000) 2019-10-13T05:45:49.534 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:49.530+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.530917+0000) 2019-10-13T05:45:49.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:49.530+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.531333+0000) 2019-10-13T05:45:49.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:49.525+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:49.526825+0000) 2019-10-13T05:45:49.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:49.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:50.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:50.551+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:50.552379+0000) 2019-10-13T05:45:51.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:51.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:51.829378+0000) 2019-10-13T05:45:52.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:52.695+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:52.696641+0000) 2019-10-13T05:45:52.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:52.695+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:52.696529+0000) 2019-10-13T05:45:54.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:54.189+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.190882+0000) 2019-10-13T05:45:54.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:54.534+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.535404+0000) 2019-10-13T05:45:54.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:54.534+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.535722+0000) 2019-10-13T05:45:54.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:54.529+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.531137+0000) 2019-10-13T05:45:54.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:54.534+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.536186+0000) 2019-10-13T05:45:54.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:54.534+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.536370+0000) 2019-10-13T05:45:54.540 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:54.535+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.536511+0000) 2019-10-13T05:45:54.541 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:54.535+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.536672+0000) 2019-10-13T05:45:54.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:54.531+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:54.533166+0000) 2019-10-13T05:45:54.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:54.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84468 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:58.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:58.235+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:58.236153+0000) 2019-10-13T05:45:58.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:58.575+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:58.576387+0000) 2019-10-13T05:45:58.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:58.575+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:58.576420+0000) 2019-10-13T05:45:59.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:45:59.539+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.540480+0000) 2019-10-13T05:45:59.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:45:59.539+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.540649+0000) 2019-10-13T05:45:59.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:45:59.539+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.541048+0000) 2019-10-13T05:45:59.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:45:59.534+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.536578+0000) 2019-10-13T05:45:59.545 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:45:59.539+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.541300+0000) 2019-10-13T05:45:59.545 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:45:59.539+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.540871+0000) 2019-10-13T05:45:59.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:45:59.540+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.541807+0000) 2019-10-13T05:45:59.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:45:59.542+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:45:59.547 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:45:59.537+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:45:59.539362+0000) 2019-10-13T05:46:00.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:00.553+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:00.554591+0000) 2019-10-13T05:46:01.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:01.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:01.829567+0000) 2019-10-13T05:46:02.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:02.695+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:02.696788+0000) 2019-10-13T05:46:02.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:02.696+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:02.696707+0000) 2019-10-13T05:46:04.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:04.190+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.191067+0000) 2019-10-13T05:46:04.549 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:04.544+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.546128+0000) 2019-10-13T05:46:04.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:04.540+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.541654+0000) 2019-10-13T05:46:04.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:04.545+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.546662+0000) 2019-10-13T05:46:04.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:04.545+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.546932+0000) 2019-10-13T05:46:04.551 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:04.545+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.547071+0000) 2019-10-13T05:46:04.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:04.545+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.547203+0000) 2019-10-13T05:46:04.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:04.546+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.547527+0000) 2019-10-13T05:46:04.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:04.542+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:04.543639+0000) 2019-10-13T05:46:04.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:04.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84532 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:08.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:08.235+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:08.236308+0000) 2019-10-13T05:46:08.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:08.575+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:08.576556+0000) 2019-10-13T05:46:08.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:08.575+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:08.576629+0000) 2019-10-13T05:46:09.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:09.549+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.551237+0000) 2019-10-13T05:46:09.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:09.550+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.551469+0000) 2019-10-13T05:46:09.555 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:09.550+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.551836+0000) 2019-10-13T05:46:09.556 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:09.545+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.546710+0000) 2019-10-13T05:46:09.556 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:09.550+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.552081+0000) 2019-10-13T05:46:09.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:09.550+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.552375+0000) 2019-10-13T05:46:09.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:09.551+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.552618+0000) 2019-10-13T05:46:09.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:09.552+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84561 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:09.559 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:09.548+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:09.549446+0000) 2019-10-13T05:46:10.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:10.556+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:10.556765+0000) 2019-10-13T05:46:11.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:11.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:11.829713+0000) 2019-10-13T05:46:12.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:12.696+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:12.696933+0000) 2019-10-13T05:46:12.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:12.696+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:12.696878+0000) 2019-10-13T05:46:14.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:14.190+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.191251+0000) 2019-10-13T05:46:14.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:14.554+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.555653+0000) 2019-10-13T05:46:14.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:14.554+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.555900+0000) 2019-10-13T05:46:14.559 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:14.554+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.555983+0000) 2019-10-13T05:46:14.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:14.554+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.556159+0000) 2019-10-13T05:46:14.560 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:14.551+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.551801+0000) 2019-10-13T05:46:14.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:14.555+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.556490+0000) 2019-10-13T05:46:14.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:14.555+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.557195+0000) 2019-10-13T05:46:14.561 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:14.552+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:14.553637+0000) 2019-10-13T05:46:14.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:14.558+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84594 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:17.232 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:46:17.233 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:17.237 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:46:17.237 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:18.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:18.235+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:18.236448+0000) 2019-10-13T05:46:18.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:18.576+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:18.576691+0000) 2019-10-13T05:46:18.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:18.576+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:18.576759+0000) 2019-10-13T05:46:19.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:19.559+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.561184+0000) 2019-10-13T05:46:19.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:19.556+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.556935+0000) 2019-10-13T05:46:19.565 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:19.560+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.561603+0000) 2019-10-13T05:46:19.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:19.560+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.561721+0000) 2019-10-13T05:46:19.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:19.560+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.561991+0000) 2019-10-13T05:46:19.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:19.560+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.562265+0000) 2019-10-13T05:46:19.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:19.561+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.562404+0000) 2019-10-13T05:46:19.567 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:19.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:19.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:19.559+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:19.559919+0000) 2019-10-13T05:46:20.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:20.558+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:20.558863+0000) 2019-10-13T05:46:21.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:21.829+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:21.829863+0000) 2019-10-13T05:46:22.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:22.696+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:22.697098+0000) 2019-10-13T05:46:22.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:22.696+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:22.697064+0000) 2019-10-13T05:46:24.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:24.191+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.191446+0000) 2019-10-13T05:46:24.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:24.565+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.565888+0000) 2019-10-13T05:46:24.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:24.565+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.566357+0000) 2019-10-13T05:46:24.570 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:24.566+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.566464+0000) 2019-10-13T05:46:24.570 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:24.560+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.561370+0000) 2019-10-13T05:46:24.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:24.566+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.566606+0000) 2019-10-13T05:46:24.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:24.566+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.566710+0000) 2019-10-13T05:46:24.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:24.566+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.567311+0000) 2019-10-13T05:46:24.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:24.563+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:24.563829+0000) 2019-10-13T05:46:24.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:24.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:28.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:28.236+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:28.236620+0000) 2019-10-13T05:46:28.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:28.576+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:28.576882+0000) 2019-10-13T05:46:28.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:28.576+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:28.576861+0000) 2019-10-13T05:46:29.573 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:29.569+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.569600+0000) 2019-10-13T05:46:29.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:29.569+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.569752+0000) 2019-10-13T05:46:29.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:29.569+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.570088+0000) 2019-10-13T05:46:29.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:29.569+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.570390+0000) 2019-10-13T05:46:29.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:29.570+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.570489+0000) 2019-10-13T05:46:29.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:29.570+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.570835+0000) 2019-10-13T05:46:29.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:29.565+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.565853+0000) 2019-10-13T05:46:29.575 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:29.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84687 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:29.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:29.566+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:29.567540+0000) 2019-10-13T05:46:30.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:30.560+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:30.561001+0000) 2019-10-13T05:46:31.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:31.829+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:31.830052+0000) 2019-10-13T05:46:32.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:32.697+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:32.697242+0000) 2019-10-13T05:46:32.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:32.697+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:32.697272+0000) 2019-10-13T05:46:34.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:34.191+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.191611+0000) 2019-10-13T05:46:34.577 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:34.568+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.569031+0000) 2019-10-13T05:46:34.577 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:34.573+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.574306+0000) 2019-10-13T05:46:34.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:34.573+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.574358+0000) 2019-10-13T05:46:34.578 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:34.574+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.574595+0000) 2019-10-13T05:46:34.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:34.574+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.575080+0000) 2019-10-13T05:46:34.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:34.575+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.575450+0000) 2019-10-13T05:46:34.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:34.575+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.575611+0000) 2019-10-13T05:46:34.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:34.571+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:34.571891+0000) 2019-10-13T05:46:34.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:34.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84720 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:38.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:38.235+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:38.236811+0000) 2019-10-13T05:46:38.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:38.576+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:38.577021+0000) 2019-10-13T05:46:38.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:38.576+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:38.577067+0000) 2019-10-13T05:46:39.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:39.578+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.580356+0000) 2019-10-13T05:46:39.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:39.578+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.580265+0000) 2019-10-13T05:46:39.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:39.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.580777+0000) 2019-10-13T05:46:39.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:39.574+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.576168+0000) 2019-10-13T05:46:39.585 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:39.579+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.581167+0000) 2019-10-13T05:46:39.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:39.579+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.581368+0000) 2019-10-13T05:46:39.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:39.580+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.581517+0000) 2019-10-13T05:46:39.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:39.575+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:39.577390+0000) 2019-10-13T05:46:39.586 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:39.581+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84749 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:40.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:40.563+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:40.564105+0000) 2019-10-13T05:46:41.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:41.829+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:41.830291+0000) 2019-10-13T05:46:42.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:42.696+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:42.697424+0000) 2019-10-13T05:46:42.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:42.696+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:42.697452+0000) 2019-10-13T05:46:44.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:44.190+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.191830+0000) 2019-10-13T05:46:44.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:44.582+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.584088+0000) 2019-10-13T05:46:44.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:44.582+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.584314+0000) 2019-10-13T05:46:44.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:44.578+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.580182+0000) 2019-10-13T05:46:44.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:44.583+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.584756+0000) 2019-10-13T05:46:44.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:44.583+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.585246+0000) 2019-10-13T05:46:44.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:44.584+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.585549+0000) 2019-10-13T05:46:44.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:44.584+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.585727+0000) 2019-10-13T05:46:44.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:44.580+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:44.582567+0000) 2019-10-13T05:46:44.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:44.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:47.370 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:46:47.370 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:47.375 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:46:47.375 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:48.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:48.236+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:48.237000+0000) 2019-10-13T05:46:48.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:48.576+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:48.577162+0000) 2019-10-13T05:46:48.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:48.576+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:48.577216+0000) 2019-10-13T05:46:49.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:49.587+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.588961+0000) 2019-10-13T05:46:49.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:49.587+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.589288+0000) 2019-10-13T05:46:49.593 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:49.588+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.589749+0000) 2019-10-13T05:46:49.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:49.588+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.589936+0000) 2019-10-13T05:46:49.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:49.583+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.585496+0000) 2019-10-13T05:46:49.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:49.588+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.590225+0000) 2019-10-13T05:46:49.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:49.589+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.590404+0000) 2019-10-13T05:46:49.594 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:49.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84813 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:49.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:49.585+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:49.587332+0000) 2019-10-13T05:46:50.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:50.565+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:50.566537+0000) 2019-10-13T05:46:51.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:51.829+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:51.830492+0000) 2019-10-13T05:46:52.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:52.696+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:52.697630+0000) 2019-10-13T05:46:52.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:52.696+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:52.697636+0000) 2019-10-13T05:46:54.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:54.191+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.192019+0000) 2019-10-13T05:46:54.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:54.592+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.593493+0000) 2019-10-13T05:46:54.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:54.588+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.588843+0000) 2019-10-13T05:46:54.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:54.592+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.593832+0000) 2019-10-13T05:46:54.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:54.592+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.594182+0000) 2019-10-13T05:46:54.598 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:54.593+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.594492+0000) 2019-10-13T05:46:54.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:54.593+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.594680+0000) 2019-10-13T05:46:54.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:54.593+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.594831+0000) 2019-10-13T05:46:54.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:54.590+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:54.591573+0000) 2019-10-13T05:46:54.600 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:54.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:46:58.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:58.236+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:58.237221+0000) 2019-10-13T05:46:58.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:58.576+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:58.577372+0000) 2019-10-13T05:46:58.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:58.576+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:58.577368+0000) 2019-10-13T05:46:59.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:46:59.597+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.598910+0000) 2019-10-13T05:46:59.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:46:59.593+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.594214+0000) 2019-10-13T05:46:59.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:46:59.598+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.599446+0000) 2019-10-13T05:46:59.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:46:59.598+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.599529+0000) 2019-10-13T05:46:59.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:46:59.598+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.600200+0000) 2019-10-13T05:46:59.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:46:59.599+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.600440+0000) 2019-10-13T05:46:59.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:46:59.599+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.600672+0000) 2019-10-13T05:46:59.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:46:59.596+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:46:59.597299+0000) 2019-10-13T05:46:59.606 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:46:59.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:00.577 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:00.568+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:00.568919+0000) 2019-10-13T05:47:01.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:01.830+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:01.830717+0000) 2019-10-13T05:47:02.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:02.696+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:02.697793+0000) 2019-10-13T05:47:02.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:02.697+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:02.697854+0000) 2019-10-13T05:47:04.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:04.190+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.192207+0000) 2019-10-13T05:47:04.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:04.603+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.605123+0000) 2019-10-13T05:47:04.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:04.603+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.605330+0000) 2019-10-13T05:47:04.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:04.600+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.601190+0000) 2019-10-13T05:47:04.610 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:04.604+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.605700+0000) 2019-10-13T05:47:04.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:04.604+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.605787+0000) 2019-10-13T05:47:04.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:04.604+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.605852+0000) 2019-10-13T05:47:04.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:04.604+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.606107+0000) 2019-10-13T05:47:04.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:04.605+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84910 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:04.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:04.603+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:04.604348+0000) 2019-10-13T05:47:08.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:08.237+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:08.237422+0000) 2019-10-13T05:47:08.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:08.577+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:08.577595+0000) 2019-10-13T05:47:08.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:08.577+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:08.577595+0000) 2019-10-13T05:47:09.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:09.608+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.608812+0000) 2019-10-13T05:47:09.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:09.608+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.609231+0000) 2019-10-13T05:47:09.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:09.608+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.609355+0000) 2019-10-13T05:47:09.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:09.609+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.609482+0000) 2019-10-13T05:47:09.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:09.609+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.609755+0000) 2019-10-13T05:47:09.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:09.604+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.604811+0000) 2019-10-13T05:47:09.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:09.609+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.610075+0000) 2019-10-13T05:47:09.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:09.606+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:09.607061+0000) 2019-10-13T05:47:09.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:09.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:10.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:10.570+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:10.571354+0000) 2019-10-13T05:47:11.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:11.830+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:11.830879+0000) 2019-10-13T05:47:12.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:12.697+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:12.697965+0000) 2019-10-13T05:47:12.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:12.697+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:12.698005+0000) 2019-10-13T05:47:14.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:14.192+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.192431+0000) 2019-10-13T05:47:14.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:14.613+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.614347+0000) 2019-10-13T05:47:14.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:14.614+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.614658+0000) 2019-10-13T05:47:14.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:14.608+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.609579+0000) 2019-10-13T05:47:14.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:14.614+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.614953+0000) 2019-10-13T05:47:14.619 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:14.614+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.615005+0000) 2019-10-13T05:47:14.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:14.614+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.615118+0000) 2019-10-13T05:47:14.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:14.614+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.615375+0000) 2019-10-13T05:47:14.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:14.610+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:14.611382+0000) 2019-10-13T05:47:14.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:14.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 84972 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:17.581 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:47:17.581 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:17.585 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:47:17.585 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:18.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:18.237+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:18.237579+0000) 2019-10-13T05:47:18.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:18.577+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:18.577789+0000) 2019-10-13T05:47:18.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:18.577+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:18.577791+0000) 2019-10-13T05:47:19.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:19.619+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.619455+0000) 2019-10-13T05:47:19.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:19.619+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.619615+0000) 2019-10-13T05:47:19.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:19.619+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.619846+0000) 2019-10-13T05:47:19.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:19.619+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.620036+0000) 2019-10-13T05:47:19.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:19.615+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.615771+0000) 2019-10-13T05:47:19.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:19.619+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.620358+0000) 2019-10-13T05:47:19.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:19.620+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.620904+0000) 2019-10-13T05:47:19.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:19.617+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:19.617786+0000) 2019-10-13T05:47:19.626 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:19.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:20.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:20.572+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:20.573645+0000) 2019-10-13T05:47:21.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:21.830+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:21.831055+0000) 2019-10-13T05:47:22.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:22.698+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:22.698149+0000) 2019-10-13T05:47:22.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:22.697+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:22.698201+0000) 2019-10-13T05:47:24.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:24.191+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.192620+0000) 2019-10-13T05:47:24.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:24.623+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.624785+0000) 2019-10-13T05:47:24.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:24.623+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.625003+0000) 2019-10-13T05:47:24.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:24.623+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.625192+0000) 2019-10-13T05:47:24.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:24.623+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.625301+0000) 2019-10-13T05:47:24.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:24.620+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.620747+0000) 2019-10-13T05:47:24.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:24.624+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.625888+0000) 2019-10-13T05:47:24.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:24.624+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.626157+0000) 2019-10-13T05:47:24.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:24.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:24.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:24.622+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:24.623551+0000) 2019-10-13T05:47:28.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:28.236+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:28.237756+0000) 2019-10-13T05:47:28.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:28.577+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:28.577968+0000) 2019-10-13T05:47:28.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:28.577+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:28.577966+0000) 2019-10-13T05:47:29.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:29.628+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.630018+0000) 2019-10-13T05:47:29.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:29.628+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.630349+0000) 2019-10-13T05:47:29.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:29.629+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.630662+0000) 2019-10-13T05:47:29.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:29.629+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.630732+0000) 2019-10-13T05:47:29.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:29.629+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.630961+0000) 2019-10-13T05:47:29.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:29.625+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.625976+0000) 2019-10-13T05:47:29.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:29.630+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.631512+0000) 2019-10-13T05:47:29.636 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:29.632+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:29.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:29.628+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:29.629100+0000) 2019-10-13T05:47:30.584 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:30.575+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:30.575980+0000) 2019-10-13T05:47:31.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:31.830+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:31.831261+0000) 2019-10-13T05:47:32.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:32.697+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:32.698278+0000) 2019-10-13T05:47:32.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:32.697+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:32.698380+0000) 2019-10-13T05:47:34.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:34.191+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.192823+0000) 2019-10-13T05:47:34.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:34.628+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.630332+0000) 2019-10-13T05:47:34.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:34.633+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.635301+0000) 2019-10-13T05:47:34.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:34.634+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.635506+0000) 2019-10-13T05:47:34.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:34.634+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.636348+0000) 2019-10-13T05:47:34.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:34.635+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.636735+0000) 2019-10-13T05:47:34.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:34.635+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.636819+0000) 2019-10-13T05:47:34.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:34.635+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.637266+0000) 2019-10-13T05:47:34.641 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:34.632+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:34.633900+0000) 2019-10-13T05:47:34.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:34.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85098 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:38.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:38.237+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:38.237978+0000) 2019-10-13T05:47:38.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:38.577+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:38.578128+0000) 2019-10-13T05:47:38.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:38.577+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:38.578129+0000) 2019-10-13T05:47:39.644 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:39.634+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.636026+0000) 2019-10-13T05:47:39.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:39.639+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.641026+0000) 2019-10-13T05:47:39.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:39.639+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.641247+0000) 2019-10-13T05:47:39.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:39.640+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.641554+0000) 2019-10-13T05:47:39.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:39.640+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.641776+0000) 2019-10-13T05:47:39.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:39.640+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.641819+0000) 2019-10-13T05:47:39.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:39.640+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.642129+0000) 2019-10-13T05:47:39.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:39.636+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:39.638640+0000) 2019-10-13T05:47:39.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:39.643+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:40.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:40.577+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:40.578372+0000) 2019-10-13T05:47:41.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:41.830+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:41.831411+0000) 2019-10-13T05:47:42.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:42.697+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:42.698438+0000) 2019-10-13T05:47:42.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:42.697+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:42.698580+0000) 2019-10-13T05:47:44.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:44.192+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.192992+0000) 2019-10-13T05:47:44.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:44.640+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.641968+0000) 2019-10-13T05:47:44.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:44.645+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.647361+0000) 2019-10-13T05:47:44.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:44.646+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.647589+0000) 2019-10-13T05:47:44.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:44.646+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.647969+0000) 2019-10-13T05:47:44.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:44.646+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.647802+0000) 2019-10-13T05:47:44.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:44.646+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.648026+0000) 2019-10-13T05:47:44.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:44.646+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.648200+0000) 2019-10-13T05:47:44.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:44.642+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:44.644183+0000) 2019-10-13T05:47:44.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:44.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:47.678 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:47:47.678 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:47.682 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:47:47.682 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:48.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:48.237+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:48.238181+0000) 2019-10-13T05:47:48.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:48.577+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:48.578315+0000) 2019-10-13T05:47:48.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:48.577+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:48.578318+0000) 2019-10-13T05:47:49.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:49.651+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.652721+0000) 2019-10-13T05:47:49.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:49.651+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.652905+0000) 2019-10-13T05:47:49.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:49.651+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.653135+0000) 2019-10-13T05:47:49.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:49.647+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.648659+0000) 2019-10-13T05:47:49.658 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:49.649+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.649917+0000) 2019-10-13T05:47:49.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:49.652+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.653625+0000) 2019-10-13T05:47:49.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:49.652+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.653909+0000) 2019-10-13T05:47:49.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:49.652+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:49.653951+0000) 2019-10-13T05:47:49.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:49.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85191 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:50.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:50.579+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:50.580538+0000) 2019-10-13T05:47:51.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:51.831+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:51.831504+0000) 2019-10-13T05:47:52.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:52.698+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:52.698584+0000) 2019-10-13T05:47:52.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:52.698+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:52.698784+0000) 2019-10-13T05:47:54.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:54.192+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.193139+0000) 2019-10-13T05:47:54.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:54.657+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.657502+0000) 2019-10-13T05:47:54.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:54.657+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.657710+0000) 2019-10-13T05:47:54.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:54.652+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.653061+0000) 2019-10-13T05:47:54.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:54.657+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.658313+0000) 2019-10-13T05:47:54.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:54.658+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.658403+0000) 2019-10-13T05:47:54.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:54.658+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.659029+0000) 2019-10-13T05:47:54.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:54.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.659071+0000) 2019-10-13T05:47:54.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:54.654+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:54.655421+0000) 2019-10-13T05:47:54.664 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:54.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:58.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:58.238+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:58.238375+0000) 2019-10-13T05:47:58.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:58.578+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:58.578510+0000) 2019-10-13T05:47:58.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:58.578+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:58.578541+0000) 2019-10-13T05:47:59.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:47:59.662+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.662967+0000) 2019-10-13T05:47:59.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:47:59.663+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.663552+0000) 2019-10-13T05:47:59.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:47:59.663+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.663683+0000) 2019-10-13T05:47:59.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:47:59.663+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.663877+0000) 2019-10-13T05:47:59.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:47:59.658+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.659373+0000) 2019-10-13T05:47:59.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:47:59.664+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.664819+0000) 2019-10-13T05:47:59.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:47:59.664+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.664953+0000) 2019-10-13T05:47:59.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:47:59.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:47:59.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:47:59.661+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:47:59.662441+0000) 2019-10-13T05:48:00.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:00.582+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:00.582585+0000) 2019-10-13T05:48:01.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:01.831+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:01.831682+0000) 2019-10-13T05:48:02.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:02.698+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:02.698768+0000) 2019-10-13T05:48:02.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:02.698+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:02.698953+0000) 2019-10-13T05:48:04.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:04.193+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.193310+0000) 2019-10-13T05:48:04.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:04.668+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.668564+0000) 2019-10-13T05:48:04.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:04.668+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.668777+0000) 2019-10-13T05:48:04.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:04.663+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.664253+0000) 2019-10-13T05:48:04.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:04.668+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.669332+0000) 2019-10-13T05:48:04.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:04.669+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.669526+0000) 2019-10-13T05:48:04.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:04.668+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.669227+0000) 2019-10-13T05:48:04.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:04.669+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.670198+0000) 2019-10-13T05:48:04.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:04.666+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:04.666931+0000) 2019-10-13T05:48:04.675 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:04.671+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:08.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:08.238+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:08.238583+0000) 2019-10-13T05:48:08.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:08.578+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:08.578718+0000) 2019-10-13T05:48:08.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:08.578+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:08.578694+0000) 2019-10-13T05:48:09.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:09.673+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.674016+0000) 2019-10-13T05:48:09.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:09.673+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.674292+0000) 2019-10-13T05:48:09.678 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:09.674+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.674509+0000) 2019-10-13T05:48:09.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:09.674+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.674931+0000) 2019-10-13T05:48:09.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:09.674+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.675022+0000) 2019-10-13T05:48:09.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:09.674+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.675120+0000) 2019-10-13T05:48:09.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:09.669+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.670561+0000) 2019-10-13T05:48:09.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:09.670+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:09.671220+0000) 2019-10-13T05:48:09.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:09.677+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:10.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:10.584+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:10.584833+0000) 2019-10-13T05:48:11.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:11.830+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:11.831921+0000) 2019-10-13T05:48:12.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:12.697+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:12.698936+0000) 2019-10-13T05:48:12.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:12.698+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:12.699133+0000) 2019-10-13T05:48:14.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:14.192+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.193529+0000) 2019-10-13T05:48:14.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:14.677+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.679176+0000) 2019-10-13T05:48:14.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:14.677+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.679206+0000) 2019-10-13T05:48:14.683 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:14.678+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.679517+0000) 2019-10-13T05:48:14.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:14.678+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.679642+0000) 2019-10-13T05:48:14.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:14.678+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.679824+0000) 2019-10-13T05:48:14.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:14.678+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.680081+0000) 2019-10-13T05:48:14.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:14.673+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.674532+0000) 2019-10-13T05:48:14.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:14.676+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:14.677198+0000) 2019-10-13T05:48:14.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:14.681+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85350 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:17.883 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:48:17.883 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:17.888 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:48:17.888 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:18.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:18.237+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:18.238772+0000) 2019-10-13T05:48:18.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:18.578+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:18.578905+0000) 2019-10-13T05:48:18.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:18.578+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:18.578904+0000) 2019-10-13T05:48:19.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:19.682+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.683708+0000) 2019-10-13T05:48:19.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:19.682+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.684018+0000) 2019-10-13T05:48:19.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:19.682+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.684326+0000) 2019-10-13T05:48:19.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:19.678+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.679282+0000) 2019-10-13T05:48:19.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:19.683+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.684497+0000) 2019-10-13T05:48:19.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:19.683+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.684602+0000) 2019-10-13T05:48:19.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:19.683+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.684733+0000) 2019-10-13T05:48:19.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:19.681+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:19.682093+0000) 2019-10-13T05:48:19.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:19.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85379 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:20.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:20.587+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:20.587187+0000) 2019-10-13T05:48:21.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:21.830+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:21.832122+0000) 2019-10-13T05:48:22.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:22.698+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:22.699131+0000) 2019-10-13T05:48:22.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:22.698+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:22.699277+0000) 2019-10-13T05:48:24.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:24.192+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.193724+0000) 2019-10-13T05:48:24.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:24.681+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.682978+0000) 2019-10-13T05:48:24.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:24.686+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.688063+0000) 2019-10-13T05:48:24.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:24.687+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.688427+0000) 2019-10-13T05:48:24.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:24.687+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.688492+0000) 2019-10-13T05:48:24.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:24.687+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.688795+0000) 2019-10-13T05:48:24.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:24.687+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.688849+0000) 2019-10-13T05:48:24.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:24.687+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.688984+0000) 2019-10-13T05:48:24.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:24.683+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:24.685462+0000) 2019-10-13T05:48:24.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:24.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85413 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:28.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:28.238+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:28.238986+0000) 2019-10-13T05:48:28.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:28.577+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:28.579105+0000) 2019-10-13T05:48:28.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:28.577+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:28.579112+0000) 2019-10-13T05:48:29.696 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:29.691+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.692917+0000) 2019-10-13T05:48:29.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:29.686+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.688305+0000) 2019-10-13T05:48:29.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:29.691+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.693362+0000) 2019-10-13T05:48:29.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:29.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.693554+0000) 2019-10-13T05:48:29.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:29.692+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.693621+0000) 2019-10-13T05:48:29.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:29.692+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.693905+0000) 2019-10-13T05:48:29.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:29.693+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.694573+0000) 2019-10-13T05:48:29.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:29.695+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85442 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:29.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:29.690+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:29.692368+0000) 2019-10-13T05:48:30.597 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:30.588+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:30.589282+0000) 2019-10-13T05:48:31.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:31.831+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:31.832288+0000) 2019-10-13T05:48:32.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:32.698+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:32.699349+0000) 2019-10-13T05:48:32.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:32.698+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:32.699468+0000) 2019-10-13T05:48:34.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:34.192+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.193901+0000) 2019-10-13T05:48:34.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:34.691+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.693537+0000) 2019-10-13T05:48:34.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:34.697+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.698505+0000) 2019-10-13T05:48:34.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:34.697+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.698818+0000) 2019-10-13T05:48:34.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:34.697+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.698846+0000) 2019-10-13T05:48:34.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:34.697+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.699058+0000) 2019-10-13T05:48:34.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:34.698+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.699393+0000) 2019-10-13T05:48:34.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:34.693+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.695601+0000) 2019-10-13T05:48:34.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:34.698+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:34.699642+0000) 2019-10-13T05:48:34.704 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:34.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85474 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:38.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:38.238+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:38.239189+0000) 2019-10-13T05:48:38.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:38.578+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:38.579304+0000) 2019-10-13T05:48:38.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:38.578+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:38.579311+0000) 2019-10-13T05:48:39.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:39.698+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.698938+0000) 2019-10-13T05:48:39.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:39.703+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.703983+0000) 2019-10-13T05:48:39.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:39.703+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.704063+0000) 2019-10-13T05:48:39.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:39.703+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.704251+0000) 2019-10-13T05:48:39.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:39.703+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.704288+0000) 2019-10-13T05:48:39.708 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:39.704+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.704604+0000) 2019-10-13T05:48:39.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:39.704+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.704778+0000) 2019-10-13T05:48:39.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:39.699+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:39.700568+0000) 2019-10-13T05:48:39.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:39.706+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:40.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:40.590+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:40.591434+0000) 2019-10-13T05:48:41.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:41.832+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:41.832479+0000) 2019-10-13T05:48:42.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:42.699+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:42.699497+0000) 2019-10-13T05:48:42.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:42.698+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:42.699629+0000) 2019-10-13T05:48:44.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:44.193+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.194103+0000) 2019-10-13T05:48:44.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:44.708+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.708410+0000) 2019-10-13T05:48:44.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:44.708+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.708608+0000) 2019-10-13T05:48:44.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:44.708+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.709140+0000) 2019-10-13T05:48:44.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:44.709+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.709394+0000) 2019-10-13T05:48:44.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:44.709+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.709605+0000) 2019-10-13T05:48:44.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:44.704+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.705000+0000) 2019-10-13T05:48:44.713 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:44.709+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.709961+0000) 2019-10-13T05:48:44.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:44.706+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:44.707106+0000) 2019-10-13T05:48:44.715 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:44.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:48.073 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:48:48.073 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:48.077 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:48:48.078 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:48.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:48.239+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:48.239391+0000) 2019-10-13T05:48:48.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:48.579+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:48.579502+0000) 2019-10-13T05:48:48.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:48.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:48.579503+0000) 2019-10-13T05:48:49.718 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:49.708+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.709664+0000) 2019-10-13T05:48:49.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:49.713+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.714377+0000) 2019-10-13T05:48:49.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:49.714+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.714622+0000) 2019-10-13T05:48:49.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:49.714+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.714950+0000) 2019-10-13T05:48:49.719 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:49.714+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.715171+0000) 2019-10-13T05:48:49.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:49.714+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.715287+0000) 2019-10-13T05:48:49.720 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:49.710+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.711542+0000) 2019-10-13T05:48:49.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:49.715+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:49.715733+0000) 2019-10-13T05:48:49.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:49.717+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:50.601 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:50.592+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:50.593514+0000) 2019-10-13T05:48:51.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:51.832+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:51.832703+0000) 2019-10-13T05:48:52.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:52.699+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:52.699682+0000) 2019-10-13T05:48:52.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:52.699+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:52.699811+0000) 2019-10-13T05:48:54.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:54.194+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.194306+0000) 2019-10-13T05:48:54.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:54.719+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.719876+0000) 2019-10-13T05:48:54.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:54.719+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.720195+0000) 2019-10-13T05:48:54.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:54.714+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.715231+0000) 2019-10-13T05:48:54.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:54.720+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.720431+0000) 2019-10-13T05:48:54.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:54.720+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.720547+0000) 2019-10-13T05:48:54.725 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:54.720+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.720633+0000) 2019-10-13T05:48:54.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:54.720+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.721043+0000) 2019-10-13T05:48:54.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:54.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:48:54.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:54.718+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:54.718792+0000) 2019-10-13T05:48:58.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:58.238+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:58.239585+0000) 2019-10-13T05:48:58.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:58.578+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:58.579689+0000) 2019-10-13T05:48:58.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:58.578+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:58.579708+0000) 2019-10-13T05:48:59.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:48:59.723+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.724411+0000) 2019-10-13T05:48:59.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:48:59.723+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.724768+0000) 2019-10-13T05:48:59.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:48:59.719+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.720422+0000) 2019-10-13T05:48:59.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:48:59.723+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.724928+0000) 2019-10-13T05:48:59.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:48:59.723+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.725296+0000) 2019-10-13T05:48:59.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:48:59.724+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.725470+0000) 2019-10-13T05:48:59.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:48:59.724+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.726058+0000) 2019-10-13T05:48:59.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:48:59.721+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:48:59.722090+0000) 2019-10-13T05:48:59.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:48:59.726+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:00.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:00.595+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:00.595974+0000) 2019-10-13T05:49:01.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:01.831+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:01.832889+0000) 2019-10-13T05:49:02.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:02.698+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:02.699876+0000) 2019-10-13T05:49:02.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:02.699+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:02.699978+0000) 2019-10-13T05:49:04.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:04.193+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.194526+0000) 2019-10-13T05:49:04.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:04.723+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.724675+0000) 2019-10-13T05:49:04.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:04.728+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.729692+0000) 2019-10-13T05:49:04.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:04.728+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.729794+0000) 2019-10-13T05:49:04.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:04.728+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.730091+0000) 2019-10-13T05:49:04.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:04.728+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.729962+0000) 2019-10-13T05:49:04.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:04.728+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.730330+0000) 2019-10-13T05:49:04.734 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:04.729+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.730483+0000) 2019-10-13T05:49:04.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:04.727+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:04.728188+0000) 2019-10-13T05:49:04.736 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:04.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85662 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:08.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:08.238+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:08.239740+0000) 2019-10-13T05:49:08.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:08.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:08.579856+0000) 2019-10-13T05:49:08.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:08.579+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:08.579919+0000) 2019-10-13T05:49:09.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:09.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.734762+0000) 2019-10-13T05:49:09.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:09.733+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.735134+0000) 2019-10-13T05:49:09.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:09.729+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.730241+0000) 2019-10-13T05:49:09.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:09.733+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.735359+0000) 2019-10-13T05:49:09.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:09.734+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.735577+0000) 2019-10-13T05:49:09.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:09.734+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.735795+0000) 2019-10-13T05:49:09.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:09.734+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.735983+0000) 2019-10-13T05:49:09.746 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:09.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:09.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:09.732+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:09.733570+0000) 2019-10-13T05:49:10.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:10.597+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:10.598236+0000) 2019-10-13T05:49:11.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:11.831+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:11.833081+0000) 2019-10-13T05:49:12.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:12.699+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:12.700057+0000) 2019-10-13T05:49:12.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:12.700+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:12.700163+0000) 2019-10-13T05:49:14.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:14.194+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.194727+0000) 2019-10-13T05:49:14.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:14.734+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.734752+0000) 2019-10-13T05:49:14.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:14.738+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.739987+0000) 2019-10-13T05:49:14.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:14.738+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.740147+0000) 2019-10-13T05:49:14.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:14.738+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.740271+0000) 2019-10-13T05:49:14.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:14.739+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.740566+0000) 2019-10-13T05:49:14.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:14.739+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.740749+0000) 2019-10-13T05:49:14.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:14.739+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.740415+0000) 2019-10-13T05:49:14.745 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:14.736+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:14.737153+0000) 2019-10-13T05:49:14.746 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:14.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85723 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:18.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:18.239+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:18.239914+0000) 2019-10-13T05:49:18.270 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:49:18.270 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:18.274 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:49:18.274 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:18.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:18.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:18.580026+0000) 2019-10-13T05:49:18.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:18.579+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:18.580079+0000) 2019-10-13T05:49:19.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:19.743+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.744883+0000) 2019-10-13T05:49:19.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:19.743+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.745223+0000) 2019-10-13T05:49:19.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:19.743+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.745115+0000) 2019-10-13T05:49:19.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:19.743+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.745318+0000) 2019-10-13T05:49:19.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:19.738+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.740661+0000) 2019-10-13T05:49:19.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:19.744+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.745594+0000) 2019-10-13T05:49:19.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:19.744+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.745831+0000) 2019-10-13T05:49:19.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:19.746+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85751 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:19.751 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:19.741+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:19.743374+0000) 2019-10-13T05:49:20.608 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:20.599+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:20.600504+0000) 2019-10-13T05:49:21.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:21.832+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:21.833243+0000) 2019-10-13T05:49:22.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:22.699+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:22.700238+0000) 2019-10-13T05:49:22.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:22.699+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:22.700358+0000) 2019-10-13T05:49:24.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:24.194+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.194902+0000) 2019-10-13T05:49:24.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:24.743+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.744793+0000) 2019-10-13T05:49:24.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:24.749+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.749763+0000) 2019-10-13T05:49:24.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:24.749+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.749954+0000) 2019-10-13T05:49:24.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:24.750+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.750433+0000) 2019-10-13T05:49:24.754 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:24.750+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.750619+0000) 2019-10-13T05:49:24.754 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:24.750+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.750783+0000) 2019-10-13T05:49:24.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:24.750+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.750883+0000) 2019-10-13T05:49:24.757 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:24.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:24.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:24.745+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:24.747610+0000) 2019-10-13T05:49:28.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:28.239+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:28.240075+0000) 2019-10-13T05:49:28.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:28.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:28.580210+0000) 2019-10-13T05:49:28.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:28.579+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:28.580227+0000) 2019-10-13T05:49:29.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:29.754+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.755176+0000) 2019-10-13T05:49:29.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:29.755+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.755494+0000) 2019-10-13T05:49:29.759 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:29.755+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.755745+0000) 2019-10-13T05:49:29.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:29.749+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.751218+0000) 2019-10-13T05:49:29.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:29.755+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.755837+0000) 2019-10-13T05:49:29.760 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:29.755+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.755756+0000) 2019-10-13T05:49:29.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:29.755+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.756158+0000) 2019-10-13T05:49:29.761 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:29.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:29.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:29.751+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:29.753634+0000) 2019-10-13T05:49:30.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:30.602+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:30.602963+0000) 2019-10-13T05:49:31.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:31.833+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:31.833434+0000) 2019-10-13T05:49:32.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:32.700+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:32.700431+0000) 2019-10-13T05:49:32.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:32.699+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:32.700526+0000) 2019-10-13T05:49:34.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:34.194+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.195105+0000) 2019-10-13T05:49:34.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:34.755+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.756078+0000) 2019-10-13T05:49:34.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:34.757+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.757839+0000) 2019-10-13T05:49:34.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:34.759+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.759767+0000) 2019-10-13T05:49:34.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:34.759+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.760014+0000) 2019-10-13T05:49:34.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:34.760+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.760431+0000) 2019-10-13T05:49:34.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:34.760+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.760524+0000) 2019-10-13T05:49:34.780 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:34.760+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.760821+0000) 2019-10-13T05:49:34.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:34.760+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:34.761131+0000) 2019-10-13T05:49:34.781 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:34.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:38.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:38.240+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:38.240215+0000) 2019-10-13T05:49:38.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:38.580+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:38.580420+0000) 2019-10-13T05:49:38.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:38.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:38.580424+0000) 2019-10-13T05:49:39.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:39.764+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.764440+0000) 2019-10-13T05:49:39.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:39.764+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.764867+0000) 2019-10-13T05:49:39.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:39.759+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.759945+0000) 2019-10-13T05:49:39.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:39.764+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.765127+0000) 2019-10-13T05:49:39.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:39.764+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.765239+0000) 2019-10-13T05:49:39.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:39.765+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.765708+0000) 2019-10-13T05:49:39.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:39.765+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.765934+0000) 2019-10-13T05:49:39.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:39.762+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:39.763055+0000) 2019-10-13T05:49:39.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:39.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:40.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:40.604+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:40.605090+0000) 2019-10-13T05:49:41.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:41.833+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:41.833632+0000) 2019-10-13T05:49:42.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:42.699+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:42.700636+0000) 2019-10-13T05:49:42.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:42.700+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:42.700747+0000) 2019-10-13T05:49:44.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:44.194+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.195287+0000) 2019-10-13T05:49:44.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:44.768+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.769676+0000) 2019-10-13T05:49:44.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:44.768+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.770137+0000) 2019-10-13T05:49:44.773 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:44.768+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.770322+0000) 2019-10-13T05:49:44.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:44.769+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.770452+0000) 2019-10-13T05:49:44.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:44.764+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.765113+0000) 2019-10-13T05:49:44.775 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:44.769+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.770678+0000) 2019-10-13T05:49:44.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:44.769+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.770994+0000) 2019-10-13T05:49:44.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:44.767+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:44.768090+0000) 2019-10-13T05:49:44.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:44.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85910 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:48.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:48.239+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:48.240362+0000) 2019-10-13T05:49:48.434 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:49:48.434 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:48.438 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:49:48.438 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:48.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:48.579+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:48.580534+0000) 2019-10-13T05:49:48.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:48.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:48.580541+0000) 2019-10-13T05:49:49.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:49.769+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.770367+0000) 2019-10-13T05:49:49.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:49.774+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.775592+0000) 2019-10-13T05:49:49.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:49.774+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.775716+0000) 2019-10-13T05:49:49.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:49.774+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.775913+0000) 2019-10-13T05:49:49.780 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:49.774+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.776139+0000) 2019-10-13T05:49:49.780 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:49.774+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.776176+0000) 2019-10-13T05:49:49.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:49.775+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.776439+0000) 2019-10-13T05:49:49.781 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:49.772+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:49.773592+0000) 2019-10-13T05:49:49.781 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:49.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85938 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:50.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:50.607+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:50.607429+0000) 2019-10-13T05:49:51.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:51.832+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:51.833807+0000) 2019-10-13T05:49:52.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:52.700+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:52.700825+0000) 2019-10-13T05:49:52.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:52.700+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:52.700895+0000) 2019-10-13T05:49:54.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:54.194+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.195478+0000) 2019-10-13T05:49:54.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:54.779+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.780502+0000) 2019-10-13T05:49:54.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:54.775+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.775933+0000) 2019-10-13T05:49:54.786 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:54.779+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.781036+0000) 2019-10-13T05:49:54.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:54.780+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.781410+0000) 2019-10-13T05:49:54.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:54.780+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.781726+0000) 2019-10-13T05:49:54.787 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:54.780+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.782065+0000) 2019-10-13T05:49:54.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:54.780+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.781727+0000) 2019-10-13T05:49:54.788 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:54.777+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:54.778509+0000) 2019-10-13T05:49:54.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:54.782+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85971 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:49:58.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:58.239+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:58.240541+0000) 2019-10-13T05:49:58.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:58.580+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:58.580746+0000) 2019-10-13T05:49:58.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:58.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:58.580746+0000) 2019-10-13T05:49:59.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:49:59.780+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.781101+0000) 2019-10-13T05:49:59.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:49:59.784+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.786241+0000) 2019-10-13T05:49:59.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:49:59.785+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.786536+0000) 2019-10-13T05:49:59.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:49:59.785+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.786910+0000) 2019-10-13T05:49:59.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:49:59.785+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.786827+0000) 2019-10-13T05:49:59.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:49:59.785+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.787110+0000) 2019-10-13T05:49:59.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:49:59.785+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.787323+0000) 2019-10-13T05:49:59.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:49:59.783+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:49:59.784051+0000) 2019-10-13T05:49:59.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:49:59.788+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 85999 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:00.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:00.609+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:00.609801+0000) 2019-10-13T05:50:01.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:01.832+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:01.833958+0000) 2019-10-13T05:50:02.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:02.700+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:02.701011+0000) 2019-10-13T05:50:02.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:02.700+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:02.701059+0000) 2019-10-13T05:50:04.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:04.195+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.195662+0000) 2019-10-13T05:50:04.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:04.790+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.791928+0000) 2019-10-13T05:50:04.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:04.790+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.792121+0000) 2019-10-13T05:50:04.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:04.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.787848+0000) 2019-10-13T05:50:04.796 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:04.791+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.792446+0000) 2019-10-13T05:50:04.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:04.791+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.793096+0000) 2019-10-13T05:50:04.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:04.791+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.793274+0000) 2019-10-13T05:50:04.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:04.791+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.792909+0000) 2019-10-13T05:50:04.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:04.789+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:04.790262+0000) 2019-10-13T05:50:04.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:04.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:08.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:08.240+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:08.240737+0000) 2019-10-13T05:50:08.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:08.580+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:08.580931+0000) 2019-10-13T05:50:08.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:08.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:08.580930+0000) 2019-10-13T05:50:09.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:09.797+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.798130+0000) 2019-10-13T05:50:09.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:09.797+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.798289+0000) 2019-10-13T05:50:09.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:09.798+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.798510+0000) 2019-10-13T05:50:09.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:09.798+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.798595+0000) 2019-10-13T05:50:09.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:09.793+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.794187+0000) 2019-10-13T05:50:09.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:09.798+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.798926+0000) 2019-10-13T05:50:09.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:09.798+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.799324+0000) 2019-10-13T05:50:09.805 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:09.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86062 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:09.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:09.795+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:09.796582+0000) 2019-10-13T05:50:10.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:10.611+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:10.612226+0000) 2019-10-13T05:50:11.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:11.833+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:11.834148+0000) 2019-10-13T05:50:12.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:12.700+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:12.701192+0000) 2019-10-13T05:50:12.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:12.700+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:12.701245+0000) 2019-10-13T05:50:14.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:14.195+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.195846+0000) 2019-10-13T05:50:14.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:14.801+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.801858+0000) 2019-10-13T05:50:14.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:14.801+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.802367+0000) 2019-10-13T05:50:14.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:14.802+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.802465+0000) 2019-10-13T05:50:14.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:14.802+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.802713+0000) 2019-10-13T05:50:14.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:14.796+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.798254+0000) 2019-10-13T05:50:14.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:14.802+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.803101+0000) 2019-10-13T05:50:14.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:14.802+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.803272+0000) 2019-10-13T05:50:14.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:14.798+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:14.799944+0000) 2019-10-13T05:50:14.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:14.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:18.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:18.240+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:18.240894+0000) 2019-10-13T05:50:18.581 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:50:18.582 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:18.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:18.580+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:18.581119+0000) 2019-10-13T05:50:18.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:18.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:18.581104+0000) 2019-10-13T05:50:18.586 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:50:18.586 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:19.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:19.806+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.807295+0000) 2019-10-13T05:50:19.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:19.807+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.807526+0000) 2019-10-13T05:50:19.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:19.807+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.807698+0000) 2019-10-13T05:50:19.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:19.807+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.807887+0000) 2019-10-13T05:50:19.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:19.802+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.803933+0000) 2019-10-13T05:50:19.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:19.808+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.808450+0000) 2019-10-13T05:50:19.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:19.808+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.808835+0000) 2019-10-13T05:50:19.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:19.810+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:19.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:19.805+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:19.806855+0000) 2019-10-13T05:50:20.622 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:20.613+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:20.614284+0000) 2019-10-13T05:50:21.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:21.834+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:21.834351+0000) 2019-10-13T05:50:22.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:22.701+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:22.701408+0000) 2019-10-13T05:50:22.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:22.700+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:22.701426+0000) 2019-10-13T05:50:24.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:24.195+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.196053+0000) 2019-10-13T05:50:24.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:24.806+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.807503+0000) 2019-10-13T05:50:24.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:24.812+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.812642+0000) 2019-10-13T05:50:24.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:24.812+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.812941+0000) 2019-10-13T05:50:24.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:24.812+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.813212+0000) 2019-10-13T05:50:24.818 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:24.812+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.813340+0000) 2019-10-13T05:50:24.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:24.813+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.813565+0000) 2019-10-13T05:50:24.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:24.813+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.813670+0000) 2019-10-13T05:50:24.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:24.809+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:24.810116+0000) 2019-10-13T05:50:24.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:24.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:28.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:28.240+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:28.241049+0000) 2019-10-13T05:50:28.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:28.580+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:28.581320+0000) 2019-10-13T05:50:28.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:28.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:28.581316+0000) 2019-10-13T05:50:29.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:29.817+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.818442+0000) 2019-10-13T05:50:29.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:29.817+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.818635+0000) 2019-10-13T05:50:29.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:29.813+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.813923+0000) 2019-10-13T05:50:29.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:29.817+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.818934+0000) 2019-10-13T05:50:29.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:29.817+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.819016+0000) 2019-10-13T05:50:29.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:29.817+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.819304+0000) 2019-10-13T05:50:29.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:29.818+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.819528+0000) 2019-10-13T05:50:29.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:29.814+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:29.815610+0000) 2019-10-13T05:50:29.824 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:29.820+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:30.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:30.616+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:30.616663+0000) 2019-10-13T05:50:31.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:31.833+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:31.834561+0000) 2019-10-13T05:50:32.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:32.700+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:32.701597+0000) 2019-10-13T05:50:32.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:32.700+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:32.701622+0000) 2019-10-13T05:50:34.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:34.195+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.196241+0000) 2019-10-13T05:50:34.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:34.821+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.823084+0000) 2019-10-13T05:50:34.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:34.821+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.823317+0000) 2019-10-13T05:50:34.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:34.817+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.818677+0000) 2019-10-13T05:50:34.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:34.822+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.823646+0000) 2019-10-13T05:50:34.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:34.822+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.823988+0000) 2019-10-13T05:50:34.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:34.822+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.824371+0000) 2019-10-13T05:50:34.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:34.823+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.824545+0000) 2019-10-13T05:50:34.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:34.820+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:34.821410+0000) 2019-10-13T05:50:34.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:34.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:38.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:38.240+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:38.241199+0000) 2019-10-13T05:50:38.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:38.580+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:38.581480+0000) 2019-10-13T05:50:38.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:38.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:38.581480+0000) 2019-10-13T05:50:39.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:39.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.828243+0000) 2019-10-13T05:50:39.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:39.827+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.828622+0000) 2019-10-13T05:50:39.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:39.823+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.824019+0000) 2019-10-13T05:50:39.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:39.827+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.829016+0000) 2019-10-13T05:50:39.833 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:39.827+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.829225+0000) 2019-10-13T05:50:39.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:39.828+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.829394+0000) 2019-10-13T05:50:39.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:39.828+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.829506+0000) 2019-10-13T05:50:39.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:39.825+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:39.826449+0000) 2019-10-13T05:50:39.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:39.830+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:40.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:40.618+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:40.619063+0000) 2019-10-13T05:50:41.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:41.834+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:41.834768+0000) 2019-10-13T05:50:42.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:42.701+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:42.701791+0000) 2019-10-13T05:50:42.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:42.701+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:42.701806+0000) 2019-10-13T05:50:44.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:44.195+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.196395+0000) 2019-10-13T05:50:44.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:44.832+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.834234+0000) 2019-10-13T05:50:44.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:44.833+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.834650+0000) 2019-10-13T05:50:44.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:44.829+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.830027+0000) 2019-10-13T05:50:44.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:44.833+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.834983+0000) 2019-10-13T05:50:44.839 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:44.833+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.835178+0000) 2019-10-13T05:50:44.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:44.834+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.835393+0000) 2019-10-13T05:50:44.839 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:44.834+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.835682+0000) 2019-10-13T05:50:44.840 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:44.832+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:44.832844+0000) 2019-10-13T05:50:44.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:44.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:48.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:48.240+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:48.241378+0000) 2019-10-13T05:50:48.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:48.581+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:48.581665+0000) 2019-10-13T05:50:48.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:48.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:48.581667+0000) 2019-10-13T05:50:48.753 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:50:48.753 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:48.758 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:50:48.758 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:49.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:49.838+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.840164+0000) 2019-10-13T05:50:49.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:49.835+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.836087+0000) 2019-10-13T05:50:49.844 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:49.839+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.840538+0000) 2019-10-13T05:50:49.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:49.839+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.840505+0000) 2019-10-13T05:50:49.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:49.839+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.840686+0000) 2019-10-13T05:50:49.845 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:49.839+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.840984+0000) 2019-10-13T05:50:49.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:49.839+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.841190+0000) 2019-10-13T05:50:49.853 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:49.837+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:49.838657+0000) 2019-10-13T05:50:49.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:49.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:50.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:50.620+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:50.621564+0000) 2019-10-13T05:50:51.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:51.833+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:51.834992+0000) 2019-10-13T05:50:52.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:52.701+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:52.701954+0000) 2019-10-13T05:50:52.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:52.701+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:52.701959+0000) 2019-10-13T05:50:54.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:54.196+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.196549+0000) 2019-10-13T05:50:54.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:54.839+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.839906+0000) 2019-10-13T05:50:54.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:54.843+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.844932+0000) 2019-10-13T05:50:54.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:54.844+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.845497+0000) 2019-10-13T05:50:54.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:54.844+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.845773+0000) 2019-10-13T05:50:54.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:54.844+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.845944+0000) 2019-10-13T05:50:54.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:54.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.846174+0000) 2019-10-13T05:50:54.850 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:54.841+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.842211+0000) 2019-10-13T05:50:54.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:54.843+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:54.845134+0000) 2019-10-13T05:50:54.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:54.846+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:50:58.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:58.241+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:58.241592+0000) 2019-10-13T05:50:58.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:58.581+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:58.581848+0000) 2019-10-13T05:50:58.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:58.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:58.581847+0000) 2019-10-13T05:50:59.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:50:59.844+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.845437+0000) 2019-10-13T05:50:59.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:50:59.849+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.850003+0000) 2019-10-13T05:50:59.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:50:59.850+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.850572+0000) 2019-10-13T05:50:59.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:50:59.850+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.850435+0000) 2019-10-13T05:50:59.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:50:59.850+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.850916+0000) 2019-10-13T05:50:59.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:50:59.850+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.851347+0000) 2019-10-13T05:50:59.855 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:50:59.851+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.851508+0000) 2019-10-13T05:50:59.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:50:59.847+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:50:59.848024+0000) 2019-10-13T05:50:59.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:50:59.853+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:00.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:00.624+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:00.624120+0000) 2019-10-13T05:51:01.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:01.834+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:01.835205+0000) 2019-10-13T05:51:02.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:02.701+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:02.702092+0000) 2019-10-13T05:51:02.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:02.701+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:02.702163+0000) 2019-10-13T05:51:04.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:04.196+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.196747+0000) 2019-10-13T05:51:04.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:04.854+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.855288+0000) 2019-10-13T05:51:04.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:04.855+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.855407+0000) 2019-10-13T05:51:04.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:04.849+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.850779+0000) 2019-10-13T05:51:04.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:04.855+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.855943+0000) 2019-10-13T05:51:04.860 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:04.855+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.856210+0000) 2019-10-13T05:51:04.860 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:04.856+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.856498+0000) 2019-10-13T05:51:04.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:04.856+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.856651+0000) 2019-10-13T05:51:04.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:04.851+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:04.853100+0000) 2019-10-13T05:51:04.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:04.858+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:08.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:08.241+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:08.241752+0000) 2019-10-13T05:51:08.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:08.581+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:08.582060+0000) 2019-10-13T05:51:08.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:08.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:08.582052+0000) 2019-10-13T05:51:09.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:09.860+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.861313+0000) 2019-10-13T05:51:09.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:09.861+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.861435+0000) 2019-10-13T05:51:09.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:09.861+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.861769+0000) 2019-10-13T05:51:09.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:09.855+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.856840+0000) 2019-10-13T05:51:09.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:09.861+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.861891+0000) 2019-10-13T05:51:09.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:09.862+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.862464+0000) 2019-10-13T05:51:09.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:09.861+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.862164+0000) 2019-10-13T05:51:09.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:09.857+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:09.859529+0000) 2019-10-13T05:51:09.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:09.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86434 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:10.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:10.625+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:10.626613+0000) 2019-10-13T05:51:11.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:11.834+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:11.835390+0000) 2019-10-13T05:51:12.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:12.702+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:12.702259+0000) 2019-10-13T05:51:12.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:12.701+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:12.702348+0000) 2019-10-13T05:51:14.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:14.195+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.196960+0000) 2019-10-13T05:51:14.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:14.865+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.866943+0000) 2019-10-13T05:51:14.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:14.865+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.867054+0000) 2019-10-13T05:51:14.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:14.860+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.862601+0000) 2019-10-13T05:51:14.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:14.866+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.867496+0000) 2019-10-13T05:51:14.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:14.866+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.867707+0000) 2019-10-13T05:51:14.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:14.866+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.868037+0000) 2019-10-13T05:51:14.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:14.866+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.868241+0000) 2019-10-13T05:51:14.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:14.862+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:14.864702+0000) 2019-10-13T05:51:14.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:14.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:18.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:18.240+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:18.241938+0000) 2019-10-13T05:51:18.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:18.581+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:18.582252+0000) 2019-10-13T05:51:18.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:18.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:18.582253+0000) 2019-10-13T05:51:18.926 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:51:18.927 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:18.934 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:51:18.934 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:19.877 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:19.871+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.872432+0000) 2019-10-13T05:51:19.878 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:19.871+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.872970+0000) 2019-10-13T05:51:19.878 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:19.871+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.873107+0000) 2019-10-13T05:51:19.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:19.871+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.873277+0000) 2019-10-13T05:51:19.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:19.872+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.873481+0000) 2019-10-13T05:51:19.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:19.872+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.874161+0000) 2019-10-13T05:51:19.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:19.866+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.867253+0000) 2019-10-13T05:51:19.879 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:19.868+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:19.869547+0000) 2019-10-13T05:51:19.879 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:19.874+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86495 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:20.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:20.628+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:20.628865+0000) 2019-10-13T05:51:21.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:21.834+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:21.835575+0000) 2019-10-13T05:51:22.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:22.701+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:22.702441+0000) 2019-10-13T05:51:22.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:22.701+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:22.702485+0000) 2019-10-13T05:51:24.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:24.196+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.197167+0000) 2019-10-13T05:51:24.881 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:24.876+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.877775+0000) 2019-10-13T05:51:24.881 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:24.876+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.877844+0000) 2019-10-13T05:51:24.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:24.872+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.873260+0000) 2019-10-13T05:51:24.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:24.877+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.878445+0000) 2019-10-13T05:51:24.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:24.876+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.878219+0000) 2019-10-13T05:51:24.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:24.877+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.878656+0000) 2019-10-13T05:51:24.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:24.877+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.879025+0000) 2019-10-13T05:51:24.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:24.874+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:24.875015+0000) 2019-10-13T05:51:24.885 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:24.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:28.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:28.241+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:28.242155+0000) 2019-10-13T05:51:28.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:28.581+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:28.582413+0000) 2019-10-13T05:51:28.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:28.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:28.582434+0000) 2019-10-13T05:51:29.886 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:29.881+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.883084+0000) 2019-10-13T05:51:29.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:29.881+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.883289+0000) 2019-10-13T05:51:29.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:29.882+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.883711+0000) 2019-10-13T05:51:29.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:29.882+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.884104+0000) 2019-10-13T05:51:29.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:29.878+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.879528+0000) 2019-10-13T05:51:29.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:29.883+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.884429+0000) 2019-10-13T05:51:29.888 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:29.883+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.884583+0000) 2019-10-13T05:51:29.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:29.880+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:29.880881+0000) 2019-10-13T05:51:29.889 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:29.885+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:30.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:30.630+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:30.631100+0000) 2019-10-13T05:51:31.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:31.835+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:31.835765+0000) 2019-10-13T05:51:32.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:32.701+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:32.702611+0000) 2019-10-13T05:51:32.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:32.702+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:32.702674+0000) 2019-10-13T05:51:34.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:34.196+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.197364+0000) 2019-10-13T05:51:34.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:34.886+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.887807+0000) 2019-10-13T05:51:34.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:34.882+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.883697+0000) 2019-10-13T05:51:34.892 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:34.886+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.888260+0000) 2019-10-13T05:51:34.892 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:34.887+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.888497+0000) 2019-10-13T05:51:34.892 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:34.887+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.888922+0000) 2019-10-13T05:51:34.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:34.887+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.889057+0000) 2019-10-13T05:51:34.893 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:34.884+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.885401+0000) 2019-10-13T05:51:34.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:34.888+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:34.889421+0000) 2019-10-13T05:51:34.894 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:34.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:38.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:38.241+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:38.242327+0000) 2019-10-13T05:51:38.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:38.581+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:38.582529+0000) 2019-10-13T05:51:38.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:38.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:38.582578+0000) 2019-10-13T05:51:39.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:39.890+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.892375+0000) 2019-10-13T05:51:39.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:39.891+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.892842+0000) 2019-10-13T05:51:39.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:39.891+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.893035+0000) 2019-10-13T05:51:39.910 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:39.891+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.893229+0000) 2019-10-13T05:51:39.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:39.892+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.893775+0000) 2019-10-13T05:51:39.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:39.892+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.894024+0000) 2019-10-13T05:51:39.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:39.888+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.888728+0000) 2019-10-13T05:51:39.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:39.890+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:39.890980+0000) 2019-10-13T05:51:39.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:39.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:40.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:40.632+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:40.633469+0000) 2019-10-13T05:51:41.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:41.835+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:41.835976+0000) 2019-10-13T05:51:42.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:42.702+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:42.702815+0000) 2019-10-13T05:51:42.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:42.702+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:42.702872+0000) 2019-10-13T05:51:44.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:44.197+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.197528+0000) 2019-10-13T05:51:44.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:44.893+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.894043+0000) 2019-10-13T05:51:44.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:44.897+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.897866+0000) 2019-10-13T05:51:44.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:44.897+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.898002+0000) 2019-10-13T05:51:44.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:44.898+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.898506+0000) 2019-10-13T05:51:44.903 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:44.898+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.899023+0000) 2019-10-13T05:51:44.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:44.898+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.899170+0000) 2019-10-13T05:51:44.904 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:44.899+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.899477+0000) 2019-10-13T05:51:44.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:44.895+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:44.896103+0000) 2019-10-13T05:51:44.904 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:44.901+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86654 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:48.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:48.242+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:48.242484+0000) 2019-10-13T05:51:48.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:48.582+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:48.582719+0000) 2019-10-13T05:51:48.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:48.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:48.582732+0000) 2019-10-13T05:51:49.040 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:51:49.040 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:49.044 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:51:49.045 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:49.907 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:49.903+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.903494+0000) 2019-10-13T05:51:49.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:49.903+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.903692+0000) 2019-10-13T05:51:49.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:49.903+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.903813+0000) 2019-10-13T05:51:49.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:49.903+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.903942+0000) 2019-10-13T05:51:49.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:49.903+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.904240+0000) 2019-10-13T05:51:49.908 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:49.904+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.904478+0000) 2019-10-13T05:51:49.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:49.899+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.900016+0000) 2019-10-13T05:51:49.909 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:49.905+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:49.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:49.900+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:49.901554+0000) 2019-10-13T05:51:50.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:50.635+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:50.635752+0000) 2019-10-13T05:51:51.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:51.835+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:51.836184+0000) 2019-10-13T05:51:52.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:52.702+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:52.703003+0000) 2019-10-13T05:51:52.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:52.702+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:52.703031+0000) 2019-10-13T05:51:54.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:54.197+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.197708+0000) 2019-10-13T05:51:54.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:54.902+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.902994+0000) 2019-10-13T05:51:54.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:54.907+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.908248+0000) 2019-10-13T05:51:54.912 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:54.907+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.908387+0000) 2019-10-13T05:51:54.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:54.908+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.908549+0000) 2019-10-13T05:51:54.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:54.908+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.908818+0000) 2019-10-13T05:51:54.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:54.908+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.908924+0000) 2019-10-13T05:51:54.913 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:54.908+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.909089+0000) 2019-10-13T05:51:54.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:54.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:54.914 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:54.906+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:54.906798+0000) 2019-10-13T05:51:58.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:58.242+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:58.242671+0000) 2019-10-13T05:51:58.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:58.582+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:58.582925+0000) 2019-10-13T05:51:58.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:58.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:58.582922+0000) 2019-10-13T05:51:59.916 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:51:59.911+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.913143+0000) 2019-10-13T05:51:59.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:51:59.907+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.908806+0000) 2019-10-13T05:51:59.917 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:51:59.912+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.913492+0000) 2019-10-13T05:51:59.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:51:59.912+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.913684+0000) 2019-10-13T05:51:59.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:51:59.912+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.914002+0000) 2019-10-13T05:51:59.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:51:59.912+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.914128+0000) 2019-10-13T05:51:59.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:51:59.913+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.914502+0000) 2019-10-13T05:51:59.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:51:59.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:51:59.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:51:59.910+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:51:59.911985+0000) 2019-10-13T05:52:00.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:00.637+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:00.638098+0000) 2019-10-13T05:52:01.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:01.835+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:01.836388+0000) 2019-10-13T05:52:02.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:02.702+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:02.703189+0000) 2019-10-13T05:52:02.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:02.702+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:02.703209+0000) 2019-10-13T05:52:04.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:04.196+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.197870+0000) 2019-10-13T05:52:04.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:04.916+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.917887+0000) 2019-10-13T05:52:04.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:04.917+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.918497+0000) 2019-10-13T05:52:04.922 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:04.917+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.918864+0000) 2019-10-13T05:52:04.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:04.912+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.914532+0000) 2019-10-13T05:52:04.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:04.917+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.919355+0000) 2019-10-13T05:52:04.923 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:04.918+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.919479+0000) 2019-10-13T05:52:04.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:04.918+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.919983+0000) 2019-10-13T05:52:04.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:04.914+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:04.916231+0000) 2019-10-13T05:52:04.924 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:04.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86778 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:08.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:08.242+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:08.242857+0000) 2019-10-13T05:52:08.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:08.582+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:08.583122+0000) 2019-10-13T05:52:08.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:08.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:08.583121+0000) 2019-10-13T05:52:09.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:09.921+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.923282+0000) 2019-10-13T05:52:09.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:09.918+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.918800+0000) 2019-10-13T05:52:09.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:09.922+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.923675+0000) 2019-10-13T05:52:09.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:09.922+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.923851+0000) 2019-10-13T05:52:09.928 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:09.922+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.923967+0000) 2019-10-13T05:52:09.928 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:09.922+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.924122+0000) 2019-10-13T05:52:09.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:09.922+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.924338+0000) 2019-10-13T05:52:09.935 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:09.925+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86807 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:09.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:09.921+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:09.922342+0000) 2019-10-13T05:52:10.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:10.639+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:10.640142+0000) 2019-10-13T05:52:11.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:11.835+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:11.836590+0000) 2019-10-13T05:52:12.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:12.702+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:12.703371+0000) 2019-10-13T05:52:12.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:12.702+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:12.703418+0000) 2019-10-13T05:52:14.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:14.197+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.198042+0000) 2019-10-13T05:52:14.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:14.927+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.929260+0000) 2019-10-13T05:52:14.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:14.927+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.929353+0000) 2019-10-13T05:52:14.933 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:14.928+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.929602+0000) 2019-10-13T05:52:14.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:14.928+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.929731+0000) 2019-10-13T05:52:14.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:14.928+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.929833+0000) 2019-10-13T05:52:14.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:14.924+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.924792+0000) 2019-10-13T05:52:14.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:14.925+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.926320+0000) 2019-10-13T05:52:14.934 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:14.928+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:14.930070+0000) 2019-10-13T05:52:14.935 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:14.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:18.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:18.242+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:18.243067+0000) 2019-10-13T05:52:18.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:18.582+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:18.583269+0000) 2019-10-13T05:52:18.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:18.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:18.583254+0000) 2019-10-13T05:52:19.204 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:52:19.204 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:19.208 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:52:19.208 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:19.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:19.928+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.929635+0000) 2019-10-13T05:52:19.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:19.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.934747+0000) 2019-10-13T05:52:19.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:19.933+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.934965+0000) 2019-10-13T05:52:19.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:19.934+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.935444+0000) 2019-10-13T05:52:19.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:19.933+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.935353+0000) 2019-10-13T05:52:19.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:19.934+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.935607+0000) 2019-10-13T05:52:19.939 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:19.934+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.935853+0000) 2019-10-13T05:52:19.940 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:19.931+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:19.932062+0000) 2019-10-13T05:52:19.941 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:19.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86868 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:20.650 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:20.641+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:20.642469+0000) 2019-10-13T05:52:21.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:21.835+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:21.836781+0000) 2019-10-13T05:52:22.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:22.703+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:22.703566+0000) 2019-10-13T05:52:22.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:22.703+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:22.703596+0000) 2019-10-13T05:52:24.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:24.197+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.198218+0000) 2019-10-13T05:52:24.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:24.938+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.940069+0000) 2019-10-13T05:52:24.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:24.939+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.940554+0000) 2019-10-13T05:52:24.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:24.939+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.940748+0000) 2019-10-13T05:52:24.944 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:24.939+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.940898+0000) 2019-10-13T05:52:24.944 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:24.939+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.941161+0000) 2019-10-13T05:52:24.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:24.935+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.936344+0000) 2019-10-13T05:52:24.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:24.940+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.941405+0000) 2019-10-13T05:52:24.946 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:24.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86902 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:24.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:24.938+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:24.939255+0000) 2019-10-13T05:52:28.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:28.242+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:28.243229+0000) 2019-10-13T05:52:28.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:28.583+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:28.583447+0000) 2019-10-13T05:52:28.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:28.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:28.583437+0000) 2019-10-13T05:52:29.947 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:29.943+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.944292+0000) 2019-10-13T05:52:29.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:29.939+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.940196+0000) 2019-10-13T05:52:29.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:29.944+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.944588+0000) 2019-10-13T05:52:29.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:29.944+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.944745+0000) 2019-10-13T05:52:29.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:29.944+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.945157+0000) 2019-10-13T05:52:29.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:29.944+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.945338+0000) 2019-10-13T05:52:29.949 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:29.945+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.945587+0000) 2019-10-13T05:52:29.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:29.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:29.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:29.943+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:29.943793+0000) 2019-10-13T05:52:30.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:30.643+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:30.644706+0000) 2019-10-13T05:52:31.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:31.836+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:31.836988+0000) 2019-10-13T05:52:32.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:32.703+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:32.703732+0000) 2019-10-13T05:52:32.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:32.703+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:32.703781+0000) 2019-10-13T05:52:34.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:34.198+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.198421+0000) 2019-10-13T05:52:34.952 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:34.948+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.948805+0000) 2019-10-13T05:52:34.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:34.943+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.944695+0000) 2019-10-13T05:52:34.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:34.948+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.948961+0000) 2019-10-13T05:52:34.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:34.948+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.949180+0000) 2019-10-13T05:52:34.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:34.949+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.949507+0000) 2019-10-13T05:52:34.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:34.949+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.949740+0000) 2019-10-13T05:52:34.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:34.949+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.950080+0000) 2019-10-13T05:52:34.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:34.946+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:34.946968+0000) 2019-10-13T05:52:34.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:34.952+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:38.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:38.243+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:38.243422+0000) 2019-10-13T05:52:38.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:38.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:38.583596+0000) 2019-10-13T05:52:38.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:38.583+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:38.583632+0000) 2019-10-13T05:52:39.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:39.953+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.954370+0000) 2019-10-13T05:52:39.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:39.954+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.954686+0000) 2019-10-13T05:52:39.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:39.954+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.954927+0000) 2019-10-13T05:52:39.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:39.954+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.955062+0000) 2019-10-13T05:52:39.960 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:39.954+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.955280+0000) 2019-10-13T05:52:39.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:39.955+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.955478+0000) 2019-10-13T05:52:39.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:39.950+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.950844+0000) 2019-10-13T05:52:39.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:39.951+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:39.952279+0000) 2019-10-13T05:52:39.962 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:39.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 86992 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:40.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:40.646+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:40.647043+0000) 2019-10-13T05:52:41.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:41.836+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:41.837172+0000) 2019-10-13T05:52:42.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:42.703+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:42.703920+0000) 2019-10-13T05:52:42.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:42.703+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:42.703968+0000) 2019-10-13T05:52:44.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:44.198+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.198597+0000) 2019-10-13T05:52:44.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:44.954+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.955330+0000) 2019-10-13T05:52:44.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:44.960+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.960506+0000) 2019-10-13T05:52:44.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:44.960+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.960738+0000) 2019-10-13T05:52:44.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:44.960+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.960836+0000) 2019-10-13T05:52:44.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:44.960+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.961060+0000) 2019-10-13T05:52:44.965 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:44.960+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.961034+0000) 2019-10-13T05:52:44.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:44.961+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.961456+0000) 2019-10-13T05:52:44.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:44.957+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:44.957778+0000) 2019-10-13T05:52:44.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:44.963+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87026 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:48.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:48.242+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:48.243637+0000) 2019-10-13T05:52:48.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:48.582+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:48.583780+0000) 2019-10-13T05:52:48.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:48.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:48.583773+0000) 2019-10-13T05:52:49.359 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:52:49.359 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:49.364 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:52:49.364 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:49.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:49.963+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.965165+0000) 2019-10-13T05:52:49.969 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:49.964+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.965526+0000) 2019-10-13T05:52:49.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:49.964+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.965508+0000) 2019-10-13T05:52:49.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:49.964+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.966116+0000) 2019-10-13T05:52:49.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:49.964+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.966331+0000) 2019-10-13T05:52:49.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:49.959+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.961461+0000) 2019-10-13T05:52:49.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:49.965+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.966608+0000) 2019-10-13T05:52:49.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:49.961+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:49.963520+0000) 2019-10-13T05:52:49.971 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:49.967+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:50.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:50.648+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:50.649495+0000) 2019-10-13T05:52:51.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:51.836+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:51.837392+0000) 2019-10-13T05:52:52.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:52.703+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:52.704128+0000) 2019-10-13T05:52:52.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:52.703+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:52.704157+0000) 2019-10-13T05:52:54.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:54.197+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.198772+0000) 2019-10-13T05:52:54.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:54.968+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.970054+0000) 2019-10-13T05:52:54.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:54.964+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.965787+0000) 2019-10-13T05:52:54.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:54.969+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.970823+0000) 2019-10-13T05:52:54.975 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:54.969+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.970673+0000) 2019-10-13T05:52:54.975 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:54.969+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.971061+0000) 2019-10-13T05:52:54.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:54.969+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.971317+0000) 2019-10-13T05:52:54.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:54.970+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.971471+0000) 2019-10-13T05:52:54.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:54.965+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:54.967639+0000) 2019-10-13T05:52:54.977 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:54.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87087 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:58.247 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:58.243+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:58.243765+0000) 2019-10-13T05:52:58.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:58.582+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:58.583957+0000) 2019-10-13T05:52:58.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:58.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:58.583962+0000) 2019-10-13T05:52:59.978 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:52:59.974+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.975394+0000) 2019-10-13T05:52:59.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:52:59.974+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.975838+0000) 2019-10-13T05:52:59.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:52:59.974+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.976076+0000) 2019-10-13T05:52:59.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:52:59.969+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.971537+0000) 2019-10-13T05:52:59.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:52:59.975+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.976551+0000) 2019-10-13T05:52:59.980 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:52:59.975+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.976899+0000) 2019-10-13T05:52:59.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:52:59.975+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.977046+0000) 2019-10-13T05:52:59.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:52:59.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:52:59.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:52:59.972+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:52:59.973782+0000) 2019-10-13T05:53:00.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:00.650+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:00.651754+0000) 2019-10-13T05:53:01.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:01.836+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:01.837560+0000) 2019-10-13T05:53:02.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:02.703+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:02.704294+0000) 2019-10-13T05:53:02.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:02.703+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:02.704318+0000) 2019-10-13T05:53:04.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:04.198+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.198978+0000) 2019-10-13T05:53:04.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:04.979+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.981172+0000) 2019-10-13T05:53:04.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:04.979+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.981376+0000) 2019-10-13T05:53:04.985 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:04.980+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.981973+0000) 2019-10-13T05:53:04.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:04.980+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.982299+0000) 2019-10-13T05:53:04.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:04.977+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.977801+0000) 2019-10-13T05:53:04.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:04.981+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.982715+0000) 2019-10-13T05:53:04.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:04.981+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.982905+0000) 2019-10-13T05:53:04.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:04.978+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:04.979052+0000) 2019-10-13T05:53:04.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:04.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:08.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:08.243+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:08.243923+0000) 2019-10-13T05:53:08.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:08.583+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:08.584100+0000) 2019-10-13T05:53:08.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:08.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:08.584098+0000) 2019-10-13T05:53:09.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:09.984+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.986015+0000) 2019-10-13T05:53:09.990 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:09.985+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.986443+0000) 2019-10-13T05:53:09.990 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:09.985+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.986788+0000) 2019-10-13T05:53:09.990 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:09.985+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.986592+0000) 2019-10-13T05:53:09.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:09.985+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.986993+0000) 2019-10-13T05:53:09.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:09.981+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.982501+0000) 2019-10-13T05:53:09.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:09.985+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.987274+0000) 2019-10-13T05:53:09.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:09.983+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:09.984512+0000) 2019-10-13T05:53:09.992 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:09.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:10.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:10.653+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:10.654174+0000) 2019-10-13T05:53:11.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:11.837+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:11.837729+0000) 2019-10-13T05:53:12.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:12.704+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:12.704472+0000) 2019-10-13T05:53:12.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:12.703+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:12.704484+0000) 2019-10-13T05:53:14.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:14.198+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.199167+0000) 2019-10-13T05:53:14.995 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:14.991+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.991468+0000) 2019-10-13T05:53:14.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:14.991+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.991510+0000) 2019-10-13T05:53:14.995 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:14.991+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.991682+0000) 2019-10-13T05:53:14.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:14.991+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.992077+0000) 2019-10-13T05:53:14.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:14.986+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.987275+0000) 2019-10-13T05:53:14.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:14.991+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.992349+0000) 2019-10-13T05:53:14.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:14.992+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.992613+0000) 2019-10-13T05:53:14.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:14.988+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:14.989527+0000) 2019-10-13T05:53:14.998 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:14.994+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:18.247 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:18.243+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:18.244116+0000) 2019-10-13T05:53:18.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:18.583+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:18.584261+0000) 2019-10-13T05:53:18.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:18.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:18.584269+0000) 2019-10-13T05:53:19.531 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:53:19.532 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:19.536 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:53:19.536 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:19.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:19.991+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.991995+0000) 2019-10-13T05:53:20.001 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:19.996+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.997018+0000) 2019-10-13T05:53:20.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:19.996+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.997055+0000) 2019-10-13T05:53:20.001 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:19.997+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.997509+0000) 2019-10-13T05:53:20.001 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:19.997+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.997509+0000) 2019-10-13T05:53:20.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:19.997+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.997726+0000) 2019-10-13T05:53:20.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:19.997+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.997918+0000) 2019-10-13T05:53:20.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:19.993+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:19.994728+0000) 2019-10-13T05:53:20.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:20.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87240 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:20.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:20.656+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:20.656458+0000) 2019-10-13T05:53:21.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:21.837+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:21.837931+0000) 2019-10-13T05:53:22.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:22.704+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:22.704565+0000) 2019-10-13T05:53:22.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:22.704+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:22.704646+0000) 2019-10-13T05:53:24.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:24.199+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:24.199299+0000) 2019-10-13T05:53:25.005 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:25.002+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:25.002420+0000) 2019-10-13T05:53:25.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:24.997+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:24.998427+0000) 2019-10-13T05:53:25.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:25.002+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:25.002845+0000) 2019-10-13T05:53:25.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:25.002+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:25.002934+0000) 2019-10-13T05:53:25.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:25.002+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:25.003200+0000) 2019-10-13T05:53:25.009 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:25.003+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:25.003392+0000) 2019-10-13T05:53:25.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:25.003+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:25.003582+0000) 2019-10-13T05:53:25.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:25.000+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:25.001138+0000) 2019-10-13T05:53:25.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:25.005+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87274 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:28.247 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:28.244+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:28.244361+0000) 2019-10-13T05:53:28.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:28.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:28.584430+0000) 2019-10-13T05:53:28.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:28.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:28.584431+0000) 2019-10-13T05:53:30.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:30.007+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.008039+0000) 2019-10-13T05:53:30.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:30.007+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.008276+0000) 2019-10-13T05:53:30.018 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:30.008+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.008553+0000) 2019-10-13T05:53:30.018 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:30.008+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.008590+0000) 2019-10-13T05:53:30.018 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:30.008+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.009053+0000) 2019-10-13T05:53:30.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:30.008+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.009242+0000) 2019-10-13T05:53:30.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:30.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87303 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:30.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:30.002+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.003240+0000) 2019-10-13T05:53:30.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:30.005+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.005820+0000) 2019-10-13T05:53:30.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:30.658+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:30.658982+0000) 2019-10-13T05:53:31.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:31.836+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:31.838102+0000) 2019-10-13T05:53:32.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:32.703+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:32.704740+0000) 2019-10-13T05:53:32.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:32.704+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:32.704826+0000) 2019-10-13T05:53:34.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:34.198+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:34.199480+0000) 2019-10-13T05:53:35.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:35.012+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.013985+0000) 2019-10-13T05:53:35.018 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:35.012+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.014320+0000) 2019-10-13T05:53:35.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:35.013+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.014494+0000) 2019-10-13T05:53:35.018 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:35.013+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.014667+0000) 2019-10-13T05:53:35.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:35.008+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.009536+0000) 2019-10-13T05:53:35.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:35.010+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.011235+0000) 2019-10-13T05:53:35.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:35.013+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.014977+0000) 2019-10-13T05:53:35.020 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:35.013+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:35.014942+0000) 2019-10-13T05:53:35.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:35.015+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:38.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:38.243+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:38.244581+0000) 2019-10-13T05:53:38.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:38.583+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:38.584554+0000) 2019-10-13T05:53:38.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:38.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:38.584623+0000) 2019-10-13T05:53:40.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:40.012+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.014294+0000) 2019-10-13T05:53:40.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:40.017+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.019216+0000) 2019-10-13T05:53:40.023 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:40.018+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.019438+0000) 2019-10-13T05:53:40.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:40.018+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.019719+0000) 2019-10-13T05:53:40.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:40.018+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.019984+0000) 2019-10-13T05:53:40.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:40.018+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.020110+0000) 2019-10-13T05:53:40.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:40.018+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.020243+0000) 2019-10-13T05:53:40.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:40.015+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.016782+0000) 2019-10-13T05:53:40.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:40.020+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87364 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:40.669 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:40.660+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:40.661504+0000) 2019-10-13T05:53:41.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:41.837+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:41.838261+0000) 2019-10-13T05:53:42.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:42.704+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:42.704875+0000) 2019-10-13T05:53:42.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:42.703+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:42.705005+0000) 2019-10-13T05:53:44.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:44.198+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:44.199649+0000) 2019-10-13T05:53:45.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:45.017+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.019156+0000) 2019-10-13T05:53:45.027 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:45.022+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.024033+0000) 2019-10-13T05:53:45.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:45.022+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.024338+0000) 2019-10-13T05:53:45.028 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:45.023+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.024550+0000) 2019-10-13T05:53:45.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:45.023+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.024863+0000) 2019-10-13T05:53:45.029 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:45.023+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.025025+0000) 2019-10-13T05:53:45.029 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:45.023+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.025122+0000) 2019-10-13T05:53:45.029 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:45.020+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:45.022100+0000) 2019-10-13T05:53:45.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:45.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:48.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:48.244+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:48.244773+0000) 2019-10-13T05:53:48.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:48.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:48.584745+0000) 2019-10-13T05:53:48.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:48.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:48.584844+0000) 2019-10-13T05:53:49.620 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:53:49.620 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:49.625 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:53:49.625 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:50.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:50.028+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.029777+0000) 2019-10-13T05:53:50.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:50.023+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.025418+0000) 2019-10-13T05:53:50.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:50.028+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.030288+0000) 2019-10-13T05:53:50.034 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:50.029+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.030515+0000) 2019-10-13T05:53:50.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:50.029+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.030701+0000) 2019-10-13T05:53:50.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:50.029+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.030960+0000) 2019-10-13T05:53:50.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:50.029+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.031139+0000) 2019-10-13T05:53:50.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:50.026+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.028035+0000) 2019-10-13T05:53:50.036 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:50.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:50.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:50.662+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:50.663721+0000) 2019-10-13T05:53:51.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:51.837+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:51.838431+0000) 2019-10-13T05:53:52.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:52.704+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:52.705064+0000) 2019-10-13T05:53:52.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:52.704+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:52.705167+0000) 2019-10-13T05:53:54.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:54.199+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:54.199823+0000) 2019-10-13T05:53:55.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:55.034+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.035783+0000) 2019-10-13T05:53:55.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:55.034+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.035950+0000) 2019-10-13T05:53:55.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:53:55.034+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.036144+0000) 2019-10-13T05:53:55.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:53:55.030+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.031631+0000) 2019-10-13T05:53:55.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:53:55.031+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.032565+0000) 2019-10-13T05:53:55.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:55.035+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.036439+0000) 2019-10-13T05:53:55.041 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:53:55.035+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.036906+0000) 2019-10-13T05:53:55.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:53:55.035+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:55.037161+0000) 2019-10-13T05:53:55.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:53:55.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:53:58.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:53:58.244+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:58.244961+0000) 2019-10-13T05:53:58.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:53:58.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:58.584953+0000) 2019-10-13T05:53:58.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:53:58.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:53:58.585009+0000) 2019-10-13T05:54:00.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:00.040+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.040683+0000) 2019-10-13T05:54:00.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:00.040+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.040856+0000) 2019-10-13T05:54:00.044 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:00.040+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.041055+0000) 2019-10-13T05:54:00.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:00.036+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.036816+0000) 2019-10-13T05:54:00.045 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:00.040+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.041337+0000) 2019-10-13T05:54:00.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:00.041+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.041710+0000) 2019-10-13T05:54:00.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:00.041+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.041944+0000) 2019-10-13T05:54:00.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:00.038+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.038784+0000) 2019-10-13T05:54:00.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:00.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87488 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:00.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:00.665+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:00.666017+0000) 2019-10-13T05:54:01.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:01.838+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:01.838637+0000) 2019-10-13T05:54:02.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:02.704+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:02.705195+0000) 2019-10-13T05:54:02.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:02.704+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:02.705354+0000) 2019-10-13T05:54:04.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:04.199+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:04.200008+0000) 2019-10-13T05:54:05.050 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:05.046+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.046393+0000) 2019-10-13T05:54:05.050 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:05.046+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.046659+0000) 2019-10-13T05:54:05.050 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:05.046+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.046867+0000) 2019-10-13T05:54:05.050 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:05.046+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.046983+0000) 2019-10-13T05:54:05.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:05.041+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.042427+0000) 2019-10-13T05:54:05.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:05.046+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.047307+0000) 2019-10-13T05:54:05.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:05.047+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.047573+0000) 2019-10-13T05:54:05.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:05.049+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:05.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:05.044+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:05.045451+0000) 2019-10-13T05:54:08.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:08.244+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:08.245138+0000) 2019-10-13T05:54:08.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:08.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:08.585137+0000) 2019-10-13T05:54:08.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:08.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:08.585162+0000) 2019-10-13T05:54:10.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:10.046+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.047093+0000) 2019-10-13T05:54:10.055 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:10.051+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.051602+0000) 2019-10-13T05:54:10.056 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:10.051+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.052211+0000) 2019-10-13T05:54:10.056 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:10.052+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.052462+0000) 2019-10-13T05:54:10.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:10.052+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.052408+0000) 2019-10-13T05:54:10.057 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:10.048+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.049371+0000) 2019-10-13T05:54:10.057 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:10.052+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.052635+0000) 2019-10-13T05:54:10.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:10.052+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.052690+0000) 2019-10-13T05:54:10.058 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:10.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:10.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:10.667+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:10.668439+0000) 2019-10-13T05:54:11.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:11.838+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:11.838793+0000) 2019-10-13T05:54:12.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:12.705+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:12.705363+0000) 2019-10-13T05:54:12.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:12.705+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:12.705518+0000) 2019-10-13T05:54:14.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:14.200+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:14.200190+0000) 2019-10-13T05:54:15.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:15.051+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.051961+0000) 2019-10-13T05:54:15.061 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:15.056+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.056983+0000) 2019-10-13T05:54:15.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:15.056+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.057198+0000) 2019-10-13T05:54:15.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:15.056+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.057343+0000) 2019-10-13T05:54:15.061 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:15.057+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.057628+0000) 2019-10-13T05:54:15.061 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:15.057+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.057693+0000) 2019-10-13T05:54:15.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:15.057+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.057968+0000) 2019-10-13T05:54:15.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:15.053+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:15.054147+0000) 2019-10-13T05:54:15.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:15.058+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:18.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:18.244+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:18.245297+0000) 2019-10-13T05:54:18.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:18.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:18.585329+0000) 2019-10-13T05:54:18.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:18.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:18.585383+0000) 2019-10-13T05:54:19.702 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:54:19.703 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:19.707 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:54:19.707 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:20.064 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:20.059+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.061024+0000) 2019-10-13T05:54:20.064 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:20.059+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.061225+0000) 2019-10-13T05:54:20.065 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:20.060+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.061454+0000) 2019-10-13T05:54:20.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:20.055+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.056653+0000) 2019-10-13T05:54:20.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:20.060+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.061868+0000) 2019-10-13T05:54:20.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:20.060+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.062045+0000) 2019-10-13T05:54:20.066 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:20.060+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.062268+0000) 2019-10-13T05:54:20.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:20.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:20.067 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:20.058+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.059395+0000) 2019-10-13T05:54:20.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:20.670+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:20.670808+0000) 2019-10-13T05:54:21.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:21.838+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:21.838960+0000) 2019-10-13T05:54:22.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:22.704+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:22.705576+0000) 2019-10-13T05:54:22.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:22.705+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:22.705693+0000) 2019-10-13T05:54:24.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:24.199+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:24.200390+0000) 2019-10-13T05:54:25.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:25.064+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.065966+0000) 2019-10-13T05:54:25.069 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:25.064+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.066134+0000) 2019-10-13T05:54:25.070 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:25.064+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.066345+0000) 2019-10-13T05:54:25.070 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:25.065+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.066482+0000) 2019-10-13T05:54:25.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:25.061+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.062571+0000) 2019-10-13T05:54:25.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:25.065+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.067046+0000) 2019-10-13T05:54:25.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:25.066+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.067588+0000) 2019-10-13T05:54:25.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:25.062+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:25.063258+0000) 2019-10-13T05:54:25.072 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:25.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:28.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:28.244+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:28.245503+0000) 2019-10-13T05:54:28.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:28.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:28.585491+0000) 2019-10-13T05:54:28.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:28.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:28.585545+0000) 2019-10-13T05:54:30.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:30.065+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.066324+0000) 2019-10-13T05:54:30.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:30.069+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.071304+0000) 2019-10-13T05:54:30.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:30.070+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.071699+0000) 2019-10-13T05:54:30.076 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:30.070+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.071748+0000) 2019-10-13T05:54:30.076 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:30.070+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.072102+0000) 2019-10-13T05:54:30.076 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:30.070+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.072319+0000) 2019-10-13T05:54:30.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:30.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.072420+0000) 2019-10-13T05:54:30.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:30.068+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.069543+0000) 2019-10-13T05:54:30.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:30.073+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:30.681 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:30.672+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:30.673104+0000) 2019-10-13T05:54:31.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:31.838+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:31.839148+0000) 2019-10-13T05:54:32.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:32.705+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:32.705767+0000) 2019-10-13T05:54:32.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:32.704+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:32.705900+0000) 2019-10-13T05:54:34.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:34.199+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:34.200616+0000) 2019-10-13T05:54:35.079 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:35.074+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.076194+0000) 2019-10-13T05:54:35.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:35.070+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.072022+0000) 2019-10-13T05:54:35.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:35.075+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.076538+0000) 2019-10-13T05:54:35.081 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:35.075+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.076918+0000) 2019-10-13T05:54:35.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:35.075+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.077007+0000) 2019-10-13T05:54:35.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:35.075+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.077211+0000) 2019-10-13T05:54:35.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:35.076+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.077539+0000) 2019-10-13T05:54:35.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:35.071+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:35.073429+0000) 2019-10-13T05:54:35.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:35.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:38.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:38.245+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:38.245701+0000) 2019-10-13T05:54:38.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:38.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:38.585679+0000) 2019-10-13T05:54:38.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:38.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:38.585697+0000) 2019-10-13T05:54:40.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:40.074+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.076432+0000) 2019-10-13T05:54:40.085 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:40.080+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.081693+0000) 2019-10-13T05:54:40.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:40.080+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.081782+0000) 2019-10-13T05:54:40.086 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:40.080+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.081986+0000) 2019-10-13T05:54:40.086 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:40.080+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.082178+0000) 2019-10-13T05:54:40.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:40.080+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.082298+0000) 2019-10-13T05:54:40.086 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:40.081+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.082469+0000) 2019-10-13T05:54:40.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:40.077+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.078826+0000) 2019-10-13T05:54:40.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:40.083+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87736 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:40.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:40.674+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:40.675461+0000) 2019-10-13T05:54:41.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:41.838+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:41.839359+0000) 2019-10-13T05:54:42.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:42.705+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:42.705974+0000) 2019-10-13T05:54:42.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:42.705+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:42.706084+0000) 2019-10-13T05:54:44.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:44.200+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:44.200789+0000) 2019-10-13T05:54:45.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:45.080+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.082402+0000) 2019-10-13T05:54:45.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:45.087+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.087436+0000) 2019-10-13T05:54:45.091 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:45.086+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.087314+0000) 2019-10-13T05:54:45.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:45.087+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.087522+0000) 2019-10-13T05:54:45.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:45.087+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.087679+0000) 2019-10-13T05:54:45.092 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:45.087+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.087832+0000) 2019-10-13T05:54:45.092 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:45.087+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.088053+0000) 2019-10-13T05:54:45.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:45.082+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:45.084248+0000) 2019-10-13T05:54:45.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:45.090+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87770 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:48.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:48.245+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:48.245895+0000) 2019-10-13T05:54:48.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:48.585+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:48.585846+0000) 2019-10-13T05:54:48.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:48.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:48.585854+0000) 2019-10-13T05:54:49.790 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:54:49.791 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:49.795 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:54:49.795 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:50.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:50.092+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.092950+0000) 2019-10-13T05:54:50.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:50.088+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.088754+0000) 2019-10-13T05:54:50.097 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:50.092+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.093225+0000) 2019-10-13T05:54:50.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:50.092+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.093374+0000) 2019-10-13T05:54:50.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:50.093+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.093593+0000) 2019-10-13T05:54:50.098 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:50.093+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.093843+0000) 2019-10-13T05:54:50.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:50.094+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.094391+0000) 2019-10-13T05:54:50.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:50.095+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:50.099 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:50.091+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.091952+0000) 2019-10-13T05:54:50.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:50.677+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:50.677792+0000) 2019-10-13T05:54:51.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:51.839+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:51.839525+0000) 2019-10-13T05:54:52.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:52.705+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:52.706137+0000) 2019-10-13T05:54:52.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:52.705+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:52.706266+0000) 2019-10-13T05:54:54.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:54.200+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:54.200948+0000) 2019-10-13T05:54:55.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:54:55.092+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.093454+0000) 2019-10-13T05:54:55.102 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:55.097+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.098218+0000) 2019-10-13T05:54:55.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:54:55.098+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.098603+0000) 2019-10-13T05:54:55.102 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:55.098+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.098616+0000) 2019-10-13T05:54:55.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:54:55.098+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.098809+0000) 2019-10-13T05:54:55.103 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:55.098+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.098869+0000) 2019-10-13T05:54:55.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:54:55.098+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.099245+0000) 2019-10-13T05:54:55.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:54:55.095+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:55.096334+0000) 2019-10-13T05:54:55.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:54:55.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:54:58.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:54:58.245+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:58.246085+0000) 2019-10-13T05:54:58.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:54:58.585+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:58.586047+0000) 2019-10-13T05:54:58.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:54:58.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:54:58.586072+0000) 2019-10-13T05:55:00.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:00.102+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.103154+0000) 2019-10-13T05:55:00.106 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:00.102+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.103313+0000) 2019-10-13T05:55:00.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:00.098+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.099047+0000) 2019-10-13T05:55:00.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:00.103+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.103886+0000) 2019-10-13T05:55:00.108 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:00.103+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.104003+0000) 2019-10-13T05:55:00.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:00.103+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.104377+0000) 2019-10-13T05:55:00.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:00.103+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.104201+0000) 2019-10-13T05:55:00.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:00.100+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.100990+0000) 2019-10-13T05:55:00.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:00.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87860 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:00.688 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:00.679+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:00.680172+0000) 2019-10-13T05:55:01.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:01.839+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:01.839686+0000) 2019-10-13T05:55:02.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:02.705+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:02.706297+0000) 2019-10-13T05:55:02.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:02.705+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:02.706454+0000) 2019-10-13T05:55:04.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:04.200+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:04.201075+0000) 2019-10-13T05:55:05.112 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:05.107+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.108833+0000) 2019-10-13T05:55:05.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:05.107+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.109008+0000) 2019-10-13T05:55:05.112 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:05.107+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.109368+0000) 2019-10-13T05:55:05.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:05.104+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.104878+0000) 2019-10-13T05:55:05.113 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:05.108+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.109903+0000) 2019-10-13T05:55:05.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:05.108+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.110057+0000) 2019-10-13T05:55:05.114 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:05.108+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.110305+0000) 2019-10-13T05:55:05.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:05.106+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:05.107172+0000) 2019-10-13T05:55:05.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:05.111+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87894 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:08.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:08.245+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:08.246273+0000) 2019-10-13T05:55:08.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:08.585+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:08.586203+0000) 2019-10-13T05:55:08.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:08.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:08.586239+0000) 2019-10-13T05:55:10.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:10.112+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.114079+0000) 2019-10-13T05:55:10.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:10.113+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.114638+0000) 2019-10-13T05:55:10.118 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:10.113+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.114784+0000) 2019-10-13T05:55:10.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:10.113+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.114946+0000) 2019-10-13T05:55:10.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:10.113+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.115083+0000) 2019-10-13T05:55:10.119 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:10.110+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.110756+0000) 2019-10-13T05:55:10.119 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:10.114+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.115770+0000) 2019-10-13T05:55:10.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:10.116+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:10.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:10.112+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.113092+0000) 2019-10-13T05:55:10.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:10.682+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:10.682716+0000) 2019-10-13T05:55:11.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:11.838+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:11.839853+0000) 2019-10-13T05:55:12.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:12.705+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:12.706454+0000) 2019-10-13T05:55:12.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:12.706+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:12.706658+0000) 2019-10-13T05:55:14.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:14.200+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:14.201283+0000) 2019-10-13T05:55:15.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:15.118+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.119575+0000) 2019-10-13T05:55:15.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:15.118+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.119622+0000) 2019-10-13T05:55:15.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:15.118+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.119887+0000) 2019-10-13T05:55:15.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:15.118+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.120183+0000) 2019-10-13T05:55:15.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:15.118+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.120362+0000) 2019-10-13T05:55:15.126 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:15.119+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.120491+0000) 2019-10-13T05:55:15.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:15.113+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.114629+0000) 2019-10-13T05:55:15.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:15.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87955 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:15.127 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:15.116+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:15.117548+0000) 2019-10-13T05:55:18.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:18.245+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:18.246471+0000) 2019-10-13T05:55:18.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:18.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:18.586377+0000) 2019-10-13T05:55:18.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:18.584+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:18.586367+0000) 2019-10-13T05:55:19.897 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:55:19.897 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:19.902 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:55:19.902 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:20.128 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:20.123+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.124485+0000) 2019-10-13T05:55:20.128 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:20.123+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.124759+0000) 2019-10-13T05:55:20.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:20.123+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.124905+0000) 2019-10-13T05:55:20.129 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:20.123+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.125207+0000) 2019-10-13T05:55:20.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:20.119+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.120127+0000) 2019-10-13T05:55:20.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:20.124+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.125495+0000) 2019-10-13T05:55:20.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:20.124+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.125652+0000) 2019-10-13T05:55:20.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:20.121+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.122548+0000) 2019-10-13T05:55:20.131 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:20.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 87985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:20.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:20.685+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:20.685185+0000) 2019-10-13T05:55:21.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:21.839+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:21.840037+0000) 2019-10-13T05:55:22.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:22.706+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:22.706577+0000) 2019-10-13T05:55:22.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:22.706+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:22.706842+0000) 2019-10-13T05:55:24.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:24.200+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:24.201440+0000) 2019-10-13T05:55:25.133 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:25.128+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.129756+0000) 2019-10-13T05:55:25.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:25.123+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.125287+0000) 2019-10-13T05:55:25.134 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:25.128+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.130327+0000) 2019-10-13T05:55:25.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:25.129+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.130535+0000) 2019-10-13T05:55:25.134 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:25.129+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.130850+0000) 2019-10-13T05:55:25.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:25.129+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.130962+0000) 2019-10-13T05:55:25.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:25.125+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.127280+0000) 2019-10-13T05:55:25.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:25.130+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:25.131528+0000) 2019-10-13T05:55:25.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:25.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:28.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:28.246+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:28.246668+0000) 2019-10-13T05:55:28.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:28.586+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:28.586538+0000) 2019-10-13T05:55:28.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:28.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:28.586546+0000) 2019-10-13T05:55:30.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:30.134+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.135171+0000) 2019-10-13T05:55:30.139 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:30.135+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.135652+0000) 2019-10-13T05:55:30.139 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:30.134+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.134782+0000) 2019-10-13T05:55:30.140 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:30.129+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.131483+0000) 2019-10-13T05:55:30.140 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:30.135+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.136374+0000) 2019-10-13T05:55:30.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:30.136+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.136486+0000) 2019-10-13T05:55:30.140 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:30.136+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.136944+0000) 2019-10-13T05:55:30.141 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:30.131+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.133539+0000) 2019-10-13T05:55:30.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:30.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:30.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:30.686+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:30.687753+0000) 2019-10-13T05:55:31.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:31.839+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:31.840174+0000) 2019-10-13T05:55:32.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:32.706+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:32.706690+0000) 2019-10-13T05:55:32.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:32.706+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:32.707015+0000) 2019-10-13T05:55:34.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:34.201+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:34.201611+0000) 2019-10-13T05:55:35.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:35.139+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.140358+0000) 2019-10-13T05:55:35.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:35.140+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.140609+0000) 2019-10-13T05:55:35.144 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:35.140+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.140779+0000) 2019-10-13T05:55:35.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:35.140+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.140968+0000) 2019-10-13T05:55:35.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:35.140+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.141172+0000) 2019-10-13T05:55:35.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:35.140+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.141293+0000) 2019-10-13T05:55:35.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:35.135+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.136848+0000) 2019-10-13T05:55:35.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:35.136+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:35.138268+0000) 2019-10-13T05:55:35.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:35.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88079 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:38.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:38.246+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:38.246880+0000) 2019-10-13T05:55:38.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:38.586+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:38.586733+0000) 2019-10-13T05:55:38.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:38.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:38.586751+0000) 2019-10-13T05:55:40.149 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:40.144+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.144912+0000) 2019-10-13T05:55:40.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:40.144+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.145128+0000) 2019-10-13T05:55:40.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:40.144+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.145276+0000) 2019-10-13T05:55:40.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:40.145+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.145710+0000) 2019-10-13T05:55:40.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:40.145+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.145811+0000) 2019-10-13T05:55:40.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:40.145+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.145982+0000) 2019-10-13T05:55:40.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:40.140+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.141570+0000) 2019-10-13T05:55:40.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:40.142+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.142933+0000) 2019-10-13T05:55:40.153 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:40.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:40.698 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:40.689+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:40.690074+0000) 2019-10-13T05:55:41.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:41.840+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:41.840339+0000) 2019-10-13T05:55:42.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:42.706+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:42.706837+0000) 2019-10-13T05:55:42.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:42.706+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:42.707198+0000) 2019-10-13T05:55:44.205 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:44.201+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:44.201797+0000) 2019-10-13T05:55:45.153 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:45.144+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.145552+0000) 2019-10-13T05:55:45.155 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:45.149+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.150336+0000) 2019-10-13T05:55:45.155 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:45.150+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.150480+0000) 2019-10-13T05:55:45.155 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:45.150+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.150798+0000) 2019-10-13T05:55:45.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:45.150+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.151080+0000) 2019-10-13T05:55:45.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:45.150+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.151296+0000) 2019-10-13T05:55:45.156 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:45.151+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.151483+0000) 2019-10-13T05:55:45.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:45.147+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:45.148401+0000) 2019-10-13T05:55:45.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:45.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88142 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:48.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:48.246+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:48.247072+0000) 2019-10-13T05:55:48.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:48.585+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:48.586945+0000) 2019-10-13T05:55:48.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:48.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:48.586943+0000) 2019-10-13T05:55:50.003 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:55:50.003 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:50.008 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:55:50.008 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:50.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:50.153+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.154940+0000) 2019-10-13T05:55:50.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:50.153+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.155106+0000) 2019-10-13T05:55:50.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:50.154+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.155404+0000) 2019-10-13T05:55:50.159 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:50.154+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.155545+0000) 2019-10-13T05:55:50.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:50.150+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.151456+0000) 2019-10-13T05:55:50.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:50.154+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.155935+0000) 2019-10-13T05:55:50.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:50.154+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.156204+0000) 2019-10-13T05:55:50.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:50.151+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.152603+0000) 2019-10-13T05:55:50.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:50.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:50.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:50.691+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:50.692319+0000) 2019-10-13T05:55:51.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:51.839+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:51.840534+0000) 2019-10-13T05:55:52.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:52.706+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:52.707025+0000) 2019-10-13T05:55:52.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:52.706+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:52.707382+0000) 2019-10-13T05:55:54.205 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:54.200+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:54.202027+0000) 2019-10-13T05:55:55.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:55:55.158+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.159740+0000) 2019-10-13T05:55:55.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:55:55.154+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.155648+0000) 2019-10-13T05:55:55.164 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:55.159+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.160407+0000) 2019-10-13T05:55:55.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:55.159+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.160529+0000) 2019-10-13T05:55:55.164 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:55.159+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.160670+0000) 2019-10-13T05:55:55.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:55:55.159+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.160836+0000) 2019-10-13T05:55:55.165 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:55:55.159+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.161133+0000) 2019-10-13T05:55:55.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:55:55.156+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:55.157465+0000) 2019-10-13T05:55:55.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:55:55.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:55:58.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:55:58.246+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:58.247257+0000) 2019-10-13T05:55:58.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:55:58.586+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:58.587132+0000) 2019-10-13T05:55:58.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:55:58.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:55:58.587125+0000) 2019-10-13T05:56:00.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:00.163+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.165198+0000) 2019-10-13T05:56:00.168 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:00.163+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.165316+0000) 2019-10-13T05:56:00.169 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:00.160+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.161542+0000) 2019-10-13T05:56:00.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:00.164+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.165675+0000) 2019-10-13T05:56:00.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:00.164+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.166322+0000) 2019-10-13T05:56:00.170 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:00.165+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.166478+0000) 2019-10-13T05:56:00.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:00.165+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.166726+0000) 2019-10-13T05:56:00.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:00.162+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.163050+0000) 2019-10-13T05:56:00.171 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:00.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:00.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:00.694+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:00.694720+0000) 2019-10-13T05:56:01.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:01.839+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:01.840700+0000) 2019-10-13T05:56:02.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:02.706+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:02.707195+0000) 2019-10-13T05:56:02.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:02.707+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:02.707577+0000) 2019-10-13T05:56:04.205 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:04.201+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:04.202218+0000) 2019-10-13T05:56:05.173 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:05.165+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.165879+0000) 2019-10-13T05:56:05.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:05.169+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.170662+0000) 2019-10-13T05:56:05.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:05.169+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.170838+0000) 2019-10-13T05:56:05.175 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:05.169+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.171063+0000) 2019-10-13T05:56:05.175 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:05.169+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.171167+0000) 2019-10-13T05:56:05.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:05.170+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.171627+0000) 2019-10-13T05:56:05.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:05.170+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.171819+0000) 2019-10-13T05:56:05.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:05.172+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88266 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:05.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:05.168+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:05.169602+0000) 2019-10-13T05:56:08.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:08.246+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:08.247433+0000) 2019-10-13T05:56:08.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:08.586+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:08.587325+0000) 2019-10-13T05:56:08.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:08.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:08.587322+0000) 2019-10-13T05:56:10.179 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:10.174+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.175855+0000) 2019-10-13T05:56:10.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:10.174+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.176026+0000) 2019-10-13T05:56:10.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:10.171+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.171816+0000) 2019-10-13T05:56:10.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:10.175+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.176475+0000) 2019-10-13T05:56:10.180 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:10.175+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.176750+0000) 2019-10-13T05:56:10.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:10.175+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.176883+0000) 2019-10-13T05:56:10.181 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:10.175+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.176995+0000) 2019-10-13T05:56:10.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:10.173+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.174166+0000) 2019-10-13T05:56:10.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:10.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88296 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:10.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:10.696+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:10.696662+0000) 2019-10-13T05:56:11.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:11.840+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:11.840858+0000) 2019-10-13T05:56:12.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:12.706+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:12.707365+0000) 2019-10-13T05:56:12.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:12.707+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:12.707740+0000) 2019-10-13T05:56:14.205 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:14.201+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:14.202440+0000) 2019-10-13T05:56:15.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:15.179+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.181368+0000) 2019-10-13T05:56:15.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:15.180+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.181622+0000) 2019-10-13T05:56:15.185 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:15.180+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.181986+0000) 2019-10-13T05:56:15.186 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:15.177+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.177966+0000) 2019-10-13T05:56:15.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:15.180+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.182178+0000) 2019-10-13T05:56:15.186 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:15.180+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.181994+0000) 2019-10-13T05:56:15.186 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:15.180+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.182389+0000) 2019-10-13T05:56:15.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:15.178+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:15.179368+0000) 2019-10-13T05:56:15.188 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:15.183+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88327 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:18.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:18.247+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:18.247627+0000) 2019-10-13T05:56:18.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:18.587+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:18.587568+0000) 2019-10-13T05:56:18.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:18.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:18.587549+0000) 2019-10-13T05:56:20.100 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:56:20.100 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:20.105 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:56:20.105 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:20.190 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:20.186+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.187201+0000) 2019-10-13T05:56:20.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:20.180+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.182305+0000) 2019-10-13T05:56:20.191 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:20.187+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.187588+0000) 2019-10-13T05:56:20.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:20.187+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.187459+0000) 2019-10-13T05:56:20.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:20.187+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.187998+0000) 2019-10-13T05:56:20.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:20.187+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.187938+0000) 2019-10-13T05:56:20.192 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:20.187+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.188132+0000) 2019-10-13T05:56:20.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:20.182+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.184567+0000) 2019-10-13T05:56:20.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:20.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:20.706 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:20.697+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:20.698957+0000) 2019-10-13T05:56:21.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:21.840+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:21.841048+0000) 2019-10-13T05:56:22.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:22.707+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:22.707546+0000) 2019-10-13T05:56:22.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:22.706+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:22.707909+0000) 2019-10-13T05:56:24.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:24.202+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:24.202630+0000) 2019-10-13T05:56:25.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:25.185+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.186950+0000) 2019-10-13T05:56:25.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:25.191+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.191876+0000) 2019-10-13T05:56:25.196 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:25.191+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.192230+0000) 2019-10-13T05:56:25.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:25.192+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.192460+0000) 2019-10-13T05:56:25.196 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:25.192+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.192740+0000) 2019-10-13T05:56:25.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:25.192+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.192715+0000) 2019-10-13T05:56:25.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:25.192+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.193017+0000) 2019-10-13T05:56:25.197 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:25.187+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:25.189680+0000) 2019-10-13T05:56:25.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:25.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:28.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:28.247+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:28.247777+0000) 2019-10-13T05:56:28.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:28.587+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:28.587739+0000) 2019-10-13T05:56:28.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:28.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:28.587732+0000) 2019-10-13T05:56:30.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:30.196+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.197367+0000) 2019-10-13T05:56:30.201 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:30.197+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.197633+0000) 2019-10-13T05:56:30.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:30.197+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.197702+0000) 2019-10-13T05:56:30.201 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:30.197+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.197910+0000) 2019-10-13T05:56:30.202 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:30.191+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.193518+0000) 2019-10-13T05:56:30.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:30.197+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.198298+0000) 2019-10-13T05:56:30.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:30.198+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.198414+0000) 2019-10-13T05:56:30.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:30.193+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.194787+0000) 2019-10-13T05:56:30.207 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:30.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88420 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:30.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:30.700+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:30.701423+0000) 2019-10-13T05:56:31.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:31.841+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:31.841261+0000) 2019-10-13T05:56:32.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:32.707+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:32.707705+0000) 2019-10-13T05:56:32.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:32.707+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:32.708089+0000) 2019-10-13T05:56:34.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:34.201+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:34.202805+0000) 2019-10-13T05:56:35.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:35.201+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.202817+0000) 2019-10-13T05:56:35.206 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:35.197+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.198293+0000) 2019-10-13T05:56:35.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:35.201+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.203303+0000) 2019-10-13T05:56:35.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:35.202+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.203429+0000) 2019-10-13T05:56:35.208 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:35.201+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.203334+0000) 2019-10-13T05:56:35.208 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:35.202+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.203872+0000) 2019-10-13T05:56:35.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:35.202+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.204065+0000) 2019-10-13T05:56:35.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:35.200+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:35.200960+0000) 2019-10-13T05:56:35.209 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:35.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88451 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:38.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:38.247+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:38.247970+0000) 2019-10-13T05:56:38.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:38.586+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:38.587931+0000) 2019-10-13T05:56:38.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:38.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:38.587925+0000) 2019-10-13T05:56:40.211 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:40.206+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.208028+0000) 2019-10-13T05:56:40.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:40.203+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.204143+0000) 2019-10-13T05:56:40.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:40.206+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.208198+0000) 2019-10-13T05:56:40.212 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:40.207+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.209007+0000) 2019-10-13T05:56:40.212 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:40.207+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.209151+0000) 2019-10-13T05:56:40.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:40.208+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.209405+0000) 2019-10-13T05:56:40.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:40.208+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.209834+0000) 2019-10-13T05:56:40.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:40.205+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.206022+0000) 2019-10-13T05:56:40.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:40.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:40.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:40.703+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:40.703638+0000) 2019-10-13T05:56:41.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:41.840+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:41.841444+0000) 2019-10-13T05:56:42.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:42.706+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:42.707794+0000) 2019-10-13T05:56:42.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:42.707+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:42.708255+0000) 2019-10-13T05:56:44.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:44.202+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:44.202994+0000) 2019-10-13T05:56:45.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:45.212+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.214000+0000) 2019-10-13T05:56:45.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:45.212+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.214364+0000) 2019-10-13T05:56:45.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:45.209+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.210459+0000) 2019-10-13T05:56:45.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:45.213+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.214541+0000) 2019-10-13T05:56:45.219 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:45.213+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.214711+0000) 2019-10-13T05:56:45.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:45.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.215143+0000) 2019-10-13T05:56:45.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:45.214+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.215440+0000) 2019-10-13T05:56:45.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:45.212+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:45.212817+0000) 2019-10-13T05:56:45.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:45.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88514 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:48.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:48.247+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:48.248169+0000) 2019-10-13T05:56:48.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:48.587+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:48.588124+0000) 2019-10-13T05:56:48.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:48.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:48.588124+0000) 2019-10-13T05:56:50.199 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:56:50.200 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:50.203 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:56:50.204 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:50.223 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:50.218+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.219652+0000) 2019-10-13T05:56:50.224 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:50.214+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.215623+0000) 2019-10-13T05:56:50.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:50.218+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.220251+0000) 2019-10-13T05:56:50.224 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:50.219+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.220416+0000) 2019-10-13T05:56:50.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:50.219+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.220596+0000) 2019-10-13T05:56:50.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:50.219+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.220831+0000) 2019-10-13T05:56:50.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:50.219+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.220992+0000) 2019-10-13T05:56:50.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:50.215+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.216725+0000) 2019-10-13T05:56:50.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:50.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:50.713 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:50.705+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:50.705919+0000) 2019-10-13T05:56:51.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:51.841+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:51.841647+0000) 2019-10-13T05:56:52.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:52.707+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:52.707977+0000) 2019-10-13T05:56:52.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:52.708+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:52.708425+0000) 2019-10-13T05:56:54.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:54.201+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:54.203195+0000) 2019-10-13T05:56:55.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:56:55.219+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.219856+0000) 2019-10-13T05:56:55.228 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:55.223+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.224706+0000) 2019-10-13T05:56:55.228 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:56:55.223+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.224959+0000) 2019-10-13T05:56:55.229 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:55.223+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.225235+0000) 2019-10-13T05:56:55.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:56:55.223+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.225324+0000) 2019-10-13T05:56:55.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:56:55.224+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.225750+0000) 2019-10-13T05:56:55.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:55.224+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.225507+0000) 2019-10-13T05:56:55.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:56:55.222+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:55.222744+0000) 2019-10-13T05:56:55.230 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:56:55.226+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88575 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:56:58.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:56:58.247+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:58.248346+0000) 2019-10-13T05:56:58.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:56:58.587+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:58.588304+0000) 2019-10-13T05:56:58.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:56:58.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:56:58.588302+0000) 2019-10-13T05:57:00.233 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:00.224+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.225196+0000) 2019-10-13T05:57:00.234 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:00.228+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.230255+0000) 2019-10-13T05:57:00.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:00.229+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.230430+0000) 2019-10-13T05:57:00.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:00.229+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.230478+0000) 2019-10-13T05:57:00.234 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:00.229+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.230656+0000) 2019-10-13T05:57:00.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:00.229+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.230919+0000) 2019-10-13T05:57:00.235 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:00.229+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.231125+0000) 2019-10-13T05:57:00.235 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:00.226+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.227665+0000) 2019-10-13T05:57:00.236 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:00.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:00.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:00.708+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:00.708278+0000) 2019-10-13T05:57:01.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:01.841+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:01.841796+0000) 2019-10-13T05:57:02.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:02.707+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:02.708161+0000) 2019-10-13T05:57:02.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:02.708+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:02.708612+0000) 2019-10-13T05:57:04.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:04.202+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:04.203372+0000) 2019-10-13T05:57:05.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:05.234+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.235097+0000) 2019-10-13T05:57:05.239 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:05.230+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.231090+0000) 2019-10-13T05:57:05.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:05.235+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.235463+0000) 2019-10-13T05:57:05.239 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:05.235+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.235791+0000) 2019-10-13T05:57:05.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:05.235+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.236210+0000) 2019-10-13T05:57:05.240 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:05.236+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.236459+0000) 2019-10-13T05:57:05.240 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:05.236+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.236652+0000) 2019-10-13T05:57:05.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:05.232+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:05.232855+0000) 2019-10-13T05:57:05.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:05.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88638 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:08.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:08.248+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:08.248508+0000) 2019-10-13T05:57:08.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:08.588+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:08.588514+0000) 2019-10-13T05:57:08.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:08.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:08.588510+0000) 2019-10-13T05:57:10.243 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:10.239+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.240173+0000) 2019-10-13T05:57:10.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:10.240+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.240586+0000) 2019-10-13T05:57:10.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:10.240+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.240755+0000) 2019-10-13T05:57:10.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:10.234+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.236275+0000) 2019-10-13T05:57:10.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:10.240+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.241105+0000) 2019-10-13T05:57:10.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:10.241+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.241393+0000) 2019-10-13T05:57:10.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:10.241+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.241602+0000) 2019-10-13T05:57:10.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:10.237+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.238826+0000) 2019-10-13T05:57:10.247 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:10.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:10.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:10.709+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:10.710548+0000) 2019-10-13T05:57:11.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:11.841+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:11.841956+0000) 2019-10-13T05:57:12.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:12.708+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:12.708347+0000) 2019-10-13T05:57:12.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:12.707+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:12.708769+0000) 2019-10-13T05:57:14.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:14.203+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:14.203542+0000) 2019-10-13T05:57:15.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:15.245+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.246186+0000) 2019-10-13T05:57:15.250 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:15.240+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.241840+0000) 2019-10-13T05:57:15.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:15.246+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.246543+0000) 2019-10-13T05:57:15.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:15.246+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.246690+0000) 2019-10-13T05:57:15.250 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:15.246+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.247316+0000) 2019-10-13T05:57:15.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:15.247+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.247691+0000) 2019-10-13T05:57:15.251 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:15.247+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.247777+0000) 2019-10-13T05:57:15.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:15.243+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:15.244798+0000) 2019-10-13T05:57:15.253 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:15.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:18.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:18.248+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:18.248693+0000) 2019-10-13T05:57:18.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:18.588+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:18.588695+0000) 2019-10-13T05:57:18.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:18.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:18.588686+0000) 2019-10-13T05:57:20.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:20.250+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.251653+0000) 2019-10-13T05:57:20.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:20.250+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.251921+0000) 2019-10-13T05:57:20.256 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:20.250+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.252108+0000) 2019-10-13T05:57:20.256 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:20.251+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.252429+0000) 2019-10-13T05:57:20.256 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:20.245+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.247630+0000) 2019-10-13T05:57:20.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:20.251+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.252634+0000) 2019-10-13T05:57:20.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:20.251+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.253051+0000) 2019-10-13T05:57:20.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:20.247+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.249384+0000) 2019-10-13T05:57:20.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:20.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:20.287 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:57:20.287 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:20.291 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:57:20.291 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:20.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:20.712+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:20.713030+0000) 2019-10-13T05:57:21.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:21.841+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:21.842165+0000) 2019-10-13T05:57:22.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:22.707+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:22.708526+0000) 2019-10-13T05:57:22.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:22.708+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:22.708936+0000) 2019-10-13T05:57:24.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:24.202+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:24.203696+0000) 2019-10-13T05:57:25.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:25.256+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.257687+0000) 2019-10-13T05:57:25.262 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:25.256+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.257881+0000) 2019-10-13T05:57:25.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:25.256+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.257963+0000) 2019-10-13T05:57:25.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:25.256+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.258252+0000) 2019-10-13T05:57:25.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:25.256+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.258363+0000) 2019-10-13T05:57:25.263 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:25.257+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.258790+0000) 2019-10-13T05:57:25.263 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:25.253+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.254012+0000) 2019-10-13T05:57:25.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:25.254+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:25.255112+0000) 2019-10-13T05:57:25.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:25.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:28.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:28.248+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:28.248855+0000) 2019-10-13T05:57:28.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:28.587+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:28.588793+0000) 2019-10-13T05:57:28.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:28.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:28.588800+0000) 2019-10-13T05:57:30.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:30.257+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.258289+0000) 2019-10-13T05:57:30.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:30.260+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.260790+0000) 2019-10-13T05:57:30.274 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:30.261+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.262728+0000) 2019-10-13T05:57:30.274 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:30.261+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.263014+0000) 2019-10-13T05:57:30.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:30.262+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.263442+0000) 2019-10-13T05:57:30.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:30.262+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.263515+0000) 2019-10-13T05:57:30.275 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:30.262+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.263750+0000) 2019-10-13T05:57:30.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:30.262+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.263897+0000) 2019-10-13T05:57:30.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:30.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:30.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:30.714+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:30.715344+0000) 2019-10-13T05:57:31.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:31.841+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:31.842327+0000) 2019-10-13T05:57:32.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:32.707+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:32.708733+0000) 2019-10-13T05:57:32.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:32.708+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:32.709127+0000) 2019-10-13T05:57:34.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:34.202+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:34.203839+0000) 2019-10-13T05:57:35.271 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:35.266+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.268283+0000) 2019-10-13T05:57:35.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:35.267+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.268441+0000) 2019-10-13T05:57:35.272 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:35.267+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.268634+0000) 2019-10-13T05:57:35.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:35.263+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.264213+0000) 2019-10-13T05:57:35.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:35.267+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.269222+0000) 2019-10-13T05:57:35.273 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:35.267+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.269328+0000) 2019-10-13T05:57:35.273 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:35.268+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.269615+0000) 2019-10-13T05:57:35.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:35.266+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:35.267185+0000) 2019-10-13T05:57:35.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:35.270+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88823 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:38.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:38.248+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:38.249094+0000) 2019-10-13T05:57:38.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:38.588+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:38.588953+0000) 2019-10-13T05:57:38.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:38.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:38.588954+0000) 2019-10-13T05:57:40.277 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:40.268+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.269300+0000) 2019-10-13T05:57:40.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:40.272+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.274114+0000) 2019-10-13T05:57:40.279 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:40.272+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.274381+0000) 2019-10-13T05:57:40.279 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:40.273+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.274419+0000) 2019-10-13T05:57:40.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:40.273+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.274728+0000) 2019-10-13T05:57:40.280 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:40.273+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.274970+0000) 2019-10-13T05:57:40.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:40.273+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.275317+0000) 2019-10-13T05:57:40.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:40.271+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.272375+0000) 2019-10-13T05:57:40.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:40.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:40.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:40.717+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:40.717872+0000) 2019-10-13T05:57:41.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:41.841+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:41.842506+0000) 2019-10-13T05:57:42.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:42.708+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:42.708918+0000) 2019-10-13T05:57:42.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:42.708+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:42.709306+0000) 2019-10-13T05:57:44.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:44.203+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:44.203964+0000) 2019-10-13T05:57:45.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:45.278+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.279826+0000) 2019-10-13T05:57:45.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:45.278+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.280013+0000) 2019-10-13T05:57:45.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:45.278+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.280216+0000) 2019-10-13T05:57:45.284 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:45.279+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.280411+0000) 2019-10-13T05:57:45.285 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:45.275+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.276003+0000) 2019-10-13T05:57:45.285 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:45.279+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.280767+0000) 2019-10-13T05:57:45.285 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:45.279+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.280854+0000) 2019-10-13T05:57:45.286 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:45.277+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:45.278602+0000) 2019-10-13T05:57:45.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:45.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:48.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:48.248+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:48.249266+0000) 2019-10-13T05:57:48.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:48.588+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:48.589151+0000) 2019-10-13T05:57:48.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:48.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:48.589160+0000) 2019-10-13T05:57:50.289 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:50.285+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.285550+0000) 2019-10-13T05:57:50.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:50.285+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.285867+0000) 2019-10-13T05:57:50.289 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:50.285+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.286111+0000) 2019-10-13T05:57:50.290 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:50.281+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.282383+0000) 2019-10-13T05:57:50.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:50.286+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.286455+0000) 2019-10-13T05:57:50.290 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:50.286+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.286587+0000) 2019-10-13T05:57:50.291 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:50.286+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.286878+0000) 2019-10-13T05:57:50.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:50.282+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.283313+0000) 2019-10-13T05:57:50.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:50.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:50.376 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:57:50.376 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:50.380 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:57:50.380 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:50.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:50.720+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:50.720322+0000) 2019-10-13T05:57:51.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:51.842+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:51.842666+0000) 2019-10-13T05:57:52.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:52.708+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:52.709099+0000) 2019-10-13T05:57:52.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:52.709+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:52.709493+0000) 2019-10-13T05:57:54.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:54.203+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:54.204146+0000) 2019-10-13T05:57:55.294 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:57:55.291+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.291552+0000) 2019-10-13T05:57:55.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:57:55.286+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.287518+0000) 2019-10-13T05:57:55.295 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:55.291+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.291921+0000) 2019-10-13T05:57:55.296 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:57:55.291+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.291814+0000) 2019-10-13T05:57:55.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:55.292+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.292581+0000) 2019-10-13T05:57:55.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:55.292+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.292756+0000) 2019-10-13T05:57:55.297 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:57:55.292+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.292932+0000) 2019-10-13T05:57:55.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:57:55.289+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:55.289997+0000) 2019-10-13T05:57:55.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:57:55.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:57:58.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:57:58.249+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:58.249423+0000) 2019-10-13T05:57:58.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:57:58.589+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:58.589345+0000) 2019-10-13T05:57:58.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:57:58.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:57:58.589346+0000) 2019-10-13T05:58:00.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:00.296+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.297058+0000) 2019-10-13T05:58:00.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:00.297+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.297494+0000) 2019-10-13T05:58:00.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:00.292+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.293096+0000) 2019-10-13T05:58:00.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:00.297+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.297929+0000) 2019-10-13T05:58:00.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:00.297+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.298069+0000) 2019-10-13T05:58:00.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:00.297+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.298192+0000) 2019-10-13T05:58:00.303 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:00.298+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.298433+0000) 2019-10-13T05:58:00.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:00.294+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.294894+0000) 2019-10-13T05:58:00.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:00.300+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 88977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:00.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:00.722+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:00.722406+0000) 2019-10-13T05:58:01.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:01.842+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:01.842879+0000) 2019-10-13T05:58:02.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:02.709+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:02.709270+0000) 2019-10-13T05:58:02.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:02.708+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:02.709675+0000) 2019-10-13T05:58:04.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:04.204+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:04.204333+0000) 2019-10-13T05:58:05.306 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:05.296+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.298391+0000) 2019-10-13T05:58:05.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:05.301+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.303158+0000) 2019-10-13T05:58:05.307 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:05.302+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.303440+0000) 2019-10-13T05:58:05.307 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:05.302+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.303567+0000) 2019-10-13T05:58:05.307 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:05.302+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.303716+0000) 2019-10-13T05:58:05.307 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:05.302+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.304115+0000) 2019-10-13T05:58:05.308 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:05.302+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.304338+0000) 2019-10-13T05:58:05.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:05.300+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:05.301961+0000) 2019-10-13T05:58:05.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:05.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:08.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:08.248+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:08.249597+0000) 2019-10-13T05:58:08.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:08.588+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:08.589484+0000) 2019-10-13T05:58:08.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:08.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:08.589484+0000) 2019-10-13T05:58:10.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:10.307+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.308494+0000) 2019-10-13T05:58:10.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:10.307+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.308861+0000) 2019-10-13T05:58:10.313 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:10.307+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.308985+0000) 2019-10-13T05:58:10.314 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:10.307+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.309246+0000) 2019-10-13T05:58:10.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:10.308+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.309506+0000) 2019-10-13T05:58:10.315 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:10.302+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.304318+0000) 2019-10-13T05:58:10.316 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:10.305+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.306976+0000) 2019-10-13T05:58:10.316 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:10.308+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.309702+0000) 2019-10-13T05:58:10.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:10.311+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89040 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:10.732 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:10.723+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:10.724702+0000) 2019-10-13T05:58:11.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:11.842+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:11.843054+0000) 2019-10-13T05:58:12.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:12.708+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:12.709436+0000) 2019-10-13T05:58:12.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:12.709+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:12.709872+0000) 2019-10-13T05:58:14.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:14.203+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:14.204464+0000) 2019-10-13T05:58:15.318 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:15.313+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.315242+0000) 2019-10-13T05:58:15.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:15.314+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.315418+0000) 2019-10-13T05:58:15.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:15.310+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.310994+0000) 2019-10-13T05:58:15.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:15.314+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.315692+0000) 2019-10-13T05:58:15.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:15.314+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.316038+0000) 2019-10-13T05:58:15.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:15.314+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.316232+0000) 2019-10-13T05:58:15.320 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:15.315+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.316623+0000) 2019-10-13T05:58:15.321 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:15.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89071 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:15.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:15.312+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:15.313878+0000) 2019-10-13T05:58:18.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:18.249+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:18.249788+0000) 2019-10-13T05:58:18.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:18.588+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:18.589636+0000) 2019-10-13T05:58:18.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:18.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:18.589638+0000) 2019-10-13T05:58:20.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:20.318+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.320068+0000) 2019-10-13T05:58:20.324 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:20.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.315894+0000) 2019-10-13T05:58:20.324 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:20.319+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.320605+0000) 2019-10-13T05:58:20.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:20.319+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.320754+0000) 2019-10-13T05:58:20.325 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:20.319+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.321078+0000) 2019-10-13T05:58:20.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:20.319+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.321336+0000) 2019-10-13T05:58:20.325 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:20.320+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.321471+0000) 2019-10-13T05:58:20.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:20.316+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.317471+0000) 2019-10-13T05:58:20.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:20.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89101 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:20.555 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:58:20.555 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:20.560 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:58:20.560 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:20.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:20.726+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:20.726954+0000) 2019-10-13T05:58:21.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:21.842+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:21.843250+0000) 2019-10-13T05:58:22.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:22.708+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:22.709609+0000) 2019-10-13T05:58:22.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:22.709+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:22.710035+0000) 2019-10-13T05:58:24.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:24.203+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:24.204575+0000) 2019-10-13T05:58:25.328 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:25.323+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.325301+0000) 2019-10-13T05:58:25.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:25.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.325569+0000) 2019-10-13T05:58:25.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:25.320+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.321365+0000) 2019-10-13T05:58:25.330 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:25.324+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.325980+0000) 2019-10-13T05:58:25.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:25.324+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.326225+0000) 2019-10-13T05:58:25.330 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:25.324+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.326341+0000) 2019-10-13T05:58:25.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:25.325+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.326474+0000) 2019-10-13T05:58:25.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:25.322+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:25.323236+0000) 2019-10-13T05:58:25.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:25.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89134 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:28.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:28.249+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:28.250027+0000) 2019-10-13T05:58:28.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:28.589+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:28.589825+0000) 2019-10-13T05:58:28.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:28.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:28.589827+0000) 2019-10-13T05:58:30.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:30.329+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.331173+0000) 2019-10-13T05:58:30.335 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:30.330+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.331414+0000) 2019-10-13T05:58:30.335 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:30.330+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.331489+0000) 2019-10-13T05:58:30.336 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:30.326+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.327279+0000) 2019-10-13T05:58:30.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:30.330+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.332018+0000) 2019-10-13T05:58:30.336 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:30.330+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.332086+0000) 2019-10-13T05:58:30.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:30.331+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.332416+0000) 2019-10-13T05:58:30.337 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:30.329+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.329775+0000) 2019-10-13T05:58:30.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:30.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:30.737 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:30.728+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:30.729324+0000) 2019-10-13T05:58:31.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:31.842+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:31.843436+0000) 2019-10-13T05:58:32.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:32.709+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:32.709791+0000) 2019-10-13T05:58:32.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:32.709+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:32.710219+0000) 2019-10-13T05:58:34.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:34.204+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:34.204745+0000) 2019-10-13T05:58:35.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:35.331+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.332019+0000) 2019-10-13T05:58:35.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:35.336+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.336963+0000) 2019-10-13T05:58:35.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:35.336+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.337097+0000) 2019-10-13T05:58:35.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:35.336+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.337226+0000) 2019-10-13T05:58:35.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:35.337+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.337404+0000) 2019-10-13T05:58:35.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:35.337+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.337637+0000) 2019-10-13T05:58:35.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:35.337+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.337834+0000) 2019-10-13T05:58:35.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:35.333+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:35.334294+0000) 2019-10-13T05:58:35.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:35.340+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:38.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:38.249+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:38.250213+0000) 2019-10-13T05:58:38.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:38.589+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:38.590023+0000) 2019-10-13T05:58:38.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:38.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:38.590041+0000) 2019-10-13T05:58:40.345 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:40.337+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.337860+0000) 2019-10-13T05:58:40.346 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:40.342+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.342590+0000) 2019-10-13T05:58:40.346 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:40.342+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.343040+0000) 2019-10-13T05:58:40.346 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:40.342+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.343276+0000) 2019-10-13T05:58:40.347 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:40.343+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.343528+0000) 2019-10-13T05:58:40.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:40.343+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.343685+0000) 2019-10-13T05:58:40.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:40.343+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.343912+0000) 2019-10-13T05:58:40.348 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:40.339+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.340316+0000) 2019-10-13T05:58:40.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:40.345+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:40.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:40.731+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:40.731716+0000) 2019-10-13T05:58:41.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:41.843+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:41.843608+0000) 2019-10-13T05:58:42.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:42.709+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:42.709889+0000) 2019-10-13T05:58:42.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:42.710+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:42.710374+0000) 2019-10-13T05:58:44.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:44.204+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:44.204931+0000) 2019-10-13T05:58:45.351 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:45.347+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.347898+0000) 2019-10-13T05:58:45.352 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:45.343+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.344031+0000) 2019-10-13T05:58:45.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:45.347+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.348173+0000) 2019-10-13T05:58:45.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:45.348+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.348639+0000) 2019-10-13T05:58:45.352 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:45.348+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.348889+0000) 2019-10-13T05:58:45.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:45.348+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.349281+0000) 2019-10-13T05:58:45.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:45.349+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.349805+0000) 2019-10-13T05:58:45.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:45.346+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:45.346947+0000) 2019-10-13T05:58:45.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:45.351+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:48.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:48.250+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:48.250430+0000) 2019-10-13T05:58:48.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:48.590+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:48.590196+0000) 2019-10-13T05:58:48.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:48.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:48.590213+0000) 2019-10-13T05:58:50.356 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:50.348+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.348868+0000) 2019-10-13T05:58:50.357 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:50.353+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.353715+0000) 2019-10-13T05:58:50.358 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:50.353+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.354214+0000) 2019-10-13T05:58:50.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:50.353+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.354360+0000) 2019-10-13T05:58:50.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:50.354+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.354571+0000) 2019-10-13T05:58:50.358 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:50.354+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.354704+0000) 2019-10-13T05:58:50.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:50.354+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.355259+0000) 2019-10-13T05:58:50.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:50.357+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:50.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:50.351+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.352326+0000) 2019-10-13T05:58:50.728 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:58:50.728 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:50.732 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:58:50.732 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:50.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:50.733+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:50.733957+0000) 2019-10-13T05:58:51.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:51.842+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:51.843793+0000) 2019-10-13T05:58:52.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:52.709+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:52.710057+0000) 2019-10-13T05:58:52.718 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:52.710+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:52.710551+0000) 2019-10-13T05:58:54.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:54.204+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:54.205080+0000) 2019-10-13T05:58:55.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:55.358+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.359770+0000) 2019-10-13T05:58:55.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:58:55.358+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.359932+0000) 2019-10-13T05:58:55.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:58:55.354+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.355874+0000) 2019-10-13T05:58:55.364 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:55.359+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.360543+0000) 2019-10-13T05:58:55.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:58:55.359+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.360540+0000) 2019-10-13T05:58:55.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:58:55.359+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.360863+0000) 2019-10-13T05:58:55.365 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:55.359+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.361050+0000) 2019-10-13T05:58:55.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:58:55.355+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:55.357231+0000) 2019-10-13T05:58:55.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:58:55.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89319 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:58:58.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:58:58.249+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:58.250644+0000) 2019-10-13T05:58:58.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:58:58.589+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:58.590383+0000) 2019-10-13T05:58:58.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:58:58.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:58:58.590390+0000) 2019-10-13T05:59:00.369 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:00.359+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.361089+0000) 2019-10-13T05:59:00.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:00.363+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.365066+0000) 2019-10-13T05:59:00.370 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:00.363+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.365344+0000) 2019-10-13T05:59:00.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:00.364+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.365474+0000) 2019-10-13T05:59:00.370 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:00.364+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.365783+0000) 2019-10-13T05:59:00.370 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:00.364+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.365997+0000) 2019-10-13T05:59:00.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:00.364+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.366180+0000) 2019-10-13T05:59:00.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:00.361+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.363471+0000) 2019-10-13T05:59:00.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:00.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:00.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:00.735+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:00.736250+0000) 2019-10-13T05:59:01.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:01.843+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:01.844018+0000) 2019-10-13T05:59:02.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:02.709+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:02.710243+0000) 2019-10-13T05:59:02.718 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:02.709+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:02.710771+0000) 2019-10-13T05:59:04.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:04.204+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:04.205257+0000) 2019-10-13T05:59:05.373 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:05.368+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.370013+0000) 2019-10-13T05:59:05.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:05.368+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.370313+0000) 2019-10-13T05:59:05.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:05.368+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.370359+0000) 2019-10-13T05:59:05.374 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:05.369+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.370538+0000) 2019-10-13T05:59:05.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:05.364+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.366155+0000) 2019-10-13T05:59:05.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:05.369+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.370880+0000) 2019-10-13T05:59:05.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:05.369+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.371071+0000) 2019-10-13T05:59:05.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:05.366+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:05.368362+0000) 2019-10-13T05:59:05.376 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:05.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:08.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:08.250+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:08.250781+0000) 2019-10-13T05:59:08.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:08.589+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:08.590594+0000) 2019-10-13T05:59:08.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:08.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:08.590602+0000) 2019-10-13T05:59:10.379 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:10.373+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.375045+0000) 2019-10-13T05:59:10.379 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:10.373+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.375203+0000) 2019-10-13T05:59:10.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:10.374+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.375431+0000) 2019-10-13T05:59:10.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:10.374+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.375784+0000) 2019-10-13T05:59:10.380 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:10.370+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.371020+0000) 2019-10-13T05:59:10.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:10.374+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.376135+0000) 2019-10-13T05:59:10.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:10.374+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.376308+0000) 2019-10-13T05:59:10.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:10.372+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.373175+0000) 2019-10-13T05:59:10.382 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:10.377+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89412 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:10.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:10.737+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:10.738564+0000) 2019-10-13T05:59:11.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:11.843+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:11.844203+0000) 2019-10-13T05:59:12.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:12.709+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:12.710457+0000) 2019-10-13T05:59:12.718 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:12.710+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:12.710958+0000) 2019-10-13T05:59:14.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:14.204+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:14.205427+0000) 2019-10-13T05:59:15.384 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:15.375+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.375829+0000) 2019-10-13T05:59:15.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:15.379+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.380760+0000) 2019-10-13T05:59:15.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:15.379+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.381003+0000) 2019-10-13T05:59:15.385 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:15.379+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.381300+0000) 2019-10-13T05:59:15.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:15.380+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.381541+0000) 2019-10-13T05:59:15.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:15.380+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.381426+0000) 2019-10-13T05:59:15.386 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:15.380+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.381678+0000) 2019-10-13T05:59:15.386 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:15.377+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:15.378627+0000) 2019-10-13T05:59:15.387 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:15.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89443 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:18.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:18.250+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:18.250931+0000) 2019-10-13T05:59:18.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:18.590+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:18.590775+0000) 2019-10-13T05:59:18.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:18.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:18.590782+0000) 2019-10-13T05:59:20.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:20.385+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.385626+0000) 2019-10-13T05:59:20.390 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:20.381+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.381723+0000) 2019-10-13T05:59:20.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:20.385+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.386053+0000) 2019-10-13T05:59:20.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:20.385+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.386386+0000) 2019-10-13T05:59:20.390 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:20.386+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.386468+0000) 2019-10-13T05:59:20.391 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:20.386+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.386560+0000) 2019-10-13T05:59:20.391 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:20.386+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.386819+0000) 2019-10-13T05:59:20.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:20.382+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.383512+0000) 2019-10-13T05:59:20.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:20.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:20.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:20.740+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:20.740866+0000) 2019-10-13T05:59:20.876 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:59:20.876 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:20.880 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:59:20.880 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:21.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:21.844+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:21.844379+0000) 2019-10-13T05:59:22.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:22.710+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:22.710621+0000) 2019-10-13T05:59:22.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:22.710+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:22.711124+0000) 2019-10-13T05:59:24.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:24.205+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:24.205546+0000) 2019-10-13T05:59:25.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:25.385+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.385874+0000) 2019-10-13T05:59:25.395 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:25.390+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.390589+0000) 2019-10-13T05:59:25.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:25.390+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.390774+0000) 2019-10-13T05:59:25.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:25.390+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.390879+0000) 2019-10-13T05:59:25.396 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:25.390+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.391095+0000) 2019-10-13T05:59:25.396 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:25.390+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.391283+0000) 2019-10-13T05:59:25.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:25.391+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.391419+0000) 2019-10-13T05:59:25.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:25.387+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:25.388327+0000) 2019-10-13T05:59:25.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:25.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89506 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:28.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:28.250+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:28.251086+0000) 2019-10-13T05:59:28.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:28.590+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:28.590970+0000) 2019-10-13T05:59:28.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:28.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:28.590976+0000) 2019-10-13T05:59:30.398 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:30.394+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.394438+0000) 2019-10-13T05:59:30.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:30.394+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.394666+0000) 2019-10-13T05:59:30.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:30.394+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.394891+0000) 2019-10-13T05:59:30.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:30.389+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.390516+0000) 2019-10-13T05:59:30.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:30.394+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.395305+0000) 2019-10-13T05:59:30.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:30.395+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.395667+0000) 2019-10-13T05:59:30.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:30.395+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.395781+0000) 2019-10-13T05:59:30.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:30.392+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.393586+0000) 2019-10-13T05:59:30.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:30.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89536 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:30.751 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:30.743+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:30.743668+0000) 2019-10-13T05:59:31.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:31.844+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:31.844590+0000) 2019-10-13T05:59:32.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:32.710+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:32.710794+0000) 2019-10-13T05:59:32.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:32.710+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:32.711295+0000) 2019-10-13T05:59:34.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:34.205+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:34.205740+0000) 2019-10-13T05:59:35.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:35.399+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.399866+0000) 2019-10-13T05:59:35.416 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:35.395+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.395778+0000) 2019-10-13T05:59:35.416 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:35.400+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.400419+0000) 2019-10-13T05:59:35.416 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:35.400+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.400610+0000) 2019-10-13T05:59:35.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:35.398+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.399408+0000) 2019-10-13T05:59:35.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:35.400+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.400794+0000) 2019-10-13T05:59:35.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:35.400+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.401169+0000) 2019-10-13T05:59:35.426 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:35.401+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:35.401617+0000) 2019-10-13T05:59:35.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:35.402+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89567 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:38.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:38.250+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:38.251254+0000) 2019-10-13T05:59:38.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:38.590+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:38.591156+0000) 2019-10-13T05:59:38.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:38.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:38.591175+0000) 2019-10-13T05:59:40.409 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:40.403+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.405087+0000) 2019-10-13T05:59:40.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:40.403+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.405240+0000) 2019-10-13T05:59:40.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:40.403+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.405377+0000) 2019-10-13T05:59:40.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:40.404+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.405721+0000) 2019-10-13T05:59:40.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:40.404+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.405936+0000) 2019-10-13T05:59:40.411 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:40.404+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.405830+0000) 2019-10-13T05:59:40.411 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:40.400+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.401217+0000) 2019-10-13T05:59:40.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:40.402+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.403724+0000) 2019-10-13T05:59:40.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:40.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89597 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:40.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:40.745+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:40.746097+0000) 2019-10-13T05:59:41.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:41.843+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:41.844819+0000) 2019-10-13T05:59:42.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:42.710+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:42.710979+0000) 2019-10-13T05:59:42.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:42.711+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:42.711498+0000) 2019-10-13T05:59:44.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:44.205+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:44.205925+0000) 2019-10-13T05:59:45.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:45.406+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.406932+0000) 2019-10-13T05:59:45.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:45.408+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.409258+0000) 2019-10-13T05:59:45.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:45.409+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.410622+0000) 2019-10-13T05:59:45.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:45.409+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.410959+0000) 2019-10-13T05:59:45.421 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:45.409+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.411270+0000) 2019-10-13T05:59:45.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:45.409+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.411163+0000) 2019-10-13T05:59:45.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:45.410+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.411888+0000) 2019-10-13T05:59:45.422 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:45.410+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:45.412025+0000) 2019-10-13T05:59:45.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:45.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89630 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:48.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:48.250+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:48.251480+0000) 2019-10-13T05:59:48.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:48.590+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:48.591346+0000) 2019-10-13T05:59:48.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:48.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:48.591348+0000) 2019-10-13T05:59:50.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:50.415+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.416721+0000) 2019-10-13T05:59:50.421 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:50.410+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.412688+0000) 2019-10-13T05:59:50.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:50.415+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.417046+0000) 2019-10-13T05:59:50.421 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:50.416+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.417453+0000) 2019-10-13T05:59:50.421 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:50.415+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.417284+0000) 2019-10-13T05:59:50.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:50.416+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.417711+0000) 2019-10-13T05:59:50.422 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:50.416+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.417995+0000) 2019-10-13T05:59:50.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:50.412+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.414221+0000) 2019-10-13T05:59:50.423 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:50.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89660 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:50.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:50.747+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:50.748403+0000) 2019-10-13T05:59:51.055 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T05:59:51.056 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:51.059 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T05:59:51.059 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:51.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:51.844+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:51.844977+0000) 2019-10-13T05:59:52.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:52.709+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:52.711176+0000) 2019-10-13T05:59:52.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:52.710+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:52.711686+0000) 2019-10-13T05:59:54.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:54.205+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:54.206117+0000) 2019-10-13T05:59:55.426 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:55.421+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.422575+0000) 2019-10-13T05:59:55.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T05:59:55.421+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.422694+0000) 2019-10-13T05:59:55.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:55.421+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.422858+0000) 2019-10-13T05:59:55.427 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T05:59:55.421+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.423208+0000) 2019-10-13T05:59:55.427 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T05:59:55.421+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.423105+0000) 2019-10-13T05:59:55.427 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T05:59:55.417+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.418870+0000) 2019-10-13T05:59:55.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T05:59:55.418+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.420108+0000) 2019-10-13T05:59:55.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:55.422+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:55.423486+0000) 2019-10-13T05:59:55.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T05:59:55.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T05:59:58.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T05:59:58.251+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:58.251648+0000) 2019-10-13T05:59:58.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T05:59:58.590+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:58.591500+0000) 2019-10-13T05:59:58.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T05:59:58.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T04:59:58.591503+0000) 2019-10-13T06:00:00.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:00.426+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.427916+0000) 2019-10-13T06:00:00.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:00.421+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.423655+0000) 2019-10-13T06:00:00.433 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:00.427+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.428408+0000) 2019-10-13T06:00:00.433 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:00.427+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.428688+0000) 2019-10-13T06:00:00.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:00.427+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.428822+0000) 2019-10-13T06:00:00.433 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:00.427+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.429137+0000) 2019-10-13T06:00:00.434 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:00.427+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.429275+0000) 2019-10-13T06:00:00.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:00.430+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:00.435 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:00.425+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.427420+0000) 2019-10-13T06:00:00.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:00.749+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:00.750586+0000) 2019-10-13T06:00:01.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:01.844+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:01.845164+0000) 2019-10-13T06:00:02.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:02.710+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:02.711375+0000) 2019-10-13T06:00:02.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:02.711+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:02.711900+0000) 2019-10-13T06:00:04.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:04.205+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:04.206302+0000) 2019-10-13T06:00:05.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:05.432+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.433263+0000) 2019-10-13T06:00:05.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:05.433+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.433509+0000) 2019-10-13T06:00:05.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:05.433+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.433659+0000) 2019-10-13T06:00:05.437 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:05.433+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.433878+0000) 2019-10-13T06:00:05.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:05.429+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.429768+0000) 2019-10-13T06:00:05.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:05.433+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.433912+0000) 2019-10-13T06:00:05.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:05.434+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.434870+0000) 2019-10-13T06:00:05.444 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:05.436+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89754 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:05.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:05.431+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:05.432239+0000) 2019-10-13T06:00:08.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:08.251+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:08.251853+0000) 2019-10-13T06:00:08.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:08.591+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:08.591698+0000) 2019-10-13T06:00:08.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:08.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:08.591697+0000) 2019-10-13T06:00:10.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:10.438+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.438743+0000) 2019-10-13T06:00:10.442 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:10.438+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.438903+0000) 2019-10-13T06:00:10.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:10.434+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.434879+0000) 2019-10-13T06:00:10.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:10.439+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.439392+0000) 2019-10-13T06:00:10.443 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:10.439+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.439670+0000) 2019-10-13T06:00:10.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:10.439+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.440162+0000) 2019-10-13T06:00:10.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:10.439+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.440377+0000) 2019-10-13T06:00:10.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:10.436+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.437493+0000) 2019-10-13T06:00:10.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:10.442+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:10.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:10.752+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:10.752923+0000) 2019-10-13T06:00:11.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:11.844+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:11.845333+0000) 2019-10-13T06:00:12.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:12.711+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:12.711574+0000) 2019-10-13T06:00:12.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:12.711+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:12.712076+0000) 2019-10-13T06:00:14.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:14.206+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:14.206507+0000) 2019-10-13T06:00:15.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:15.443+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.444119+0000) 2019-10-13T06:00:15.448 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:15.443+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.444371+0000) 2019-10-13T06:00:15.449 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:15.444+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.444630+0000) 2019-10-13T06:00:15.449 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:15.444+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.444580+0000) 2019-10-13T06:00:15.449 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:15.444+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.445077+0000) 2019-10-13T06:00:15.450 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:15.440+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.440778+0000) 2019-10-13T06:00:15.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:15.445+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.445490+0000) 2019-10-13T06:00:15.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:15.442+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:15.443094+0000) 2019-10-13T06:00:15.452 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:15.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:18.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:18.251+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:18.252045+0000) 2019-10-13T06:00:18.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:18.591+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:18.591846+0000) 2019-10-13T06:00:18.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:18.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:18.591864+0000) 2019-10-13T06:00:20.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:20.449+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.449687+0000) 2019-10-13T06:00:20.454 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:20.449+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.449964+0000) 2019-10-13T06:00:20.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:20.449+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.450107+0000) 2019-10-13T06:00:20.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:20.450+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.450439+0000) 2019-10-13T06:00:20.454 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:20.450+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.450795+0000) 2019-10-13T06:00:20.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:20.450+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.451003+0000) 2019-10-13T06:00:20.455 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:20.445+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.446168+0000) 2019-10-13T06:00:20.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:20.447+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.447933+0000) 2019-10-13T06:00:20.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:20.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:20.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:20.754+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:20.755244+0000) 2019-10-13T06:00:21.203 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:00:21.203 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:21.207 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:00:21.208 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:21.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:21.845+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:21.845509+0000) 2019-10-13T06:00:22.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:22.710+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:22.711728+0000) 2019-10-13T06:00:22.720 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:22.711+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:22.712292+0000) 2019-10-13T06:00:24.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:24.205+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:24.206715+0000) 2019-10-13T06:00:25.460 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:25.450+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.450809+0000) 2019-10-13T06:00:25.460 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:25.453+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.455127+0000) 2019-10-13T06:00:25.460 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:25.454+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.455577+0000) 2019-10-13T06:00:25.461 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:25.454+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.455955+0000) 2019-10-13T06:00:25.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:25.454+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.456052+0000) 2019-10-13T06:00:25.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:25.454+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.456259+0000) 2019-10-13T06:00:25.461 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:25.455+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.456675+0000) 2019-10-13T06:00:25.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:25.453+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:25.454168+0000) 2019-10-13T06:00:25.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:25.457+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:28.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:28.251+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:28.252248+0000) 2019-10-13T06:00:28.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:28.591+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:28.592033+0000) 2019-10-13T06:00:28.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:28.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:28.592037+0000) 2019-10-13T06:00:30.464 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:30.459+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.460815+0000) 2019-10-13T06:00:30.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:30.459+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.461164+0000) 2019-10-13T06:00:30.465 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:30.459+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.461378+0000) 2019-10-13T06:00:30.466 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:30.455+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.456609+0000) 2019-10-13T06:00:30.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:30.460+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.461793+0000) 2019-10-13T06:00:30.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:30.460+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.461922+0000) 2019-10-13T06:00:30.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:30.460+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.462138+0000) 2019-10-13T06:00:30.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:30.458+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.459265+0000) 2019-10-13T06:00:30.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:30.462+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89908 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:30.766 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:30.758+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:30.758274+0000) 2019-10-13T06:00:31.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:31.844+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:31.845666+0000) 2019-10-13T06:00:32.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:32.711+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:32.711905+0000) 2019-10-13T06:00:32.720 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:32.711+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:32.712498+0000) 2019-10-13T06:00:34.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:34.206+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:34.206888+0000) 2019-10-13T06:00:35.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:35.464+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.465752+0000) 2019-10-13T06:00:35.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:35.464+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.466142+0000) 2019-10-13T06:00:35.470 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:35.465+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.466433+0000) 2019-10-13T06:00:35.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:35.465+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.466813+0000) 2019-10-13T06:00:35.471 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:35.465+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.467029+0000) 2019-10-13T06:00:35.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:35.465+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.467241+0000) 2019-10-13T06:00:35.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:35.461+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.461830+0000) 2019-10-13T06:00:35.473 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:35.464+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:35.465576+0000) 2019-10-13T06:00:35.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:35.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:38.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:38.251+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:38.252464+0000) 2019-10-13T06:00:38.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:38.591+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:38.592219+0000) 2019-10-13T06:00:38.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:38.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:38.592219+0000) 2019-10-13T06:00:40.476 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:40.470+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.472067+0000) 2019-10-13T06:00:40.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:40.470+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.472285+0000) 2019-10-13T06:00:40.476 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:40.471+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.472513+0000) 2019-10-13T06:00:40.477 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:40.471+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.472701+0000) 2019-10-13T06:00:40.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:40.471+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.473036+0000) 2019-10-13T06:00:40.478 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:40.465+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.467698+0000) 2019-10-13T06:00:40.478 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:40.472+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.473609+0000) 2019-10-13T06:00:40.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:40.469+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.470906+0000) 2019-10-13T06:00:40.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:40.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 89971 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:40.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:40.759+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:40.760544+0000) 2019-10-13T06:00:41.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:41.845+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:41.845848+0000) 2019-10-13T06:00:42.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:42.711+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:42.712075+0000) 2019-10-13T06:00:42.720 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:42.710+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:42.712673+0000) 2019-10-13T06:00:44.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:44.206+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:44.207000+0000) 2019-10-13T06:00:45.480 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:45.471+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.472743+0000) 2019-10-13T06:00:45.481 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:45.476+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.477610+0000) 2019-10-13T06:00:45.481 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:45.476+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.477879+0000) 2019-10-13T06:00:45.481 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:45.476+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.478044+0000) 2019-10-13T06:00:45.482 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:45.476+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.478188+0000) 2019-10-13T06:00:45.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:45.473+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.474866+0000) 2019-10-13T06:00:45.482 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:45.477+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.478700+0000) 2019-10-13T06:00:45.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:45.476+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:45.477895+0000) 2019-10-13T06:00:45.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:45.480+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:48.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:48.252+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:48.252654+0000) 2019-10-13T06:00:48.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:48.591+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:48.592428+0000) 2019-10-13T06:00:48.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:48.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:48.592433+0000) 2019-10-13T06:00:50.486 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:50.476+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.478431+0000) 2019-10-13T06:00:50.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:50.482+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.483192+0000) 2019-10-13T06:00:50.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:50.483+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.483461+0000) 2019-10-13T06:00:50.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:50.483+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.483636+0000) 2019-10-13T06:00:50.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:50.483+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.483732+0000) 2019-10-13T06:00:50.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:50.483+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.484051+0000) 2019-10-13T06:00:50.488 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:50.483+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.484332+0000) 2019-10-13T06:00:50.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:50.479+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.481520+0000) 2019-10-13T06:00:50.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:50.485+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90032 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:50.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:50.762+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:50.762976+0000) 2019-10-13T06:00:51.359 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:00:51.359 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:51.373 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:00:51.373 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:51.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:51.845+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:51.846081+0000) 2019-10-13T06:00:52.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:52.711+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:52.712263+0000) 2019-10-13T06:00:52.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:52.712+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:52.712887+0000) 2019-10-13T06:00:54.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:54.206+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:54.207177+0000) 2019-10-13T06:00:55.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:00:55.487+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.487765+0000) 2019-10-13T06:00:55.491 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:55.487+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.487961+0000) 2019-10-13T06:00:55.491 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:55.487+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.488180+0000) 2019-10-13T06:00:55.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:55.487+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.488261+0000) 2019-10-13T06:00:55.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:00:55.483+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.484196+0000) 2019-10-13T06:00:55.493 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:00:55.488+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.488731+0000) 2019-10-13T06:00:55.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:00:55.488+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.489141+0000) 2019-10-13T06:00:55.494 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:00:55.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90063 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:00:55.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:00:55.486+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:55.487186+0000) 2019-10-13T06:00:58.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:00:58.252+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:58.252851+0000) 2019-10-13T06:00:58.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:00:58.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:58.592623+0000) 2019-10-13T06:00:58.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:00:58.592+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:00:58.592623+0000) 2019-10-13T06:01:00.496 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:00.492+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.493002+0000) 2019-10-13T06:01:00.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:00.492+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.493142+0000) 2019-10-13T06:01:00.497 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:00.493+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.493458+0000) 2019-10-13T06:01:00.497 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:00.493+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.493703+0000) 2019-10-13T06:01:00.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:00.493+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.493892+0000) 2019-10-13T06:01:00.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:00.493+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.494161+0000) 2019-10-13T06:01:00.498 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:00.489+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.490027+0000) 2019-10-13T06:01:00.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:00.490+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.491076+0000) 2019-10-13T06:01:00.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:00.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:00.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:00.764+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:00.764957+0000) 2019-10-13T06:01:01.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:01.845+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:01.846313+0000) 2019-10-13T06:01:02.716 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:02.712+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:02.712406+0000) 2019-10-13T06:01:02.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:02.712+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:02.713076+0000) 2019-10-13T06:01:04.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:04.207+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:04.207356+0000) 2019-10-13T06:01:05.506 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:05.498+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.499168+0000) 2019-10-13T06:01:05.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:05.499+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.499391+0000) 2019-10-13T06:01:05.507 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:05.499+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.499697+0000) 2019-10-13T06:01:05.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:05.499+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.499813+0000) 2019-10-13T06:01:05.508 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:05.499+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.500062+0000) 2019-10-13T06:01:05.508 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:05.499+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.500268+0000) 2019-10-13T06:01:05.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:05.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:05.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:05.493+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.494438+0000) 2019-10-13T06:01:05.525 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:05.496+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:05.497285+0000) 2019-10-13T06:01:08.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:08.252+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:08.253046+0000) 2019-10-13T06:01:08.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:08.591+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:08.592789+0000) 2019-10-13T06:01:08.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:08.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:08.592790+0000) 2019-10-13T06:01:10.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:10.503+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.504823+0000) 2019-10-13T06:01:10.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:10.503+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.504972+0000) 2019-10-13T06:01:10.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:10.500+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.500927+0000) 2019-10-13T06:01:10.509 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:10.504+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.505699+0000) 2019-10-13T06:01:10.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:10.504+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.505757+0000) 2019-10-13T06:01:10.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:10.504+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.505856+0000) 2019-10-13T06:01:10.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:10.504+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.506051+0000) 2019-10-13T06:01:10.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:10.501+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.502145+0000) 2019-10-13T06:01:10.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:10.507+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:10.776 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:10.766+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:10.767263+0000) 2019-10-13T06:01:11.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:11.845+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:11.846539+0000) 2019-10-13T06:01:12.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:12.711+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:12.712536+0000) 2019-10-13T06:01:12.720 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:12.712+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:12.713245+0000) 2019-10-13T06:01:14.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:14.206+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:14.207563+0000) 2019-10-13T06:01:15.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:15.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.509872+0000) 2019-10-13T06:01:15.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:15.508+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.510160+0000) 2019-10-13T06:01:15.514 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:15.508+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.510347+0000) 2019-10-13T06:01:15.514 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:15.509+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.510569+0000) 2019-10-13T06:01:15.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:15.505+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.506408+0000) 2019-10-13T06:01:15.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:15.508+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.508781+0000) 2019-10-13T06:01:15.517 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:15.509+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.510447+0000) 2019-10-13T06:01:15.517 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:15.509+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:15.510670+0000) 2019-10-13T06:01:15.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:15.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:18.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:18.252+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:18.253242+0000) 2019-10-13T06:01:18.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:18.592+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:18.592912+0000) 2019-10-13T06:01:18.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:18.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:18.592992+0000) 2019-10-13T06:01:20.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:20.509+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.510404+0000) 2019-10-13T06:01:20.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:20.512+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.514214+0000) 2019-10-13T06:01:20.520 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:20.512+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.514364+0000) 2019-10-13T06:01:20.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:20.513+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.514600+0000) 2019-10-13T06:01:20.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:20.512+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.513974+0000) 2019-10-13T06:01:20.521 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:20.513+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.514805+0000) 2019-10-13T06:01:20.521 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:20.513+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.515065+0000) 2019-10-13T06:01:20.521 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:20.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:20.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:20.512+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.513334+0000) 2019-10-13T06:01:20.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:20.769+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:20.769802+0000) 2019-10-13T06:01:21.508 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:01:21.509 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:21.516 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:01:21.516 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:21.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:21.846+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:21.846748+0000) 2019-10-13T06:01:22.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:22.711+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:22.712710+0000) 2019-10-13T06:01:22.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:22.713+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:22.713392+0000) 2019-10-13T06:01:24.211 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:24.207+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:24.207759+0000) 2019-10-13T06:01:25.523 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:25.517+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.519328+0000) 2019-10-13T06:01:25.523 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:25.518+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.519511+0000) 2019-10-13T06:01:25.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:25.517+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.519175+0000) 2019-10-13T06:01:25.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:25.518+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.519655+0000) 2019-10-13T06:01:25.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:25.518+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.520003+0000) 2019-10-13T06:01:25.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:25.515+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.515875+0000) 2019-10-13T06:01:25.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:25.519+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.520627+0000) 2019-10-13T06:01:25.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:25.517+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:25.517881+0000) 2019-10-13T06:01:25.526 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:25.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90250 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:28.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:28.252+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:28.253419+0000) 2019-10-13T06:01:28.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:28.592+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:28.593113+0000) 2019-10-13T06:01:28.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:28.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:28.593155+0000) 2019-10-13T06:01:30.527 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:30.522+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.523992+0000) 2019-10-13T06:01:30.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:30.522+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.524180+0000) 2019-10-13T06:01:30.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:30.522+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.524323+0000) 2019-10-13T06:01:30.529 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:30.519+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.520144+0000) 2019-10-13T06:01:30.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:30.523+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.524758+0000) 2019-10-13T06:01:30.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:30.523+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.524965+0000) 2019-10-13T06:01:30.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:30.523+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.525386+0000) 2019-10-13T06:01:30.547 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:30.522+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.522937+0000) 2019-10-13T06:01:30.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:30.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90280 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:30.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:30.771+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:30.772070+0000) 2019-10-13T06:01:31.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:31.845+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:31.846912+0000) 2019-10-13T06:01:32.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:32.712+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:32.712894+0000) 2019-10-13T06:01:32.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:32.712+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:32.713567+0000) 2019-10-13T06:01:34.211 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:34.207+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:34.207917+0000) 2019-10-13T06:01:35.537 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:35.523+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.525647+0000) 2019-10-13T06:01:35.538 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:35.526+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.528024+0000) 2019-10-13T06:01:35.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:35.527+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.529366+0000) 2019-10-13T06:01:35.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:35.528+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.529804+0000) 2019-10-13T06:01:35.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:35.528+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.530155+0000) 2019-10-13T06:01:35.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:35.528+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.530250+0000) 2019-10-13T06:01:35.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:35.529+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.530478+0000) 2019-10-13T06:01:35.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:35.529+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:35.530582+0000) 2019-10-13T06:01:35.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:35.531+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90311 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:38.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:38.253+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:38.253596+0000) 2019-10-13T06:01:38.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:38.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:38.593257+0000) 2019-10-13T06:01:38.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:38.592+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:38.593283+0000) 2019-10-13T06:01:40.538 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:40.534+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.534820+0000) 2019-10-13T06:01:40.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:40.534+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.535029+0000) 2019-10-13T06:01:40.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:40.534+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.535352+0000) 2019-10-13T06:01:40.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:40.529+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.531245+0000) 2019-10-13T06:01:40.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:40.535+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.535929+0000) 2019-10-13T06:01:40.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:40.535+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.536031+0000) 2019-10-13T06:01:40.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:40.535+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.536185+0000) 2019-10-13T06:01:40.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:40.531+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.532983+0000) 2019-10-13T06:01:40.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:40.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:40.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:40.773+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:40.774493+0000) 2019-10-13T06:01:41.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:41.846+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:41.847083+0000) 2019-10-13T06:01:42.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:42.712+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:42.713052+0000) 2019-10-13T06:01:42.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:42.712+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:42.713737+0000) 2019-10-13T06:01:44.211 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:44.207+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:44.208098+0000) 2019-10-13T06:01:45.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:45.539+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.540374+0000) 2019-10-13T06:01:45.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:45.540+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.540651+0000) 2019-10-13T06:01:45.544 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:45.540+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.540826+0000) 2019-10-13T06:01:45.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:45.540+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.540995+0000) 2019-10-13T06:01:45.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:45.534+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.536692+0000) 2019-10-13T06:01:45.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:45.541+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.541526+0000) 2019-10-13T06:01:45.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:45.541+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.541750+0000) 2019-10-13T06:01:45.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:45.537+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:45.539299+0000) 2019-10-13T06:01:45.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:45.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:48.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:48.253+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:48.253797+0000) 2019-10-13T06:01:48.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:48.593+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:48.593446+0000) 2019-10-13T06:01:48.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:48.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:48.593415+0000) 2019-10-13T06:01:50.549 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:50.545+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.546232+0000) 2019-10-13T06:01:50.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:50.546+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.546566+0000) 2019-10-13T06:01:50.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:50.546+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.546771+0000) 2019-10-13T06:01:50.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:50.541+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.541967+0000) 2019-10-13T06:01:50.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:50.546+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.546965+0000) 2019-10-13T06:01:50.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:50.546+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.547088+0000) 2019-10-13T06:01:50.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:50.547+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.547843+0000) 2019-10-13T06:01:50.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:50.544+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.545010+0000) 2019-10-13T06:01:50.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:50.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:50.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:50.776+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:50.776931+0000) 2019-10-13T06:01:51.641 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:01:51.641 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:51.645 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:01:51.645 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:51.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:51.846+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:51.847256+0000) 2019-10-13T06:01:52.716 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:52.713+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:52.713213+0000) 2019-10-13T06:01:52.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:52.713+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:52.713909+0000) 2019-10-13T06:01:54.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:54.207+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:54.208309+0000) 2019-10-13T06:01:55.555 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:01:55.550+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.552243+0000) 2019-10-13T06:01:55.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:55.551+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.552431+0000) 2019-10-13T06:01:55.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:01:55.547+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.548317+0000) 2019-10-13T06:01:55.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:55.551+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.552899+0000) 2019-10-13T06:01:55.558 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:01:55.551+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.553121+0000) 2019-10-13T06:01:55.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:55.552+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.553415+0000) 2019-10-13T06:01:55.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:01:55.552+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.553638+0000) 2019-10-13T06:01:55.559 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:01:55.549+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:55.549762+0000) 2019-10-13T06:01:55.560 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:01:55.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:01:58.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:01:58.253+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:58.253987+0000) 2019-10-13T06:01:58.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:01:58.592+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:58.593613+0000) 2019-10-13T06:01:58.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:01:58.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:01:58.593609+0000) 2019-10-13T06:02:00.561 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:00.556+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.558103+0000) 2019-10-13T06:02:00.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:00.557+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.558393+0000) 2019-10-13T06:02:00.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:00.557+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.558499+0000) 2019-10-13T06:02:00.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:00.557+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.558612+0000) 2019-10-13T06:02:00.562 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:00.557+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.558828+0000) 2019-10-13T06:02:00.563 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:00.553+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.554641+0000) 2019-10-13T06:02:00.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:00.557+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.559362+0000) 2019-10-13T06:02:00.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:00.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:00.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:00.556+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.556933+0000) 2019-10-13T06:02:00.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:00.778+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:00.779249+0000) 2019-10-13T06:02:01.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:01.846+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:01.847429+0000) 2019-10-13T06:02:02.716 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:02.712+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:02.713402+0000) 2019-10-13T06:02:02.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:02.713+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:02.714107+0000) 2019-10-13T06:02:04.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:04.207+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:04.208503+0000) 2019-10-13T06:02:05.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:05.561+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.562953+0000) 2019-10-13T06:02:05.567 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:05.558+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.558926+0000) 2019-10-13T06:02:05.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:05.562+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.563557+0000) 2019-10-13T06:02:05.568 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:05.562+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.563777+0000) 2019-10-13T06:02:05.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:05.562+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.564005+0000) 2019-10-13T06:02:05.568 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:05.562+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.564183+0000) 2019-10-13T06:02:05.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:05.562+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.564359+0000) 2019-10-13T06:02:05.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:05.560+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:05.560835+0000) 2019-10-13T06:02:05.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:05.565+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90498 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:08.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:08.252+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:08.254221+0000) 2019-10-13T06:02:08.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:08.593+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:08.593829+0000) 2019-10-13T06:02:08.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:08.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:08.593832+0000) 2019-10-13T06:02:10.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:10.566+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.568139+0000) 2019-10-13T06:02:10.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:10.566+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.568311+0000) 2019-10-13T06:02:10.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:10.567+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.568905+0000) 2019-10-13T06:02:10.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:10.567+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.568997+0000) 2019-10-13T06:02:10.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:10.567+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.569136+0000) 2019-10-13T06:02:10.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:10.567+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.569264+0000) 2019-10-13T06:02:10.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:10.563+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.564018+0000) 2019-10-13T06:02:10.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:10.565+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.566266+0000) 2019-10-13T06:02:10.575 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:10.570+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:10.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:10.781+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:10.781622+0000) 2019-10-13T06:02:11.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:11.846+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:11.847642+0000) 2019-10-13T06:02:12.716 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:12.712+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:12.713554+0000) 2019-10-13T06:02:12.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:12.714+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:12.714294+0000) 2019-10-13T06:02:14.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:14.208+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:14.208693+0000) 2019-10-13T06:02:15.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:15.571+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.573192+0000) 2019-10-13T06:02:15.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:15.572+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.573446+0000) 2019-10-13T06:02:15.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:15.571+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.573343+0000) 2019-10-13T06:02:15.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:15.572+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.573764+0000) 2019-10-13T06:02:15.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:15.572+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.573903+0000) 2019-10-13T06:02:15.578 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:15.572+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.574027+0000) 2019-10-13T06:02:15.579 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:15.568+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.569408+0000) 2019-10-13T06:02:15.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:15.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90559 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:15.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:15.571+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:15.572108+0000) 2019-10-13T06:02:18.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:18.253+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:18.254424+0000) 2019-10-13T06:02:18.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:18.593+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:18.594019+0000) 2019-10-13T06:02:18.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:18.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:18.594021+0000) 2019-10-13T06:02:20.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:20.576+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.578188+0000) 2019-10-13T06:02:20.583 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:20.577+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.578435+0000) 2019-10-13T06:02:20.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:20.577+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.578831+0000) 2019-10-13T06:02:20.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:20.577+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.578957+0000) 2019-10-13T06:02:20.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:20.577+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.579118+0000) 2019-10-13T06:02:20.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:20.573+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.574166+0000) 2019-10-13T06:02:20.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:20.578+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.579583+0000) 2019-10-13T06:02:20.585 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:20.576+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.577513+0000) 2019-10-13T06:02:20.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:20.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:20.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:20.783+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:20.784045+0000) 2019-10-13T06:02:21.777 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:02:21.777 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:21.781 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:02:21.781 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:21.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:21.847+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:21.847783+0000) 2019-10-13T06:02:22.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:22.713+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:22.713742+0000) 2019-10-13T06:02:22.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:22.714+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:22.714482+0000) 2019-10-13T06:02:24.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:24.208+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:24.208870+0000) 2019-10-13T06:02:25.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:25.583+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.584169+0000) 2019-10-13T06:02:25.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:25.584+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.584410+0000) 2019-10-13T06:02:25.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:25.584+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.584513+0000) 2019-10-13T06:02:25.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:25.584+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.584713+0000) 2019-10-13T06:02:25.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:25.584+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.584970+0000) 2019-10-13T06:02:25.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:25.584+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.585169+0000) 2019-10-13T06:02:25.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:25.578+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.580515+0000) 2019-10-13T06:02:25.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:25.581+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:25.583077+0000) 2019-10-13T06:02:25.591 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:25.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90622 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:28.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:28.254+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:28.254589+0000) 2019-10-13T06:02:28.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:28.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:28.594156+0000) 2019-10-13T06:02:28.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:28.593+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:28.594180+0000) 2019-10-13T06:02:30.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:30.588+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.589132+0000) 2019-10-13T06:02:30.592 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:30.589+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.589531+0000) 2019-10-13T06:02:30.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:30.589+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.589781+0000) 2019-10-13T06:02:30.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:30.589+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.590115+0000) 2019-10-13T06:02:30.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:30.589+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.590225+0000) 2019-10-13T06:02:30.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:30.584+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.586172+0000) 2019-10-13T06:02:30.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:30.590+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.590906+0000) 2019-10-13T06:02:30.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:30.585+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.587127+0000) 2019-10-13T06:02:30.595 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:30.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90652 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:30.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:30.785+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:30.786423+0000) 2019-10-13T06:02:31.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:31.847+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:31.847987+0000) 2019-10-13T06:02:32.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:32.713+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:32.713942+0000) 2019-10-13T06:02:32.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:32.713+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:32.714685+0000) 2019-10-13T06:02:34.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:34.208+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:34.209034+0000) 2019-10-13T06:02:35.597 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:35.593+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.594328+0000) 2019-10-13T06:02:35.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:35.594+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.594635+0000) 2019-10-13T06:02:35.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:35.588+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.590442+0000) 2019-10-13T06:02:35.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:35.594+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.595066+0000) 2019-10-13T06:02:35.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:35.594+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.595130+0000) 2019-10-13T06:02:35.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:35.594+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.595267+0000) 2019-10-13T06:02:35.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:35.595+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.596022+0000) 2019-10-13T06:02:35.600 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:35.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:35.601 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:35.592+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:35.593807+0000) 2019-10-13T06:02:38.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:38.254+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:38.254771+0000) 2019-10-13T06:02:38.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:38.594+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:38.594332+0000) 2019-10-13T06:02:38.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:38.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:38.594329+0000) 2019-10-13T06:02:40.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:40.598+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.599458+0000) 2019-10-13T06:02:40.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:40.598+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.600009+0000) 2019-10-13T06:02:40.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:40.598+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.600281+0000) 2019-10-13T06:02:40.604 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:40.599+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.600392+0000) 2019-10-13T06:02:40.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:40.599+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.600538+0000) 2019-10-13T06:02:40.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:40.595+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.596403+0000) 2019-10-13T06:02:40.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:40.599+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.601008+0000) 2019-10-13T06:02:40.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:40.597+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.597939+0000) 2019-10-13T06:02:40.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:40.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:40.796 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:40.788+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:40.788861+0000) 2019-10-13T06:02:41.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:41.846+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:41.848184+0000) 2019-10-13T06:02:42.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:42.713+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:42.714125+0000) 2019-10-13T06:02:42.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:42.714+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:42.714895+0000) 2019-10-13T06:02:44.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:44.208+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:44.209192+0000) 2019-10-13T06:02:45.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:45.599+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.600388+0000) 2019-10-13T06:02:45.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:45.603+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.605132+0000) 2019-10-13T06:02:45.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:45.603+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.605292+0000) 2019-10-13T06:02:45.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:45.604+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.605565+0000) 2019-10-13T06:02:45.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:45.604+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.605764+0000) 2019-10-13T06:02:45.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:45.604+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.605854+0000) 2019-10-13T06:02:45.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:45.604+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.606028+0000) 2019-10-13T06:02:45.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:45.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:45.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:45.602+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:45.603421+0000) 2019-10-13T06:02:48.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:48.254+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:48.254947+0000) 2019-10-13T06:02:48.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:48.593+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:48.594519+0000) 2019-10-13T06:02:48.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:48.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:48.594521+0000) 2019-10-13T06:02:50.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:50.608+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.609885+0000) 2019-10-13T06:02:50.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:50.608+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.610129+0000) 2019-10-13T06:02:50.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:50.605+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.605845+0000) 2019-10-13T06:02:50.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:50.608+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.610378+0000) 2019-10-13T06:02:50.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:50.609+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.610498+0000) 2019-10-13T06:02:50.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:50.609+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.610829+0000) 2019-10-13T06:02:50.615 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:50.609+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.610805+0000) 2019-10-13T06:02:50.616 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:50.608+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.609028+0000) 2019-10-13T06:02:50.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:50.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90776 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:50.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:50.790+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:50.790908+0000) 2019-10-13T06:02:51.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:51.847+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:51.848367+0000) 2019-10-13T06:02:51.915 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:02:51.916 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:51.920 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:02:51.920 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:52.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:52.712+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:52.714333+0000) 2019-10-13T06:02:52.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:52.714+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:52.715079+0000) 2019-10-13T06:02:54.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:54.208+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:54.209383+0000) 2019-10-13T06:02:55.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:02:55.610+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.610985+0000) 2019-10-13T06:02:55.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:55.614+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.615506+0000) 2019-10-13T06:02:55.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:02:55.614+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.616115+0000) 2019-10-13T06:02:55.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:02:55.615+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.616423+0000) 2019-10-13T06:02:55.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:55.615+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.616596+0000) 2019-10-13T06:02:55.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:55.615+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.616849+0000) 2019-10-13T06:02:55.620 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:02:55.615+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.616995+0000) 2019-10-13T06:02:55.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:02:55.612+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:55.613594+0000) 2019-10-13T06:02:55.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:02:55.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90807 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:02:58.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:02:58.254+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:58.255147+0000) 2019-10-13T06:02:58.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:02:58.594+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:58.594676+0000) 2019-10-13T06:02:58.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:02:58.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:02:58.594675+0000) 2019-10-13T06:03:00.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:00.620+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.621464+0000) 2019-10-13T06:03:00.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:00.620+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.621707+0000) 2019-10-13T06:03:00.625 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:00.620+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.621854+0000) 2019-10-13T06:03:00.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:00.616+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.617689+0000) 2019-10-13T06:03:00.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:00.621+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.622407+0000) 2019-10-13T06:03:00.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:00.621+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.622572+0000) 2019-10-13T06:03:00.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:00.621+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.622979+0000) 2019-10-13T06:03:00.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:00.619+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.619918+0000) 2019-10-13T06:03:00.628 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:00.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:00.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:00.792+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:00.792946+0000) 2019-10-13T06:03:01.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:01.847+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:01.848557+0000) 2019-10-13T06:03:02.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:02.713+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:02.714523+0000) 2019-10-13T06:03:02.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:02.714+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:02.715290+0000) 2019-10-13T06:03:04.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:04.209+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:04.209580+0000) 2019-10-13T06:03:05.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:05.622+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.622851+0000) 2019-10-13T06:03:05.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:05.626+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.628213+0000) 2019-10-13T06:03:05.631 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:05.626+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.628373+0000) 2019-10-13T06:03:05.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:05.627+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.628548+0000) 2019-10-13T06:03:05.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:05.627+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.628722+0000) 2019-10-13T06:03:05.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:05.627+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.628929+0000) 2019-10-13T06:03:05.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:05.627+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.629150+0000) 2019-10-13T06:03:05.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:05.624+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:05.625044+0000) 2019-10-13T06:03:05.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:05.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90870 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:08.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:08.254+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:08.255294+0000) 2019-10-13T06:03:08.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:08.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:08.594863+0000) 2019-10-13T06:03:08.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:08.594+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:08.594862+0000) 2019-10-13T06:03:10.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:10.633+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.633615+0000) 2019-10-13T06:03:10.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:10.633+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.633848+0000) 2019-10-13T06:03:10.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:10.633+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.634118+0000) 2019-10-13T06:03:10.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:10.629+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.630083+0000) 2019-10-13T06:03:10.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:10.633+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.634300+0000) 2019-10-13T06:03:10.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:10.634+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.634559+0000) 2019-10-13T06:03:10.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:10.634+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.634964+0000) 2019-10-13T06:03:10.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:10.631+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.632395+0000) 2019-10-13T06:03:10.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:10.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:10.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:10.794+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:10.794964+0000) 2019-10-13T06:03:11.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:11.848+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:11.848753+0000) 2019-10-13T06:03:12.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:12.714+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:12.714700+0000) 2019-10-13T06:03:12.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:12.715+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:12.715473+0000) 2019-10-13T06:03:14.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:14.209+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:14.209770+0000) 2019-10-13T06:03:15.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:15.638+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.638804+0000) 2019-10-13T06:03:15.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:15.638+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.639049+0000) 2019-10-13T06:03:15.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:15.638+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.639254+0000) 2019-10-13T06:03:15.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:15.634+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.635015+0000) 2019-10-13T06:03:15.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:15.639+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.639562+0000) 2019-10-13T06:03:15.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:15.639+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.639739+0000) 2019-10-13T06:03:15.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:15.639+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.639937+0000) 2019-10-13T06:03:15.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:15.636+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:15.637007+0000) 2019-10-13T06:03:15.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:15.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:18.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:18.255+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:18.255468+0000) 2019-10-13T06:03:18.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:18.594+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:18.595048+0000) 2019-10-13T06:03:18.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:18.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:18.595049+0000) 2019-10-13T06:03:20.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:20.643+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.644321+0000) 2019-10-13T06:03:20.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:20.644+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.644531+0000) 2019-10-13T06:03:20.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:20.638+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.640672+0000) 2019-10-13T06:03:20.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:20.644+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.644905+0000) 2019-10-13T06:03:20.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:20.644+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.645191+0000) 2019-10-13T06:03:20.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:20.645+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.645526+0000) 2019-10-13T06:03:20.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:20.645+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.645662+0000) 2019-10-13T06:03:20.650 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:20.641+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.643106+0000) 2019-10-13T06:03:20.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:20.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:20.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:20.795+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:20.797391+0000) 2019-10-13T06:03:21.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:21.848+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:21.848957+0000) 2019-10-13T06:03:22.047 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:03:22.047 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:22.052 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:03:22.052 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:22.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:22.714+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:22.714880+0000) 2019-10-13T06:03:22.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:22.714+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:22.715671+0000) 2019-10-13T06:03:24.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:24.209+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:24.209932+0000) 2019-10-13T06:03:25.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:25.648+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.649951+0000) 2019-10-13T06:03:25.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:25.648+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.650177+0000) 2019-10-13T06:03:25.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:25.649+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.650553+0000) 2019-10-13T06:03:25.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:25.649+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.650660+0000) 2019-10-13T06:03:25.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:25.649+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.650789+0000) 2019-10-13T06:03:25.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:25.649+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.650966+0000) 2019-10-13T06:03:25.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:25.645+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.647071+0000) 2019-10-13T06:03:25.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:25.647+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:25.648890+0000) 2019-10-13T06:03:25.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:25.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 90994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:28.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:28.254+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:28.255654+0000) 2019-10-13T06:03:28.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:28.594+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:28.595257+0000) 2019-10-13T06:03:28.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:28.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:28.595234+0000) 2019-10-13T06:03:30.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:30.653+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.655017+0000) 2019-10-13T06:03:30.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:30.653+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.655246+0000) 2019-10-13T06:03:30.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:30.649+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.651596+0000) 2019-10-13T06:03:30.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:30.654+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.655561+0000) 2019-10-13T06:03:30.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:30.654+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.655697+0000) 2019-10-13T06:03:30.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:30.654+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.656083+0000) 2019-10-13T06:03:30.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:30.654+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.656319+0000) 2019-10-13T06:03:30.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:30.651+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.653429+0000) 2019-10-13T06:03:30.661 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:30.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91024 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:30.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:30.798+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:30.799435+0000) 2019-10-13T06:03:31.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:31.847+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:31.849180+0000) 2019-10-13T06:03:32.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:32.713+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:32.715084+0000) 2019-10-13T06:03:32.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:32.715+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:32.715893+0000) 2019-10-13T06:03:34.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:34.209+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:34.210101+0000) 2019-10-13T06:03:35.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:35.654+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.655632+0000) 2019-10-13T06:03:35.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:35.659+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.660391+0000) 2019-10-13T06:03:35.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:35.659+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.660528+0000) 2019-10-13T06:03:35.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:35.659+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.660704+0000) 2019-10-13T06:03:35.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:35.659+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.660940+0000) 2019-10-13T06:03:35.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:35.659+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.661332+0000) 2019-10-13T06:03:35.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:35.659+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.661336+0000) 2019-10-13T06:03:35.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:35.657+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:35.658658+0000) 2019-10-13T06:03:35.667 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:35.662+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:38.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:38.255+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:38.255804+0000) 2019-10-13T06:03:38.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:38.594+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:38.595430+0000) 2019-10-13T06:03:38.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:38.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:38.595389+0000) 2019-10-13T06:03:40.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:40.663+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.665373+0000) 2019-10-13T06:03:40.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:40.664+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.665631+0000) 2019-10-13T06:03:40.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:40.664+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.666083+0000) 2019-10-13T06:03:40.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:40.664+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.666211+0000) 2019-10-13T06:03:40.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:40.661+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.662229+0000) 2019-10-13T06:03:40.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:40.665+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.666847+0000) 2019-10-13T06:03:40.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:40.665+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.667089+0000) 2019-10-13T06:03:40.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:40.664+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.664993+0000) 2019-10-13T06:03:40.673 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:40.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91087 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:40.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:40.800+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:40.801528+0000) 2019-10-13T06:03:41.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:41.848+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:41.849371+0000) 2019-10-13T06:03:42.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:42.713+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:42.715295+0000) 2019-10-13T06:03:42.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:42.715+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:42.716059+0000) 2019-10-13T06:03:44.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:44.209+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:44.210250+0000) 2019-10-13T06:03:45.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:45.670+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.671866+0000) 2019-10-13T06:03:45.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:45.670+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.671731+0000) 2019-10-13T06:03:45.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:45.670+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.672067+0000) 2019-10-13T06:03:45.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:45.670+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.672190+0000) 2019-10-13T06:03:45.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:45.671+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.672466+0000) 2019-10-13T06:03:45.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:45.666+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.667579+0000) 2019-10-13T06:03:45.677 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:45.668+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.669244+0000) 2019-10-13T06:03:45.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:45.671+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:45.672617+0000) 2019-10-13T06:03:45.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:45.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:48.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:48.254+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:48.256004+0000) 2019-10-13T06:03:48.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:48.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:48.595551+0000) 2019-10-13T06:03:48.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:48.595+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:48.595587+0000) 2019-10-13T06:03:50.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:50.671+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.672548+0000) 2019-10-13T06:03:50.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:50.675+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.677164+0000) 2019-10-13T06:03:50.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:50.675+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.677328+0000) 2019-10-13T06:03:50.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:50.676+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.677600+0000) 2019-10-13T06:03:50.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:50.676+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.677915+0000) 2019-10-13T06:03:50.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:50.676+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.678269+0000) 2019-10-13T06:03:50.682 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:50.676+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.678362+0000) 2019-10-13T06:03:50.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:50.674+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.674864+0000) 2019-10-13T06:03:50.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:50.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91148 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:50.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:50.803+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:50.803582+0000) 2019-10-13T06:03:51.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:51.849+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:51.849522+0000) 2019-10-13T06:03:52.169 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:03:52.169 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:52.173 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:03:52.173 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:52.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:52.715+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:52.715477+0000) 2019-10-13T06:03:52.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:52.715+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:52.716265+0000) 2019-10-13T06:03:54.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:54.210+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:54.210421+0000) 2019-10-13T06:03:55.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:55.682+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.682450+0000) 2019-10-13T06:03:55.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:55.682+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.682602+0000) 2019-10-13T06:03:55.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:03:55.682+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.683023+0000) 2019-10-13T06:03:55.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:03:55.678+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.679171+0000) 2019-10-13T06:03:55.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:03:55.682+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.683239+0000) 2019-10-13T06:03:55.688 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:55.683+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.683787+0000) 2019-10-13T06:03:55.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:03:55.683+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.683960+0000) 2019-10-13T06:03:55.688 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:03:55.680+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:55.681469+0000) 2019-10-13T06:03:55.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:03:55.686+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:03:58.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:03:58.255+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:58.256167+0000) 2019-10-13T06:03:58.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:03:58.595+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:58.595724+0000) 2019-10-13T06:03:58.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:03:58.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:03:58.595718+0000) 2019-10-13T06:04:00.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:00.688+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.688609+0000) 2019-10-13T06:04:00.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:00.688+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.688694+0000) 2019-10-13T06:04:00.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:00.688+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.688936+0000) 2019-10-13T06:04:00.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:00.684+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.685482+0000) 2019-10-13T06:04:00.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:00.685+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.685965+0000) 2019-10-13T06:04:00.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:00.688+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.689249+0000) 2019-10-13T06:04:00.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:00.688+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.689362+0000) 2019-10-13T06:04:00.694 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:00.689+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.689518+0000) 2019-10-13T06:04:00.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:00.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:00.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:00.805+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:00.805834+0000) 2019-10-13T06:04:01.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:01.849+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:01.849702+0000) 2019-10-13T06:04:02.719 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:02.715+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:02.715627+0000) 2019-10-13T06:04:02.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:02.716+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:02.716454+0000) 2019-10-13T06:04:04.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:04.210+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:04.210583+0000) 2019-10-13T06:04:05.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:05.694+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.694543+0000) 2019-10-13T06:04:05.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:05.689+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.690365+0000) 2019-10-13T06:04:05.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:05.694+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.695096+0000) 2019-10-13T06:04:05.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:05.695+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.695960+0000) 2019-10-13T06:04:05.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:05.695+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.696202+0000) 2019-10-13T06:04:05.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:05.696+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.696393+0000) 2019-10-13T06:04:05.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:05.691+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.692693+0000) 2019-10-13T06:04:05.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:05.696+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:05.696625+0000) 2019-10-13T06:04:05.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:05.698+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:08.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:08.255+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:08.256364+0000) 2019-10-13T06:04:08.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:08.595+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:08.595887+0000) 2019-10-13T06:04:08.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:08.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:08.595877+0000) 2019-10-13T06:04:10.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:10.694+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.696129+0000) 2019-10-13T06:04:10.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:10.700+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.700688+0000) 2019-10-13T06:04:10.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:10.700+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.701122+0000) 2019-10-13T06:04:10.705 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:10.700+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.701269+0000) 2019-10-13T06:04:10.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:10.701+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.701501+0000) 2019-10-13T06:04:10.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:10.701+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.702020+0000) 2019-10-13T06:04:10.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:10.701+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.702086+0000) 2019-10-13T06:04:10.708 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:10.697+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.699715+0000) 2019-10-13T06:04:10.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:10.704+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91272 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:10.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:10.807+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:10.808288+0000) 2019-10-13T06:04:11.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:11.848+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:11.849912+0000) 2019-10-13T06:04:12.719 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:12.714+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:12.715777+0000) 2019-10-13T06:04:12.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:12.715+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:12.716641+0000) 2019-10-13T06:04:14.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:14.209+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:14.210757+0000) 2019-10-13T06:04:15.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:15.704+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.705900+0000) 2019-10-13T06:04:15.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:15.704+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.706035+0000) 2019-10-13T06:04:15.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:15.704+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.706256+0000) 2019-10-13T06:04:15.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:15.700+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.701805+0000) 2019-10-13T06:04:15.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:15.705+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.706533+0000) 2019-10-13T06:04:15.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:15.705+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.706970+0000) 2019-10-13T06:04:15.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:15.705+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.707106+0000) 2019-10-13T06:04:15.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:15.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91303 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:15.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:15.702+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:15.704539+0000) 2019-10-13T06:04:18.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:18.255+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:18.256561+0000) 2019-10-13T06:04:18.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:18.595+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:18.596102+0000) 2019-10-13T06:04:18.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:18.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:18.596097+0000) 2019-10-13T06:04:20.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:20.708+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.710303+0000) 2019-10-13T06:04:20.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:20.709+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.710639+0000) 2019-10-13T06:04:20.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:20.704+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.706117+0000) 2019-10-13T06:04:20.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:20.709+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.711127+0000) 2019-10-13T06:04:20.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:20.709+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.711145+0000) 2019-10-13T06:04:20.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:20.709+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.711355+0000) 2019-10-13T06:04:20.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:20.710+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.711521+0000) 2019-10-13T06:04:20.717 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:20.708+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.709844+0000) 2019-10-13T06:04:20.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:20.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:20.818 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:20.809+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:20.810633+0000) 2019-10-13T06:04:21.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:21.848+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:21.850114+0000) 2019-10-13T06:04:22.283 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:04:22.283 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:22.288 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:04:22.288 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:22.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:22.715+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:22.715956+0000) 2019-10-13T06:04:22.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:22.716+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:22.716829+0000) 2019-10-13T06:04:24.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:24.210+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:24.210949+0000) 2019-10-13T06:04:25.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:25.714+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.716030+0000) 2019-10-13T06:04:25.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:25.715+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.716433+0000) 2019-10-13T06:04:25.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:25.715+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.716607+0000) 2019-10-13T06:04:25.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:25.711+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.711926+0000) 2019-10-13T06:04:25.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:25.715+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.716916+0000) 2019-10-13T06:04:25.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:25.715+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.716945+0000) 2019-10-13T06:04:25.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:25.715+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.717346+0000) 2019-10-13T06:04:25.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:25.718+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:25.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:25.714+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:25.715354+0000) 2019-10-13T06:04:28.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:28.255+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:28.256725+0000) 2019-10-13T06:04:28.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:28.595+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:28.596329+0000) 2019-10-13T06:04:28.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:28.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:28.596329+0000) 2019-10-13T06:04:30.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:30.716+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.716829+0000) 2019-10-13T06:04:30.725 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:30.719+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.721331+0000) 2019-10-13T06:04:30.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:30.720+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.721616+0000) 2019-10-13T06:04:30.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:30.720+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.721765+0000) 2019-10-13T06:04:30.725 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:30.720+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.722158+0000) 2019-10-13T06:04:30.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:30.721+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.722533+0000) 2019-10-13T06:04:30.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:30.721+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.722617+0000) 2019-10-13T06:04:30.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:30.719+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.719957+0000) 2019-10-13T06:04:30.728 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:30.723+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:30.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:30.812+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:30.812770+0000) 2019-10-13T06:04:31.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:31.849+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:31.850279+0000) 2019-10-13T06:04:32.719 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:32.715+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:32.716186+0000) 2019-10-13T06:04:32.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:32.716+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:32.717029+0000) 2019-10-13T06:04:34.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:34.210+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:34.211134+0000) 2019-10-13T06:04:35.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:35.721+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.722421+0000) 2019-10-13T06:04:35.731 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:35.725+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.726932+0000) 2019-10-13T06:04:35.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:35.725+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.727106+0000) 2019-10-13T06:04:35.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:35.726+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.727606+0000) 2019-10-13T06:04:35.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:35.726+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.727826+0000) 2019-10-13T06:04:35.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:35.726+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.727858+0000) 2019-10-13T06:04:35.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:35.726+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.728288+0000) 2019-10-13T06:04:35.732 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:35.724+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:35.725256+0000) 2019-10-13T06:04:35.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:35.729+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:38.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:38.256+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:38.256919+0000) 2019-10-13T06:04:38.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:38.596+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:38.596502+0000) 2019-10-13T06:04:38.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:38.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:38.596504+0000) 2019-10-13T06:04:40.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:40.728+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.729010+0000) 2019-10-13T06:04:40.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:40.732+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.733137+0000) 2019-10-13T06:04:40.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:40.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.733599+0000) 2019-10-13T06:04:40.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:40.733+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.733786+0000) 2019-10-13T06:04:40.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:40.733+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.733827+0000) 2019-10-13T06:04:40.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:40.733+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.733993+0000) 2019-10-13T06:04:40.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:40.733+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.734129+0000) 2019-10-13T06:04:40.740 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:40.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:40.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:40.731+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.732372+0000) 2019-10-13T06:04:40.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:40.814+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:40.815081+0000) 2019-10-13T06:04:41.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:41.850+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:41.850470+0000) 2019-10-13T06:04:42.719 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:42.716+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:42.716455+0000) 2019-10-13T06:04:42.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:42.716+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:42.717219+0000) 2019-10-13T06:04:44.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:44.210+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:44.211315+0000) 2019-10-13T06:04:45.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:45.738+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.739161+0000) 2019-10-13T06:04:45.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:45.739+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.739567+0000) 2019-10-13T06:04:45.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:45.739+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.740061+0000) 2019-10-13T06:04:45.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:45.735+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.735885+0000) 2019-10-13T06:04:45.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:45.739+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.740313+0000) 2019-10-13T06:04:45.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:45.739+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.740236+0000) 2019-10-13T06:04:45.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:45.740+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.740588+0000) 2019-10-13T06:04:45.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:45.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91490 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:45.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:45.738+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:45.738988+0000) 2019-10-13T06:04:48.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:48.256+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:48.257120+0000) 2019-10-13T06:04:48.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:48.596+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:48.596660+0000) 2019-10-13T06:04:48.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:48.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:48.596678+0000) 2019-10-13T06:04:50.749 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:50.743+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.744006+0000) 2019-10-13T06:04:50.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:50.743+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.744278+0000) 2019-10-13T06:04:50.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:50.744+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.744688+0000) 2019-10-13T06:04:50.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:50.744+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.744840+0000) 2019-10-13T06:04:50.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:50.744+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.744949+0000) 2019-10-13T06:04:50.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:50.744+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.745205+0000) 2019-10-13T06:04:50.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:50.738+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.739449+0000) 2019-10-13T06:04:50.752 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:50.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91520 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:50.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:50.742+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.743294+0000) 2019-10-13T06:04:50.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:50.817+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:50.817333+0000) 2019-10-13T06:04:51.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:51.850+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:51.850662+0000) 2019-10-13T06:04:52.364 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:04:52.364 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:52.369 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:04:52.369 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:52.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:52.716+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:52.716657+0000) 2019-10-13T06:04:52.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:52.717+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:52.717423+0000) 2019-10-13T06:04:54.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:54.211+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:54.211519+0000) 2019-10-13T06:04:55.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:55.749+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.749751+0000) 2019-10-13T06:04:55.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:04:55.745+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.745786+0000) 2019-10-13T06:04:55.754 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:55.750+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.750442+0000) 2019-10-13T06:04:55.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:04:55.749+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.750195+0000) 2019-10-13T06:04:55.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:55.750+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.750800+0000) 2019-10-13T06:04:55.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:04:55.750+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.751111+0000) 2019-10-13T06:04:55.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:04:55.750+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.751221+0000) 2019-10-13T06:04:55.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:04:55.747+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:55.748705+0000) 2019-10-13T06:04:55.756 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:04:55.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:04:58.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:04:58.255+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:58.257309+0000) 2019-10-13T06:04:58.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:04:58.595+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:58.596842+0000) 2019-10-13T06:04:58.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:04:58.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:04:58.596847+0000) 2019-10-13T06:05:00.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:00.753+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.755110+0000) 2019-10-13T06:05:00.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:00.754+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.755559+0000) 2019-10-13T06:05:00.759 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:00.754+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.755729+0000) 2019-10-13T06:05:00.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:00.750+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.751636+0000) 2019-10-13T06:05:00.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:00.754+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.756069+0000) 2019-10-13T06:05:00.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:00.754+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.756330+0000) 2019-10-13T06:05:00.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:00.755+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.756394+0000) 2019-10-13T06:05:00.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:00.753+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.754382+0000) 2019-10-13T06:05:00.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:00.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:00.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:00.819+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:00.819560+0000) 2019-10-13T06:05:01.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:01.849+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:01.850820+0000) 2019-10-13T06:05:02.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:02.715+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:02.716884+0000) 2019-10-13T06:05:02.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:02.715+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:02.717633+0000) 2019-10-13T06:05:04.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:04.210+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:04.211694+0000) 2019-10-13T06:05:05.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:05.759+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.760906+0000) 2019-10-13T06:05:05.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:05.759+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.761044+0000) 2019-10-13T06:05:05.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:05.759+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.761351+0000) 2019-10-13T06:05:05.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:05.755+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.756841+0000) 2019-10-13T06:05:05.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:05.760+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.761764+0000) 2019-10-13T06:05:05.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:05.760+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.762036+0000) 2019-10-13T06:05:05.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:05.760+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.762207+0000) 2019-10-13T06:05:05.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:05.757+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:05.759403+0000) 2019-10-13T06:05:05.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:05.763+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91614 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:08.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:08.256+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:08.257456+0000) 2019-10-13T06:05:08.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:08.596+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:08.597016+0000) 2019-10-13T06:05:08.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:08.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:08.597017+0000) 2019-10-13T06:05:10.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:10.764+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.766113+0000) 2019-10-13T06:05:10.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:10.764+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.766301+0000) 2019-10-13T06:05:10.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:10.760+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.762290+0000) 2019-10-13T06:05:10.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:10.765+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.766879+0000) 2019-10-13T06:05:10.771 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:10.765+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.767072+0000) 2019-10-13T06:05:10.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:10.766+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.767472+0000) 2019-10-13T06:05:10.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:10.766+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.767604+0000) 2019-10-13T06:05:10.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:10.763+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.765106+0000) 2019-10-13T06:05:10.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:10.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:10.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:10.820+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:10.821772+0000) 2019-10-13T06:05:11.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:11.850+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:11.850998+0000) 2019-10-13T06:05:12.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:12.716+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:12.717043+0000) 2019-10-13T06:05:12.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:12.716+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:12.717817+0000) 2019-10-13T06:05:14.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:14.211+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:14.211872+0000) 2019-10-13T06:05:15.774 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:15.769+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.771328+0000) 2019-10-13T06:05:15.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:15.765+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.767310+0000) 2019-10-13T06:05:15.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:15.770+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.772204+0000) 2019-10-13T06:05:15.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:15.770+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.772318+0000) 2019-10-13T06:05:15.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:15.771+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.772449+0000) 2019-10-13T06:05:15.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:15.771+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.772709+0000) 2019-10-13T06:05:15.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:15.771+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.772917+0000) 2019-10-13T06:05:15.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:15.767+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:15.769724+0000) 2019-10-13T06:05:15.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:15.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:18.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:18.257+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:18.257632+0000) 2019-10-13T06:05:18.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:18.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:18.597167+0000) 2019-10-13T06:05:18.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:18.596+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:18.597168+0000) 2019-10-13T06:05:20.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:20.775+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.777141+0000) 2019-10-13T06:05:20.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:20.772+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.773302+0000) 2019-10-13T06:05:20.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:20.776+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.777732+0000) 2019-10-13T06:05:20.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:20.776+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.778114+0000) 2019-10-13T06:05:20.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:20.776+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.778249+0000) 2019-10-13T06:05:20.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:20.777+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.778532+0000) 2019-10-13T06:05:20.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:20.777+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.778802+0000) 2019-10-13T06:05:20.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:20.775+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.775809+0000) 2019-10-13T06:05:20.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:20.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:20.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:20.823+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:20.824160+0000) 2019-10-13T06:05:21.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:21.850+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:21.851189+0000) 2019-10-13T06:05:22.452 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:05:22.453 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:22.457 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:05:22.457 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:22.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:22.716+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:22.717219+0000) 2019-10-13T06:05:22.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:22.717+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:22.718008+0000) 2019-10-13T06:05:24.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:24.211+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:24.212034+0000) 2019-10-13T06:05:25.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:25.777+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.778019+0000) 2019-10-13T06:05:25.787 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:25.782+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.782750+0000) 2019-10-13T06:05:25.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:25.782+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.783092+0000) 2019-10-13T06:05:25.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:25.782+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.783384+0000) 2019-10-13T06:05:25.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:25.783+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.783750+0000) 2019-10-13T06:05:25.787 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:25.783+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.783951+0000) 2019-10-13T06:05:25.788 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:25.779+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.780329+0000) 2019-10-13T06:05:25.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:25.783+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:25.784094+0000) 2019-10-13T06:05:25.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:25.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:28.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:28.257+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:28.257794+0000) 2019-10-13T06:05:28.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:28.596+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:28.597330+0000) 2019-10-13T06:05:28.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:28.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:28.597331+0000) 2019-10-13T06:05:30.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:30.787+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.787833+0000) 2019-10-13T06:05:30.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:30.787+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.787903+0000) 2019-10-13T06:05:30.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:30.787+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.788119+0000) 2019-10-13T06:05:30.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:30.788+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.788443+0000) 2019-10-13T06:05:30.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:30.783+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.783954+0000) 2019-10-13T06:05:30.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:30.788+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.789155+0000) 2019-10-13T06:05:30.793 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:30.789+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.789515+0000) 2019-10-13T06:05:30.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:30.785+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.786490+0000) 2019-10-13T06:05:30.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:30.790+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91768 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:30.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:30.825+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:30.826212+0000) 2019-10-13T06:05:31.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:31.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:31.851375+0000) 2019-10-13T06:05:32.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:32.717+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:32.717444+0000) 2019-10-13T06:05:32.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:32.717+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:32.718208+0000) 2019-10-13T06:05:34.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:34.211+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:34.212207+0000) 2019-10-13T06:05:35.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:35.789+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.789954+0000) 2019-10-13T06:05:35.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:35.792+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.793249+0000) 2019-10-13T06:05:35.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:35.792+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.793338+0000) 2019-10-13T06:05:35.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:35.793+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.793534+0000) 2019-10-13T06:05:35.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:35.793+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.793785+0000) 2019-10-13T06:05:35.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:35.793+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.793938+0000) 2019-10-13T06:05:35.804 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:35.794+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.794902+0000) 2019-10-13T06:05:35.804 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:35.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:35.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:35.791+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:35.792508+0000) 2019-10-13T06:05:38.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:38.257+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:38.257943+0000) 2019-10-13T06:05:38.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:38.597+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:38.597551+0000) 2019-10-13T06:05:38.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:38.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:38.597561+0000) 2019-10-13T06:05:40.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:40.793+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.794164+0000) 2019-10-13T06:05:40.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:40.797+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.798316+0000) 2019-10-13T06:05:40.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:40.798+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.798577+0000) 2019-10-13T06:05:40.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:40.798+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.798671+0000) 2019-10-13T06:05:40.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:40.798+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.798849+0000) 2019-10-13T06:05:40.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:40.798+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.799044+0000) 2019-10-13T06:05:40.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:40.799+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.799540+0000) 2019-10-13T06:05:40.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:40.797+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.798066+0000) 2019-10-13T06:05:40.806 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:40.802+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:40.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:40.827+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:40.828372+0000) 2019-10-13T06:05:41.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:41.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:41.851585+0000) 2019-10-13T06:05:42.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:42.716+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:42.717599+0000) 2019-10-13T06:05:42.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:42.717+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:42.718392+0000) 2019-10-13T06:05:44.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:44.211+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:44.212394+0000) 2019-10-13T06:05:45.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:45.799+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.800265+0000) 2019-10-13T06:05:45.809 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:45.803+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.804686+0000) 2019-10-13T06:05:45.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:45.803+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.804802+0000) 2019-10-13T06:05:45.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:45.803+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.805159+0000) 2019-10-13T06:05:45.809 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:45.803+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.805282+0000) 2019-10-13T06:05:45.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:45.804+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.805459+0000) 2019-10-13T06:05:45.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:45.804+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.805730+0000) 2019-10-13T06:05:45.810 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:45.802+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:45.802938+0000) 2019-10-13T06:05:45.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:45.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:48.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:48.256+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:48.258074+0000) 2019-10-13T06:05:48.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:48.596+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:48.597757+0000) 2019-10-13T06:05:48.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:48.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:48.597784+0000) 2019-10-13T06:05:50.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:50.808+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.809867+0000) 2019-10-13T06:05:50.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:50.808+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.810040+0000) 2019-10-13T06:05:50.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:50.808+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.810177+0000) 2019-10-13T06:05:50.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:50.806+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.806763+0000) 2019-10-13T06:05:50.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:50.809+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.810906+0000) 2019-10-13T06:05:50.815 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:50.810+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.811440+0000) 2019-10-13T06:05:50.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:50.810+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.811619+0000) 2019-10-13T06:05:50.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:50.808+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.809043+0000) 2019-10-13T06:05:50.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:50.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:50.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:50.830+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:50.830505+0000) 2019-10-13T06:05:51.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:51.850+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:51.851778+0000) 2019-10-13T06:05:52.558 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:05:52.558 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:52.562 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:05:52.563 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:52.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:52.716+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:52.717785+0000) 2019-10-13T06:05:52.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:52.717+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:52.718591+0000) 2019-10-13T06:05:54.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:54.211+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:54.212586+0000) 2019-10-13T06:05:55.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:55.814+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.815881+0000) 2019-10-13T06:05:55.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:05:55.814+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.816058+0000) 2019-10-13T06:05:55.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:55.814+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.816346+0000) 2019-10-13T06:05:55.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:05:55.810+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.812439+0000) 2019-10-13T06:05:55.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:05:55.815+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.816909+0000) 2019-10-13T06:05:55.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:05:55.815+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.817080+0000) 2019-10-13T06:05:55.821 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:55.815+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.816654+0000) 2019-10-13T06:05:55.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:05:55.812+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:55.814312+0000) 2019-10-13T06:05:55.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:05:55.818+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:05:58.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:05:58.257+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:58.258232+0000) 2019-10-13T06:05:58.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:05:58.596+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:58.597904+0000) 2019-10-13T06:05:58.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:05:58.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:05:58.597921+0000) 2019-10-13T06:06:00.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:00.819+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.821375+0000) 2019-10-13T06:06:00.825 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:00.820+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.821725+0000) 2019-10-13T06:06:00.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:00.816+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.817777+0000) 2019-10-13T06:06:00.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:00.820+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.822260+0000) 2019-10-13T06:06:00.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:00.821+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.822390+0000) 2019-10-13T06:06:00.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:00.821+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.822574+0000) 2019-10-13T06:06:00.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:00.821+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.822777+0000) 2019-10-13T06:06:00.827 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:00.817+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.819681+0000) 2019-10-13T06:06:00.828 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:00.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:00.840 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:00.831+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:00.832601+0000) 2019-10-13T06:06:01.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:01.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:01.851973+0000) 2019-10-13T06:06:02.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:02.717+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:02.717955+0000) 2019-10-13T06:06:02.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:02.717+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:02.718804+0000) 2019-10-13T06:06:04.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:04.211+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:04.212761+0000) 2019-10-13T06:06:05.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:05.826+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.827535+0000) 2019-10-13T06:06:05.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:05.826+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.827750+0000) 2019-10-13T06:06:05.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:05.821+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.823673+0000) 2019-10-13T06:06:05.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:05.826+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.828098+0000) 2019-10-13T06:06:05.833 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:05.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.828189+0000) 2019-10-13T06:06:05.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:05.827+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.828443+0000) 2019-10-13T06:06:05.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:05.827+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.828592+0000) 2019-10-13T06:06:05.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:05.825+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:05.826873+0000) 2019-10-13T06:06:05.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:05.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 91986 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:08.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:08.257+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:08.258380+0000) 2019-10-13T06:06:08.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:08.597+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:08.598079+0000) 2019-10-13T06:06:08.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:08.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:08.598078+0000) 2019-10-13T06:06:10.836 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:10.832+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.832760+0000) 2019-10-13T06:06:10.836 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:10.832+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.832895+0000) 2019-10-13T06:06:10.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:10.832+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.833095+0000) 2019-10-13T06:06:10.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:10.832+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.833347+0000) 2019-10-13T06:06:10.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:10.828+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.829282+0000) 2019-10-13T06:06:10.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:10.833+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.833712+0000) 2019-10-13T06:06:10.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:10.833+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.833879+0000) 2019-10-13T06:06:10.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:10.830+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.831201+0000) 2019-10-13T06:06:10.839 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:10.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:10.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:10.834+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:10.834802+0000) 2019-10-13T06:06:11.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:11.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:11.852168+0000) 2019-10-13T06:06:12.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:12.718+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:12.718990+0000) 2019-10-13T06:06:12.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:12.717+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:12.718181+0000) 2019-10-13T06:06:14.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:14.212+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:14.212945+0000) 2019-10-13T06:06:15.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:15.837+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.838166+0000) 2019-10-13T06:06:15.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:15.837+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.838377+0000) 2019-10-13T06:06:15.842 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:15.833+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.834690+0000) 2019-10-13T06:06:15.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:15.838+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.838768+0000) 2019-10-13T06:06:15.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:15.838+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.839332+0000) 2019-10-13T06:06:15.843 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:15.839+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.839553+0000) 2019-10-13T06:06:15.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:15.839+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.839768+0000) 2019-10-13T06:06:15.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:15.835+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:15.836371+0000) 2019-10-13T06:06:15.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:15.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:18.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:18.258+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:18.258546+0000) 2019-10-13T06:06:18.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:18.598+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:18.598272+0000) 2019-10-13T06:06:18.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:18.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:18.598272+0000) 2019-10-13T06:06:20.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:20.836+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.837265+0000) 2019-10-13T06:06:20.846 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:20.843+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.843420+0000) 2019-10-13T06:06:20.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:20.843+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.843757+0000) 2019-10-13T06:06:20.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:20.843+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.843860+0000) 2019-10-13T06:06:20.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:20.839+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.839968+0000) 2019-10-13T06:06:20.848 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:20.844+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.844478+0000) 2019-10-13T06:06:20.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:20.844+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.844712+0000) 2019-10-13T06:06:20.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:20.844+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.844900+0000) 2019-10-13T06:06:20.850 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:20.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:20.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:20.842+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:20.843185+0000) 2019-10-13T06:06:21.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:21.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:21.852330+0000) 2019-10-13T06:06:22.672 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:06:22.672 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:22.676 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:06:22.677 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:22.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:22.717+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:22.718361+0000) 2019-10-13T06:06:22.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:22.718+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:22.719175+0000) 2019-10-13T06:06:24.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:24.213+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:24.213115+0000) 2019-10-13T06:06:25.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:25.848+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.848933+0000) 2019-10-13T06:06:25.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:25.844+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.844867+0000) 2019-10-13T06:06:25.854 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:25.849+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.849434+0000) 2019-10-13T06:06:25.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:25.849+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.849625+0000) 2019-10-13T06:06:25.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:25.849+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.849994+0000) 2019-10-13T06:06:25.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:25.849+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.850095+0000) 2019-10-13T06:06:25.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:25.849+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.849770+0000) 2019-10-13T06:06:25.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:25.847+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:25.847921+0000) 2019-10-13T06:06:25.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:25.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:28.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:28.257+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:28.258726+0000) 2019-10-13T06:06:28.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:28.597+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:28.598483+0000) 2019-10-13T06:06:28.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:28.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:28.598476+0000) 2019-10-13T06:06:30.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:30.839+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.839773+0000) 2019-10-13T06:06:30.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:30.849+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.849969+0000) 2019-10-13T06:06:30.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:30.853+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.854426+0000) 2019-10-13T06:06:30.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:30.853+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.854595+0000) 2019-10-13T06:06:30.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:30.853+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.854975+0000) 2019-10-13T06:06:30.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:30.853+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.854843+0000) 2019-10-13T06:06:30.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:30.851+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.851824+0000) 2019-10-13T06:06:30.859 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:30.854+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.855448+0000) 2019-10-13T06:06:30.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:30.853+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:30.855195+0000) 2019-10-13T06:06:30.860 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:30.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:31.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:31.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:31.852499+0000) 2019-10-13T06:06:32.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:32.717+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:32.718519+0000) 2019-10-13T06:06:32.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:32.719+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:32.719361+0000) 2019-10-13T06:06:34.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:34.212+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:34.213292+0000) 2019-10-13T06:06:35.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:35.854+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.855406+0000) 2019-10-13T06:06:35.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:35.858+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.859812+0000) 2019-10-13T06:06:35.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:35.858+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.860038+0000) 2019-10-13T06:06:35.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:35.858+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.860114+0000) 2019-10-13T06:06:35.864 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:35.858+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.860373+0000) 2019-10-13T06:06:35.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:35.859+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.860701+0000) 2019-10-13T06:06:35.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:35.859+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.860903+0000) 2019-10-13T06:06:35.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:35.858+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:35.859162+0000) 2019-10-13T06:06:35.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:35.862+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:38.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:38.258+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:38.258906+0000) 2019-10-13T06:06:38.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:38.597+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:38.598715+0000) 2019-10-13T06:06:38.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:38.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:38.598715+0000) 2019-10-13T06:06:40.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:40.842+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.842233+0000) 2019-10-13T06:06:40.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:40.863+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.865048+0000) 2019-10-13T06:06:40.868 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:40.863+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.865317+0000) 2019-10-13T06:06:40.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:40.860+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.861325+0000) 2019-10-13T06:06:40.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:40.864+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.865703+0000) 2019-10-13T06:06:40.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:40.864+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.865817+0000) 2019-10-13T06:06:40.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:40.864+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.866285+0000) 2019-10-13T06:06:40.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:40.865+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.866403+0000) 2019-10-13T06:06:40.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:40.862+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:40.863629+0000) 2019-10-13T06:06:40.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:40.867+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92204 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:41.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:41.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:41.852663+0000) 2019-10-13T06:06:42.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:42.717+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:42.718666+0000) 2019-10-13T06:06:42.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:42.719+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:42.719564+0000) 2019-10-13T06:06:44.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:44.212+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:44.213452+0000) 2019-10-13T06:06:45.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:45.869+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.870967+0000) 2019-10-13T06:06:45.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:45.869+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.871104+0000) 2019-10-13T06:06:45.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:45.866+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.867143+0000) 2019-10-13T06:06:45.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:45.870+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.871565+0000) 2019-10-13T06:06:45.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:45.870+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.871718+0000) 2019-10-13T06:06:45.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:45.870+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.871859+0000) 2019-10-13T06:06:45.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:45.870+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.872037+0000) 2019-10-13T06:06:45.876 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:45.868+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:45.869586+0000) 2019-10-13T06:06:45.877 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:45.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:48.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:48.258+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:48.259095+0000) 2019-10-13T06:06:48.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:48.597+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:48.598949+0000) 2019-10-13T06:06:48.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:48.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:48.598947+0000) 2019-10-13T06:06:50.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:50.843+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.844547+0000) 2019-10-13T06:06:50.879 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:50.874+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.875727+0000) 2019-10-13T06:06:50.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:50.874+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.876001+0000) 2019-10-13T06:06:50.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:50.870+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.872402+0000) 2019-10-13T06:06:50.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:50.875+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.877123+0000) 2019-10-13T06:06:50.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:50.875+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.877295+0000) 2019-10-13T06:06:50.881 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:50.876+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.877477+0000) 2019-10-13T06:06:50.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:50.876+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.878204+0000) 2019-10-13T06:06:50.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:50.872+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:50.874629+0000) 2019-10-13T06:06:50.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:50.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:51.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:51.852+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:51.852813+0000) 2019-10-13T06:06:52.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:52.718+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:52.718840+0000) 2019-10-13T06:06:52.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:52.718+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:52.719766+0000) 2019-10-13T06:06:52.769 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:06:52.770 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:52.774 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:06:52.774 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:54.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:54.213+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:54.213662+0000) 2019-10-13T06:06:55.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:06:55.882+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.882471+0000) 2019-10-13T06:06:55.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:55.882+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.882666+0000) 2019-10-13T06:06:55.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:06:55.877+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.878814+0000) 2019-10-13T06:06:55.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:55.882+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.882857+0000) 2019-10-13T06:06:55.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:06:55.883+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.883465+0000) 2019-10-13T06:06:55.887 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:06:55.883+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.883743+0000) 2019-10-13T06:06:55.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:55.883+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.883951+0000) 2019-10-13T06:06:55.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:06:55.879+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:55.881124+0000) 2019-10-13T06:06:55.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:06:55.886+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92295 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:06:58.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:06:58.258+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:58.259288+0000) 2019-10-13T06:06:58.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:06:58.598+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:58.599142+0000) 2019-10-13T06:06:58.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:06:58.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:06:58.599141+0000) 2019-10-13T06:07:00.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:00.846+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.846924+0000) 2019-10-13T06:07:00.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:00.887+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.888212+0000) 2019-10-13T06:07:00.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:00.883+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.884605+0000) 2019-10-13T06:07:00.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:00.888+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.888647+0000) 2019-10-13T06:07:00.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:00.888+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.888745+0000) 2019-10-13T06:07:00.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:00.888+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.889289+0000) 2019-10-13T06:07:00.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:00.889+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.889465+0000) 2019-10-13T06:07:00.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:00.889+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.889702+0000) 2019-10-13T06:07:00.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:00.885+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:00.886188+0000) 2019-10-13T06:07:00.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:00.891+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:01.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:01.852+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:01.852977+0000) 2019-10-13T06:07:02.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:02.718+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:02.719027+0000) 2019-10-13T06:07:02.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:02.719+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:02.719966+0000) 2019-10-13T06:07:04.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:04.213+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:04.213853+0000) 2019-10-13T06:07:05.896 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:05.892+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.893187+0000) 2019-10-13T06:07:05.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:05.888+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.889379+0000) 2019-10-13T06:07:05.897 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:05.893+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.893947+0000) 2019-10-13T06:07:05.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:05.893+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.894306+0000) 2019-10-13T06:07:05.898 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:05.894+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.894580+0000) 2019-10-13T06:07:05.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:05.894+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.894763+0000) 2019-10-13T06:07:05.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:05.894+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.894947+0000) 2019-10-13T06:07:05.899 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:05.891+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:05.892050+0000) 2019-10-13T06:07:05.900 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:05.896+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92358 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:08.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:08.259+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:08.259474+0000) 2019-10-13T06:07:08.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:08.598+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:08.599361+0000) 2019-10-13T06:07:08.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:08.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:08.599358+0000) 2019-10-13T06:07:10.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:10.848+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.849257+0000) 2019-10-13T06:07:10.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:10.898+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.898436+0000) 2019-10-13T06:07:10.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:10.893+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.894302+0000) 2019-10-13T06:07:10.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:10.898+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.898912+0000) 2019-10-13T06:07:10.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:10.898+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.899149+0000) 2019-10-13T06:07:10.903 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:10.898+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.899178+0000) 2019-10-13T06:07:10.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:10.898+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.899198+0000) 2019-10-13T06:07:10.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:10.899+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.899775+0000) 2019-10-13T06:07:10.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:10.895+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:10.896396+0000) 2019-10-13T06:07:10.904 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:10.901+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92389 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:11.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:11.852+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:11.853115+0000) 2019-10-13T06:07:12.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:12.719+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:12.719221+0000) 2019-10-13T06:07:12.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:12.719+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:12.720139+0000) 2019-10-13T06:07:14.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:14.213+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:14.214046+0000) 2019-10-13T06:07:15.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:15.899+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.899857+0000) 2019-10-13T06:07:15.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:15.902+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.903900+0000) 2019-10-13T06:07:15.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:15.902+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.904240+0000) 2019-10-13T06:07:15.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:15.903+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.904426+0000) 2019-10-13T06:07:15.909 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:15.903+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.904612+0000) 2019-10-13T06:07:15.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:15.903+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.904821+0000) 2019-10-13T06:07:15.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:15.903+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.905059+0000) 2019-10-13T06:07:15.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:15.900+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:15.901235+0000) 2019-10-13T06:07:15.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:15.905+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:18.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:18.258+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:18.259668+0000) 2019-10-13T06:07:18.603 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:18.598+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:18.599563+0000) 2019-10-13T06:07:18.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:18.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:18.599565+0000) 2019-10-13T06:07:20.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:20.851+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.851621+0000) 2019-10-13T06:07:20.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:20.907+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.908752+0000) 2019-10-13T06:07:20.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:20.904+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.904921+0000) 2019-10-13T06:07:20.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:20.907+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.909303+0000) 2019-10-13T06:07:20.913 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:20.908+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.909718+0000) 2019-10-13T06:07:20.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:20.908+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.909851+0000) 2019-10-13T06:07:20.914 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:20.908+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.910004+0000) 2019-10-13T06:07:20.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:20.908+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.910207+0000) 2019-10-13T06:07:20.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:20.906+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:20.906821+0000) 2019-10-13T06:07:20.915 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:20.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:21.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:21.851+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:21.853303+0000) 2019-10-13T06:07:22.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:22.718+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:22.719429+0000) 2019-10-13T06:07:22.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:22.720+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:22.720377+0000) 2019-10-13T06:07:22.925 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:07:22.925 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:22.929 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:07:22.929 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:24.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:24.213+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:24.214233+0000) 2019-10-13T06:07:25.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:25.909+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.910149+0000) 2019-10-13T06:07:25.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:25.913+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.914631+0000) 2019-10-13T06:07:25.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:25.913+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.914801+0000) 2019-10-13T06:07:25.919 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:25.913+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.915068+0000) 2019-10-13T06:07:25.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:25.913+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.915059+0000) 2019-10-13T06:07:25.919 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:25.914+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.915660+0000) 2019-10-13T06:07:25.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:25.914+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.915848+0000) 2019-10-13T06:07:25.920 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:25.911+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:25.912468+0000) 2019-10-13T06:07:25.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:25.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92482 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:28.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:28.258+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:28.259825+0000) 2019-10-13T06:07:28.603 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:28.599+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:28.599773+0000) 2019-10-13T06:07:28.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:28.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:28.599763+0000) 2019-10-13T06:07:30.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:30.853+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.853945+0000) 2019-10-13T06:07:30.923 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:30.918+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.919769+0000) 2019-10-13T06:07:30.923 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:30.918+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.919803+0000) 2019-10-13T06:07:30.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:30.918+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.920165+0000) 2019-10-13T06:07:30.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:30.915+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.915854+0000) 2019-10-13T06:07:30.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:30.919+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.920439+0000) 2019-10-13T06:07:30.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:30.919+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.920623+0000) 2019-10-13T06:07:30.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:30.919+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.920918+0000) 2019-10-13T06:07:30.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:30.917+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:30.918146+0000) 2019-10-13T06:07:30.926 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:30.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:31.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:31.852+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:31.853489+0000) 2019-10-13T06:07:32.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:32.718+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:32.719562+0000) 2019-10-13T06:07:32.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:32.720+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:32.720548+0000) 2019-10-13T06:07:34.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:34.213+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:34.214445+0000) 2019-10-13T06:07:35.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:35.923+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.925123+0000) 2019-10-13T06:07:35.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:35.924+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.925487+0000) 2019-10-13T06:07:35.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:35.924+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.925768+0000) 2019-10-13T06:07:35.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:35.924+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.926198+0000) 2019-10-13T06:07:35.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:35.920+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.921710+0000) 2019-10-13T06:07:35.930 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:35.925+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.926393+0000) 2019-10-13T06:07:35.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:35.925+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.926554+0000) 2019-10-13T06:07:35.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:35.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:35.932 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:35.923+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:35.924592+0000) 2019-10-13T06:07:38.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:38.259+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:38.259988+0000) 2019-10-13T06:07:38.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:38.599+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:38.599979+0000) 2019-10-13T06:07:38.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:38.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:38.599980+0000) 2019-10-13T06:07:40.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:40.855+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.856292+0000) 2019-10-13T06:07:40.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:40.929+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.930478+0000) 2019-10-13T06:07:40.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:40.929+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.930631+0000) 2019-10-13T06:07:40.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:40.925+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.926795+0000) 2019-10-13T06:07:40.935 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:40.930+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.931441+0000) 2019-10-13T06:07:40.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:40.930+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.931647+0000) 2019-10-13T06:07:40.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:40.930+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.931835+0000) 2019-10-13T06:07:40.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:40.930+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.932129+0000) 2019-10-13T06:07:40.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:40.928+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:40.929821+0000) 2019-10-13T06:07:40.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:40.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92576 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:41.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:41.853+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:41.853628+0000) 2019-10-13T06:07:42.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:42.719+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:42.719733+0000) 2019-10-13T06:07:42.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:42.719+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:42.720719+0000) 2019-10-13T06:07:44.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:44.214+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:44.214632+0000) 2019-10-13T06:07:45.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:45.930+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.932011+0000) 2019-10-13T06:07:45.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:45.935+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.936371+0000) 2019-10-13T06:07:45.940 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:45.936+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.936552+0000) 2019-10-13T06:07:45.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:45.936+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.936735+0000) 2019-10-13T06:07:45.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:45.936+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.936944+0000) 2019-10-13T06:07:45.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:45.936+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.937305+0000) 2019-10-13T06:07:45.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:45.937+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.937777+0000) 2019-10-13T06:07:45.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:45.933+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:45.935528+0000) 2019-10-13T06:07:45.943 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:45.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:48.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:48.259+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:48.260167+0000) 2019-10-13T06:07:48.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:48.599+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:48.600188+0000) 2019-10-13T06:07:48.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:48.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:48.600189+0000) 2019-10-13T06:07:50.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:50.857+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.858633+0000) 2019-10-13T06:07:50.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:50.941+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.941606+0000) 2019-10-13T06:07:50.945 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:50.941+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.941986+0000) 2019-10-13T06:07:50.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:50.936+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.937886+0000) 2019-10-13T06:07:50.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:50.941+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.942239+0000) 2019-10-13T06:07:50.946 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:50.942+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.942612+0000) 2019-10-13T06:07:50.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:50.942+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.942794+0000) 2019-10-13T06:07:50.947 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:50.942+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.942940+0000) 2019-10-13T06:07:50.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:50.938+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:50.940349+0000) 2019-10-13T06:07:50.948 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:50.944+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:51.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:51.853+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:51.853831+0000) 2019-10-13T06:07:52.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:52.719+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:52.719959+0000) 2019-10-13T06:07:52.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:52.720+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:52.720927+0000) 2019-10-13T06:07:53.121 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:07:53.122 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:53.126 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:07:53.126 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:54.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:54.214+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:54.214824+0000) 2019-10-13T06:07:55.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:07:55.941+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.942366+0000) 2019-10-13T06:07:55.951 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:55.946+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.946760+0000) 2019-10-13T06:07:55.951 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:55.946+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.946848+0000) 2019-10-13T06:07:55.951 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:07:55.947+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.947395+0000) 2019-10-13T06:07:55.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:07:55.947+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.947607+0000) 2019-10-13T06:07:55.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:55.947+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.947881+0000) 2019-10-13T06:07:55.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:07:55.947+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.948215+0000) 2019-10-13T06:07:55.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:07:55.944+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:55.944974+0000) 2019-10-13T06:07:55.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:07:55.950+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:07:58.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:07:58.260+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:58.260377+0000) 2019-10-13T06:07:58.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:07:58.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:58.600350+0000) 2019-10-13T06:07:58.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:07:58.600+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:07:58.600377+0000) 2019-10-13T06:08:00.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:00.860+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.860918+0000) 2019-10-13T06:08:00.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:00.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.952863+0000) 2019-10-13T06:08:00.956 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:00.951+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.952995+0000) 2019-10-13T06:08:00.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:00.951+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.953185+0000) 2019-10-13T06:08:00.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:00.948+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.949197+0000) 2019-10-13T06:08:00.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:00.952+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.953920+0000) 2019-10-13T06:08:00.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:00.952+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.954109+0000) 2019-10-13T06:08:00.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:00.952+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.954341+0000) 2019-10-13T06:08:00.958 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:00.950+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:00.951342+0000) 2019-10-13T06:08:00.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:00.954+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:01.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:01.852+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:01.854019+0000) 2019-10-13T06:08:02.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:02.719+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:02.720082+0000) 2019-10-13T06:08:02.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:02.720+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:02.721115+0000) 2019-10-13T06:08:04.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:04.214+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:04.215016+0000) 2019-10-13T06:08:05.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:05.956+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.958273+0000) 2019-10-13T06:08:05.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:05.957+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.958621+0000) 2019-10-13T06:08:05.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:05.957+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.958832+0000) 2019-10-13T06:08:05.963 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:05.957+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.959066+0000) 2019-10-13T06:08:05.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:05.958+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.959485+0000) 2019-10-13T06:08:05.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:05.958+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.959701+0000) 2019-10-13T06:08:05.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:05.954+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.955118+0000) 2019-10-13T06:08:05.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:05.955+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:05.955993+0000) 2019-10-13T06:08:05.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:05.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92730 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:08.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:08.259+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:08.260560+0000) 2019-10-13T06:08:08.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:08.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:08.600542+0000) 2019-10-13T06:08:08.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:08.599+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:08.600611+0000) 2019-10-13T06:08:10.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:10.862+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.863216+0000) 2019-10-13T06:08:10.968 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:10.962+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.964113+0000) 2019-10-13T06:08:10.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:10.962+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.964261+0000) 2019-10-13T06:08:10.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:10.963+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.964769+0000) 2019-10-13T06:08:10.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:10.963+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.964950+0000) 2019-10-13T06:08:10.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:10.960+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.960920+0000) 2019-10-13T06:08:10.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:10.963+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.965329+0000) 2019-10-13T06:08:10.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:10.964+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.965693+0000) 2019-10-13T06:08:10.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:10.961+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:10.962389+0000) 2019-10-13T06:08:10.971 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:10.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:11.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:11.852+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:11.854209+0000) 2019-10-13T06:08:12.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:12.719+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:12.720248+0000) 2019-10-13T06:08:12.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:12.720+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:12.721282+0000) 2019-10-13T06:08:14.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:14.214+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:14.215210+0000) 2019-10-13T06:08:15.973 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:15.968+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.969782+0000) 2019-10-13T06:08:15.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:15.968+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.970072+0000) 2019-10-13T06:08:15.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:15.965+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.965728+0000) 2019-10-13T06:08:15.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:15.968+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.970231+0000) 2019-10-13T06:08:15.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:15.968+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.970353+0000) 2019-10-13T06:08:15.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:15.969+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.970724+0000) 2019-10-13T06:08:15.975 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:15.970+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.971608+0000) 2019-10-13T06:08:15.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:15.968+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:15.969022+0000) 2019-10-13T06:08:15.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:15.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92791 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:18.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:18.260+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:18.260773+0000) 2019-10-13T06:08:18.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:18.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:18.600728+0000) 2019-10-13T06:08:18.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:18.600+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:18.600806+0000) 2019-10-13T06:08:20.874 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:20.865+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.865663+0000) 2019-10-13T06:08:20.978 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:20.973+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.975161+0000) 2019-10-13T06:08:20.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:20.970+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.971676+0000) 2019-10-13T06:08:20.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:20.974+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.975521+0000) 2019-10-13T06:08:20.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:20.974+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.975728+0000) 2019-10-13T06:08:20.980 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:20.974+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.976063+0000) 2019-10-13T06:08:20.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:20.974+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.976214+0000) 2019-10-13T06:08:20.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:20.972+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.973181+0000) 2019-10-13T06:08:20.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:20.975+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:20.976667+0000) 2019-10-13T06:08:20.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:20.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:21.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:21.853+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:21.854381+0000) 2019-10-13T06:08:22.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:22.719+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:22.720422+0000) 2019-10-13T06:08:22.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:22.721+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:22.721468+0000) 2019-10-13T06:08:23.268 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:08:23.269 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:23.273 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:08:23.273 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:24.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:24.214+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:24.215431+0000) 2019-10-13T06:08:25.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:25.976+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.976773+0000) 2019-10-13T06:08:25.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:25.979+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.980970+0000) 2019-10-13T06:08:25.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:25.979+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.981076+0000) 2019-10-13T06:08:25.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:25.979+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.981154+0000) 2019-10-13T06:08:25.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:25.979+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.981285+0000) 2019-10-13T06:08:25.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:25.980+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.981477+0000) 2019-10-13T06:08:25.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:25.980+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.981753+0000) 2019-10-13T06:08:25.989 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:25.979+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:25.979839+0000) 2019-10-13T06:08:25.989 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:25.982+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:28.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:28.260+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:28.260969+0000) 2019-10-13T06:08:28.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:28.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:28.600886+0000) 2019-10-13T06:08:28.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:28.600+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:28.600966+0000) 2019-10-13T06:08:30.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:30.867+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.867978+0000) 2019-10-13T06:08:30.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:30.985+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.985711+0000) 2019-10-13T06:08:30.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:30.985+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.986197+0000) 2019-10-13T06:08:30.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:30.986+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.986516+0000) 2019-10-13T06:08:30.992 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:30.986+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.986786+0000) 2019-10-13T06:08:30.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:30.986+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.986451+0000) 2019-10-13T06:08:30.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:30.986+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.986831+0000) 2019-10-13T06:08:30.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:30.980+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.981175+0000) 2019-10-13T06:08:30.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:30.983+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:30.983799+0000) 2019-10-13T06:08:30.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:30.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:31.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:31.854+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:31.854557+0000) 2019-10-13T06:08:32.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:32.720+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:32.720603+0000) 2019-10-13T06:08:32.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:32.720+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:32.721692+0000) 2019-10-13T06:08:34.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:34.215+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:34.215621+0000) 2019-10-13T06:08:35.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:35.984+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.986704+0000) 2019-10-13T06:08:35.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:35.990+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.991126+0000) 2019-10-13T06:08:35.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:35.990+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.991222+0000) 2019-10-13T06:08:35.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:35.991+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.991592+0000) 2019-10-13T06:08:35.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:35.991+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.991790+0000) 2019-10-13T06:08:35.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:35.991+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.992075+0000) 2019-10-13T06:08:35.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:35.991+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.991901+0000) 2019-10-13T06:08:35.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:35.988+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:35.990201+0000) 2019-10-13T06:08:35.998 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:35.994+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:38.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:38.260+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:38.261172+0000) 2019-10-13T06:08:38.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:38.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:38.601056+0000) 2019-10-13T06:08:38.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:38.600+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:38.601131+0000) 2019-10-13T06:08:40.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:40.869+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.870424+0000) 2019-10-13T06:08:41.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:40.990+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.991927+0000) 2019-10-13T06:08:41.001 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:40.996+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.996689+0000) 2019-10-13T06:08:41.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:40.996+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.997016+0000) 2019-10-13T06:08:41.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:40.996+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.997001+0000) 2019-10-13T06:08:41.001 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:40.997+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.997484+0000) 2019-10-13T06:08:41.002 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:40.997+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.997535+0000) 2019-10-13T06:08:41.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:40.997+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.997963+0000) 2019-10-13T06:08:41.003 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:40.993+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:40.994958+0000) 2019-10-13T06:08:41.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:41.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:41.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:41.854+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:41.854762+0000) 2019-10-13T06:08:42.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:42.720+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:42.720788+0000) 2019-10-13T06:08:42.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:42.721+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:42.721880+0000) 2019-10-13T06:08:44.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:44.215+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:44.215809+0000) 2019-10-13T06:08:46.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:45.997+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:45.998121+0000) 2019-10-13T06:08:46.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:46.000+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:46.002235+0000) 2019-10-13T06:08:46.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:46.001+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:46.002547+0000) 2019-10-13T06:08:46.007 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:46.001+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:46.002961+0000) 2019-10-13T06:08:46.011 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:46.001+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:46.003157+0000) 2019-10-13T06:08:46.012 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:46.001+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:46.003298+0000) 2019-10-13T06:08:46.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:46.002+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:46.003513+0000) 2019-10-13T06:08:46.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:46.000+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:46.001586+0000) 2019-10-13T06:08:46.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:46.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 92978 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:48.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:48.260+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:48.261337+0000) 2019-10-13T06:08:48.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:48.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:48.601246+0000) 2019-10-13T06:08:48.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:48.600+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:48.601313+0000) 2019-10-13T06:08:50.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:50.872+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:50.872909+0000) 2019-10-13T06:08:51.011 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:51.006+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.007701+0000) 2019-10-13T06:08:51.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:51.006+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.007873+0000) 2019-10-13T06:08:51.011 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:51.006+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.008075+0000) 2019-10-13T06:08:51.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:51.006+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.008078+0000) 2019-10-13T06:08:51.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:51.003+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.004037+0000) 2019-10-13T06:08:51.016 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:51.007+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.008453+0000) 2019-10-13T06:08:51.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:51.007+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.008557+0000) 2019-10-13T06:08:51.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:51.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:51.017 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:51.005+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.006188+0000) 2019-10-13T06:08:51.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:51.853+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:51.854926+0000) 2019-10-13T06:08:52.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:52.719+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:52.721001+0000) 2019-10-13T06:08:52.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:52.721+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:52.722007+0000) 2019-10-13T06:08:53.363 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:08:53.363 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:53.369 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:08:53.369 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:54.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:54.215+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:54.215995+0000) 2019-10-13T06:08:56.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:08:56.010+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.012104+0000) 2019-10-13T06:08:56.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:56.010+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.012344+0000) 2019-10-13T06:08:56.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:08:56.011+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.012486+0000) 2019-10-13T06:08:56.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:08:56.008+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.008884+0000) 2019-10-13T06:08:56.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:08:56.011+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.013045+0000) 2019-10-13T06:08:56.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:56.011+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.012905+0000) 2019-10-13T06:08:56.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:56.011+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.013338+0000) 2019-10-13T06:08:56.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:08:56.009+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:56.009865+0000) 2019-10-13T06:08:56.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:08:56.014+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93039 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:08:58.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:08:58.260+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:58.261562+0000) 2019-10-13T06:08:58.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:08:58.600+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:58.601469+0000) 2019-10-13T06:08:58.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:08:58.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:08:58.601428+0000) 2019-10-13T06:09:00.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:00.874+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:00.875327+0000) 2019-10-13T06:09:01.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:01.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.013442+0000) 2019-10-13T06:09:01.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:01.016+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.017912+0000) 2019-10-13T06:09:01.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:01.016+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.018124+0000) 2019-10-13T06:09:01.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:01.016+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.018345+0000) 2019-10-13T06:09:01.022 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:01.017+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.018503+0000) 2019-10-13T06:09:01.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:01.017+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.018681+0000) 2019-10-13T06:09:01.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:01.017+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.018823+0000) 2019-10-13T06:09:01.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:01.015+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.016017+0000) 2019-10-13T06:09:01.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:01.020+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:01.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:01.854+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:01.855084+0000) 2019-10-13T06:09:02.724 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:02.720+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:02.721198+0000) 2019-10-13T06:09:02.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:02.721+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:02.722202+0000) 2019-10-13T06:09:04.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:04.214+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:04.216183+0000) 2019-10-13T06:09:06.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:06.018+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.019220+0000) 2019-10-13T06:09:06.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:06.022+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.023512+0000) 2019-10-13T06:09:06.028 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:06.022+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.023774+0000) 2019-10-13T06:09:06.028 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:06.022+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.023949+0000) 2019-10-13T06:09:06.028 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:06.022+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.024171+0000) 2019-10-13T06:09:06.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:06.022+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.024225+0000) 2019-10-13T06:09:06.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:06.023+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.024569+0000) 2019-10-13T06:09:06.030 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:06.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93102 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:06.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:06.022+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:06.022972+0000) 2019-10-13T06:09:08.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:08.261+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:08.261759+0000) 2019-10-13T06:09:08.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:08.601+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:08.601655+0000) 2019-10-13T06:09:08.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:08.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:08.601594+0000) 2019-10-13T06:09:10.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:10.877+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:10.877747+0000) 2019-10-13T06:09:11.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:11.025+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.025888+0000) 2019-10-13T06:09:11.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:11.026+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.026873+0000) 2019-10-13T06:09:11.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:11.028+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.029515+0000) 2019-10-13T06:09:11.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:11.028+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.029666+0000) 2019-10-13T06:09:11.039 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:11.028+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.029884+0000) 2019-10-13T06:09:11.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:11.028+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.030181+0000) 2019-10-13T06:09:11.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:11.029+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.030464+0000) 2019-10-13T06:09:11.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:11.029+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.030556+0000) 2019-10-13T06:09:11.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:11.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93134 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:11.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:11.854+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:11.855253+0000) 2019-10-13T06:09:12.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:12.720+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:12.721351+0000) 2019-10-13T06:09:12.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:12.722+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:12.722357+0000) 2019-10-13T06:09:14.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:14.215+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:14.216299+0000) 2019-10-13T06:09:16.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:16.030+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.030768+0000) 2019-10-13T06:09:16.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:16.034+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.035202+0000) 2019-10-13T06:09:16.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:16.034+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.035302+0000) 2019-10-13T06:09:16.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:16.035+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.035561+0000) 2019-10-13T06:09:16.040 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:16.035+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.035921+0000) 2019-10-13T06:09:16.040 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:16.035+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.036065+0000) 2019-10-13T06:09:16.040 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:16.035+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.036191+0000) 2019-10-13T06:09:16.041 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:16.033+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:16.034317+0000) 2019-10-13T06:09:16.042 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:16.038+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:18.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:18.261+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:18.261934+0000) 2019-10-13T06:09:18.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:18.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:18.601741+0000) 2019-10-13T06:09:18.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:18.601+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:18.601803+0000) 2019-10-13T06:09:20.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:20.880+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:20.880212+0000) 2019-10-13T06:09:21.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:21.036+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.037177+0000) 2019-10-13T06:09:21.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:21.040+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.041235+0000) 2019-10-13T06:09:21.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:21.041+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.041516+0000) 2019-10-13T06:09:21.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:21.041+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.041708+0000) 2019-10-13T06:09:21.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:21.041+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.041786+0000) 2019-10-13T06:09:21.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:21.041+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.042018+0000) 2019-10-13T06:09:21.047 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:21.042+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.042429+0000) 2019-10-13T06:09:21.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:21.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93197 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:21.048 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:21.039+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.040192+0000) 2019-10-13T06:09:21.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:21.855+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:21.855436+0000) 2019-10-13T06:09:22.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:22.721+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:22.721553+0000) 2019-10-13T06:09:22.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:22.722+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:22.722530+0000) 2019-10-13T06:09:23.509 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:09:23.510 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:23.514 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:09:23.514 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:24.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:24.216+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:24.216472+0000) 2019-10-13T06:09:26.050 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:26.045+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.046292+0000) 2019-10-13T06:09:26.050 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:26.046+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.046478+0000) 2019-10-13T06:09:26.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:26.046+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.046813+0000) 2019-10-13T06:09:26.050 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:26.046+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.046728+0000) 2019-10-13T06:09:26.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:26.041+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.042775+0000) 2019-10-13T06:09:26.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:26.046+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.047382+0000) 2019-10-13T06:09:26.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:26.046+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.047151+0000) 2019-10-13T06:09:26.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:26.043+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:26.045373+0000) 2019-10-13T06:09:26.053 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:26.049+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:28.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:28.261+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:28.262145+0000) 2019-10-13T06:09:28.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:28.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:28.601887+0000) 2019-10-13T06:09:28.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:28.601+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:28.601950+0000) 2019-10-13T06:09:30.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:30.881+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:30.882684+0000) 2019-10-13T06:09:31.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:31.046+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.048434+0000) 2019-10-13T06:09:31.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:31.050+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.052332+0000) 2019-10-13T06:09:31.058 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:31.051+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.052555+0000) 2019-10-13T06:09:31.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:31.051+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.052863+0000) 2019-10-13T06:09:31.059 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:31.051+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.053063+0000) 2019-10-13T06:09:31.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:31.051+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.053244+0000) 2019-10-13T06:09:31.059 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:31.052+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.053665+0000) 2019-10-13T06:09:31.060 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:31.049+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.051205+0000) 2019-10-13T06:09:31.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:31.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:31.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:31.854+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:31.855638+0000) 2019-10-13T06:09:32.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:32.720+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:32.721739+0000) 2019-10-13T06:09:32.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:32.721+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:32.722686+0000) 2019-10-13T06:09:34.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:34.215+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:34.216659+0000) 2019-10-13T06:09:36.061 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:36.056+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.057744+0000) 2019-10-13T06:09:36.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:36.052+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.054054+0000) 2019-10-13T06:09:36.062 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:36.056+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.058114+0000) 2019-10-13T06:09:36.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:36.056+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.058367+0000) 2019-10-13T06:09:36.062 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:36.057+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.058494+0000) 2019-10-13T06:09:36.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:36.057+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.058666+0000) 2019-10-13T06:09:36.063 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:36.057+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.058990+0000) 2019-10-13T06:09:36.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:36.053+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:36.055724+0000) 2019-10-13T06:09:36.064 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:36.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93287 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:38.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:38.260+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:38.262344+0000) 2019-10-13T06:09:38.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:38.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:38.602062+0000) 2019-10-13T06:09:38.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:38.601+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:38.602109+0000) 2019-10-13T06:09:40.892 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:40.884+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:40.885064+0000) 2019-10-13T06:09:41.066 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:41.061+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.062966+0000) 2019-10-13T06:09:41.066 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:41.061+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.063323+0000) 2019-10-13T06:09:41.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:41.058+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.059311+0000) 2019-10-13T06:09:41.068 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:41.062+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.063774+0000) 2019-10-13T06:09:41.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:41.062+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.063823+0000) 2019-10-13T06:09:41.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:41.062+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.064100+0000) 2019-10-13T06:09:41.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:41.062+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.064369+0000) 2019-10-13T06:09:41.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:41.061+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.061979+0000) 2019-10-13T06:09:41.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:41.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:41.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:41.855+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:41.855809+0000) 2019-10-13T06:09:42.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:42.721+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:42.721900+0000) 2019-10-13T06:09:42.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:42.722+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:42.722856+0000) 2019-10-13T06:09:44.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:44.216+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:44.216845+0000) 2019-10-13T06:09:46.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:46.063+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.063901+0000) 2019-10-13T06:09:46.081 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:46.066+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.068229+0000) 2019-10-13T06:09:46.083 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:46.067+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.068466+0000) 2019-10-13T06:09:46.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:46.067+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.068561+0000) 2019-10-13T06:09:46.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:46.067+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.068924+0000) 2019-10-13T06:09:46.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:46.067+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.069129+0000) 2019-10-13T06:09:46.087 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:46.068+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.069730+0000) 2019-10-13T06:09:46.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:46.070+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93350 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:46.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:46.066+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:46.067125+0000) 2019-10-13T06:09:48.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:48.261+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:48.262513+0000) 2019-10-13T06:09:48.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:48.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:48.602248+0000) 2019-10-13T06:09:48.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:48.601+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:48.602335+0000) 2019-10-13T06:09:50.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:50.886+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:50.887473+0000) 2019-10-13T06:09:51.077 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:51.072+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.073648+0000) 2019-10-13T06:09:51.077 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:51.072+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.073811+0000) 2019-10-13T06:09:51.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:51.069+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.069791+0000) 2019-10-13T06:09:51.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:51.072+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.074383+0000) 2019-10-13T06:09:51.078 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:51.073+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.074797+0000) 2019-10-13T06:09:51.078 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:51.073+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.074912+0000) 2019-10-13T06:09:51.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:51.073+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.075097+0000) 2019-10-13T06:09:51.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:51.071+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.072392+0000) 2019-10-13T06:09:51.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:51.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:51.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:51.854+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:51.856031+0000) 2019-10-13T06:09:52.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:52.721+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:52.722049+0000) 2019-10-13T06:09:52.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:52.722+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:52.723061+0000) 2019-10-13T06:09:53.682 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:09:53.683 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:53.697 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:09:53.697 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:54.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:54.216+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:54.217041+0000) 2019-10-13T06:09:56.084 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:09:56.078+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.079656+0000) 2019-10-13T06:09:56.084 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:09:56.078+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.079802+0000) 2019-10-13T06:09:56.084 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:56.078+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.079942+0000) 2019-10-13T06:09:56.085 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:56.078+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.080060+0000) 2019-10-13T06:09:56.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:09:56.079+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.080559+0000) 2019-10-13T06:09:56.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:56.079+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.080785+0000) 2019-10-13T06:09:56.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:09:56.075+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.076208+0000) 2019-10-13T06:09:56.086 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:09:56.076+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:56.077234+0000) 2019-10-13T06:09:56.087 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:09:56.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:09:58.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:09:58.262+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:58.262663+0000) 2019-10-13T06:09:58.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:09:58.602+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:58.602489+0000) 2019-10-13T06:09:58.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:09:58.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:09:58.602491+0000) 2019-10-13T06:10:00.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:00.889+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:00.889898+0000) 2019-10-13T06:10:01.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:01.085+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.085774+0000) 2019-10-13T06:10:01.089 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:01.085+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.085927+0000) 2019-10-13T06:10:01.090 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:01.085+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.086231+0000) 2019-10-13T06:10:01.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:01.081+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.082680+0000) 2019-10-13T06:10:01.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:01.085+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.086320+0000) 2019-10-13T06:10:01.091 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:01.086+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.086502+0000) 2019-10-13T06:10:01.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:01.086+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.086673+0000) 2019-10-13T06:10:01.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:01.083+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.084363+0000) 2019-10-13T06:10:01.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:01.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:01.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:01.855+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:01.856218+0000) 2019-10-13T06:10:02.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:02.721+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:02.722201+0000) 2019-10-13T06:10:02.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:02.722+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:02.723270+0000) 2019-10-13T06:10:04.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:04.216+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:04.217254+0000) 2019-10-13T06:10:06.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:06.090+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.090962+0000) 2019-10-13T06:10:06.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:06.090+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.091296+0000) 2019-10-13T06:10:06.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:06.086+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.086969+0000) 2019-10-13T06:10:06.096 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:06.091+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.091977+0000) 2019-10-13T06:10:06.096 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:06.091+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.092093+0000) 2019-10-13T06:10:06.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:06.091+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.092229+0000) 2019-10-13T06:10:06.097 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:06.092+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.092549+0000) 2019-10-13T06:10:06.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:06.088+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:06.089475+0000) 2019-10-13T06:10:06.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:06.094+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93474 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:08.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:08.262+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:08.262837+0000) 2019-10-13T06:10:08.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:08.602+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:08.602709+0000) 2019-10-13T06:10:08.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:08.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:08.602715+0000) 2019-10-13T06:10:10.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:10.892+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:10.892196+0000) 2019-10-13T06:10:11.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:11.096+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.096682+0000) 2019-10-13T06:10:11.100 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:11.096+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.096989+0000) 2019-10-13T06:10:11.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:11.092+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.093294+0000) 2019-10-13T06:10:11.102 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:11.097+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.097463+0000) 2019-10-13T06:10:11.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:11.097+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.097668+0000) 2019-10-13T06:10:11.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:11.097+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.097913+0000) 2019-10-13T06:10:11.102 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:11.097+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.097953+0000) 2019-10-13T06:10:11.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:11.094+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.095236+0000) 2019-10-13T06:10:11.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:11.100+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:11.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:11.856+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:11.856417+0000) 2019-10-13T06:10:12.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:12.722+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:12.722400+0000) 2019-10-13T06:10:12.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:12.723+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:12.723450+0000) 2019-10-13T06:10:14.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:14.217+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:14.217467+0000) 2019-10-13T06:10:16.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:16.096+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.098237+0000) 2019-10-13T06:10:16.106 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:16.102+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.102886+0000) 2019-10-13T06:10:16.106 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:16.102+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.103265+0000) 2019-10-13T06:10:16.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:16.102+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.103353+0000) 2019-10-13T06:10:16.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:16.103+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.103767+0000) 2019-10-13T06:10:16.107 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:16.103+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.103912+0000) 2019-10-13T06:10:16.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:16.103+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.104170+0000) 2019-10-13T06:10:16.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:16.098+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:16.100538+0000) 2019-10-13T06:10:16.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:16.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:18.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:18.262+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:18.263025+0000) 2019-10-13T06:10:18.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:18.601+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:18.602900+0000) 2019-10-13T06:10:18.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:18.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:18.602905+0000) 2019-10-13T06:10:20.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:20.893+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:20.894612+0000) 2019-10-13T06:10:21.111 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:21.106+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.107836+0000) 2019-10-13T06:10:21.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:21.106+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.107973+0000) 2019-10-13T06:10:21.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:21.106+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.108214+0000) 2019-10-13T06:10:21.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:21.102+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.104197+0000) 2019-10-13T06:10:21.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:21.107+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.108572+0000) 2019-10-13T06:10:21.113 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:21.107+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.108961+0000) 2019-10-13T06:10:21.113 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:21.107+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.109155+0000) 2019-10-13T06:10:21.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:21.104+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.106465+0000) 2019-10-13T06:10:21.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:21.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93570 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:21.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:21.855+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:21.856541+0000) 2019-10-13T06:10:22.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:22.721+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:22.722551+0000) 2019-10-13T06:10:22.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:22.722+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:22.723617+0000) 2019-10-13T06:10:23.843 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:10:23.843 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:23.848 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:10:23.849 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:24.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:24.216+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:24.217626+0000) 2019-10-13T06:10:26.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:26.106+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.107909+0000) 2019-10-13T06:10:26.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:26.110+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.112205+0000) 2019-10-13T06:10:26.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:26.111+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.112706+0000) 2019-10-13T06:10:26.116 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:26.111+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.112852+0000) 2019-10-13T06:10:26.117 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:26.111+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.113042+0000) 2019-10-13T06:10:26.117 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:26.112+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.113390+0000) 2019-10-13T06:10:26.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:26.112+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.113675+0000) 2019-10-13T06:10:26.118 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:26.108+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:26.110341+0000) 2019-10-13T06:10:26.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:26.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:28.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:28.262+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:28.263191+0000) 2019-10-13T06:10:28.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:28.602+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:28.603117+0000) 2019-10-13T06:10:28.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:28.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:28.603126+0000) 2019-10-13T06:10:30.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:30.896+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:30.896863+0000) 2019-10-13T06:10:31.122 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:31.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.118124+0000) 2019-10-13T06:10:31.123 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:31.117+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.118449+0000) 2019-10-13T06:10:31.123 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:31.117+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.118565+0000) 2019-10-13T06:10:31.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:31.117+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.119020+0000) 2019-10-13T06:10:31.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:31.117+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.118999+0000) 2019-10-13T06:10:31.124 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:31.117+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.119195+0000) 2019-10-13T06:10:31.124 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:31.113+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.114108+0000) 2019-10-13T06:10:31.124 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:31.115+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.116438+0000) 2019-10-13T06:10:31.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:31.120+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:31.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:31.856+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:31.856709+0000) 2019-10-13T06:10:32.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:32.722+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:32.722718+0000) 2019-10-13T06:10:32.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:32.723+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:32.723823+0000) 2019-10-13T06:10:34.222 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:34.216+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:34.217787+0000) 2019-10-13T06:10:36.131 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:36.122+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.124053+0000) 2019-10-13T06:10:36.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:36.123+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.124460+0000) 2019-10-13T06:10:36.133 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:36.123+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.124630+0000) 2019-10-13T06:10:36.133 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:36.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.124846+0000) 2019-10-13T06:10:36.133 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:36.123+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.124928+0000) 2019-10-13T06:10:36.133 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:36.123+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.125251+0000) 2019-10-13T06:10:36.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:36.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.119809+0000) 2019-10-13T06:10:36.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:36.121+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:36.122279+0000) 2019-10-13T06:10:36.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:36.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:38.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:38.262+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:38.263336+0000) 2019-10-13T06:10:38.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:38.602+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:38.603246+0000) 2019-10-13T06:10:38.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:38.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:38.603281+0000) 2019-10-13T06:10:40.906 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:40.898+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:40.899116+0000) 2019-10-13T06:10:41.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:41.128+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.130161+0000) 2019-10-13T06:10:41.134 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:41.128+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.130333+0000) 2019-10-13T06:10:41.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:41.125+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.126505+0000) 2019-10-13T06:10:41.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:41.129+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.130991+0000) 2019-10-13T06:10:41.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:41.129+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.131090+0000) 2019-10-13T06:10:41.135 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:41.129+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.131184+0000) 2019-10-13T06:10:41.135 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:41.129+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.131352+0000) 2019-10-13T06:10:41.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:41.127+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.128616+0000) 2019-10-13T06:10:41.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:41.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93694 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:41.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:41.856+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:41.856866+0000) 2019-10-13T06:10:42.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:42.722+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:42.722880+0000) 2019-10-13T06:10:42.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:42.723+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:42.723988+0000) 2019-10-13T06:10:44.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:44.217+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:44.217938+0000) 2019-10-13T06:10:46.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:46.134+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.135155+0000) 2019-10-13T06:10:46.139 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:46.130+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.131146+0000) 2019-10-13T06:10:46.139 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:46.135+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.135725+0000) 2019-10-13T06:10:46.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:46.135+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.135800+0000) 2019-10-13T06:10:46.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:46.135+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.135937+0000) 2019-10-13T06:10:46.140 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:46.135+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.136079+0000) 2019-10-13T06:10:46.140 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:46.135+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.136214+0000) 2019-10-13T06:10:46.141 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:46.132+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:46.133227+0000) 2019-10-13T06:10:46.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:46.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93722 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:48.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:48.263+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:48.263529+0000) 2019-10-13T06:10:48.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:48.603+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:48.603434+0000) 2019-10-13T06:10:48.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:48.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:48.603437+0000) 2019-10-13T06:10:50.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:50.901+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:50.901402+0000) 2019-10-13T06:10:51.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:51.134+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.135713+0000) 2019-10-13T06:10:51.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:51.139+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.140115+0000) 2019-10-13T06:10:51.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:51.139+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.140163+0000) 2019-10-13T06:10:51.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:51.139+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.140343+0000) 2019-10-13T06:10:51.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:51.140+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.140653+0000) 2019-10-13T06:10:51.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:51.140+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.140867+0000) 2019-10-13T06:10:51.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:51.140+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.141120+0000) 2019-10-13T06:10:51.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:51.138+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.138939+0000) 2019-10-13T06:10:51.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:51.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:51.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:51.856+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:51.857007+0000) 2019-10-13T06:10:52.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:52.722+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:52.723056+0000) 2019-10-13T06:10:52.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:52.723+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:52.724118+0000) 2019-10-13T06:10:54.024 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:10:54.025 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:54.029 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:10:54.029 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:54.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:54.217+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:54.218109+0000) 2019-10-13T06:10:56.148 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:10:56.140+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.141259+0000) 2019-10-13T06:10:56.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:56.145+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.145549+0000) 2019-10-13T06:10:56.150 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:10:56.145+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.146116+0000) 2019-10-13T06:10:56.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:56.145+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.146333+0000) 2019-10-13T06:10:56.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:56.146+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.146421+0000) 2019-10-13T06:10:56.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:10:56.146+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.146690+0000) 2019-10-13T06:10:56.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:10:56.142+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.143632+0000) 2019-10-13T06:10:56.151 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:10:56.146+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:56.146941+0000) 2019-10-13T06:10:56.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:10:56.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:10:58.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:10:58.263+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:58.263728+0000) 2019-10-13T06:10:58.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:10:58.603+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:58.603610+0000) 2019-10-13T06:10:58.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:10:58.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:10:58.603610+0000) 2019-10-13T06:11:00.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:00.903+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:00.903911+0000) 2019-10-13T06:11:01.154 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:01.150+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.151223+0000) 2019-10-13T06:11:01.155 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:01.146+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.147547+0000) 2019-10-13T06:11:01.155 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:01.151+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.151934+0000) 2019-10-13T06:11:01.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:01.151+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.152207+0000) 2019-10-13T06:11:01.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:01.151+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.152344+0000) 2019-10-13T06:11:01.156 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:01.152+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.152518+0000) 2019-10-13T06:11:01.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:01.152+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.152774+0000) 2019-10-13T06:11:01.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:01.149+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.149922+0000) 2019-10-13T06:11:01.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:01.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:01.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:01.857+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:01.857145+0000) 2019-10-13T06:11:02.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:02.722+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:02.723196+0000) 2019-10-13T06:11:02.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:02.724+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:02.724261+0000) 2019-10-13T06:11:04.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:04.217+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:04.218301+0000) 2019-10-13T06:11:06.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:06.151+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.152501+0000) 2019-10-13T06:11:06.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:06.155+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.156816+0000) 2019-10-13T06:11:06.161 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:06.155+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.157165+0000) 2019-10-13T06:11:06.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:06.156+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.157420+0000) 2019-10-13T06:11:06.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:06.156+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.157538+0000) 2019-10-13T06:11:06.162 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:06.156+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.157661+0000) 2019-10-13T06:11:06.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:06.156+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.157797+0000) 2019-10-13T06:11:06.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:06.155+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:06.156115+0000) 2019-10-13T06:11:06.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:06.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:08.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:08.263+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:08.263936+0000) 2019-10-13T06:11:08.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:08.602+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:08.603754+0000) 2019-10-13T06:11:08.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:08.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:08.603771+0000) 2019-10-13T06:11:10.914 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:10.904+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:10.906293+0000) 2019-10-13T06:11:11.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:11.156+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.158269+0000) 2019-10-13T06:11:11.166 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:11.161+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.162562+0000) 2019-10-13T06:11:11.167 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:11.161+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.162869+0000) 2019-10-13T06:11:11.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:11.161+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.163012+0000) 2019-10-13T06:11:11.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:11.161+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.163091+0000) 2019-10-13T06:11:11.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:11.161+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.163204+0000) 2019-10-13T06:11:11.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:11.162+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.163427+0000) 2019-10-13T06:11:11.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:11.159+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.161254+0000) 2019-10-13T06:11:11.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:11.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93879 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:11.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:11.856+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:11.857288+0000) 2019-10-13T06:11:12.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:12.722+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:12.723315+0000) 2019-10-13T06:11:12.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:12.723+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:12.724421+0000) 2019-10-13T06:11:14.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:14.217+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:14.218513+0000) 2019-10-13T06:11:16.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:16.165+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.166872+0000) 2019-10-13T06:11:16.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:16.165+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.167193+0000) 2019-10-13T06:11:16.171 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:16.165+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.167256+0000) 2019-10-13T06:11:16.171 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:16.166+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.167484+0000) 2019-10-13T06:11:16.172 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:16.161+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.163471+0000) 2019-10-13T06:11:16.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:16.166+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.167749+0000) 2019-10-13T06:11:16.172 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:16.166+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.168246+0000) 2019-10-13T06:11:16.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:16.164+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:16.165875+0000) 2019-10-13T06:11:16.174 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:16.169+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:18.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:18.263+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:18.264181+0000) 2019-10-13T06:11:18.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:18.603+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:18.603925+0000) 2019-10-13T06:11:18.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:18.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:18.603917+0000) 2019-10-13T06:11:20.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:20.907+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:20.908363+0000) 2019-10-13T06:11:21.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:21.171+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.172742+0000) 2019-10-13T06:11:21.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:21.171+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.172828+0000) 2019-10-13T06:11:21.177 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:21.171+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.173059+0000) 2019-10-13T06:11:21.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:21.171+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.173013+0000) 2019-10-13T06:11:21.178 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:21.171+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.173132+0000) 2019-10-13T06:11:21.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:21.171+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.173293+0000) 2019-10-13T06:11:21.178 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:21.167+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.169411+0000) 2019-10-13T06:11:21.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:21.168+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.170561+0000) 2019-10-13T06:11:21.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:21.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93942 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:21.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:21.856+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:21.857432+0000) 2019-10-13T06:11:22.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:22.722+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:22.723501+0000) 2019-10-13T06:11:22.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:22.723+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:22.724601+0000) 2019-10-13T06:11:24.161 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:11:24.162 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:24.166 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:11:24.166 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:24.222 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:24.217+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:24.218615+0000) 2019-10-13T06:11:26.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:26.175+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.176965+0000) 2019-10-13T06:11:26.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:26.175+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.177116+0000) 2019-10-13T06:11:26.181 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:26.175+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.177236+0000) 2019-10-13T06:11:26.181 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:26.176+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.177512+0000) 2019-10-13T06:11:26.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:26.172+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.173505+0000) 2019-10-13T06:11:26.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:26.176+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.177779+0000) 2019-10-13T06:11:26.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:26.176+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.177886+0000) 2019-10-13T06:11:26.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:26.174+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:26.175410+0000) 2019-10-13T06:11:26.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:26.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 93970 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:28.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:28.263+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:28.264391+0000) 2019-10-13T06:11:28.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:28.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:28.604111+0000) 2019-10-13T06:11:28.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:28.603+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:28.604113+0000) 2019-10-13T06:11:30.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:30.910+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:30.910838+0000) 2019-10-13T06:11:31.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:31.176+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.177358+0000) 2019-10-13T06:11:31.186 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:31.181+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.181708+0000) 2019-10-13T06:11:31.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:31.181+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.181988+0000) 2019-10-13T06:11:31.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:31.181+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.182196+0000) 2019-10-13T06:11:31.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:31.181+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.182234+0000) 2019-10-13T06:11:31.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:31.182+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.182506+0000) 2019-10-13T06:11:31.187 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:31.182+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.182837+0000) 2019-10-13T06:11:31.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:31.179+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.180514+0000) 2019-10-13T06:11:31.188 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:31.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94003 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:31.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:31.857+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:31.857622+0000) 2019-10-13T06:11:32.727 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:32.723+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:32.723709+0000) 2019-10-13T06:11:32.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:32.724+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:32.724789+0000) 2019-10-13T06:11:34.222 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:34.218+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:34.218754+0000) 2019-10-13T06:11:36.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:36.182+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.183226+0000) 2019-10-13T06:11:36.191 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:36.187+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.187524+0000) 2019-10-13T06:11:36.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:36.187+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.187792+0000) 2019-10-13T06:11:36.192 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:36.187+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.187950+0000) 2019-10-13T06:11:36.192 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:36.187+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.188083+0000) 2019-10-13T06:11:36.192 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:36.188+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.188723+0000) 2019-10-13T06:11:36.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:36.188+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.188867+0000) 2019-10-13T06:11:36.193 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:36.185+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:36.186554+0000) 2019-10-13T06:11:36.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:36.190+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94031 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:38.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:38.264+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:38.264613+0000) 2019-10-13T06:11:38.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:38.604+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:38.604283+0000) 2019-10-13T06:11:38.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:38.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:38.604305+0000) 2019-10-13T06:11:40.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:40.912+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:40.913359+0000) 2019-10-13T06:11:41.196 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:41.192+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.192661+0000) 2019-10-13T06:11:41.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:41.192+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.192581+0000) 2019-10-13T06:11:41.197 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:41.187+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.188681+0000) 2019-10-13T06:11:41.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:41.192+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.193237+0000) 2019-10-13T06:11:41.197 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:41.192+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.193381+0000) 2019-10-13T06:11:41.198 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:41.192+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.193282+0000) 2019-10-13T06:11:41.198 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:41.193+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.193515+0000) 2019-10-13T06:11:41.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:41.190+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.190926+0000) 2019-10-13T06:11:41.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:41.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:41.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:41.857+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:41.857767+0000) 2019-10-13T06:11:42.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:42.723+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:42.723895+0000) 2019-10-13T06:11:42.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:42.724+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:42.724969+0000) 2019-10-13T06:11:44.222 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:44.218+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:44.218868+0000) 2019-10-13T06:11:46.200 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:46.196+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.197173+0000) 2019-10-13T06:11:46.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:46.196+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.197313+0000) 2019-10-13T06:11:46.201 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:46.197+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.197427+0000) 2019-10-13T06:11:46.202 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:46.193+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.193742+0000) 2019-10-13T06:11:46.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:46.197+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.198105+0000) 2019-10-13T06:11:46.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:46.197+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.198277+0000) 2019-10-13T06:11:46.202 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:46.198+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.198424+0000) 2019-10-13T06:11:46.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:46.194+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:46.195008+0000) 2019-10-13T06:11:46.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:46.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:48.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:48.263+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:48.264771+0000) 2019-10-13T06:11:48.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:48.603+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:48.604466+0000) 2019-10-13T06:11:48.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:48.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:48.604470+0000) 2019-10-13T06:11:50.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:50.915+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:50.915676+0000) 2019-10-13T06:11:51.205 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:51.200+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.202338+0000) 2019-10-13T06:11:51.206 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:51.201+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.202512+0000) 2019-10-13T06:11:51.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:51.197+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.198583+0000) 2019-10-13T06:11:51.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:51.201+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.202985+0000) 2019-10-13T06:11:51.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:51.201+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.203165+0000) 2019-10-13T06:11:51.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:51.202+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.203487+0000) 2019-10-13T06:11:51.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:51.202+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.203633+0000) 2019-10-13T06:11:51.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:51.200+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.201526+0000) 2019-10-13T06:11:51.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:51.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:51.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:51.857+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:51.857959+0000) 2019-10-13T06:11:52.727 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:52.723+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:52.724046+0000) 2019-10-13T06:11:52.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:52.724+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:52.725127+0000) 2019-10-13T06:11:54.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:54.218+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:54.219072+0000) 2019-10-13T06:11:54.324 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:11:54.324 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:54.329 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:11:54.329 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:56.210 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:11:56.205+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.206883+0000) 2019-10-13T06:11:56.211 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:11:56.205+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.207345+0000) 2019-10-13T06:11:56.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:11:56.203+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.203900+0000) 2019-10-13T06:11:56.212 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:11:56.206+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.207843+0000) 2019-10-13T06:11:56.212 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:56.206+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.208125+0000) 2019-10-13T06:11:56.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:56.206+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.208299+0000) 2019-10-13T06:11:56.213 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:56.207+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.208811+0000) 2019-10-13T06:11:56.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:11:56.204+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:56.205508+0000) 2019-10-13T06:11:56.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:11:56.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:11:58.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:11:58.264+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:58.264965+0000) 2019-10-13T06:11:58.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:11:58.603+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:58.604659+0000) 2019-10-13T06:11:58.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:11:58.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:11:58.604652+0000) 2019-10-13T06:12:00.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:00.917+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:00.918112+0000) 2019-10-13T06:12:01.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:01.211+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.213364+0000) 2019-10-13T06:12:01.217 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:01.212+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.213719+0000) 2019-10-13T06:12:01.218 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:01.207+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.209708+0000) 2019-10-13T06:12:01.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:01.212+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.214063+0000) 2019-10-13T06:12:01.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:01.212+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.214029+0000) 2019-10-13T06:12:01.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:01.213+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.214456+0000) 2019-10-13T06:12:01.220 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:01.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.214523+0000) 2019-10-13T06:12:01.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:01.210+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.212141+0000) 2019-10-13T06:12:01.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:01.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94190 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:01.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:01.857+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:01.858122+0000) 2019-10-13T06:12:02.727 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:02.723+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:02.724239+0000) 2019-10-13T06:12:02.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:02.724+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:02.725297+0000) 2019-10-13T06:12:04.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:04.218+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:04.219233+0000) 2019-10-13T06:12:06.222 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:06.213+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.214991+0000) 2019-10-13T06:12:06.224 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:06.218+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.219407+0000) 2019-10-13T06:12:06.224 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:06.218+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.219694+0000) 2019-10-13T06:12:06.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:06.218+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.219623+0000) 2019-10-13T06:12:06.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:06.218+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.219872+0000) 2019-10-13T06:12:06.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:06.218+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.220197+0000) 2019-10-13T06:12:06.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:06.219+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.220577+0000) 2019-10-13T06:12:06.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:06.215+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:06.217258+0000) 2019-10-13T06:12:06.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:06.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:08.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:08.263+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:08.265155+0000) 2019-10-13T06:12:08.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:08.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:08.604782+0000) 2019-10-13T06:12:08.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:08.604+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:08.604806+0000) 2019-10-13T06:12:10.928 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:10.919+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:10.920565+0000) 2019-10-13T06:12:11.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:11.223+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.225017+0000) 2019-10-13T06:12:11.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:11.219+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.221200+0000) 2019-10-13T06:12:11.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:11.224+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.225477+0000) 2019-10-13T06:12:11.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:11.224+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.225675+0000) 2019-10-13T06:12:11.230 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:11.224+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.225751+0000) 2019-10-13T06:12:11.230 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:11.224+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.226089+0000) 2019-10-13T06:12:11.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:11.225+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.226479+0000) 2019-10-13T06:12:11.231 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:11.222+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.224369+0000) 2019-10-13T06:12:11.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:11.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:11.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:11.857+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:11.858295+0000) 2019-10-13T06:12:12.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:12.723+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:12.724437+0000) 2019-10-13T06:12:12.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:12.724+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:12.725472+0000) 2019-10-13T06:12:14.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:14.218+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:14.219412+0000) 2019-10-13T06:12:16.233 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:16.230+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.230449+0000) 2019-10-13T06:12:16.234 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:16.225+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.226558+0000) 2019-10-13T06:12:16.235 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:16.230+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.231099+0000) 2019-10-13T06:12:16.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:16.231+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.231435+0000) 2019-10-13T06:12:16.235 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:16.231+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.231827+0000) 2019-10-13T06:12:16.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:16.231+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.231907+0000) 2019-10-13T06:12:16.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:16.230+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.230824+0000) 2019-10-13T06:12:16.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:16.228+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:16.228955+0000) 2019-10-13T06:12:16.240 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:16.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94280 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:18.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:18.264+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:18.265271+0000) 2019-10-13T06:12:18.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:18.604+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:18.604967+0000) 2019-10-13T06:12:18.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:18.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:18.604967+0000) 2019-10-13T06:12:20.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:20.922+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:20.922972+0000) 2019-10-13T06:12:21.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:21.236+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.236723+0000) 2019-10-13T06:12:21.242 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:21.232+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.233576+0000) 2019-10-13T06:12:21.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:21.236+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.237213+0000) 2019-10-13T06:12:21.242 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:21.236+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.237033+0000) 2019-10-13T06:12:21.243 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:21.237+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.237406+0000) 2019-10-13T06:12:21.243 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:21.237+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.237517+0000) 2019-10-13T06:12:21.243 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:21.237+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.237738+0000) 2019-10-13T06:12:21.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:21.234+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.234991+0000) 2019-10-13T06:12:21.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:21.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:21.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:21.858+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:21.858486+0000) 2019-10-13T06:12:22.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:22.724+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:22.724606+0000) 2019-10-13T06:12:22.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:22.725+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:22.725618+0000) 2019-10-13T06:12:24.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:24.219+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:24.219566+0000) 2019-10-13T06:12:24.436 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:12:24.436 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:24.441 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:12:24.441 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:26.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:26.241+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.241890+0000) 2019-10-13T06:12:26.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:26.241+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.242270+0000) 2019-10-13T06:12:26.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:26.242+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.242393+0000) 2019-10-13T06:12:26.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:26.237+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.238472+0000) 2019-10-13T06:12:26.247 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:26.242+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.242834+0000) 2019-10-13T06:12:26.247 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:26.242+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.243066+0000) 2019-10-13T06:12:26.248 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:26.242+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.242983+0000) 2019-10-13T06:12:26.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:26.239+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:26.240100+0000) 2019-10-13T06:12:26.249 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:26.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:28.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:28.265+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:28.265457+0000) 2019-10-13T06:12:28.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:28.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:28.605145+0000) 2019-10-13T06:12:28.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:28.605+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:28.605146+0000) 2019-10-13T06:12:30.932 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:30.924+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:30.925318+0000) 2019-10-13T06:12:31.251 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:31.247+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.247468+0000) 2019-10-13T06:12:31.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:31.247+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.247644+0000) 2019-10-13T06:12:31.252 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:31.242+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.243636+0000) 2019-10-13T06:12:31.252 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:31.247+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.248074+0000) 2019-10-13T06:12:31.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:31.247+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.248200+0000) 2019-10-13T06:12:31.253 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:31.248+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.248412+0000) 2019-10-13T06:12:31.253 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:31.248+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.248611+0000) 2019-10-13T06:12:31.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:31.245+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.246525+0000) 2019-10-13T06:12:31.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:31.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94375 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:31.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:31.858+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:31.858631+0000) 2019-10-13T06:12:32.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:32.724+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:32.724780+0000) 2019-10-13T06:12:32.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:32.725+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:32.725804+0000) 2019-10-13T06:12:34.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:34.218+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:34.219756+0000) 2019-10-13T06:12:36.257 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:36.251+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.253226+0000) 2019-10-13T06:12:36.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:36.252+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.253653+0000) 2019-10-13T06:12:36.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:36.252+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.253717+0000) 2019-10-13T06:12:36.258 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:36.252+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.253479+0000) 2019-10-13T06:12:36.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:36.252+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.253804+0000) 2019-10-13T06:12:36.259 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:36.249+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.249882+0000) 2019-10-13T06:12:36.259 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:36.252+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.254352+0000) 2019-10-13T06:12:36.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:36.251+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:36.252164+0000) 2019-10-13T06:12:36.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:36.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:38.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:38.264+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:38.265611+0000) 2019-10-13T06:12:38.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:38.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:38.605297+0000) 2019-10-13T06:12:38.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:38.604+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:38.605296+0000) 2019-10-13T06:12:40.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:40.927+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:40.927687+0000) 2019-10-13T06:12:41.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:41.257+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.258855+0000) 2019-10-13T06:12:41.264 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:41.257+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.259101+0000) 2019-10-13T06:12:41.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:41.257+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.259310+0000) 2019-10-13T06:12:41.264 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:41.258+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.259464+0000) 2019-10-13T06:12:41.265 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:41.258+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.259680+0000) 2019-10-13T06:12:41.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:41.258+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.259998+0000) 2019-10-13T06:12:41.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:41.254+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.254779+0000) 2019-10-13T06:12:41.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:41.255+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.256345+0000) 2019-10-13T06:12:41.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:41.261+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:41.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:41.857+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:41.858816+0000) 2019-10-13T06:12:42.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:42.724+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:42.724934+0000) 2019-10-13T06:12:42.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:42.725+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:42.725977+0000) 2019-10-13T06:12:44.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:44.219+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:44.219939+0000) 2019-10-13T06:12:46.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:46.263+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.264507+0000) 2019-10-13T06:12:46.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:46.259+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.260584+0000) 2019-10-13T06:12:46.269 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:46.263+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.265044+0000) 2019-10-13T06:12:46.269 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:46.263+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.265195+0000) 2019-10-13T06:12:46.269 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:46.264+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.265440+0000) 2019-10-13T06:12:46.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:46.264+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.265643+0000) 2019-10-13T06:12:46.270 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:46.264+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.265770+0000) 2019-10-13T06:12:46.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:46.261+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:46.262308+0000) 2019-10-13T06:12:46.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:46.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:48.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:48.265+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:48.265825+0000) 2019-10-13T06:12:48.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:48.604+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:48.605439+0000) 2019-10-13T06:12:48.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:48.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:48.605446+0000) 2019-10-13T06:12:50.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:50.930+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:50.930176+0000) 2019-10-13T06:12:51.277 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:51.266+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.266943+0000) 2019-10-13T06:12:51.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:51.268+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.269371+0000) 2019-10-13T06:12:51.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:51.268+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.270316+0000) 2019-10-13T06:12:51.280 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:51.269+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.270538+0000) 2019-10-13T06:12:51.280 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:51.269+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.270924+0000) 2019-10-13T06:12:51.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:51.269+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.271233+0000) 2019-10-13T06:12:51.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:51.270+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.271424+0000) 2019-10-13T06:12:51.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:51.270+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.271846+0000) 2019-10-13T06:12:51.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:51.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94499 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:51.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:51.858+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:51.858983+0000) 2019-10-13T06:12:52.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:52.724+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:52.725098+0000) 2019-10-13T06:12:52.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:52.725+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:52.726159+0000) 2019-10-13T06:12:54.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:54.219+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:54.220104+0000) 2019-10-13T06:12:54.609 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:12:54.609 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:54.613 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:12:54.614 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:56.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:56.274+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.276154+0000) 2019-10-13T06:12:56.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:12:56.274+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.276303+0000) 2019-10-13T06:12:56.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:56.275+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.276665+0000) 2019-10-13T06:12:56.281 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:12:56.270+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.272349+0000) 2019-10-13T06:12:56.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:56.275+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.276871+0000) 2019-10-13T06:12:56.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:12:56.275+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.277001+0000) 2019-10-13T06:12:56.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:12:56.276+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.277583+0000) 2019-10-13T06:12:56.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:12:56.273+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:56.274809+0000) 2019-10-13T06:12:56.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:12:56.278+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:12:58.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:12:58.265+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:58.266026+0000) 2019-10-13T06:12:58.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:12:58.605+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:58.605604+0000) 2019-10-13T06:12:58.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:12:58.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:12:58.605611+0000) 2019-10-13T06:13:00.940 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:00.931+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:00.932688+0000) 2019-10-13T06:13:01.285 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:01.275+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.277692+0000) 2019-10-13T06:13:01.286 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:01.281+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.282106+0000) 2019-10-13T06:13:01.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:01.281+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.282245+0000) 2019-10-13T06:13:01.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:01.282+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.282464+0000) 2019-10-13T06:13:01.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:01.282+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.282661+0000) 2019-10-13T06:13:01.287 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:01.282+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.282539+0000) 2019-10-13T06:13:01.287 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:01.282+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.283342+0000) 2019-10-13T06:13:01.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:01.278+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.279925+0000) 2019-10-13T06:13:01.289 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:01.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:01.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:01.858+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:01.859174+0000) 2019-10-13T06:13:02.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:02.724+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:02.725253+0000) 2019-10-13T06:13:02.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:02.725+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:02.726344+0000) 2019-10-13T06:13:04.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:04.219+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:04.220249+0000) 2019-10-13T06:13:06.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:06.281+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.283606+0000) 2019-10-13T06:13:06.292 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:06.287+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.287769+0000) 2019-10-13T06:13:06.292 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:06.287+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.287976+0000) 2019-10-13T06:13:06.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:06.287+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.288148+0000) 2019-10-13T06:13:06.292 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:06.287+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.288287+0000) 2019-10-13T06:13:06.292 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:06.287+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.288356+0000) 2019-10-13T06:13:06.293 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:06.288+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.288581+0000) 2019-10-13T06:13:06.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:06.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:06.294 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:06.285+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:06.286549+0000) 2019-10-13T06:13:08.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:08.265+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:08.266202+0000) 2019-10-13T06:13:08.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:08.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:08.605831+0000) 2019-10-13T06:13:08.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:08.605+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:08.605832+0000) 2019-10-13T06:13:10.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:10.934+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:10.935016+0000) 2019-10-13T06:13:11.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:11.291+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.292015+0000) 2019-10-13T06:13:11.295 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:11.291+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.292123+0000) 2019-10-13T06:13:11.296 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:11.287+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.288719+0000) 2019-10-13T06:13:11.297 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:11.292+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.292558+0000) 2019-10-13T06:13:11.297 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:11.292+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.292664+0000) 2019-10-13T06:13:11.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:11.292+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.292958+0000) 2019-10-13T06:13:11.297 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:11.292+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.293012+0000) 2019-10-13T06:13:11.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:11.289+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.290282+0000) 2019-10-13T06:13:11.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:11.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:11.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:11.859+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:11.859313+0000) 2019-10-13T06:13:12.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:12.725+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:12.725434+0000) 2019-10-13T06:13:12.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:12.725+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:12.726547+0000) 2019-10-13T06:13:14.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:14.220+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:14.220435+0000) 2019-10-13T06:13:16.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:16.295+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.296332+0000) 2019-10-13T06:13:16.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:16.296+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.296491+0000) 2019-10-13T06:13:16.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:16.296+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.296680+0000) 2019-10-13T06:13:16.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:16.292+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.293231+0000) 2019-10-13T06:13:16.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:16.296+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.297019+0000) 2019-10-13T06:13:16.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:16.296+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.297316+0000) 2019-10-13T06:13:16.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:16.297+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.297392+0000) 2019-10-13T06:13:16.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:16.295+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:16.295778+0000) 2019-10-13T06:13:16.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:16.300+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94652 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:18.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:18.266+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:18.266393+0000) 2019-10-13T06:13:18.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:18.605+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:18.606037+0000) 2019-10-13T06:13:18.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:18.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:18.606062+0000) 2019-10-13T06:13:20.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:20.936+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:20.937391+0000) 2019-10-13T06:13:21.306 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:21.297+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.298166+0000) 2019-10-13T06:13:21.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:21.300+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.302163+0000) 2019-10-13T06:13:21.306 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:21.301+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.302631+0000) 2019-10-13T06:13:21.307 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:21.301+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.302869+0000) 2019-10-13T06:13:21.307 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:21.301+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.303126+0000) 2019-10-13T06:13:21.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:21.301+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.303362+0000) 2019-10-13T06:13:21.307 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:21.302+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.303649+0000) 2019-10-13T06:13:21.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:21.300+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.300962+0000) 2019-10-13T06:13:21.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:21.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94686 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:21.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:21.858+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:21.859461+0000) 2019-10-13T06:13:22.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:22.724+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:22.725585+0000) 2019-10-13T06:13:22.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:22.725+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:22.726724+0000) 2019-10-13T06:13:24.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:24.219+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:24.220614+0000) 2019-10-13T06:13:24.772 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:13:24.772 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:24.776 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:13:24.776 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:26.311 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:26.302+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.303636+0000) 2019-10-13T06:13:26.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:26.306+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.308039+0000) 2019-10-13T06:13:26.312 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:26.306+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.308362+0000) 2019-10-13T06:13:26.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:26.307+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.308627+0000) 2019-10-13T06:13:26.313 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:26.307+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.308866+0000) 2019-10-13T06:13:26.313 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:26.307+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.308941+0000) 2019-10-13T06:13:26.313 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:26.307+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.309103+0000) 2019-10-13T06:13:26.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:26.305+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:26.306355+0000) 2019-10-13T06:13:26.315 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:26.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:28.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:28.265+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:28.266567+0000) 2019-10-13T06:13:28.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:28.605+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:28.606207+0000) 2019-10-13T06:13:28.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:28.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:28.606208+0000) 2019-10-13T06:13:30.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:30.939+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:30.939690+0000) 2019-10-13T06:13:31.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:31.312+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.314177+0000) 2019-10-13T06:13:31.318 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:31.312+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.314388+0000) 2019-10-13T06:13:31.318 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:31.313+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.314493+0000) 2019-10-13T06:13:31.319 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:31.313+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.314844+0000) 2019-10-13T06:13:31.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:31.310+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.310955+0000) 2019-10-13T06:13:31.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:31.313+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.315173+0000) 2019-10-13T06:13:31.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:31.314+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.315524+0000) 2019-10-13T06:13:31.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:31.311+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.312593+0000) 2019-10-13T06:13:31.321 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:31.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:31.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:31.858+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:31.859628+0000) 2019-10-13T06:13:32.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:32.724+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:32.725734+0000) 2019-10-13T06:13:32.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:32.726+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:32.726958+0000) 2019-10-13T06:13:34.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:34.220+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:34.220804+0000) 2019-10-13T06:13:36.323 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:36.318+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.319661+0000) 2019-10-13T06:13:36.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:36.318+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.320307+0000) 2019-10-13T06:13:36.324 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:36.318+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.320035+0000) 2019-10-13T06:13:36.324 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:36.319+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.320444+0000) 2019-10-13T06:13:36.325 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:36.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.316700+0000) 2019-10-13T06:13:36.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:36.319+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.321011+0000) 2019-10-13T06:13:36.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:36.319+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.321196+0000) 2019-10-13T06:13:36.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:36.317+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:36.318169+0000) 2019-10-13T06:13:36.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:36.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94776 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:38.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:38.266+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:38.266761+0000) 2019-10-13T06:13:38.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:38.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:38.606407+0000) 2019-10-13T06:13:38.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:38.605+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:38.606428+0000) 2019-10-13T06:13:40.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:40.941+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:40.942061+0000) 2019-10-13T06:13:41.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:41.322+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.323533+0000) 2019-10-13T06:13:41.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:41.324+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.324884+0000) 2019-10-13T06:13:41.333 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:41.324+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.326233+0000) 2019-10-13T06:13:41.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:41.325+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.326509+0000) 2019-10-13T06:13:41.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:41.325+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.326561+0000) 2019-10-13T06:13:41.334 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:41.325+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.326813+0000) 2019-10-13T06:13:41.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:41.325+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.327006+0000) 2019-10-13T06:13:41.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:41.325+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.327360+0000) 2019-10-13T06:13:41.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:41.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:41.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:41.859+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:41.859825+0000) 2019-10-13T06:13:42.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:42.725+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:42.725897+0000) 2019-10-13T06:13:42.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:42.727+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:42.727147+0000) 2019-10-13T06:13:44.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:44.219+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:44.221004+0000) 2019-10-13T06:13:46.336 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:46.330+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.332263+0000) 2019-10-13T06:13:46.336 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:46.330+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.332373+0000) 2019-10-13T06:13:46.336 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:46.331+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.332473+0000) 2019-10-13T06:13:46.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:46.331+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.332641+0000) 2019-10-13T06:13:46.337 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:46.331+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.332882+0000) 2019-10-13T06:13:46.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:46.327+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.329136+0000) 2019-10-13T06:13:46.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:46.328+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.330082+0000) 2019-10-13T06:13:46.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:46.331+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:46.333344+0000) 2019-10-13T06:13:46.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:46.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:48.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:48.266+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:48.266977+0000) 2019-10-13T06:13:48.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:48.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:48.606600+0000) 2019-10-13T06:13:48.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:48.606+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:48.606626+0000) 2019-10-13T06:13:50.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:50.942+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:50.944473+0000) 2019-10-13T06:13:51.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:51.336+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.337100+0000) 2019-10-13T06:13:51.340 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:51.336+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.337363+0000) 2019-10-13T06:13:51.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:51.337+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.337628+0000) 2019-10-13T06:13:51.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:51.337+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.338050+0000) 2019-10-13T06:13:51.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:51.337+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.338293+0000) 2019-10-13T06:13:51.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:51.333+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.334863+0000) 2019-10-13T06:13:51.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:51.338+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.339050+0000) 2019-10-13T06:13:51.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:51.334+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.336404+0000) 2019-10-13T06:13:51.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:51.340+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:51.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:51.859+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:51.860010+0000) 2019-10-13T06:13:52.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:52.725+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:52.726079+0000) 2019-10-13T06:13:52.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:52.726+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:52.727299+0000) 2019-10-13T06:13:54.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:54.220+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:54.221198+0000) 2019-10-13T06:13:54.927 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:13:54.927 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:54.932 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:13:54.932 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:56.346 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:13:56.342+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.343153+0000) 2019-10-13T06:13:56.347 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:13:56.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.339382+0000) 2019-10-13T06:13:56.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:13:56.343+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.343458+0000) 2019-10-13T06:13:56.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:56.343+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.343638+0000) 2019-10-13T06:13:56.348 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:56.343+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.343971+0000) 2019-10-13T06:13:56.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:13:56.343+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.344304+0000) 2019-10-13T06:13:56.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:56.343+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.344195+0000) 2019-10-13T06:13:56.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:13:56.340+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:56.341874+0000) 2019-10-13T06:13:56.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:13:56.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:13:58.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:13:58.266+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:58.267128+0000) 2019-10-13T06:13:58.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:13:58.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:58.606780+0000) 2019-10-13T06:13:58.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:13:58.606+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:13:58.606841+0000) 2019-10-13T06:14:00.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:00.946+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:00.946780+0000) 2019-10-13T06:14:01.352 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:01.348+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.348743+0000) 2019-10-13T06:14:01.353 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:01.344+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.344995+0000) 2019-10-13T06:14:01.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:01.348+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.349294+0000) 2019-10-13T06:14:01.354 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:01.348+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.349389+0000) 2019-10-13T06:14:01.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:01.349+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.349745+0000) 2019-10-13T06:14:01.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:01.349+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.349786+0000) 2019-10-13T06:14:01.354 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:01.349+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.350264+0000) 2019-10-13T06:14:01.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:01.347+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.347884+0000) 2019-10-13T06:14:01.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:01.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:01.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:01.860+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:01.860179+0000) 2019-10-13T06:14:02.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:02.726+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:02.726260+0000) 2019-10-13T06:14:02.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:02.726+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:02.727484+0000) 2019-10-13T06:14:04.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:04.221+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:04.221357+0000) 2019-10-13T06:14:06.359 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:06.353+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.355070+0000) 2019-10-13T06:14:06.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:06.353+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.355318+0000) 2019-10-13T06:14:06.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:06.354+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.355422+0000) 2019-10-13T06:14:06.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:06.354+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.355539+0000) 2019-10-13T06:14:06.360 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:06.354+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.355870+0000) 2019-10-13T06:14:06.360 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:06.354+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.356085+0000) 2019-10-13T06:14:06.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:06.350+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.351640+0000) 2019-10-13T06:14:06.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:06.353+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:06.354022+0000) 2019-10-13T06:14:06.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:06.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94964 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:08.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:08.266+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:08.267306+0000) 2019-10-13T06:14:08.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:08.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:08.606958+0000) 2019-10-13T06:14:08.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:08.606+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:08.607002+0000) 2019-10-13T06:14:10.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:10.948+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:10.949149+0000) 2019-10-13T06:14:11.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:11.358+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.359835+0000) 2019-10-13T06:14:11.363 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:11.358+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.360039+0000) 2019-10-13T06:14:11.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:11.356+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.356828+0000) 2019-10-13T06:14:11.365 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:11.358+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.360361+0000) 2019-10-13T06:14:11.365 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:11.358+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.360175+0000) 2019-10-13T06:14:11.365 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:11.359+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.360746+0000) 2019-10-13T06:14:11.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:11.359+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.361214+0000) 2019-10-13T06:14:11.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:11.357+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.358414+0000) 2019-10-13T06:14:11.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:11.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 94995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:11.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:11.859+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:11.860372+0000) 2019-10-13T06:14:12.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:12.725+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:12.726419+0000) 2019-10-13T06:14:12.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:12.726+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:12.727657+0000) 2019-10-13T06:14:14.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:14.220+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:14.221497+0000) 2019-10-13T06:14:16.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:16.364+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.366078+0000) 2019-10-13T06:14:16.371 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:16.364+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.366265+0000) 2019-10-13T06:14:16.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:16.365+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.366620+0000) 2019-10-13T06:14:16.371 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:16.365+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.367022+0000) 2019-10-13T06:14:16.371 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:16.365+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.367119+0000) 2019-10-13T06:14:16.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:16.366+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.367612+0000) 2019-10-13T06:14:16.372 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:16.362+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.363283+0000) 2019-10-13T06:14:16.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:16.363+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:16.364392+0000) 2019-10-13T06:14:16.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:16.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:18.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:18.266+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:18.267500+0000) 2019-10-13T06:14:18.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:18.606+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:18.607164+0000) 2019-10-13T06:14:18.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:18.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:18.607159+0000) 2019-10-13T06:14:20.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:20.951+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:20.951443+0000) 2019-10-13T06:14:21.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:21.370+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.371738+0000) 2019-10-13T06:14:21.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:21.367+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.367849+0000) 2019-10-13T06:14:21.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:21.370+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.372102+0000) 2019-10-13T06:14:21.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:21.371+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.373039+0000) 2019-10-13T06:14:21.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:21.371+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.373158+0000) 2019-10-13T06:14:21.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:21.371+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.373257+0000) 2019-10-13T06:14:21.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:21.372+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.373508+0000) 2019-10-13T06:14:21.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:21.369+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.370565+0000) 2019-10-13T06:14:21.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:21.374+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95058 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:21.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:21.859+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:21.860584+0000) 2019-10-13T06:14:22.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:22.725+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:22.726574+0000) 2019-10-13T06:14:22.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:22.727+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:22.727842+0000) 2019-10-13T06:14:24.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:24.221+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:24.221650+0000) 2019-10-13T06:14:25.026 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:14:25.026 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:25.051 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:14:25.051 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:26.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:26.376+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.377912+0000) 2019-10-13T06:14:26.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:26.373+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.374191+0000) 2019-10-13T06:14:26.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:26.377+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.378537+0000) 2019-10-13T06:14:26.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:26.377+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.378621+0000) 2019-10-13T06:14:26.383 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:26.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.378745+0000) 2019-10-13T06:14:26.383 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:26.377+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.378997+0000) 2019-10-13T06:14:26.383 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:26.377+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.379147+0000) 2019-10-13T06:14:26.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:26.376+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:26.377462+0000) 2019-10-13T06:14:26.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:26.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:28.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:28.267+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:28.267703+0000) 2019-10-13T06:14:28.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:28.606+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:28.607331+0000) 2019-10-13T06:14:28.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:28.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:28.607332+0000) 2019-10-13T06:14:30.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:30.953+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:30.953765+0000) 2019-10-13T06:14:31.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:31.382+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.383879+0000) 2019-10-13T06:14:31.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:31.382+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.384054+0000) 2019-10-13T06:14:31.388 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:31.382+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.384162+0000) 2019-10-13T06:14:31.388 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:31.379+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.380313+0000) 2019-10-13T06:14:31.389 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:31.383+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.384437+0000) 2019-10-13T06:14:31.389 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:31.383+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.384614+0000) 2019-10-13T06:14:31.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:31.383+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.384904+0000) 2019-10-13T06:14:31.390 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:31.382+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.382764+0000) 2019-10-13T06:14:31.390 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:31.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95119 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:31.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:31.860+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:31.860723+0000) 2019-10-13T06:14:32.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:32.726+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:32.726742+0000) 2019-10-13T06:14:32.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:32.727+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:32.728024+0000) 2019-10-13T06:14:34.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:34.221+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:34.221781+0000) 2019-10-13T06:14:36.392 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:36.388+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.388515+0000) 2019-10-13T06:14:36.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:36.388+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.388808+0000) 2019-10-13T06:14:36.392 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:36.388+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.388911+0000) 2019-10-13T06:14:36.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:36.388+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.389083+0000) 2019-10-13T06:14:36.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:36.385+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.385757+0000) 2019-10-13T06:14:36.394 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:36.389+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.389471+0000) 2019-10-13T06:14:36.394 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:36.388+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.389340+0000) 2019-10-13T06:14:36.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:36.386+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:36.387644+0000) 2019-10-13T06:14:36.395 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:36.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:38.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:38.267+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:38.267892+0000) 2019-10-13T06:14:38.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:38.607+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:38.607444+0000) 2019-10-13T06:14:38.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:38.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:38.607486+0000) 2019-10-13T06:14:40.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:40.955+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:40.956097+0000) 2019-10-13T06:14:41.397 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:41.388+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.390033+0000) 2019-10-13T06:14:41.398 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:41.393+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.394313+0000) 2019-10-13T06:14:41.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:41.394+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.394412+0000) 2019-10-13T06:14:41.398 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:41.394+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.394584+0000) 2019-10-13T06:14:41.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:41.394+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.394855+0000) 2019-10-13T06:14:41.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:41.394+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.395198+0000) 2019-10-13T06:14:41.399 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:41.395+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.395520+0000) 2019-10-13T06:14:41.400 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:41.391+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.393369+0000) 2019-10-13T06:14:41.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:41.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95182 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:41.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:41.860+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:41.860866+0000) 2019-10-13T06:14:42.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:42.726+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:42.726870+0000) 2019-10-13T06:14:42.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:42.727+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:42.728228+0000) 2019-10-13T06:14:44.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:44.221+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:44.221976+0000) 2019-10-13T06:14:46.403 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:46.394+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.395812+0000) 2019-10-13T06:14:46.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:46.399+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.400035+0000) 2019-10-13T06:14:46.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:46.399+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.400375+0000) 2019-10-13T06:14:46.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:46.400+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.400824+0000) 2019-10-13T06:14:46.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:46.400+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.400827+0000) 2019-10-13T06:14:46.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:46.400+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.401055+0000) 2019-10-13T06:14:46.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:46.400+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.400887+0000) 2019-10-13T06:14:46.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:46.396+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:46.398269+0000) 2019-10-13T06:14:46.406 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:46.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95212 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:48.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:48.267+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:48.268061+0000) 2019-10-13T06:14:48.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:48.607+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:48.607637+0000) 2019-10-13T06:14:48.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:48.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:48.607638+0000) 2019-10-13T06:14:50.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:50.957+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:50.958355+0000) 2019-10-13T06:14:51.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:51.404+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.405491+0000) 2019-10-13T06:14:51.410 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:51.404+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.405607+0000) 2019-10-13T06:14:51.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:51.404+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.405844+0000) 2019-10-13T06:14:51.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:51.404+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.406134+0000) 2019-10-13T06:14:51.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:51.404+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.406354+0000) 2019-10-13T06:14:51.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:51.405+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.406464+0000) 2019-10-13T06:14:51.411 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:51.400+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.401100+0000) 2019-10-13T06:14:51.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:51.402+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.403249+0000) 2019-10-13T06:14:51.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:51.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95243 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:51.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:51.860+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:51.861039+0000) 2019-10-13T06:14:52.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:52.726+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:52.727016+0000) 2019-10-13T06:14:52.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:52.727+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:52.728389+0000) 2019-10-13T06:14:54.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:54.221+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:54.222150+0000) 2019-10-13T06:14:55.124 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:14:55.124 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:55.129 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:14:55.129 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:56.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:14:56.404+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.405684+0000) 2019-10-13T06:14:56.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:56.408+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.410009+0000) 2019-10-13T06:14:56.414 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:56.408+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.410185+0000) 2019-10-13T06:14:56.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:14:56.409+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.410441+0000) 2019-10-13T06:14:56.414 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:56.409+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.410693+0000) 2019-10-13T06:14:56.415 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:14:56.409+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.410954+0000) 2019-10-13T06:14:56.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:14:56.409+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.411140+0000) 2019-10-13T06:14:56.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:14:56.408+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:56.409142+0000) 2019-10-13T06:14:56.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:14:56.412+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:14:58.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:14:58.266+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:58.268270+0000) 2019-10-13T06:14:58.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:14:58.606+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:58.607853+0000) 2019-10-13T06:14:58.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:14:58.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:14:58.607850+0000) 2019-10-13T06:15:00.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:00.960+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:00.960613+0000) 2019-10-13T06:15:01.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:01.413+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.415293+0000) 2019-10-13T06:15:01.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:01.414+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.415626+0000) 2019-10-13T06:15:01.419 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:01.414+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.415977+0000) 2019-10-13T06:15:01.420 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:01.414+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.416129+0000) 2019-10-13T06:15:01.420 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:01.414+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.416154+0000) 2019-10-13T06:15:01.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:01.415+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.416564+0000) 2019-10-13T06:15:01.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:01.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.412510+0000) 2019-10-13T06:15:01.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:01.414+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.414758+0000) 2019-10-13T06:15:01.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:01.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:01.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:01.860+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:01.861186+0000) 2019-10-13T06:15:02.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:02.726+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:02.727201+0000) 2019-10-13T06:15:02.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:02.728+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:02.728559+0000) 2019-10-13T06:15:04.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:04.220+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:04.222351+0000) 2019-10-13T06:15:06.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:06.419+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.421186+0000) 2019-10-13T06:15:06.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:06.419+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.421290+0000) 2019-10-13T06:15:06.425 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:06.420+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.421727+0000) 2019-10-13T06:15:06.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:06.420+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.421837+0000) 2019-10-13T06:15:06.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:06.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.418404+0000) 2019-10-13T06:15:06.427 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:06.420+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.422188+0000) 2019-10-13T06:15:06.427 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:06.421+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.422494+0000) 2019-10-13T06:15:06.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:06.418+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:06.419568+0000) 2019-10-13T06:15:06.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:06.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:08.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:08.267+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:08.268492+0000) 2019-10-13T06:15:08.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:08.607+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:08.608066+0000) 2019-10-13T06:15:08.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:08.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:08.608064+0000) 2019-10-13T06:15:10.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:10.962+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:10.963116+0000) 2019-10-13T06:15:11.430 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:11.425+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.426816+0000) 2019-10-13T06:15:11.430 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:11.425+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.427362+0000) 2019-10-13T06:15:11.431 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:11.426+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.427507+0000) 2019-10-13T06:15:11.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:11.422+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.423650+0000) 2019-10-13T06:15:11.432 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:11.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.427955+0000) 2019-10-13T06:15:11.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:11.426+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.428096+0000) 2019-10-13T06:15:11.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:11.427+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.428460+0000) 2019-10-13T06:15:11.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:11.425+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.426362+0000) 2019-10-13T06:15:11.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:11.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:11.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:11.860+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:11.861339+0000) 2019-10-13T06:15:12.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:12.726+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:12.727386+0000) 2019-10-13T06:15:12.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:12.728+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:12.728761+0000) 2019-10-13T06:15:14.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:14.221+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:14.222503+0000) 2019-10-13T06:15:16.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:16.431+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.432744+0000) 2019-10-13T06:15:16.437 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:16.428+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.429078+0000) 2019-10-13T06:15:16.437 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:16.431+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.433348+0000) 2019-10-13T06:15:16.437 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:16.432+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.433492+0000) 2019-10-13T06:15:16.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:16.432+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.433700+0000) 2019-10-13T06:15:16.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:16.432+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.433919+0000) 2019-10-13T06:15:16.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:16.432+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.434044+0000) 2019-10-13T06:15:16.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:16.431+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:16.431784+0000) 2019-10-13T06:15:16.440 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:16.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95397 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:18.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:18.268+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:18.268645+0000) 2019-10-13T06:15:18.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:18.607+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:18.608216+0000) 2019-10-13T06:15:18.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:18.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:18.608211+0000) 2019-10-13T06:15:20.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:20.964+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:20.965548+0000) 2019-10-13T06:15:21.442 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:21.437+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.438381+0000) 2019-10-13T06:15:21.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:21.438+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.438514+0000) 2019-10-13T06:15:21.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:21.438+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.438622+0000) 2019-10-13T06:15:21.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:21.434+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.434817+0000) 2019-10-13T06:15:21.443 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:21.438+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.438896+0000) 2019-10-13T06:15:21.443 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:21.438+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.439007+0000) 2019-10-13T06:15:21.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:21.438+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.439228+0000) 2019-10-13T06:15:21.444 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:21.440+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:21.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:21.436+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.437285+0000) 2019-10-13T06:15:21.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:21.861+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:21.861532+0000) 2019-10-13T06:15:22.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:22.727+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:22.727589+0000) 2019-10-13T06:15:22.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:22.728+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:22.728955+0000) 2019-10-13T06:15:24.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:24.222+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:24.222640+0000) 2019-10-13T06:15:25.256 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:15:25.256 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:25.260 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:15:25.260 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:26.446 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:26.442+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.442857+0000) 2019-10-13T06:15:26.446 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:26.442+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.443026+0000) 2019-10-13T06:15:26.447 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:26.442+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.443364+0000) 2019-10-13T06:15:26.447 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:26.439+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.439827+0000) 2019-10-13T06:15:26.448 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:26.443+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.443590+0000) 2019-10-13T06:15:26.448 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:26.443+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.443675+0000) 2019-10-13T06:15:26.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:26.443+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.444226+0000) 2019-10-13T06:15:26.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:26.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:26.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:26.442+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:26.442951+0000) 2019-10-13T06:15:28.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:28.268+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:28.268803+0000) 2019-10-13T06:15:28.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:28.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:28.608358+0000) 2019-10-13T06:15:28.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:28.608+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:28.608357+0000) 2019-10-13T06:15:30.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:30.967+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:30.968054+0000) 2019-10-13T06:15:31.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:31.448+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.448640+0000) 2019-10-13T06:15:31.453 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:31.448+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.448804+0000) 2019-10-13T06:15:31.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:31.448+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.449026+0000) 2019-10-13T06:15:31.454 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:31.444+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.445798+0000) 2019-10-13T06:15:31.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:31.449+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.449425+0000) 2019-10-13T06:15:31.455 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:31.449+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.449700+0000) 2019-10-13T06:15:31.455 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:31.449+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.450029+0000) 2019-10-13T06:15:31.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:31.445+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.447186+0000) 2019-10-13T06:15:31.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:31.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:31.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:31.861+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:31.861686+0000) 2019-10-13T06:15:32.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:32.727+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:32.727747+0000) 2019-10-13T06:15:32.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:32.728+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:32.729135+0000) 2019-10-13T06:15:34.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:34.222+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:34.222793+0000) 2019-10-13T06:15:36.457 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:36.452+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.453854+0000) 2019-10-13T06:15:36.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:36.452+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.454061+0000) 2019-10-13T06:15:36.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:36.452+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.454209+0000) 2019-10-13T06:15:36.459 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:36.453+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.454462+0000) 2019-10-13T06:15:36.460 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:36.448+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.450574+0000) 2019-10-13T06:15:36.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:36.450+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.452424+0000) 2019-10-13T06:15:36.460 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:36.453+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.455060+0000) 2019-10-13T06:15:36.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:36.453+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:36.455229+0000) 2019-10-13T06:15:36.461 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:36.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:38.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:38.267+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:38.268996+0000) 2019-10-13T06:15:38.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:38.607+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:38.608503+0000) 2019-10-13T06:15:38.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:38.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:38.608508+0000) 2019-10-13T06:15:40.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:40.969+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:40.970468+0000) 2019-10-13T06:15:41.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:41.453+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.455174+0000) 2019-10-13T06:15:41.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:41.457+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.459372+0000) 2019-10-13T06:15:41.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:41.458+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.459687+0000) 2019-10-13T06:15:41.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:41.458+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.459911+0000) 2019-10-13T06:15:41.464 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:41.458+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.459983+0000) 2019-10-13T06:15:41.464 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:41.458+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.460262+0000) 2019-10-13T06:15:41.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:41.459+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.460588+0000) 2019-10-13T06:15:41.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:41.456+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.458635+0000) 2019-10-13T06:15:41.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:41.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:41.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:41.860+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:41.861863+0000) 2019-10-13T06:15:42.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:42.726+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:42.727948+0000) 2019-10-13T06:15:42.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:42.728+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:42.729314+0000) 2019-10-13T06:15:44.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:44.222+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:44.222946+0000) 2019-10-13T06:15:46.468 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:46.463+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.464718+0000) 2019-10-13T06:15:46.468 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:46.463+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.464881+0000) 2019-10-13T06:15:46.469 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:46.460+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.461020+0000) 2019-10-13T06:15:46.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:46.464+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.465428+0000) 2019-10-13T06:15:46.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:46.464+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.465581+0000) 2019-10-13T06:15:46.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:46.464+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.465782+0000) 2019-10-13T06:15:46.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:46.464+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.466168+0000) 2019-10-13T06:15:46.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:46.462+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:46.463363+0000) 2019-10-13T06:15:46.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:46.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:48.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:48.268+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:48.269206+0000) 2019-10-13T06:15:48.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:48.607+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:48.608698+0000) 2019-10-13T06:15:48.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:48.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:48.608706+0000) 2019-10-13T06:15:50.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:50.972+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:50.972893+0000) 2019-10-13T06:15:51.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:51.468+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.470103+0000) 2019-10-13T06:15:51.474 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:51.469+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.470428+0000) 2019-10-13T06:15:51.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:51.465+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.466722+0000) 2019-10-13T06:15:51.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:51.469+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.471111+0000) 2019-10-13T06:15:51.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:51.469+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.471288+0000) 2019-10-13T06:15:51.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:51.469+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.471330+0000) 2019-10-13T06:15:51.475 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:51.470+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.471470+0000) 2019-10-13T06:15:51.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:51.467+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.468649+0000) 2019-10-13T06:15:51.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:51.472+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:51.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:51.861+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:51.862049+0000) 2019-10-13T06:15:52.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:52.727+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:52.728123+0000) 2019-10-13T06:15:52.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:52.728+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:52.729463+0000) 2019-10-13T06:15:54.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:54.222+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:54.223137+0000) 2019-10-13T06:15:55.354 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:15:55.354 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:55.358 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:15:55.358 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:56.481 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:15:56.474+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.475595+0000) 2019-10-13T06:15:56.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:56.474+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.475759+0000) 2019-10-13T06:15:56.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:56.473+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.475296+0000) 2019-10-13T06:15:56.482 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:15:56.474+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.475958+0000) 2019-10-13T06:15:56.482 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:56.475+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.476514+0000) 2019-10-13T06:15:56.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:15:56.475+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.476415+0000) 2019-10-13T06:15:56.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:15:56.471+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.472612+0000) 2019-10-13T06:15:56.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:15:56.473+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:56.474293+0000) 2019-10-13T06:15:56.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:15:56.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:15:58.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:15:58.268+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:58.269393+0000) 2019-10-13T06:15:58.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:15:58.608+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:58.608835+0000) 2019-10-13T06:15:58.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:15:58.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:15:58.608913+0000) 2019-10-13T06:16:00.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:00.974+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:00.975279+0000) 2019-10-13T06:16:01.484 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:01.476+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.477342+0000) 2019-10-13T06:16:01.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:01.480+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.481586+0000) 2019-10-13T06:16:01.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:01.480+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.481827+0000) 2019-10-13T06:16:01.486 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:01.480+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.482209+0000) 2019-10-13T06:16:01.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:01.480+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.482390+0000) 2019-10-13T06:16:01.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:01.481+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.482789+0000) 2019-10-13T06:16:01.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:01.481+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.483061+0000) 2019-10-13T06:16:01.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:01.479+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.480321+0000) 2019-10-13T06:16:01.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:01.484+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:01.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:01.861+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:01.862182+0000) 2019-10-13T06:16:02.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:02.727+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:02.728267+0000) 2019-10-13T06:16:02.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:02.729+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:02.729625+0000) 2019-10-13T06:16:04.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:04.222+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:04.223316+0000) 2019-10-13T06:16:06.491 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:06.482+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.483573+0000) 2019-10-13T06:16:06.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:06.487+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.487755+0000) 2019-10-13T06:16:06.492 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:06.487+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.487969+0000) 2019-10-13T06:16:06.492 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:06.487+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.488001+0000) 2019-10-13T06:16:06.492 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:06.487+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.488238+0000) 2019-10-13T06:16:06.493 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:06.488+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.488392+0000) 2019-10-13T06:16:06.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:06.488+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.489104+0000) 2019-10-13T06:16:06.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:06.486+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:06.487475+0000) 2019-10-13T06:16:06.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:06.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95708 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:08.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:08.269+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:08.269638+0000) 2019-10-13T06:16:08.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:08.608+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:08.609003+0000) 2019-10-13T06:16:08.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:08.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:08.609071+0000) 2019-10-13T06:16:10.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:10.977+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:10.977708+0000) 2019-10-13T06:16:11.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:11.492+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.493349+0000) 2019-10-13T06:16:11.497 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:11.493+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.493449+0000) 2019-10-13T06:16:11.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:11.493+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.493728+0000) 2019-10-13T06:16:11.498 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:11.493+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.493897+0000) 2019-10-13T06:16:11.498 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:11.488+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.489591+0000) 2019-10-13T06:16:11.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:11.493+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.494288+0000) 2019-10-13T06:16:11.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:11.494+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.494699+0000) 2019-10-13T06:16:11.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:11.491+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.492152+0000) 2019-10-13T06:16:11.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:11.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95739 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:11.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:11.862+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:11.862306+0000) 2019-10-13T06:16:12.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:12.728+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:12.728429+0000) 2019-10-13T06:16:12.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:12.729+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:12.729810+0000) 2019-10-13T06:16:14.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:14.223+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:14.223504+0000) 2019-10-13T06:16:16.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:16.498+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.498826+0000) 2019-10-13T06:16:16.502 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:16.498+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.499056+0000) 2019-10-13T06:16:16.503 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:16.494+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.495188+0000) 2019-10-13T06:16:16.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:16.499+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.499550+0000) 2019-10-13T06:16:16.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:16.499+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.499636+0000) 2019-10-13T06:16:16.504 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:16.499+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.499945+0000) 2019-10-13T06:16:16.504 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:16.499+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.499917+0000) 2019-10-13T06:16:16.505 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:16.497+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:16.498325+0000) 2019-10-13T06:16:16.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:16.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95770 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:18.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:18.269+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:18.269851+0000) 2019-10-13T06:16:18.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:18.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:18.609189+0000) 2019-10-13T06:16:18.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:18.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:18.609217+0000) 2019-10-13T06:16:20.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:20.979+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:20.980181+0000) 2019-10-13T06:16:21.507 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:21.504+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.504406+0000) 2019-10-13T06:16:21.508 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:21.498+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.500543+0000) 2019-10-13T06:16:21.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:21.504+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.505033+0000) 2019-10-13T06:16:21.509 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:21.504+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.505176+0000) 2019-10-13T06:16:21.509 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:21.505+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.505589+0000) 2019-10-13T06:16:21.509 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:21.505+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.505698+0000) 2019-10-13T06:16:21.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:21.505+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.505878+0000) 2019-10-13T06:16:21.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:21.501+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.502772+0000) 2019-10-13T06:16:21.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:21.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:21.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:21.861+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:21.862446+0000) 2019-10-13T06:16:22.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:22.727+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:22.728577+0000) 2019-10-13T06:16:22.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:22.728+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:22.729936+0000) 2019-10-13T06:16:24.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:24.222+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:24.223684+0000) 2019-10-13T06:16:25.460 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:16:25.461 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:25.465 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:16:25.465 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:26.513 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:26.508+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.510078+0000) 2019-10-13T06:16:26.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:26.504+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.506314+0000) 2019-10-13T06:16:26.514 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:26.509+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.510493+0000) 2019-10-13T06:16:26.515 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:26.509+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.510657+0000) 2019-10-13T06:16:26.515 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:26.509+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.510880+0000) 2019-10-13T06:16:26.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:26.509+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.511276+0000) 2019-10-13T06:16:26.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:26.509+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.511125+0000) 2019-10-13T06:16:26.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:26.508+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:26.509845+0000) 2019-10-13T06:16:26.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:26.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95832 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:28.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:28.269+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:28.270007+0000) 2019-10-13T06:16:28.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:28.608+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:28.609377+0000) 2019-10-13T06:16:28.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:28.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:28.609424+0000) 2019-10-13T06:16:30.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:30.981+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:30.982551+0000) 2019-10-13T06:16:31.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:31.510+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.512347+0000) 2019-10-13T06:16:31.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:31.514+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.515954+0000) 2019-10-13T06:16:31.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:31.515+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.516457+0000) 2019-10-13T06:16:31.521 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:31.515+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.516720+0000) 2019-10-13T06:16:31.521 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:31.515+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.516900+0000) 2019-10-13T06:16:31.521 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:31.515+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.517136+0000) 2019-10-13T06:16:31.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:31.515+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.517222+0000) 2019-10-13T06:16:31.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:31.512+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.514143+0000) 2019-10-13T06:16:31.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:31.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:31.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:31.861+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:31.862573+0000) 2019-10-13T06:16:32.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:32.727+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:32.728749+0000) 2019-10-13T06:16:32.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:32.729+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:32.730085+0000) 2019-10-13T06:16:34.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:34.223+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:34.223871+0000) 2019-10-13T06:16:36.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:36.519+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.521313+0000) 2019-10-13T06:16:36.525 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:36.520+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.521570+0000) 2019-10-13T06:16:36.525 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:36.520+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.521824+0000) 2019-10-13T06:16:36.526 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:36.520+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.521924+0000) 2019-10-13T06:16:36.526 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:36.517+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.518371+0000) 2019-10-13T06:16:36.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:36.521+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.522702+0000) 2019-10-13T06:16:36.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:36.521+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.522739+0000) 2019-10-13T06:16:36.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:36.519+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:36.520283+0000) 2019-10-13T06:16:36.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:36.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:38.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:38.269+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:38.270186+0000) 2019-10-13T06:16:38.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:38.608+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:38.609534+0000) 2019-10-13T06:16:38.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:38.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:38.609632+0000) 2019-10-13T06:16:40.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:40.984+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:40.984901+0000) 2019-10-13T06:16:41.530 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:41.525+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.526719+0000) 2019-10-13T06:16:41.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:41.525+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.527056+0000) 2019-10-13T06:16:41.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:41.525+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.527175+0000) 2019-10-13T06:16:41.531 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:41.522+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.522822+0000) 2019-10-13T06:16:41.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:41.526+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.527465+0000) 2019-10-13T06:16:41.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:41.526+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.527611+0000) 2019-10-13T06:16:41.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:41.526+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.527796+0000) 2019-10-13T06:16:41.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:41.523+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.524715+0000) 2019-10-13T06:16:41.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:41.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:41.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:41.862+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:41.862739+0000) 2019-10-13T06:16:42.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:42.728+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:42.728915+0000) 2019-10-13T06:16:42.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:42.729+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:42.730264+0000) 2019-10-13T06:16:44.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:44.222+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:44.224048+0000) 2019-10-13T06:16:46.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:46.526+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.527722+0000) 2019-10-13T06:16:46.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:46.530+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.531971+0000) 2019-10-13T06:16:46.536 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:46.530+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.532115+0000) 2019-10-13T06:16:46.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:46.531+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.532959+0000) 2019-10-13T06:16:46.537 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:46.531+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.533183+0000) 2019-10-13T06:16:46.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:46.531+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.533377+0000) 2019-10-13T06:16:46.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:46.532+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.533492+0000) 2019-10-13T06:16:46.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:46.531+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:46.531895+0000) 2019-10-13T06:16:46.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:46.534+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:48.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:48.269+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:48.270384+0000) 2019-10-13T06:16:48.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:48.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:48.609702+0000) 2019-10-13T06:16:48.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:48.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:48.609766+0000) 2019-10-13T06:16:50.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:50.986+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:50.987382+0000) 2019-10-13T06:16:51.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:51.532+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.533457+0000) 2019-10-13T06:16:51.542 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:51.537+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.537633+0000) 2019-10-13T06:16:51.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:51.537+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.537953+0000) 2019-10-13T06:16:51.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:51.537+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.538063+0000) 2019-10-13T06:16:51.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:51.537+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.538326+0000) 2019-10-13T06:16:51.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:51.538+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.538423+0000) 2019-10-13T06:16:51.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:51.538+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.538824+0000) 2019-10-13T06:16:51.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:51.536+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.536838+0000) 2019-10-13T06:16:51.544 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:51.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 95987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:51.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:51.862+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:51.862924+0000) 2019-10-13T06:16:52.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:52.728+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:52.729155+0000) 2019-10-13T06:16:52.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:52.729+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:52.730455+0000) 2019-10-13T06:16:54.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:54.223+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:54.224208+0000) 2019-10-13T06:16:55.574 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:16:55.574 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:55.578 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:16:55.578 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:56.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:16:56.542+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.542635+0000) 2019-10-13T06:16:56.547 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:16:56.538+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.538948+0000) 2019-10-13T06:16:56.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:16:56.542+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.543115+0000) 2019-10-13T06:16:56.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:56.542+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.543310+0000) 2019-10-13T06:16:56.548 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:16:56.543+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.543504+0000) 2019-10-13T06:16:56.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:56.543+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.543604+0000) 2019-10-13T06:16:56.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:56.543+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.543718+0000) 2019-10-13T06:16:56.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:16:56.540+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:56.540953+0000) 2019-10-13T06:16:56.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:16:56.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:16:58.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:16:58.270+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:58.270594+0000) 2019-10-13T06:16:58.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:16:58.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:58.609896+0000) 2019-10-13T06:16:58.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:16:58.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:16:58.609991+0000) 2019-10-13T06:17:00.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:00.989+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:00.989745+0000) 2019-10-13T06:17:01.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:01.543+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.543860+0000) 2019-10-13T06:17:01.552 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:01.546+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.547247+0000) 2019-10-13T06:17:01.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:01.547+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.547403+0000) 2019-10-13T06:17:01.553 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:01.547+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.547533+0000) 2019-10-13T06:17:01.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:01.547+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.547896+0000) 2019-10-13T06:17:01.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:01.547+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.548349+0000) 2019-10-13T06:17:01.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:01.548+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.548530+0000) 2019-10-13T06:17:01.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:01.545+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.546369+0000) 2019-10-13T06:17:01.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:01.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:01.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:01.862+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:01.863069+0000) 2019-10-13T06:17:02.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:02.728+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:02.729367+0000) 2019-10-13T06:17:02.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:02.730+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:02.730606+0000) 2019-10-13T06:17:04.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:04.224+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:04.224366+0000) 2019-10-13T06:17:06.562 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:06.549+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.549851+0000) 2019-10-13T06:17:06.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:06.551+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.552560+0000) 2019-10-13T06:17:06.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:06.552+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.553315+0000) 2019-10-13T06:17:06.563 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:06.553+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.553517+0000) 2019-10-13T06:17:06.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:06.553+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.553784+0000) 2019-10-13T06:17:06.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:06.553+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.554223+0000) 2019-10-13T06:17:06.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:06.554+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.554586+0000) 2019-10-13T06:17:06.564 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:06.554+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:06.554799+0000) 2019-10-13T06:17:06.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:06.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:08.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:08.269+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:08.270803+0000) 2019-10-13T06:17:08.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:08.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:08.610061+0000) 2019-10-13T06:17:08.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:08.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:08.610163+0000) 2019-10-13T06:17:10.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:10.991+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:10.991989+0000) 2019-10-13T06:17:11.563 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:11.558+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.560069+0000) 2019-10-13T06:17:11.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:11.558+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.560295+0000) 2019-10-13T06:17:11.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:11.556+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.557209+0000) 2019-10-13T06:17:11.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:11.559+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.560740+0000) 2019-10-13T06:17:11.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:11.559+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.560586+0000) 2019-10-13T06:17:11.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:11.559+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.560835+0000) 2019-10-13T06:17:11.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:11.559+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.561090+0000) 2019-10-13T06:17:11.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:11.557+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.558157+0000) 2019-10-13T06:17:11.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:11.563+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:11.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:11.861+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:11.863262+0000) 2019-10-13T06:17:12.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:12.728+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:12.729553+0000) 2019-10-13T06:17:12.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:12.729+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:12.730812+0000) 2019-10-13T06:17:14.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:14.223+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:14.224575+0000) 2019-10-13T06:17:16.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:16.560+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.562035+0000) 2019-10-13T06:17:16.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:16.564+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.566337+0000) 2019-10-13T06:17:16.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:16.565+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.566534+0000) 2019-10-13T06:17:16.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:16.565+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.566571+0000) 2019-10-13T06:17:16.571 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:16.565+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.566670+0000) 2019-10-13T06:17:16.571 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:16.565+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.566824+0000) 2019-10-13T06:17:16.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:16.565+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.567070+0000) 2019-10-13T06:17:16.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:16.563+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:16.565000+0000) 2019-10-13T06:17:16.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:16.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:18.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:18.270+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:18.271052+0000) 2019-10-13T06:17:18.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:18.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:18.610204+0000) 2019-10-13T06:17:18.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:18.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:18.610283+0000) 2019-10-13T06:17:21.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:20.992+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:20.994287+0000) 2019-10-13T06:17:21.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:21.565+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.566818+0000) 2019-10-13T06:17:21.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:21.569+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.571208+0000) 2019-10-13T06:17:21.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:21.570+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.571755+0000) 2019-10-13T06:17:21.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:21.570+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.571636+0000) 2019-10-13T06:17:21.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:21.570+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.571887+0000) 2019-10-13T06:17:21.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:21.570+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.572014+0000) 2019-10-13T06:17:21.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:21.567+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.569151+0000) 2019-10-13T06:17:21.576 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:21.571+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.572462+0000) 2019-10-13T06:17:21.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:21.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96174 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:21.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:21.862+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:21.863430+0000) 2019-10-13T06:17:22.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:22.729+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:22.729705+0000) 2019-10-13T06:17:22.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:22.729+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:22.730973+0000) 2019-10-13T06:17:24.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:24.224+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:24.224767+0000) 2019-10-13T06:17:25.689 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:17:25.689 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:25.697 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:17:25.697 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:26.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:26.575+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.576946+0000) 2019-10-13T06:17:26.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:26.575+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.577192+0000) 2019-10-13T06:17:26.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:26.575+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.577277+0000) 2019-10-13T06:17:26.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:26.576+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.577479+0000) 2019-10-13T06:17:26.581 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:26.576+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.577543+0000) 2019-10-13T06:17:26.582 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:26.573+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.574194+0000) 2019-10-13T06:17:26.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:26.576+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.578032+0000) 2019-10-13T06:17:26.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:26.575+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:26.575947+0000) 2019-10-13T06:17:26.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:26.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96204 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:28.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:28.270+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:28.271243+0000) 2019-10-13T06:17:28.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:28.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:28.610376+0000) 2019-10-13T06:17:28.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:28.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:28.610450+0000) 2019-10-13T06:17:31.004 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:30.996+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:30.996833+0000) 2019-10-13T06:17:31.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:31.580+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.582192+0000) 2019-10-13T06:17:31.586 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:31.581+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.582457+0000) 2019-10-13T06:17:31.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:31.581+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.582840+0000) 2019-10-13T06:17:31.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:31.581+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.583119+0000) 2019-10-13T06:17:31.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:31.581+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.583305+0000) 2019-10-13T06:17:31.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:31.582+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.583548+0000) 2019-10-13T06:17:31.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:31.578+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.579372+0000) 2019-10-13T06:17:31.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:31.579+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.580706+0000) 2019-10-13T06:17:31.589 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:31.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:31.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:31.863+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:31.863600+0000) 2019-10-13T06:17:32.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:32.729+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:32.729861+0000) 2019-10-13T06:17:32.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:32.730+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:32.731157+0000) 2019-10-13T06:17:34.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:34.224+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:34.224932+0000) 2019-10-13T06:17:36.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:36.583+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.584282+0000) 2019-10-13T06:17:36.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:36.587+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.587992+0000) 2019-10-13T06:17:36.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:36.587+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.587899+0000) 2019-10-13T06:17:36.594 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:36.588+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.588450+0000) 2019-10-13T06:17:36.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:36.588+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.588989+0000) 2019-10-13T06:17:36.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:36.588+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.589050+0000) 2019-10-13T06:17:36.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:36.588+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.589196+0000) 2019-10-13T06:17:36.596 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:36.585+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:36.586709+0000) 2019-10-13T06:17:36.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:36.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:38.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:38.271+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:38.271435+0000) 2019-10-13T06:17:38.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:38.610+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:38.610556+0000) 2019-10-13T06:17:38.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:38.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:38.610618+0000) 2019-10-13T06:17:41.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:40.998+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:40.999304+0000) 2019-10-13T06:17:41.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:41.593+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.594021+0000) 2019-10-13T06:17:41.597 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:41.593+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.594373+0000) 2019-10-13T06:17:41.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:41.589+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.590256+0000) 2019-10-13T06:17:41.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:41.594+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.594635+0000) 2019-10-13T06:17:41.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:41.594+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.594966+0000) 2019-10-13T06:17:41.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:41.594+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.594666+0000) 2019-10-13T06:17:41.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:41.594+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.595069+0000) 2019-10-13T06:17:41.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:41.592+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.592801+0000) 2019-10-13T06:17:41.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:41.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:41.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:41.863+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:41.863734+0000) 2019-10-13T06:17:42.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:42.729+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:42.729982+0000) 2019-10-13T06:17:42.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:42.730+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:42.731335+0000) 2019-10-13T06:17:44.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:44.224+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:44.225080+0000) 2019-10-13T06:17:46.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:46.595+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.595742+0000) 2019-10-13T06:17:46.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:46.599+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.599925+0000) 2019-10-13T06:17:46.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:46.600+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.600409+0000) 2019-10-13T06:17:46.604 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:46.600+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.600540+0000) 2019-10-13T06:17:46.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:46.600+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.600673+0000) 2019-10-13T06:17:46.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:46.600+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.600804+0000) 2019-10-13T06:17:46.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:46.600+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.601034+0000) 2019-10-13T06:17:46.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:46.597+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:46.598309+0000) 2019-10-13T06:17:46.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:46.603+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:48.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:48.271+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:48.271625+0000) 2019-10-13T06:17:48.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:48.610+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:48.610714+0000) 2019-10-13T06:17:48.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:48.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:48.610763+0000) 2019-10-13T06:17:51.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:51.001+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.001864+0000) 2019-10-13T06:17:51.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:51.604+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.605245+0000) 2019-10-13T06:17:51.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:51.604+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.605359+0000) 2019-10-13T06:17:51.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:51.605+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.605485+0000) 2019-10-13T06:17:51.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:51.601+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.602359+0000) 2019-10-13T06:17:51.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:51.605+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.606047+0000) 2019-10-13T06:17:51.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:51.606+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.606988+0000) 2019-10-13T06:17:51.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:51.606+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.607148+0000) 2019-10-13T06:17:51.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:51.603+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.603878+0000) 2019-10-13T06:17:51.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:51.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96359 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:51.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:51.863+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:51.863902+0000) 2019-10-13T06:17:52.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:52.730+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:52.730148+0000) 2019-10-13T06:17:52.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:52.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:52.731515+0000) 2019-10-13T06:17:54.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:54.223+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:54.225249+0000) 2019-10-13T06:17:55.996 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:17:55.996 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:56.000 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:17:56.001 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:56.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:56.608+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.609984+0000) 2019-10-13T06:17:56.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:56.608+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.610210+0000) 2019-10-13T06:17:56.614 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:17:56.609+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.610523+0000) 2019-10-13T06:17:56.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:17:56.605+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.606712+0000) 2019-10-13T06:17:56.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:56.609+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.610992+0000) 2019-10-13T06:17:56.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:17:56.609+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.611198+0000) 2019-10-13T06:17:56.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:17:56.609+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.611269+0000) 2019-10-13T06:17:56.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:17:56.608+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:56.609229+0000) 2019-10-13T06:17:56.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:17:56.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:17:58.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:17:58.270+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:58.271819+0000) 2019-10-13T06:17:58.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:17:58.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:58.610857+0000) 2019-10-13T06:17:58.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:17:58.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:17:58.610872+0000) 2019-10-13T06:18:01.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:01.003+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.004234+0000) 2019-10-13T06:18:01.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:01.614+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.615700+0000) 2019-10-13T06:18:01.619 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:01.614+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.615981+0000) 2019-10-13T06:18:01.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:01.614+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.616140+0000) 2019-10-13T06:18:01.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:01.611+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.611779+0000) 2019-10-13T06:18:01.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:01.615+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.616456+0000) 2019-10-13T06:18:01.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:01.615+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.616668+0000) 2019-10-13T06:18:01.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:01.615+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.616858+0000) 2019-10-13T06:18:01.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:01.613+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.614173+0000) 2019-10-13T06:18:01.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:01.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:01.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:01.863+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:01.864087+0000) 2019-10-13T06:18:02.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:02.729+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:02.730341+0000) 2019-10-13T06:18:02.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:02.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:02.731694+0000) 2019-10-13T06:18:04.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:04.224+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:04.225455+0000) 2019-10-13T06:18:06.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:06.615+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.616828+0000) 2019-10-13T06:18:06.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:06.619+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.620668+0000) 2019-10-13T06:18:06.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:06.619+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.621008+0000) 2019-10-13T06:18:06.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:06.619+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.621172+0000) 2019-10-13T06:18:06.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:06.620+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.621484+0000) 2019-10-13T06:18:06.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:06.620+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.621741+0000) 2019-10-13T06:18:06.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:06.620+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.622044+0000) 2019-10-13T06:18:06.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:06.618+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:06.620022+0000) 2019-10-13T06:18:06.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:06.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:08.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:08.271+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:08.272014+0000) 2019-10-13T06:18:08.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:08.609+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:08.611003+0000) 2019-10-13T06:18:08.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:08.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:08.611041+0000) 2019-10-13T06:18:11.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:11.005+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.006553+0000) 2019-10-13T06:18:11.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:11.624+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.625946+0000) 2019-10-13T06:18:11.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:11.624+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.626266+0000) 2019-10-13T06:18:11.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:11.620+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.622172+0000) 2019-10-13T06:18:11.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:11.625+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.626617+0000) 2019-10-13T06:18:11.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:11.625+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.626749+0000) 2019-10-13T06:18:11.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:11.625+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.627084+0000) 2019-10-13T06:18:11.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:11.625+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.627224+0000) 2019-10-13T06:18:11.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:11.623+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.625253+0000) 2019-10-13T06:18:11.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:11.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:11.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:11.863+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:11.864269+0000) 2019-10-13T06:18:12.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:12.729+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:12.730494+0000) 2019-10-13T06:18:12.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:12.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:12.731905+0000) 2019-10-13T06:18:14.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:14.225+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:14.225638+0000) 2019-10-13T06:18:16.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:16.629+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.630769+0000) 2019-10-13T06:18:16.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:16.629+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.630973+0000) 2019-10-13T06:18:16.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:16.630+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.631426+0000) 2019-10-13T06:18:16.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:16.630+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.631615+0000) 2019-10-13T06:18:16.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:16.630+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.631717+0000) 2019-10-13T06:18:16.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:16.625+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.627255+0000) 2019-10-13T06:18:16.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:16.631+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.632454+0000) 2019-10-13T06:18:16.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:16.628+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:16.630407+0000) 2019-10-13T06:18:16.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:16.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:18.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:18.271+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:18.272183+0000) 2019-10-13T06:18:18.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:18.610+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:18.611198+0000) 2019-10-13T06:18:18.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:18.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:18.611201+0000) 2019-10-13T06:18:21.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:21.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.008822+0000) 2019-10-13T06:18:21.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:21.631+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.632648+0000) 2019-10-13T06:18:21.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:21.636+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.636474+0000) 2019-10-13T06:18:21.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:21.636+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.636772+0000) 2019-10-13T06:18:21.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:21.636+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.637000+0000) 2019-10-13T06:18:21.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:21.636+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.637324+0000) 2019-10-13T06:18:21.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:21.637+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.637683+0000) 2019-10-13T06:18:21.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:21.637+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.637889+0000) 2019-10-13T06:18:21.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:21.634+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.635190+0000) 2019-10-13T06:18:21.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:21.640+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:21.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:21.864+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:21.864457+0000) 2019-10-13T06:18:22.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:22.730+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:22.730646+0000) 2019-10-13T06:18:22.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:22.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:22.732092+0000) 2019-10-13T06:18:24.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:24.225+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:24.225829+0000) 2019-10-13T06:18:26.174 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:18:26.174 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:26.179 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:18:26.179 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:26.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:26.642+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.642498+0000) 2019-10-13T06:18:26.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:26.642+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.642637+0000) 2019-10-13T06:18:26.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:26.638+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.639365+0000) 2019-10-13T06:18:26.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:26.642+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.643199+0000) 2019-10-13T06:18:26.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:26.643+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.643482+0000) 2019-10-13T06:18:26.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:26.643+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.643592+0000) 2019-10-13T06:18:26.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:26.643+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.643917+0000) 2019-10-13T06:18:26.649 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:26.641+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:26.641879+0000) 2019-10-13T06:18:26.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:26.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96576 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:28.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:28.271+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:28.272379+0000) 2019-10-13T06:18:28.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:28.611+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:28.611421+0000) 2019-10-13T06:18:28.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:28.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:28.611433+0000) 2019-10-13T06:18:31.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:31.010+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.011281+0000) 2019-10-13T06:18:31.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:31.648+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.648728+0000) 2019-10-13T06:18:31.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:31.648+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.648848+0000) 2019-10-13T06:18:31.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:31.648+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.649018+0000) 2019-10-13T06:18:31.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:31.644+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.645192+0000) 2019-10-13T06:18:31.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:31.648+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.649313+0000) 2019-10-13T06:18:31.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:31.649+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.649651+0000) 2019-10-13T06:18:31.654 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:31.649+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.649730+0000) 2019-10-13T06:18:31.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:31.647+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.648045+0000) 2019-10-13T06:18:31.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:31.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:31.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:31.864+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:31.864625+0000) 2019-10-13T06:18:32.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:32.730+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:32.730837+0000) 2019-10-13T06:18:32.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:32.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:32.732273+0000) 2019-10-13T06:18:34.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:34.225+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:34.226020+0000) 2019-10-13T06:18:36.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:36.654+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.655220+0000) 2019-10-13T06:18:36.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:36.654+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.655355+0000) 2019-10-13T06:18:36.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:36.655+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.655642+0000) 2019-10-13T06:18:36.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:36.651+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.651959+0000) 2019-10-13T06:18:36.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:36.655+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.656078+0000) 2019-10-13T06:18:36.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:36.655+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.656222+0000) 2019-10-13T06:18:36.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:36.656+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.656447+0000) 2019-10-13T06:18:36.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:36.652+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:36.653277+0000) 2019-10-13T06:18:36.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:36.658+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:38.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:38.272+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:38.272588+0000) 2019-10-13T06:18:38.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:38.611+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:38.611556+0000) 2019-10-13T06:18:38.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:38.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:38.611568+0000) 2019-10-13T06:18:41.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:41.013+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.013806+0000) 2019-10-13T06:18:41.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:41.655+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.656245+0000) 2019-10-13T06:18:41.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:41.659+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.660424+0000) 2019-10-13T06:18:41.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:41.659+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.660510+0000) 2019-10-13T06:18:41.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:41.659+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.660622+0000) 2019-10-13T06:18:41.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:41.659+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.660815+0000) 2019-10-13T06:18:41.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:41.659+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.660859+0000) 2019-10-13T06:18:41.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:41.659+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.661201+0000) 2019-10-13T06:18:41.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:41.657+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.658477+0000) 2019-10-13T06:18:41.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:41.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:41.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:41.863+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:41.864788+0000) 2019-10-13T06:18:42.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:42.730+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:42.731022+0000) 2019-10-13T06:18:42.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:42.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:42.732434+0000) 2019-10-13T06:18:44.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:44.225+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:44.226209+0000) 2019-10-13T06:18:46.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:46.660+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.660823+0000) 2019-10-13T06:18:46.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:46.663+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.665062+0000) 2019-10-13T06:18:46.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:46.663+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.665336+0000) 2019-10-13T06:18:46.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:46.663+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.665225+0000) 2019-10-13T06:18:46.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:46.664+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.665477+0000) 2019-10-13T06:18:46.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:46.664+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.665750+0000) 2019-10-13T06:18:46.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:46.664+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.666188+0000) 2019-10-13T06:18:46.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:46.663+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:46.664387+0000) 2019-10-13T06:18:46.672 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:46.667+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:48.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:48.271+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:48.272780+0000) 2019-10-13T06:18:48.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:48.610+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:48.611728+0000) 2019-10-13T06:18:48.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:48.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:48.611750+0000) 2019-10-13T06:18:51.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:51.015+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.016070+0000) 2019-10-13T06:18:51.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:51.665+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.666676+0000) 2019-10-13T06:18:51.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:51.669+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.670913+0000) 2019-10-13T06:18:51.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:51.669+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.671201+0000) 2019-10-13T06:18:51.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:51.669+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.671339+0000) 2019-10-13T06:18:51.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:51.670+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.671690+0000) 2019-10-13T06:18:51.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:51.670+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.671826+0000) 2019-10-13T06:18:51.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:51.670+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.672114+0000) 2019-10-13T06:18:51.677 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:51.669+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.670084+0000) 2019-10-13T06:18:51.677 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:51.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:51.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:51.864+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:51.864969+0000) 2019-10-13T06:18:52.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:52.729+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:52.731250+0000) 2019-10-13T06:18:52.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:52.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:52.732561+0000) 2019-10-13T06:18:54.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:54.225+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:54.226395+0000) 2019-10-13T06:18:56.341 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:18:56.341 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:56.346 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:18:56.346 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:56.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:18:56.674+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.676119+0000) 2019-10-13T06:18:56.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:56.675+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.676633+0000) 2019-10-13T06:18:56.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:18:56.670+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.672551+0000) 2019-10-13T06:18:56.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:18:56.675+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.676827+0000) 2019-10-13T06:18:56.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:56.675+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.676958+0000) 2019-10-13T06:18:56.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:56.675+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.677332+0000) 2019-10-13T06:18:56.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:18:56.676+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.677680+0000) 2019-10-13T06:18:56.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:18:56.672+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:56.674410+0000) 2019-10-13T06:18:56.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:18:56.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:18:58.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:18:58.272+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:58.272976+0000) 2019-10-13T06:18:58.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:18:58.611+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:58.611911+0000) 2019-10-13T06:18:58.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:18:58.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:18:58.611912+0000) 2019-10-13T06:19:01.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:01.017+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.018344+0000) 2019-10-13T06:19:01.686 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:01.680+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.682146+0000) 2019-10-13T06:19:01.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:01.680+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.682303+0000) 2019-10-13T06:19:01.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:01.681+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.682507+0000) 2019-10-13T06:19:01.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:01.681+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.682591+0000) 2019-10-13T06:19:01.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:01.681+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.682918+0000) 2019-10-13T06:19:01.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:01.681+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.683021+0000) 2019-10-13T06:19:01.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:01.676+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.678081+0000) 2019-10-13T06:19:01.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:01.683+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96794 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:01.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:01.679+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.680944+0000) 2019-10-13T06:19:01.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:01.864+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:01.865128+0000) 2019-10-13T06:19:02.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:02.730+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:02.731444+0000) 2019-10-13T06:19:02.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:02.731+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:02.732778+0000) 2019-10-13T06:19:04.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:04.226+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:04.226562+0000) 2019-10-13T06:19:06.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:06.686+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.686427+0000) 2019-10-13T06:19:06.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:06.686+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.686767+0000) 2019-10-13T06:19:06.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:06.680+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.682655+0000) 2019-10-13T06:19:06.691 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:06.686+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.687095+0000) 2019-10-13T06:19:06.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:06.686+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.687283+0000) 2019-10-13T06:19:06.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:06.686+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.687334+0000) 2019-10-13T06:19:06.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:06.687+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.687583+0000) 2019-10-13T06:19:06.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:06.683+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:06.684961+0000) 2019-10-13T06:19:06.693 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:06.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:08.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:08.272+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:08.273184+0000) 2019-10-13T06:19:08.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:08.611+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:08.612093+0000) 2019-10-13T06:19:08.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:08.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:08.612094+0000) 2019-10-13T06:19:11.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:11.019+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.020613+0000) 2019-10-13T06:19:11.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:11.691+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.691567+0000) 2019-10-13T06:19:11.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:11.691+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.691730+0000) 2019-10-13T06:19:11.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:11.687+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.688209+0000) 2019-10-13T06:19:11.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:11.692+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.692527+0000) 2019-10-13T06:19:11.696 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:11.692+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.692843+0000) 2019-10-13T06:19:11.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:11.692+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.692878+0000) 2019-10-13T06:19:11.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:11.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.692369+0000) 2019-10-13T06:19:11.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:11.689+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.690090+0000) 2019-10-13T06:19:11.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:11.695+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96855 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:11.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:11.864+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:11.865262+0000) 2019-10-13T06:19:12.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:12.731+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:12.731652+0000) 2019-10-13T06:19:12.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:12.732+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:12.732951+0000) 2019-10-13T06:19:14.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:14.226+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:14.226746+0000) 2019-10-13T06:19:16.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:16.697+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.697831+0000) 2019-10-13T06:19:16.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:16.697+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.698044+0000) 2019-10-13T06:19:16.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:16.697+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.698296+0000) 2019-10-13T06:19:16.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:16.698+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.698513+0000) 2019-10-13T06:19:16.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:16.694+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.694988+0000) 2019-10-13T06:19:16.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:16.698+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.698677+0000) 2019-10-13T06:19:16.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:16.698+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.698815+0000) 2019-10-13T06:19:16.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:16.695+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:16.696559+0000) 2019-10-13T06:19:16.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:16.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:18.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:18.272+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:18.273377+0000) 2019-10-13T06:19:18.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:18.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:18.612219+0000) 2019-10-13T06:19:18.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:18.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:18.612276+0000) 2019-10-13T06:19:21.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:21.022+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.022922+0000) 2019-10-13T06:19:21.707 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:21.702+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.703296+0000) 2019-10-13T06:19:21.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:21.703+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.703527+0000) 2019-10-13T06:19:21.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:21.703+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.703823+0000) 2019-10-13T06:19:21.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:21.699+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.700140+0000) 2019-10-13T06:19:21.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:21.703+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.703915+0000) 2019-10-13T06:19:21.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:21.704+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.704428+0000) 2019-10-13T06:19:21.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:21.704+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.704589+0000) 2019-10-13T06:19:21.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:21.700+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.701603+0000) 2019-10-13T06:19:21.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:21.706+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:21.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:21.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:21.865426+0000) 2019-10-13T06:19:22.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:22.731+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:22.731843+0000) 2019-10-13T06:19:22.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:22.732+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:22.733108+0000) 2019-10-13T06:19:24.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:24.226+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:24.226941+0000) 2019-10-13T06:19:26.521 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:19:26.521 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:26.526 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:19:26.527 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:26.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:26.706+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.708231+0000) 2019-10-13T06:19:26.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:26.707+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.708450+0000) 2019-10-13T06:19:26.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:26.707+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.708829+0000) 2019-10-13T06:19:26.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:26.707+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.709030+0000) 2019-10-13T06:19:26.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:26.704+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.705297+0000) 2019-10-13T06:19:26.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:26.706+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.707387+0000) 2019-10-13T06:19:26.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:26.707+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.708691+0000) 2019-10-13T06:19:26.716 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:26.708+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:26.709459+0000) 2019-10-13T06:19:26.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:26.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96948 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:28.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:28.272+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:28.273609+0000) 2019-10-13T06:19:28.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:28.611+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:28.612421+0000) 2019-10-13T06:19:28.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:28.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:28.612371+0000) 2019-10-13T06:19:31.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:31.024+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.025328+0000) 2019-10-13T06:19:31.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:31.708+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.709444+0000) 2019-10-13T06:19:31.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:31.712+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.714093+0000) 2019-10-13T06:19:31.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:31.712+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.714253+0000) 2019-10-13T06:19:31.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:31.713+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.714516+0000) 2019-10-13T06:19:31.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:31.713+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.714657+0000) 2019-10-13T06:19:31.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:31.713+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.714814+0000) 2019-10-13T06:19:31.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:31.713+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.715194+0000) 2019-10-13T06:19:31.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:31.710+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.711627+0000) 2019-10-13T06:19:31.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:31.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 96979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:31.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:31.864+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:31.865591+0000) 2019-10-13T06:19:32.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:32.731+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:32.732030+0000) 2019-10-13T06:19:32.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:32.732+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:32.733289+0000) 2019-10-13T06:19:34.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:34.226+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:34.227131+0000) 2019-10-13T06:19:36.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:36.717+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.718946+0000) 2019-10-13T06:19:36.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:36.717+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.719348+0000) 2019-10-13T06:19:36.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:36.718+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.719558+0000) 2019-10-13T06:19:36.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:36.718+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.719791+0000) 2019-10-13T06:19:36.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:36.718+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.720098+0000) 2019-10-13T06:19:36.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:36.715+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.715991+0000) 2019-10-13T06:19:36.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:36.718+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.720248+0000) 2019-10-13T06:19:36.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:36.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97011 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:36.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:36.717+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:36.718599+0000) 2019-10-13T06:19:38.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:38.273+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:38.273810+0000) 2019-10-13T06:19:38.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:38.611+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:38.612581+0000) 2019-10-13T06:19:38.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:38.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:38.612556+0000) 2019-10-13T06:19:41.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:41.027+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.027742+0000) 2019-10-13T06:19:41.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:41.720+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.720848+0000) 2019-10-13T06:19:41.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:41.722+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.723702+0000) 2019-10-13T06:19:41.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:41.722+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.723829+0000) 2019-10-13T06:19:41.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:41.722+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.724107+0000) 2019-10-13T06:19:41.730 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:41.722+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.724299+0000) 2019-10-13T06:19:41.730 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:41.723+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.724453+0000) 2019-10-13T06:19:41.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:41.723+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.724599+0000) 2019-10-13T06:19:41.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:41.722+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.722973+0000) 2019-10-13T06:19:41.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:41.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97042 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:41.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:41.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:41.865753+0000) 2019-10-13T06:19:42.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:42.731+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:42.732213+0000) 2019-10-13T06:19:42.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:42.733+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:42.733477+0000) 2019-10-13T06:19:44.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:44.225+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:44.227340+0000) 2019-10-13T06:19:46.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:46.723+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.724535+0000) 2019-10-13T06:19:46.733 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:46.727+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.728726+0000) 2019-10-13T06:19:46.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:46.727+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.729166+0000) 2019-10-13T06:19:46.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:46.727+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.729001+0000) 2019-10-13T06:19:46.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:46.727+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.729349+0000) 2019-10-13T06:19:46.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:46.728+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.729601+0000) 2019-10-13T06:19:46.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:46.727+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.728841+0000) 2019-10-13T06:19:46.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:46.727+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:46.727896+0000) 2019-10-13T06:19:46.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:46.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97072 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:48.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:48.273+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:48.274002+0000) 2019-10-13T06:19:48.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:48.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:48.612697+0000) 2019-10-13T06:19:48.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:48.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:48.612780+0000) 2019-10-13T06:19:51.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:51.029+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.030109+0000) 2019-10-13T06:19:51.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:51.732+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.733922+0000) 2019-10-13T06:19:51.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:51.728+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.730552+0000) 2019-10-13T06:19:51.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:51.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.735010+0000) 2019-10-13T06:19:51.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:51.732+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.733997+0000) 2019-10-13T06:19:51.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:51.733+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.735192+0000) 2019-10-13T06:19:51.740 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:51.733+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.734612+0000) 2019-10-13T06:19:51.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:51.732+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.734205+0000) 2019-10-13T06:19:51.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:51.730+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.732550+0000) 2019-10-13T06:19:51.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:51.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:51.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:51.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:51.865921+0000) 2019-10-13T06:19:52.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:52.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:52.732418+0000) 2019-10-13T06:19:52.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:52.732+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:52.733679+0000) 2019-10-13T06:19:54.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:54.227+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:54.227536+0000) 2019-10-13T06:19:56.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:19:56.734+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.735826+0000) 2019-10-13T06:19:56.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:19:56.739+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.740163+0000) 2019-10-13T06:19:56.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:56.739+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.740293+0000) 2019-10-13T06:19:56.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:19:56.740+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.740601+0000) 2019-10-13T06:19:56.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:19:56.740+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.740822+0000) 2019-10-13T06:19:56.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:56.740+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.740882+0000) 2019-10-13T06:19:56.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:56.740+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.741100+0000) 2019-10-13T06:19:56.746 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:19:56.746 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:56.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:19:56.736+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:56.738093+0000) 2019-10-13T06:19:56.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:19:56.743+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:19:56.752 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:19:56.752 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:58.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:19:58.273+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:58.274222+0000) 2019-10-13T06:19:58.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:19:58.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:58.612807+0000) 2019-10-13T06:19:58.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:19:58.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:19:58.612879+0000) 2019-10-13T06:20:01.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:01.031+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.032602+0000) 2019-10-13T06:20:01.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:01.745+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.745842+0000) 2019-10-13T06:20:01.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:01.745+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.746001+0000) 2019-10-13T06:20:01.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:01.740+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.742359+0000) 2019-10-13T06:20:01.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:01.745+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.746343+0000) 2019-10-13T06:20:01.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:01.746+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.746564+0000) 2019-10-13T06:20:01.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:01.746+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.746846+0000) 2019-10-13T06:20:01.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:01.746+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.747193+0000) 2019-10-13T06:20:01.752 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:01.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:01.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:01.743+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.745650+0000) 2019-10-13T06:20:01.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:01.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:01.866087+0000) 2019-10-13T06:20:02.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:02.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:02.732574+0000) 2019-10-13T06:20:02.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:02.733+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:02.733860+0000) 2019-10-13T06:20:04.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:04.227+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:04.227729+0000) 2019-10-13T06:20:06.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:06.751+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.751574+0000) 2019-10-13T06:20:06.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:06.751+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.751764+0000) 2019-10-13T06:20:06.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:06.751+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.752006+0000) 2019-10-13T06:20:06.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:06.747+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.748194+0000) 2019-10-13T06:20:06.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:06.751+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.752356+0000) 2019-10-13T06:20:06.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:06.752+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.752628+0000) 2019-10-13T06:20:06.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:06.752+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.752950+0000) 2019-10-13T06:20:06.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:06.749+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:06.749964+0000) 2019-10-13T06:20:06.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:06.755+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:08.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:08.274+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:08.274429+0000) 2019-10-13T06:20:08.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:08.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:08.612978+0000) 2019-10-13T06:20:08.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:08.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:08.613059+0000) 2019-10-13T06:20:11.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:11.034+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.035122+0000) 2019-10-13T06:20:11.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:11.755+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.757370+0000) 2019-10-13T06:20:11.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:11.756+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.757482+0000) 2019-10-13T06:20:11.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:11.753+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.754589+0000) 2019-10-13T06:20:11.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:11.756+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.758019+0000) 2019-10-13T06:20:11.762 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:11.756+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.757775+0000) 2019-10-13T06:20:11.762 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:11.756+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.758167+0000) 2019-10-13T06:20:11.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:11.757+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.758824+0000) 2019-10-13T06:20:11.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:11.760+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:11.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:11.756+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.757297+0000) 2019-10-13T06:20:11.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:11.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:11.866257+0000) 2019-10-13T06:20:12.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:12.731+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:12.732701+0000) 2019-10-13T06:20:12.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:12.733+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:12.734025+0000) 2019-10-13T06:20:14.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:14.226+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:14.227922+0000) 2019-10-13T06:20:16.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:16.761+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.762963+0000) 2019-10-13T06:20:16.767 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:16.761+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.763203+0000) 2019-10-13T06:20:16.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:16.761+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.763363+0000) 2019-10-13T06:20:16.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:16.759+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.760015+0000) 2019-10-13T06:20:16.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:16.762+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.763492+0000) 2019-10-13T06:20:16.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:16.762+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.763683+0000) 2019-10-13T06:20:16.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:16.762+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.764230+0000) 2019-10-13T06:20:16.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:16.765+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97259 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:16.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:16.762+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:16.763362+0000) 2019-10-13T06:20:18.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:18.273+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:18.274617+0000) 2019-10-13T06:20:18.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:18.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:18.613108+0000) 2019-10-13T06:20:18.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:18.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:18.613234+0000) 2019-10-13T06:20:21.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:21.036+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.037409+0000) 2019-10-13T06:20:21.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:21.766+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.768148+0000) 2019-10-13T06:20:21.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:21.766+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.768343+0000) 2019-10-13T06:20:21.772 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:21.767+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.768541+0000) 2019-10-13T06:20:21.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:21.767+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.769216+0000) 2019-10-13T06:20:21.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:21.768+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.769413+0000) 2019-10-13T06:20:21.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:21.768+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.769591+0000) 2019-10-13T06:20:21.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:21.765+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.766341+0000) 2019-10-13T06:20:21.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:21.767+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.768180+0000) 2019-10-13T06:20:21.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:21.770+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97290 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:21.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:21.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:21.866430+0000) 2019-10-13T06:20:22.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:22.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:22.732831+0000) 2019-10-13T06:20:22.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:22.733+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:22.734162+0000) 2019-10-13T06:20:24.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:24.227+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:24.228092+0000) 2019-10-13T06:20:26.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:26.772+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.774006+0000) 2019-10-13T06:20:26.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:26.772+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.774243+0000) 2019-10-13T06:20:26.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:26.773+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.774419+0000) 2019-10-13T06:20:26.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:26.769+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.770626+0000) 2019-10-13T06:20:26.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:26.773+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.774734+0000) 2019-10-13T06:20:26.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:26.773+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.775024+0000) 2019-10-13T06:20:26.779 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:26.774+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.775736+0000) 2019-10-13T06:20:26.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:26.772+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:26.773285+0000) 2019-10-13T06:20:26.781 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:26.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97320 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:26.918 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:20:26.918 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:26.923 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:20:26.923 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:28.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:28.274+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:28.274830+0000) 2019-10-13T06:20:28.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:28.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:28.613292+0000) 2019-10-13T06:20:28.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:28.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:28.613427+0000) 2019-10-13T06:20:31.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:31.039+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.039848+0000) 2019-10-13T06:20:31.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:31.774+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.775649+0000) 2019-10-13T06:20:31.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:31.778+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.779806+0000) 2019-10-13T06:20:31.784 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:31.778+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.780071+0000) 2019-10-13T06:20:31.784 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:31.778+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.780241+0000) 2019-10-13T06:20:31.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:31.779+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.780538+0000) 2019-10-13T06:20:31.785 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:31.779+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.780783+0000) 2019-10-13T06:20:31.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:31.779+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.781098+0000) 2019-10-13T06:20:31.785 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:31.777+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.778064+0000) 2019-10-13T06:20:31.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:31.782+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:31.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:31.866+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:31.866568+0000) 2019-10-13T06:20:32.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:32.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:32.732975+0000) 2019-10-13T06:20:32.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:32.734+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:32.734325+0000) 2019-10-13T06:20:34.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:34.227+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:34.228256+0000) 2019-10-13T06:20:36.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:36.784+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.785570+0000) 2019-10-13T06:20:36.789 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:36.784+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.786263+0000) 2019-10-13T06:20:36.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:36.784+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.786352+0000) 2019-10-13T06:20:36.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:36.782+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.782767+0000) 2019-10-13T06:20:36.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:36.784+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.786315+0000) 2019-10-13T06:20:36.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:36.785+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.786836+0000) 2019-10-13T06:20:36.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:36.785+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.786998+0000) 2019-10-13T06:20:36.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:36.783+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:36.783973+0000) 2019-10-13T06:20:36.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:36.788+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97383 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:38.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:38.274+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:38.275026+0000) 2019-10-13T06:20:38.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:38.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:38.613484+0000) 2019-10-13T06:20:38.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:38.613+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:38.613532+0000) 2019-10-13T06:20:41.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:41.041+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.042172+0000) 2019-10-13T06:20:41.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:41.790+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.791095+0000) 2019-10-13T06:20:41.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:41.790+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.791237+0000) 2019-10-13T06:20:41.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:41.791+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.791482+0000) 2019-10-13T06:20:41.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:41.791+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.791416+0000) 2019-10-13T06:20:41.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:41.791+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.791900+0000) 2019-10-13T06:20:41.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:41.786+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.787812+0000) 2019-10-13T06:20:41.797 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:41.791+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.792275+0000) 2019-10-13T06:20:41.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:41.788+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.790368+0000) 2019-10-13T06:20:41.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:41.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:41.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:41.866+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:41.866751+0000) 2019-10-13T06:20:42.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:42.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:42.733140+0000) 2019-10-13T06:20:42.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:42.733+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:42.734480+0000) 2019-10-13T06:20:44.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:44.228+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:44.228436+0000) 2019-10-13T06:20:46.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:46.790+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.792502+0000) 2019-10-13T06:20:46.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:46.796+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.796645+0000) 2019-10-13T06:20:46.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:46.796+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.796908+0000) 2019-10-13T06:20:46.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:46.796+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.797139+0000) 2019-10-13T06:20:46.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:46.796+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.797212+0000) 2019-10-13T06:20:46.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:46.797+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.797790+0000) 2019-10-13T06:20:46.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:46.797+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.797889+0000) 2019-10-13T06:20:46.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:46.794+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:46.796336+0000) 2019-10-13T06:20:46.804 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:46.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97444 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:48.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:48.274+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:48.275162+0000) 2019-10-13T06:20:48.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:48.613+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:48.613634+0000) 2019-10-13T06:20:48.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:48.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:48.613633+0000) 2019-10-13T06:20:51.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:51.043+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.044588+0000) 2019-10-13T06:20:51.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:51.801+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.801958+0000) 2019-10-13T06:20:51.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:51.801+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.802094+0000) 2019-10-13T06:20:51.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:51.796+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.798517+0000) 2019-10-13T06:20:51.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:51.802+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.802760+0000) 2019-10-13T06:20:51.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:51.802+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.802897+0000) 2019-10-13T06:20:51.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:51.802+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.803101+0000) 2019-10-13T06:20:51.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:51.802+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.803166+0000) 2019-10-13T06:20:51.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:51.799+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.801450+0000) 2019-10-13T06:20:51.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:51.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97475 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:51.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:51.866+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:51.866901+0000) 2019-10-13T06:20:52.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:52.733+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:52.733304+0000) 2019-10-13T06:20:52.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:52.733+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:52.734648+0000) 2019-10-13T06:20:54.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:54.228+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:54.228642+0000) 2019-10-13T06:20:56.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:20:56.802+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.803370+0000) 2019-10-13T06:20:56.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:56.806+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.807603+0000) 2019-10-13T06:20:56.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:56.806+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.807675+0000) 2019-10-13T06:20:56.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:20:56.806+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.807915+0000) 2019-10-13T06:20:56.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:20:56.807+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.808603+0000) 2019-10-13T06:20:56.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:20:56.807+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.808819+0000) 2019-10-13T06:20:56.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:56.807+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.809075+0000) 2019-10-13T06:20:56.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:20:56.805+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:56.806415+0000) 2019-10-13T06:20:56.815 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:20:56.810+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:20:57.073 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:20:57.073 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:57.078 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:20:57.078 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:58.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:20:58.274+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:58.275341+0000) 2019-10-13T06:20:58.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:20:58.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:58.613815+0000) 2019-10-13T06:20:58.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:20:58.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:20:58.613816+0000) 2019-10-13T06:21:01.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:01.046+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.047089+0000) 2019-10-13T06:21:01.817 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:01.812+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.813974+0000) 2019-10-13T06:21:01.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:01.812+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.814207+0000) 2019-10-13T06:21:01.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:01.813+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.814446+0000) 2019-10-13T06:21:01.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:01.812+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.814266+0000) 2019-10-13T06:21:01.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:01.809+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.810663+0000) 2019-10-13T06:21:01.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:01.810+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.811635+0000) 2019-10-13T06:21:01.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:01.813+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.814901+0000) 2019-10-13T06:21:01.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:01.814+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.815603+0000) 2019-10-13T06:21:01.821 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:01.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97538 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:01.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:01.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:01.867075+0000) 2019-10-13T06:21:02.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:02.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:02.733463+0000) 2019-10-13T06:21:02.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:02.734+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:02.734826+0000) 2019-10-13T06:21:04.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:04.227+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:04.228852+0000) 2019-10-13T06:21:06.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:06.818+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.819463+0000) 2019-10-13T06:21:06.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:06.818+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.819619+0000) 2019-10-13T06:21:06.823 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:06.818+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.819762+0000) 2019-10-13T06:21:06.824 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:06.815+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.816218+0000) 2019-10-13T06:21:06.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:06.818+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.820289+0000) 2019-10-13T06:21:06.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:06.819+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.820558+0000) 2019-10-13T06:21:06.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:06.819+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.820552+0000) 2019-10-13T06:21:06.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:06.817+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:06.818123+0000) 2019-10-13T06:21:06.828 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:06.821+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97568 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:08.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:08.274+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:08.275561+0000) 2019-10-13T06:21:08.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:08.612+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:08.613953+0000) 2019-10-13T06:21:08.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:08.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:08.614024+0000) 2019-10-13T06:21:11.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:11.049+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.049482+0000) 2019-10-13T06:21:11.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:11.823+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.824875+0000) 2019-10-13T06:21:11.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:11.823+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.825315+0000) 2019-10-13T06:21:11.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:11.824+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.825540+0000) 2019-10-13T06:21:11.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:11.824+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.825817+0000) 2019-10-13T06:21:11.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:11.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.822173+0000) 2019-10-13T06:21:11.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:11.824+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.826168+0000) 2019-10-13T06:21:11.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:11.824+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.826299+0000) 2019-10-13T06:21:11.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:11.823+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.823991+0000) 2019-10-13T06:21:11.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:11.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97599 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:11.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:11.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:11.867226+0000) 2019-10-13T06:21:12.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:12.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:12.733662+0000) 2019-10-13T06:21:12.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:12.734+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:12.734979+0000) 2019-10-13T06:21:14.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:14.228+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:14.229075+0000) 2019-10-13T06:21:16.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:16.829+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.831364+0000) 2019-10-13T06:21:16.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:16.830+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.831857+0000) 2019-10-13T06:21:16.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:16.830+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.831956+0000) 2019-10-13T06:21:16.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:16.831+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.832422+0000) 2019-10-13T06:21:16.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:16.831+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.832416+0000) 2019-10-13T06:21:16.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:16.828+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.828914+0000) 2019-10-13T06:21:16.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:16.831+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.832644+0000) 2019-10-13T06:21:16.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:16.830+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:16.831330+0000) 2019-10-13T06:21:16.839 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:16.834+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:18.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:18.275+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:18.275740+0000) 2019-10-13T06:21:18.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:18.613+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:18.614109+0000) 2019-10-13T06:21:18.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:18.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:18.614194+0000) 2019-10-13T06:21:21.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:21.051+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.051790+0000) 2019-10-13T06:21:21.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:21.836+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.837528+0000) 2019-10-13T06:21:21.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:21.836+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.837718+0000) 2019-10-13T06:21:21.842 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:21.836+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.837872+0000) 2019-10-13T06:21:21.842 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:21.836+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.838019+0000) 2019-10-13T06:21:21.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:21.836+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.838188+0000) 2019-10-13T06:21:21.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:21.833+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.834603+0000) 2019-10-13T06:21:21.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:21.837+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.838821+0000) 2019-10-13T06:21:21.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:21.836+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.836898+0000) 2019-10-13T06:21:21.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:21.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97662 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:21.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:21.865+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:21.867371+0000) 2019-10-13T06:21:22.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:22.733+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:22.733813+0000) 2019-10-13T06:21:22.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:22.734+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:22.735137+0000) 2019-10-13T06:21:24.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:24.228+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:24.229282+0000) 2019-10-13T06:21:26.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:26.838+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.839706+0000) 2019-10-13T06:21:26.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:26.843+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.843771+0000) 2019-10-13T06:21:26.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:26.843+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.844152+0000) 2019-10-13T06:21:26.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:26.844+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.844459+0000) 2019-10-13T06:21:26.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:26.844+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.844554+0000) 2019-10-13T06:21:26.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:26.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.844723+0000) 2019-10-13T06:21:26.849 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:26.844+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.844954+0000) 2019-10-13T06:21:26.850 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:26.842+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:26.843080+0000) 2019-10-13T06:21:26.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:26.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97692 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:27.237 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:21:27.237 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:27.242 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:21:27.243 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:28.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:28.275+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:28.275930+0000) 2019-10-13T06:21:28.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:28.613+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:28.614230+0000) 2019-10-13T06:21:28.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:28.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:28.614320+0000) 2019-10-13T06:21:31.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:31.054+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.054224+0000) 2019-10-13T06:21:31.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:31.844+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.845851+0000) 2019-10-13T06:21:31.854 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:31.849+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.849598+0000) 2019-10-13T06:21:31.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:31.849+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.849897+0000) 2019-10-13T06:21:31.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:31.849+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.850209+0000) 2019-10-13T06:21:31.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:31.849+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.850277+0000) 2019-10-13T06:21:31.854 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:31.850+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.850438+0000) 2019-10-13T06:21:31.855 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:31.845+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.847713+0000) 2019-10-13T06:21:31.855 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:31.850+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.850747+0000) 2019-10-13T06:21:31.857 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:31.853+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97723 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:31.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:31.867+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:31.867488+0000) 2019-10-13T06:21:32.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:32.733+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:32.733982+0000) 2019-10-13T06:21:32.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:32.733+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:32.735255+0000) 2019-10-13T06:21:34.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:34.229+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:34.229456+0000) 2019-10-13T06:21:36.859 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:36.855+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.855492+0000) 2019-10-13T06:21:36.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:36.855+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.855700+0000) 2019-10-13T06:21:36.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:36.855+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.855993+0000) 2019-10-13T06:21:36.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:36.851+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.852805+0000) 2019-10-13T06:21:36.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:36.856+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.856656+0000) 2019-10-13T06:21:36.861 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:36.856+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.856831+0000) 2019-10-13T06:21:36.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:36.856+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.857010+0000) 2019-10-13T06:21:36.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:36.852+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:36.854115+0000) 2019-10-13T06:21:36.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:36.859+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:38.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:38.276+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:38.276112+0000) 2019-10-13T06:21:38.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:38.614+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:38.614434+0000) 2019-10-13T06:21:38.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:38.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:38.614478+0000) 2019-10-13T06:21:41.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:41.055+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.056506+0000) 2019-10-13T06:21:41.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:41.855+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.857486+0000) 2019-10-13T06:21:41.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:41.860+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.861588+0000) 2019-10-13T06:21:41.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:41.860+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.861770+0000) 2019-10-13T06:21:41.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:41.860+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.862190+0000) 2019-10-13T06:21:41.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:41.861+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.862472+0000) 2019-10-13T06:21:41.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:41.861+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.862664+0000) 2019-10-13T06:21:41.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:41.861+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.862819+0000) 2019-10-13T06:21:41.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:41.858+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.860112+0000) 2019-10-13T06:21:41.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:41.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:41.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:41.866+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:41.867601+0000) 2019-10-13T06:21:42.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:42.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:42.734099+0000) 2019-10-13T06:21:42.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:42.734+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:42.735456+0000) 2019-10-13T06:21:44.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:44.228+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:44.229641+0000) 2019-10-13T06:21:46.870 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:46.865+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.866919+0000) 2019-10-13T06:21:46.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:46.865+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.867155+0000) 2019-10-13T06:21:46.871 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:46.865+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.867362+0000) 2019-10-13T06:21:46.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:46.866+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.867410+0000) 2019-10-13T06:21:46.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:46.866+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.867617+0000) 2019-10-13T06:21:46.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:46.863+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.864356+0000) 2019-10-13T06:21:46.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:46.866+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.868029+0000) 2019-10-13T06:21:46.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:46.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:46.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:46.865+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:46.866262+0000) 2019-10-13T06:21:48.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:48.275+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:48.276286+0000) 2019-10-13T06:21:48.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:48.613+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:48.614600+0000) 2019-10-13T06:21:48.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:48.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:48.614655+0000) 2019-10-13T06:21:51.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:51.058+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.058811+0000) 2019-10-13T06:21:51.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:51.867+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.868505+0000) 2019-10-13T06:21:51.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:51.869+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.871288+0000) 2019-10-13T06:21:51.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:51.870+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.871460+0000) 2019-10-13T06:21:51.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:51.870+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.871805+0000) 2019-10-13T06:21:51.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:51.870+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.871979+0000) 2019-10-13T06:21:51.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:51.870+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.872123+0000) 2019-10-13T06:21:51.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:51.871+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.872752+0000) 2019-10-13T06:21:51.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:51.867+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.868569+0000) 2019-10-13T06:21:51.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:51.869+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:51.869941+0000) 2019-10-13T06:21:51.879 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:51.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:52.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:52.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:52.734215+0000) 2019-10-13T06:21:52.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:52.734+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:52.735594+0000) 2019-10-13T06:21:54.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:54.229+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:54.229813+0000) 2019-10-13T06:21:56.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:21:56.875+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.876876+0000) 2019-10-13T06:21:56.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:21:56.872+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.873612+0000) 2019-10-13T06:21:56.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:21:56.875+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.877341+0000) 2019-10-13T06:21:56.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:21:56.875+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.877227+0000) 2019-10-13T06:21:56.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:56.876+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.877654+0000) 2019-10-13T06:21:56.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:56.876+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.877960+0000) 2019-10-13T06:21:56.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:56.876+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.878293+0000) 2019-10-13T06:21:56.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:21:56.874+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:56.875371+0000) 2019-10-13T06:21:56.885 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:21:56.880+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:21:57.401 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:21:57.401 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:57.405 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:21:57.406 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:58.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:21:58.275+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:58.276458+0000) 2019-10-13T06:21:58.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:21:58.614+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:58.614794+0000) 2019-10-13T06:21:58.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:21:58.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:21:58.614813+0000) 2019-10-13T06:22:01.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:01.060+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.061120+0000) 2019-10-13T06:22:01.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:01.868+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.868686+0000) 2019-10-13T06:22:01.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:01.878+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.879469+0000) 2019-10-13T06:22:01.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:01.882+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.883539+0000) 2019-10-13T06:22:01.888 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:01.882+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.883750+0000) 2019-10-13T06:22:01.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:01.882+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.883896+0000) 2019-10-13T06:22:01.888 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:01.882+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.884025+0000) 2019-10-13T06:22:01.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:01.882+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.884328+0000) 2019-10-13T06:22:01.889 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:01.883+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.884521+0000) 2019-10-13T06:22:01.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:01.882+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:01.882965+0000) 2019-10-13T06:22:01.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:01.886+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:02.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:02.732+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:02.734308+0000) 2019-10-13T06:22:02.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:02.735+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:02.735766+0000) 2019-10-13T06:22:04.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:04.229+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:04.229998+0000) 2019-10-13T06:22:06.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:06.888+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.889568+0000) 2019-10-13T06:22:06.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:06.888+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.889766+0000) 2019-10-13T06:22:06.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:06.885+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.886033+0000) 2019-10-13T06:22:06.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:06.888+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.890226+0000) 2019-10-13T06:22:06.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:06.889+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.890408+0000) 2019-10-13T06:22:06.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:06.889+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.890672+0000) 2019-10-13T06:22:06.895 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:06.889+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.890814+0000) 2019-10-13T06:22:06.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:06.888+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:06.889202+0000) 2019-10-13T06:22:06.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:06.892+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:08.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:08.276+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:08.276643+0000) 2019-10-13T06:22:08.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:08.614+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:08.615021+0000) 2019-10-13T06:22:08.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:08.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:08.615018+0000) 2019-10-13T06:22:11.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:11.063+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.063555+0000) 2019-10-13T06:22:11.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:11.868+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.868875+0000) 2019-10-13T06:22:11.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:11.890+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.891264+0000) 2019-10-13T06:22:11.899 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:11.895+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.895629+0000) 2019-10-13T06:22:11.899 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:11.895+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.895914+0000) 2019-10-13T06:22:11.900 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:11.895+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.896153+0000) 2019-10-13T06:22:11.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:11.895+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.896327+0000) 2019-10-13T06:22:11.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:11.896+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.896456+0000) 2019-10-13T06:22:11.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:11.896+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.896743+0000) 2019-10-13T06:22:11.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:11.893+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:11.894083+0000) 2019-10-13T06:22:11.902 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:11.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:12.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:12.733+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:12.734402+0000) 2019-10-13T06:22:12.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:12.735+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:12.735923+0000) 2019-10-13T06:22:14.233 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:14.229+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:14.230194+0000) 2019-10-13T06:22:16.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:16.900+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.900973+0000) 2019-10-13T06:22:16.905 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:16.900+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.901287+0000) 2019-10-13T06:22:16.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:16.900+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.901387+0000) 2019-10-13T06:22:16.905 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:16.901+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.901574+0000) 2019-10-13T06:22:16.906 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:16.901+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.901643+0000) 2019-10-13T06:22:16.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:16.897+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.898077+0000) 2019-10-13T06:22:16.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:16.898+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.899361+0000) 2019-10-13T06:22:16.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:16.901+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:16.902350+0000) 2019-10-13T06:22:16.907 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:16.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 97998 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:18.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:18.276+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:18.276797+0000) 2019-10-13T06:22:18.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:18.614+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:18.615233+0000) 2019-10-13T06:22:18.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:18.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:18.615226+0000) 2019-10-13T06:22:21.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:21.065+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.065841+0000) 2019-10-13T06:22:21.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:21.868+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.869033+0000) 2019-10-13T06:22:21.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:21.901+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.901816+0000) 2019-10-13T06:22:21.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:21.905+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.906130+0000) 2019-10-13T06:22:21.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:21.905+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.906234+0000) 2019-10-13T06:22:21.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:21.906+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.906716+0000) 2019-10-13T06:22:21.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:21.906+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.906978+0000) 2019-10-13T06:22:21.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:21.906+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.907052+0000) 2019-10-13T06:22:21.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:21.906+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.907190+0000) 2019-10-13T06:22:21.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:21.903+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:21.903982+0000) 2019-10-13T06:22:21.913 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:21.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:22.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:22.734+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:22.734545+0000) 2019-10-13T06:22:22.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:22.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:22.736076+0000) 2019-10-13T06:22:24.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:24.230+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:24.230368+0000) 2019-10-13T06:22:26.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:26.906+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.907829+0000) 2019-10-13T06:22:26.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:26.911+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.911581+0000) 2019-10-13T06:22:26.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:26.911+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.911899+0000) 2019-10-13T06:22:26.916 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:26.911+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.912278+0000) 2019-10-13T06:22:26.916 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:26.911+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.912160+0000) 2019-10-13T06:22:26.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:26.912+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.912576+0000) 2019-10-13T06:22:26.917 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:26.912+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.912940+0000) 2019-10-13T06:22:26.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:26.909+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:26.910914+0000) 2019-10-13T06:22:26.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:26.915+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:27.531 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:22:27.531 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:27.535 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:22:27.536 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:28.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:28.275+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:28.276993+0000) 2019-10-13T06:22:28.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:28.614+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:28.615441+0000) 2019-10-13T06:22:28.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:28.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:28.615438+0000) 2019-10-13T06:22:31.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:31.067+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.068184+0000) 2019-10-13T06:22:31.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:31.868+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.869212+0000) 2019-10-13T06:22:31.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:31.916+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.917822+0000) 2019-10-13T06:22:31.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:31.916+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.917956+0000) 2019-10-13T06:22:31.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:31.916+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.918121+0000) 2019-10-13T06:22:31.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:31.912+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.914402+0000) 2019-10-13T06:22:31.922 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:31.917+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.918695+0000) 2019-10-13T06:22:31.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:31.917+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.918807+0000) 2019-10-13T06:22:31.923 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:31.917+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.919001+0000) 2019-10-13T06:22:31.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:31.915+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:31.917312+0000) 2019-10-13T06:22:31.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:31.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:32.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:32.733+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:32.734715+0000) 2019-10-13T06:22:32.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:32.735+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:32.736241+0000) 2019-10-13T06:22:34.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:34.229+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:34.230576+0000) 2019-10-13T06:22:36.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:36.918+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.920244+0000) 2019-10-13T06:22:36.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:36.923+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.924414+0000) 2019-10-13T06:22:36.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:36.923+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.924527+0000) 2019-10-13T06:22:36.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:36.923+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.924795+0000) 2019-10-13T06:22:36.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:36.923+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.924950+0000) 2019-10-13T06:22:36.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:36.923+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.925145+0000) 2019-10-13T06:22:36.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:36.923+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.925341+0000) 2019-10-13T06:22:36.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:36.920+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:36.921984+0000) 2019-10-13T06:22:36.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:36.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:38.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:38.276+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:38.277158+0000) 2019-10-13T06:22:38.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:38.614+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:38.615658+0000) 2019-10-13T06:22:38.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:38.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:38.615660+0000) 2019-10-13T06:22:41.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:41.069+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.070664+0000) 2019-10-13T06:22:41.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:41.868+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.869416+0000) 2019-10-13T06:22:41.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:41.929+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.930635+0000) 2019-10-13T06:22:41.934 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:41.929+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.930834+0000) 2019-10-13T06:22:41.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:41.929+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.931097+0000) 2019-10-13T06:22:41.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:41.929+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.931211+0000) 2019-10-13T06:22:41.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:41.926+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.927012+0000) 2019-10-13T06:22:41.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:41.927+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.928638+0000) 2019-10-13T06:22:41.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:41.930+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.931459+0000) 2019-10-13T06:22:41.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:41.930+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:41.931834+0000) 2019-10-13T06:22:41.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:41.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:42.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:42.734+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:42.734886+0000) 2019-10-13T06:22:42.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:42.735+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:42.736398+0000) 2019-10-13T06:22:44.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:44.230+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:44.230765+0000) 2019-10-13T06:22:46.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:46.931+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.932157+0000) 2019-10-13T06:22:46.940 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:46.934+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.936219+0000) 2019-10-13T06:22:46.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:46.935+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.936688+0000) 2019-10-13T06:22:46.940 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:46.935+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.937044+0000) 2019-10-13T06:22:46.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:46.935+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.937067+0000) 2019-10-13T06:22:46.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:46.935+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.937195+0000) 2019-10-13T06:22:46.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:46.933+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.934577+0000) 2019-10-13T06:22:46.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:46.935+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:46.937385+0000) 2019-10-13T06:22:46.943 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:46.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98177 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:48.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:48.276+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:48.277423+0000) 2019-10-13T06:22:48.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:48.614+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:48.615870+0000) 2019-10-13T06:22:48.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:48.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:48.615850+0000) 2019-10-13T06:22:51.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:51.072+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.073105+0000) 2019-10-13T06:22:51.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:51.869+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.869587+0000) 2019-10-13T06:22:51.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:51.940+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.942275+0000) 2019-10-13T06:22:51.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:51.941+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.942855+0000) 2019-10-13T06:22:51.947 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:51.941+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.942715+0000) 2019-10-13T06:22:51.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:51.941+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.943091+0000) 2019-10-13T06:22:51.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:51.941+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.943223+0000) 2019-10-13T06:22:51.948 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:51.941+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.942956+0000) 2019-10-13T06:22:51.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:51.938+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.939546+0000) 2019-10-13T06:22:51.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:51.940+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:51.940830+0000) 2019-10-13T06:22:51.949 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:51.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:52.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:52.734+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:52.735031+0000) 2019-10-13T06:22:52.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:52.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:52.736557+0000) 2019-10-13T06:22:54.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:54.230+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:54.230952+0000) 2019-10-13T06:22:56.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:22:56.947+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.948218+0000) 2019-10-13T06:22:56.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:22:56.944+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.944780+0000) 2019-10-13T06:22:56.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:56.948+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.948822+0000) 2019-10-13T06:22:56.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:22:56.948+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.948974+0000) 2019-10-13T06:22:56.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:56.948+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.949211+0000) 2019-10-13T06:22:56.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:22:56.949+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.949514+0000) 2019-10-13T06:22:56.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:56.949+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.949934+0000) 2019-10-13T06:22:56.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:22:56.947+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:56.948050+0000) 2019-10-13T06:22:56.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:22:56.952+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98238 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:22:57.722 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:22:57.723 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:57.727 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:22:57.727 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:58.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:22:58.277+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:58.277637+0000) 2019-10-13T06:22:58.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:22:58.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:58.615993+0000) 2019-10-13T06:22:58.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:22:58.615+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:22:58.616058+0000) 2019-10-13T06:23:01.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:01.075+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.075450+0000) 2019-10-13T06:23:01.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:01.869+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.869783+0000) 2019-10-13T06:23:01.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:01.949+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.950436+0000) 2019-10-13T06:23:01.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:01.954+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.954909+0000) 2019-10-13T06:23:01.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:01.954+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.955102+0000) 2019-10-13T06:23:01.959 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:01.955+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.955476+0000) 2019-10-13T06:23:01.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:01.954+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.954958+0000) 2019-10-13T06:23:01.959 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:01.955+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.955735+0000) 2019-10-13T06:23:01.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:01.952+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.952811+0000) 2019-10-13T06:23:01.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:01.955+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:01.956130+0000) 2019-10-13T06:23:01.961 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:01.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:02.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:02.734+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:02.735174+0000) 2019-10-13T06:23:02.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:02.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:02.736752+0000) 2019-10-13T06:23:04.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:04.230+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:04.231144+0000) 2019-10-13T06:23:06.964 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:06.960+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.960593+0000) 2019-10-13T06:23:06.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:06.960+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.960804+0000) 2019-10-13T06:23:06.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:06.960+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.960963+0000) 2019-10-13T06:23:06.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:06.960+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.961101+0000) 2019-10-13T06:23:06.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:06.956+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.956968+0000) 2019-10-13T06:23:06.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:06.960+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.961337+0000) 2019-10-13T06:23:06.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:06.961+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.961548+0000) 2019-10-13T06:23:06.966 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:06.958+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:06.959632+0000) 2019-10-13T06:23:06.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:06.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:08.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:08.277+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:08.277835+0000) 2019-10-13T06:23:08.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:08.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:08.616111+0000) 2019-10-13T06:23:08.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:08.615+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:08.616169+0000) 2019-10-13T06:23:11.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:11.077+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.077876+0000) 2019-10-13T06:23:11.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:11.869+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.869986+0000) 2019-10-13T06:23:11.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:11.961+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.962429+0000) 2019-10-13T06:23:11.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:11.966+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.966864+0000) 2019-10-13T06:23:11.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:11.966+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.966871+0000) 2019-10-13T06:23:11.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:11.966+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.967065+0000) 2019-10-13T06:23:11.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:11.966+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.967232+0000) 2019-10-13T06:23:11.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:11.967+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.967435+0000) 2019-10-13T06:23:11.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:11.963+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.964637+0000) 2019-10-13T06:23:11.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:11.967+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:11.967747+0000) 2019-10-13T06:23:11.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:11.970+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:12.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:12.734+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:12.735318+0000) 2019-10-13T06:23:12.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:12.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:12.736901+0000) 2019-10-13T06:23:14.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:14.229+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:14.231336+0000) 2019-10-13T06:23:16.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:16.970+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.972326+0000) 2019-10-13T06:23:16.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:16.967+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.968720+0000) 2019-10-13T06:23:16.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:16.971+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.972970+0000) 2019-10-13T06:23:16.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:16.971+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.973123+0000) 2019-10-13T06:23:16.977 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:16.971+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.973237+0000) 2019-10-13T06:23:16.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:16.972+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.973514+0000) 2019-10-13T06:23:16.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:16.972+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.973742+0000) 2019-10-13T06:23:16.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:16.969+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:16.971048+0000) 2019-10-13T06:23:16.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:16.975+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98358 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:18.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:18.276+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:18.278042+0000) 2019-10-13T06:23:18.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:18.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:18.616256+0000) 2019-10-13T06:23:18.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:18.615+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:18.616305+0000) 2019-10-13T06:23:21.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:21.079+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.080177+0000) 2019-10-13T06:23:21.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:21.869+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.870205+0000) 2019-10-13T06:23:21.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:21.972+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.974358+0000) 2019-10-13T06:23:21.982 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:21.976+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.978125+0000) 2019-10-13T06:23:21.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:21.977+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.978417+0000) 2019-10-13T06:23:21.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:21.977+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.978747+0000) 2019-10-13T06:23:21.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:21.977+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.978975+0000) 2019-10-13T06:23:21.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:21.977+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.979132+0000) 2019-10-13T06:23:21.983 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:21.978+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.980097+0000) 2019-10-13T06:23:21.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:21.975+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:21.976936+0000) 2019-10-13T06:23:21.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:21.980+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98389 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:22.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:22.734+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:22.735487+0000) 2019-10-13T06:23:22.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:22.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:22.737049+0000) 2019-10-13T06:23:24.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:24.230+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:24.231549+0000) 2019-10-13T06:23:26.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:26.982+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.983401+0000) 2019-10-13T06:23:26.987 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:26.982+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.983683+0000) 2019-10-13T06:23:26.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:26.982+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.983822+0000) 2019-10-13T06:23:26.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:26.982+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.983960+0000) 2019-10-13T06:23:26.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:26.978+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.980260+0000) 2019-10-13T06:23:26.989 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:26.982+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.984331+0000) 2019-10-13T06:23:26.989 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:26.983+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.984672+0000) 2019-10-13T06:23:26.989 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:26.980+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:26.982460+0000) 2019-10-13T06:23:26.990 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:26.985+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98417 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:27.937 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:23:27.937 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:27.942 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:23:27.942 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:28.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:28.276+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:28.278265+0000) 2019-10-13T06:23:28.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:28.615+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:28.616450+0000) 2019-10-13T06:23:28.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:28.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:28.616429+0000) 2019-10-13T06:23:31.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:31.081+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.082607+0000) 2019-10-13T06:23:31.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:31.869+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.870300+0000) 2019-10-13T06:23:31.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:31.987+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.988758+0000) 2019-10-13T06:23:31.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:31.987+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.988916+0000) 2019-10-13T06:23:31.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:31.984+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.985214+0000) 2019-10-13T06:23:31.993 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:31.987+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.989302+0000) 2019-10-13T06:23:31.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:31.988+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.989600+0000) 2019-10-13T06:23:31.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:31.988+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.989946+0000) 2019-10-13T06:23:31.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:31.988+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.990304+0000) 2019-10-13T06:23:31.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:31.991+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:31.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:31.987+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:31.987938+0000) 2019-10-13T06:23:32.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:32.735+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:32.735671+0000) 2019-10-13T06:23:32.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:32.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:32.737208+0000) 2019-10-13T06:23:34.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:34.231+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:34.231749+0000) 2019-10-13T06:23:36.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:36.990+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.991104+0000) 2019-10-13T06:23:36.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:36.993+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.994613+0000) 2019-10-13T06:23:36.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:36.993+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.994750+0000) 2019-10-13T06:23:37.000 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:36.993+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.995061+0000) 2019-10-13T06:23:37.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:36.994+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.995476+0000) 2019-10-13T06:23:37.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:36.994+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.995635+0000) 2019-10-13T06:23:37.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:36.994+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.996023+0000) 2019-10-13T06:23:37.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:36.992+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:36.993523+0000) 2019-10-13T06:23:37.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:36.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98478 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:38.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:38.277+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:38.278462+0000) 2019-10-13T06:23:38.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:38.616+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:38.616632+0000) 2019-10-13T06:23:38.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:38.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:38.616653+0000) 2019-10-13T06:23:41.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:41.084+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:41.084990+0000) 2019-10-13T06:23:41.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:41.870+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:41.870476+0000) 2019-10-13T06:23:42.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:41.995+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:41.996562+0000) 2019-10-13T06:23:42.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:42.000+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.000627+0000) 2019-10-13T06:23:42.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:42.000+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.000800+0000) 2019-10-13T06:23:42.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:42.001+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.001530+0000) 2019-10-13T06:23:42.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:42.001+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.001770+0000) 2019-10-13T06:23:42.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:42.001+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.002032+0000) 2019-10-13T06:23:42.006 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:42.001+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.002304+0000) 2019-10-13T06:23:42.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:41.998+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:41.999243+0000) 2019-10-13T06:23:42.008 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:42.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:42.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:42.735+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.735828+0000) 2019-10-13T06:23:42.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:42.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:42.737346+0000) 2019-10-13T06:23:44.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:44.231+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:44.231936+0000) 2019-10-13T06:23:47.010 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:47.006+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.006476+0000) 2019-10-13T06:23:47.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:47.006+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.006771+0000) 2019-10-13T06:23:47.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:47.002+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.003279+0000) 2019-10-13T06:23:47.011 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:47.006+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.007233+0000) 2019-10-13T06:23:47.011 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:47.006+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.007324+0000) 2019-10-13T06:23:47.012 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:47.007+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.007509+0000) 2019-10-13T06:23:47.012 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:47.007+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.007825+0000) 2019-10-13T06:23:47.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:47.003+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:47.004519+0000) 2019-10-13T06:23:47.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:47.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:48.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:48.278+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:48.278652+0000) 2019-10-13T06:23:48.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:48.616+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:48.616833+0000) 2019-10-13T06:23:48.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:48.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:48.616856+0000) 2019-10-13T06:23:51.094 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:51.087+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:51.087400+0000) 2019-10-13T06:23:51.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:51.870+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:51.870671+0000) 2019-10-13T06:23:52.016 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:52.011+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.012128+0000) 2019-10-13T06:23:52.016 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:52.011+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.012318+0000) 2019-10-13T06:23:52.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:52.012+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.012424+0000) 2019-10-13T06:23:52.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:52.008+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.009560+0000) 2019-10-13T06:23:52.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:52.012+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.012735+0000) 2019-10-13T06:23:52.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:52.012+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.013060+0000) 2019-10-13T06:23:52.017 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:52.012+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.013268+0000) 2019-10-13T06:23:52.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:52.010+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.010776+0000) 2019-10-13T06:23:52.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:52.015+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98569 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:52.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:52.735+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.735959+0000) 2019-10-13T06:23:52.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:52.737+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:52.737461+0000) 2019-10-13T06:23:54.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:54.231+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:54.232086+0000) 2019-10-13T06:23:57.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:23:57.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.013467+0000) 2019-10-13T06:23:57.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:57.017+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.017496+0000) 2019-10-13T06:23:57.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:23:57.017+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.017718+0000) 2019-10-13T06:23:57.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:23:57.017+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.017788+0000) 2019-10-13T06:23:57.022 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:57.017+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.017948+0000) 2019-10-13T06:23:57.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:57.018+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.018520+0000) 2019-10-13T06:23:57.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:23:57.018+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.018687+0000) 2019-10-13T06:23:57.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:23:57.016+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:57.017216+0000) 2019-10-13T06:23:57.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:23:57.021+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98599 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:23:58.109 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:23:58.109 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:58.113 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:23:58.113 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:58.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:23:58.278+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:58.278812+0000) 2019-10-13T06:23:58.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:23:58.615+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:58.616968+0000) 2019-10-13T06:23:58.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:23:58.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:23:58.617037+0000) 2019-10-13T06:24:01.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:01.088+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:01.089695+0000) 2019-10-13T06:24:01.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:01.869+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:01.870863+0000) 2019-10-13T06:24:02.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:02.018+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.019506+0000) 2019-10-13T06:24:02.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:02.022+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.023585+0000) 2019-10-13T06:24:02.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:02.022+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.023720+0000) 2019-10-13T06:24:02.028 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:02.022+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.023843+0000) 2019-10-13T06:24:02.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:02.022+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.024056+0000) 2019-10-13T06:24:02.029 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:02.023+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.024604+0000) 2019-10-13T06:24:02.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:02.023+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.024776+0000) 2019-10-13T06:24:02.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:02.022+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.022819+0000) 2019-10-13T06:24:02.030 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:02.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:02.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:02.734+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.736074+0000) 2019-10-13T06:24:02.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:02.737+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:02.737582+0000) 2019-10-13T06:24:04.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:04.231+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:04.232309+0000) 2019-10-13T06:24:07.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:07.024+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.024869+0000) 2019-10-13T06:24:07.033 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:07.027+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.028530+0000) 2019-10-13T06:24:07.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:07.028+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.029411+0000) 2019-10-13T06:24:07.033 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:07.028+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.029603+0000) 2019-10-13T06:24:07.033 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:07.028+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.029813+0000) 2019-10-13T06:24:07.034 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:07.026+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.027153+0000) 2019-10-13T06:24:07.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:07.028+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.030190+0000) 2019-10-13T06:24:07.035 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:07.028+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:07.030348+0000) 2019-10-13T06:24:07.036 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:07.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:08.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:08.278+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:08.278996+0000) 2019-10-13T06:24:08.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:08.616+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:08.617149+0000) 2019-10-13T06:24:08.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:08.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:08.617206+0000) 2019-10-13T06:24:11.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:11.091+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:11.092235+0000) 2019-10-13T06:24:11.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:11.870+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:11.871025+0000) 2019-10-13T06:24:12.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:12.032+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.034308+0000) 2019-10-13T06:24:12.039 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:12.033+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.034787+0000) 2019-10-13T06:24:12.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:12.029+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.031141+0000) 2019-10-13T06:24:12.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:12.033+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.035144+0000) 2019-10-13T06:24:12.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:12.032+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.033832+0000) 2019-10-13T06:24:12.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:12.033+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.035314+0000) 2019-10-13T06:24:12.045 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:12.034+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.035686+0000) 2019-10-13T06:24:12.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:12.034+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.036040+0000) 2019-10-13T06:24:12.046 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:12.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:12.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:12.735+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.736220+0000) 2019-10-13T06:24:12.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:12.736+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:12.737724+0000) 2019-10-13T06:24:14.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:14.231+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:14.232493+0000) 2019-10-13T06:24:17.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:17.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.036638+0000) 2019-10-13T06:24:17.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:17.036+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.037827+0000) 2019-10-13T06:24:17.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:17.038+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.039793+0000) 2019-10-13T06:24:17.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:17.038+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.039941+0000) 2019-10-13T06:24:17.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:17.038+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.040047+0000) 2019-10-13T06:24:17.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:17.038+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.040284+0000) 2019-10-13T06:24:17.047 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:17.039+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.040662+0000) 2019-10-13T06:24:17.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:17.039+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:17.040941+0000) 2019-10-13T06:24:17.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:17.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:18.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:18.278+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:18.279188+0000) 2019-10-13T06:24:18.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:18.616+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:18.617335+0000) 2019-10-13T06:24:18.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:18.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:18.617416+0000) 2019-10-13T06:24:21.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:21.093+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:21.094532+0000) 2019-10-13T06:24:21.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:21.870+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:21.871129+0000) 2019-10-13T06:24:22.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:22.044+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.045389+0000) 2019-10-13T06:24:22.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:22.044+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.045523+0000) 2019-10-13T06:24:22.049 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:22.044+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.045813+0000) 2019-10-13T06:24:22.050 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:22.044+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.045920+0000) 2019-10-13T06:24:22.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:22.041+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.042192+0000) 2019-10-13T06:24:22.051 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:22.044+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.046270+0000) 2019-10-13T06:24:22.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:22.045+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.046572+0000) 2019-10-13T06:24:22.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:22.043+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.044653+0000) 2019-10-13T06:24:22.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:22.047+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98753 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:22.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:22.735+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.736354+0000) 2019-10-13T06:24:22.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:22.737+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:22.737870+0000) 2019-10-13T06:24:24.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:24.232+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:24.232685+0000) 2019-10-13T06:24:27.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:27.049+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.049977+0000) 2019-10-13T06:24:27.054 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:27.049+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.050151+0000) 2019-10-13T06:24:27.054 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:27.049+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.050365+0000) 2019-10-13T06:24:27.054 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:27.050+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.050807+0000) 2019-10-13T06:24:27.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:27.046+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.047152+0000) 2019-10-13T06:24:27.055 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:27.050+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.051174+0000) 2019-10-13T06:24:27.055 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:27.051+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.051549+0000) 2019-10-13T06:24:27.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:27.048+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:27.049597+0000) 2019-10-13T06:24:27.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:27.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98781 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:28.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:28.278+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:28.279380+0000) 2019-10-13T06:24:28.315 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:24:28.315 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:28.319 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:24:28.320 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:28.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:28.617+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:28.617499+0000) 2019-10-13T06:24:28.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:28.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:28.617569+0000) 2019-10-13T06:24:31.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:31.096+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:31.096881+0000) 2019-10-13T06:24:31.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:31.870+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:31.871276+0000) 2019-10-13T06:24:32.059 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:32.055+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.056192+0000) 2019-10-13T06:24:32.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:32.052+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.052816+0000) 2019-10-13T06:24:32.061 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:32.056+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.056539+0000) 2019-10-13T06:24:32.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:32.056+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.056863+0000) 2019-10-13T06:24:32.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:32.056+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.057153+0000) 2019-10-13T06:24:32.061 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:32.056+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.057059+0000) 2019-10-13T06:24:32.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:32.057+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.057539+0000) 2019-10-13T06:24:32.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:32.053+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.054575+0000) 2019-10-13T06:24:32.063 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:32.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98813 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:32.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:32.736+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.736496+0000) 2019-10-13T06:24:32.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:32.737+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:32.738013+0000) 2019-10-13T06:24:34.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:34.232+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:34.232859+0000) 2019-10-13T06:24:37.065 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:37.061+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.062192+0000) 2019-10-13T06:24:37.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:37.058+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.058824+0000) 2019-10-13T06:24:37.066 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:37.062+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.062964+0000) 2019-10-13T06:24:37.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:37.062+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.062999+0000) 2019-10-13T06:24:37.067 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:37.063+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.063425+0000) 2019-10-13T06:24:37.067 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:37.063+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.063551+0000) 2019-10-13T06:24:37.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:37.063+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.063861+0000) 2019-10-13T06:24:37.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:37.061+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:37.061773+0000) 2019-10-13T06:24:37.070 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:37.066+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:38.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:38.279+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:38.279557+0000) 2019-10-13T06:24:38.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:38.617+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:38.617653+0000) 2019-10-13T06:24:38.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:38.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:38.617749+0000) 2019-10-13T06:24:41.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:41.098+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:41.099375+0000) 2019-10-13T06:24:41.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:41.871+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:41.871484+0000) 2019-10-13T06:24:42.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:42.068+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.068615+0000) 2019-10-13T06:24:42.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:42.064+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.065146+0000) 2019-10-13T06:24:42.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:42.068+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.069226+0000) 2019-10-13T06:24:42.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:42.068+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.069158+0000) 2019-10-13T06:24:42.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:42.069+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.069422+0000) 2019-10-13T06:24:42.074 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:42.069+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.069665+0000) 2019-10-13T06:24:42.074 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:42.069+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.069910+0000) 2019-10-13T06:24:42.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:42.067+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.068498+0000) 2019-10-13T06:24:42.076 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:42.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:42.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:42.736+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.736641+0000) 2019-10-13T06:24:42.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:42.737+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:42.738153+0000) 2019-10-13T06:24:44.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:44.232+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:44.233057+0000) 2019-10-13T06:24:47.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:47.073+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.074844+0000) 2019-10-13T06:24:47.079 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:47.073+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.074743+0000) 2019-10-13T06:24:47.079 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:47.073+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.075253+0000) 2019-10-13T06:24:47.079 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:47.073+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.075173+0000) 2019-10-13T06:24:47.080 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:47.073+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.075373+0000) 2019-10-13T06:24:47.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:47.074+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.075523+0000) 2019-10-13T06:24:47.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:47.071+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.071869+0000) 2019-10-13T06:24:47.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:47.072+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:47.072736+0000) 2019-10-13T06:24:47.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:47.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:48.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:48.278+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:48.279726+0000) 2019-10-13T06:24:48.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:48.616+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:48.617841+0000) 2019-10-13T06:24:48.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:48.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:48.617901+0000) 2019-10-13T06:24:51.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:51.101+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:51.101771+0000) 2019-10-13T06:24:51.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:51.870+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:51.871672+0000) 2019-10-13T06:24:52.083 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:52.078+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.080099+0000) 2019-10-13T06:24:52.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:52.076+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.076917+0000) 2019-10-13T06:24:52.085 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:52.078+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.080315+0000) 2019-10-13T06:24:52.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:52.079+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.080448+0000) 2019-10-13T06:24:52.085 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:52.079+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.080628+0000) 2019-10-13T06:24:52.085 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:52.079+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.081046+0000) 2019-10-13T06:24:52.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:52.079+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.081313+0000) 2019-10-13T06:24:52.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:52.079+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.079808+0000) 2019-10-13T06:24:52.087 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:52.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:52.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:52.735+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.736773+0000) 2019-10-13T06:24:52.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:52.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:52.738317+0000) 2019-10-13T06:24:54.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:54.232+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:54.233248+0000) 2019-10-13T06:24:57.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:24:57.084+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.085842+0000) 2019-10-13T06:24:57.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:24:57.085+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.086411+0000) 2019-10-13T06:24:57.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:24:57.085+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.086526+0000) 2019-10-13T06:24:57.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:57.085+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.086662+0000) 2019-10-13T06:24:57.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:57.085+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.087207+0000) 2019-10-13T06:24:57.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:24:57.082+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.083277+0000) 2019-10-13T06:24:57.092 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:57.084+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.086047+0000) 2019-10-13T06:24:57.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:24:57.084+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:57.085303+0000) 2019-10-13T06:24:57.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:24:57.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:24:58.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:24:58.279+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:58.279911+0000) 2019-10-13T06:24:58.513 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:24:58.515 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:58.520 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:24:58.520 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:58.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:24:58.617+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:58.618091+0000) 2019-10-13T06:24:58.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:24:58.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:24:58.618093+0000) 2019-10-13T06:25:01.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:01.103+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:01.104328+0000) 2019-10-13T06:25:01.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:01.871+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:01.871867+0000) 2019-10-13T06:25:02.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:02.090+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.091516+0000) 2019-10-13T06:25:02.095 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:02.090+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.091807+0000) 2019-10-13T06:25:02.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:02.090+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.092063+0000) 2019-10-13T06:25:02.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:02.090+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.092275+0000) 2019-10-13T06:25:02.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:02.086+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.087825+0000) 2019-10-13T06:25:02.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:02.091+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.092451+0000) 2019-10-13T06:25:02.097 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:02.091+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.092633+0000) 2019-10-13T06:25:02.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:02.088+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.090314+0000) 2019-10-13T06:25:02.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:02.093+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 98997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:02.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:02.736+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.736940+0000) 2019-10-13T06:25:02.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:02.737+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:02.738515+0000) 2019-10-13T06:25:04.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:04.232+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:04.233440+0000) 2019-10-13T06:25:07.101 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:07.096+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.097431+0000) 2019-10-13T06:25:07.101 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:07.095+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.097238+0000) 2019-10-13T06:25:07.101 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:07.096+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.097916+0000) 2019-10-13T06:25:07.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:07.096+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.098078+0000) 2019-10-13T06:25:07.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:07.092+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.094401+0000) 2019-10-13T06:25:07.103 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:07.096+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.098354+0000) 2019-10-13T06:25:07.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:07.097+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.098561+0000) 2019-10-13T06:25:07.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:07.094+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:07.096001+0000) 2019-10-13T06:25:07.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:07.099+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:08.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:08.279+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:08.280100+0000) 2019-10-13T06:25:08.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:08.617+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:08.618276+0000) 2019-10-13T06:25:08.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:08.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:08.618294+0000) 2019-10-13T06:25:11.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:11.106+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:11.106800+0000) 2019-10-13T06:25:11.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:11.871+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:11.872021+0000) 2019-10-13T06:25:12.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:12.097+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.098752+0000) 2019-10-13T06:25:12.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:12.102+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.103194+0000) 2019-10-13T06:25:12.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:12.103+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.103532+0000) 2019-10-13T06:25:12.107 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:12.103+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.103449+0000) 2019-10-13T06:25:12.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:12.103+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.103631+0000) 2019-10-13T06:25:12.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:12.103+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.103888+0000) 2019-10-13T06:25:12.108 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:12.103+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.104030+0000) 2019-10-13T06:25:12.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:12.099+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.101145+0000) 2019-10-13T06:25:12.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:12.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99057 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:12.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:12.736+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.737099+0000) 2019-10-13T06:25:12.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:12.737+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:12.738672+0000) 2019-10-13T06:25:14.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:14.233+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:14.233632+0000) 2019-10-13T06:25:17.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:17.108+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.108482+0000) 2019-10-13T06:25:17.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:17.104+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.104853+0000) 2019-10-13T06:25:17.113 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:17.108+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.109179+0000) 2019-10-13T06:25:17.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:17.109+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.109457+0000) 2019-10-13T06:25:17.114 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:17.109+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.109898+0000) 2019-10-13T06:25:17.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:17.109+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.110015+0000) 2019-10-13T06:25:17.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:17.109+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.110192+0000) 2019-10-13T06:25:17.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:17.107+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:17.107894+0000) 2019-10-13T06:25:17.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:17.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99087 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:18.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:18.279+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:18.280312+0000) 2019-10-13T06:25:18.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:18.618+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:18.618492+0000) 2019-10-13T06:25:18.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:18.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:18.618492+0000) 2019-10-13T06:25:21.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:21.108+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:21.109165+0000) 2019-10-13T06:25:21.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:21.871+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:21.872166+0000) 2019-10-13T06:25:22.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:22.114+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.114722+0000) 2019-10-13T06:25:22.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:22.114+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.114885+0000) 2019-10-13T06:25:22.119 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:22.114+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.115102+0000) 2019-10-13T06:25:22.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:22.115+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.115733+0000) 2019-10-13T06:25:22.120 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:22.115+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.115876+0000) 2019-10-13T06:25:22.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:22.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.116467+0000) 2019-10-13T06:25:22.121 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:22.111+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.112469+0000) 2019-10-13T06:25:22.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:22.114+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.114745+0000) 2019-10-13T06:25:22.122 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:22.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99119 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:22.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:22.737+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.737263+0000) 2019-10-13T06:25:22.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:22.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:22.738768+0000) 2019-10-13T06:25:24.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:24.233+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:24.233838+0000) 2019-10-13T06:25:27.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:27.120+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.120525+0000) 2019-10-13T06:25:27.125 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:27.120+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.120753+0000) 2019-10-13T06:25:27.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:27.120+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.121206+0000) 2019-10-13T06:25:27.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:27.120+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.121350+0000) 2019-10-13T06:25:27.126 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:27.121+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.121471+0000) 2019-10-13T06:25:27.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:27.121+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.121619+0000) 2019-10-13T06:25:27.127 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:27.117+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.117977+0000) 2019-10-13T06:25:27.127 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:27.118+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:27.119351+0000) 2019-10-13T06:25:27.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:27.124+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:28.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:28.280+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:28.280500+0000) 2019-10-13T06:25:28.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:28.618+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:28.618682+0000) 2019-10-13T06:25:28.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:28.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:28.618677+0000) 2019-10-13T06:25:28.694 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:25:28.694 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:28.699 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:25:28.699 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:31.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:31.110+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:31.111416+0000) 2019-10-13T06:25:31.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:31.871+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:31.872321+0000) 2019-10-13T06:25:32.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:32.124+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.126330+0000) 2019-10-13T06:25:32.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:32.125+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.126786+0000) 2019-10-13T06:25:32.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:32.125+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.126946+0000) 2019-10-13T06:25:32.131 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:32.125+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.126822+0000) 2019-10-13T06:25:32.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:32.122+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.123470+0000) 2019-10-13T06:25:32.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:32.126+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.127539+0000) 2019-10-13T06:25:32.132 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:32.126+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.128079+0000) 2019-10-13T06:25:32.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:32.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:32.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:32.126+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.126788+0000) 2019-10-13T06:25:32.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:32.736+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.737426+0000) 2019-10-13T06:25:32.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:32.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:32.738910+0000) 2019-10-13T06:25:34.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:34.233+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:34.234057+0000) 2019-10-13T06:25:37.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:37.130+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.131502+0000) 2019-10-13T06:25:37.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:37.130+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.131634+0000) 2019-10-13T06:25:37.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:37.130+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.131729+0000) 2019-10-13T06:25:37.136 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:37.130+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.131995+0000) 2019-10-13T06:25:37.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:37.130+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.132037+0000) 2019-10-13T06:25:37.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:37.127+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.128561+0000) 2019-10-13T06:25:37.138 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:37.131+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.132465+0000) 2019-10-13T06:25:37.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:37.130+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:37.131035+0000) 2019-10-13T06:25:37.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:37.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:38.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:38.279+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:38.280770+0000) 2019-10-13T06:25:38.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:38.617+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:38.618883+0000) 2019-10-13T06:25:38.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:38.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:38.618870+0000) 2019-10-13T06:25:41.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:41.113+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:41.113868+0000) 2019-10-13T06:25:41.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:41.871+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:41.872497+0000) 2019-10-13T06:25:42.140 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:42.134+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.136117+0000) 2019-10-13T06:25:42.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:42.134+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.136149+0000) 2019-10-13T06:25:42.140 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:42.134+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.136286+0000) 2019-10-13T06:25:42.141 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:42.135+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.136403+0000) 2019-10-13T06:25:42.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:42.135+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.136546+0000) 2019-10-13T06:25:42.141 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:42.135+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.136773+0000) 2019-10-13T06:25:42.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:42.132+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.133428+0000) 2019-10-13T06:25:42.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:42.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:42.143 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:42.134+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.135709+0000) 2019-10-13T06:25:42.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:42.736+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.737592+0000) 2019-10-13T06:25:42.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:42.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:42.739040+0000) 2019-10-13T06:25:44.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:44.233+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:44.234265+0000) 2019-10-13T06:25:47.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:47.139+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.141155+0000) 2019-10-13T06:25:47.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:47.140+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.141577+0000) 2019-10-13T06:25:47.146 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:47.140+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.141864+0000) 2019-10-13T06:25:47.146 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:47.140+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.142087+0000) 2019-10-13T06:25:47.146 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:47.140+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.142242+0000) 2019-10-13T06:25:47.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:47.140+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.142384+0000) 2019-10-13T06:25:47.147 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:47.138+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.139177+0000) 2019-10-13T06:25:47.148 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:47.140+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:47.141040+0000) 2019-10-13T06:25:47.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:47.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:48.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:48.280+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:48.280984+0000) 2019-10-13T06:25:48.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:48.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:48.619063+0000) 2019-10-13T06:25:48.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:48.617+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:48.619079+0000) 2019-10-13T06:25:51.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:51.114+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:51.116089+0000) 2019-10-13T06:25:51.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:51.872+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:51.872673+0000) 2019-10-13T06:25:52.150 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:52.145+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.147051+0000) 2019-10-13T06:25:52.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:52.145+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.147361+0000) 2019-10-13T06:25:52.151 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:52.146+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.147587+0000) 2019-10-13T06:25:52.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:52.146+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.147747+0000) 2019-10-13T06:25:52.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:52.146+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.147921+0000) 2019-10-13T06:25:52.152 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:52.146+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.148115+0000) 2019-10-13T06:25:52.152 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:52.143+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.144993+0000) 2019-10-13T06:25:52.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:52.145+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.147019+0000) 2019-10-13T06:25:52.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:52.150+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99301 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:52.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:52.736+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.737728+0000) 2019-10-13T06:25:52.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:52.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:52.739199+0000) 2019-10-13T06:25:54.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:54.233+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:54.234475+0000) 2019-10-13T06:25:57.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:25:57.153+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.153437+0000) 2019-10-13T06:25:57.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:25:57.153+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.153959+0000) 2019-10-13T06:25:57.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:57.153+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.154085+0000) 2019-10-13T06:25:57.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:57.153+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.154374+0000) 2019-10-13T06:25:57.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:25:57.149+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.150800+0000) 2019-10-13T06:25:57.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:57.154+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.154868+0000) 2019-10-13T06:25:57.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:25:57.154+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.155066+0000) 2019-10-13T06:25:57.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:25:57.150+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:57.152595+0000) 2019-10-13T06:25:57.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:25:57.157+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99331 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:25:58.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:25:58.280+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:58.281176+0000) 2019-10-13T06:25:58.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:25:58.618+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:58.619267+0000) 2019-10-13T06:25:58.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:25:58.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:25:58.619260+0000) 2019-10-13T06:25:58.887 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:25:58.887 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:58.893 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:25:58.893 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:01.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:01.117+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:01.118617+0000) 2019-10-13T06:26:01.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:01.872+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:01.872836+0000) 2019-10-13T06:26:02.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:02.159+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.159744+0000) 2019-10-13T06:26:02.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:02.159+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.159846+0000) 2019-10-13T06:26:02.164 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:02.159+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.160195+0000) 2019-10-13T06:26:02.164 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:02.160+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.160483+0000) 2019-10-13T06:26:02.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:02.154+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.156513+0000) 2019-10-13T06:26:02.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:02.160+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.160830+0000) 2019-10-13T06:26:02.165 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:02.160+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.161163+0000) 2019-10-13T06:26:02.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:02.157+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.159247+0000) 2019-10-13T06:26:02.167 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:02.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:02.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:02.737+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.737875+0000) 2019-10-13T06:26:02.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:02.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:02.739357+0000) 2019-10-13T06:26:04.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:04.234+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:04.234662+0000) 2019-10-13T06:26:07.168 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:07.164+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.165357+0000) 2019-10-13T06:26:07.169 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:07.160+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.161701+0000) 2019-10-13T06:26:07.170 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:07.165+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.166020+0000) 2019-10-13T06:26:07.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:07.165+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.166148+0000) 2019-10-13T06:26:07.170 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:07.165+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.166335+0000) 2019-10-13T06:26:07.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:07.166+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.166809+0000) 2019-10-13T06:26:07.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:07.166+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.167163+0000) 2019-10-13T06:26:07.174 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:07.169+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:07.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:07.164+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:07.164983+0000) 2019-10-13T06:26:08.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:08.281+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:08.281389+0000) 2019-10-13T06:26:08.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:08.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:08.619430+0000) 2019-10-13T06:26:08.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:08.619+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:08.619464+0000) 2019-10-13T06:26:11.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:11.120+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:11.120939+0000) 2019-10-13T06:26:11.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:11.872+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:11.873031+0000) 2019-10-13T06:26:12.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:12.171+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.171484+0000) 2019-10-13T06:26:12.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:12.167+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.167827+0000) 2019-10-13T06:26:12.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:12.171+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.171934+0000) 2019-10-13T06:26:12.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:12.171+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.172094+0000) 2019-10-13T06:26:12.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:12.172+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.172702+0000) 2019-10-13T06:26:12.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:12.172+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.172878+0000) 2019-10-13T06:26:12.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:12.173+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.173463+0000) 2019-10-13T06:26:12.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:12.169+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.170642+0000) 2019-10-13T06:26:12.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:12.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:12.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:12.737+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.738042+0000) 2019-10-13T06:26:12.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:12.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:12.739541+0000) 2019-10-13T06:26:14.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:14.234+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:14.234850+0000) 2019-10-13T06:26:17.181 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:17.176+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.177875+0000) 2019-10-13T06:26:17.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:17.176+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.178128+0000) 2019-10-13T06:26:17.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:17.176+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.178230+0000) 2019-10-13T06:26:17.182 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:17.177+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.178440+0000) 2019-10-13T06:26:17.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:17.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.174894+0000) 2019-10-13T06:26:17.183 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:17.177+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.178742+0000) 2019-10-13T06:26:17.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:17.177+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.179168+0000) 2019-10-13T06:26:17.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:17.176+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:17.177268+0000) 2019-10-13T06:26:17.185 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:17.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99453 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:18.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:18.280+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:18.281618+0000) 2019-10-13T06:26:18.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:18.618+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:18.619620+0000) 2019-10-13T06:26:18.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:18.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:18.619619+0000) 2019-10-13T06:26:21.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:21.123+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:21.123517+0000) 2019-10-13T06:26:21.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:21.871+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:21.873248+0000) 2019-10-13T06:26:22.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:22.182+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.183903+0000) 2019-10-13T06:26:22.187 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:22.182+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.184200+0000) 2019-10-13T06:26:22.188 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:22.179+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.180228+0000) 2019-10-13T06:26:22.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:22.183+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.184522+0000) 2019-10-13T06:26:22.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:22.183+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.184965+0000) 2019-10-13T06:26:22.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:22.183+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.185011+0000) 2019-10-13T06:26:22.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:22.183+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.185366+0000) 2019-10-13T06:26:22.190 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:22.181+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.182599+0000) 2019-10-13T06:26:22.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:22.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:22.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:22.737+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.738198+0000) 2019-10-13T06:26:22.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:22.738+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:22.739696+0000) 2019-10-13T06:26:24.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:24.234+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:24.235045+0000) 2019-10-13T06:26:27.194 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:27.188+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.190120+0000) 2019-10-13T06:26:27.194 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:27.189+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.190411+0000) 2019-10-13T06:26:27.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:27.188+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.190328+0000) 2019-10-13T06:26:27.195 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:27.189+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.190817+0000) 2019-10-13T06:26:27.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:27.189+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.190924+0000) 2019-10-13T06:26:27.196 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:27.185+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.186681+0000) 2019-10-13T06:26:27.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:27.188+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.188952+0000) 2019-10-13T06:26:27.197 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:27.190+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:27.191422+0000) 2019-10-13T06:26:27.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:27.192+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:28.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:28.281+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:28.281816+0000) 2019-10-13T06:26:28.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:28.619+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:28.619809+0000) 2019-10-13T06:26:28.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:28.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:28.619812+0000) 2019-10-13T06:26:29.308 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:26:29.309 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:29.313 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:26:29.314 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:31.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:31.125+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:31.125888+0000) 2019-10-13T06:26:31.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:31.872+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:31.873441+0000) 2019-10-13T06:26:32.199 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:32.190+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.191426+0000) 2019-10-13T06:26:32.199 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:32.193+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.195109+0000) 2019-10-13T06:26:32.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:32.194+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.195676+0000) 2019-10-13T06:26:32.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:32.194+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.195964+0000) 2019-10-13T06:26:32.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:32.194+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.196081+0000) 2019-10-13T06:26:32.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:32.194+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.196227+0000) 2019-10-13T06:26:32.201 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:32.195+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.196628+0000) 2019-10-13T06:26:32.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:32.193+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.193812+0000) 2019-10-13T06:26:32.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:32.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:32.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:32.737+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.738342+0000) 2019-10-13T06:26:32.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:32.739+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:32.739869+0000) 2019-10-13T06:26:34.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:34.233+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:34.235247+0000) 2019-10-13T06:26:37.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:37.200+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.201483+0000) 2019-10-13T06:26:37.206 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:37.200+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.201531+0000) 2019-10-13T06:26:37.206 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:37.200+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.201721+0000) 2019-10-13T06:26:37.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:37.200+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.201943+0000) 2019-10-13T06:26:37.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:37.200+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.202335+0000) 2019-10-13T06:26:37.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:37.196+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.197367+0000) 2019-10-13T06:26:37.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:37.201+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.202987+0000) 2019-10-13T06:26:37.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:37.200+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:37.200937+0000) 2019-10-13T06:26:37.209 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:37.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99575 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:38.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:38.281+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:38.282016+0000) 2019-10-13T06:26:38.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:38.618+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:38.619970+0000) 2019-10-13T06:26:38.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:38.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:38.619980+0000) 2019-10-13T06:26:41.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:41.128+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:41.128341+0000) 2019-10-13T06:26:41.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:41.873+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:41.873633+0000) 2019-10-13T06:26:42.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:42.205+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.207212+0000) 2019-10-13T06:26:42.211 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:42.206+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.207439+0000) 2019-10-13T06:26:42.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:42.206+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.207632+0000) 2019-10-13T06:26:42.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:42.202+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.203958+0000) 2019-10-13T06:26:42.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:42.206+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.207972+0000) 2019-10-13T06:26:42.212 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:42.206+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.208095+0000) 2019-10-13T06:26:42.213 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:42.206+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.208233+0000) 2019-10-13T06:26:42.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:42.205+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.206805+0000) 2019-10-13T06:26:42.214 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:42.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:42.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:42.738+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.738520+0000) 2019-10-13T06:26:42.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:42.739+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:42.740028+0000) 2019-10-13T06:26:44.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:44.235+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:44.235450+0000) 2019-10-13T06:26:47.216 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:47.212+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.212562+0000) 2019-10-13T06:26:47.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:47.212+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.212760+0000) 2019-10-13T06:26:47.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:47.207+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.209057+0000) 2019-10-13T06:26:47.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:47.212+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.213299+0000) 2019-10-13T06:26:47.218 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:47.213+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.213488+0000) 2019-10-13T06:26:47.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:47.213+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.213691+0000) 2019-10-13T06:26:47.218 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:47.213+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.214053+0000) 2019-10-13T06:26:47.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:47.210+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:47.212135+0000) 2019-10-13T06:26:47.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:47.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:48.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:48.281+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:48.282182+0000) 2019-10-13T06:26:48.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:48.619+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:48.620069+0000) 2019-10-13T06:26:48.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:48.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:48.620151+0000) 2019-10-13T06:26:51.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:51.129+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:51.130766+0000) 2019-10-13T06:26:51.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:51.873+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:51.873823+0000) 2019-10-13T06:26:52.223 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:52.218+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.219299+0000) 2019-10-13T06:26:52.223 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:52.219+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.219465+0000) 2019-10-13T06:26:52.224 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:52.214+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.215855+0000) 2019-10-13T06:26:52.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:52.219+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.219970+0000) 2019-10-13T06:26:52.224 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:52.215+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.217130+0000) 2019-10-13T06:26:52.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:52.219+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.220191+0000) 2019-10-13T06:26:52.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:52.220+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.220411+0000) 2019-10-13T06:26:52.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:52.220+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.220786+0000) 2019-10-13T06:26:52.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:52.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:52.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:52.738+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.738691+0000) 2019-10-13T06:26:52.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:52.739+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:52.740185+0000) 2019-10-13T06:26:54.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:54.235+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:54.235641+0000) 2019-10-13T06:26:57.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:57.225+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.225646+0000) 2019-10-13T06:26:57.230 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:26:57.225+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.226074+0000) 2019-10-13T06:26:57.230 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:57.225+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.226210+0000) 2019-10-13T06:26:57.231 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:26:57.221+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.222028+0000) 2019-10-13T06:26:57.232 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:57.226+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.226566+0000) 2019-10-13T06:26:57.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:26:57.226+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.226838+0000) 2019-10-13T06:26:57.232 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:26:57.226+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.227100+0000) 2019-10-13T06:26:57.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:26:57.224+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:57.224748+0000) 2019-10-13T06:26:57.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:26:57.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:26:58.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:26:58.282+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:58.282355+0000) 2019-10-13T06:26:58.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:26:58.620+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:58.620237+0000) 2019-10-13T06:26:58.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:26:58.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:26:58.620303+0000) 2019-10-13T06:26:59.472 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:26:59.472 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:59.476 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:26:59.477 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:01.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:01.132+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:01.133016+0000) 2019-10-13T06:27:01.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:01.872+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:01.874012+0000) 2019-10-13T06:27:02.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:02.230+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.231402+0000) 2019-10-13T06:27:02.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:02.226+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.227710+0000) 2019-10-13T06:27:02.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:02.230+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.232300+0000) 2019-10-13T06:27:02.236 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:02.230+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.232293+0000) 2019-10-13T06:27:02.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:02.231+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.232657+0000) 2019-10-13T06:27:02.237 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:02.231+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.232798+0000) 2019-10-13T06:27:02.237 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:02.231+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.233014+0000) 2019-10-13T06:27:02.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:02.229+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.230047+0000) 2019-10-13T06:27:02.239 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:02.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:02.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:02.737+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.738840+0000) 2019-10-13T06:27:02.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:02.739+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:02.740342+0000) 2019-10-13T06:27:04.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:04.234+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:04.235801+0000) 2019-10-13T06:27:07.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:07.236+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.237457+0000) 2019-10-13T06:27:07.243 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:07.236+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.237865+0000) 2019-10-13T06:27:07.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:07.236+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.238005+0000) 2019-10-13T06:27:07.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:07.236+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.238154+0000) 2019-10-13T06:27:07.244 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:07.237+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.238638+0000) 2019-10-13T06:27:07.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:07.237+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.238846+0000) 2019-10-13T06:27:07.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:07.233+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.234680+0000) 2019-10-13T06:27:07.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:07.235+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:07.235914+0000) 2019-10-13T06:27:07.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:07.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:08.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:08.281+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:08.282518+0000) 2019-10-13T06:27:08.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:08.619+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:08.620464+0000) 2019-10-13T06:27:08.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:08.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:08.620500+0000) 2019-10-13T06:27:11.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:11.134+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:11.135387+0000) 2019-10-13T06:27:11.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:11.873+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:11.874176+0000) 2019-10-13T06:27:12.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:12.241+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.243182+0000) 2019-10-13T06:27:12.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:12.238+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.239640+0000) 2019-10-13T06:27:12.248 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:12.242+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.243642+0000) 2019-10-13T06:27:12.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:12.242+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.243724+0000) 2019-10-13T06:27:12.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:12.242+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.243871+0000) 2019-10-13T06:27:12.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:12.242+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.244008+0000) 2019-10-13T06:27:12.249 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:12.243+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.244390+0000) 2019-10-13T06:27:12.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:12.242+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.242791+0000) 2019-10-13T06:27:12.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:12.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99789 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:12.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:12.738+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.739027+0000) 2019-10-13T06:27:12.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:12.739+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:12.740473+0000) 2019-10-13T06:27:14.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:14.235+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:14.235989+0000) 2019-10-13T06:27:17.255 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:17.248+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.249446+0000) 2019-10-13T06:27:17.255 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:17.248+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.249631+0000) 2019-10-13T06:27:17.256 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:17.248+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.249765+0000) 2019-10-13T06:27:17.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:17.248+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.250069+0000) 2019-10-13T06:27:17.256 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:17.249+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.250593+0000) 2019-10-13T06:27:17.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:17.249+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.250833+0000) 2019-10-13T06:27:17.257 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:17.244+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.245496+0000) 2019-10-13T06:27:17.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:17.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99819 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:17.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:17.247+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:17.248665+0000) 2019-10-13T06:27:18.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:18.282+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:18.282702+0000) 2019-10-13T06:27:18.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:18.619+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:18.620648+0000) 2019-10-13T06:27:18.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:18.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:18.620655+0000) 2019-10-13T06:27:21.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:21.137+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:21.137814+0000) 2019-10-13T06:27:21.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:21.873+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:21.874351+0000) 2019-10-13T06:27:22.258 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:22.253+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.255062+0000) 2019-10-13T06:27:22.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:22.253+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.255251+0000) 2019-10-13T06:27:22.259 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:22.250+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.251588+0000) 2019-10-13T06:27:22.260 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:22.254+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.255669+0000) 2019-10-13T06:27:22.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:22.254+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.255847+0000) 2019-10-13T06:27:22.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:22.254+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.256171+0000) 2019-10-13T06:27:22.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:22.255+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.256426+0000) 2019-10-13T06:27:22.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:22.253+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.254628+0000) 2019-10-13T06:27:22.263 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:22.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:22.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:22.738+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.739188+0000) 2019-10-13T06:27:22.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:22.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:22.740629+0000) 2019-10-13T06:27:24.240 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:24.235+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:24.236176+0000) 2019-10-13T06:27:27.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:27.259+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.261163+0000) 2019-10-13T06:27:27.265 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:27.259+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.261300+0000) 2019-10-13T06:27:27.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:27.260+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.261471+0000) 2019-10-13T06:27:27.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:27.260+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.262291+0000) 2019-10-13T06:27:27.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:27.260+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.262335+0000) 2019-10-13T06:27:27.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:27.261+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.262681+0000) 2019-10-13T06:27:27.267 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:27.257+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.258158+0000) 2019-10-13T06:27:27.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:27.263+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99879 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:27.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:27.259+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:27.260387+0000) 2019-10-13T06:27:28.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:28.282+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:28.282907+0000) 2019-10-13T06:27:28.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:28.620+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:28.620847+0000) 2019-10-13T06:27:28.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:28.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:28.620822+0000) 2019-10-13T06:27:29.560 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:27:29.560 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:29.565 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:27:29.565 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:31.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:31.140+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:31.140195+0000) 2019-10-13T06:27:31.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:31.874+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:31.874551+0000) 2019-10-13T06:27:32.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:32.261+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.262402+0000) 2019-10-13T06:27:32.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:32.266+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.266429+0000) 2019-10-13T06:27:32.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:32.266+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.266604+0000) 2019-10-13T06:27:32.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:32.266+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.266768+0000) 2019-10-13T06:27:32.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:32.266+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.266951+0000) 2019-10-13T06:27:32.272 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:32.267+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.267443+0000) 2019-10-13T06:27:32.273 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:32.267+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.267617+0000) 2019-10-13T06:27:32.273 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:32.264+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.265562+0000) 2019-10-13T06:27:32.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:32.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:32.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:32.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.739353+0000) 2019-10-13T06:27:32.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:32.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:32.740779+0000) 2019-10-13T06:27:34.240 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:34.236+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:34.236360+0000) 2019-10-13T06:27:37.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:37.265+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.267406+0000) 2019-10-13T06:27:37.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:37.270+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.271275+0000) 2019-10-13T06:27:37.275 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:37.271+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.271589+0000) 2019-10-13T06:27:37.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:37.271+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.271624+0000) 2019-10-13T06:27:37.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:37.271+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.271845+0000) 2019-10-13T06:27:37.276 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:37.271+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.272030+0000) 2019-10-13T06:27:37.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:37.272+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.272860+0000) 2019-10-13T06:27:37.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:37.269+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:37.271187+0000) 2019-10-13T06:27:37.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:37.275+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99941 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:38.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:38.282+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:38.283135+0000) 2019-10-13T06:27:38.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:38.620+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:38.621046+0000) 2019-10-13T06:27:38.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:38.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:38.621042+0000) 2019-10-13T06:27:41.150 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:41.141+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:41.142575+0000) 2019-10-13T06:27:41.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:41.874+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:41.874713+0000) 2019-10-13T06:27:42.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:42.276+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.277217+0000) 2019-10-13T06:27:42.281 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:42.271+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.273692+0000) 2019-10-13T06:27:42.281 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:42.277+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.277638+0000) 2019-10-13T06:27:42.282 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:42.277+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.278192+0000) 2019-10-13T06:27:42.282 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:42.278+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.278432+0000) 2019-10-13T06:27:42.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:42.278+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.278678+0000) 2019-10-13T06:27:42.283 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:42.278+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.278919+0000) 2019-10-13T06:27:42.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:42.275+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.277070+0000) 2019-10-13T06:27:42.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:42.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 99973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:42.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:42.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.739508+0000) 2019-10-13T06:27:42.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:42.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:42.740935+0000) 2019-10-13T06:27:44.240 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:44.236+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:44.236501+0000) 2019-10-13T06:27:47.287 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:47.281+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.283330+0000) 2019-10-13T06:27:47.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:47.282+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.283614+0000) 2019-10-13T06:27:47.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:47.282+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.283821+0000) 2019-10-13T06:27:47.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:47.282+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.284114+0000) 2019-10-13T06:27:47.289 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:47.282+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.284349+0000) 2019-10-13T06:27:47.290 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:47.279+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.280136+0000) 2019-10-13T06:27:47.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:47.283+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.284683+0000) 2019-10-13T06:27:47.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:47.281+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:47.283294+0000) 2019-10-13T06:27:47.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:47.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:48.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:48.281+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:48.283344+0000) 2019-10-13T06:27:48.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:48.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:48.621258+0000) 2019-10-13T06:27:48.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:48.620+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:48.621264+0000) 2019-10-13T06:27:51.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:51.144+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:51.145011+0000) 2019-10-13T06:27:51.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:51.873+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:51.874903+0000) 2019-10-13T06:27:52.292 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:52.287+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.288908+0000) 2019-10-13T06:27:52.293 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:52.285+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.285790+0000) 2019-10-13T06:27:52.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:52.288+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.289492+0000) 2019-10-13T06:27:52.293 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:52.288+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.289903+0000) 2019-10-13T06:27:52.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:52.288+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.289908+0000) 2019-10-13T06:27:52.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:52.288+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.290148+0000) 2019-10-13T06:27:52.294 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:52.286+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.287373+0000) 2019-10-13T06:27:52.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:52.288+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.290314+0000) 2019-10-13T06:27:52.297 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:52.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:52.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:52.738+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.739646+0000) 2019-10-13T06:27:52.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:52.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:52.741124+0000) 2019-10-13T06:27:54.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:54.235+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:54.236624+0000) 2019-10-13T06:27:57.298 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:27:57.293+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.294907+0000) 2019-10-13T06:27:57.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:57.293+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.295020+0000) 2019-10-13T06:27:57.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:27:57.293+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.295299+0000) 2019-10-13T06:27:57.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:57.294+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.295393+0000) 2019-10-13T06:27:57.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:27:57.291+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.292267+0000) 2019-10-13T06:27:57.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:57.294+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.295762+0000) 2019-10-13T06:27:57.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:27:57.294+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.295844+0000) 2019-10-13T06:27:57.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:27:57.294+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:57.295342+0000) 2019-10-13T06:27:57.302 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:27:57.297+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100063 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:27:58.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:27:58.282+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:58.283523+0000) 2019-10-13T06:27:58.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:27:58.620+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:58.621472+0000) 2019-10-13T06:27:58.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:27:58.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:27:58.621472+0000) 2019-10-13T06:27:59.715 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:27:59.716 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:59.721 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:27:59.721 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:01.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:01.146+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:01.147362+0000) 2019-10-13T06:28:01.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:01.874+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:01.875031+0000) 2019-10-13T06:28:02.305 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:02.296+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.297682+0000) 2019-10-13T06:28:02.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:02.300+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.301623+0000) 2019-10-13T06:28:02.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:02.300+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.301771+0000) 2019-10-13T06:28:02.306 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:02.300+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.302149+0000) 2019-10-13T06:28:02.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:02.300+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.302168+0000) 2019-10-13T06:28:02.307 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:02.301+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.302759+0000) 2019-10-13T06:28:02.307 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:02.301+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.302757+0000) 2019-10-13T06:28:02.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:02.299+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.300588+0000) 2019-10-13T06:28:02.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:02.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:02.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:02.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.739762+0000) 2019-10-13T06:28:02.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:02.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:02.741308+0000) 2019-10-13T06:28:04.240 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:04.236+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:04.236796+0000) 2019-10-13T06:28:07.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:07.305+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.306988+0000) 2019-10-13T06:28:07.312 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:07.302+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.303383+0000) 2019-10-13T06:28:07.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:07.306+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.307808+0000) 2019-10-13T06:28:07.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:07.306+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.307789+0000) 2019-10-13T06:28:07.313 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:07.306+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.307992+0000) 2019-10-13T06:28:07.313 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:07.306+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.308159+0000) 2019-10-13T06:28:07.314 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:07.307+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.308404+0000) 2019-10-13T06:28:07.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:07.304+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:07.305595+0000) 2019-10-13T06:28:07.315 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:07.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:08.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:08.282+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:08.283645+0000) 2019-10-13T06:28:08.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:08.621+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:08.621680+0000) 2019-10-13T06:28:08.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:08.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:08.621671+0000) 2019-10-13T06:28:11.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:11.149+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:11.149747+0000) 2019-10-13T06:28:11.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:11.874+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:11.875206+0000) 2019-10-13T06:28:12.316 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:12.308+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.309277+0000) 2019-10-13T06:28:12.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:12.311+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.313367+0000) 2019-10-13T06:28:12.318 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:12.312+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.313764+0000) 2019-10-13T06:28:12.318 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:12.312+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.313937+0000) 2019-10-13T06:28:12.319 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:12.312+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.314051+0000) 2019-10-13T06:28:12.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:12.312+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.314212+0000) 2019-10-13T06:28:12.319 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:12.313+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.314707+0000) 2019-10-13T06:28:12.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:12.310+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.311332+0000) 2019-10-13T06:28:12.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:12.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:12.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:12.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.739924+0000) 2019-10-13T06:28:12.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:12.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:12.741462+0000) 2019-10-13T06:28:14.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:14.236+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:14.237025+0000) 2019-10-13T06:28:17.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:17.318+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.319085+0000) 2019-10-13T06:28:17.324 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:17.319+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.319437+0000) 2019-10-13T06:28:17.324 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:17.319+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.319467+0000) 2019-10-13T06:28:17.324 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:17.319+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.319557+0000) 2019-10-13T06:28:17.325 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:17.319+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.319842+0000) 2019-10-13T06:28:17.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:17.319+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.320025+0000) 2019-10-13T06:28:17.326 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:17.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.316209+0000) 2019-10-13T06:28:17.326 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:17.316+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:17.317505+0000) 2019-10-13T06:28:17.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:17.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:18.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:18.283+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:18.283793+0000) 2019-10-13T06:28:18.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:18.621+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:18.621890+0000) 2019-10-13T06:28:18.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:18.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:18.621892+0000) 2019-10-13T06:28:21.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:21.151+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:21.151725+0000) 2019-10-13T06:28:21.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:21.875+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:21.875427+0000) 2019-10-13T06:28:22.327 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:22.323+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.323915+0000) 2019-10-13T06:28:22.327 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:22.320+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.320800+0000) 2019-10-13T06:28:22.328 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:22.324+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.324779+0000) 2019-10-13T06:28:22.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:22.324+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.324916+0000) 2019-10-13T06:28:22.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:22.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.325319+0000) 2019-10-13T06:28:22.329 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:22.325+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.325423+0000) 2019-10-13T06:28:22.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:22.325+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.325680+0000) 2019-10-13T06:28:22.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:22.322+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.323453+0000) 2019-10-13T06:28:22.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:22.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:22.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:22.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.740062+0000) 2019-10-13T06:28:22.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:22.741+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:22.741607+0000) 2019-10-13T06:28:24.240 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:24.236+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:24.237245+0000) 2019-10-13T06:28:27.333 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:27.329+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.330137+0000) 2019-10-13T06:28:27.335 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:27.324+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.326461+0000) 2019-10-13T06:28:27.335 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:27.330+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.330776+0000) 2019-10-13T06:28:27.335 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:27.330+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.331010+0000) 2019-10-13T06:28:27.335 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:27.330+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.331068+0000) 2019-10-13T06:28:27.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:27.331+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.331538+0000) 2019-10-13T06:28:27.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:27.331+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.331678+0000) 2019-10-13T06:28:27.336 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:27.327+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:27.329672+0000) 2019-10-13T06:28:27.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:27.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:28.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:28.283+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:28.283962+0000) 2019-10-13T06:28:28.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:28.621+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:28.622086+0000) 2019-10-13T06:28:28.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:28.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:28.622089+0000) 2019-10-13T06:28:29.894 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:28:29.895 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:29.899 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:28:29.899 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:31.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:31.153+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:31.154057+0000) 2019-10-13T06:28:31.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:31.875+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:31.875620+0000) 2019-10-13T06:28:32.340 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:32.336+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.336659+0000) 2019-10-13T06:28:32.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:32.331+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.332971+0000) 2019-10-13T06:28:32.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:32.336+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.337090+0000) 2019-10-13T06:28:32.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:32.336+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.336903+0000) 2019-10-13T06:28:32.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:32.336+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.337312+0000) 2019-10-13T06:28:32.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:32.337+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.337760+0000) 2019-10-13T06:28:32.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:32.338+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.338499+0000) 2019-10-13T06:28:32.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:32.334+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.336390+0000) 2019-10-13T06:28:32.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:32.340+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100277 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:32.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:32.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.740257+0000) 2019-10-13T06:28:32.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:32.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:32.741761+0000) 2019-10-13T06:28:34.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:34.236+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:34.237425+0000) 2019-10-13T06:28:37.346 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:37.340+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.342334+0000) 2019-10-13T06:28:37.347 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:37.341+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.342474+0000) 2019-10-13T06:28:37.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:37.341+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.342694+0000) 2019-10-13T06:28:37.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:37.341+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.343175+0000) 2019-10-13T06:28:37.347 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:37.341+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.343271+0000) 2019-10-13T06:28:37.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:37.341+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.343388+0000) 2019-10-13T06:28:37.348 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:37.337+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.339081+0000) 2019-10-13T06:28:37.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:37.339+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:37.341005+0000) 2019-10-13T06:28:37.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:37.343+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100307 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:38.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:38.283+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:38.284107+0000) 2019-10-13T06:28:38.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:38.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:38.622264+0000) 2019-10-13T06:28:38.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:38.621+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:38.622257+0000) 2019-10-13T06:28:41.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:41.155+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:41.156466+0000) 2019-10-13T06:28:41.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:41.874+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:41.875795+0000) 2019-10-13T06:28:42.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:42.345+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.347130+0000) 2019-10-13T06:28:42.351 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:42.346+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.347422+0000) 2019-10-13T06:28:42.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:42.346+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.347772+0000) 2019-10-13T06:28:42.352 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:42.346+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.347917+0000) 2019-10-13T06:28:42.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:42.346+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.348052+0000) 2019-10-13T06:28:42.352 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:42.347+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.348567+0000) 2019-10-13T06:28:42.352 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:42.343+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.344566+0000) 2019-10-13T06:28:42.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:42.346+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.347372+0000) 2019-10-13T06:28:42.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:42.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100339 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:42.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:42.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.740444+0000) 2019-10-13T06:28:42.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:42.741+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:42.741941+0000) 2019-10-13T06:28:44.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:44.236+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:44.237566+0000) 2019-10-13T06:28:47.356 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:47.351+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.352745+0000) 2019-10-13T06:28:47.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:47.351+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.353047+0000) 2019-10-13T06:28:47.357 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:47.351+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.353275+0000) 2019-10-13T06:28:47.357 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:47.352+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.353459+0000) 2019-10-13T06:28:47.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:47.349+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.350285+0000) 2019-10-13T06:28:47.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:47.352+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.353811+0000) 2019-10-13T06:28:47.359 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:47.352+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.354338+0000) 2019-10-13T06:28:47.360 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:47.351+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:47.352722+0000) 2019-10-13T06:28:47.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:47.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:48.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:48.282+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:48.284266+0000) 2019-10-13T06:28:48.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:48.621+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:48.622445+0000) 2019-10-13T06:28:48.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:48.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:48.622440+0000) 2019-10-13T06:28:51.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:51.158+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:51.158817+0000) 2019-10-13T06:28:51.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:51.875+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:51.875980+0000) 2019-10-13T06:28:52.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:52.357+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.359278+0000) 2019-10-13T06:28:52.363 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:52.358+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.359548+0000) 2019-10-13T06:28:52.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:52.355+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.355869+0000) 2019-10-13T06:28:52.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:52.358+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.359801+0000) 2019-10-13T06:28:52.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:52.358+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.359968+0000) 2019-10-13T06:28:52.364 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:52.358+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.360079+0000) 2019-10-13T06:28:52.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:52.358+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.360252+0000) 2019-10-13T06:28:52.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:52.356+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.357678+0000) 2019-10-13T06:28:52.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:52.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:52.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:52.739+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.740585+0000) 2019-10-13T06:28:52.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:52.741+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:52.742118+0000) 2019-10-13T06:28:54.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:54.237+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:54.237697+0000) 2019-10-13T06:28:57.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:28:57.363+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.365271+0000) 2019-10-13T06:28:57.369 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:57.364+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.365516+0000) 2019-10-13T06:28:57.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:28:57.364+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.365629+0000) 2019-10-13T06:28:57.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:57.364+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.365748+0000) 2019-10-13T06:28:57.370 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:28:57.361+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.362099+0000) 2019-10-13T06:28:57.370 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:57.364+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.366264+0000) 2019-10-13T06:28:57.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:28:57.365+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.366477+0000) 2019-10-13T06:28:57.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:28:57.363+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:57.364290+0000) 2019-10-13T06:28:57.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:28:57.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:28:58.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:28:58.283+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:58.284449+0000) 2019-10-13T06:28:58.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:28:58.622+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:58.622659+0000) 2019-10-13T06:28:58.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:28:58.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:28:58.622649+0000) 2019-10-13T06:28:59.994 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:28:59.994 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:59.998 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:28:59.998 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:01.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:01.160+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:01.161163+0000) 2019-10-13T06:29:01.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:01.875+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:01.876194+0000) 2019-10-13T06:29:02.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:02.370+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.371195+0000) 2019-10-13T06:29:02.375 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:02.370+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.371226+0000) 2019-10-13T06:29:02.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:02.370+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.371302+0000) 2019-10-13T06:29:02.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:02.371+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.371692+0000) 2019-10-13T06:29:02.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:02.371+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.371860+0000) 2019-10-13T06:29:02.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:02.371+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.372097+0000) 2019-10-13T06:29:02.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:02.367+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.368493+0000) 2019-10-13T06:29:02.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:02.368+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.369152+0000) 2019-10-13T06:29:02.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:02.374+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:02.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:02.740+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.740744+0000) 2019-10-13T06:29:02.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:02.741+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:02.742272+0000) 2019-10-13T06:29:04.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:04.237+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:04.237876+0000) 2019-10-13T06:29:07.380 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:07.375+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.376292+0000) 2019-10-13T06:29:07.380 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:07.376+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.376541+0000) 2019-10-13T06:29:07.381 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:07.376+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.377260+0000) 2019-10-13T06:29:07.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:07.377+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.377651+0000) 2019-10-13T06:29:07.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:07.377+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.377962+0000) 2019-10-13T06:29:07.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:07.373+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.374123+0000) 2019-10-13T06:29:07.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:07.378+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.378440+0000) 2019-10-13T06:29:07.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:07.374+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:07.375477+0000) 2019-10-13T06:29:07.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:07.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:08.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:08.284+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:08.284585+0000) 2019-10-13T06:29:08.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:08.622+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:08.622885+0000) 2019-10-13T06:29:08.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:08.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:08.622877+0000) 2019-10-13T06:29:11.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:11.163+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:11.163546+0000) 2019-10-13T06:29:11.880 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:11.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:11.876326+0000) 2019-10-13T06:29:12.386 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:12.382+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.382586+0000) 2019-10-13T06:29:12.387 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:12.382+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.382911+0000) 2019-10-13T06:29:12.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:12.382+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.383139+0000) 2019-10-13T06:29:12.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:12.382+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.382979+0000) 2019-10-13T06:29:12.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:12.382+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.383388+0000) 2019-10-13T06:29:12.388 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:12.379+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.379809+0000) 2019-10-13T06:29:12.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:12.383+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.383919+0000) 2019-10-13T06:29:12.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:12.381+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.381809+0000) 2019-10-13T06:29:12.390 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:12.386+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:12.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:12.740+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.740909+0000) 2019-10-13T06:29:12.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:12.742+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:12.742373+0000) 2019-10-13T06:29:14.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:14.237+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:14.237997+0000) 2019-10-13T06:29:17.392 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:17.388+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.388485+0000) 2019-10-13T06:29:17.392 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:17.388+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.389220+0000) 2019-10-13T06:29:17.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:17.383+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.385647+0000) 2019-10-13T06:29:17.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:17.389+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.389906+0000) 2019-10-13T06:29:17.394 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:17.389+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.390156+0000) 2019-10-13T06:29:17.394 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:17.390+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.390504+0000) 2019-10-13T06:29:17.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:17.390+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.390842+0000) 2019-10-13T06:29:17.395 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:17.386+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:17.388039+0000) 2019-10-13T06:29:17.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:17.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:18.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:18.284+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:18.284753+0000) 2019-10-13T06:29:18.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:18.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:18.623053+0000) 2019-10-13T06:29:18.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:18.623+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:18.623063+0000) 2019-10-13T06:29:21.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:21.165+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:21.166192+0000) 2019-10-13T06:29:21.880 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:21.875+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:21.876488+0000) 2019-10-13T06:29:22.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:22.393+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.394980+0000) 2019-10-13T06:29:22.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:22.394+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.396387+0000) 2019-10-13T06:29:22.400 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:22.394+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.396252+0000) 2019-10-13T06:29:22.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:22.395+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.396675+0000) 2019-10-13T06:29:22.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:22.395+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.396602+0000) 2019-10-13T06:29:22.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:22.395+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.396903+0000) 2019-10-13T06:29:22.401 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:22.391+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.393361+0000) 2019-10-13T06:29:22.402 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:22.393+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.395222+0000) 2019-10-13T06:29:22.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:22.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:22.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:22.740+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.741077+0000) 2019-10-13T06:29:22.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:22.740+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:22.742454+0000) 2019-10-13T06:29:24.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:24.237+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:24.238173+0000) 2019-10-13T06:29:27.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:27.395+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.397394+0000) 2019-10-13T06:29:27.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:27.399+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.401318+0000) 2019-10-13T06:29:27.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:27.400+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.401450+0000) 2019-10-13T06:29:27.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:27.400+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.401601+0000) 2019-10-13T06:29:27.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:27.400+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.401800+0000) 2019-10-13T06:29:27.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:27.400+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.402157+0000) 2019-10-13T06:29:27.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:27.401+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.402510+0000) 2019-10-13T06:29:27.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:27.397+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:27.399392+0000) 2019-10-13T06:29:27.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:27.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:28.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:28.284+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:28.284901+0000) 2019-10-13T06:29:28.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:28.622+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:28.623219+0000) 2019-10-13T06:29:28.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:28.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:28.623220+0000) 2019-10-13T06:29:30.083 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:29:30.083 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:30.087 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:29:30.087 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:31.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:31.167+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:31.168490+0000) 2019-10-13T06:29:31.880 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:31.875+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:31.876658+0000) 2019-10-13T06:29:32.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:32.403+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.404196+0000) 2019-10-13T06:29:32.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:32.406+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.407766+0000) 2019-10-13T06:29:32.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:32.406+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.408234+0000) 2019-10-13T06:29:32.414 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:32.407+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.408519+0000) 2019-10-13T06:29:32.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:32.407+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.408705+0000) 2019-10-13T06:29:32.414 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:32.407+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.408935+0000) 2019-10-13T06:29:32.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:32.406+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.408363+0000) 2019-10-13T06:29:32.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:32.406+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.407182+0000) 2019-10-13T06:29:32.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:32.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:32.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:32.740+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.741260+0000) 2019-10-13T06:29:32.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:32.741+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:32.742622+0000) 2019-10-13T06:29:34.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:34.237+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:34.238340+0000) 2019-10-13T06:29:37.417 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:37.409+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.410104+0000) 2019-10-13T06:29:37.418 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:37.412+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.414123+0000) 2019-10-13T06:29:37.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:37.412+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.414254+0000) 2019-10-13T06:29:37.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:37.413+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.414415+0000) 2019-10-13T06:29:37.419 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:37.413+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.414653+0000) 2019-10-13T06:29:37.419 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:37.413+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.414913+0000) 2019-10-13T06:29:37.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:37.413+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.415304+0000) 2019-10-13T06:29:37.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:37.411+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:37.412005+0000) 2019-10-13T06:29:37.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:37.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:38.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:38.284+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:38.285087+0000) 2019-10-13T06:29:38.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:38.622+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:38.623449+0000) 2019-10-13T06:29:38.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:38.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:38.623447+0000) 2019-10-13T06:29:41.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:41.170+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:41.170816+0000) 2019-10-13T06:29:41.880 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:41.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:41.876816+0000) 2019-10-13T06:29:42.423 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:42.417+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.419345+0000) 2019-10-13T06:29:42.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:42.415+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.416000+0000) 2019-10-13T06:29:42.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:42.418+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.420248+0000) 2019-10-13T06:29:42.424 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:42.419+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.420442+0000) 2019-10-13T06:29:42.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:42.419+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.420710+0000) 2019-10-13T06:29:42.425 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:42.419+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.420604+0000) 2019-10-13T06:29:42.425 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:42.419+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.420887+0000) 2019-10-13T06:29:42.426 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:42.417+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.417966+0000) 2019-10-13T06:29:42.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:42.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:42.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:42.740+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.741473+0000) 2019-10-13T06:29:42.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:42.742+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:42.742808+0000) 2019-10-13T06:29:44.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:44.238+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:44.238572+0000) 2019-10-13T06:29:47.430 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:47.421+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.422625+0000) 2019-10-13T06:29:47.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:47.426+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.426580+0000) 2019-10-13T06:29:47.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:47.426+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.426944+0000) 2019-10-13T06:29:47.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:47.426+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.427342+0000) 2019-10-13T06:29:47.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:47.427+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.427591+0000) 2019-10-13T06:29:47.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:47.427+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.427710+0000) 2019-10-13T06:29:47.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:47.423+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.424611+0000) 2019-10-13T06:29:47.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:47.426+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:47.427079+0000) 2019-10-13T06:29:47.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:47.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100734 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:48.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:48.284+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:48.285287+0000) 2019-10-13T06:29:48.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:48.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:48.623626+0000) 2019-10-13T06:29:48.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:48.623+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:48.623650+0000) 2019-10-13T06:29:51.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:51.172+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:51.173285+0000) 2019-10-13T06:29:51.880 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:51.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:51.877037+0000) 2019-10-13T06:29:52.436 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:52.431+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.431507+0000) 2019-10-13T06:29:52.436 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:52.431+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.431790+0000) 2019-10-13T06:29:52.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:52.431+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.431658+0000) 2019-10-13T06:29:52.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:52.431+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.432011+0000) 2019-10-13T06:29:52.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:52.432+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.432391+0000) 2019-10-13T06:29:52.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:52.432+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.432672+0000) 2019-10-13T06:29:52.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:52.427+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.428369+0000) 2019-10-13T06:29:52.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:52.429+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.430309+0000) 2019-10-13T06:29:52.439 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:52.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:52.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:52.741+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.741593+0000) 2019-10-13T06:29:52.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:52.742+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:52.742973+0000) 2019-10-13T06:29:54.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:54.238+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:54.238765+0000) 2019-10-13T06:29:57.440 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:29:57.431+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.432568+0000) 2019-10-13T06:29:57.441 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:57.436+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.436534+0000) 2019-10-13T06:29:57.441 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:57.436+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.436684+0000) 2019-10-13T06:29:57.442 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:29:57.436+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.436926+0000) 2019-10-13T06:29:57.442 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:29:57.437+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.437401+0000) 2019-10-13T06:29:57.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:57.437+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.437706+0000) 2019-10-13T06:29:57.442 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:29:57.437+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.437794+0000) 2019-10-13T06:29:57.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:29:57.435+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:57.435867+0000) 2019-10-13T06:29:57.444 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:29:57.440+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:29:58.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:29:58.285+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:58.285431+0000) 2019-10-13T06:29:58.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:29:58.623+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:58.623818+0000) 2019-10-13T06:29:58.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:29:58.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:29:58.623811+0000) 2019-10-13T06:30:00.189 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:30:00.189 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:00.193 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:30:00.193 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:01.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:01.175+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:01.175751+0000) 2019-10-13T06:30:01.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:01.877+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:01.877187+0000) 2019-10-13T06:30:02.448 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:02.439+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.439749+0000) 2019-10-13T06:30:02.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:02.442+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.442825+0000) 2019-10-13T06:30:02.449 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:02.442+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.443015+0000) 2019-10-13T06:30:02.449 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:02.443+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.443394+0000) 2019-10-13T06:30:02.450 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:02.443+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.443806+0000) 2019-10-13T06:30:02.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:02.443+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.444113+0000) 2019-10-13T06:30:02.450 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:02.443+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.444091+0000) 2019-10-13T06:30:02.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:02.441+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.442513+0000) 2019-10-13T06:30:02.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:02.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:02.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:02.741+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.741735+0000) 2019-10-13T06:30:02.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:02.742+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:02.743130+0000) 2019-10-13T06:30:04.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:04.238+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:04.238905+0000) 2019-10-13T06:30:07.453 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:07.447+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.449148+0000) 2019-10-13T06:30:07.454 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:07.448+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.449465+0000) 2019-10-13T06:30:07.454 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:07.448+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.449561+0000) 2019-10-13T06:30:07.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:07.448+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.449752+0000) 2019-10-13T06:30:07.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:07.448+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.450044+0000) 2019-10-13T06:30:07.455 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:07.445+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.446437+0000) 2019-10-13T06:30:07.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:07.446+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.447585+0000) 2019-10-13T06:30:07.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:07.449+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:07.450412+0000) 2019-10-13T06:30:07.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:07.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:08.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:08.284+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:08.285596+0000) 2019-10-13T06:30:08.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:08.623+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:08.624014+0000) 2019-10-13T06:30:08.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:08.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:08.624009+0000) 2019-10-13T06:30:11.186 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:11.177+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:11.178754+0000) 2019-10-13T06:30:11.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:11.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:11.877418+0000) 2019-10-13T06:30:12.460 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:12.449+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.451180+0000) 2019-10-13T06:30:12.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:12.450+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.452441+0000) 2019-10-13T06:30:12.460 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:12.452+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.453819+0000) 2019-10-13T06:30:12.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:12.452+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.454115+0000) 2019-10-13T06:30:12.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:12.452+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.454266+0000) 2019-10-13T06:30:12.461 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:12.453+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.454424+0000) 2019-10-13T06:30:12.461 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:12.453+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.454635+0000) 2019-10-13T06:30:12.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:12.453+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.455114+0000) 2019-10-13T06:30:12.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:12.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:12.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:12.741+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.741898+0000) 2019-10-13T06:30:12.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:12.741+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:12.743267+0000) 2019-10-13T06:30:14.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:14.238+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:14.239090+0000) 2019-10-13T06:30:17.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:17.458+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.460073+0000) 2019-10-13T06:30:17.464 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:17.458+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.460246+0000) 2019-10-13T06:30:17.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:17.459+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.460423+0000) 2019-10-13T06:30:17.464 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:17.459+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.460672+0000) 2019-10-13T06:30:17.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:17.455+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.457136+0000) 2019-10-13T06:30:17.465 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:17.459+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.461071+0000) 2019-10-13T06:30:17.465 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:17.459+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.461151+0000) 2019-10-13T06:30:17.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:17.456+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:17.458359+0000) 2019-10-13T06:30:17.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:17.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:18.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:18.285+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:18.285771+0000) 2019-10-13T06:30:18.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:18.623+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:18.624222+0000) 2019-10-13T06:30:18.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:18.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:18.624202+0000) 2019-10-13T06:30:21.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:21.180+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:21.181284+0000) 2019-10-13T06:30:21.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:21.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:21.877563+0000) 2019-10-13T06:30:22.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:22.464+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.466345+0000) 2019-10-13T06:30:22.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:22.461+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.462715+0000) 2019-10-13T06:30:22.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:22.465+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.466899+0000) 2019-10-13T06:30:22.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:22.465+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.467032+0000) 2019-10-13T06:30:22.472 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:22.465+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.467077+0000) 2019-10-13T06:30:22.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:22.465+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.467217+0000) 2019-10-13T06:30:22.472 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:22.466+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.467743+0000) 2019-10-13T06:30:22.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:22.464+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.464996+0000) 2019-10-13T06:30:22.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:22.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:22.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:22.741+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.742081+0000) 2019-10-13T06:30:22.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:22.742+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:22.743420+0000) 2019-10-13T06:30:24.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:24.238+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:24.239278+0000) 2019-10-13T06:30:27.475 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:27.467+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.468470+0000) 2019-10-13T06:30:27.476 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:27.471+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.472468+0000) 2019-10-13T06:30:27.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:27.471+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.472543+0000) 2019-10-13T06:30:27.477 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:27.471+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.472739+0000) 2019-10-13T06:30:27.477 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:27.471+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.472843+0000) 2019-10-13T06:30:27.477 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:27.471+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.473041+0000) 2019-10-13T06:30:27.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:27.471+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.473225+0000) 2019-10-13T06:30:27.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:27.471+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:27.472433+0000) 2019-10-13T06:30:27.480 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:27.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 100974 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:28.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:28.285+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:28.285909+0000) 2019-10-13T06:30:28.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:28.623+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:28.624380+0000) 2019-10-13T06:30:28.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:28.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:28.624378+0000) 2019-10-13T06:30:30.303 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:30:30.304 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:30.309 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:30:30.309 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:31.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:31.182+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:31.183635+0000) 2019-10-13T06:30:31.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:31.877+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:31.877706+0000) 2019-10-13T06:30:32.481 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:32.477+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.477697+0000) 2019-10-13T06:30:32.481 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:32.477+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.477842+0000) 2019-10-13T06:30:32.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:32.473+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.474681+0000) 2019-10-13T06:30:32.483 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:32.477+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.478261+0000) 2019-10-13T06:30:32.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:32.478+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.478553+0000) 2019-10-13T06:30:32.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:32.478+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.478705+0000) 2019-10-13T06:30:32.483 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:32.478+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.479016+0000) 2019-10-13T06:30:32.485 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:32.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101005 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:32.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:32.477+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.478327+0000) 2019-10-13T06:30:32.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:32.741+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.742277+0000) 2019-10-13T06:30:32.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:32.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:32.743580+0000) 2019-10-13T06:30:34.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:34.239+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:34.239445+0000) 2019-10-13T06:30:37.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:37.483+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.483979+0000) 2019-10-13T06:30:37.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:37.483+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.484262+0000) 2019-10-13T06:30:37.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:37.483+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.484328+0000) 2019-10-13T06:30:37.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:37.484+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.484469+0000) 2019-10-13T06:30:37.490 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:37.480+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.481584+0000) 2019-10-13T06:30:37.490 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:37.484+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.484819+0000) 2019-10-13T06:30:37.490 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:37.484+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.484974+0000) 2019-10-13T06:30:37.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:37.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101035 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:37.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:37.482+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:37.483090+0000) 2019-10-13T06:30:38.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:38.285+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:38.286093+0000) 2019-10-13T06:30:38.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:38.624+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:38.624523+0000) 2019-10-13T06:30:38.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:38.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:38.624577+0000) 2019-10-13T06:30:41.193 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:41.185+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:41.185939+0000) 2019-10-13T06:30:41.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:41.877+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:41.877849+0000) 2019-10-13T06:30:42.492 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:42.487+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.488180+0000) 2019-10-13T06:30:42.492 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:42.487+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.488345+0000) 2019-10-13T06:30:42.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:42.488+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.488594+0000) 2019-10-13T06:30:42.492 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:42.488+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.488676+0000) 2019-10-13T06:30:42.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:42.488+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.488756+0000) 2019-10-13T06:30:42.493 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:42.484+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.485350+0000) 2019-10-13T06:30:42.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:42.488+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.489349+0000) 2019-10-13T06:30:42.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:42.487+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.487950+0000) 2019-10-13T06:30:42.496 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:42.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:42.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:42.742+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.742444+0000) 2019-10-13T06:30:42.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:42.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:42.743767+0000) 2019-10-13T06:30:44.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:44.239+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:44.239587+0000) 2019-10-13T06:30:47.498 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:47.493+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.494253+0000) 2019-10-13T06:30:47.499 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:47.490+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.490829+0000) 2019-10-13T06:30:47.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:47.494+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.494918+0000) 2019-10-13T06:30:47.499 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:47.494+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.495041+0000) 2019-10-13T06:30:47.499 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:47.494+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.495275+0000) 2019-10-13T06:30:47.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:47.495+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.495425+0000) 2019-10-13T06:30:47.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:47.491+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.492691+0000) 2019-10-13T06:30:47.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:47.495+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:47.495843+0000) 2019-10-13T06:30:47.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:47.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101094 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:48.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:48.286+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:48.286281+0000) 2019-10-13T06:30:48.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:48.624+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:48.624728+0000) 2019-10-13T06:30:48.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:48.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:48.624735+0000) 2019-10-13T06:30:51.195 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:51.188+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:51.188330+0000) 2019-10-13T06:30:51.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:51.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:51.878040+0000) 2019-10-13T06:30:52.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:52.499+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.500431+0000) 2019-10-13T06:30:52.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:52.499+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.500650+0000) 2019-10-13T06:30:52.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:52.499+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.500862+0000) 2019-10-13T06:30:52.507 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:52.499+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.501183+0000) 2019-10-13T06:30:52.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:52.499+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.501258+0000) 2019-10-13T06:30:52.507 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:52.500+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.501527+0000) 2019-10-13T06:30:52.508 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:52.497+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.497887+0000) 2019-10-13T06:30:52.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:52.499+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.499956+0000) 2019-10-13T06:30:52.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:52.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:52.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:52.741+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.742638+0000) 2019-10-13T06:30:52.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:52.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:52.743928+0000) 2019-10-13T06:30:54.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:54.238+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:54.239751+0000) 2019-10-13T06:30:57.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:57.504+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.505821+0000) 2019-10-13T06:30:57.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:30:57.504+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.506029+0000) 2019-10-13T06:30:57.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:30:57.504+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.506269+0000) 2019-10-13T06:30:57.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:30:57.505+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.506572+0000) 2019-10-13T06:30:57.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:57.505+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.506642+0000) 2019-10-13T06:30:57.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:57.505+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.506811+0000) 2019-10-13T06:30:57.511 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:30:57.502+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.503661+0000) 2019-10-13T06:30:57.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:30:57.504+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:57.505177+0000) 2019-10-13T06:30:57.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:30:57.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:30:58.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:30:58.285+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:58.286462+0000) 2019-10-13T06:30:58.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:30:58.624+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:58.624925+0000) 2019-10-13T06:30:58.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:30:58.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:30:58.624923+0000) 2019-10-13T06:31:00.489 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:31:00.490 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:00.496 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:31:00.497 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:01.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:01.189+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:01.190628+0000) 2019-10-13T06:31:01.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:01.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:01.878201+0000) 2019-10-13T06:31:02.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:02.510+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.511673+0000) 2019-10-13T06:31:02.515 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:02.510+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.511982+0000) 2019-10-13T06:31:02.516 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:02.506+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.508026+0000) 2019-10-13T06:31:02.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:02.510+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.512294+0000) 2019-10-13T06:31:02.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:02.511+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.512416+0000) 2019-10-13T06:31:02.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:02.511+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.512943+0000) 2019-10-13T06:31:02.517 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:02.511+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.512590+0000) 2019-10-13T06:31:02.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:02.508+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.510433+0000) 2019-10-13T06:31:02.519 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:02.514+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:02.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:02.742+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.742837+0000) 2019-10-13T06:31:02.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:02.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:02.744084+0000) 2019-10-13T06:31:04.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:04.239+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:04.239925+0000) 2019-10-13T06:31:07.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:07.516+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.517675+0000) 2019-10-13T06:31:07.522 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:07.516+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.518253+0000) 2019-10-13T06:31:07.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:07.517+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.518468+0000) 2019-10-13T06:31:07.522 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:07.517+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.518660+0000) 2019-10-13T06:31:07.523 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:07.513+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.514758+0000) 2019-10-13T06:31:07.523 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:07.517+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.518963+0000) 2019-10-13T06:31:07.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:07.517+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.519131+0000) 2019-10-13T06:31:07.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:07.515+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:07.517448+0000) 2019-10-13T06:31:07.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:07.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:08.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:08.285+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:08.286602+0000) 2019-10-13T06:31:08.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:08.624+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:08.625120+0000) 2019-10-13T06:31:08.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:08.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:08.625121+0000) 2019-10-13T06:31:11.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:11.192+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:11.192974+0000) 2019-10-13T06:31:11.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:11.876+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:11.878342+0000) 2019-10-13T06:31:12.527 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:12.522+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.523889+0000) 2019-10-13T06:31:12.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:12.518+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.520418+0000) 2019-10-13T06:31:12.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:12.522+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.524361+0000) 2019-10-13T06:31:12.529 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:12.523+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.524473+0000) 2019-10-13T06:31:12.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:12.523+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.524604+0000) 2019-10-13T06:31:12.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:12.523+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.524668+0000) 2019-10-13T06:31:12.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:12.523+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.524913+0000) 2019-10-13T06:31:12.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:12.521+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.523152+0000) 2019-10-13T06:31:12.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:12.525+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101246 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:12.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:12.742+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.742974+0000) 2019-10-13T06:31:12.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:12.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:12.744246+0000) 2019-10-13T06:31:14.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:14.239+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:14.240117+0000) 2019-10-13T06:31:17.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:17.527+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.528865+0000) 2019-10-13T06:31:17.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:17.527+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.528772+0000) 2019-10-13T06:31:17.534 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:17.524+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.525241+0000) 2019-10-13T06:31:17.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:17.527+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.529221+0000) 2019-10-13T06:31:17.534 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:17.528+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.529494+0000) 2019-10-13T06:31:17.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:17.528+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.529780+0000) 2019-10-13T06:31:17.535 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:17.528+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.529889+0000) 2019-10-13T06:31:17.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:17.526+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:17.527701+0000) 2019-10-13T06:31:17.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:17.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:18.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:18.286+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:18.286747+0000) 2019-10-13T06:31:18.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:18.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:18.625249+0000) 2019-10-13T06:31:18.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:18.624+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:18.625301+0000) 2019-10-13T06:31:21.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:21.194+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:21.195239+0000) 2019-10-13T06:31:21.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:21.878+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:21.878569+0000) 2019-10-13T06:31:22.537 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:22.529+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.530404+0000) 2019-10-13T06:31:22.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:22.533+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.534364+0000) 2019-10-13T06:31:22.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:22.534+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.534850+0000) 2019-10-13T06:31:22.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:22.534+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.535038+0000) 2019-10-13T06:31:22.539 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:22.534+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.535072+0000) 2019-10-13T06:31:22.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:22.535+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.535484+0000) 2019-10-13T06:31:22.540 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:22.535+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.535521+0000) 2019-10-13T06:31:22.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:22.532+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.532968+0000) 2019-10-13T06:31:22.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:22.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:22.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:22.742+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.743143+0000) 2019-10-13T06:31:22.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:22.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:22.744400+0000) 2019-10-13T06:31:24.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:24.239+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:24.240325+0000) 2019-10-13T06:31:27.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:27.540+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.541262+0000) 2019-10-13T06:31:27.546 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:27.537+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.538302+0000) 2019-10-13T06:31:27.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:27.541+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.541627+0000) 2019-10-13T06:31:27.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:27.541+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.541798+0000) 2019-10-13T06:31:27.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:27.541+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.541894+0000) 2019-10-13T06:31:27.547 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:27.541+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.542193+0000) 2019-10-13T06:31:27.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:27.542+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.542483+0000) 2019-10-13T06:31:27.547 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:27.539+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:27.539819+0000) 2019-10-13T06:31:27.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:27.543+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:28.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:28.286+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:28.286903+0000) 2019-10-13T06:31:28.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:28.625+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:28.625464+0000) 2019-10-13T06:31:28.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:28.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:28.625460+0000) 2019-10-13T06:31:30.572 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:31:30.573 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:30.577 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:31:30.577 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:31.205 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:31.197+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:31.197695+0000) 2019-10-13T06:31:31.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:31.878+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:31.878763+0000) 2019-10-13T06:31:32.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:32.545+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.546277+0000) 2019-10-13T06:31:32.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:32.542+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.543293+0000) 2019-10-13T06:31:32.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:32.546+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.546919+0000) 2019-10-13T06:31:32.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:32.546+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.546857+0000) 2019-10-13T06:31:32.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:32.546+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.547105+0000) 2019-10-13T06:31:32.552 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:32.546+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.547241+0000) 2019-10-13T06:31:32.552 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:32.547+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.547564+0000) 2019-10-13T06:31:32.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:32.544+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.545012+0000) 2019-10-13T06:31:32.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:32.549+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:32.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:32.743+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.743322+0000) 2019-10-13T06:31:32.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:32.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:32.744576+0000) 2019-10-13T06:31:34.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:34.240+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:34.240515+0000) 2019-10-13T06:31:37.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:37.550+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.551823+0000) 2019-10-13T06:31:37.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:37.550+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.551977+0000) 2019-10-13T06:31:37.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:37.550+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.552254+0000) 2019-10-13T06:31:37.556 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:37.551+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.552561+0000) 2019-10-13T06:31:37.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:37.548+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.549524+0000) 2019-10-13T06:31:37.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:37.552+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.553516+0000) 2019-10-13T06:31:37.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:37.552+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.553696+0000) 2019-10-13T06:31:37.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:37.550+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:37.551053+0000) 2019-10-13T06:31:37.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:37.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:38.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:38.286+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:38.287048+0000) 2019-10-13T06:31:38.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:38.624+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:38.625675+0000) 2019-10-13T06:31:38.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:38.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:38.625674+0000) 2019-10-13T06:31:41.207 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:41.199+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:41.200203+0000) 2019-10-13T06:31:41.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:41.878+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:41.878960+0000) 2019-10-13T06:31:42.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:42.556+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.557865+0000) 2019-10-13T06:31:42.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:42.556+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.557957+0000) 2019-10-13T06:31:42.562 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:42.553+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.554428+0000) 2019-10-13T06:31:42.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:42.557+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.558585+0000) 2019-10-13T06:31:42.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:42.557+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.558779+0000) 2019-10-13T06:31:42.563 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:42.557+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.558766+0000) 2019-10-13T06:31:42.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:42.557+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.558973+0000) 2019-10-13T06:31:42.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:42.555+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.556652+0000) 2019-10-13T06:31:42.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:42.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:42.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:42.742+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.743507+0000) 2019-10-13T06:31:42.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:42.744+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:42.744750+0000) 2019-10-13T06:31:44.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:44.239+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:44.240696+0000) 2019-10-13T06:31:47.567 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:47.559+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.559988+0000) 2019-10-13T06:31:47.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:47.562+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.564054+0000) 2019-10-13T06:31:47.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:47.562+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.564015+0000) 2019-10-13T06:31:47.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:47.562+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.564295+0000) 2019-10-13T06:31:47.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:47.563+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.564505+0000) 2019-10-13T06:31:47.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:47.562+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.564334+0000) 2019-10-13T06:31:47.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:47.563+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.564941+0000) 2019-10-13T06:31:47.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:47.562+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:47.562834+0000) 2019-10-13T06:31:47.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:47.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:48.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:48.286+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:48.287241+0000) 2019-10-13T06:31:48.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:48.625+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:48.625867+0000) 2019-10-13T06:31:48.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:48.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:48.625869+0000) 2019-10-13T06:31:51.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:51.202+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:51.202530+0000) 2019-10-13T06:31:51.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:51.878+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:51.879129+0000) 2019-10-13T06:31:52.573 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:52.564+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.566142+0000) 2019-10-13T06:31:52.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:52.568+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.570093+0000) 2019-10-13T06:31:52.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:52.568+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.570280+0000) 2019-10-13T06:31:52.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:52.569+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.570492+0000) 2019-10-13T06:31:52.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:52.569+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.570830+0000) 2019-10-13T06:31:52.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:52.569+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.570525+0000) 2019-10-13T06:31:52.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:52.569+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.570989+0000) 2019-10-13T06:31:52.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:52.566+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.568190+0000) 2019-10-13T06:31:52.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:52.572+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:52.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:52.742+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.743670+0000) 2019-10-13T06:31:52.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:52.743+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:52.744907+0000) 2019-10-13T06:31:54.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:54.240+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:54.240870+0000) 2019-10-13T06:31:57.579 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:31:57.569+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.571716+0000) 2019-10-13T06:31:57.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:31:57.574+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.575807+0000) 2019-10-13T06:31:57.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:57.574+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.576028+0000) 2019-10-13T06:31:57.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:31:57.575+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.576546+0000) 2019-10-13T06:31:57.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:31:57.575+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.576722+0000) 2019-10-13T06:31:57.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:57.575+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.576895+0000) 2019-10-13T06:31:57.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:57.575+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.577100+0000) 2019-10-13T06:31:57.584 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:31:57.572+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:57.574526+0000) 2019-10-13T06:31:57.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:31:57.578+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:31:58.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:31:58.286+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:58.287428+0000) 2019-10-13T06:31:58.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:31:58.625+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:58.626049+0000) 2019-10-13T06:31:58.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:31:58.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:31:58.626051+0000) 2019-10-13T06:32:00.748 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:32:00.748 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:00.753 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:32:00.753 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:01.212 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:01.203+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:01.204751+0000) 2019-10-13T06:32:01.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:01.878+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:01.879252+0000) 2019-10-13T06:32:02.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:02.580+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.581765+0000) 2019-10-13T06:32:02.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:02.580+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.582109+0000) 2019-10-13T06:32:02.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:02.580+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.582222+0000) 2019-10-13T06:32:02.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:02.577+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.578817+0000) 2019-10-13T06:32:02.587 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:02.581+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.582647+0000) 2019-10-13T06:32:02.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:02.581+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.582758+0000) 2019-10-13T06:32:02.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:02.581+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.583012+0000) 2019-10-13T06:32:02.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:02.578+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.580554+0000) 2019-10-13T06:32:02.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:02.583+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:02.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:02.743+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.743809+0000) 2019-10-13T06:32:02.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:02.744+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:02.745063+0000) 2019-10-13T06:32:04.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:04.240+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:04.241026+0000) 2019-10-13T06:32:07.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:07.582+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.582926+0000) 2019-10-13T06:32:07.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:07.586+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.586878+0000) 2019-10-13T06:32:07.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:07.586+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.587216+0000) 2019-10-13T06:32:07.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:07.587+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.587414+0000) 2019-10-13T06:32:07.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:07.587+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.587694+0000) 2019-10-13T06:32:07.592 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:07.587+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.588150+0000) 2019-10-13T06:32:07.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:07.588+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.588919+0000) 2019-10-13T06:32:07.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:07.584+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:07.585372+0000) 2019-10-13T06:32:07.595 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:07.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:08.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:08.287+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:08.287600+0000) 2019-10-13T06:32:08.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:08.625+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:08.626254+0000) 2019-10-13T06:32:08.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:08.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:08.626252+0000) 2019-10-13T06:32:11.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:11.206+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:11.207160+0000) 2019-10-13T06:32:11.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:11.879+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:11.879428+0000) 2019-10-13T06:32:12.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:12.593+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.594329+0000) 2019-10-13T06:32:12.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:12.594+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.594423+0000) 2019-10-13T06:32:12.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:12.594+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.594532+0000) 2019-10-13T06:32:12.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:12.594+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.594677+0000) 2019-10-13T06:32:12.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:12.590+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.591154+0000) 2019-10-13T06:32:12.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:12.591+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.592475+0000) 2019-10-13T06:32:12.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:12.595+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.595489+0000) 2019-10-13T06:32:12.600 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:12.595+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.595570+0000) 2019-10-13T06:32:12.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:12.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:12.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:12.743+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.743987+0000) 2019-10-13T06:32:12.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:12.744+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:12.745219+0000) 2019-10-13T06:32:14.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:14.240+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:14.241203+0000) 2019-10-13T06:32:17.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:17.595+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.595938+0000) 2019-10-13T06:32:17.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:17.599+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.599507+0000) 2019-10-13T06:32:17.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:17.599+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.599824+0000) 2019-10-13T06:32:17.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:17.599+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.600073+0000) 2019-10-13T06:32:17.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:17.600+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.600554+0000) 2019-10-13T06:32:17.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:17.600+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.600925+0000) 2019-10-13T06:32:17.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:17.600+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.601223+0000) 2019-10-13T06:32:17.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:17.597+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:17.598447+0000) 2019-10-13T06:32:17.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:17.603+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:18.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:18.287+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:18.287760+0000) 2019-10-13T06:32:18.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:18.626+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:18.626456+0000) 2019-10-13T06:32:18.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:18.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:18.626453+0000) 2019-10-13T06:32:21.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:21.209+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:21.209589+0000) 2019-10-13T06:32:21.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:21.879+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:21.879621+0000) 2019-10-13T06:32:22.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:22.604+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.605773+0000) 2019-10-13T06:32:22.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:22.604+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.606047+0000) 2019-10-13T06:32:22.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:22.601+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.602171+0000) 2019-10-13T06:32:22.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:22.605+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.606546+0000) 2019-10-13T06:32:22.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:22.605+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.606642+0000) 2019-10-13T06:32:22.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:22.605+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.607306+0000) 2019-10-13T06:32:22.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:22.606+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.607418+0000) 2019-10-13T06:32:22.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:22.604+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.605382+0000) 2019-10-13T06:32:22.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:22.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:22.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:22.743+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.744171+0000) 2019-10-13T06:32:22.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:22.744+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:22.745384+0000) 2019-10-13T06:32:24.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:24.240+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:24.241395+0000) 2019-10-13T06:32:27.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:27.606+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.607425+0000) 2019-10-13T06:32:27.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:27.609+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.611322+0000) 2019-10-13T06:32:27.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:27.610+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.611762+0000) 2019-10-13T06:32:27.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:27.610+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.611860+0000) 2019-10-13T06:32:27.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:27.610+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.611981+0000) 2019-10-13T06:32:27.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:27.610+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.612093+0000) 2019-10-13T06:32:27.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:27.610+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.612301+0000) 2019-10-13T06:32:27.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:27.609+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:27.609903+0000) 2019-10-13T06:32:27.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:27.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101694 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:28.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:28.287+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:28.287936+0000) 2019-10-13T06:32:28.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:28.625+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:28.626654+0000) 2019-10-13T06:32:28.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:28.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:28.626659+0000) 2019-10-13T06:32:30.835 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:32:30.835 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:30.839 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:32:30.840 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:31.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:31.211+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:31.211966+0000) 2019-10-13T06:32:31.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:31.878+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:31.879810+0000) 2019-10-13T06:32:32.619 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:32.614+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.615867+0000) 2019-10-13T06:32:32.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:32.611+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.612287+0000) 2019-10-13T06:32:32.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:32.614+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.616253+0000) 2019-10-13T06:32:32.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:32.614+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.616324+0000) 2019-10-13T06:32:32.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:32.615+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.616643+0000) 2019-10-13T06:32:32.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:32.615+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.616932+0000) 2019-10-13T06:32:32.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:32.615+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.617022+0000) 2019-10-13T06:32:32.622 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:32.614+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.615129+0000) 2019-10-13T06:32:32.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:32.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:32.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:32.743+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.744337+0000) 2019-10-13T06:32:32.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:32.745+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:32.745462+0000) 2019-10-13T06:32:34.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:34.240+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:34.241618+0000) 2019-10-13T06:32:37.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:37.620+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.621407+0000) 2019-10-13T06:32:37.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:37.617+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.618352+0000) 2019-10-13T06:32:37.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:37.620+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.621724+0000) 2019-10-13T06:32:37.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:37.620+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.621856+0000) 2019-10-13T06:32:37.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:37.620+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.622363+0000) 2019-10-13T06:32:37.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:37.621+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.622716+0000) 2019-10-13T06:32:37.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:37.621+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.623229+0000) 2019-10-13T06:32:37.628 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:37.619+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:37.620686+0000) 2019-10-13T06:32:37.628 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:37.624+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:38.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:38.287+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:38.288081+0000) 2019-10-13T06:32:38.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:38.626+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:38.626826+0000) 2019-10-13T06:32:38.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:38.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:38.626841+0000) 2019-10-13T06:32:41.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:41.214+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:41.214360+0000) 2019-10-13T06:32:41.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:41.879+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:41.880026+0000) 2019-10-13T06:32:42.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:42.625+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.627153+0000) 2019-10-13T06:32:42.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:42.625+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.627311+0000) 2019-10-13T06:32:42.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:42.626+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.627792+0000) 2019-10-13T06:32:42.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:42.626+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.627963+0000) 2019-10-13T06:32:42.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:42.626+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.628203+0000) 2019-10-13T06:32:42.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:42.624+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.625396+0000) 2019-10-13T06:32:42.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:42.626+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.628291+0000) 2019-10-13T06:32:42.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:42.626+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.627464+0000) 2019-10-13T06:32:42.635 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:42.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101787 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:42.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:42.743+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.744475+0000) 2019-10-13T06:32:42.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:42.745+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:42.745595+0000) 2019-10-13T06:32:44.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:44.241+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:44.241780+0000) 2019-10-13T06:32:47.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:47.632+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.633981+0000) 2019-10-13T06:32:47.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:47.632+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.634189+0000) 2019-10-13T06:32:47.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:47.633+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.634511+0000) 2019-10-13T06:32:47.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:47.628+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.630583+0000) 2019-10-13T06:32:47.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:47.633+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.634742+0000) 2019-10-13T06:32:47.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:47.633+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.635092+0000) 2019-10-13T06:32:47.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:47.633+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.635200+0000) 2019-10-13T06:32:47.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:47.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:47.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:47.631+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:47.633301+0000) 2019-10-13T06:32:48.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:48.287+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:48.288206+0000) 2019-10-13T06:32:48.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:48.626+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:48.626926+0000) 2019-10-13T06:32:48.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:48.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:48.626971+0000) 2019-10-13T06:32:51.224 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:51.214+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:51.216649+0000) 2019-10-13T06:32:51.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:51.879+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:51.880172+0000) 2019-10-13T06:32:52.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:52.637+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.638183+0000) 2019-10-13T06:32:52.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:52.638+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.638507+0000) 2019-10-13T06:32:52.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:52.638+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.638830+0000) 2019-10-13T06:32:52.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:52.638+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.639155+0000) 2019-10-13T06:32:52.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:52.639+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.639546+0000) 2019-10-13T06:32:52.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:52.633+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.635638+0000) 2019-10-13T06:32:52.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:52.639+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.639771+0000) 2019-10-13T06:32:52.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:52.635+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.637049+0000) 2019-10-13T06:32:52.647 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:52.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101848 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:52.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:52.744+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.744631+0000) 2019-10-13T06:32:52.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:52.744+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:52.745760+0000) 2019-10-13T06:32:54.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:54.241+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:54.241969+0000) 2019-10-13T06:32:57.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:32:57.644+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.644793+0000) 2019-10-13T06:32:57.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:32:57.640+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.641179+0000) 2019-10-13T06:32:57.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:32:57.645+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.645745+0000) 2019-10-13T06:32:57.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:32:57.645+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.645824+0000) 2019-10-13T06:32:57.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:57.645+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.645968+0000) 2019-10-13T06:32:57.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:57.645+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.646050+0000) 2019-10-13T06:32:57.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:57.645+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.646229+0000) 2019-10-13T06:32:57.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:32:57.642+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:57.643388+0000) 2019-10-13T06:32:57.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:32:57.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:32:58.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:32:58.288+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:58.288358+0000) 2019-10-13T06:32:58.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:32:58.626+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:58.627071+0000) 2019-10-13T06:32:58.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:32:58.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:32:58.627119+0000) 2019-10-13T06:33:00.982 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:33:00.982 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:00.987 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:33:00.987 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:01.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:01.218+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:01.219026+0000) 2019-10-13T06:33:01.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:01.880+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:01.880336+0000) 2019-10-13T06:33:02.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:02.650+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.650756+0000) 2019-10-13T06:33:02.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:02.650+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.651041+0000) 2019-10-13T06:33:02.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:02.650+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.650429+0000) 2019-10-13T06:33:02.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:02.651+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.651453+0000) 2019-10-13T06:33:02.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:02.651+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.651622+0000) 2019-10-13T06:33:02.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:02.651+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.651919+0000) 2019-10-13T06:33:02.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:02.646+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.647504+0000) 2019-10-13T06:33:02.658 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:02.649+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.650217+0000) 2019-10-13T06:33:02.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:02.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:02.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:02.744+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.744819+0000) 2019-10-13T06:33:02.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:02.745+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:02.745894+0000) 2019-10-13T06:33:04.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:04.242+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:04.242154+0000) 2019-10-13T06:33:07.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:07.656+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.657044+0000) 2019-10-13T06:33:07.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:07.656+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.657167+0000) 2019-10-13T06:33:07.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:07.657+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.657522+0000) 2019-10-13T06:33:07.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:07.653+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.653889+0000) 2019-10-13T06:33:07.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:07.657+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.657859+0000) 2019-10-13T06:33:07.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:07.657+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.657984+0000) 2019-10-13T06:33:07.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:07.657+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.658332+0000) 2019-10-13T06:33:07.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:07.655+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:07.656512+0000) 2019-10-13T06:33:07.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:07.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:08.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:08.288+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:08.288538+0000) 2019-10-13T06:33:08.631 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:08.626+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:08.627240+0000) 2019-10-13T06:33:08.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:08.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:08.627281+0000) 2019-10-13T06:33:11.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:11.220+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:11.221340+0000) 2019-10-13T06:33:11.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:11.879+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:11.880515+0000) 2019-10-13T06:33:12.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:12.661+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.663122+0000) 2019-10-13T06:33:12.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:12.661+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.663338+0000) 2019-10-13T06:33:12.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:12.659+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.660359+0000) 2019-10-13T06:33:12.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:12.662+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.663512+0000) 2019-10-13T06:33:12.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:12.662+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.663708+0000) 2019-10-13T06:33:12.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:12.662+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.663776+0000) 2019-10-13T06:33:12.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:12.662+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.664005+0000) 2019-10-13T06:33:12.669 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:12.661+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.662339+0000) 2019-10-13T06:33:12.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:12.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:12.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:12.744+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.744980+0000) 2019-10-13T06:33:12.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:12.745+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:12.746060+0000) 2019-10-13T06:33:14.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:14.241+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:14.242329+0000) 2019-10-13T06:33:17.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:17.667+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.668949+0000) 2019-10-13T06:33:17.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:17.667+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.668864+0000) 2019-10-13T06:33:17.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:17.668+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.669480+0000) 2019-10-13T06:33:17.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:17.668+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.669793+0000) 2019-10-13T06:33:17.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:17.665+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.665880+0000) 2019-10-13T06:33:17.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:17.668+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.669955+0000) 2019-10-13T06:33:17.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:17.668+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.670133+0000) 2019-10-13T06:33:17.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:17.666+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:17.667604+0000) 2019-10-13T06:33:17.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:17.671+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 101994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:18.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:18.287+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:18.288730+0000) 2019-10-13T06:33:18.631 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:18.626+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:18.627430+0000) 2019-10-13T06:33:18.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:18.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:18.627436+0000) 2019-10-13T06:33:21.231 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:21.223+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:21.223821+0000) 2019-10-13T06:33:21.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:21.879+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:21.880710+0000) 2019-10-13T06:33:22.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:22.670+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.671154+0000) 2019-10-13T06:33:22.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:22.673+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.675327+0000) 2019-10-13T06:33:22.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:22.674+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.675560+0000) 2019-10-13T06:33:22.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:22.674+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.675711+0000) 2019-10-13T06:33:22.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:22.674+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.675828+0000) 2019-10-13T06:33:22.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:22.674+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.676112+0000) 2019-10-13T06:33:22.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:22.674+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.676272+0000) 2019-10-13T06:33:22.681 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:22.673+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.674565+0000) 2019-10-13T06:33:22.682 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:22.677+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102026 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:22.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:22.744+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.745124+0000) 2019-10-13T06:33:22.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:22.745+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:22.746214+0000) 2019-10-13T06:33:24.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:24.241+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:24.242517+0000) 2019-10-13T06:33:27.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:27.679+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.681275+0000) 2019-10-13T06:33:27.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:27.680+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.681563+0000) 2019-10-13T06:33:27.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:27.677+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.677941+0000) 2019-10-13T06:33:27.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:27.680+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.681895+0000) 2019-10-13T06:33:27.688 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:27.680+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.682016+0000) 2019-10-13T06:33:27.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:27.680+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.682381+0000) 2019-10-13T06:33:27.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:27.680+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.682229+0000) 2019-10-13T06:33:27.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:27.678+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:27.679514+0000) 2019-10-13T06:33:27.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:27.683+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:28.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:28.288+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:28.288898+0000) 2019-10-13T06:33:28.631 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:28.627+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:28.627639+0000) 2019-10-13T06:33:28.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:28.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:28.627639+0000) 2019-10-13T06:33:31.145 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:33:31.145 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:31.150 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:33:31.150 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:31.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:31.226+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:31.226285+0000) 2019-10-13T06:33:31.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:31.880+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:31.880921+0000) 2019-10-13T06:33:32.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:32.685+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.686663+0000) 2019-10-13T06:33:32.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:32.685+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.686881+0000) 2019-10-13T06:33:32.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:32.682+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.683254+0000) 2019-10-13T06:33:32.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:32.685+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.687298+0000) 2019-10-13T06:33:32.692 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:32.686+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.687468+0000) 2019-10-13T06:33:32.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:32.686+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.687606+0000) 2019-10-13T06:33:32.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:32.686+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.688276+0000) 2019-10-13T06:33:32.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:32.685+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.686421+0000) 2019-10-13T06:33:32.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:32.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:32.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:32.744+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.745294+0000) 2019-10-13T06:33:32.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:32.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:32.746368+0000) 2019-10-13T06:33:34.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:34.242+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:34.242663+0000) 2019-10-13T06:33:37.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:37.692+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.692991+0000) 2019-10-13T06:33:37.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:37.692+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.693310+0000) 2019-10-13T06:33:37.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:37.688+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.689925+0000) 2019-10-13T06:33:37.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:37.693+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.693822+0000) 2019-10-13T06:33:37.698 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:37.693+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.693946+0000) 2019-10-13T06:33:37.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:37.693+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.694243+0000) 2019-10-13T06:33:37.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:37.694+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.694522+0000) 2019-10-13T06:33:37.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:37.690+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:37.692290+0000) 2019-10-13T06:33:37.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:37.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:38.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:38.288+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:38.289062+0000) 2019-10-13T06:33:38.631 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:38.627+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:38.627804+0000) 2019-10-13T06:33:38.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:38.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:38.627801+0000) 2019-10-13T06:33:41.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:41.227+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:41.228421+0000) 2019-10-13T06:33:41.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:41.880+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:41.881099+0000) 2019-10-13T06:33:42.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:42.698+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.698927+0000) 2019-10-13T06:33:42.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:42.694+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.695863+0000) 2019-10-13T06:33:42.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:42.698+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.699188+0000) 2019-10-13T06:33:42.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:42.698+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.699335+0000) 2019-10-13T06:33:42.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:42.699+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.699794+0000) 2019-10-13T06:33:42.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:42.699+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.700013+0000) 2019-10-13T06:33:42.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:42.699+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.700312+0000) 2019-10-13T06:33:42.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:42.696+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.697802+0000) 2019-10-13T06:33:42.712 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:42.702+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:42.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:42.745+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.745398+0000) 2019-10-13T06:33:42.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:42.745+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:42.746511+0000) 2019-10-13T06:33:44.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:44.242+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:44.242831+0000) 2019-10-13T06:33:47.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:47.704+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.704711+0000) 2019-10-13T06:33:47.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:47.699+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.701239+0000) 2019-10-13T06:33:47.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:47.704+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.705225+0000) 2019-10-13T06:33:47.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:47.705+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.705415+0000) 2019-10-13T06:33:47.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:47.705+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.705556+0000) 2019-10-13T06:33:47.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:47.705+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.705855+0000) 2019-10-13T06:33:47.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:47.705+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.706052+0000) 2019-10-13T06:33:47.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:47.702+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:47.704424+0000) 2019-10-13T06:33:47.712 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:47.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:48.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:48.289+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:48.289249+0000) 2019-10-13T06:33:48.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:48.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:48.627927+0000) 2019-10-13T06:33:48.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:48.627+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:48.627951+0000) 2019-10-13T06:33:51.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:51.230+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:51.230972+0000) 2019-10-13T06:33:51.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:51.881+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:51.881275+0000) 2019-10-13T06:33:52.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:52.710+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.711335+0000) 2019-10-13T06:33:52.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:52.710+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.711377+0000) 2019-10-13T06:33:52.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:52.711+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.711730+0000) 2019-10-13T06:33:52.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:52.707+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.708342+0000) 2019-10-13T06:33:52.716 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:52.711+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.712016+0000) 2019-10-13T06:33:52.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:52.711+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.712201+0000) 2019-10-13T06:33:52.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:52.712+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.712504+0000) 2019-10-13T06:33:52.717 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:52.709+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.710002+0000) 2019-10-13T06:33:52.718 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:52.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:52.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:52.745+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.745564+0000) 2019-10-13T06:33:52.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:52.745+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:52.746592+0000) 2019-10-13T06:33:54.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:54.241+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:54.243002+0000) 2019-10-13T06:33:57.722 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:57.715+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.717231+0000) 2019-10-13T06:33:57.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:33:57.715+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.717366+0000) 2019-10-13T06:33:57.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:57.716+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.717595+0000) 2019-10-13T06:33:57.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:33:57.716+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.717936+0000) 2019-10-13T06:33:57.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:33:57.716+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.718062+0000) 2019-10-13T06:33:57.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:57.717+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.718506+0000) 2019-10-13T06:33:57.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:33:57.714+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.714753+0000) 2019-10-13T06:33:57.724 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:33:57.716+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:57.716834+0000) 2019-10-13T06:33:57.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:33:57.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:33:58.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:33:58.288+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:58.289433+0000) 2019-10-13T06:33:58.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:33:58.627+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:58.628074+0000) 2019-10-13T06:33:58.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:33:58.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:33:58.628066+0000) 2019-10-13T06:34:01.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:01.232+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:01.233344+0000) 2019-10-13T06:34:01.290 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:34:01.291 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:01.299 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:34:01.300 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:01.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:01.880+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:01.881464+0000) 2019-10-13T06:34:02.727 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:02.721+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.723167+0000) 2019-10-13T06:34:02.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:02.721+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.723331+0000) 2019-10-13T06:34:02.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:02.722+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.723706+0000) 2019-10-13T06:34:02.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:02.719+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.719753+0000) 2019-10-13T06:34:02.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:02.722+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.723961+0000) 2019-10-13T06:34:02.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:02.722+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.724071+0000) 2019-10-13T06:34:02.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:02.723+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.724438+0000) 2019-10-13T06:34:02.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:02.722+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.723478+0000) 2019-10-13T06:34:02.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:02.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:02.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:02.744+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.745749+0000) 2019-10-13T06:34:02.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:02.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:02.746760+0000) 2019-10-13T06:34:04.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:04.242+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:04.243183+0000) 2019-10-13T06:34:07.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:07.727+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.729251+0000) 2019-10-13T06:34:07.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:07.727+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.729246+0000) 2019-10-13T06:34:07.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:07.728+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.729439+0000) 2019-10-13T06:34:07.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:07.728+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.729702+0000) 2019-10-13T06:34:07.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:07.726+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.726836+0000) 2019-10-13T06:34:07.735 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:07.728+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.730094+0000) 2019-10-13T06:34:07.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:07.729+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.730747+0000) 2019-10-13T06:34:07.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:07.727+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:07.728094+0000) 2019-10-13T06:34:07.737 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:07.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102292 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:08.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:08.288+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:08.289622+0000) 2019-10-13T06:34:08.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:08.627+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:08.628230+0000) 2019-10-13T06:34:08.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:08.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:08.628224+0000) 2019-10-13T06:34:11.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:11.235+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:11.235777+0000) 2019-10-13T06:34:11.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:11.880+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:11.881652+0000) 2019-10-13T06:34:12.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:12.733+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.735339+0000) 2019-10-13T06:34:12.739 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:12.734+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.736165+0000) 2019-10-13T06:34:12.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:12.735+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.736492+0000) 2019-10-13T06:34:12.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:12.735+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.737170+0000) 2019-10-13T06:34:12.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:12.733+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.733944+0000) 2019-10-13T06:34:12.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:12.736+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.737980+0000) 2019-10-13T06:34:12.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:12.737+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.738562+0000) 2019-10-13T06:34:12.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:12.735+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.736214+0000) 2019-10-13T06:34:12.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:12.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:12.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:12.744+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.745855+0000) 2019-10-13T06:34:12.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:12.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:12.746892+0000) 2019-10-13T06:34:14.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:14.242+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:14.243329+0000) 2019-10-13T06:34:17.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:17.741+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.742406+0000) 2019-10-13T06:34:17.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:17.741+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.742851+0000) 2019-10-13T06:34:17.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:17.741+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.743003+0000) 2019-10-13T06:34:17.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:17.738+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.739460+0000) 2019-10-13T06:34:17.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:17.742+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.743649+0000) 2019-10-13T06:34:17.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:17.742+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.743838+0000) 2019-10-13T06:34:17.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:17.742+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.744084+0000) 2019-10-13T06:34:17.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:17.740+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:17.741142+0000) 2019-10-13T06:34:17.750 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:17.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102353 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:18.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:18.289+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:18.289797+0000) 2019-10-13T06:34:18.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:18.627+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:18.628427+0000) 2019-10-13T06:34:18.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:18.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:18.628426+0000) 2019-10-13T06:34:21.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:21.237+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:21.238099+0000) 2019-10-13T06:34:21.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:21.881+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:21.881797+0000) 2019-10-13T06:34:22.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:22.745+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.746021+0000) 2019-10-13T06:34:22.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:22.743+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.744443+0000) 2019-10-13T06:34:22.753 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:22.748+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.748423+0000) 2019-10-13T06:34:22.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:22.748+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.748782+0000) 2019-10-13T06:34:22.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:22.748+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.748964+0000) 2019-10-13T06:34:22.754 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:22.748+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.749097+0000) 2019-10-13T06:34:22.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:22.748+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.748516+0000) 2019-10-13T06:34:22.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:22.749+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.749502+0000) 2019-10-13T06:34:22.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:22.745+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.746398+0000) 2019-10-13T06:34:22.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:22.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:22.746998+0000) 2019-10-13T06:34:22.785 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:22.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:24.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:24.243+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:24.243483+0000) 2019-10-13T06:34:27.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:27.753+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.753650+0000) 2019-10-13T06:34:27.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:27.753+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.753940+0000) 2019-10-13T06:34:27.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:27.748+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.750340+0000) 2019-10-13T06:34:27.759 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:27.753+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.754346+0000) 2019-10-13T06:34:27.759 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:27.754+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.754515+0000) 2019-10-13T06:34:27.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:27.754+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.754675+0000) 2019-10-13T06:34:27.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:27.754+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.755328+0000) 2019-10-13T06:34:27.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:27.751+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:27.753373+0000) 2019-10-13T06:34:27.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:27.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102412 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:28.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:28.289+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:28.289964+0000) 2019-10-13T06:34:28.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:28.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:28.628619+0000) 2019-10-13T06:34:28.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:28.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:28.628617+0000) 2019-10-13T06:34:31.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:31.239+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:31.240459+0000) 2019-10-13T06:34:31.470 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:34:31.470 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:31.485 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:34:31.485 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:31.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:31.881+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:31.881952+0000) 2019-10-13T06:34:32.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:32.745+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.746230+0000) 2019-10-13T06:34:32.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:32.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.747140+0000) 2019-10-13T06:34:32.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:32.759+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.759954+0000) 2019-10-13T06:34:32.763 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:32.759+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.760099+0000) 2019-10-13T06:34:32.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:32.759+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.760244+0000) 2019-10-13T06:34:32.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:32.755+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.756948+0000) 2019-10-13T06:34:32.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:32.760+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.760437+0000) 2019-10-13T06:34:32.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:32.760+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.760564+0000) 2019-10-13T06:34:32.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:32.760+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.760898+0000) 2019-10-13T06:34:32.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:32.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:32.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:32.758+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:32.760648+0000) 2019-10-13T06:34:34.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:34.243+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:34.243681+0000) 2019-10-13T06:34:37.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:37.764+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.764658+0000) 2019-10-13T06:34:37.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:37.764+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.764961+0000) 2019-10-13T06:34:37.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:37.759+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.761381+0000) 2019-10-13T06:34:37.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:37.765+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.765570+0000) 2019-10-13T06:34:37.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:37.765+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.765957+0000) 2019-10-13T06:34:37.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:37.765+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.766187+0000) 2019-10-13T06:34:37.770 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:37.766+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.766564+0000) 2019-10-13T06:34:37.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:37.761+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:37.763708+0000) 2019-10-13T06:34:37.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:37.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:38.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:38.289+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:38.290149+0000) 2019-10-13T06:34:38.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:38.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:38.628791+0000) 2019-10-13T06:34:38.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:38.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:38.628789+0000) 2019-10-13T06:34:41.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:41.241+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:41.242714+0000) 2019-10-13T06:34:41.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:41.880+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:41.882182+0000) 2019-10-13T06:34:42.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:42.745+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.746426+0000) 2019-10-13T06:34:42.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:42.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.747303+0000) 2019-10-13T06:34:42.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:42.766+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.767299+0000) 2019-10-13T06:34:42.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:42.769+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.770801+0000) 2019-10-13T06:34:42.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:42.769+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.771326+0000) 2019-10-13T06:34:42.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:42.770+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.771452+0000) 2019-10-13T06:34:42.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:42.770+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.771747+0000) 2019-10-13T06:34:42.776 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:42.770+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.771549+0000) 2019-10-13T06:34:42.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:42.768+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.769570+0000) 2019-10-13T06:34:42.777 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:42.770+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:42.772224+0000) 2019-10-13T06:34:42.779 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:42.774+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:44.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:44.242+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:44.243894+0000) 2019-10-13T06:34:47.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:47.775+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.777042+0000) 2019-10-13T06:34:47.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:47.775+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.777152+0000) 2019-10-13T06:34:47.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:47.775+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.777370+0000) 2019-10-13T06:34:47.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:47.776+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.777625+0000) 2019-10-13T06:34:47.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:47.773+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.773978+0000) 2019-10-13T06:34:47.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:47.777+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.778508+0000) 2019-10-13T06:34:47.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:47.777+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.778578+0000) 2019-10-13T06:34:47.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:47.775+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:47.776467+0000) 2019-10-13T06:34:47.785 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:47.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102532 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:48.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:48.289+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:48.290342+0000) 2019-10-13T06:34:48.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:48.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:48.628959+0000) 2019-10-13T06:34:48.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:48.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:48.628959+0000) 2019-10-13T06:34:51.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:51.244+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:51.245108+0000) 2019-10-13T06:34:51.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:51.881+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:51.882365+0000) 2019-10-13T06:34:52.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:52.745+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.746614+0000) 2019-10-13T06:34:52.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:52.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.747471+0000) 2019-10-13T06:34:52.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:52.779+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.780125+0000) 2019-10-13T06:34:52.788 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:52.782+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.784004+0000) 2019-10-13T06:34:52.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:52.782+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.784301+0000) 2019-10-13T06:34:52.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:52.783+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.784512+0000) 2019-10-13T06:34:52.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:52.783+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.784620+0000) 2019-10-13T06:34:52.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:52.783+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.784868+0000) 2019-10-13T06:34:52.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:52.781+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.782497+0000) 2019-10-13T06:34:52.790 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:52.783+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:52.785113+0000) 2019-10-13T06:34:52.791 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:52.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:54.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:54.243+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:54.244046+0000) 2019-10-13T06:34:57.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:57.788+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.789499+0000) 2019-10-13T06:34:57.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:34:57.785+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.786026+0000) 2019-10-13T06:34:57.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:57.788+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.790006+0000) 2019-10-13T06:34:57.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:57.788+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.790254+0000) 2019-10-13T06:34:57.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:34:57.788+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.790383+0000) 2019-10-13T06:34:57.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:34:57.789+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.790487+0000) 2019-10-13T06:34:57.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:34:57.789+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.790754+0000) 2019-10-13T06:34:57.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:34:57.787+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:57.788330+0000) 2019-10-13T06:34:57.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:34:57.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:34:58.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:34:58.289+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:58.290546+0000) 2019-10-13T06:34:58.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:34:58.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:58.629135+0000) 2019-10-13T06:34:58.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:34:58.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:34:58.629137+0000) 2019-10-13T06:35:01.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:01.247+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:01.247474+0000) 2019-10-13T06:35:01.649 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:35:01.649 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:01.654 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:35:01.654 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:01.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:01.881+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:01.882514+0000) 2019-10-13T06:35:02.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:02.746+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.746800+0000) 2019-10-13T06:35:02.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:02.746+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.747677+0000) 2019-10-13T06:35:02.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:02.793+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.795102+0000) 2019-10-13T06:35:02.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:02.791+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.791930+0000) 2019-10-13T06:35:02.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:02.794+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.795514+0000) 2019-10-13T06:35:02.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:02.794+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.795792+0000) 2019-10-13T06:35:02.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:02.794+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.795989+0000) 2019-10-13T06:35:02.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:02.794+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.796113+0000) 2019-10-13T06:35:02.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:02.794+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.796285+0000) 2019-10-13T06:35:02.801 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:02.792+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:02.793664+0000) 2019-10-13T06:35:02.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:02.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:04.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:04.243+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:04.244244+0000) 2019-10-13T06:35:07.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:07.801+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.802127+0000) 2019-10-13T06:35:07.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:07.801+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.802237+0000) 2019-10-13T06:35:07.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:07.802+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.802710+0000) 2019-10-13T06:35:07.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:07.802+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.802828+0000) 2019-10-13T06:35:07.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:07.802+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.802954+0000) 2019-10-13T06:35:07.808 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:07.802+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.803155+0000) 2019-10-13T06:35:07.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:07.798+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.798759+0000) 2019-10-13T06:35:07.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:07.800+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:07.801068+0000) 2019-10-13T06:35:07.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:07.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102648 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:08.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:08.290+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:08.290725+0000) 2019-10-13T06:35:08.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:08.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:08.629278+0000) 2019-10-13T06:35:08.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:08.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:08.629312+0000) 2019-10-13T06:35:11.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:11.249+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:11.249821+0000) 2019-10-13T06:35:11.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:11.882+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:11.882722+0000) 2019-10-13T06:35:12.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:12.746+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.746987+0000) 2019-10-13T06:35:12.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:12.747+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.747886+0000) 2019-10-13T06:35:12.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:12.805+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.806272+0000) 2019-10-13T06:35:12.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:12.806+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.806529+0000) 2019-10-13T06:35:12.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:12.806+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.806763+0000) 2019-10-13T06:35:12.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:12.802+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.803167+0000) 2019-10-13T06:35:12.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:12.807+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.807669+0000) 2019-10-13T06:35:12.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:12.807+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.807748+0000) 2019-10-13T06:35:12.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:12.807+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.808039+0000) 2019-10-13T06:35:12.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:12.806+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:12.807022+0000) 2019-10-13T06:35:12.814 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:12.811+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:14.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:14.244+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:14.244456+0000) 2019-10-13T06:35:17.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:17.812+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.813241+0000) 2019-10-13T06:35:17.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:17.809+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.810166+0000) 2019-10-13T06:35:17.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:17.813+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.813745+0000) 2019-10-13T06:35:17.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:17.813+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.813991+0000) 2019-10-13T06:35:17.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:17.813+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.814245+0000) 2019-10-13T06:35:17.819 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:17.814+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.814452+0000) 2019-10-13T06:35:17.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:17.814+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.814773+0000) 2019-10-13T06:35:17.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:17.811+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:17.812517+0000) 2019-10-13T06:35:17.821 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:17.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:18.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:18.290+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:18.290866+0000) 2019-10-13T06:35:18.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:18.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:18.629386+0000) 2019-10-13T06:35:18.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:18.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:18.629455+0000) 2019-10-13T06:35:21.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:21.250+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:21.252074+0000) 2019-10-13T06:35:21.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:21.882+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:21.882906+0000) 2019-10-13T06:35:22.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:22.747+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.747158+0000) 2019-10-13T06:35:22.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:22.747+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.748066+0000) 2019-10-13T06:35:22.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:22.814+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.816031+0000) 2019-10-13T06:35:22.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:22.819+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.819479+0000) 2019-10-13T06:35:22.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:22.819+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.820283+0000) 2019-10-13T06:35:22.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:22.819+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.820341+0000) 2019-10-13T06:35:22.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:22.820+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.820511+0000) 2019-10-13T06:35:22.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:22.820+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.820684+0000) 2019-10-13T06:35:22.825 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:22.820+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.820911+0000) 2019-10-13T06:35:22.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:22.816+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:22.818328+0000) 2019-10-13T06:35:22.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:22.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:24.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:24.244+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:24.244644+0000) 2019-10-13T06:35:27.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:27.820+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.822128+0000) 2019-10-13T06:35:27.831 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:27.825+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.826445+0000) 2019-10-13T06:35:27.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:27.825+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.826555+0000) 2019-10-13T06:35:27.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:27.825+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.826813+0000) 2019-10-13T06:35:27.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:27.825+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.826920+0000) 2019-10-13T06:35:27.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:27.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.827208+0000) 2019-10-13T06:35:27.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:27.826+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.827494+0000) 2019-10-13T06:35:27.833 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:27.822+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:27.824667+0000) 2019-10-13T06:35:27.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:27.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:28.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:28.289+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:28.291029+0000) 2019-10-13T06:35:28.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:28.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:28.629536+0000) 2019-10-13T06:35:28.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:28.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:28.629607+0000) 2019-10-13T06:35:31.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:31.253+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:31.254364+0000) 2019-10-13T06:35:31.812 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:35:31.812 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:31.817 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:35:31.817 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:31.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:31.881+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:31.883122+0000) 2019-10-13T06:35:32.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:32.746+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.747344+0000) 2019-10-13T06:35:32.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:32.747+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.748250+0000) 2019-10-13T06:35:32.838 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:32.831+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.832747+0000) 2019-10-13T06:35:32.838 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:32.831+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.832635+0000) 2019-10-13T06:35:32.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:32.831+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.832979+0000) 2019-10-13T06:35:32.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:32.831+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.833112+0000) 2019-10-13T06:35:32.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:32.832+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.833460+0000) 2019-10-13T06:35:32.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:32.831+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.833293+0000) 2019-10-13T06:35:32.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:32.827+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.828642+0000) 2019-10-13T06:35:32.840 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:32.830+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:32.830923+0000) 2019-10-13T06:35:32.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:32.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:34.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:34.243+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:34.244836+0000) 2019-10-13T06:35:37.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:37.837+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.838423+0000) 2019-10-13T06:35:37.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:37.837+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.838582+0000) 2019-10-13T06:35:37.843 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:37.837+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.838850+0000) 2019-10-13T06:35:37.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:37.833+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.834488+0000) 2019-10-13T06:35:37.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:37.838+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.839460+0000) 2019-10-13T06:35:37.844 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:37.838+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.839776+0000) 2019-10-13T06:35:37.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:37.838+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.839812+0000) 2019-10-13T06:35:37.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:37.836+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:37.837606+0000) 2019-10-13T06:35:37.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:37.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102823 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:38.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:38.290+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:38.291177+0000) 2019-10-13T06:35:38.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:38.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:38.629677+0000) 2019-10-13T06:35:38.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:38.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:38.629772+0000) 2019-10-13T06:35:41.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:41.256+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:41.256614+0000) 2019-10-13T06:35:41.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:41.882+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:41.883346+0000) 2019-10-13T06:35:42.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:42.746+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.747490+0000) 2019-10-13T06:35:42.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:42.747+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.748437+0000) 2019-10-13T06:35:42.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:42.839+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.840619+0000) 2019-10-13T06:35:42.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:42.843+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.844704+0000) 2019-10-13T06:35:42.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:42.843+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.844808+0000) 2019-10-13T06:35:42.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:42.843+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.845044+0000) 2019-10-13T06:35:42.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:42.844+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.845425+0000) 2019-10-13T06:35:42.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:42.844+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.845671+0000) 2019-10-13T06:35:42.850 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:42.844+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.845960+0000) 2019-10-13T06:35:42.850 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:42.842+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:42.842932+0000) 2019-10-13T06:35:42.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:42.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:44.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:44.244+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:44.245041+0000) 2019-10-13T06:35:47.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:47.848+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.850327+0000) 2019-10-13T06:35:47.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:47.849+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.850630+0000) 2019-10-13T06:35:47.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:47.849+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.850771+0000) 2019-10-13T06:35:47.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:47.849+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.851130+0000) 2019-10-13T06:35:47.855 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:47.849+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.850965+0000) 2019-10-13T06:35:47.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:47.850+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.851488+0000) 2019-10-13T06:35:47.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:47.847+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.848171+0000) 2019-10-13T06:35:47.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:47.848+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:47.848929+0000) 2019-10-13T06:35:47.858 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:47.853+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102880 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:48.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:48.290+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:48.291349+0000) 2019-10-13T06:35:48.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:48.629+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:48.629865+0000) 2019-10-13T06:35:48.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:48.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:48.629924+0000) 2019-10-13T06:35:51.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:51.258+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:51.258969+0000) 2019-10-13T06:35:51.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:51.883+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:51.883533+0000) 2019-10-13T06:35:52.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:52.747+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.747665+0000) 2019-10-13T06:35:52.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:52.748+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.748621+0000) 2019-10-13T06:35:52.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:52.855+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.856552+0000) 2019-10-13T06:35:52.861 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:52.855+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.856650+0000) 2019-10-13T06:35:52.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:52.855+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.856960+0000) 2019-10-13T06:35:52.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:52.855+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.857141+0000) 2019-10-13T06:35:52.861 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:52.855+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.857228+0000) 2019-10-13T06:35:52.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:52.852+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.853123+0000) 2019-10-13T06:35:52.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:52.856+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.857623+0000) 2019-10-13T06:35:52.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:52.855+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:52.856249+0000) 2019-10-13T06:35:52.863 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:52.858+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:54.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:54.244+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:54.245207+0000) 2019-10-13T06:35:57.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:35:57.857+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.858267+0000) 2019-10-13T06:35:57.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:35:57.862+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.862467+0000) 2019-10-13T06:35:57.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:57.862+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.862728+0000) 2019-10-13T06:35:57.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:57.862+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.862825+0000) 2019-10-13T06:35:57.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:35:57.862+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.863154+0000) 2019-10-13T06:35:57.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:35:57.862+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.863297+0000) 2019-10-13T06:35:57.868 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:57.863+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.863746+0000) 2019-10-13T06:35:57.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:35:57.860+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:57.861721+0000) 2019-10-13T06:35:57.870 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:35:57.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:35:58.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:35:58.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:58.291518+0000) 2019-10-13T06:35:58.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:35:58.629+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:58.629972+0000) 2019-10-13T06:35:58.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:35:58.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:35:58.630069+0000) 2019-10-13T06:36:01.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:01.260+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:01.261216+0000) 2019-10-13T06:36:01.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:01.883+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:01.883729+0000) 2019-10-13T06:36:01.968 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:36:01.968 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:01.972 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:36:01.973 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:02.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:02.747+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.747848+0000) 2019-10-13T06:36:02.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:02.748+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.748811+0000) 2019-10-13T06:36:02.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:02.863+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.864648+0000) 2019-10-13T06:36:02.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:02.868+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.868563+0000) 2019-10-13T06:36:02.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:02.868+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.868846+0000) 2019-10-13T06:36:02.874 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:02.868+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.869043+0000) 2019-10-13T06:36:02.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:02.868+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.869265+0000) 2019-10-13T06:36:02.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:02.869+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.869456+0000) 2019-10-13T06:36:02.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:02.869+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.869766+0000) 2019-10-13T06:36:02.876 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:02.867+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:02.868396+0000) 2019-10-13T06:36:02.876 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:02.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:04.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:04.245+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:04.245423+0000) 2019-10-13T06:36:07.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:07.873+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.874264+0000) 2019-10-13T06:36:07.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:07.874+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.874455+0000) 2019-10-13T06:36:07.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:07.874+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.874562+0000) 2019-10-13T06:36:07.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:07.874+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.874951+0000) 2019-10-13T06:36:07.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:07.870+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.871228+0000) 2019-10-13T06:36:07.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:07.874+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.875168+0000) 2019-10-13T06:36:07.880 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:07.874+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.875168+0000) 2019-10-13T06:36:07.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:07.872+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:07.873295+0000) 2019-10-13T06:36:07.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:07.877+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 102996 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:08.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:08.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:08.291719+0000) 2019-10-13T06:36:08.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:08.630+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:08.630108+0000) 2019-10-13T06:36:08.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:08.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:08.630208+0000) 2019-10-13T06:36:11.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:11.263+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:11.263577+0000) 2019-10-13T06:36:11.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:11.882+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:11.883904+0000) 2019-10-13T06:36:12.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:12.746+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.747981+0000) 2019-10-13T06:36:12.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:12.747+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.748992+0000) 2019-10-13T06:36:12.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:12.878+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.879673+0000) 2019-10-13T06:36:12.883 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:12.878+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.879993+0000) 2019-10-13T06:36:12.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:12.878+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.879977+0000) 2019-10-13T06:36:12.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:12.875+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.876799+0000) 2019-10-13T06:36:12.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:12.878+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.880311+0000) 2019-10-13T06:36:12.885 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:12.879+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.881040+0000) 2019-10-13T06:36:12.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:12.879+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.881092+0000) 2019-10-13T06:36:12.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:12.876+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:12.878558+0000) 2019-10-13T06:36:12.887 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:12.883+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:14.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:14.244+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:14.245614+0000) 2019-10-13T06:36:17.890 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:17.884+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.886334+0000) 2019-10-13T06:36:17.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:17.885+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.886481+0000) 2019-10-13T06:36:17.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:17.881+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.883665+0000) 2019-10-13T06:36:17.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:17.885+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.886987+0000) 2019-10-13T06:36:17.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:17.885+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.887208+0000) 2019-10-13T06:36:17.892 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:17.886+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.887598+0000) 2019-10-13T06:36:17.892 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:17.886+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.887856+0000) 2019-10-13T06:36:17.892 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:17.883+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:17.885111+0000) 2019-10-13T06:36:17.894 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:17.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:18.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:18.290+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:18.291889+0000) 2019-10-13T06:36:18.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:18.628+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:18.630297+0000) 2019-10-13T06:36:18.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:18.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:18.630388+0000) 2019-10-13T06:36:21.273 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:21.265+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:21.265901+0000) 2019-10-13T06:36:21.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:21.883+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:21.884113+0000) 2019-10-13T06:36:22.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:22.747+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.748132+0000) 2019-10-13T06:36:22.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:22.748+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.749173+0000) 2019-10-13T06:36:22.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:22.891+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.892415+0000) 2019-10-13T06:36:22.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:22.887+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.889493+0000) 2019-10-13T06:36:22.897 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:22.891+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.892863+0000) 2019-10-13T06:36:22.897 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:22.891+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.892992+0000) 2019-10-13T06:36:22.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:22.891+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.893267+0000) 2019-10-13T06:36:22.898 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:22.891+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.893361+0000) 2019-10-13T06:36:22.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:22.892+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.893632+0000) 2019-10-13T06:36:22.899 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:22.890+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:22.892301+0000) 2019-10-13T06:36:22.900 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:22.895+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:24.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:24.244+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:24.245815+0000) 2019-10-13T06:36:27.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:27.897+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.898746+0000) 2019-10-13T06:36:27.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:27.897+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.898882+0000) 2019-10-13T06:36:27.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:27.897+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.899068+0000) 2019-10-13T06:36:27.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:27.894+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.895615+0000) 2019-10-13T06:36:27.904 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:27.898+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.899801+0000) 2019-10-13T06:36:27.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:27.898+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.900214+0000) 2019-10-13T06:36:27.904 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:27.898+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.899425+0000) 2019-10-13T06:36:27.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:27.897+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:27.898471+0000) 2019-10-13T06:36:27.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:27.901+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:28.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:28.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:28.292035+0000) 2019-10-13T06:36:28.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:28.629+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:28.630461+0000) 2019-10-13T06:36:28.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:28.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:28.630579+0000) 2019-10-13T06:36:31.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:31.267+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:31.268197+0000) 2019-10-13T06:36:31.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:31.883+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:31.884281+0000) 2019-10-13T06:36:32.145 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:36:32.146 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:32.150 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:36:32.150 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:32.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:32.747+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.748310+0000) 2019-10-13T06:36:32.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:32.748+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.749360+0000) 2019-10-13T06:36:32.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:32.903+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.905079+0000) 2019-10-13T06:36:32.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:32.901+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.901957+0000) 2019-10-13T06:36:32.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:32.904+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.905619+0000) 2019-10-13T06:36:32.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:32.904+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.905777+0000) 2019-10-13T06:36:32.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:32.904+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.905946+0000) 2019-10-13T06:36:32.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:32.904+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.906246+0000) 2019-10-13T06:36:32.910 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:32.905+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.906562+0000) 2019-10-13T06:36:32.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:32.904+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:32.905155+0000) 2019-10-13T06:36:32.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:32.907+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:34.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:34.245+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:34.246010+0000) 2019-10-13T06:36:37.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:37.906+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.907490+0000) 2019-10-13T06:36:37.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:37.909+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.910777+0000) 2019-10-13T06:36:37.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:37.909+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.910988+0000) 2019-10-13T06:36:37.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:37.910+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.911483+0000) 2019-10-13T06:36:37.917 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:37.910+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.911593+0000) 2019-10-13T06:36:37.917 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:37.910+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.911757+0000) 2019-10-13T06:36:37.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:37.910+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.911978+0000) 2019-10-13T06:36:37.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:37.909+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:37.910538+0000) 2019-10-13T06:36:37.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:37.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103174 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:38.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:38.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:38.292182+0000) 2019-10-13T06:36:38.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:38.630+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:38.630619+0000) 2019-10-13T06:36:38.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:38.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:38.630748+0000) 2019-10-13T06:36:41.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:41.270+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:41.270556+0000) 2019-10-13T06:36:41.888 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:41.884+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:41.884475+0000) 2019-10-13T06:36:42.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:42.748+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.748531+0000) 2019-10-13T06:36:42.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:42.749+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.749562+0000) 2019-10-13T06:36:42.921 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:42.916+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.917151+0000) 2019-10-13T06:36:42.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:42.916+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.917357+0000) 2019-10-13T06:36:42.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:42.917+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.917432+0000) 2019-10-13T06:36:42.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:42.917+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.917837+0000) 2019-10-13T06:36:42.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:42.913+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.914020+0000) 2019-10-13T06:36:42.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:42.917+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.918030+0000) 2019-10-13T06:36:42.923 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:42.917+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.918286+0000) 2019-10-13T06:36:42.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:42.915+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:42.916629+0000) 2019-10-13T06:36:42.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:42.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:44.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:44.245+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:44.246203+0000) 2019-10-13T06:36:47.926 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:47.922+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.923012+0000) 2019-10-13T06:36:47.927 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:47.923+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.924193+0000) 2019-10-13T06:36:47.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:47.924+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.924420+0000) 2019-10-13T06:36:47.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:47.920+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.921184+0000) 2019-10-13T06:36:47.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:47.924+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.924623+0000) 2019-10-13T06:36:47.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:47.924+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.924632+0000) 2019-10-13T06:36:47.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:47.924+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.924988+0000) 2019-10-13T06:36:47.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:47.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:47.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:47.923+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:47.924272+0000) 2019-10-13T06:36:48.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:48.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:48.292283+0000) 2019-10-13T06:36:48.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:48.630+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:48.630779+0000) 2019-10-13T06:36:48.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:48.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:48.630897+0000) 2019-10-13T06:36:51.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:51.272+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:51.272897+0000) 2019-10-13T06:36:51.888 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:51.884+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:51.884644+0000) 2019-10-13T06:36:52.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:52.748+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.748739+0000) 2019-10-13T06:36:52.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:52.749+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.749785+0000) 2019-10-13T06:36:52.933 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:52.929+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.929585+0000) 2019-10-13T06:36:52.934 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:52.929+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.929675+0000) 2019-10-13T06:36:52.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:52.929+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.929813+0000) 2019-10-13T06:36:52.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:52.929+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.929984+0000) 2019-10-13T06:36:52.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:52.929+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.930178+0000) 2019-10-13T06:36:52.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:52.926+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.926969+0000) 2019-10-13T06:36:52.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:52.927+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.927915+0000) 2019-10-13T06:36:52.937 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:52.930+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:52.930779+0000) 2019-10-13T06:36:52.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:52.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:54.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:54.246+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:54.246369+0000) 2019-10-13T06:36:57.940 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:57.933+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.935002+0000) 2019-10-13T06:36:57.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:57.933+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.935190+0000) 2019-10-13T06:36:57.941 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:57.934+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.935519+0000) 2019-10-13T06:36:57.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:36:57.934+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.936190+0000) 2019-10-13T06:36:57.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:36:57.935+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.936467+0000) 2019-10-13T06:36:57.942 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:36:57.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.936739+0000) 2019-10-13T06:36:57.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:36:57.932+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.932741+0000) 2019-10-13T06:36:57.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:36:57.933+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:57.934133+0000) 2019-10-13T06:36:57.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:36:57.938+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:36:58.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:36:58.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:58.292442+0000) 2019-10-13T06:36:58.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:36:58.629+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:58.630908+0000) 2019-10-13T06:36:58.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:36:58.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:36:58.631036+0000) 2019-10-13T06:37:01.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:01.275+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:01.275292+0000) 2019-10-13T06:37:01.888 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:01.883+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:01.884818+0000) 2019-10-13T06:37:02.325 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:37:02.325 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:02.329 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:37:02.330 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:02.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:02.747+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.748924+0000) 2019-10-13T06:37:02.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:02.749+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.749947+0000) 2019-10-13T06:37:02.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:02.939+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.941377+0000) 2019-10-13T06:37:02.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:02.940+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.941575+0000) 2019-10-13T06:37:02.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:02.940+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.941696+0000) 2019-10-13T06:37:02.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:02.940+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.941876+0000) 2019-10-13T06:37:02.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:02.938+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.939031+0000) 2019-10-13T06:37:02.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:02.941+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.942553+0000) 2019-10-13T06:37:02.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:02.941+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.942774+0000) 2019-10-13T06:37:02.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:02.940+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:02.940909+0000) 2019-10-13T06:37:02.949 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:02.944+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:04.250 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:04.245+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:04.246579+0000) 2019-10-13T06:37:07.951 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:07.946+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.947520+0000) 2019-10-13T06:37:07.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:07.946+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.947854+0000) 2019-10-13T06:37:07.952 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:07.946+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.948046+0000) 2019-10-13T06:37:07.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:07.942+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.944633+0000) 2019-10-13T06:37:07.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:07.947+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.948499+0000) 2019-10-13T06:37:07.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:07.947+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.948812+0000) 2019-10-13T06:37:07.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:07.947+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.949196+0000) 2019-10-13T06:37:07.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:07.945+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:07.946997+0000) 2019-10-13T06:37:07.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:07.950+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103353 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:08.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:08.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:08.292646+0000) 2019-10-13T06:37:08.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:08.630+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:08.631074+0000) 2019-10-13T06:37:08.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:08.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:08.631200+0000) 2019-10-13T06:37:11.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:11.276+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:11.277821+0000) 2019-10-13T06:37:11.888 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:11.884+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:11.885041+0000) 2019-10-13T06:37:12.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:12.748+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.749125+0000) 2019-10-13T06:37:12.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:12.749+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.750108+0000) 2019-10-13T06:37:12.956 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:12.951+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.953055+0000) 2019-10-13T06:37:12.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:12.951+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.953376+0000) 2019-10-13T06:37:12.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:12.952+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.953614+0000) 2019-10-13T06:37:12.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:12.948+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.950450+0000) 2019-10-13T06:37:12.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:12.952+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.954309+0000) 2019-10-13T06:37:12.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:12.953+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.954526+0000) 2019-10-13T06:37:12.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:12.953+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.954871+0000) 2019-10-13T06:37:12.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:12.950+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:12.952459+0000) 2019-10-13T06:37:12.961 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:12.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:14.250 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:14.246+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:14.246800+0000) 2019-10-13T06:37:17.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:17.958+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.959730+0000) 2019-10-13T06:37:17.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:17.958+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.959897+0000) 2019-10-13T06:37:17.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:17.958+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.960139+0000) 2019-10-13T06:37:17.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:17.954+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.956607+0000) 2019-10-13T06:37:17.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:17.959+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.960697+0000) 2019-10-13T06:37:17.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:17.959+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.960874+0000) 2019-10-13T06:37:17.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:17.959+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.961115+0000) 2019-10-13T06:37:17.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:17.957+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:17.959615+0000) 2019-10-13T06:37:17.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:17.962+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:18.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:18.291+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:18.292791+0000) 2019-10-13T06:37:18.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:18.630+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:18.631244+0000) 2019-10-13T06:37:18.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:18.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:18.631323+0000) 2019-10-13T06:37:21.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:21.279+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:21.280117+0000) 2019-10-13T06:37:21.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:21.884+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:21.885262+0000) 2019-10-13T06:37:22.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:22.747+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.749299+0000) 2019-10-13T06:37:22.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:22.749+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.750219+0000) 2019-10-13T06:37:22.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:22.964+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.965604+0000) 2019-10-13T06:37:22.969 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:22.964+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.965877+0000) 2019-10-13T06:37:22.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:22.961+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.962699+0000) 2019-10-13T06:37:22.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:22.964+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.966204+0000) 2019-10-13T06:37:22.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:22.965+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.966459+0000) 2019-10-13T06:37:22.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:22.965+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.966645+0000) 2019-10-13T06:37:22.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:22.966+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.967561+0000) 2019-10-13T06:37:22.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:22.964+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:22.965327+0000) 2019-10-13T06:37:22.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:22.969+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:24.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:24.246+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:24.246977+0000) 2019-10-13T06:37:27.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:27.971+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.972220+0000) 2019-10-13T06:37:27.977 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:27.972+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.972571+0000) 2019-10-13T06:37:27.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:27.972+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.972705+0000) 2019-10-13T06:37:27.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:27.972+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.972935+0000) 2019-10-13T06:37:27.978 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:27.972+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.973285+0000) 2019-10-13T06:37:27.978 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:27.973+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.973589+0000) 2019-10-13T06:37:27.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:27.969+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.970243+0000) 2019-10-13T06:37:27.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:27.970+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:27.970915+0000) 2019-10-13T06:37:27.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:27.976+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:28.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:28.292+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:28.292946+0000) 2019-10-13T06:37:28.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:28.631+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:28.631438+0000) 2019-10-13T06:37:28.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:28.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:28.631469+0000) 2019-10-13T06:37:31.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:31.281+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:31.282509+0000) 2019-10-13T06:37:31.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:31.885+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:31.885474+0000) 2019-10-13T06:37:32.478 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:37:32.479 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:32.483 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:37:32.484 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:32.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:32.749+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.749461+0000) 2019-10-13T06:37:32.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:32.749+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.750408+0000) 2019-10-13T06:37:32.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:32.974+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.974853+0000) 2019-10-13T06:37:32.982 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:32.978+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.978736+0000) 2019-10-13T06:37:32.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:32.978+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.979010+0000) 2019-10-13T06:37:32.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:32.978+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.979356+0000) 2019-10-13T06:37:32.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:32.979+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.979584+0000) 2019-10-13T06:37:32.984 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:32.979+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.979757+0000) 2019-10-13T06:37:32.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:32.979+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.980009+0000) 2019-10-13T06:37:32.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:32.977+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:32.978477+0000) 2019-10-13T06:37:32.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:32.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:34.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:34.246+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:34.247170+0000) 2019-10-13T06:37:37.989 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:37.984+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.985284+0000) 2019-10-13T06:37:37.989 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:37.985+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.985685+0000) 2019-10-13T06:37:37.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:37.985+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.985742+0000) 2019-10-13T06:37:37.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:37.981+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.982615+0000) 2019-10-13T06:37:37.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:37.985+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.986023+0000) 2019-10-13T06:37:37.991 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:37.986+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.986488+0000) 2019-10-13T06:37:37.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:37.986+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.986757+0000) 2019-10-13T06:37:37.991 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:37.983+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:37.984686+0000) 2019-10-13T06:37:37.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:37.989+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103534 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:38.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:38.292+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:38.293112+0000) 2019-10-13T06:37:38.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:38.631+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:38.631604+0000) 2019-10-13T06:37:38.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:38.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:38.631629+0000) 2019-10-13T06:37:41.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:41.284+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:41.284849+0000) 2019-10-13T06:37:41.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:41.885+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:41.885668+0000) 2019-10-13T06:37:42.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:42.749+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.749637+0000) 2019-10-13T06:37:42.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:42.750+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.750596+0000) 2019-10-13T06:37:42.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:42.991+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.991716+0000) 2019-10-13T06:37:42.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:42.991+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.991895+0000) 2019-10-13T06:37:42.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:42.991+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.992234+0000) 2019-10-13T06:37:42.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:42.991+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.992360+0000) 2019-10-13T06:37:42.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:42.992+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.992931+0000) 2019-10-13T06:37:42.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:42.992+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.993037+0000) 2019-10-13T06:37:42.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:42.988+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.989317+0000) 2019-10-13T06:37:42.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:42.990+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:42.991349+0000) 2019-10-13T06:37:42.999 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:42.995+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:44.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:44.246+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:44.247363+0000) 2019-10-13T06:37:48.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:47.996+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.997611+0000) 2019-10-13T06:37:48.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:47.993+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.994676+0000) 2019-10-13T06:37:48.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:47.996+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.998154+0000) 2019-10-13T06:37:48.002 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:47.997+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.998428+0000) 2019-10-13T06:37:48.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:47.997+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.998761+0000) 2019-10-13T06:37:48.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:47.997+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.999102+0000) 2019-10-13T06:37:48.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:47.997+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.999378+0000) 2019-10-13T06:37:48.003 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:47.995+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:47.996247+0000) 2019-10-13T06:37:48.005 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:48.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103593 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:48.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:48.292+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:48.293293+0000) 2019-10-13T06:37:48.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:48.630+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:48.631749+0000) 2019-10-13T06:37:48.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:48.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:48.631786+0000) 2019-10-13T06:37:51.294 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:51.286+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:51.287199+0000) 2019-10-13T06:37:51.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:51.885+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:51.885856+0000) 2019-10-13T06:37:52.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:52.749+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:52.749860+0000) 2019-10-13T06:37:52.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:52.750+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:52.750777+0000) 2019-10-13T06:37:53.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:53.002+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.003494+0000) 2019-10-13T06:37:53.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:53.002+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.003973+0000) 2019-10-13T06:37:53.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:53.003+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.004434+0000) 2019-10-13T06:37:53.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:53.003+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.004721+0000) 2019-10-13T06:37:53.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:53.003+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.004919+0000) 2019-10-13T06:37:53.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:53.003+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.005180+0000) 2019-10-13T06:37:53.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:53.001+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.001978+0000) 2019-10-13T06:37:53.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:53.003+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:53.004250+0000) 2019-10-13T06:37:53.012 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:53.007+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:54.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:54.246+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:54.247602+0000) 2019-10-13T06:37:58.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:37:58.006+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.007273+0000) 2019-10-13T06:37:58.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:37:58.009+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.011094+0000) 2019-10-13T06:37:58.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:37:58.009+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.011266+0000) 2019-10-13T06:37:58.016 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:58.010+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.011430+0000) 2019-10-13T06:37:58.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:58.009+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.011378+0000) 2019-10-13T06:37:58.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:37:58.010+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.011760+0000) 2019-10-13T06:37:58.016 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:58.010+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.012203+0000) 2019-10-13T06:37:58.017 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:37:58.008+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.009018+0000) 2019-10-13T06:37:58.018 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:37:58.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103654 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:37:58.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:37:58.292+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.293431+0000) 2019-10-13T06:37:58.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:37:58.631+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.631883+0000) 2019-10-13T06:37:58.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:37:58.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:37:58.631901+0000) 2019-10-13T06:38:01.296 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:01.288+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:01.289631+0000) 2019-10-13T06:38:01.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:01.884+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:01.886085+0000) 2019-10-13T06:38:02.641 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:38:02.642 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:02.646 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:38:02.646 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:02.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:02.749+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:02.750050+0000) 2019-10-13T06:38:02.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:02.750+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:02.750996+0000) 2019-10-13T06:38:03.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:03.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.013848+0000) 2019-10-13T06:38:03.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:03.015+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.017152+0000) 2019-10-13T06:38:03.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:03.016+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.017632+0000) 2019-10-13T06:38:03.023 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:03.016+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.017783+0000) 2019-10-13T06:38:03.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:03.016+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.018122+0000) 2019-10-13T06:38:03.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:03.016+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.018206+0000) 2019-10-13T06:38:03.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:03.017+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.018455+0000) 2019-10-13T06:38:03.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:03.014+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:03.016121+0000) 2019-10-13T06:38:03.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:03.020+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103686 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:04.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:04.247+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:04.247795+0000) 2019-10-13T06:38:08.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:08.018+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.020389+0000) 2019-10-13T06:38:08.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:08.022+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.024291+0000) 2019-10-13T06:38:08.028 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:08.023+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.024443+0000) 2019-10-13T06:38:08.029 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:08.023+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.024559+0000) 2019-10-13T06:38:08.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:08.023+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.024846+0000) 2019-10-13T06:38:08.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:08.023+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.025285+0000) 2019-10-13T06:38:08.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:08.024+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.025574+0000) 2019-10-13T06:38:08.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:08.021+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.023648+0000) 2019-10-13T06:38:08.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:08.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:08.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:08.293+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.293569+0000) 2019-10-13T06:38:08.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:08.630+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.632051+0000) 2019-10-13T06:38:08.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:08.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:08.632052+0000) 2019-10-13T06:38:11.299 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:11.291+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:11.291890+0000) 2019-10-13T06:38:11.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:11.885+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:11.886260+0000) 2019-10-13T06:38:12.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:12.749+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:12.750281+0000) 2019-10-13T06:38:12.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:12.750+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:12.751207+0000) 2019-10-13T06:38:13.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:13.030+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.030526+0000) 2019-10-13T06:38:13.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:13.025+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.027139+0000) 2019-10-13T06:38:13.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:13.030+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.031126+0000) 2019-10-13T06:38:13.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:13.031+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.031799+0000) 2019-10-13T06:38:13.035 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:13.031+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.031686+0000) 2019-10-13T06:38:13.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:13.031+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.032083+0000) 2019-10-13T06:38:13.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:13.028+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.030103+0000) 2019-10-13T06:38:13.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:13.032+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:13.032475+0000) 2019-10-13T06:38:13.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:13.034+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:14.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:14.247+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:14.247994+0000) 2019-10-13T06:38:18.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:18.032+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.033078+0000) 2019-10-13T06:38:18.041 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:18.036+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.037034+0000) 2019-10-13T06:38:18.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:18.036+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.037207+0000) 2019-10-13T06:38:18.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:18.036+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.037365+0000) 2019-10-13T06:38:18.042 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:18.037+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.037576+0000) 2019-10-13T06:38:18.042 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:18.037+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.037814+0000) 2019-10-13T06:38:18.042 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:18.037+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.038045+0000) 2019-10-13T06:38:18.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:18.034+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.035438+0000) 2019-10-13T06:38:18.044 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:18.040+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103774 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:18.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:18.293+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.293706+0000) 2019-10-13T06:38:18.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:18.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.632214+0000) 2019-10-13T06:38:18.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:18.631+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:18.632223+0000) 2019-10-13T06:38:21.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:21.293+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:21.294363+0000) 2019-10-13T06:38:21.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:21.886+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:21.886475+0000) 2019-10-13T06:38:22.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:22.750+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:22.750430+0000) 2019-10-13T06:38:22.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:22.750+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:22.751359+0000) 2019-10-13T06:38:23.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:23.042+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.042548+0000) 2019-10-13T06:38:23.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:23.038+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.039101+0000) 2019-10-13T06:38:23.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:23.042+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.043021+0000) 2019-10-13T06:38:23.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:23.043+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.043432+0000) 2019-10-13T06:38:23.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:23.043+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.043781+0000) 2019-10-13T06:38:23.048 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:23.043+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.043907+0000) 2019-10-13T06:38:23.048 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:23.044+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.044488+0000) 2019-10-13T06:38:23.050 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:23.042+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:23.042941+0000) 2019-10-13T06:38:23.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:23.047+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:24.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:24.248+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:24.248186+0000) 2019-10-13T06:38:28.054 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:28.049+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.049522+0000) 2019-10-13T06:38:28.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:28.049+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.049692+0000) 2019-10-13T06:38:28.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:28.049+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.049810+0000) 2019-10-13T06:38:28.055 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:28.049+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.050106+0000) 2019-10-13T06:38:28.055 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:28.049+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.050284+0000) 2019-10-13T06:38:28.055 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:28.050+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.050438+0000) 2019-10-13T06:38:28.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:28.045+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.046430+0000) 2019-10-13T06:38:28.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:28.048+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.049227+0000) 2019-10-13T06:38:28.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:28.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:28.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:28.293+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.293855+0000) 2019-10-13T06:38:28.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:28.632+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.632384+0000) 2019-10-13T06:38:28.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:28.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:28.632386+0000) 2019-10-13T06:38:31.305 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:31.296+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:31.296670+0000) 2019-10-13T06:38:31.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:31.885+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:31.886649+0000) 2019-10-13T06:38:32.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:32.749+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:32.750583+0000) 2019-10-13T06:38:32.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:32.750+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:32.751559+0000) 2019-10-13T06:38:32.796 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:38:32.797 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:32.802 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:38:32.802 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:33.059 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:33.053+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.054998+0000) 2019-10-13T06:38:33.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:33.054+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.055416+0000) 2019-10-13T06:38:33.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:33.054+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.055505+0000) 2019-10-13T06:38:33.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:33.054+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.055784+0000) 2019-10-13T06:38:33.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:33.054+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.055919+0000) 2019-10-13T06:38:33.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:33.054+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.056008+0000) 2019-10-13T06:38:33.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:33.051+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.052190+0000) 2019-10-13T06:38:33.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:33.052+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:33.053572+0000) 2019-10-13T06:38:33.061 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:33.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:34.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:34.247+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:34.248362+0000) 2019-10-13T06:38:38.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:38.058+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.059942+0000) 2019-10-13T06:38:38.064 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:38.058+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.060131+0000) 2019-10-13T06:38:38.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:38.056+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.056918+0000) 2019-10-13T06:38:38.065 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:38.059+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.060617+0000) 2019-10-13T06:38:38.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:38.059+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.060936+0000) 2019-10-13T06:38:38.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:38.059+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.061068+0000) 2019-10-13T06:38:38.066 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:38.059+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.061334+0000) 2019-10-13T06:38:38.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:38.058+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.059372+0000) 2019-10-13T06:38:38.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:38.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103894 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:38.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:38.293+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.294027+0000) 2019-10-13T06:38:38.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:38.631+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.632575+0000) 2019-10-13T06:38:38.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:38.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:38.632574+0000) 2019-10-13T06:38:41.306 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:41.298+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:41.298928+0000) 2019-10-13T06:38:41.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:41.886+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:41.886848+0000) 2019-10-13T06:38:42.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:42.749+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:42.750734+0000) 2019-10-13T06:38:42.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:42.751+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:42.751753+0000) 2019-10-13T06:38:43.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:43.062+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.062924+0000) 2019-10-13T06:38:43.071 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:43.065+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.066885+0000) 2019-10-13T06:38:43.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:43.065+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.066986+0000) 2019-10-13T06:38:43.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:43.065+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.067267+0000) 2019-10-13T06:38:43.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:43.065+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.067301+0000) 2019-10-13T06:38:43.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:43.066+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.067698+0000) 2019-10-13T06:38:43.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:43.064+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.065334+0000) 2019-10-13T06:38:43.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:43.066+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:43.067943+0000) 2019-10-13T06:38:43.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:43.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:44.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:44.247+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:44.248549+0000) 2019-10-13T06:38:48.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:48.067+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.068607+0000) 2019-10-13T06:38:48.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:48.070+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.072005+0000) 2019-10-13T06:38:48.080 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:48.070+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.072242+0000) 2019-10-13T06:38:48.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:48.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.072722+0000) 2019-10-13T06:38:48.080 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:48.071+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.072963+0000) 2019-10-13T06:38:48.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:48.071+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.073030+0000) 2019-10-13T06:38:48.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:48.071+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.073363+0000) 2019-10-13T06:38:48.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:48.074+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103953 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:48.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:48.070+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.071335+0000) 2019-10-13T06:38:48.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:48.293+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.294200+0000) 2019-10-13T06:38:48.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:48.631+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.632704+0000) 2019-10-13T06:38:48.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:48.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:48.632731+0000) 2019-10-13T06:38:51.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:51.301+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:51.301428+0000) 2019-10-13T06:38:51.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:51.886+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:51.887035+0000) 2019-10-13T06:38:52.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:52.750+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:52.750920+0000) 2019-10-13T06:38:52.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:52.751+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:52.751886+0000) 2019-10-13T06:38:53.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:53.073+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.074656+0000) 2019-10-13T06:38:53.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:53.076+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.078370+0000) 2019-10-13T06:38:53.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:53.077+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.078559+0000) 2019-10-13T06:38:53.083 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:53.077+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.078790+0000) 2019-10-13T06:38:53.083 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:53.077+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.078877+0000) 2019-10-13T06:38:53.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:53.077+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.079058+0000) 2019-10-13T06:38:53.084 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:53.078+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.079460+0000) 2019-10-13T06:38:53.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:53.077+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:53.078014+0000) 2019-10-13T06:38:53.085 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:53.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 103987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:54.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:54.248+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:54.248726+0000) 2019-10-13T06:38:58.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:38:58.078+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.080503+0000) 2019-10-13T06:38:58.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:38:58.083+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.083497+0000) 2019-10-13T06:38:58.090 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:58.083+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.083828+0000) 2019-10-13T06:38:58.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:38:58.084+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.084449+0000) 2019-10-13T06:38:58.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:58.084+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.084526+0000) 2019-10-13T06:38:58.091 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:38:58.084+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.084620+0000) 2019-10-13T06:38:58.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:58.084+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.084875+0000) 2019-10-13T06:38:58.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:38:58.081+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.082850+0000) 2019-10-13T06:38:58.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:38:58.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:38:58.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:38:58.293+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.294344+0000) 2019-10-13T06:38:58.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:38:58.632+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.632866+0000) 2019-10-13T06:38:58.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:38:58.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:38:58.632908+0000) 2019-10-13T06:39:01.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:01.303+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:01.303971+0000) 2019-10-13T06:39:01.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:01.886+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:01.887225+0000) 2019-10-13T06:39:02.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:02.750+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:02.751102+0000) 2019-10-13T06:39:02.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:02.751+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:02.752033+0000) 2019-10-13T06:39:02.942 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:39:02.943 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:02.947 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:39:02.947 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:03.094 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:03.088+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.088826+0000) 2019-10-13T06:39:03.094 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:03.088+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.088968+0000) 2019-10-13T06:39:03.094 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:03.088+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.089066+0000) 2019-10-13T06:39:03.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:03.089+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.089499+0000) 2019-10-13T06:39:03.095 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:03.089+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.089895+0000) 2019-10-13T06:39:03.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:03.089+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.090066+0000) 2019-10-13T06:39:03.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:03.084+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.086123+0000) 2019-10-13T06:39:03.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:03.086+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:03.088225+0000) 2019-10-13T06:39:03.097 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:03.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:04.253 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:04.248+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:04.248933+0000) 2019-10-13T06:39:08.097 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:08.093+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.093934+0000) 2019-10-13T06:39:08.098 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:08.094+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.094421+0000) 2019-10-13T06:39:08.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:08.094+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.094589+0000) 2019-10-13T06:39:08.098 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:08.094+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.094778+0000) 2019-10-13T06:39:08.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:08.094+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.095137+0000) 2019-10-13T06:39:08.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:08.089+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.091505+0000) 2019-10-13T06:39:08.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:08.095+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.095461+0000) 2019-10-13T06:39:08.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:08.091+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.093121+0000) 2019-10-13T06:39:08.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:08.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:08.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:08.294+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.294515+0000) 2019-10-13T06:39:08.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:08.632+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.633034+0000) 2019-10-13T06:39:08.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:08.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:08.633074+0000) 2019-10-13T06:39:11.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:11.305+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:11.306298+0000) 2019-10-13T06:39:11.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:11.887+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:11.887417+0000) 2019-10-13T06:39:12.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:12.751+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:12.751295+0000) 2019-10-13T06:39:12.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:12.751+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:12.752204+0000) 2019-10-13T06:39:13.103 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:13.099+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.099788+0000) 2019-10-13T06:39:13.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:13.099+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.099945+0000) 2019-10-13T06:39:13.104 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:13.099+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.100382+0000) 2019-10-13T06:39:13.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:13.100+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.100725+0000) 2019-10-13T06:39:13.105 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:13.100+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.101166+0000) 2019-10-13T06:39:13.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:13.100+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.101328+0000) 2019-10-13T06:39:13.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:13.096+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.097468+0000) 2019-10-13T06:39:13.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:13.098+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:13.098908+0000) 2019-10-13T06:39:13.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:13.104+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:14.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:14.249+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:14.249105+0000) 2019-10-13T06:39:18.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:18.102+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.103070+0000) 2019-10-13T06:39:18.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:18.105+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.106862+0000) 2019-10-13T06:39:18.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:18.105+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.106729+0000) 2019-10-13T06:39:18.112 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:18.105+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.107071+0000) 2019-10-13T06:39:18.112 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:18.105+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.107233+0000) 2019-10-13T06:39:18.113 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:18.106+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.107417+0000) 2019-10-13T06:39:18.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:18.106+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.107608+0000) 2019-10-13T06:39:18.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:18.104+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.104988+0000) 2019-10-13T06:39:18.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:18.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104134 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:18.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:18.293+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.294696+0000) 2019-10-13T06:39:18.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:18.632+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.633219+0000) 2019-10-13T06:39:18.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:18.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:18.633250+0000) 2019-10-13T06:39:21.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:21.307+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:21.308658+0000) 2019-10-13T06:39:21.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:21.886+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:21.887596+0000) 2019-10-13T06:39:22.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:22.750+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:22.751476+0000) 2019-10-13T06:39:22.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:22.751+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:22.752388+0000) 2019-10-13T06:39:23.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:23.111+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.112830+0000) 2019-10-13T06:39:23.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:23.111+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.113320+0000) 2019-10-13T06:39:23.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:23.112+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.113479+0000) 2019-10-13T06:39:23.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:23.109+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.110351+0000) 2019-10-13T06:39:23.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:23.112+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.114006+0000) 2019-10-13T06:39:23.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:23.112+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.114198+0000) 2019-10-13T06:39:23.118 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:23.113+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.114422+0000) 2019-10-13T06:39:23.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:23.111+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:23.111775+0000) 2019-10-13T06:39:23.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:23.116+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:24.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:24.248+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:24.249278+0000) 2019-10-13T06:39:28.123 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:28.117+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.119351+0000) 2019-10-13T06:39:28.123 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:28.118+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.119628+0000) 2019-10-13T06:39:28.124 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:28.118+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.119854+0000) 2019-10-13T06:39:28.125 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:28.115+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.116450+0000) 2019-10-13T06:39:28.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:28.119+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.120423+0000) 2019-10-13T06:39:28.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:28.119+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.120635+0000) 2019-10-13T06:39:28.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:28.119+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.120873+0000) 2019-10-13T06:39:28.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:28.117+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.118269+0000) 2019-10-13T06:39:28.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:28.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104193 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:28.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:28.294+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.294861+0000) 2019-10-13T06:39:28.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:28.632+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.633436+0000) 2019-10-13T06:39:28.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:28.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:28.633432+0000) 2019-10-13T06:39:31.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:31.310+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:31.311008+0000) 2019-10-13T06:39:31.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:31.887+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:31.887789+0000) 2019-10-13T06:39:32.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:32.751+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:32.751683+0000) 2019-10-13T06:39:32.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:32.752+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:32.752596+0000) 2019-10-13T06:39:33.106 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:39:33.106 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:33.110 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:39:33.110 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:33.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:33.123+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.124974+0000) 2019-10-13T06:39:33.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:33.124+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.125494+0000) 2019-10-13T06:39:33.129 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:33.124+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.125720+0000) 2019-10-13T06:39:33.130 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:33.121+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.122119+0000) 2019-10-13T06:39:33.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:33.124+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.125847+0000) 2019-10-13T06:39:33.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:33.124+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.126327+0000) 2019-10-13T06:39:33.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:33.125+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.126566+0000) 2019-10-13T06:39:33.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:33.123+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:33.123921+0000) 2019-10-13T06:39:33.132 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:33.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:34.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:34.248+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:34.249446+0000) 2019-10-13T06:39:38.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:38.126+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.126924+0000) 2019-10-13T06:39:38.134 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:38.129+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.130569+0000) 2019-10-13T06:39:38.135 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:38.129+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.130841+0000) 2019-10-13T06:39:38.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:38.129+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.131106+0000) 2019-10-13T06:39:38.135 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:38.130+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.131625+0000) 2019-10-13T06:39:38.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:38.130+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.131800+0000) 2019-10-13T06:39:38.136 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:38.130+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.131990+0000) 2019-10-13T06:39:38.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:38.128+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.129626+0000) 2019-10-13T06:39:38.138 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:38.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:38.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:38.294+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.295029+0000) 2019-10-13T06:39:38.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:38.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.633609+0000) 2019-10-13T06:39:38.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:38.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:38.633608+0000) 2019-10-13T06:39:41.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:41.313+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:41.313251+0000) 2019-10-13T06:39:41.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:41.887+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:41.888047+0000) 2019-10-13T06:39:42.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:42.751+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:42.751860+0000) 2019-10-13T06:39:42.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:42.752+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:42.752796+0000) 2019-10-13T06:39:43.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:43.132+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.133176+0000) 2019-10-13T06:39:43.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:43.136+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.136521+0000) 2019-10-13T06:39:43.143 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:43.136+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.136773+0000) 2019-10-13T06:39:43.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:43.136+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.136894+0000) 2019-10-13T06:39:43.144 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:43.136+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.137303+0000) 2019-10-13T06:39:43.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:43.137+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.137561+0000) 2019-10-13T06:39:43.144 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:43.137+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.137806+0000) 2019-10-13T06:39:43.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:43.135+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:43.135801+0000) 2019-10-13T06:39:43.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:43.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:44.253 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:44.249+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:44.249639+0000) 2019-10-13T06:39:48.146 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:48.142+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.142424+0000) 2019-10-13T06:39:48.147 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:48.142+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.142591+0000) 2019-10-13T06:39:48.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:48.142+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.142837+0000) 2019-10-13T06:39:48.147 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:48.142+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.143147+0000) 2019-10-13T06:39:48.147 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:48.142+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.143258+0000) 2019-10-13T06:39:48.148 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:48.142+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.143284+0000) 2019-10-13T06:39:48.148 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:48.138+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.140233+0000) 2019-10-13T06:39:48.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:48.140+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.142517+0000) 2019-10-13T06:39:48.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:48.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:48.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:48.294+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.295199+0000) 2019-10-13T06:39:48.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:48.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.633754+0000) 2019-10-13T06:39:48.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:48.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:48.633729+0000) 2019-10-13T06:39:51.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:51.314+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:51.315667+0000) 2019-10-13T06:39:51.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:51.887+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:51.888199+0000) 2019-10-13T06:39:52.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:52.751+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:52.752020+0000) 2019-10-13T06:39:52.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:52.752+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:52.752979+0000) 2019-10-13T06:39:53.151 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:53.147+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.147984+0000) 2019-10-13T06:39:53.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:53.147+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.148334+0000) 2019-10-13T06:39:53.153 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:53.143+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.145481+0000) 2019-10-13T06:39:53.153 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:53.148+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.148568+0000) 2019-10-13T06:39:53.153 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:53.148+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.148800+0000) 2019-10-13T06:39:53.153 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:53.148+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.148833+0000) 2019-10-13T06:39:53.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:53.148+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.149176+0000) 2019-10-13T06:39:53.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:53.145+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:53.146817+0000) 2019-10-13T06:39:53.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:53.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:54.253 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:54.249+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:54.249845+0000) 2019-10-13T06:39:58.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:58.153+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.154145+0000) 2019-10-13T06:39:58.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:39:58.153+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.154248+0000) 2019-10-13T06:39:58.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:58.153+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.154367+0000) 2019-10-13T06:39:58.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:39:58.154+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.154535+0000) 2019-10-13T06:39:58.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:58.154+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.154624+0000) 2019-10-13T06:39:58.159 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:39:58.154+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.154873+0000) 2019-10-13T06:39:58.167 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:39:58.149+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.151377+0000) 2019-10-13T06:39:58.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:39:58.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:39:58.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:39:58.152+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.153834+0000) 2019-10-13T06:39:58.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:39:58.294+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.295365+0000) 2019-10-13T06:39:58.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:39:58.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.633871+0000) 2019-10-13T06:39:58.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:39:58.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:39:58.633866+0000) 2019-10-13T06:40:01.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:01.317+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:01.318049+0000) 2019-10-13T06:40:01.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:01.887+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:01.888378+0000) 2019-10-13T06:40:02.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:02.751+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:02.752162+0000) 2019-10-13T06:40:02.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:02.752+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:02.753162+0000) 2019-10-13T06:40:03.162 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:03.154+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.155108+0000) 2019-10-13T06:40:03.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:03.157+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.158688+0000) 2019-10-13T06:40:03.163 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:03.157+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.158834+0000) 2019-10-13T06:40:03.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:03.157+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.159141+0000) 2019-10-13T06:40:03.164 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:03.157+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.159299+0000) 2019-10-13T06:40:03.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:03.158+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.159760+0000) 2019-10-13T06:40:03.164 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:03.158+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.159860+0000) 2019-10-13T06:40:03.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:03.157+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:03.158200+0000) 2019-10-13T06:40:03.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:03.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:03.251 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:40:03.251 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:03.257 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:40:03.257 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:04.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:04.249+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:04.250084+0000) 2019-10-13T06:40:08.168 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:08.163+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.164795+0000) 2019-10-13T06:40:08.169 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:08.163+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.165158+0000) 2019-10-13T06:40:08.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:08.163+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.165386+0000) 2019-10-13T06:40:08.170 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:08.161+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.162284+0000) 2019-10-13T06:40:08.170 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:08.164+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.165635+0000) 2019-10-13T06:40:08.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:08.164+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.165902+0000) 2019-10-13T06:40:08.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:08.164+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.166046+0000) 2019-10-13T06:40:08.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:08.162+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.163709+0000) 2019-10-13T06:40:08.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:08.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104433 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:08.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:08.294+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.295541+0000) 2019-10-13T06:40:08.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:08.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.634002+0000) 2019-10-13T06:40:08.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:08.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:08.634003+0000) 2019-10-13T06:40:11.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:11.319+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:11.320546+0000) 2019-10-13T06:40:11.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:11.887+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:11.888591+0000) 2019-10-13T06:40:12.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:12.750+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:12.752350+0000) 2019-10-13T06:40:12.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:12.752+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:12.753324+0000) 2019-10-13T06:40:13.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:13.169+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.171302+0000) 2019-10-13T06:40:13.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:13.170+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.171556+0000) 2019-10-13T06:40:13.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:13.170+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.171645+0000) 2019-10-13T06:40:13.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:13.170+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.171844+0000) 2019-10-13T06:40:13.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:13.170+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.171935+0000) 2019-10-13T06:40:13.177 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:13.168+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.169357+0000) 2019-10-13T06:40:13.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:13.171+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.172404+0000) 2019-10-13T06:40:13.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:13.170+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:13.171224+0000) 2019-10-13T06:40:13.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:13.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:14.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:14.249+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:14.250287+0000) 2019-10-13T06:40:18.181 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:18.175+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.177166+0000) 2019-10-13T06:40:18.181 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:18.175+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.177349+0000) 2019-10-13T06:40:18.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:18.176+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.177622+0000) 2019-10-13T06:40:18.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:18.176+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.177850+0000) 2019-10-13T06:40:18.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:18.174+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.175277+0000) 2019-10-13T06:40:18.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:18.176+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.178200+0000) 2019-10-13T06:40:18.183 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:18.177+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.178527+0000) 2019-10-13T06:40:18.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:18.176+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.177183+0000) 2019-10-13T06:40:18.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:18.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:18.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:18.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.295707+0000) 2019-10-13T06:40:18.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:18.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.634186+0000) 2019-10-13T06:40:18.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:18.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:18.634186+0000) 2019-10-13T06:40:21.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:21.322+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:21.323062+0000) 2019-10-13T06:40:21.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:21.888+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:21.888797+0000) 2019-10-13T06:40:22.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:22.751+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:22.752533+0000) 2019-10-13T06:40:22.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:22.753+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:22.753491+0000) 2019-10-13T06:40:23.187 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:23.181+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.182982+0000) 2019-10-13T06:40:23.187 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:23.181+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.183388+0000) 2019-10-13T06:40:23.187 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:23.182+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.183673+0000) 2019-10-13T06:40:23.187 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:23.179+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.179728+0000) 2019-10-13T06:40:23.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:23.182+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.183979+0000) 2019-10-13T06:40:23.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:23.182+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.184208+0000) 2019-10-13T06:40:23.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:23.183+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.184430+0000) 2019-10-13T06:40:23.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:23.181+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:23.182262+0000) 2019-10-13T06:40:23.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:23.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:24.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:24.249+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:24.250485+0000) 2019-10-13T06:40:28.192 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:28.184+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.185274+0000) 2019-10-13T06:40:28.193 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:28.187+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.188971+0000) 2019-10-13T06:40:28.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:28.187+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.189215+0000) 2019-10-13T06:40:28.193 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:28.188+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.189468+0000) 2019-10-13T06:40:28.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:28.188+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.189785+0000) 2019-10-13T06:40:28.194 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:28.188+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.189952+0000) 2019-10-13T06:40:28.194 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:28.188+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.190108+0000) 2019-10-13T06:40:28.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:28.187+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.187969+0000) 2019-10-13T06:40:28.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:28.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:28.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:28.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.295877+0000) 2019-10-13T06:40:28.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:28.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.634354+0000) 2019-10-13T06:40:28.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:28.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:28.634355+0000) 2019-10-13T06:40:31.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:31.325+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:31.325297+0000) 2019-10-13T06:40:31.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:31.888+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:31.888962+0000) 2019-10-13T06:40:32.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:32.752+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:32.752675+0000) 2019-10-13T06:40:32.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:32.753+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:32.753713+0000) 2019-10-13T06:40:33.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:33.194+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.195249+0000) 2019-10-13T06:40:33.199 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:33.195+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.195642+0000) 2019-10-13T06:40:33.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:33.195+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.195903+0000) 2019-10-13T06:40:33.201 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:33.191+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.192376+0000) 2019-10-13T06:40:33.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:33.195+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.196072+0000) 2019-10-13T06:40:33.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:33.195+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.196197+0000) 2019-10-13T06:40:33.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:33.196+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.196538+0000) 2019-10-13T06:40:33.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:33.194+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:33.195394+0000) 2019-10-13T06:40:33.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:33.199+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:33.405 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:40:33.405 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:33.411 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:40:33.411 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:34.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:34.250+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:34.250671+0000) 2019-10-13T06:40:38.206 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:38.201+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.201474+0000) 2019-10-13T06:40:38.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:38.201+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.201726+0000) 2019-10-13T06:40:38.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:38.201+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.202086+0000) 2019-10-13T06:40:38.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:38.201+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.202370+0000) 2019-10-13T06:40:38.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:38.202+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.202531+0000) 2019-10-13T06:40:38.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:38.202+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.202691+0000) 2019-10-13T06:40:38.209 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:38.198+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.198820+0000) 2019-10-13T06:40:38.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:38.199+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.199878+0000) 2019-10-13T06:40:38.209 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:38.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104614 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:38.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:38.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.296070+0000) 2019-10-13T06:40:38.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:38.634+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.634459+0000) 2019-10-13T06:40:38.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:38.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:38.634540+0000) 2019-10-13T06:40:41.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:41.327+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:41.327681+0000) 2019-10-13T06:40:41.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:41.888+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:41.889138+0000) 2019-10-13T06:40:42.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:42.752+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:42.752864+0000) 2019-10-13T06:40:42.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:42.752+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:42.753911+0000) 2019-10-13T06:40:43.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:43.207+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.207822+0000) 2019-10-13T06:40:43.213 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:43.207+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.208222+0000) 2019-10-13T06:40:43.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:43.207+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.208074+0000) 2019-10-13T06:40:43.214 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:43.208+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.208620+0000) 2019-10-13T06:40:43.214 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:43.208+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.208784+0000) 2019-10-13T06:40:43.214 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:43.208+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.208988+0000) 2019-10-13T06:40:43.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:43.202+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.204565+0000) 2019-10-13T06:40:43.215 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:43.206+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:43.208143+0000) 2019-10-13T06:40:43.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:43.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:44.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:44.250+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:44.250862+0000) 2019-10-13T06:40:48.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:48.212+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.214233+0000) 2019-10-13T06:40:48.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:48.209+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.210990+0000) 2019-10-13T06:40:48.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:48.213+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.214767+0000) 2019-10-13T06:40:48.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:48.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.214884+0000) 2019-10-13T06:40:48.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:48.213+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.215077+0000) 2019-10-13T06:40:48.220 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:48.214+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.215522+0000) 2019-10-13T06:40:48.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:48.214+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.215680+0000) 2019-10-13T06:40:48.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:48.211+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.213310+0000) 2019-10-13T06:40:48.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:48.217+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104673 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:48.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:48.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.296257+0000) 2019-10-13T06:40:48.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:48.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.634574+0000) 2019-10-13T06:40:48.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:48.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:48.634663+0000) 2019-10-13T06:40:51.337 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:51.329+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:51.329953+0000) 2019-10-13T06:40:51.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:51.887+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:51.889357+0000) 2019-10-13T06:40:52.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:52.751+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:52.753060+0000) 2019-10-13T06:40:52.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:52.753+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:52.754094+0000) 2019-10-13T06:40:53.224 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:53.215+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.217075+0000) 2019-10-13T06:40:53.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:53.219+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.220747+0000) 2019-10-13T06:40:53.225 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:53.219+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.220876+0000) 2019-10-13T06:40:53.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:53.219+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.221387+0000) 2019-10-13T06:40:53.226 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:53.220+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.221619+0000) 2019-10-13T06:40:53.226 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:53.220+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.221849+0000) 2019-10-13T06:40:53.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:53.220+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.222021+0000) 2019-10-13T06:40:53.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:53.218+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:53.219804+0000) 2019-10-13T06:40:53.228 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:53.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:54.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:54.250+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:54.251068+0000) 2019-10-13T06:40:58.230 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:58.225+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.226724+0000) 2019-10-13T06:40:58.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:58.225+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.227068+0000) 2019-10-13T06:40:58.231 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:40:58.225+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.227248+0000) 2019-10-13T06:40:58.231 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:40:58.225+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.227383+0000) 2019-10-13T06:40:58.232 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:40:58.222+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.223506+0000) 2019-10-13T06:40:58.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:40:58.226+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.227881+0000) 2019-10-13T06:40:58.232 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:58.226+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.228019+0000) 2019-10-13T06:40:58.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:40:58.225+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.226275+0000) 2019-10-13T06:40:58.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:40:58.230+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104734 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:40:58.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:40:58.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.296410+0000) 2019-10-13T06:40:58.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:40:58.633+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.634757+0000) 2019-10-13T06:40:58.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:40:58.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:40:58.634812+0000) 2019-10-13T06:41:01.339 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:01.331+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:01.332241+0000) 2019-10-13T06:41:01.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:01.888+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:01.889476+0000) 2019-10-13T06:41:02.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:02.752+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:02.753263+0000) 2019-10-13T06:41:02.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:02.753+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:02.754309+0000) 2019-10-13T06:41:03.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:03.231+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.232978+0000) 2019-10-13T06:41:03.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:03.231+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.233195+0000) 2019-10-13T06:41:03.238 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:03.229+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.230000+0000) 2019-10-13T06:41:03.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:03.232+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.233543+0000) 2019-10-13T06:41:03.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:03.232+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.233683+0000) 2019-10-13T06:41:03.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:03.232+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.233872+0000) 2019-10-13T06:41:03.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:03.233+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.234431+0000) 2019-10-13T06:41:03.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:03.232+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:03.233148+0000) 2019-10-13T06:41:03.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:03.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:03.517 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:41:03.517 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:03.521 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:41:03.522 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:04.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:04.250+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:04.251253+0000) 2019-10-13T06:41:08.242 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:08.237+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.238903+0000) 2019-10-13T06:41:08.243 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:08.238+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.239433+0000) 2019-10-13T06:41:08.243 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:08.238+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.239581+0000) 2019-10-13T06:41:08.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:08.238+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.240062+0000) 2019-10-13T06:41:08.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:08.238+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.240222+0000) 2019-10-13T06:41:08.244 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:08.236+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.237114+0000) 2019-10-13T06:41:08.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:08.239+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.240809+0000) 2019-10-13T06:41:08.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:08.237+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.238539+0000) 2019-10-13T06:41:08.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:08.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:08.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:08.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.296555+0000) 2019-10-13T06:41:08.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:08.634+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.634931+0000) 2019-10-13T06:41:08.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:08.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:08.635019+0000) 2019-10-13T06:41:11.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:11.333+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:11.334700+0000) 2019-10-13T06:41:11.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:11.889+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:11.889652+0000) 2019-10-13T06:41:12.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:12.752+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:12.753422+0000) 2019-10-13T06:41:12.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:12.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:12.754487+0000) 2019-10-13T06:41:13.249 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:13.244+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.245393+0000) 2019-10-13T06:41:13.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:13.244+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.245668+0000) 2019-10-13T06:41:13.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:13.244+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.245792+0000) 2019-10-13T06:41:13.250 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:13.244+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.245969+0000) 2019-10-13T06:41:13.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:13.244+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.246045+0000) 2019-10-13T06:41:13.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:13.244+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.246310+0000) 2019-10-13T06:41:13.251 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:13.242+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.242817+0000) 2019-10-13T06:41:13.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:13.244+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:13.245405+0000) 2019-10-13T06:41:13.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:13.247+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104827 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:14.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:14.251+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:14.251444+0000) 2019-10-13T06:41:18.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:18.250+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.251356+0000) 2019-10-13T06:41:18.255 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:18.251+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.251482+0000) 2019-10-13T06:41:18.256 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:18.251+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.251636+0000) 2019-10-13T06:41:18.256 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:18.251+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.252102+0000) 2019-10-13T06:41:18.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:18.251+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.252233+0000) 2019-10-13T06:41:18.257 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:18.247+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.248562+0000) 2019-10-13T06:41:18.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:18.249+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.250296+0000) 2019-10-13T06:41:18.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:18.252+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.252612+0000) 2019-10-13T06:41:18.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:18.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:18.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:18.296+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.296713+0000) 2019-10-13T06:41:18.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:18.634+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.635042+0000) 2019-10-13T06:41:18.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:18.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:18.635134+0000) 2019-10-13T06:41:21.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:21.336+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:21.337050+0000) 2019-10-13T06:41:21.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:21.889+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:21.889838+0000) 2019-10-13T06:41:22.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:22.753+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:22.753570+0000) 2019-10-13T06:41:22.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:22.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:22.754671+0000) 2019-10-13T06:41:23.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:23.253+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.254376+0000) 2019-10-13T06:41:23.647 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:23.255+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.256559+0000) 2019-10-13T06:41:23.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:23.256+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.256874+0000) 2019-10-13T06:41:23.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:23.256+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.257231+0000) 2019-10-13T06:41:23.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:23.256+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.257341+0000) 2019-10-13T06:41:23.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:23.257+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.257483+0000) 2019-10-13T06:41:23.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:23.257+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.257803+0000) 2019-10-13T06:41:23.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:23.257+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:23.258103+0000) 2019-10-13T06:41:23.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:23.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:24.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:24.251+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:24.251631+0000) 2019-10-13T06:41:28.266 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:28.261+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.262263+0000) 2019-10-13T06:41:28.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:28.262+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.262458+0000) 2019-10-13T06:41:28.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:28.262+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.262559+0000) 2019-10-13T06:41:28.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:28.262+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.262864+0000) 2019-10-13T06:41:28.267 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:28.258+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.259448+0000) 2019-10-13T06:41:28.268 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:28.262+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.263002+0000) 2019-10-13T06:41:28.268 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:28.262+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.263131+0000) 2019-10-13T06:41:28.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:28.260+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.261524+0000) 2019-10-13T06:41:28.270 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:28.266+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104913 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:28.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:28.296+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.296894+0000) 2019-10-13T06:41:28.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:28.635+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.635186+0000) 2019-10-13T06:41:28.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:28.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:28.635233+0000) 2019-10-13T06:41:31.347 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:31.339+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:31.339529+0000) 2019-10-13T06:41:31.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:31.889+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:31.890057+0000) 2019-10-13T06:41:32.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:32.753+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:32.753746+0000) 2019-10-13T06:41:32.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:32.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:32.754840+0000) 2019-10-13T06:41:33.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:33.266+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.268314+0000) 2019-10-13T06:41:33.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:33.267+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.268484+0000) 2019-10-13T06:41:33.272 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:33.267+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.268656+0000) 2019-10-13T06:41:33.273 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:33.267+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.268817+0000) 2019-10-13T06:41:33.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:33.265+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.266207+0000) 2019-10-13T06:41:33.273 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:33.267+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.269318+0000) 2019-10-13T06:41:33.274 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:33.268+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.269450+0000) 2019-10-13T06:41:33.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:33.266+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:33.267650+0000) 2019-10-13T06:41:33.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:33.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:33.673 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:41:33.673 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:33.677 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:41:33.677 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:34.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:34.250+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:34.251766+0000) 2019-10-13T06:41:38.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:38.270+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.271903+0000) 2019-10-13T06:41:38.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:38.274+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.275439+0000) 2019-10-13T06:41:38.280 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:38.274+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.275762+0000) 2019-10-13T06:41:38.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:38.274+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.275928+0000) 2019-10-13T06:41:38.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:38.274+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.276045+0000) 2019-10-13T06:41:38.281 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:38.274+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.276256+0000) 2019-10-13T06:41:38.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:38.275+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.276426+0000) 2019-10-13T06:41:38.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:38.272+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.274394+0000) 2019-10-13T06:41:38.283 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:38.278+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 104974 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:38.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:38.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.297030+0000) 2019-10-13T06:41:38.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:38.634+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.635318+0000) 2019-10-13T06:41:38.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:38.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:38.635364+0000) 2019-10-13T06:41:41.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:41.340+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:41.341869+0000) 2019-10-13T06:41:41.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:41.889+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:41.890263+0000) 2019-10-13T06:41:42.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:42.753+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:42.753931+0000) 2019-10-13T06:41:42.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:42.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:42.754990+0000) 2019-10-13T06:41:43.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:43.280+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.281672+0000) 2019-10-13T06:41:43.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:43.280+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.281778+0000) 2019-10-13T06:41:43.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:43.280+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.282078+0000) 2019-10-13T06:41:43.289 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:43.280+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.282340+0000) 2019-10-13T06:41:43.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:43.281+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.282537+0000) 2019-10-13T06:41:43.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:43.281+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.282866+0000) 2019-10-13T06:41:43.290 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:43.276+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.277965+0000) 2019-10-13T06:41:43.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:43.278+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:43.280038+0000) 2019-10-13T06:41:43.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:43.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:44.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:44.250+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:44.251884+0000) 2019-10-13T06:41:48.294 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:48.282+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.284399+0000) 2019-10-13T06:41:48.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:48.286+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.287658+0000) 2019-10-13T06:41:48.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:48.286+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.287981+0000) 2019-10-13T06:41:48.294 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:48.287+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.288418+0000) 2019-10-13T06:41:48.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:48.287+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.288778+0000) 2019-10-13T06:41:48.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:48.287+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.288960+0000) 2019-10-13T06:41:48.295 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:48.288+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.289504+0000) 2019-10-13T06:41:48.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:48.285+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.287365+0000) 2019-10-13T06:41:48.296 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:48.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:48.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:48.295+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.297123+0000) 2019-10-13T06:41:48.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:48.634+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.635451+0000) 2019-10-13T06:41:48.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:48.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:48.635452+0000) 2019-10-13T06:41:51.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:51.343+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:51.344327+0000) 2019-10-13T06:41:51.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:51.889+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:51.890432+0000) 2019-10-13T06:41:52.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:52.753+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:52.754138+0000) 2019-10-13T06:41:52.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:52.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:52.755147+0000) 2019-10-13T06:41:53.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:53.290+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.290901+0000) 2019-10-13T06:41:53.298 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:53.293+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.294479+0000) 2019-10-13T06:41:53.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:53.293+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.295000+0000) 2019-10-13T06:41:53.299 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:53.293+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.295132+0000) 2019-10-13T06:41:53.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:53.293+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.295285+0000) 2019-10-13T06:41:53.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:53.294+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.295433+0000) 2019-10-13T06:41:53.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:53.294+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.295745+0000) 2019-10-13T06:41:53.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:53.293+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:53.294309+0000) 2019-10-13T06:41:53.302 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:53.297+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105067 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:54.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:54.250+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:54.252060+0000) 2019-10-13T06:41:58.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:58.296+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.297267+0000) 2019-10-13T06:41:58.303 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:58.298+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.299903+0000) 2019-10-13T06:41:58.304 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:58.298+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.300013+0000) 2019-10-13T06:41:58.304 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:41:58.298+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.300287+0000) 2019-10-13T06:41:58.304 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:41:58.299+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.300689+0000) 2019-10-13T06:41:58.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:41:58.299+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.300869+0000) 2019-10-13T06:41:58.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:41:58.299+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.300981+0000) 2019-10-13T06:41:58.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:41:58.296+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.297470+0000) 2019-10-13T06:41:58.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:41:58.298+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.299319+0000) 2019-10-13T06:41:58.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:41:58.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105094 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:41:58.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:41:58.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.635560+0000) 2019-10-13T06:41:58.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:41:58.635+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:41:58.635632+0000) 2019-10-13T06:42:01.353 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:01.346+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:01.346728+0000) 2019-10-13T06:42:01.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:01.890+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:01.890616+0000) 2019-10-13T06:42:02.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:02.753+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:02.754289+0000) 2019-10-13T06:42:02.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:02.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:02.755332+0000) 2019-10-13T06:42:03.308 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:03.304+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.304840+0000) 2019-10-13T06:42:03.309 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:03.304+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.305117+0000) 2019-10-13T06:42:03.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:03.304+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.305171+0000) 2019-10-13T06:42:03.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:03.301+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.302343+0000) 2019-10-13T06:42:03.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:03.305+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.305462+0000) 2019-10-13T06:42:03.310 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:03.305+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.305778+0000) 2019-10-13T06:42:03.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:03.305+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.306130+0000) 2019-10-13T06:42:03.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:03.303+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:03.304215+0000) 2019-10-13T06:42:03.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:03.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:03.818 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:42:03.818 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:03.822 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:42:03.822 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:04.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:04.251+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:04.252236+0000) 2019-10-13T06:42:08.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:08.297+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.297434+0000) 2019-10-13T06:42:08.315 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:08.310+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.311036+0000) 2019-10-13T06:42:08.315 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:08.310+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.311262+0000) 2019-10-13T06:42:08.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:08.310+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.311377+0000) 2019-10-13T06:42:08.316 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:08.308+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.308816+0000) 2019-10-13T06:42:08.316 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:08.311+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.311913+0000) 2019-10-13T06:42:08.317 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:08.311+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.312044+0000) 2019-10-13T06:42:08.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:08.311+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.312309+0000) 2019-10-13T06:42:08.317 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:08.309+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.310574+0000) 2019-10-13T06:42:08.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:08.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105153 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:08.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:08.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.635729+0000) 2019-10-13T06:42:08.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:08.635+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:08.635767+0000) 2019-10-13T06:42:11.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:11.348+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:11.348990+0000) 2019-10-13T06:42:11.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:11.890+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:11.890729+0000) 2019-10-13T06:42:12.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:12.754+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:12.754474+0000) 2019-10-13T06:42:12.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:12.755+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:12.755456+0000) 2019-10-13T06:42:13.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:13.317+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.317466+0000) 2019-10-13T06:42:13.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:13.317+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.317658+0000) 2019-10-13T06:42:13.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:13.317+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.317917+0000) 2019-10-13T06:42:13.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:13.317+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.318016+0000) 2019-10-13T06:42:13.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:13.317+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.318187+0000) 2019-10-13T06:42:13.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:13.314+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.315327+0000) 2019-10-13T06:42:13.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:13.318+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.318898+0000) 2019-10-13T06:42:13.324 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:13.316+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:13.317302+0000) 2019-10-13T06:42:13.325 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:13.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:14.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:14.252+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:14.252430+0000) 2019-10-13T06:42:18.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:18.297+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.297623+0000) 2019-10-13T06:42:18.327 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:18.322+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.323286+0000) 2019-10-13T06:42:18.327 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:18.323+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.323466+0000) 2019-10-13T06:42:18.327 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:18.323+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.323616+0000) 2019-10-13T06:42:18.328 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:18.319+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.320532+0000) 2019-10-13T06:42:18.328 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:18.323+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.323951+0000) 2019-10-13T06:42:18.328 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:18.323+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.324218+0000) 2019-10-13T06:42:18.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:18.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.324566+0000) 2019-10-13T06:42:18.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:18.323+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.323945+0000) 2019-10-13T06:42:18.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:18.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:18.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:18.635+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.635913+0000) 2019-10-13T06:42:18.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:18.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:18.635910+0000) 2019-10-13T06:42:21.358 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:21.350+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:21.351211+0000) 2019-10-13T06:42:21.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:21.889+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:21.890840+0000) 2019-10-13T06:42:22.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:22.753+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:22.754602+0000) 2019-10-13T06:42:22.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:22.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:22.755588+0000) 2019-10-13T06:42:23.332 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:23.325+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.325817+0000) 2019-10-13T06:42:24.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:23.328+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.329655+0000) 2019-10-13T06:42:24.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:24.251+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:24.252601+0000) 2019-10-13T06:42:24.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:23.328+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.329664+0000) 2019-10-13T06:42:24.691 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:23.328+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.329719+0000) 2019-10-13T06:42:24.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:23.328+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.330170+0000) 2019-10-13T06:42:24.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:23.328+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.330366+0000) 2019-10-13T06:42:24.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:23.328+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.330317+0000) 2019-10-13T06:42:24.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:23.331+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105246 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:24.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:23.327+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:23.328330+0000) 2019-10-13T06:42:28.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:28.296+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.297819+0000) 2019-10-13T06:42:28.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:28.333+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.334486+0000) 2019-10-13T06:42:28.338 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:28.330+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.331633+0000) 2019-10-13T06:42:28.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:28.333+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.334635+0000) 2019-10-13T06:42:28.339 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:28.333+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.335056+0000) 2019-10-13T06:42:28.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:28.334+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.335416+0000) 2019-10-13T06:42:28.339 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:28.333+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.335232+0000) 2019-10-13T06:42:28.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:28.334+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.335971+0000) 2019-10-13T06:42:28.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:28.337+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:28.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:28.334+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.334839+0000) 2019-10-13T06:42:28.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:28.635+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.636091+0000) 2019-10-13T06:42:28.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:28.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:28.636090+0000) 2019-10-13T06:42:31.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:31.352+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:31.353537+0000) 2019-10-13T06:42:31.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:31.890+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:31.891012+0000) 2019-10-13T06:42:32.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:32.754+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:32.754742+0000) 2019-10-13T06:42:32.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:32.754+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:32.755770+0000) 2019-10-13T06:42:33.345 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:33.340+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.341406+0000) 2019-10-13T06:42:33.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:33.340+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.341736+0000) 2019-10-13T06:42:33.346 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:33.340+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.341550+0000) 2019-10-13T06:42:33.346 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:33.340+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.342124+0000) 2019-10-13T06:42:33.348 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:33.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.339030+0000) 2019-10-13T06:42:33.348 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:33.338+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.340174+0000) 2019-10-13T06:42:33.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:33.341+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.342569+0000) 2019-10-13T06:42:33.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:33.340+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:33.342242+0000) 2019-10-13T06:42:33.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:33.343+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105307 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:33.972 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:42:33.973 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:33.977 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:42:33.977 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:34.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:34.251+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:34.252791+0000) 2019-10-13T06:42:38.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:38.297+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.297984+0000) 2019-10-13T06:42:38.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:38.345+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.346583+0000) 2019-10-13T06:42:38.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:38.345+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.346727+0000) 2019-10-13T06:42:38.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:38.342+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.343725+0000) 2019-10-13T06:42:38.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:38.345+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.347196+0000) 2019-10-13T06:42:38.351 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:38.345+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.347254+0000) 2019-10-13T06:42:38.352 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:38.346+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.347480+0000) 2019-10-13T06:42:38.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:38.346+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.347653+0000) 2019-10-13T06:42:38.352 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:38.343+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.345419+0000) 2019-10-13T06:42:38.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:38.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:38.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:38.634+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.636284+0000) 2019-10-13T06:42:38.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:38.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:38.636284+0000) 2019-10-13T06:42:41.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:41.355+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:41.355821+0000) 2019-10-13T06:42:41.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:41.890+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:41.891203+0000) 2019-10-13T06:42:42.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:42.754+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:42.754866+0000) 2019-10-13T06:42:42.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:42.755+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:42.755948+0000) 2019-10-13T06:42:43.357 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:43.347+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.348896+0000) 2019-10-13T06:42:43.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:43.350+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.352077+0000) 2019-10-13T06:42:43.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:43.350+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.352309+0000) 2019-10-13T06:42:43.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:43.351+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.352421+0000) 2019-10-13T06:42:43.382 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:43.351+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.352540+0000) 2019-10-13T06:42:43.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:43.351+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.352753+0000) 2019-10-13T06:42:43.382 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:43.351+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.353159+0000) 2019-10-13T06:42:43.383 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:43.351+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:43.352357+0000) 2019-10-13T06:42:43.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:43.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:44.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:44.252+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:44.252980+0000) 2019-10-13T06:42:48.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:48.297+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.298164+0000) 2019-10-13T06:42:48.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:48.357+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.357639+0000) 2019-10-13T06:42:48.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:48.357+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.357887+0000) 2019-10-13T06:42:48.362 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:48.357+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.358235+0000) 2019-10-13T06:42:48.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:48.358+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.358470+0000) 2019-10-13T06:42:48.363 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:48.353+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.354690+0000) 2019-10-13T06:42:48.363 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:48.358+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.358785+0000) 2019-10-13T06:42:48.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:48.358+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.359065+0000) 2019-10-13T06:42:48.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:48.357+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.358184+0000) 2019-10-13T06:42:48.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:48.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:48.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:48.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.636421+0000) 2019-10-13T06:42:48.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:48.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:48.636425+0000) 2019-10-13T06:42:51.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:51.357+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:51.358214+0000) 2019-10-13T06:42:51.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:51.891+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:51.891421+0000) 2019-10-13T06:42:52.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:52.754+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:52.755048+0000) 2019-10-13T06:42:52.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:52.755+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:52.756096+0000) 2019-10-13T06:42:53.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:53.363+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.364016+0000) 2019-10-13T06:42:53.368 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:53.363+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.364218+0000) 2019-10-13T06:42:53.368 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:53.363+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.364360+0000) 2019-10-13T06:42:53.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:53.360+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.361574+0000) 2019-10-13T06:42:53.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:53.364+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.365160+0000) 2019-10-13T06:42:53.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:53.365+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.365580+0000) 2019-10-13T06:42:53.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:53.365+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.365885+0000) 2019-10-13T06:42:53.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:53.363+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:53.363902+0000) 2019-10-13T06:42:53.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:53.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:54.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:54.252+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:54.253140+0000) 2019-10-13T06:42:58.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:58.298+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.298327+0000) 2019-10-13T06:42:58.376 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:42:58.367+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.367881+0000) 2019-10-13T06:42:58.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:42:58.369+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.370273+0000) 2019-10-13T06:42:58.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:42:58.369+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.370382+0000) 2019-10-13T06:42:58.377 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:58.370+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.370672+0000) 2019-10-13T06:42:58.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:42:58.370+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.371117+0000) 2019-10-13T06:42:58.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:58.370+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.371308+0000) 2019-10-13T06:42:58.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:42:58.371+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.371491+0000) 2019-10-13T06:42:58.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:42:58.369+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.370366+0000) 2019-10-13T06:42:58.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:42:58.374+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:42:58.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:42:58.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.636556+0000) 2019-10-13T06:42:58.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:42:58.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:42:58.636580+0000) 2019-10-13T06:43:01.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:01.360+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:01.360579+0000) 2019-10-13T06:43:01.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:01.891+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:01.891577+0000) 2019-10-13T06:43:02.759 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:02.755+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:02.755243+0000) 2019-10-13T06:43:02.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:02.755+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:02.756276+0000) 2019-10-13T06:43:03.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:03.376+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.377263+0000) 2019-10-13T06:43:03.381 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:03.377+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.377404+0000) 2019-10-13T06:43:03.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:03.377+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.377578+0000) 2019-10-13T06:43:03.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:03.377+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.377760+0000) 2019-10-13T06:43:03.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:03.374+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.375215+0000) 2019-10-13T06:43:03.382 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:03.377+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.378294+0000) 2019-10-13T06:43:03.383 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:03.378+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.378868+0000) 2019-10-13T06:43:03.383 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:03.376+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:03.377045+0000) 2019-10-13T06:43:03.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:03.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:04.135 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:43:04.135 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:04.139 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:43:04.139 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:04.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:04.252+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:04.253321+0000) 2019-10-13T06:43:08.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:08.297+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.298518+0000) 2019-10-13T06:43:08.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:08.381+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.382456+0000) 2019-10-13T06:43:08.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:08.381+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.382793+0000) 2019-10-13T06:43:08.387 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:08.378+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.379390+0000) 2019-10-13T06:43:08.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:08.381+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.382984+0000) 2019-10-13T06:43:08.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:08.381+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.383166+0000) 2019-10-13T06:43:08.388 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:08.381+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.383065+0000) 2019-10-13T06:43:08.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:08.381+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.383354+0000) 2019-10-13T06:43:08.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:08.382+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.383100+0000) 2019-10-13T06:43:08.390 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:08.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:08.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:08.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.636665+0000) 2019-10-13T06:43:08.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:08.635+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:08.636742+0000) 2019-10-13T06:43:11.369 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:11.362+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:11.362907+0000) 2019-10-13T06:43:11.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:11.890+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:11.891731+0000) 2019-10-13T06:43:12.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:12.754+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:12.755412+0000) 2019-10-13T06:43:12.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:12.755+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:12.756427+0000) 2019-10-13T06:43:13.392 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:13.384+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.385390+0000) 2019-10-13T06:43:13.393 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:13.387+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.389022+0000) 2019-10-13T06:43:13.394 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:13.387+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.389357+0000) 2019-10-13T06:43:13.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:13.388+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.389547+0000) 2019-10-13T06:43:13.394 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:13.388+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.389624+0000) 2019-10-13T06:43:13.394 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:13.388+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.390049+0000) 2019-10-13T06:43:13.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:13.388+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.390285+0000) 2019-10-13T06:43:13.395 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:13.386+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:13.387749+0000) 2019-10-13T06:43:13.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:13.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105547 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:14.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:14.252+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:14.253496+0000) 2019-10-13T06:43:18.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:18.297+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.298715+0000) 2019-10-13T06:43:18.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:18.391+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.392510+0000) 2019-10-13T06:43:18.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:18.394+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.396044+0000) 2019-10-13T06:43:18.400 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:18.394+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.396174+0000) 2019-10-13T06:43:18.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:18.395+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.396512+0000) 2019-10-13T06:43:18.401 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:18.395+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.396573+0000) 2019-10-13T06:43:18.401 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:18.395+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.396849+0000) 2019-10-13T06:43:18.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:18.395+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.396954+0000) 2019-10-13T06:43:18.402 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:18.394+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.394798+0000) 2019-10-13T06:43:18.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:18.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:18.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:18.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.636755+0000) 2019-10-13T06:43:18.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:18.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:18.636844+0000) 2019-10-13T06:43:21.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:21.365+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:21.365289+0000) 2019-10-13T06:43:21.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:21.891+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:21.891908+0000) 2019-10-13T06:43:22.759 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:22.754+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:22.755567+0000) 2019-10-13T06:43:22.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:22.756+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:22.756591+0000) 2019-10-13T06:43:23.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:23.396+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.398705+0000) 2019-10-13T06:43:23.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:23.400+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.402188+0000) 2019-10-13T06:43:23.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:23.400+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.402337+0000) 2019-10-13T06:43:23.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:23.401+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.402392+0000) 2019-10-13T06:43:23.407 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:23.401+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.402649+0000) 2019-10-13T06:43:23.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:23.401+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.402721+0000) 2019-10-13T06:43:23.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:23.401+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.403238+0000) 2019-10-13T06:43:23.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:23.400+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:23.401795+0000) 2019-10-13T06:43:23.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:23.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:24.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:24.253+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:24.253666+0000) 2019-10-13T06:43:28.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:28.298+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.298933+0000) 2019-10-13T06:43:28.412 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:28.407+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.408405+0000) 2019-10-13T06:43:28.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:28.407+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.408684+0000) 2019-10-13T06:43:28.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:28.407+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.408854+0000) 2019-10-13T06:43:28.413 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:28.407+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.408958+0000) 2019-10-13T06:43:28.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:28.404+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.406559+0000) 2019-10-13T06:43:28.413 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:28.408+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.409574+0000) 2019-10-13T06:43:28.414 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:28.408+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.409951+0000) 2019-10-13T06:43:28.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:28.405+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.407450+0000) 2019-10-13T06:43:28.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:28.411+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105632 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:28.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:28.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.636922+0000) 2019-10-13T06:43:28.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:28.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:28.636983+0000) 2019-10-13T06:43:31.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:31.366+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:31.367784+0000) 2019-10-13T06:43:31.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:31.891+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:31.892119+0000) 2019-10-13T06:43:32.759 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:32.755+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:32.755759+0000) 2019-10-13T06:43:32.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:32.755+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:32.756773+0000) 2019-10-13T06:43:33.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:33.414+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.415237+0000) 2019-10-13T06:43:33.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:33.414+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.415368+0000) 2019-10-13T06:43:33.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:33.415+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.415531+0000) 2019-10-13T06:43:33.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:33.415+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.415615+0000) 2019-10-13T06:43:33.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:33.415+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.416214+0000) 2019-10-13T06:43:33.422 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:33.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.412768+0000) 2019-10-13T06:43:33.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:33.416+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.416742+0000) 2019-10-13T06:43:33.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:33.413+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:33.414833+0000) 2019-10-13T06:43:33.425 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:33.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:34.246 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:43:34.246 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:34.251 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:43:34.251 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:34.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:34.253+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:34.253811+0000) 2019-10-13T06:43:38.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:38.298+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.299120+0000) 2019-10-13T06:43:38.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:38.421+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.421493+0000) 2019-10-13T06:43:38.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:38.421+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.421702+0000) 2019-10-13T06:43:38.425 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:38.421+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.422050+0000) 2019-10-13T06:43:38.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:38.418+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.418977+0000) 2019-10-13T06:43:38.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:38.421+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.422347+0000) 2019-10-13T06:43:38.427 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:38.422+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.422640+0000) 2019-10-13T06:43:38.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:38.422+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.422895+0000) 2019-10-13T06:43:38.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:38.420+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.421343+0000) 2019-10-13T06:43:38.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:38.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105692 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:38.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:38.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.637107+0000) 2019-10-13T06:43:38.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:38.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:38.637150+0000) 2019-10-13T06:43:41.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:41.369+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:41.370158+0000) 2019-10-13T06:43:41.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:41.892+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:41.892313+0000) 2019-10-13T06:43:42.759 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:42.755+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:42.755961+0000) 2019-10-13T06:43:42.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:42.756+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:42.756954+0000) 2019-10-13T06:43:43.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:43.427+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.427936+0000) 2019-10-13T06:43:43.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:43.427+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.428074+0000) 2019-10-13T06:43:43.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:43.428+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.428408+0000) 2019-10-13T06:43:43.433 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:43.424+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.425355+0000) 2019-10-13T06:43:43.433 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:43.427+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.428298+0000) 2019-10-13T06:43:43.433 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:43.428+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.428965+0000) 2019-10-13T06:43:43.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:43.431+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.432161+0000) 2019-10-13T06:43:43.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:43.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105724 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:43.437 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:43.430+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:43.430868+0000) 2019-10-13T06:43:44.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:44.253+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:44.253962+0000) 2019-10-13T06:43:48.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:48.299+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.299311+0000) 2019-10-13T06:43:48.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:48.434+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.434491+0000) 2019-10-13T06:43:48.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:48.434+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.434867+0000) 2019-10-13T06:43:48.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:48.430+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.431418+0000) 2019-10-13T06:43:48.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:48.434+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.435151+0000) 2019-10-13T06:43:48.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:48.435+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.435469+0000) 2019-10-13T06:43:48.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:48.435+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.435522+0000) 2019-10-13T06:43:48.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:48.435+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.435896+0000) 2019-10-13T06:43:48.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:48.434+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.434933+0000) 2019-10-13T06:43:48.442 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:48.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105750 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:48.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:48.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.637308+0000) 2019-10-13T06:43:48.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:48.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:48.637276+0000) 2019-10-13T06:43:51.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:51.371+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:51.372448+0000) 2019-10-13T06:43:51.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:51.891+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:51.892499+0000) 2019-10-13T06:43:52.759 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:52.755+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:52.756159+0000) 2019-10-13T06:43:52.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:52.756+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:52.757134+0000) 2019-10-13T06:43:53.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:53.439+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.440741+0000) 2019-10-13T06:43:53.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:53.439+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.440844+0000) 2019-10-13T06:43:53.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:53.439+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.441032+0000) 2019-10-13T06:43:53.446 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:53.437+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.438488+0000) 2019-10-13T06:43:53.446 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:53.440+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.441405+0000) 2019-10-13T06:43:53.446 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:53.440+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.441605+0000) 2019-10-13T06:43:53.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:53.440+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.441956+0000) 2019-10-13T06:43:53.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:53.440+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:53.441095+0000) 2019-10-13T06:43:53.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:53.444+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:54.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:54.253+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:54.254142+0000) 2019-10-13T06:43:58.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:58.298+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.299521+0000) 2019-10-13T06:43:58.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:43:58.443+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.444313+0000) 2019-10-13T06:43:58.451 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:43:58.445+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.447212+0000) 2019-10-13T06:43:58.451 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:58.445+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.447387+0000) 2019-10-13T06:43:58.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:43:58.446+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.447894+0000) 2019-10-13T06:43:58.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:43:58.446+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.448159+0000) 2019-10-13T06:43:58.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:58.446+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.448328+0000) 2019-10-13T06:43:58.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:43:58.447+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.448390+0000) 2019-10-13T06:43:58.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:43:58.446+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.447585+0000) 2019-10-13T06:43:58.455 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:43:58.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:43:58.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:43:58.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.637411+0000) 2019-10-13T06:43:58.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:43:58.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:43:58.637416+0000) 2019-10-13T06:44:01.381 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:01.374+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:01.374742+0000) 2019-10-13T06:44:01.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:01.891+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:01.892699+0000) 2019-10-13T06:44:02.759 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:02.755+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:02.756332+0000) 2019-10-13T06:44:02.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:02.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:02.757296+0000) 2019-10-13T06:44:03.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:03.450+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.450940+0000) 2019-10-13T06:44:03.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:03.452+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.453730+0000) 2019-10-13T06:44:03.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:03.452+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.453734+0000) 2019-10-13T06:44:03.459 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:03.453+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.454498+0000) 2019-10-13T06:44:03.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:03.453+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.454746+0000) 2019-10-13T06:44:03.460 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:03.453+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.454939+0000) 2019-10-13T06:44:03.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:03.453+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.455293+0000) 2019-10-13T06:44:03.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:03.453+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:03.453910+0000) 2019-10-13T06:44:03.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:03.457+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:04.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:04.253+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:04.254293+0000) 2019-10-13T06:44:04.402 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:44:04.402 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:04.406 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:44:04.407 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:08.303 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:08.299+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.299700+0000) 2019-10-13T06:44:08.464 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:08.459+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.460574+0000) 2019-10-13T06:44:08.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:08.459+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.460922+0000) 2019-10-13T06:44:08.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:08.457+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.458120+0000) 2019-10-13T06:44:08.465 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:08.459+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.461109+0000) 2019-10-13T06:44:08.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:08.459+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.461167+0000) 2019-10-13T06:44:08.466 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:08.460+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.461472+0000) 2019-10-13T06:44:08.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:08.460+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.461675+0000) 2019-10-13T06:44:08.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:08.458+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.459554+0000) 2019-10-13T06:44:08.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:08.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105870 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:08.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:08.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.637533+0000) 2019-10-13T06:44:08.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:08.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:08.637563+0000) 2019-10-13T06:44:11.383 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:11.376+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:11.376985+0000) 2019-10-13T06:44:11.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:11.892+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:11.892881+0000) 2019-10-13T06:44:12.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:12.755+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:12.756515+0000) 2019-10-13T06:44:12.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:12.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:12.757468+0000) 2019-10-13T06:44:13.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:13.464+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.466231+0000) 2019-10-13T06:44:13.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:13.464+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.466048+0000) 2019-10-13T06:44:13.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:13.465+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.466514+0000) 2019-10-13T06:44:13.471 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:13.465+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.466667+0000) 2019-10-13T06:44:13.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:13.465+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.466944+0000) 2019-10-13T06:44:13.471 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:13.465+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.466846+0000) 2019-10-13T06:44:13.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:13.463+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.463909+0000) 2019-10-13T06:44:13.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:13.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105904 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:13.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:13.465+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:13.466396+0000) 2019-10-13T06:44:14.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:14.253+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:14.254464+0000) 2019-10-13T06:44:18.303 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:18.299+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.299868+0000) 2019-10-13T06:44:18.475 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:18.467+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.469206+0000) 2019-10-13T06:44:18.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:18.472+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.472709+0000) 2019-10-13T06:44:18.477 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:18.472+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.472909+0000) 2019-10-13T06:44:18.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:18.472+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.472949+0000) 2019-10-13T06:44:18.478 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:18.472+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.473194+0000) 2019-10-13T06:44:18.478 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:18.473+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.473526+0000) 2019-10-13T06:44:18.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:18.473+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.473718+0000) 2019-10-13T06:44:18.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:18.469+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.471396+0000) 2019-10-13T06:44:18.480 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:18.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:18.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:18.637+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.637652+0000) 2019-10-13T06:44:18.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:18.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:18.637729+0000) 2019-10-13T06:44:21.386 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:21.379+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:21.380080+0000) 2019-10-13T06:44:21.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:21.892+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:21.893055+0000) 2019-10-13T06:44:22.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:22.756+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:22.756694+0000) 2019-10-13T06:44:22.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:22.756+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:22.757701+0000) 2019-10-13T06:44:23.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:23.478+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.478728+0000) 2019-10-13T06:44:23.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:23.478+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.479059+0000) 2019-10-13T06:44:23.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:23.478+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.479170+0000) 2019-10-13T06:44:23.484 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:23.474+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.476134+0000) 2019-10-13T06:44:23.484 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:23.479+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.479529+0000) 2019-10-13T06:44:23.485 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:23.479+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.479760+0000) 2019-10-13T06:44:23.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:23.479+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.479900+0000) 2019-10-13T06:44:23.486 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:23.476+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:23.478493+0000) 2019-10-13T06:44:23.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:23.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:24.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:24.254+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:24.254636+0000) 2019-10-13T06:44:28.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:28.299+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.300041+0000) 2019-10-13T06:44:28.489 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:28.484+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.485036+0000) 2019-10-13T06:44:28.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:28.484+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.485314+0000) 2019-10-13T06:44:28.490 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:28.485+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.485663+0000) 2019-10-13T06:44:28.490 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:28.485+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.485925+0000) 2019-10-13T06:44:28.490 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:28.485+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.485922+0000) 2019-10-13T06:44:28.490 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:28.485+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.486218+0000) 2019-10-13T06:44:28.491 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:28.480+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.482359+0000) 2019-10-13T06:44:28.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:28.482+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.484377+0000) 2019-10-13T06:44:28.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:28.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 105990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:28.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:28.637+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.637846+0000) 2019-10-13T06:44:28.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:28.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:28.637874+0000) 2019-10-13T06:44:31.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:31.381+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:31.382378+0000) 2019-10-13T06:44:31.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:31.893+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:31.893184+0000) 2019-10-13T06:44:32.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:32.756+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:32.756809+0000) 2019-10-13T06:44:32.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:32.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:32.757876+0000) 2019-10-13T06:44:33.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:33.489+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.489872+0000) 2019-10-13T06:44:33.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:33.489+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.489785+0000) 2019-10-13T06:44:33.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:33.489+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.490247+0000) 2019-10-13T06:44:33.494 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:33.486+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.487628+0000) 2019-10-13T06:44:33.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:33.490+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.490644+0000) 2019-10-13T06:44:33.496 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:33.490+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.491024+0000) 2019-10-13T06:44:33.496 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:33.490+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.491179+0000) 2019-10-13T06:44:33.496 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:33.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:33.489359+0000) 2019-10-13T06:44:33.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:33.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106024 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:34.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:34.254+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:34.254806+0000) 2019-10-13T06:44:34.557 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:44:34.557 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:34.564 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:44:34.564 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:38.303 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:38.299+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.300222+0000) 2019-10-13T06:44:38.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:38.494+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.495692+0000) 2019-10-13T06:44:38.499 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:38.494+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.496055+0000) 2019-10-13T06:44:38.500 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:38.494+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.495950+0000) 2019-10-13T06:44:38.500 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:38.492+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.493383+0000) 2019-10-13T06:44:38.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:38.494+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.496384+0000) 2019-10-13T06:44:38.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:38.495+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.496825+0000) 2019-10-13T06:44:38.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:38.495+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.496717+0000) 2019-10-13T06:44:38.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:38.495+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.495886+0000) 2019-10-13T06:44:38.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:38.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:38.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:38.636+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.637994+0000) 2019-10-13T06:44:38.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:38.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:38.637960+0000) 2019-10-13T06:44:41.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:41.384+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:41.384666+0000) 2019-10-13T06:44:41.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:41.892+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:41.893352+0000) 2019-10-13T06:44:42.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:42.756+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:42.756987+0000) 2019-10-13T06:44:42.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:42.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:42.758062+0000) 2019-10-13T06:44:43.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:43.500+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.501452+0000) 2019-10-13T06:44:43.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:43.500+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.502041+0000) 2019-10-13T06:44:43.505 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:43.500+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.502173+0000) 2019-10-13T06:44:43.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:43.500+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.502350+0000) 2019-10-13T06:44:43.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:43.498+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.499417+0000) 2019-10-13T06:44:43.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:43.501+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.502576+0000) 2019-10-13T06:44:43.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:43.501+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.502953+0000) 2019-10-13T06:44:43.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:43.499+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:43.500591+0000) 2019-10-13T06:44:43.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:43.504+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:44.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:44.254+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:44.254978+0000) 2019-10-13T06:44:48.303 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:48.299+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.300387+0000) 2019-10-13T06:44:48.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:48.506+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.507916+0000) 2019-10-13T06:44:48.512 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:48.506+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.508105+0000) 2019-10-13T06:44:48.512 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:48.507+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.508464+0000) 2019-10-13T06:44:48.512 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:48.507+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.508610+0000) 2019-10-13T06:44:48.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:48.507+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.508751+0000) 2019-10-13T06:44:48.513 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:48.507+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.508887+0000) 2019-10-13T06:44:48.513 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:48.505+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.505863+0000) 2019-10-13T06:44:48.514 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:48.507+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.508266+0000) 2019-10-13T06:44:48.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:48.510+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:48.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:48.637+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.638131+0000) 2019-10-13T06:44:48.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:48.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:48.638133+0000) 2019-10-13T06:44:51.393 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:51.386+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:51.387007+0000) 2019-10-13T06:44:51.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:51.892+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:51.893532+0000) 2019-10-13T06:44:52.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:52.756+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:52.757172+0000) 2019-10-13T06:44:52.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:52.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:52.758243+0000) 2019-10-13T06:44:53.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:53.511+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.513274+0000) 2019-10-13T06:44:53.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:53.511+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.513347+0000) 2019-10-13T06:44:53.517 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:53.512+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.513522+0000) 2019-10-13T06:44:53.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:53.510+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.511113+0000) 2019-10-13T06:44:53.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:53.513+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.514482+0000) 2019-10-13T06:44:53.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:53.513+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.514644+0000) 2019-10-13T06:44:53.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:53.513+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.514810+0000) 2019-10-13T06:44:53.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:53.513+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:53.513921+0000) 2019-10-13T06:44:53.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:53.516+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106144 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:54.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:54.253+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:54.255167+0000) 2019-10-13T06:44:58.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:58.300+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.300584+0000) 2019-10-13T06:44:58.523 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:44:58.515+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.516295+0000) 2019-10-13T06:44:58.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:58.518+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.519653+0000) 2019-10-13T06:44:58.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:44:58.518+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.519894+0000) 2019-10-13T06:44:58.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:44:58.518+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.520271+0000) 2019-10-13T06:44:58.525 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:58.519+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.520402+0000) 2019-10-13T06:44:58.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:44:58.519+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.520495+0000) 2019-10-13T06:44:58.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:44:58.519+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.520637+0000) 2019-10-13T06:44:58.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:44:58.518+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.518950+0000) 2019-10-13T06:44:58.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:44:58.522+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:44:58.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:44:58.637+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.638274+0000) 2019-10-13T06:44:58.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:44:58.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:44:58.638294+0000) 2019-10-13T06:45:01.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:01.389+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:01.389487+0000) 2019-10-13T06:45:01.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:01.893+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:01.893700+0000) 2019-10-13T06:45:02.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:02.755+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:02.757357+0000) 2019-10-13T06:45:02.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:02.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:02.758417+0000) 2019-10-13T06:45:03.533 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:03.522+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.523140+0000) 2019-10-13T06:45:03.533 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:03.524+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.526131+0000) 2019-10-13T06:45:03.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:03.525+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.526466+0000) 2019-10-13T06:45:03.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:03.525+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.526718+0000) 2019-10-13T06:45:03.534 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:03.525+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.526844+0000) 2019-10-13T06:45:03.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:03.525+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.527220+0000) 2019-10-13T06:45:03.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:03.526+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.527687+0000) 2019-10-13T06:45:03.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:03.526+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:03.526817+0000) 2019-10-13T06:45:03.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:03.529+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:04.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:04.254+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:04.255337+0000) 2019-10-13T06:45:04.695 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:45:04.695 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:04.700 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:45:04.700 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:08.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:08.300+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.300778+0000) 2019-10-13T06:45:08.537 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:08.528+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.529623+0000) 2019-10-13T06:45:08.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:08.531+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.532299+0000) 2019-10-13T06:45:08.537 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:08.532+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.532545+0000) 2019-10-13T06:45:08.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:08.532+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.532637+0000) 2019-10-13T06:45:08.537 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:08.532+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.533104+0000) 2019-10-13T06:45:08.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:08.532+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.533319+0000) 2019-10-13T06:45:08.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:08.533+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.533682+0000) 2019-10-13T06:45:08.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:08.531+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.532477+0000) 2019-10-13T06:45:08.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:08.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:08.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:08.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.638424+0000) 2019-10-13T06:45:08.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:08.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:08.638442+0000) 2019-10-13T06:45:11.398 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:11.390+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:11.391928+0000) 2019-10-13T06:45:11.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:11.893+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:11.893864+0000) 2019-10-13T06:45:12.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:12.757+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:12.757561+0000) 2019-10-13T06:45:12.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:12.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:12.758587+0000) 2019-10-13T06:45:13.546 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:13.534+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.535868+0000) 2019-10-13T06:45:13.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:13.537+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.538743+0000) 2019-10-13T06:45:13.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:13.538+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.538447+0000) 2019-10-13T06:45:13.546 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:13.538+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.538687+0000) 2019-10-13T06:45:13.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:13.538+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.538864+0000) 2019-10-13T06:45:13.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:13.538+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.539138+0000) 2019-10-13T06:45:13.547 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:13.538+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.539305+0000) 2019-10-13T06:45:13.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:13.539+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:13.539688+0000) 2019-10-13T06:45:13.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:13.541+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106264 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:14.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:14.255+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:14.255512+0000) 2019-10-13T06:45:18.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:18.300+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.300954+0000) 2019-10-13T06:45:18.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:18.543+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.544022+0000) 2019-10-13T06:45:18.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:18.544+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.544403+0000) 2019-10-13T06:45:18.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:18.544+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.544676+0000) 2019-10-13T06:45:18.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:18.544+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.544778+0000) 2019-10-13T06:45:18.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:18.544+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.545172+0000) 2019-10-13T06:45:18.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:18.545+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.545457+0000) 2019-10-13T06:45:18.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:18.540+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.541786+0000) 2019-10-13T06:45:18.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:18.542+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.544595+0000) 2019-10-13T06:45:18.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:18.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:18.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:18.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.638575+0000) 2019-10-13T06:45:18.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:18.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:18.638585+0000) 2019-10-13T06:45:21.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:21.393+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:21.394253+0000) 2019-10-13T06:45:21.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:21.893+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:21.894011+0000) 2019-10-13T06:45:22.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:22.757+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:22.757738+0000) 2019-10-13T06:45:22.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:22.757+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:22.758767+0000) 2019-10-13T06:45:23.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:23.549+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.551076+0000) 2019-10-13T06:45:23.555 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:23.546+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.547990+0000) 2019-10-13T06:45:23.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:23.549+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.551319+0000) 2019-10-13T06:45:23.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:23.550+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.551680+0000) 2019-10-13T06:45:23.556 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:23.550+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.551839+0000) 2019-10-13T06:45:23.556 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:23.550+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.551929+0000) 2019-10-13T06:45:23.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:23.550+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.552258+0000) 2019-10-13T06:45:23.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:23.549+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:23.551155+0000) 2019-10-13T06:45:23.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:23.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106323 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:24.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:24.254+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:24.255717+0000) 2019-10-13T06:45:28.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:28.300+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.301123+0000) 2019-10-13T06:45:28.560 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:28.555+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.557222+0000) 2019-10-13T06:45:28.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:28.555+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.557383+0000) 2019-10-13T06:45:28.562 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:28.553+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.554645+0000) 2019-10-13T06:45:28.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:28.556+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.557705+0000) 2019-10-13T06:45:28.562 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:28.556+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.557903+0000) 2019-10-13T06:45:28.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:28.557+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.558416+0000) 2019-10-13T06:45:28.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:28.557+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.558545+0000) 2019-10-13T06:45:28.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:28.556+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.557085+0000) 2019-10-13T06:45:28.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:28.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:28.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:28.637+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.638699+0000) 2019-10-13T06:45:28.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:28.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:28.638734+0000) 2019-10-13T06:45:31.403 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:31.395+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:31.396473+0000) 2019-10-13T06:45:31.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:31.893+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:31.894155+0000) 2019-10-13T06:45:32.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:32.757+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:32.757908+0000) 2019-10-13T06:45:32.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:32.758+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:32.758943+0000) 2019-10-13T06:45:33.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:33.561+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.562669+0000) 2019-10-13T06:45:33.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:33.561+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.562701+0000) 2019-10-13T06:45:33.567 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:33.561+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.563018+0000) 2019-10-13T06:45:33.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:33.561+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.563007+0000) 2019-10-13T06:45:33.568 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:33.559+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.559995+0000) 2019-10-13T06:45:33.568 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:33.560+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.561316+0000) 2019-10-13T06:45:33.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:33.562+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.563521+0000) 2019-10-13T06:45:33.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:33.562+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:33.563429+0000) 2019-10-13T06:45:33.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:33.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:34.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:34.255+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:34.255898+0000) 2019-10-13T06:45:34.856 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:45:34.857 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:34.861 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:45:34.861 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:38.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:38.300+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.301288+0000) 2019-10-13T06:45:38.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:38.566+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.568019+0000) 2019-10-13T06:45:38.572 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:38.566+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.568070+0000) 2019-10-13T06:45:38.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:38.566+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.568199+0000) 2019-10-13T06:45:38.573 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:38.565+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.566267+0000) 2019-10-13T06:45:38.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:38.567+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.568787+0000) 2019-10-13T06:45:38.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:38.567+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.569171+0000) 2019-10-13T06:45:38.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:38.568+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.569664+0000) 2019-10-13T06:45:38.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:38.567+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.567832+0000) 2019-10-13T06:45:38.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:38.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:38.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:38.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.638833+0000) 2019-10-13T06:45:38.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:38.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:38.638872+0000) 2019-10-13T06:45:41.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:41.398+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:41.398762+0000) 2019-10-13T06:45:41.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:41.893+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:41.894309+0000) 2019-10-13T06:45:42.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:42.757+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:42.758082+0000) 2019-10-13T06:45:42.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:42.758+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:42.759125+0000) 2019-10-13T06:45:43.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:43.572+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.574328+0000) 2019-10-13T06:45:43.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:43.573+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.574598+0000) 2019-10-13T06:45:43.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:43.570+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.571354+0000) 2019-10-13T06:45:43.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:43.573+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.575054+0000) 2019-10-13T06:45:43.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:43.573+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.575294+0000) 2019-10-13T06:45:43.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:43.574+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.575712+0000) 2019-10-13T06:45:43.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:43.574+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.575993+0000) 2019-10-13T06:45:43.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:43.574+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:43.575189+0000) 2019-10-13T06:45:43.582 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:43.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106443 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:44.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:44.255+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:44.256034+0000) 2019-10-13T06:45:48.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:48.300+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.301461+0000) 2019-10-13T06:45:48.586 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:48.579+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.581340+0000) 2019-10-13T06:45:48.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:48.580+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.581570+0000) 2019-10-13T06:45:48.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:48.580+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.581804+0000) 2019-10-13T06:45:48.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:48.580+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.581849+0000) 2019-10-13T06:45:48.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:48.580+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.582190+0000) 2019-10-13T06:45:48.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:48.581+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.582614+0000) 2019-10-13T06:45:48.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:48.577+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.578051+0000) 2019-10-13T06:45:48.587 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:48.580+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.581258+0000) 2019-10-13T06:45:48.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:48.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106472 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:48.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:48.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.639026+0000) 2019-10-13T06:45:48.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:48.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:48.639026+0000) 2019-10-13T06:45:51.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:51.400+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:51.401023+0000) 2019-10-13T06:45:51.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:51.894+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:51.894496+0000) 2019-10-13T06:45:52.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:52.757+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:52.758240+0000) 2019-10-13T06:45:52.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:52.759+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:52.759329+0000) 2019-10-13T06:45:53.590 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:53.586+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.586636+0000) 2019-10-13T06:45:53.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:53.583+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.583956+0000) 2019-10-13T06:45:53.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:53.586+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.587234+0000) 2019-10-13T06:45:53.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:53.587+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.587419+0000) 2019-10-13T06:45:53.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:53.587+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.587540+0000) 2019-10-13T06:45:53.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:53.587+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.587615+0000) 2019-10-13T06:45:53.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:53.587+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.587828+0000) 2019-10-13T06:45:53.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:53.585+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:53.586274+0000) 2019-10-13T06:45:53.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:53.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106504 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:54.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:54.255+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:54.256204+0000) 2019-10-13T06:45:58.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:58.301+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.301650+0000) 2019-10-13T06:45:58.596 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:45:58.591+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.592129+0000) 2019-10-13T06:45:58.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:58.591+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.592318+0000) 2019-10-13T06:45:58.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:45:58.588+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.589541+0000) 2019-10-13T06:45:58.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:58.592+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.592937+0000) 2019-10-13T06:45:58.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:45:58.592+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.593304+0000) 2019-10-13T06:45:58.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:45:58.593+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.593567+0000) 2019-10-13T06:45:58.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:45:58.593+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.593590+0000) 2019-10-13T06:45:58.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:45:58.591+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.591753+0000) 2019-10-13T06:45:58.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:45:58.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:45:58.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:45:58.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.639160+0000) 2019-10-13T06:45:58.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:45:58.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:45:58.639161+0000) 2019-10-13T06:46:01.410 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:01.403+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:01.403325+0000) 2019-10-13T06:46:01.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:01.894+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:01.894631+0000) 2019-10-13T06:46:02.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:02.758+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:02.758384+0000) 2019-10-13T06:46:02.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:02.759+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:02.759507+0000) 2019-10-13T06:46:03.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:03.598+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.598641+0000) 2019-10-13T06:46:03.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:03.598+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.598992+0000) 2019-10-13T06:46:03.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:03.595+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.596385+0000) 2019-10-13T06:46:03.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:03.598+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.599185+0000) 2019-10-13T06:46:03.603 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:03.598+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.599366+0000) 2019-10-13T06:46:03.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:03.599+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.599727+0000) 2019-10-13T06:46:03.604 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:03.599+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.600036+0000) 2019-10-13T06:46:03.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:03.597+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:03.598100+0000) 2019-10-13T06:46:03.606 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:03.602+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106563 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:04.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:04.256+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:04.256373+0000) 2019-10-13T06:46:05.011 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:46:05.011 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:05.015 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:46:05.015 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:08.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:08.301+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.301840+0000) 2019-10-13T06:46:08.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:08.604+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.604408+0000) 2019-10-13T06:46:08.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:08.604+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.604658+0000) 2019-10-13T06:46:08.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:08.599+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.601668+0000) 2019-10-13T06:46:08.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:08.604+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.605057+0000) 2019-10-13T06:46:08.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:08.604+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.605300+0000) 2019-10-13T06:46:08.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:08.605+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.605719+0000) 2019-10-13T06:46:08.610 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:08.605+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.605933+0000) 2019-10-13T06:46:08.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:08.602+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.604509+0000) 2019-10-13T06:46:08.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:08.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106592 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:08.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:08.639+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.639320+0000) 2019-10-13T06:46:08.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:08.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:08.639321+0000) 2019-10-13T06:46:11.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:11.404+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:11.405828+0000) 2019-10-13T06:46:11.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:11.893+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:11.894777+0000) 2019-10-13T06:46:12.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:12.757+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:12.758603+0000) 2019-10-13T06:46:12.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:12.758+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:12.759749+0000) 2019-10-13T06:46:13.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:13.609+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.610868+0000) 2019-10-13T06:46:13.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:13.609+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.611081+0000) 2019-10-13T06:46:13.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:13.606+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.608723+0000) 2019-10-13T06:46:13.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:13.609+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.611291+0000) 2019-10-13T06:46:13.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:13.610+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.611715+0000) 2019-10-13T06:46:13.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:13.610+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.612012+0000) 2019-10-13T06:46:13.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:13.610+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.612146+0000) 2019-10-13T06:46:13.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:13.608+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:13.610313+0000) 2019-10-13T06:46:13.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:13.614+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106624 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:14.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:14.255+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:14.256531+0000) 2019-10-13T06:46:18.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:18.301+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.302033+0000) 2019-10-13T06:46:18.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:18.613+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.614258+0000) 2019-10-13T06:46:18.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:18.616+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.617632+0000) 2019-10-13T06:46:18.622 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:18.616+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.617757+0000) 2019-10-13T06:46:18.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:18.616+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.618098+0000) 2019-10-13T06:46:18.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:18.616+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.618339+0000) 2019-10-13T06:46:18.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:18.617+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.618484+0000) 2019-10-13T06:46:18.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:18.617+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.618753+0000) 2019-10-13T06:46:18.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:18.616+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.616929+0000) 2019-10-13T06:46:18.625 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:18.620+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:18.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:18.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.639463+0000) 2019-10-13T06:46:18.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:18.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:18.639463+0000) 2019-10-13T06:46:21.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:21.407+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:21.408254+0000) 2019-10-13T06:46:21.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:21.894+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:21.894925+0000) 2019-10-13T06:46:22.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:22.758+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:22.758808+0000) 2019-10-13T06:46:22.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:22.759+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:22.759896+0000) 2019-10-13T06:46:23.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:23.620+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.620762+0000) 2019-10-13T06:46:23.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:23.622+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.624073+0000) 2019-10-13T06:46:23.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:23.622+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.624229+0000) 2019-10-13T06:46:23.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:23.623+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.624564+0000) 2019-10-13T06:46:23.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:23.623+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.624625+0000) 2019-10-13T06:46:23.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:23.623+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.624785+0000) 2019-10-13T06:46:23.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:23.623+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.625143+0000) 2019-10-13T06:46:23.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:23.622+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:23.622967+0000) 2019-10-13T06:46:23.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:23.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:24.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:24.255+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:24.256711+0000) 2019-10-13T06:46:28.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:28.301+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.302252+0000) 2019-10-13T06:46:28.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:28.626+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.626745+0000) 2019-10-13T06:46:28.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:28.628+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.630360+0000) 2019-10-13T06:46:28.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:28.629+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.630697+0000) 2019-10-13T06:46:28.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:28.629+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.630847+0000) 2019-10-13T06:46:28.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:28.629+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.631051+0000) 2019-10-13T06:46:28.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:28.629+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.631171+0000) 2019-10-13T06:46:28.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:28.629+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.631249+0000) 2019-10-13T06:46:28.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:28.629+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.629878+0000) 2019-10-13T06:46:28.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:28.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106712 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:28.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:28.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.639583+0000) 2019-10-13T06:46:28.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:28.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:28.639601+0000) 2019-10-13T06:46:31.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:31.410+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:31.410563+0000) 2019-10-13T06:46:31.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:31.894+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:31.895089+0000) 2019-10-13T06:46:32.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:32.757+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:32.759011+0000) 2019-10-13T06:46:32.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:32.759+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:32.760059+0000) 2019-10-13T06:46:33.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:33.635+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.636511+0000) 2019-10-13T06:46:33.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:33.635+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.636765+0000) 2019-10-13T06:46:33.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:33.635+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.636408+0000) 2019-10-13T06:46:33.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:33.632+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.633689+0000) 2019-10-13T06:46:33.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:33.635+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.637077+0000) 2019-10-13T06:46:33.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:33.635+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.637275+0000) 2019-10-13T06:46:33.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:33.636+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.637464+0000) 2019-10-13T06:46:33.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:33.636+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:33.637105+0000) 2019-10-13T06:46:33.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:33.639+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:34.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:34.256+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:34.256913+0000) 2019-10-13T06:46:35.156 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:46:35.157 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:35.161 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:46:35.161 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:38.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:38.302+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.302435+0000) 2019-10-13T06:46:38.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:38.639+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.639738+0000) 2019-10-13T06:46:38.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:38.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.639729+0000) 2019-10-13T06:46:38.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:38.641+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.642318+0000) 2019-10-13T06:46:38.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:38.642+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.642480+0000) 2019-10-13T06:46:38.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:38.639+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.639853+0000) 2019-10-13T06:46:38.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:38.642+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.643216+0000) 2019-10-13T06:46:38.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:38.642+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.643382+0000) 2019-10-13T06:46:38.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:38.643+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.643551+0000) 2019-10-13T06:46:38.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:38.643+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.643705+0000) 2019-10-13T06:46:38.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:38.641+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:38.642019+0000) 2019-10-13T06:46:38.649 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:38.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:41.419 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:41.412+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:41.413036+0000) 2019-10-13T06:46:41.898 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:41.894+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:41.895256+0000) 2019-10-13T06:46:42.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:42.758+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:42.759213+0000) 2019-10-13T06:46:42.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:42.759+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:42.760224+0000) 2019-10-13T06:46:43.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:43.647+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.648029+0000) 2019-10-13T06:46:43.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:43.648+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.648406+0000) 2019-10-13T06:46:43.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:43.648+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.648511+0000) 2019-10-13T06:46:43.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:43.648+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.648716+0000) 2019-10-13T06:46:43.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:43.648+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.648943+0000) 2019-10-13T06:46:43.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:43.648+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.649109+0000) 2019-10-13T06:46:43.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:43.645+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.646347+0000) 2019-10-13T06:46:43.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:43.647+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:43.648642+0000) 2019-10-13T06:46:43.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:43.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106803 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:44.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:44.256+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:44.257062+0000) 2019-10-13T06:46:48.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:48.302+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.302645+0000) 2019-10-13T06:46:48.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:48.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.639853+0000) 2019-10-13T06:46:48.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:48.639+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.639875+0000) 2019-10-13T06:46:48.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:48.654+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.654405+0000) 2019-10-13T06:46:48.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:48.654+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.654655+0000) 2019-10-13T06:46:48.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:48.654+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.655039+0000) 2019-10-13T06:46:48.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:48.654+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.655189+0000) 2019-10-13T06:46:48.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:48.651+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.652242+0000) 2019-10-13T06:46:48.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:48.655+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.655453+0000) 2019-10-13T06:46:48.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:48.655+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.655731+0000) 2019-10-13T06:46:48.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:48.653+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:48.653866+0000) 2019-10-13T06:46:48.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:48.658+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106832 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:51.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:51.415+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:51.415370+0000) 2019-10-13T06:46:51.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:51.895+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:51.895438+0000) 2019-10-13T06:46:52.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:52.759+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:52.759424+0000) 2019-10-13T06:46:52.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:52.760+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:52.760392+0000) 2019-10-13T06:46:53.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:53.660+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.660489+0000) 2019-10-13T06:46:53.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:53.660+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.660664+0000) 2019-10-13T06:46:53.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:53.657+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.658362+0000) 2019-10-13T06:46:53.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:53.660+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.661120+0000) 2019-10-13T06:46:53.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:53.660+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.661065+0000) 2019-10-13T06:46:53.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:53.660+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.661272+0000) 2019-10-13T06:46:53.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:53.661+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.661894+0000) 2019-10-13T06:46:53.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:53.658+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:53.659445+0000) 2019-10-13T06:46:53.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:53.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:46:54.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:54.257+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:54.257218+0000) 2019-10-13T06:46:58.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:58.301+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.302834+0000) 2019-10-13T06:46:58.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:58.639+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.640054+0000) 2019-10-13T06:46:58.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:58.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.640038+0000) 2019-10-13T06:46:58.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:46:58.666+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.667429+0000) 2019-10-13T06:46:58.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:46:58.662+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.664654+0000) 2019-10-13T06:46:58.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:46:58.666+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.667642+0000) 2019-10-13T06:46:58.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:46:58.666+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.667879+0000) 2019-10-13T06:46:58.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:46:58.666+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.667999+0000) 2019-10-13T06:46:58.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:46:58.666+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.668202+0000) 2019-10-13T06:46:58.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:46:58.667+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.668592+0000) 2019-10-13T06:46:58.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:46:58.666+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:46:58.667876+0000) 2019-10-13T06:46:58.675 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:46:58.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:01.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:01.416+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:01.417874+0000) 2019-10-13T06:47:01.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:01.894+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:01.895626+0000) 2019-10-13T06:47:02.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:02.758+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:02.759624+0000) 2019-10-13T06:47:02.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:02.759+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:02.760548+0000) 2019-10-13T06:47:03.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:03.672+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.673671+0000) 2019-10-13T06:47:03.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:03.672+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.673932+0000) 2019-10-13T06:47:03.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:03.669+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.670999+0000) 2019-10-13T06:47:03.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:03.672+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.674234+0000) 2019-10-13T06:47:03.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:03.673+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.674458+0000) 2019-10-13T06:47:03.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:03.673+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.674734+0000) 2019-10-13T06:47:03.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:03.673+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.675022+0000) 2019-10-13T06:47:03.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:03.671+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:03.673557+0000) 2019-10-13T06:47:03.681 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:03.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:04.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:04.255+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:04.257350+0000) 2019-10-13T06:47:05.287 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:47:05.287 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:05.291 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:47:05.291 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:08.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:08.302+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.303006+0000) 2019-10-13T06:47:08.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:08.638+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.640261+0000) 2019-10-13T06:47:08.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:08.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.640249+0000) 2019-10-13T06:47:08.683 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:08.678+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.679842+0000) 2019-10-13T06:47:08.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:08.679+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.680457+0000) 2019-10-13T06:47:08.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:08.678+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.680327+0000) 2019-10-13T06:47:08.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:08.679+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.680757+0000) 2019-10-13T06:47:08.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:08.676+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.677854+0000) 2019-10-13T06:47:08.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:08.679+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.681122+0000) 2019-10-13T06:47:08.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:08.679+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.681233+0000) 2019-10-13T06:47:08.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:08.678+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:08.679769+0000) 2019-10-13T06:47:08.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:08.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106953 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:11.426 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:11.419+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:11.420354+0000) 2019-10-13T06:47:11.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:11.895+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:11.895836+0000) 2019-10-13T06:47:12.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:12.758+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:12.759800+0000) 2019-10-13T06:47:12.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:12.760+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:12.760745+0000) 2019-10-13T06:47:13.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:13.682+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.683043+0000) 2019-10-13T06:47:13.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:13.684+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.686288+0000) 2019-10-13T06:47:13.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:13.685+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.686670+0000) 2019-10-13T06:47:13.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:13.685+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.686764+0000) 2019-10-13T06:47:13.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:13.685+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.687157+0000) 2019-10-13T06:47:13.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:13.685+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.687222+0000) 2019-10-13T06:47:13.691 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:13.685+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.687351+0000) 2019-10-13T06:47:13.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:13.685+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:13.685920+0000) 2019-10-13T06:47:13.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:13.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 106984 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:14.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:14.256+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:14.257519+0000) 2019-10-13T06:47:18.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:18.302+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.303137+0000) 2019-10-13T06:47:18.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:18.639+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.640429+0000) 2019-10-13T06:47:18.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:18.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.640430+0000) 2019-10-13T06:47:18.696 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:18.690+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.692300+0000) 2019-10-13T06:47:18.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:18.691+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.692422+0000) 2019-10-13T06:47:18.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:18.691+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.692734+0000) 2019-10-13T06:47:18.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:18.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.692903+0000) 2019-10-13T06:47:18.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:18.691+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.692984+0000) 2019-10-13T06:47:18.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:18.691+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.693176+0000) 2019-10-13T06:47:18.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:18.688+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.689560+0000) 2019-10-13T06:47:18.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:18.691+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:18.692703+0000) 2019-10-13T06:47:18.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:18.694+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107012 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:21.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:21.421+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:21.422405+0000) 2019-10-13T06:47:21.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:21.895+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:21.896013+0000) 2019-10-13T06:47:22.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:22.759+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:22.759945+0000) 2019-10-13T06:47:22.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:22.760+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:22.760888+0000) 2019-10-13T06:47:23.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:23.693+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.694592+0000) 2019-10-13T06:47:23.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:23.697+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.697834+0000) 2019-10-13T06:47:23.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:23.697+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.698301+0000) 2019-10-13T06:47:23.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:23.698+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.698484+0000) 2019-10-13T06:47:23.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:23.698+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.698620+0000) 2019-10-13T06:47:23.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:23.698+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.698874+0000) 2019-10-13T06:47:23.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:23.698+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.698949+0000) 2019-10-13T06:47:23.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:23.696+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:23.697012+0000) 2019-10-13T06:47:23.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:23.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:24.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:24.257+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:24.257693+0000) 2019-10-13T06:47:28.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:28.303+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.303337+0000) 2019-10-13T06:47:28.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:28.640+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.640558+0000) 2019-10-13T06:47:28.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:28.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.640535+0000) 2019-10-13T06:47:28.707 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:28.703+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.703598+0000) 2019-10-13T06:47:28.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:28.703+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.703952+0000) 2019-10-13T06:47:28.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:28.703+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.704189+0000) 2019-10-13T06:47:28.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:28.703+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.704099+0000) 2019-10-13T06:47:28.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:28.700+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.701581+0000) 2019-10-13T06:47:28.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:28.704+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.704857+0000) 2019-10-13T06:47:28.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:28.704+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.705115+0000) 2019-10-13T06:47:28.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:28.702+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:28.703448+0000) 2019-10-13T06:47:28.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:28.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:31.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:31.424+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:31.424722+0000) 2019-10-13T06:47:31.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:31.895+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:31.896193+0000) 2019-10-13T06:47:32.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:32.759+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:32.760113+0000) 2019-10-13T06:47:32.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:32.760+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:32.761074+0000) 2019-10-13T06:47:33.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:33.709+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.709566+0000) 2019-10-13T06:47:33.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:33.709+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.709842+0000) 2019-10-13T06:47:33.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:33.709+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.710039+0000) 2019-10-13T06:47:33.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:33.709+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.710303+0000) 2019-10-13T06:47:33.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:33.706+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.707163+0000) 2019-10-13T06:47:33.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:33.710+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.710561+0000) 2019-10-13T06:47:33.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:33.710+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.710966+0000) 2019-10-13T06:47:33.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:33.708+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:33.709411+0000) 2019-10-13T06:47:33.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:33.713+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107105 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:34.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:34.257+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:34.257843+0000) 2019-10-13T06:47:35.449 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:47:35.449 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:35.454 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:47:35.454 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:38.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:38.303+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.303550+0000) 2019-10-13T06:47:38.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:38.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.640680+0000) 2019-10-13T06:47:38.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:38.640+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.640733+0000) 2019-10-13T06:47:38.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:38.715+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.716163+0000) 2019-10-13T06:47:38.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:38.715+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.716233+0000) 2019-10-13T06:47:38.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:38.716+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.716393+0000) 2019-10-13T06:47:38.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:38.713+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.713876+0000) 2019-10-13T06:47:38.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:38.716+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.717084+0000) 2019-10-13T06:47:38.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:38.716+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.717213+0000) 2019-10-13T06:47:38.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:38.717+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.717524+0000) 2019-10-13T06:47:38.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:38.715+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:38.716492+0000) 2019-10-13T06:47:38.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:38.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107132 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:41.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:41.426+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:41.427208+0000) 2019-10-13T06:47:41.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:41.895+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:41.896350+0000) 2019-10-13T06:47:42.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:42.759+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:42.760291+0000) 2019-10-13T06:47:42.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:42.761+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:42.761281+0000) 2019-10-13T06:47:43.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:43.721+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.722439+0000) 2019-10-13T06:47:43.726 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:43.721+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.722747+0000) 2019-10-13T06:47:43.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:43.719+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.720046+0000) 2019-10-13T06:47:43.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:43.721+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.723279+0000) 2019-10-13T06:47:43.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:43.722+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.723548+0000) 2019-10-13T06:47:43.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:43.722+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.723872+0000) 2019-10-13T06:47:43.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:43.722+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.724235+0000) 2019-10-13T06:47:43.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:43.721+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:43.721986+0000) 2019-10-13T06:47:43.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:43.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:44.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:44.256+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:44.258040+0000) 2019-10-13T06:47:48.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:48.302+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.303750+0000) 2019-10-13T06:47:48.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:48.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.640867+0000) 2019-10-13T06:47:48.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:48.639+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.640893+0000) 2019-10-13T06:47:48.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:48.725+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.726283+0000) 2019-10-13T06:47:48.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:48.727+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.728647+0000) 2019-10-13T06:47:48.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:48.727+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.729011+0000) 2019-10-13T06:47:48.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:48.727+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.729207+0000) 2019-10-13T06:47:48.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:48.728+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.729770+0000) 2019-10-13T06:47:48.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:48.728+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.729923+0000) 2019-10-13T06:47:48.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:48.728+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.730195+0000) 2019-10-13T06:47:48.737 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:48.728+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:48.729214+0000) 2019-10-13T06:47:48.737 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:48.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107193 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:51.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:51.428+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:51.429521+0000) 2019-10-13T06:47:51.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:51.895+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:51.896530+0000) 2019-10-13T06:47:52.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:52.759+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:52.760463+0000) 2019-10-13T06:47:52.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:52.760+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:52.761467+0000) 2019-10-13T06:47:53.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:53.733+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.734718+0000) 2019-10-13T06:47:53.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:53.733+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.734891+0000) 2019-10-13T06:47:53.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:53.733+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.735159+0000) 2019-10-13T06:47:53.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:53.730+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.732426+0000) 2019-10-13T06:47:53.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:53.734+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.735663+0000) 2019-10-13T06:47:53.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:53.734+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.735752+0000) 2019-10-13T06:47:53.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:53.734+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.735975+0000) 2019-10-13T06:47:53.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:53.732+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:53.734240+0000) 2019-10-13T06:47:53.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:53.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:47:54.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:54.256+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:54.258242+0000) 2019-10-13T06:47:58.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:58.303+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.303892+0000) 2019-10-13T06:47:58.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:58.640+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.641054+0000) 2019-10-13T06:47:58.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:58.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.641055+0000) 2019-10-13T06:47:58.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:47:58.736+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.738069+0000) 2019-10-13T06:47:58.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:47:58.740+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.741440+0000) 2019-10-13T06:47:58.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:47:58.740+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.741647+0000) 2019-10-13T06:47:58.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:47:58.740+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.742044+0000) 2019-10-13T06:47:58.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:47:58.740+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.742363+0000) 2019-10-13T06:47:58.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:47:58.741+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.742707+0000) 2019-10-13T06:47:58.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:47:58.741+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.742874+0000) 2019-10-13T06:47:58.747 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:47:58.738+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:47:58.740618+0000) 2019-10-13T06:47:58.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:47:58.744+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107252 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:01.438 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:01.430+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:01.431781+0000) 2019-10-13T06:48:01.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:01.896+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:01.896714+0000) 2019-10-13T06:48:02.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:02.760+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:02.760635+0000) 2019-10-13T06:48:02.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:02.760+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:02.761671+0000) 2019-10-13T06:48:03.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:03.746+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.747617+0000) 2019-10-13T06:48:03.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:03.746+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.747990+0000) 2019-10-13T06:48:03.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:03.746+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.748128+0000) 2019-10-13T06:48:03.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:03.743+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.744737+0000) 2019-10-13T06:48:03.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:03.747+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.748643+0000) 2019-10-13T06:48:03.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:03.747+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.748885+0000) 2019-10-13T06:48:03.753 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:03.747+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.749091+0000) 2019-10-13T06:48:03.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:03.746+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:03.748314+0000) 2019-10-13T06:48:03.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:03.750+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:04.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:04.257+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:04.258427+0000) 2019-10-13T06:48:05.646 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:48:05.646 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:05.650 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:48:05.650 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:08.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:08.303+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.304061+0000) 2019-10-13T06:48:08.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:08.640+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.641261+0000) 2019-10-13T06:48:08.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:08.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.641261+0000) 2019-10-13T06:48:08.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:08.750+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.750841+0000) 2019-10-13T06:48:08.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:08.752+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.754073+0000) 2019-10-13T06:48:08.758 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:08.753+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.754465+0000) 2019-10-13T06:48:08.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:08.753+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.754701+0000) 2019-10-13T06:48:08.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:08.753+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.754861+0000) 2019-10-13T06:48:08.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:08.753+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.755151+0000) 2019-10-13T06:48:08.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:08.752+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.753281+0000) 2019-10-13T06:48:08.759 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:08.754+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:08.755474+0000) 2019-10-13T06:48:08.761 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:08.756+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:11.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:11.433+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:11.434236+0000) 2019-10-13T06:48:11.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:11.896+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:11.896874+0000) 2019-10-13T06:48:12.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:12.760+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:12.760858+0000) 2019-10-13T06:48:12.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:12.761+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:12.761868+0000) 2019-10-13T06:48:13.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:13.755+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.756701+0000) 2019-10-13T06:48:13.764 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:13.759+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.759922+0000) 2019-10-13T06:48:13.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:13.759+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.760127+0000) 2019-10-13T06:48:13.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:13.759+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.760375+0000) 2019-10-13T06:48:13.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:13.760+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.760609+0000) 2019-10-13T06:48:13.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:13.760+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.760827+0000) 2019-10-13T06:48:13.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:13.760+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.760975+0000) 2019-10-13T06:48:13.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:13.758+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:13.759211+0000) 2019-10-13T06:48:13.767 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:13.763+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:14.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:14.258+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:14.258589+0000) 2019-10-13T06:48:18.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:18.304+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.304256+0000) 2019-10-13T06:48:18.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:18.641+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.641416+0000) 2019-10-13T06:48:18.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:18.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.641460+0000) 2019-10-13T06:48:18.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:18.762+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.762883+0000) 2019-10-13T06:48:18.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:18.765+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.766260+0000) 2019-10-13T06:48:18.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:18.766+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.766691+0000) 2019-10-13T06:48:18.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:18.766+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.767019+0000) 2019-10-13T06:48:18.771 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:18.766+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.767155+0000) 2019-10-13T06:48:18.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:18.766+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.767340+0000) 2019-10-13T06:48:18.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:18.767+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.767495+0000) 2019-10-13T06:48:18.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:18.764+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:18.765620+0000) 2019-10-13T06:48:18.773 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:18.770+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:21.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:21.436+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:21.436563+0000) 2019-10-13T06:48:21.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:21.896+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:21.897047+0000) 2019-10-13T06:48:22.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:22.760+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:22.760982+0000) 2019-10-13T06:48:22.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:22.761+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:22.761991+0000) 2019-10-13T06:48:23.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:23.771+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.772303+0000) 2019-10-13T06:48:23.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:23.772+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.772621+0000) 2019-10-13T06:48:23.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:23.769+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.769788+0000) 2019-10-13T06:48:23.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:23.772+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.772999+0000) 2019-10-13T06:48:23.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:23.772+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.773130+0000) 2019-10-13T06:48:23.777 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:23.773+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.773437+0000) 2019-10-13T06:48:23.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:23.773+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.773566+0000) 2019-10-13T06:48:23.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:23.770+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:23.771577+0000) 2019-10-13T06:48:23.780 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:23.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:24.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:24.258+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:24.258721+0000) 2019-10-13T06:48:28.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:28.303+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.304389+0000) 2019-10-13T06:48:28.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:28.640+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.641593+0000) 2019-10-13T06:48:28.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:28.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.641579+0000) 2019-10-13T06:48:28.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:28.775+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.776208+0000) 2019-10-13T06:48:28.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:28.777+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.778498+0000) 2019-10-13T06:48:28.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:28.777+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.778840+0000) 2019-10-13T06:48:28.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:28.777+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.779021+0000) 2019-10-13T06:48:28.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:28.777+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.779373+0000) 2019-10-13T06:48:28.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:28.778+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.779630+0000) 2019-10-13T06:48:28.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:28.778+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.779970+0000) 2019-10-13T06:48:28.787 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:28.778+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:28.779854+0000) 2019-10-13T06:48:28.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:28.781+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107433 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:31.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:31.438+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:31.439080+0000) 2019-10-13T06:48:31.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:31.896+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:31.897266+0000) 2019-10-13T06:48:32.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:32.760+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:32.761120+0000) 2019-10-13T06:48:32.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:32.761+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:32.762155+0000) 2019-10-13T06:48:33.788 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:33.783+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.784863+0000) 2019-10-13T06:48:33.789 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:33.783+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.784998+0000) 2019-10-13T06:48:33.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:33.783+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.785118+0000) 2019-10-13T06:48:33.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:33.783+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.785288+0000) 2019-10-13T06:48:33.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:33.784+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.785577+0000) 2019-10-13T06:48:33.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:33.781+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.782369+0000) 2019-10-13T06:48:33.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:33.784+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.786210+0000) 2019-10-13T06:48:33.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:33.784+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:33.785375+0000) 2019-10-13T06:48:33.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:33.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:34.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:34.257+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:34.258913+0000) 2019-10-13T06:48:35.731 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:48:35.731 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:35.736 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:48:35.736 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:38.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:38.303+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.304580+0000) 2019-10-13T06:48:38.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:38.640+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.641764+0000) 2019-10-13T06:48:38.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:38.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.641777+0000) 2019-10-13T06:48:38.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:38.788+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.790284+0000) 2019-10-13T06:48:38.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:38.789+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.790478+0000) 2019-10-13T06:48:38.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:38.789+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.790806+0000) 2019-10-13T06:48:38.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:38.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.788686+0000) 2019-10-13T06:48:38.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:38.789+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.791078+0000) 2019-10-13T06:48:38.796 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:38.789+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.791174+0000) 2019-10-13T06:48:38.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:38.790+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.791488+0000) 2019-10-13T06:48:38.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:38.789+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:38.789826+0000) 2019-10-13T06:48:38.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:38.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:41.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:41.440+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:41.441417+0000) 2019-10-13T06:48:41.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:41.896+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:41.897416+0000) 2019-10-13T06:48:42.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:42.760+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:42.761258+0000) 2019-10-13T06:48:42.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:42.762+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:42.762339+0000) 2019-10-13T06:48:43.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:43.794+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.796334+0000) 2019-10-13T06:48:43.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:43.795+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.796521+0000) 2019-10-13T06:48:43.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:43.795+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.796717+0000) 2019-10-13T06:48:43.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:43.795+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.796949+0000) 2019-10-13T06:48:43.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:43.793+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.794185+0000) 2019-10-13T06:48:43.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:43.795+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.797096+0000) 2019-10-13T06:48:43.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:43.796+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.797541+0000) 2019-10-13T06:48:43.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:43.796+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:43.796892+0000) 2019-10-13T06:48:43.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:43.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:44.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:44.258+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:44.259086+0000) 2019-10-13T06:48:48.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:48.304+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.304742+0000) 2019-10-13T06:48:48.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:48.641+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.641933+0000) 2019-10-13T06:48:48.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:48.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.641953+0000) 2019-10-13T06:48:48.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:48.797+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.799132+0000) 2019-10-13T06:48:48.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:48.800+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.802283+0000) 2019-10-13T06:48:48.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:48.801+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.802495+0000) 2019-10-13T06:48:48.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:48.801+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.802636+0000) 2019-10-13T06:48:48.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:48.801+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.802980+0000) 2019-10-13T06:48:48.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:48.801+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.803199+0000) 2019-10-13T06:48:48.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:48.802+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.803747+0000) 2019-10-13T06:48:48.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:48.800+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:48.802601+0000) 2019-10-13T06:48:48.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:48.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107557 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:51.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:51.442+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:51.443811+0000) 2019-10-13T06:48:51.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:51.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:51.897583+0000) 2019-10-13T06:48:52.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:52.760+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:52.761429+0000) 2019-10-13T06:48:52.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:52.761+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:52.762543+0000) 2019-10-13T06:48:53.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:53.803+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.804974+0000) 2019-10-13T06:48:53.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:53.806+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.808154+0000) 2019-10-13T06:48:53.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:53.806+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.808361+0000) 2019-10-13T06:48:53.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:53.807+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.809228+0000) 2019-10-13T06:48:53.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:53.808+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.809434+0000) 2019-10-13T06:48:53.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:53.808+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.809636+0000) 2019-10-13T06:48:53.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:53.808+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.809648+0000) 2019-10-13T06:48:53.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:53.806+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:53.808137+0000) 2019-10-13T06:48:53.815 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:53.811+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:48:54.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:54.258+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:54.259280+0000) 2019-10-13T06:48:58.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:58.304+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.304926+0000) 2019-10-13T06:48:58.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:58.641+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.642102+0000) 2019-10-13T06:48:58.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:58.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.642103+0000) 2019-10-13T06:48:58.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:48:58.809+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.811625+0000) 2019-10-13T06:48:58.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:48:58.814+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.814864+0000) 2019-10-13T06:48:58.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:48:58.814+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.815015+0000) 2019-10-13T06:48:58.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:48:58.814+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.815262+0000) 2019-10-13T06:48:58.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:48:58.815+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.815489+0000) 2019-10-13T06:48:58.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:48:58.815+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.815561+0000) 2019-10-13T06:48:58.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:48:58.815+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.815679+0000) 2019-10-13T06:48:58.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:48:58.811+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:48:58.813600+0000) 2019-10-13T06:48:58.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:48:58.819+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:01.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:01.445+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:01.446100+0000) 2019-10-13T06:49:01.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:01.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:01.897713+0000) 2019-10-13T06:49:02.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:02.761+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:02.761571+0000) 2019-10-13T06:49:02.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:02.762+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:02.762760+0000) 2019-10-13T06:49:03.824 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:03.817+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.818108+0000) 2019-10-13T06:49:03.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:03.820+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.821319+0000) 2019-10-13T06:49:03.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:03.821+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.821767+0000) 2019-10-13T06:49:03.826 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:03.821+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.821924+0000) 2019-10-13T06:49:03.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:03.821+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.822087+0000) 2019-10-13T06:49:03.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:03.821+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.822131+0000) 2019-10-13T06:49:03.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:03.821+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.822371+0000) 2019-10-13T06:49:03.827 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:03.819+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:03.820040+0000) 2019-10-13T06:49:03.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:03.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:04.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:04.259+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:04.259451+0000) 2019-10-13T06:49:05.888 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:49:05.889 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:05.893 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:49:05.893 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:08.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:08.304+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.305103+0000) 2019-10-13T06:49:08.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:08.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.642299+0000) 2019-10-13T06:49:08.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:08.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.642300+0000) 2019-10-13T06:49:08.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:08.827+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.828117+0000) 2019-10-13T06:49:08.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:08.828+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.828459+0000) 2019-10-13T06:49:08.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:08.828+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.828734+0000) 2019-10-13T06:49:08.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:08.828+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.828832+0000) 2019-10-13T06:49:08.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:08.828+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.829006+0000) 2019-10-13T06:49:08.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:08.825+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.825857+0000) 2019-10-13T06:49:08.833 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:08.826+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.827004+0000) 2019-10-13T06:49:08.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:08.829+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:08.829512+0000) 2019-10-13T06:49:08.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:08.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:11.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:11.447+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:11.448346+0000) 2019-10-13T06:49:11.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:11.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:11.897880+0000) 2019-10-13T06:49:12.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:12.761+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:12.761780+0000) 2019-10-13T06:49:12.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:12.762+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:12.762949+0000) 2019-10-13T06:49:13.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:13.830+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.830953+0000) 2019-10-13T06:49:13.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:13.832+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.834097+0000) 2019-10-13T06:49:13.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:13.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.834361+0000) 2019-10-13T06:49:13.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:13.833+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.834965+0000) 2019-10-13T06:49:13.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:13.833+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.835215+0000) 2019-10-13T06:49:13.839 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:13.834+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.835615+0000) 2019-10-13T06:49:13.839 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:13.834+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.836375+0000) 2019-10-13T06:49:13.840 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:13.834+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:13.834763+0000) 2019-10-13T06:49:13.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:13.837+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:14.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:14.258+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:14.259638+0000) 2019-10-13T06:49:18.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:18.304+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.305273+0000) 2019-10-13T06:49:18.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:18.641+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.642512+0000) 2019-10-13T06:49:18.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:18.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.642505+0000) 2019-10-13T06:49:18.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:18.839+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.840468+0000) 2019-10-13T06:49:18.844 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:18.839+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.840826+0000) 2019-10-13T06:49:18.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:18.836+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.837694+0000) 2019-10-13T06:49:18.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:18.839+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.841197+0000) 2019-10-13T06:49:18.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:18.839+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.841389+0000) 2019-10-13T06:49:18.845 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:18.840+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.842127+0000) 2019-10-13T06:49:18.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:18.840+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.842374+0000) 2019-10-13T06:49:18.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:18.840+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:18.840862+0000) 2019-10-13T06:49:18.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:18.844+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:21.457 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:21.450+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:21.450830+0000) 2019-10-13T06:49:21.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:21.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:21.898075+0000) 2019-10-13T06:49:22.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:22.760+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:22.761999+0000) 2019-10-13T06:49:22.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:22.762+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:22.763128+0000) 2019-10-13T06:49:23.851 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:23.846+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.848077+0000) 2019-10-13T06:49:23.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:23.846+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.848252+0000) 2019-10-13T06:49:23.852 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:23.847+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.848454+0000) 2019-10-13T06:49:23.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:23.847+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.848633+0000) 2019-10-13T06:49:23.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:23.847+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.848837+0000) 2019-10-13T06:49:23.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:23.847+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.848939+0000) 2019-10-13T06:49:23.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:23.845+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.846123+0000) 2019-10-13T06:49:23.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:23.845+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:23.846632+0000) 2019-10-13T06:49:23.855 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:23.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107774 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:24.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:24.258+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:24.259782+0000) 2019-10-13T06:49:28.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:28.304+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.305429+0000) 2019-10-13T06:49:28.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:28.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.642657+0000) 2019-10-13T06:49:28.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:28.641+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.642722+0000) 2019-10-13T06:49:28.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:28.850+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.851027+0000) 2019-10-13T06:49:28.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:28.853+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.854540+0000) 2019-10-13T06:49:28.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:28.853+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.854756+0000) 2019-10-13T06:49:28.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:28.853+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.854983+0000) 2019-10-13T06:49:28.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:28.853+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.855161+0000) 2019-10-13T06:49:28.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:28.853+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.855387+0000) 2019-10-13T06:49:28.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:28.854+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.855488+0000) 2019-10-13T06:49:28.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:28.853+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:28.854145+0000) 2019-10-13T06:49:28.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:28.857+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:31.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:31.453+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:31.453304+0000) 2019-10-13T06:49:31.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:31.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:31.898258+0000) 2019-10-13T06:49:32.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:32.761+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:32.762183+0000) 2019-10-13T06:49:32.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:32.763+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:32.763313+0000) 2019-10-13T06:49:33.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:33.858+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.860215+0000) 2019-10-13T06:49:33.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:33.858+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.860387+0000) 2019-10-13T06:49:33.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:33.859+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.860612+0000) 2019-10-13T06:49:33.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:33.857+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.857913+0000) 2019-10-13T06:49:33.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:33.859+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.861099+0000) 2019-10-13T06:49:33.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:33.859+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.861262+0000) 2019-10-13T06:49:33.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:33.860+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.861485+0000) 2019-10-13T06:49:33.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:33.859+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:33.859983+0000) 2019-10-13T06:49:33.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:33.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:34.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:34.259+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:34.259961+0000) 2019-10-13T06:49:36.033 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:49:36.033 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:36.038 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:49:36.038 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:38.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:38.305+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.305569+0000) 2019-10-13T06:49:38.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:38.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.642831+0000) 2019-10-13T06:49:38.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:38.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.642920+0000) 2019-10-13T06:49:38.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:38.861+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.862771+0000) 2019-10-13T06:49:38.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:38.865+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.866623+0000) 2019-10-13T06:49:38.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:38.865+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.866875+0000) 2019-10-13T06:49:38.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:38.865+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.867122+0000) 2019-10-13T06:49:38.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:38.865+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.867130+0000) 2019-10-13T06:49:38.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:38.865+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.867270+0000) 2019-10-13T06:49:38.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:38.866+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.867456+0000) 2019-10-13T06:49:38.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:38.863+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:38.865250+0000) 2019-10-13T06:49:38.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:38.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:41.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:41.454+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:41.455779+0000) 2019-10-13T06:49:41.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:41.898+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:41.898394+0000) 2019-10-13T06:49:42.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:42.761+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:42.762332+0000) 2019-10-13T06:49:42.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:42.762+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:42.763492+0000) 2019-10-13T06:49:43.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:43.871+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.871812+0000) 2019-10-13T06:49:43.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:43.871+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.872110+0000) 2019-10-13T06:49:43.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:43.871+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.872320+0000) 2019-10-13T06:49:43.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:43.872+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.872429+0000) 2019-10-13T06:49:43.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:43.872+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.872569+0000) 2019-10-13T06:49:43.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:43.868+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.869796+0000) 2019-10-13T06:49:43.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:43.872+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.872988+0000) 2019-10-13T06:49:43.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:43.870+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:43.872420+0000) 2019-10-13T06:49:43.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:43.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:44.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:44.259+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:44.260134+0000) 2019-10-13T06:49:48.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:48.305+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.305751+0000) 2019-10-13T06:49:48.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:48.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.643019+0000) 2019-10-13T06:49:48.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:48.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.643085+0000) 2019-10-13T06:49:48.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:48.878+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.878483+0000) 2019-10-13T06:49:48.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:48.878+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.878654+0000) 2019-10-13T06:49:48.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:48.878+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.879030+0000) 2019-10-13T06:49:48.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:48.874+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.875814+0000) 2019-10-13T06:49:48.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:48.878+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.879152+0000) 2019-10-13T06:49:48.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:48.879+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.879492+0000) 2019-10-13T06:49:48.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:48.879+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.880299+0000) 2019-10-13T06:49:48.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:48.877+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:48.878955+0000) 2019-10-13T06:49:48.887 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:48.883+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107930 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:51.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:51.458+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:51.458967+0000) 2019-10-13T06:49:51.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:51.898+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:51.898541+0000) 2019-10-13T06:49:52.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:52.762+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:52.762477+0000) 2019-10-13T06:49:52.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:52.762+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:52.763676+0000) 2019-10-13T06:49:53.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:53.884+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.884940+0000) 2019-10-13T06:49:53.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:53.884+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.885269+0000) 2019-10-13T06:49:53.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:53.885+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.885548+0000) 2019-10-13T06:49:53.889 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:53.885+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.885604+0000) 2019-10-13T06:49:53.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:53.884+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.885306+0000) 2019-10-13T06:49:53.890 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:53.885+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.885939+0000) 2019-10-13T06:49:53.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:53.882+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.883130+0000) 2019-10-13T06:49:53.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:53.883+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:53.884089+0000) 2019-10-13T06:49:53.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:53.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:49:54.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:54.260+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:54.260330+0000) 2019-10-13T06:49:58.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:58.305+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.305955+0000) 2019-10-13T06:49:58.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:58.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.643154+0000) 2019-10-13T06:49:58.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:58.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.643250+0000) 2019-10-13T06:49:58.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:49:58.889+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.889665+0000) 2019-10-13T06:49:58.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:49:58.889+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.889853+0000) 2019-10-13T06:49:58.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:49:58.889+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.890136+0000) 2019-10-13T06:49:58.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:49:58.886+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.887723+0000) 2019-10-13T06:49:58.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:49:58.890+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.890403+0000) 2019-10-13T06:49:58.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:49:58.890+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.890463+0000) 2019-10-13T06:49:58.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:49:58.890+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.891090+0000) 2019-10-13T06:49:58.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:49:58.889+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:49:58.890215+0000) 2019-10-13T06:49:58.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:49:58.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 107991 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:01.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:01.460+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:01.461253+0000) 2019-10-13T06:50:01.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:01.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:01.898703+0000) 2019-10-13T06:50:02.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:02.761+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:02.762648+0000) 2019-10-13T06:50:02.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:02.763+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:02.763861+0000) 2019-10-13T06:50:03.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:03.894+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.895704+0000) 2019-10-13T06:50:03.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:03.894+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.895987+0000) 2019-10-13T06:50:03.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:03.892+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.893164+0000) 2019-10-13T06:50:03.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:03.895+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.896412+0000) 2019-10-13T06:50:03.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:03.895+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.896609+0000) 2019-10-13T06:50:03.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:03.895+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.896879+0000) 2019-10-13T06:50:03.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:03.895+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.896792+0000) 2019-10-13T06:50:03.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:03.894+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:03.895576+0000) 2019-10-13T06:50:03.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:03.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:04.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:04.259+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:04.260523+0000) 2019-10-13T06:50:06.195 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:50:06.196 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:06.201 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:50:06.201 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:08.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:08.305+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.306149+0000) 2019-10-13T06:50:08.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:08.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.643349+0000) 2019-10-13T06:50:08.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:08.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.643435+0000) 2019-10-13T06:50:08.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:08.900+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.902166+0000) 2019-10-13T06:50:08.906 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:08.900+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.902335+0000) 2019-10-13T06:50:08.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:08.899+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.899976+0000) 2019-10-13T06:50:08.907 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:08.901+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.902712+0000) 2019-10-13T06:50:08.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:08.901+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.903034+0000) 2019-10-13T06:50:08.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:08.901+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.903232+0000) 2019-10-13T06:50:08.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:08.902+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.903446+0000) 2019-10-13T06:50:08.908 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:08.901+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:08.901840+0000) 2019-10-13T06:50:08.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:08.905+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108054 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:11.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:11.463+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:11.463599+0000) 2019-10-13T06:50:11.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:11.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:11.898856+0000) 2019-10-13T06:50:12.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:12.762+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:12.762813+0000) 2019-10-13T06:50:12.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:12.763+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:12.764020+0000) 2019-10-13T06:50:13.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:13.907+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.908634+0000) 2019-10-13T06:50:13.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:13.907+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.908965+0000) 2019-10-13T06:50:13.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:13.905+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.906253+0000) 2019-10-13T06:50:13.913 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:13.908+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.909539+0000) 2019-10-13T06:50:13.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:13.908+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.909885+0000) 2019-10-13T06:50:13.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:13.908+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.910004+0000) 2019-10-13T06:50:13.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:13.908+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.910302+0000) 2019-10-13T06:50:13.914 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:13.907+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:13.908688+0000) 2019-10-13T06:50:13.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:13.911+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:14.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:14.259+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:14.260696+0000) 2019-10-13T06:50:18.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:18.305+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.306340+0000) 2019-10-13T06:50:18.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:18.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.643525+0000) 2019-10-13T06:50:18.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:18.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.643565+0000) 2019-10-13T06:50:18.919 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:18.913+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.914873+0000) 2019-10-13T06:50:18.919 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:18.913+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.915189+0000) 2019-10-13T06:50:18.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:18.913+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.915350+0000) 2019-10-13T06:50:18.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:18.914+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.915523+0000) 2019-10-13T06:50:18.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:18.914+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.915745+0000) 2019-10-13T06:50:18.921 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:18.914+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.915641+0000) 2019-10-13T06:50:18.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:18.911+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.912085+0000) 2019-10-13T06:50:18.922 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:18.914+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:18.915203+0000) 2019-10-13T06:50:18.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:18.917+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108115 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:21.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:21.465+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:21.466051+0000) 2019-10-13T06:50:21.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:21.897+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:21.899014+0000) 2019-10-13T06:50:22.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:22.762+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:22.762971+0000) 2019-10-13T06:50:22.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:22.763+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:22.764199+0000) 2019-10-13T06:50:23.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:23.917+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.917927+0000) 2019-10-13T06:50:23.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:23.919+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.921081+0000) 2019-10-13T06:50:23.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:23.919+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.921259+0000) 2019-10-13T06:50:23.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:23.920+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.921411+0000) 2019-10-13T06:50:23.925 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:23.920+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.921645+0000) 2019-10-13T06:50:23.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:23.920+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.921674+0000) 2019-10-13T06:50:23.926 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:23.920+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.921884+0000) 2019-10-13T06:50:23.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:23.918+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:23.919602+0000) 2019-10-13T06:50:23.928 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:23.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:24.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:24.260+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:24.260869+0000) 2019-10-13T06:50:28.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:28.306+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.306475+0000) 2019-10-13T06:50:28.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:28.643+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.643723+0000) 2019-10-13T06:50:28.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:28.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.643725+0000) 2019-10-13T06:50:28.930 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:28.926+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.927198+0000) 2019-10-13T06:50:28.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:28.927+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.927668+0000) 2019-10-13T06:50:28.931 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:28.927+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.927835+0000) 2019-10-13T06:50:28.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:28.924+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.925204+0000) 2019-10-13T06:50:28.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:28.927+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.928339+0000) 2019-10-13T06:50:28.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:28.928+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.928508+0000) 2019-10-13T06:50:28.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:28.928+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.928843+0000) 2019-10-13T06:50:28.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:28.925+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:28.926629+0000) 2019-10-13T06:50:28.935 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:28.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:31.474 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:31.468+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:31.468425+0000) 2019-10-13T06:50:31.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:31.898+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:31.899156+0000) 2019-10-13T06:50:32.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:32.762+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:32.763137+0000) 2019-10-13T06:50:32.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:32.763+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:32.764340+0000) 2019-10-13T06:50:33.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:33.929+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.931551+0000) 2019-10-13T06:50:33.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:33.931+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.933409+0000) 2019-10-13T06:50:33.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:33.933+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.933455+0000) 2019-10-13T06:50:33.943 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:33.933+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.933926+0000) 2019-10-13T06:50:33.943 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:33.933+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.934153+0000) 2019-10-13T06:50:33.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:33.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.934309+0000) 2019-10-13T06:50:33.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:33.934+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.934660+0000) 2019-10-13T06:50:33.944 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:33.934+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:33.934869+0000) 2019-10-13T06:50:33.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:33.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:34.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:34.260+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:34.261045+0000) 2019-10-13T06:50:36.350 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:50:36.350 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:36.354 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:50:36.355 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:38.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:38.306+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.306652+0000) 2019-10-13T06:50:38.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:38.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.643907+0000) 2019-10-13T06:50:38.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:38.643+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.643912+0000) 2019-10-13T06:50:38.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:38.939+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.940207+0000) 2019-10-13T06:50:38.944 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:38.940+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.940539+0000) 2019-10-13T06:50:38.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:38.940+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.940631+0000) 2019-10-13T06:50:38.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:38.940+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.940790+0000) 2019-10-13T06:50:38.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:38.935+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.937332+0000) 2019-10-13T06:50:38.945 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:38.940+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.941137+0000) 2019-10-13T06:50:38.946 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:38.941+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.941416+0000) 2019-10-13T06:50:38.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:38.937+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:38.939021+0000) 2019-10-13T06:50:38.948 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:38.944+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:41.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:41.469+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:41.470742+0000) 2019-10-13T06:50:41.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:41.898+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:41.899353+0000) 2019-10-13T06:50:42.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:42.762+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:42.763301+0000) 2019-10-13T06:50:42.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:42.763+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:42.764488+0000) 2019-10-13T06:50:43.950 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:43.946+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.946785+0000) 2019-10-13T06:50:43.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:43.946+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.947141+0000) 2019-10-13T06:50:43.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:43.947+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.947416+0000) 2019-10-13T06:50:43.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:43.942+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.944535+0000) 2019-10-13T06:50:43.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:43.947+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.947684+0000) 2019-10-13T06:50:43.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:43.947+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.947899+0000) 2019-10-13T06:50:43.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:43.947+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.948031+0000) 2019-10-13T06:50:43.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:43.945+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:43.946795+0000) 2019-10-13T06:50:43.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:43.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:44.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:44.260+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:44.261248+0000) 2019-10-13T06:50:48.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:48.305+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.306847+0000) 2019-10-13T06:50:48.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:48.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.644092+0000) 2019-10-13T06:50:48.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:48.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.644093+0000) 2019-10-13T06:50:48.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:48.951+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.952881+0000) 2019-10-13T06:50:48.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:48.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.953051+0000) 2019-10-13T06:50:48.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:48.951+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.953170+0000) 2019-10-13T06:50:48.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:48.951+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.953308+0000) 2019-10-13T06:50:48.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:48.949+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.950610+0000) 2019-10-13T06:50:48.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:48.952+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.953923+0000) 2019-10-13T06:50:48.959 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:48.952+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.954205+0000) 2019-10-13T06:50:48.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:48.952+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:48.953138+0000) 2019-10-13T06:50:48.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:48.955+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:51.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:51.472+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:51.473138+0000) 2019-10-13T06:50:51.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:51.898+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:51.899593+0000) 2019-10-13T06:50:52.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:52.762+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:52.763472+0000) 2019-10-13T06:50:52.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:52.763+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:52.764671+0000) 2019-10-13T06:50:53.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:53.956+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.957829+0000) 2019-10-13T06:50:53.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:53.956+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.958121+0000) 2019-10-13T06:50:53.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:53.956+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.958025+0000) 2019-10-13T06:50:53.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:53.956+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.958237+0000) 2019-10-13T06:50:53.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:53.954+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.955711+0000) 2019-10-13T06:50:53.963 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:53.957+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.958778+0000) 2019-10-13T06:50:53.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:53.957+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.959057+0000) 2019-10-13T06:50:53.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:53.956+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:53.957220+0000) 2019-10-13T06:50:53.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:53.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:50:54.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:54.260+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:54.261457+0000) 2019-10-13T06:50:58.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:58.306+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.307061+0000) 2019-10-13T06:50:58.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:58.642+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.644261+0000) 2019-10-13T06:50:58.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:58.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.644258+0000) 2019-10-13T06:50:58.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:50:58.959+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.960347+0000) 2019-10-13T06:50:58.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:50:58.962+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.963591+0000) 2019-10-13T06:50:58.967 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:50:58.962+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.963672+0000) 2019-10-13T06:50:58.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:50:58.962+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.964163+0000) 2019-10-13T06:50:58.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:50:58.962+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.964029+0000) 2019-10-13T06:50:58.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:50:58.963+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.964483+0000) 2019-10-13T06:50:58.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:50:58.963+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.964578+0000) 2019-10-13T06:50:58.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:50:58.963+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:50:58.963903+0000) 2019-10-13T06:50:58.971 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:50:58.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:01.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:01.475+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:01.475529+0000) 2019-10-13T06:51:01.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:01.898+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:01.899771+0000) 2019-10-13T06:51:02.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:02.762+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:02.763603+0000) 2019-10-13T06:51:02.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:02.764+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:02.764835+0000) 2019-10-13T06:51:03.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:03.968+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.969600+0000) 2019-10-13T06:51:03.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:03.968+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.969974+0000) 2019-10-13T06:51:03.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:03.966+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.966882+0000) 2019-10-13T06:51:03.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:03.968+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.970356+0000) 2019-10-13T06:51:03.974 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:03.969+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.970440+0000) 2019-10-13T06:51:03.975 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:03.969+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.970920+0000) 2019-10-13T06:51:03.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:03.969+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.971134+0000) 2019-10-13T06:51:03.975 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:03.968+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:03.969366+0000) 2019-10-13T06:51:03.978 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:03.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108394 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:04.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:04.260+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:04.261653+0000) 2019-10-13T06:51:06.445 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:51:06.445 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:06.450 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:51:06.450 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:08.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:08.306+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.307276+0000) 2019-10-13T06:51:08.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:08.643+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.644437+0000) 2019-10-13T06:51:08.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:08.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.644441+0000) 2019-10-13T06:51:08.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:08.973+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.975361+0000) 2019-10-13T06:51:08.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:08.974+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.975649+0000) 2019-10-13T06:51:08.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:08.974+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.975977+0000) 2019-10-13T06:51:08.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:08.972+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.972827+0000) 2019-10-13T06:51:08.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:08.974+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.976227+0000) 2019-10-13T06:51:08.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:08.975+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.976516+0000) 2019-10-13T06:51:08.981 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:08.975+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.976797+0000) 2019-10-13T06:51:08.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:08.974+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:08.975250+0000) 2019-10-13T06:51:08.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:08.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:11.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:11.477+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:11.477914+0000) 2019-10-13T06:51:11.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:11.899+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:11.899931+0000) 2019-10-13T06:51:12.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:12.763+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:12.763726+0000) 2019-10-13T06:51:12.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:12.764+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:12.765011+0000) 2019-10-13T06:51:13.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:13.978+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.978960+0000) 2019-10-13T06:51:13.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:13.981+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.982151+0000) 2019-10-13T06:51:13.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:13.981+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.982288+0000) 2019-10-13T06:51:13.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:13.982+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.982513+0000) 2019-10-13T06:51:13.987 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:13.982+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.982602+0000) 2019-10-13T06:51:13.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:13.982+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.982827+0000) 2019-10-13T06:51:13.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:13.982+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.982975+0000) 2019-10-13T06:51:13.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:13.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:13.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:13.982+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:13.983312+0000) 2019-10-13T06:51:14.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:14.261+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:14.261798+0000) 2019-10-13T06:51:18.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:18.307+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.307475+0000) 2019-10-13T06:51:18.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:18.644+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.644598+0000) 2019-10-13T06:51:18.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:18.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.644598+0000) 2019-10-13T06:51:18.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:18.984+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.985227+0000) 2019-10-13T06:51:18.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:18.987+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.988308+0000) 2019-10-13T06:51:18.992 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:18.988+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.988478+0000) 2019-10-13T06:51:18.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:18.988+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.988734+0000) 2019-10-13T06:51:18.993 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:18.988+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.988964+0000) 2019-10-13T06:51:18.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:18.988+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.989368+0000) 2019-10-13T06:51:18.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:18.989+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.989796+0000) 2019-10-13T06:51:18.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:18.987+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:18.988432+0000) 2019-10-13T06:51:18.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:18.992+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:21.486 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:21.480+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:21.480253+0000) 2019-10-13T06:51:21.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:21.899+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:21.900125+0000) 2019-10-13T06:51:22.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:22.763+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:22.763898+0000) 2019-10-13T06:51:22.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:22.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:22.765169+0000) 2019-10-13T06:51:23.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:23.991+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.991924+0000) 2019-10-13T06:51:23.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:23.994+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.995130+0000) 2019-10-13T06:51:23.999 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:23.994+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.995040+0000) 2019-10-13T06:51:23.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:23.995+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.995387+0000) 2019-10-13T06:51:23.999 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:23.995+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.995596+0000) 2019-10-13T06:51:24.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:23.995+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.995912+0000) 2019-10-13T06:51:24.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:23.995+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.995818+0000) 2019-10-13T06:51:24.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:23.994+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:23.995236+0000) 2019-10-13T06:51:24.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:23.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:24.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:24.261+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:24.261977+0000) 2019-10-13T06:51:28.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:28.307+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:28.307672+0000) 2019-10-13T06:51:28.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:28.644+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:28.644770+0000) 2019-10-13T06:51:28.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:28.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:28.644770+0000) 2019-10-13T06:51:29.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:29.000+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:29.000975+0000) 2019-10-13T06:51:29.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:29.000+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:29.001123+0000) 2019-10-13T06:51:29.005 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:29.000+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:29.001329+0000) 2019-10-13T06:51:29.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:28.997+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:28.999047+0000) 2019-10-13T06:51:29.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:29.001+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:29.001553+0000) 2019-10-13T06:51:29.006 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:29.001+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:29.001720+0000) 2019-10-13T06:51:29.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:29.002+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:29.002448+0000) 2019-10-13T06:51:29.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:28.999+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:29.001367+0000) 2019-10-13T06:51:29.008 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:29.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108550 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:31.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:31.481+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:31.482679+0000) 2019-10-13T06:51:31.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:31.898+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:31.900303+0000) 2019-10-13T06:51:32.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:32.763+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:32.764086+0000) 2019-10-13T06:51:32.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:32.764+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:32.765350+0000) 2019-10-13T06:51:34.010 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:34.005+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.007053+0000) 2019-10-13T06:51:34.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:34.005+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.007357+0000) 2019-10-13T06:51:34.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:34.003+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.005077+0000) 2019-10-13T06:51:34.011 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:34.006+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.007531+0000) 2019-10-13T06:51:34.012 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:34.006+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.007630+0000) 2019-10-13T06:51:34.012 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:34.006+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.008227+0000) 2019-10-13T06:51:34.012 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:34.006+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.008283+0000) 2019-10-13T06:51:34.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:34.005+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.007140+0000) 2019-10-13T06:51:34.015 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:34.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:34.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:34.261+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:34.262143+0000) 2019-10-13T06:51:36.593 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:51:36.593 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:36.597 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:51:36.597 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:38.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:38.306+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:38.307862+0000) 2019-10-13T06:51:38.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:38.643+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:38.644955+0000) 2019-10-13T06:51:38.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:38.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:38.644955+0000) 2019-10-13T06:51:39.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:39.011+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.013309+0000) 2019-10-13T06:51:39.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:39.012+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.013518+0000) 2019-10-13T06:51:39.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:39.012+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.013617+0000) 2019-10-13T06:51:39.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:39.009+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.011307+0000) 2019-10-13T06:51:39.018 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:39.013+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.014421+0000) 2019-10-13T06:51:39.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:39.013+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.014569+0000) 2019-10-13T06:51:39.018 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:39.013+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.014707+0000) 2019-10-13T06:51:39.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:39.011+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:39.013248+0000) 2019-10-13T06:51:39.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:39.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:41.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:41.484+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:41.485030+0000) 2019-10-13T06:51:41.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:41.899+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:41.900481+0000) 2019-10-13T06:51:42.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:42.763+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:42.764276+0000) 2019-10-13T06:51:42.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:42.764+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:42.765501+0000) 2019-10-13T06:51:44.022 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:44.018+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.019503+0000) 2019-10-13T06:51:44.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:44.016+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.016747+0000) 2019-10-13T06:51:44.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:44.018+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.019989+0000) 2019-10-13T06:51:44.024 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:44.018+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.020201+0000) 2019-10-13T06:51:44.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:44.019+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.020481+0000) 2019-10-13T06:51:44.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:44.019+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.020866+0000) 2019-10-13T06:51:44.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:44.019+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.021072+0000) 2019-10-13T06:51:44.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:44.018+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.019317+0000) 2019-10-13T06:51:44.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:44.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108642 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:44.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:44.260+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:44.262334+0000) 2019-10-13T06:51:48.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:48.307+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:48.308067+0000) 2019-10-13T06:51:48.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:48.644+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:48.645149+0000) 2019-10-13T06:51:48.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:48.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:48.645150+0000) 2019-10-13T06:51:49.029 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:49.025+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.026414+0000) 2019-10-13T06:51:49.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:49.022+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.023665+0000) 2019-10-13T06:51:49.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:49.025+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.027062+0000) 2019-10-13T06:51:49.031 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:49.025+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.027233+0000) 2019-10-13T06:51:49.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:49.025+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.027374+0000) 2019-10-13T06:51:49.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:49.026+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.027495+0000) 2019-10-13T06:51:49.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:49.026+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.027765+0000) 2019-10-13T06:51:49.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:49.025+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:49.026018+0000) 2019-10-13T06:51:49.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:49.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:51.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:51.486+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:51.487289+0000) 2019-10-13T06:51:51.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:51.899+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:51.900629+0000) 2019-10-13T06:51:52.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:52.763+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:52.764452+0000) 2019-10-13T06:51:52.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:52.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:52.765665+0000) 2019-10-13T06:51:54.036 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:54.031+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.033017+0000) 2019-10-13T06:51:54.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:54.031+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.033219+0000) 2019-10-13T06:51:54.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:54.031+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.033287+0000) 2019-10-13T06:51:54.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:54.032+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.033477+0000) 2019-10-13T06:51:54.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:54.032+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.033711+0000) 2019-10-13T06:51:54.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:54.030+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.030999+0000) 2019-10-13T06:51:54.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:54.031+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.032111+0000) 2019-10-13T06:51:54.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:54.032+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.034243+0000) 2019-10-13T06:51:54.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:54.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:51:54.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:54.261+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:54.262504+0000) 2019-10-13T06:51:58.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:58.307+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:58.308239+0000) 2019-10-13T06:51:58.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:58.643+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:58.645300+0000) 2019-10-13T06:51:58.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:58.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:58.645300+0000) 2019-10-13T06:51:59.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:51:59.038+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.040216+0000) 2019-10-13T06:51:59.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:51:59.039+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.040627+0000) 2019-10-13T06:51:59.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:51:59.039+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.040788+0000) 2019-10-13T06:51:59.044 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:51:59.039+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.041093+0000) 2019-10-13T06:51:59.045 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:51:59.039+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.041184+0000) 2019-10-13T06:51:59.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:51:59.037+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.038092+0000) 2019-10-13T06:51:59.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:51:59.039+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.041367+0000) 2019-10-13T06:51:59.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:51:59.040+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:51:59.040986+0000) 2019-10-13T06:51:59.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:51:59.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:01.496 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:01.489+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:01.489634+0000) 2019-10-13T06:52:01.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:01.900+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:01.900805+0000) 2019-10-13T06:52:02.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:02.764+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:02.764641+0000) 2019-10-13T06:52:02.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:02.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:02.765852+0000) 2019-10-13T06:52:04.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:04.045+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.046326+0000) 2019-10-13T06:52:04.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:04.042+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.043631+0000) 2019-10-13T06:52:04.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:04.046+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.046810+0000) 2019-10-13T06:52:04.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:04.046+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.047006+0000) 2019-10-13T06:52:04.051 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:04.046+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.047350+0000) 2019-10-13T06:52:04.051 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:04.046+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.047143+0000) 2019-10-13T06:52:04.051 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:04.047+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.047499+0000) 2019-10-13T06:52:04.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:04.045+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.045891+0000) 2019-10-13T06:52:04.053 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:04.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:04.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:04.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:04.262658+0000) 2019-10-13T06:52:06.705 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:52:06.705 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:06.710 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:52:06.710 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:08.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:08.308+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:08.308384+0000) 2019-10-13T06:52:08.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:08.645+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:08.645490+0000) 2019-10-13T06:52:08.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:08.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:08.645490+0000) 2019-10-13T06:52:09.056 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:09.052+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.052511+0000) 2019-10-13T06:52:09.056 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:09.052+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.052640+0000) 2019-10-13T06:52:09.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:09.052+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.052893+0000) 2019-10-13T06:52:09.057 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:09.052+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.053065+0000) 2019-10-13T06:52:09.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:09.052+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.053190+0000) 2019-10-13T06:52:09.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:09.049+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.049864+0000) 2019-10-13T06:52:09.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:09.053+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.053855+0000) 2019-10-13T06:52:09.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:09.051+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:09.052448+0000) 2019-10-13T06:52:09.059 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:09.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:11.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:11.491+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:11.492043+0000) 2019-10-13T06:52:11.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:11.900+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:11.901011+0000) 2019-10-13T06:52:12.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:12.764+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:12.764849+0000) 2019-10-13T06:52:12.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:12.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:12.766037+0000) 2019-10-13T06:52:14.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:14.057+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.058288+0000) 2019-10-13T06:52:14.062 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:14.058+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.058673+0000) 2019-10-13T06:52:14.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:14.058+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.058534+0000) 2019-10-13T06:52:14.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:14.054+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.055183+0000) 2019-10-13T06:52:14.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:14.058+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.059067+0000) 2019-10-13T06:52:14.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:14.058+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.059228+0000) 2019-10-13T06:52:14.063 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:14.058+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.059290+0000) 2019-10-13T06:52:14.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:14.058+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.058892+0000) 2019-10-13T06:52:14.065 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:14.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:14.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:14.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:14.262806+0000) 2019-10-13T06:52:18.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:18.307+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:18.308552+0000) 2019-10-13T06:52:18.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:18.644+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:18.645676+0000) 2019-10-13T06:52:18.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:18.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:18.645676+0000) 2019-10-13T06:52:19.068 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:19.063+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.064722+0000) 2019-10-13T06:52:19.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:19.063+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.064603+0000) 2019-10-13T06:52:19.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:19.063+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.064985+0000) 2019-10-13T06:52:19.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:19.060+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.062393+0000) 2019-10-13T06:52:19.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:19.064+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.065579+0000) 2019-10-13T06:52:19.070 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:19.064+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.065474+0000) 2019-10-13T06:52:19.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:19.064+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.065676+0000) 2019-10-13T06:52:19.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:19.061+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:19.063666+0000) 2019-10-13T06:52:19.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:19.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:21.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:21.493+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:21.494384+0000) 2019-10-13T06:52:21.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:21.900+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:21.901204+0000) 2019-10-13T06:52:22.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:22.764+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:22.765045+0000) 2019-10-13T06:52:22.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:22.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:22.766165+0000) 2019-10-13T06:52:24.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:24.069+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.070417+0000) 2019-10-13T06:52:24.074 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:24.069+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.070661+0000) 2019-10-13T06:52:24.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:24.066+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.068051+0000) 2019-10-13T06:52:24.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:24.069+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.070806+0000) 2019-10-13T06:52:24.075 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:24.069+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.071159+0000) 2019-10-13T06:52:24.075 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:24.069+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.071249+0000) 2019-10-13T06:52:24.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:24.070+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.071418+0000) 2019-10-13T06:52:24.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:24.069+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.071212+0000) 2019-10-13T06:52:24.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:24.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:24.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:24.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:24.262984+0000) 2019-10-13T06:52:28.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:28.307+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:28.308747+0000) 2019-10-13T06:52:28.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:28.645+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:28.645866+0000) 2019-10-13T06:52:28.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:28.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:28.645866+0000) 2019-10-13T06:52:29.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:29.071+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.073135+0000) 2019-10-13T06:52:29.080 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:29.075+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.076416+0000) 2019-10-13T06:52:29.081 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:29.075+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.076721+0000) 2019-10-13T06:52:29.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:29.075+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.076918+0000) 2019-10-13T06:52:29.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:29.075+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.077072+0000) 2019-10-13T06:52:29.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:29.075+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.077364+0000) 2019-10-13T06:52:29.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:29.076+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.077586+0000) 2019-10-13T06:52:29.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:29.073+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:29.075455+0000) 2019-10-13T06:52:29.083 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:29.079+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108922 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:31.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:31.495+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:31.496768+0000) 2019-10-13T06:52:31.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:31.900+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:31.901426+0000) 2019-10-13T06:52:32.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:32.763+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:32.765239+0000) 2019-10-13T06:52:32.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:32.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:32.766333+0000) 2019-10-13T06:52:34.086 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:34.080+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.082259+0000) 2019-10-13T06:52:34.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:34.081+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.082596+0000) 2019-10-13T06:52:34.086 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:34.081+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.082713+0000) 2019-10-13T06:52:34.086 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:34.081+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.082788+0000) 2019-10-13T06:52:34.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:34.079+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.080105+0000) 2019-10-13T06:52:34.087 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:34.082+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.083489+0000) 2019-10-13T06:52:34.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:34.082+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.083713+0000) 2019-10-13T06:52:34.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:34.081+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.082589+0000) 2019-10-13T06:52:34.090 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:34.085+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108953 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:34.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:34.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:34.263115+0000) 2019-10-13T06:52:36.851 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:52:36.851 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:36.855 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:52:36.857 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:38.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:38.308+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:38.308939+0000) 2019-10-13T06:52:38.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:38.645+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:38.646080+0000) 2019-10-13T06:52:38.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:38.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:38.646080+0000) 2019-10-13T06:52:39.092 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:39.087+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.088791+0000) 2019-10-13T06:52:39.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:39.087+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.089036+0000) 2019-10-13T06:52:39.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:39.087+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.089104+0000) 2019-10-13T06:52:39.093 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:39.087+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.089350+0000) 2019-10-13T06:52:39.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:39.086+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.087031+0000) 2019-10-13T06:52:39.094 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:39.088+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.089657+0000) 2019-10-13T06:52:39.094 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:39.088+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.089814+0000) 2019-10-13T06:52:39.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:39.088+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:39.088965+0000) 2019-10-13T06:52:39.096 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:39.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 108983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:41.505 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:41.498+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:41.499179+0000) 2019-10-13T06:52:41.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:41.901+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:41.901641+0000) 2019-10-13T06:52:42.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:42.764+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:42.765453+0000) 2019-10-13T06:52:42.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:42.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:42.766498+0000) 2019-10-13T06:52:44.098 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:44.093+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.095365+0000) 2019-10-13T06:52:44.099 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:44.094+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.095521+0000) 2019-10-13T06:52:44.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:44.094+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.095674+0000) 2019-10-13T06:52:44.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:44.094+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.095963+0000) 2019-10-13T06:52:44.100 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:44.094+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.096263+0000) 2019-10-13T06:52:44.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:44.093+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.093908+0000) 2019-10-13T06:52:44.100 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:44.095+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.097005+0000) 2019-10-13T06:52:44.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:44.095+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.096560+0000) 2019-10-13T06:52:44.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:44.099+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:44.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:44.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:44.263279+0000) 2019-10-13T06:52:48.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:48.308+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:48.309120+0000) 2019-10-13T06:52:48.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:48.645+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:48.646319+0000) 2019-10-13T06:52:48.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:48.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:48.646319+0000) 2019-10-13T06:52:49.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:49.101+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.102329+0000) 2019-10-13T06:52:49.106 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:49.102+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.102606+0000) 2019-10-13T06:52:49.106 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:49.102+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.102829+0000) 2019-10-13T06:52:49.106 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:49.102+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.102864+0000) 2019-10-13T06:52:49.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:49.098+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.099612+0000) 2019-10-13T06:52:49.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:49.102+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.103186+0000) 2019-10-13T06:52:49.107 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:49.103+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.103489+0000) 2019-10-13T06:52:49.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:49.101+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:49.102605+0000) 2019-10-13T06:52:49.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:49.105+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:51.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:51.501+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:51.501465+0000) 2019-10-13T06:52:51.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:51.901+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:51.901827+0000) 2019-10-13T06:52:52.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:52.765+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:52.765627+0000) 2019-10-13T06:52:52.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:52.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:52.766647+0000) 2019-10-13T06:52:54.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:54.103+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.104429+0000) 2019-10-13T06:52:54.111 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:54.107+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.107604+0000) 2019-10-13T06:52:54.112 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:54.107+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.107667+0000) 2019-10-13T06:52:54.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:54.107+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.108072+0000) 2019-10-13T06:52:54.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:54.107+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.108291+0000) 2019-10-13T06:52:54.112 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:54.108+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.108407+0000) 2019-10-13T06:52:54.112 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:54.108+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.108585+0000) 2019-10-13T06:52:54.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:54.106+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.107022+0000) 2019-10-13T06:52:54.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:54.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:54.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:54.263+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:54.263453+0000) 2019-10-13T06:52:58.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:58.309+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:58.309287+0000) 2019-10-13T06:52:58.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:58.646+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:58.646422+0000) 2019-10-13T06:52:58.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:58.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:58.646423+0000) 2019-10-13T06:52:59.115 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:52:59.111+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.112309+0000) 2019-10-13T06:52:59.116 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:52:59.112+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.112464+0000) 2019-10-13T06:52:59.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:52:59.109+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.109774+0000) 2019-10-13T06:52:59.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:52:59.112+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.112901+0000) 2019-10-13T06:52:59.117 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:52:59.112+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.112971+0000) 2019-10-13T06:52:59.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:52:59.112+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.113066+0000) 2019-10-13T06:52:59.117 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:52:59.112+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.113273+0000) 2019-10-13T06:52:59.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:52:59.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:52:59.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:52:59.111+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:52:59.112688+0000) 2019-10-13T06:53:01.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:01.503+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:01.503751+0000) 2019-10-13T06:53:01.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:01.901+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:01.902017+0000) 2019-10-13T06:53:02.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:02.765+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:02.765847+0000) 2019-10-13T06:53:02.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:02.766+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:02.766766+0000) 2019-10-13T06:53:04.121 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:04.116+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.117611+0000) 2019-10-13T06:53:04.125 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:04.116+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.117998+0000) 2019-10-13T06:53:04.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:04.116+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.118357+0000) 2019-10-13T06:53:04.125 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:04.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.118291+0000) 2019-10-13T06:53:04.126 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:04.117+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.118534+0000) 2019-10-13T06:53:04.126 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:04.117+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.119144+0000) 2019-10-13T06:53:04.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:04.120+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109138 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:04.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:04.115+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.115830+0000) 2019-10-13T06:53:04.127 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:04.116+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.116752+0000) 2019-10-13T06:53:04.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:04.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:04.263614+0000) 2019-10-13T06:53:06.961 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:53:06.961 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:06.965 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:53:06.965 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:08.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:08.308+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:08.309485+0000) 2019-10-13T06:53:08.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:08.645+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:08.646609+0000) 2019-10-13T06:53:08.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:08.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:08.646608+0000) 2019-10-13T06:53:09.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:09.122+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.124100+0000) 2019-10-13T06:53:09.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:09.120+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.121435+0000) 2019-10-13T06:53:09.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:09.123+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.124678+0000) 2019-10-13T06:53:09.128 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:09.123+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.124807+0000) 2019-10-13T06:53:09.129 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:09.123+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.125108+0000) 2019-10-13T06:53:09.129 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:09.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.125117+0000) 2019-10-13T06:53:09.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:09.124+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.125578+0000) 2019-10-13T06:53:09.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:09.123+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:09.124637+0000) 2019-10-13T06:53:09.131 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:09.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:11.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:11.505+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:11.506025+0000) 2019-10-13T06:53:11.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:11.901+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:11.902192+0000) 2019-10-13T06:53:12.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:12.765+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:12.766047+0000) 2019-10-13T06:53:12.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:12.765+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:12.766953+0000) 2019-10-13T06:53:14.134 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:14.129+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.130488+0000) 2019-10-13T06:53:14.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:14.129+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.130662+0000) 2019-10-13T06:53:14.134 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:14.129+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.130827+0000) 2019-10-13T06:53:14.135 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:14.126+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.128091+0000) 2019-10-13T06:53:14.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:14.129+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.131132+0000) 2019-10-13T06:53:14.135 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:14.129+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.131192+0000) 2019-10-13T06:53:14.135 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:14.130+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.131672+0000) 2019-10-13T06:53:14.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:14.129+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.131413+0000) 2019-10-13T06:53:14.138 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:14.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:14.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:14.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:14.263753+0000) 2019-10-13T06:53:18.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:18.308+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:18.309675+0000) 2019-10-13T06:53:18.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:18.645+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:18.646791+0000) 2019-10-13T06:53:18.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:18.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:18.646757+0000) 2019-10-13T06:53:19.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:19.135+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.136654+0000) 2019-10-13T06:53:19.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:19.135+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.136850+0000) 2019-10-13T06:53:19.141 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:19.135+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.137020+0000) 2019-10-13T06:53:19.141 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:19.135+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.137371+0000) 2019-10-13T06:53:19.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:19.133+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.135027+0000) 2019-10-13T06:53:19.142 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:19.136+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.137565+0000) 2019-10-13T06:53:19.142 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:19.136+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.137669+0000) 2019-10-13T06:53:19.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:19.134+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:19.136454+0000) 2019-10-13T06:53:19.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:19.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:21.514 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:21.507+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:21.508328+0000) 2019-10-13T06:53:21.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:21.901+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:21.902417+0000) 2019-10-13T06:53:22.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:22.765+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:22.766237+0000) 2019-10-13T06:53:22.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:22.766+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:22.767170+0000) 2019-10-13T06:53:24.146 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:24.141+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.142559+0000) 2019-10-13T06:53:24.146 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:24.141+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.142857+0000) 2019-10-13T06:53:24.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:24.141+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.142739+0000) 2019-10-13T06:53:24.147 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:24.141+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.142954+0000) 2019-10-13T06:53:24.147 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:24.138+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.140317+0000) 2019-10-13T06:53:24.148 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:24.142+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.143589+0000) 2019-10-13T06:53:24.148 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:24.142+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.143808+0000) 2019-10-13T06:53:24.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:24.140+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.142666+0000) 2019-10-13T06:53:24.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:24.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:24.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:24.263+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:24.263883+0000) 2019-10-13T06:53:28.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:28.308+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:28.309793+0000) 2019-10-13T06:53:28.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:28.645+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:28.646993+0000) 2019-10-13T06:53:28.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:28.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:28.646995+0000) 2019-10-13T06:53:29.151 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:29.146+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.148070+0000) 2019-10-13T06:53:29.151 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:29.147+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.148557+0000) 2019-10-13T06:53:29.152 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:29.145+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.146034+0000) 2019-10-13T06:53:29.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:29.147+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.148753+0000) 2019-10-13T06:53:29.153 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:29.147+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.148997+0000) 2019-10-13T06:53:29.153 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:29.147+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.149162+0000) 2019-10-13T06:53:29.153 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:29.147+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.149333+0000) 2019-10-13T06:53:29.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:29.147+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:29.148010+0000) 2019-10-13T06:53:29.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:29.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:31.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:31.509+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:31.510668+0000) 2019-10-13T06:53:31.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:31.902+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:31.902606+0000) 2019-10-13T06:53:32.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:32.766+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:32.766447+0000) 2019-10-13T06:53:32.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:32.766+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:32.767350+0000) 2019-10-13T06:53:34.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:34.154+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.154393+0000) 2019-10-13T06:53:34.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:34.154+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.154609+0000) 2019-10-13T06:53:34.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:34.154+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.154950+0000) 2019-10-13T06:53:34.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:34.151+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.152316+0000) 2019-10-13T06:53:34.160 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:34.154+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.155384+0000) 2019-10-13T06:53:34.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:34.155+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.155544+0000) 2019-10-13T06:53:34.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:34.155+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.155768+0000) 2019-10-13T06:53:34.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:34.153+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.154516+0000) 2019-10-13T06:53:34.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:34.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:34.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:34.263+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:34.264067+0000) 2019-10-13T06:53:37.115 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:53:37.115 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:37.120 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:53:37.120 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:38.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:38.309+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:38.309969+0000) 2019-10-13T06:53:38.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:38.646+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:38.647188+0000) 2019-10-13T06:53:38.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:38.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:38.647198+0000) 2019-10-13T06:53:39.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:39.160+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.160434+0000) 2019-10-13T06:53:39.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:39.160+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.160824+0000) 2019-10-13T06:53:39.164 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:39.160+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.161078+0000) 2019-10-13T06:53:39.164 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:39.160+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.161357+0000) 2019-10-13T06:53:39.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:39.157+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.158402+0000) 2019-10-13T06:53:39.165 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:39.161+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.161649+0000) 2019-10-13T06:53:39.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:39.161+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.161862+0000) 2019-10-13T06:53:39.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:39.160+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:39.161311+0000) 2019-10-13T06:53:39.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:39.164+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:41.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:41.512+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:41.513067+0000) 2019-10-13T06:53:41.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:41.902+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:41.902778+0000) 2019-10-13T06:53:42.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:42.766+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:42.766649+0000) 2019-10-13T06:53:42.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:42.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:42.767501+0000) 2019-10-13T06:53:44.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:44.163+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.164263+0000) 2019-10-13T06:53:44.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:44.165+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.166209+0000) 2019-10-13T06:53:44.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:44.166+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.166781+0000) 2019-10-13T06:53:44.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:44.166+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.167024+0000) 2019-10-13T06:53:44.175 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:44.166+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.167255+0000) 2019-10-13T06:53:44.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:44.166+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.167172+0000) 2019-10-13T06:53:44.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:44.167+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.167410+0000) 2019-10-13T06:53:44.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:44.167+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.167556+0000) 2019-10-13T06:53:44.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:44.169+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109386 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:44.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:44.264+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:44.264224+0000) 2019-10-13T06:53:48.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:48.310+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:48.310198+0000) 2019-10-13T06:53:48.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:48.647+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:48.647376+0000) 2019-10-13T06:53:48.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:48.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:48.647412+0000) 2019-10-13T06:53:49.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:49.171+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.171759+0000) 2019-10-13T06:53:49.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:49.171+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.171900+0000) 2019-10-13T06:53:49.177 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:49.171+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.172224+0000) 2019-10-13T06:53:49.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:49.171+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.172320+0000) 2019-10-13T06:53:49.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:49.172+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.172479+0000) 2019-10-13T06:53:49.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:49.172+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.172702+0000) 2019-10-13T06:53:49.178 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:49.168+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.168840+0000) 2019-10-13T06:53:49.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:49.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109418 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:49.179 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:49.170+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:49.171388+0000) 2019-10-13T06:53:51.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:51.515+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:51.516211+0000) 2019-10-13T06:53:51.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:51.901+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:51.902965+0000) 2019-10-13T06:53:52.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:52.765+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:52.766852+0000) 2019-10-13T06:53:52.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:52.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:52.767688+0000) 2019-10-13T06:53:54.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:54.176+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.177799+0000) 2019-10-13T06:53:54.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:54.176+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.178162+0000) 2019-10-13T06:53:54.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:54.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.175171+0000) 2019-10-13T06:53:54.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:54.177+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.178531+0000) 2019-10-13T06:53:54.182 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:54.177+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.178753+0000) 2019-10-13T06:53:54.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:54.177+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.178963+0000) 2019-10-13T06:53:54.183 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:54.177+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.179183+0000) 2019-10-13T06:53:54.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:54.176+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.177603+0000) 2019-10-13T06:53:54.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:54.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109449 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:53:54.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:54.262+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:54.264373+0000) 2019-10-13T06:53:58.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:58.309+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:58.310425+0000) 2019-10-13T06:53:58.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:58.646+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:58.647569+0000) 2019-10-13T06:53:58.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:58.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:58.647574+0000) 2019-10-13T06:53:59.187 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:53:59.183+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.184443+0000) 2019-10-13T06:53:59.188 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:53:59.183+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.184947+0000) 2019-10-13T06:53:59.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:53:59.183+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.185134+0000) 2019-10-13T06:53:59.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:53:59.183+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.185282+0000) 2019-10-13T06:53:59.189 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:53:59.184+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.185578+0000) 2019-10-13T06:53:59.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:53:59.181+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.182693+0000) 2019-10-13T06:53:59.190 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:53:59.183+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.184195+0000) 2019-10-13T06:53:59.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:53:59.184+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:53:59.185690+0000) 2019-10-13T06:53:59.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:53:59.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109479 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:01.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:01.518+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:01.518563+0000) 2019-10-13T06:54:01.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:01.902+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:01.903143+0000) 2019-10-13T06:54:02.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:02.766+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:02.767033+0000) 2019-10-13T06:54:02.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:02.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:02.767880+0000) 2019-10-13T06:54:04.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:04.187+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.188117+0000) 2019-10-13T06:54:04.195 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:04.189+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.191333+0000) 2019-10-13T06:54:04.195 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:04.190+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.191407+0000) 2019-10-13T06:54:04.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:04.190+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.191570+0000) 2019-10-13T06:54:04.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:04.190+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.191782+0000) 2019-10-13T06:54:04.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:04.190+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.191992+0000) 2019-10-13T06:54:04.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:04.190+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.192243+0000) 2019-10-13T06:54:04.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:04.189+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.190505+0000) 2019-10-13T06:54:04.199 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:04.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:04.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:04.263+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:04.264549+0000) 2019-10-13T06:54:07.231 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:54:07.231 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:07.235 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:54:07.236 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:08.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:08.309+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:08.310648+0000) 2019-10-13T06:54:08.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:08.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:08.647680+0000) 2019-10-13T06:54:08.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:08.646+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:08.647707+0000) 2019-10-13T06:54:09.202 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:09.195+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.197194+0000) 2019-10-13T06:54:09.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:09.196+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.197556+0000) 2019-10-13T06:54:09.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:09.196+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.197587+0000) 2019-10-13T06:54:09.203 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:09.196+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.197640+0000) 2019-10-13T06:54:09.203 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:09.196+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.198134+0000) 2019-10-13T06:54:09.203 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:09.196+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.198378+0000) 2019-10-13T06:54:09.204 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:09.193+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.195464+0000) 2019-10-13T06:54:09.205 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:09.196+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:09.197828+0000) 2019-10-13T06:54:09.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:09.199+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109542 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:11.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:11.520+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:11.520945+0000) 2019-10-13T06:54:11.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:11.902+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:11.903355+0000) 2019-10-13T06:54:12.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:12.766+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:12.767217+0000) 2019-10-13T06:54:12.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:12.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:12.768040+0000) 2019-10-13T06:54:14.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:14.198+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.199851+0000) 2019-10-13T06:54:14.212 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:14.200+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.202410+0000) 2019-10-13T06:54:14.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:14.201+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.202481+0000) 2019-10-13T06:54:14.213 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:14.200+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.202216+0000) 2019-10-13T06:54:14.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:14.201+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.202508+0000) 2019-10-13T06:54:14.213 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:14.201+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.202777+0000) 2019-10-13T06:54:14.213 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:14.201+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.202939+0000) 2019-10-13T06:54:14.214 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:14.201+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.203230+0000) 2019-10-13T06:54:14.214 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:14.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:14.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:14.264+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:14.264714+0000) 2019-10-13T06:54:18.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:18.310+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:18.310850+0000) 2019-10-13T06:54:18.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:18.647+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:18.647849+0000) 2019-10-13T06:54:18.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:18.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:18.647829+0000) 2019-10-13T06:54:19.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:19.207+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.208233+0000) 2019-10-13T06:54:19.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:19.208+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.208618+0000) 2019-10-13T06:54:19.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:19.208+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.208759+0000) 2019-10-13T06:54:19.213 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:19.204+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.206128+0000) 2019-10-13T06:54:19.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:19.208+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.209098+0000) 2019-10-13T06:54:19.213 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:19.208+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.209160+0000) 2019-10-13T06:54:19.213 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:19.209+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.209768+0000) 2019-10-13T06:54:19.214 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:19.207+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:19.208340+0000) 2019-10-13T06:54:19.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:19.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:21.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:21.522+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:21.523212+0000) 2019-10-13T06:54:21.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:21.903+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:21.903573+0000) 2019-10-13T06:54:22.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:22.767+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:22.767341+0000) 2019-10-13T06:54:22.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:22.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:22.768187+0000) 2019-10-13T06:54:24.217 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:24.213+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.214213+0000) 2019-10-13T06:54:24.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:24.214+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.214666+0000) 2019-10-13T06:54:24.218 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:24.214+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.214757+0000) 2019-10-13T06:54:24.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:24.214+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.214953+0000) 2019-10-13T06:54:24.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:24.214+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.215366+0000) 2019-10-13T06:54:24.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:24.211+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.212505+0000) 2019-10-13T06:54:24.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:24.212+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.213606+0000) 2019-10-13T06:54:24.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:24.215+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.215875+0000) 2019-10-13T06:54:24.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:24.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:24.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:24.264+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:24.264863+0000) 2019-10-13T06:54:28.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:28.310+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:28.311051+0000) 2019-10-13T06:54:28.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:28.647+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:28.648047+0000) 2019-10-13T06:54:28.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:28.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:28.648049+0000) 2019-10-13T06:54:29.223 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:29.216+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.217185+0000) 2019-10-13T06:54:29.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:29.219+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.220293+0000) 2019-10-13T06:54:29.224 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:29.220+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.220632+0000) 2019-10-13T06:54:29.224 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:29.220+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.220502+0000) 2019-10-13T06:54:29.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:29.220+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.220978+0000) 2019-10-13T06:54:29.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:29.220+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.221127+0000) 2019-10-13T06:54:29.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:29.221+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.221514+0000) 2019-10-13T06:54:29.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:29.220+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:29.221330+0000) 2019-10-13T06:54:29.228 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:29.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:31.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:31.525+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:31.525570+0000) 2019-10-13T06:54:31.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:31.903+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:31.903755+0000) 2019-10-13T06:54:32.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:32.767+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:32.767506+0000) 2019-10-13T06:54:32.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:32.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:32.768338+0000) 2019-10-13T06:54:34.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:34.222+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.223615+0000) 2019-10-13T06:54:34.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:34.226+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.226969+0000) 2019-10-13T06:54:34.231 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:34.226+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.227105+0000) 2019-10-13T06:54:34.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:34.226+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.227363+0000) 2019-10-13T06:54:34.232 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:34.227+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.227490+0000) 2019-10-13T06:54:34.232 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:34.227+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.227676+0000) 2019-10-13T06:54:34.232 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:34.227+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.228011+0000) 2019-10-13T06:54:34.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:34.224+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.225586+0000) 2019-10-13T06:54:34.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:34.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:34.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:34.264+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:34.264977+0000) 2019-10-13T06:54:37.386 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:54:37.386 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:37.390 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:54:37.390 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:38.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:38.310+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:38.311191+0000) 2019-10-13T06:54:38.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:38.646+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:38.648238+0000) 2019-10-13T06:54:38.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:38.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:38.648244+0000) 2019-10-13T06:54:39.236 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:39.229+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.230214+0000) 2019-10-13T06:54:39.237 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:39.231+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.233299+0000) 2019-10-13T06:54:39.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:39.232+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.233696+0000) 2019-10-13T06:54:39.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:39.232+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.233904+0000) 2019-10-13T06:54:39.237 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:39.232+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.234080+0000) 2019-10-13T06:54:39.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:39.232+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.233763+0000) 2019-10-13T06:54:39.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:39.232+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.234310+0000) 2019-10-13T06:54:39.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:39.232+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:39.232903+0000) 2019-10-13T06:54:39.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:39.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:41.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:41.527+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:41.527850+0000) 2019-10-13T06:54:41.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:41.902+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:41.903883+0000) 2019-10-13T06:54:42.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:42.766+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:42.767688+0000) 2019-10-13T06:54:42.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:42.768+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:42.768526+0000) 2019-10-13T06:54:44.243 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:44.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.237066+0000) 2019-10-13T06:54:44.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:44.238+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.240003+0000) 2019-10-13T06:54:44.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:44.238+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.240191+0000) 2019-10-13T06:54:44.244 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:44.238+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.240380+0000) 2019-10-13T06:54:44.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:44.239+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.240598+0000) 2019-10-13T06:54:44.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:44.239+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.240748+0000) 2019-10-13T06:54:44.245 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:44.239+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.241173+0000) 2019-10-13T06:54:44.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:44.239+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.240669+0000) 2019-10-13T06:54:44.247 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:44.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:44.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:44.264+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:44.265111+0000) 2019-10-13T06:54:48.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:48.310+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:48.311374+0000) 2019-10-13T06:54:48.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:48.647+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:48.648394+0000) 2019-10-13T06:54:48.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:48.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:48.648387+0000) 2019-10-13T06:54:49.249 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:49.242+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.243433+0000) 2019-10-13T06:54:49.250 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:49.245+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.246557+0000) 2019-10-13T06:54:49.250 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:49.245+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.246829+0000) 2019-10-13T06:54:49.251 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:49.245+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.247004+0000) 2019-10-13T06:54:49.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:49.245+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.247191+0000) 2019-10-13T06:54:49.251 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:49.246+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.247499+0000) 2019-10-13T06:54:49.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:49.245+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.246973+0000) 2019-10-13T06:54:49.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:49.246+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:49.247109+0000) 2019-10-13T06:54:49.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:49.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:51.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:51.530+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:51.530249+0000) 2019-10-13T06:54:51.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:51.903+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:51.904036+0000) 2019-10-13T06:54:52.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:52.767+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:52.767871+0000) 2019-10-13T06:54:52.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:52.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:52.768719+0000) 2019-10-13T06:54:54.256 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:54.249+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.250067+0000) 2019-10-13T06:54:54.257 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:54.251+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.253152+0000) 2019-10-13T06:54:54.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:54.251+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.253289+0000) 2019-10-13T06:54:54.257 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:54.252+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.253455+0000) 2019-10-13T06:54:54.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:54.252+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.253655+0000) 2019-10-13T06:54:54.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:54.252+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.253706+0000) 2019-10-13T06:54:54.258 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:54.252+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.253851+0000) 2019-10-13T06:54:54.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:54.251+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.252370+0000) 2019-10-13T06:54:54.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:54.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:54:54.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:54.263+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:54.265234+0000) 2019-10-13T06:54:58.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:58.311+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:58.311603+0000) 2019-10-13T06:54:58.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:58.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:58.648529+0000) 2019-10-13T06:54:58.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:58.647+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:58.648543+0000) 2019-10-13T06:54:59.261 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:54:59.256+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.258045+0000) 2019-10-13T06:54:59.261 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:54:59.256+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.258149+0000) 2019-10-13T06:54:59.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:54:59.257+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.258497+0000) 2019-10-13T06:54:59.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:54:59.255+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.255830+0000) 2019-10-13T06:54:59.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:54:59.257+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.258940+0000) 2019-10-13T06:54:59.263 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:54:59.257+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.259178+0000) 2019-10-13T06:54:59.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:54:59.258+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.259896+0000) 2019-10-13T06:54:59.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:54:59.257+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:54:59.258648+0000) 2019-10-13T06:54:59.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:54:59.261+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:01.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:01.531+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:01.533505+0000) 2019-10-13T06:55:01.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:01.903+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:01.904166+0000) 2019-10-13T06:55:02.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:02.767+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:02.768054+0000) 2019-10-13T06:55:02.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:02.767+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:02.768926+0000) 2019-10-13T06:55:04.268 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:04.264+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.264561+0000) 2019-10-13T06:55:04.268 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:04.264+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.264659+0000) 2019-10-13T06:55:04.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:04.260+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.262269+0000) 2019-10-13T06:55:04.269 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:04.264+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.265060+0000) 2019-10-13T06:55:04.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:04.265+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.265542+0000) 2019-10-13T06:55:04.269 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:04.265+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.265949+0000) 2019-10-13T06:55:04.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:04.262+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.264254+0000) 2019-10-13T06:55:04.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:04.266+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:04.266547+0000) 2019-10-13T06:55:04.272 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:04.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:07.548 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:55:07.549 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:07.553 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:55:07.553 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:08.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:08.311+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:08.311773+0000) 2019-10-13T06:55:08.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:08.648+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:08.648749+0000) 2019-10-13T06:55:08.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:08.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:08.648750+0000) 2019-10-13T06:55:09.275 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:09.271+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.271608+0000) 2019-10-13T06:55:09.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:09.271+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.271781+0000) 2019-10-13T06:55:09.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:09.271+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.271942+0000) 2019-10-13T06:55:09.277 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:09.268+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.269962+0000) 2019-10-13T06:55:09.277 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:09.272+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.272447+0000) 2019-10-13T06:55:09.277 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:09.272+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.272546+0000) 2019-10-13T06:55:09.278 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:09.272+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.273191+0000) 2019-10-13T06:55:09.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:09.270+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:09.272259+0000) 2019-10-13T06:55:09.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:09.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109914 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:11.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:11.015+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:11.015629+0000) 2019-10-13T06:55:11.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:11.534+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:11.535773+0000) 2019-10-13T06:55:11.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:11.904+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:11.904347+0000) 2019-10-13T06:55:12.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:12.768+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:12.768240+0000) 2019-10-13T06:55:12.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:12.768+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:12.769107+0000) 2019-10-13T06:55:14.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:14.277+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.278041+0000) 2019-10-13T06:55:14.282 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:14.278+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.278478+0000) 2019-10-13T06:55:14.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:14.275+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.276324+0000) 2019-10-13T06:55:14.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:14.278+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.278886+0000) 2019-10-13T06:55:14.283 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:14.278+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.279267+0000) 2019-10-13T06:55:14.283 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:14.279+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.279569+0000) 2019-10-13T06:55:14.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:14.279+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.279641+0000) 2019-10-13T06:55:14.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:14.277+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:14.277933+0000) 2019-10-13T06:55:14.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:14.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:18.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:18.311+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:18.311966+0000) 2019-10-13T06:55:18.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:18.648+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:18.648942+0000) 2019-10-13T06:55:18.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:18.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:18.648942+0000) 2019-10-13T06:55:19.288 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:19.284+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.284556+0000) 2019-10-13T06:55:19.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:19.284+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.285095+0000) 2019-10-13T06:55:19.289 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:19.284+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.285191+0000) 2019-10-13T06:55:19.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:19.285+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.285488+0000) 2019-10-13T06:55:19.290 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:19.282+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.283310+0000) 2019-10-13T06:55:19.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:19.283+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.284330+0000) 2019-10-13T06:55:19.291 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:19.285+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.285716+0000) 2019-10-13T06:55:19.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:19.285+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:19.285782+0000) 2019-10-13T06:55:19.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:19.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 109975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:21.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:21.015+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:21.015815+0000) 2019-10-13T06:55:21.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:21.537+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:21.538284+0000) 2019-10-13T06:55:21.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:21.904+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:21.904546+0000) 2019-10-13T06:55:22.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:22.767+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:22.768450+0000) 2019-10-13T06:55:22.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:22.768+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:22.769291+0000) 2019-10-13T06:55:24.294 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:24.289+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.290491+0000) 2019-10-13T06:55:24.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:24.289+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.290409+0000) 2019-10-13T06:55:24.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:24.289+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.290768+0000) 2019-10-13T06:55:24.295 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:24.289+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.290944+0000) 2019-10-13T06:55:24.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:24.289+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.291052+0000) 2019-10-13T06:55:24.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:24.290+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.291557+0000) 2019-10-13T06:55:24.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:24.287+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.288124+0000) 2019-10-13T06:55:24.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:24.289+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:24.289993+0000) 2019-10-13T06:55:24.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:24.292+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:28.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:28.310+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:28.312120+0000) 2019-10-13T06:55:28.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:28.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:28.649111+0000) 2019-10-13T06:55:28.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:28.648+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:28.649141+0000) 2019-10-13T06:55:29.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:29.294+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.295908+0000) 2019-10-13T06:55:29.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:29.294+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.296273+0000) 2019-10-13T06:55:29.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:29.295+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.296469+0000) 2019-10-13T06:55:29.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:29.295+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.296451+0000) 2019-10-13T06:55:29.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:29.295+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.296606+0000) 2019-10-13T06:55:29.302 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:29.295+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.297196+0000) 2019-10-13T06:55:29.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:29.293+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.294245+0000) 2019-10-13T06:55:29.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:29.295+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:29.296056+0000) 2019-10-13T06:55:29.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:29.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:31.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:31.015+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:31.016004+0000) 2019-10-13T06:55:31.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:31.539+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:31.540638+0000) 2019-10-13T06:55:31.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:31.903+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:31.904746+0000) 2019-10-13T06:55:32.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:32.767+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:32.768632+0000) 2019-10-13T06:55:32.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:32.769+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:32.769460+0000) 2019-10-13T06:55:34.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:34.300+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.302196+0000) 2019-10-13T06:55:34.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:34.301+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.302498+0000) 2019-10-13T06:55:34.306 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:34.301+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.302678+0000) 2019-10-13T06:55:34.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:34.301+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.302982+0000) 2019-10-13T06:55:34.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:34.301+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.303064+0000) 2019-10-13T06:55:34.307 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:34.299+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.300594+0000) 2019-10-13T06:55:34.307 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:34.302+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.304020+0000) 2019-10-13T06:55:34.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:34.302+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:34.303663+0000) 2019-10-13T06:55:34.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:34.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:37.677 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:55:37.677 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:37.685 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:55:37.685 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:38.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:38.311+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:38.312296+0000) 2019-10-13T06:55:38.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:38.648+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:38.649327+0000) 2019-10-13T06:55:38.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:38.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:38.649319+0000) 2019-10-13T06:55:39.311 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:39.306+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.308298+0000) 2019-10-13T06:55:39.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:39.307+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.308613+0000) 2019-10-13T06:55:39.313 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:39.305+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.306601+0000) 2019-10-13T06:55:39.313 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:39.307+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.309153+0000) 2019-10-13T06:55:39.313 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:39.308+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.309439+0000) 2019-10-13T06:55:39.313 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:39.308+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.309603+0000) 2019-10-13T06:55:39.313 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:39.308+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.309719+0000) 2019-10-13T06:55:39.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:39.307+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:39.308017+0000) 2019-10-13T06:55:39.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:39.311+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:41.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:41.014+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:41.016185+0000) 2019-10-13T06:55:41.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:41.542+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:41.542970+0000) 2019-10-13T06:55:41.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:41.904+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:41.904920+0000) 2019-10-13T06:55:42.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:42.768+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:42.768834+0000) 2019-10-13T06:55:42.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:42.769+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:42.769660+0000) 2019-10-13T06:55:44.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:44.313+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.314705+0000) 2019-10-13T06:55:44.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:44.311+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.312582+0000) 2019-10-13T06:55:44.319 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:44.313+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.315156+0000) 2019-10-13T06:55:44.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:44.314+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.315473+0000) 2019-10-13T06:55:44.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:44.314+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.315760+0000) 2019-10-13T06:55:44.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:44.314+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.315829+0000) 2019-10-13T06:55:44.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:44.314+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.316117+0000) 2019-10-13T06:55:44.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:44.313+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:44.314519+0000) 2019-10-13T06:55:44.322 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:44.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110130 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:48.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:48.311+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:48.312475+0000) 2019-10-13T06:55:48.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:48.649+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:48.649518+0000) 2019-10-13T06:55:48.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:48.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:48.649511+0000) 2019-10-13T06:55:49.324 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:49.319+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.320884+0000) 2019-10-13T06:55:49.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:49.319+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.321037+0000) 2019-10-13T06:55:49.325 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:49.319+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.321162+0000) 2019-10-13T06:55:49.325 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:49.318+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.318743+0000) 2019-10-13T06:55:49.326 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:49.320+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.321489+0000) 2019-10-13T06:55:49.326 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:49.320+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.321885+0000) 2019-10-13T06:55:49.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:49.320+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.322102+0000) 2019-10-13T06:55:49.326 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:49.320+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:49.321159+0000) 2019-10-13T06:55:49.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:49.323+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:51.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:51.015+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:51.016324+0000) 2019-10-13T06:55:51.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:51.545+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:51.545311+0000) 2019-10-13T06:55:51.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:51.904+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:51.905128+0000) 2019-10-13T06:55:52.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:52.768+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:52.768959+0000) 2019-10-13T06:55:52.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:52.769+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:52.769837+0000) 2019-10-13T06:55:54.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:54.321+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.323386+0000) 2019-10-13T06:55:54.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:54.325+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.326238+0000) 2019-10-13T06:55:54.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:54.326+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.326539+0000) 2019-10-13T06:55:54.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:54.326+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.327118+0000) 2019-10-13T06:55:54.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:54.326+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.327318+0000) 2019-10-13T06:55:54.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:54.326+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.327195+0000) 2019-10-13T06:55:54.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:54.327+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.327447+0000) 2019-10-13T06:55:54.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:54.324+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:54.326131+0000) 2019-10-13T06:55:54.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:54.330+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110193 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:55:58.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:58.312+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:58.312631+0000) 2019-10-13T06:55:58.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:58.649+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:58.649731+0000) 2019-10-13T06:55:58.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:58.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:58.649730+0000) 2019-10-13T06:55:59.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:55:59.331+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.332382+0000) 2019-10-13T06:55:59.336 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:55:59.332+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.332538+0000) 2019-10-13T06:55:59.336 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:55:59.332+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.332822+0000) 2019-10-13T06:55:59.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:55:59.328+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.329959+0000) 2019-10-13T06:55:59.337 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:55:59.332+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.333341+0000) 2019-10-13T06:55:59.338 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:55:59.333+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.333525+0000) 2019-10-13T06:55:59.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:55:59.333+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.333772+0000) 2019-10-13T06:55:59.339 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:55:59.331+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:55:59.333267+0000) 2019-10-13T06:55:59.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:55:59.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:01.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:01.016+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:01.016515+0000) 2019-10-13T06:56:01.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:01.546+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:01.547635+0000) 2019-10-13T06:56:01.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:01.904+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:01.905322+0000) 2019-10-13T06:56:02.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:02.768+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:02.769125+0000) 2019-10-13T06:56:02.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:02.769+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:02.770017+0000) 2019-10-13T06:56:04.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:04.338+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.338943+0000) 2019-10-13T06:56:04.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:04.338+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.339105+0000) 2019-10-13T06:56:04.343 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:04.338+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.339237+0000) 2019-10-13T06:56:04.343 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:04.338+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.339390+0000) 2019-10-13T06:56:04.343 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:04.335+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.336838+0000) 2019-10-13T06:56:04.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:04.339+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.340341+0000) 2019-10-13T06:56:04.344 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:04.340+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.340910+0000) 2019-10-13T06:56:04.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:04.338+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:04.340055+0000) 2019-10-13T06:56:04.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:04.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:07.764 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:56:07.765 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:07.769 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:56:07.769 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:08.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:08.311+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:08.312787+0000) 2019-10-13T06:56:08.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:08.648+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:08.649934+0000) 2019-10-13T06:56:08.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:08.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:08.649938+0000) 2019-10-13T06:56:09.350 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:09.345+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.346465+0000) 2019-10-13T06:56:09.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:09.345+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.346808+0000) 2019-10-13T06:56:09.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:09.344+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.344788+0000) 2019-10-13T06:56:09.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:09.345+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.347048+0000) 2019-10-13T06:56:09.351 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:09.345+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.347301+0000) 2019-10-13T06:56:09.351 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:09.346+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.347649+0000) 2019-10-13T06:56:09.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:09.346+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.347931+0000) 2019-10-13T06:56:09.352 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:09.345+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:09.346078+0000) 2019-10-13T06:56:09.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:09.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:11.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:11.015+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:11.016716+0000) 2019-10-13T06:56:11.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:11.549+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:11.550127+0000) 2019-10-13T06:56:11.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:11.904+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:11.905510+0000) 2019-10-13T06:56:12.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:12.768+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:12.769309+0000) 2019-10-13T06:56:12.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:12.769+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:12.770186+0000) 2019-10-13T06:56:14.356 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:14.351+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.353059+0000) 2019-10-13T06:56:14.357 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:14.352+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.353418+0000) 2019-10-13T06:56:14.357 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:14.352+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.353554+0000) 2019-10-13T06:56:14.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:14.352+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.353716+0000) 2019-10-13T06:56:14.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:14.352+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.354065+0000) 2019-10-13T06:56:14.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:14.350+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.351189+0000) 2019-10-13T06:56:14.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:14.353+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.354439+0000) 2019-10-13T06:56:14.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:14.352+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:14.353643+0000) 2019-10-13T06:56:14.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:14.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:18.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:18.312+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:18.313018+0000) 2019-10-13T06:56:18.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:18.649+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:18.650114+0000) 2019-10-13T06:56:18.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:18.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:18.650113+0000) 2019-10-13T06:56:19.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:19.358+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.359444+0000) 2019-10-13T06:56:19.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:19.358+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.359709+0000) 2019-10-13T06:56:19.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:19.358+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.360033+0000) 2019-10-13T06:56:19.364 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:19.358+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.359995+0000) 2019-10-13T06:56:19.364 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:19.359+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.360800+0000) 2019-10-13T06:56:19.365 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:19.357+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.357787+0000) 2019-10-13T06:56:19.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:19.359+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.361226+0000) 2019-10-13T06:56:19.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:19.359+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:19.359961+0000) 2019-10-13T06:56:19.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:19.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:21.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:21.016+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:21.016878+0000) 2019-10-13T06:56:21.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:21.552+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:21.552546+0000) 2019-10-13T06:56:21.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:21.904+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:21.905682+0000) 2019-10-13T06:56:22.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:22.768+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:22.769486+0000) 2019-10-13T06:56:22.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:22.769+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:22.770387+0000) 2019-10-13T06:56:24.371 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:24.362+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.363703+0000) 2019-10-13T06:56:24.372 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:24.364+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.365654+0000) 2019-10-13T06:56:24.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:24.364+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.365858+0000) 2019-10-13T06:56:24.372 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:24.364+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.366046+0000) 2019-10-13T06:56:24.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:24.365+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.367194+0000) 2019-10-13T06:56:24.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:24.365+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.367132+0000) 2019-10-13T06:56:24.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:24.366+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.367506+0000) 2019-10-13T06:56:24.373 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:24.365+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:24.366097+0000) 2019-10-13T06:56:24.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:24.369+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110378 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:28.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:28.312+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:28.313197+0000) 2019-10-13T06:56:28.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:28.649+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:28.650334+0000) 2019-10-13T06:56:28.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:28.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:28.650334+0000) 2019-10-13T06:56:29.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:29.368+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.369555+0000) 2019-10-13T06:56:29.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:29.371+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.372522+0000) 2019-10-13T06:56:29.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:29.371+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.372782+0000) 2019-10-13T06:56:29.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:29.371+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.372993+0000) 2019-10-13T06:56:29.377 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:29.371+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.373109+0000) 2019-10-13T06:56:29.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:29.371+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.373330+0000) 2019-10-13T06:56:29.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:29.372+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.373458+0000) 2019-10-13T06:56:29.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:29.371+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:29.372152+0000) 2019-10-13T06:56:29.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:29.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:31.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:31.016+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:31.017036+0000) 2019-10-13T06:56:31.561 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:31.554+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:31.554859+0000) 2019-10-13T06:56:31.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:31.905+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:31.905872+0000) 2019-10-13T06:56:32.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:32.769+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:32.769642+0000) 2019-10-13T06:56:32.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:32.770+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:32.770570+0000) 2019-10-13T06:56:34.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:34.377+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.378671+0000) 2019-10-13T06:56:34.383 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:34.375+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.376237+0000) 2019-10-13T06:56:34.383 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:34.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.379054+0000) 2019-10-13T06:56:34.383 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:34.377+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.379325+0000) 2019-10-13T06:56:34.383 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:34.377+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.379184+0000) 2019-10-13T06:56:34.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:34.378+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.379683+0000) 2019-10-13T06:56:34.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:34.378+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.379772+0000) 2019-10-13T06:56:34.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:34.378+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:34.378916+0000) 2019-10-13T06:56:34.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:34.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:37.944 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:56:37.944 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:37.948 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:56:37.948 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:38.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:38.313+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:38.313389+0000) 2019-10-13T06:56:38.654 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:38.650+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:38.650504+0000) 2019-10-13T06:56:38.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:38.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:38.650503+0000) 2019-10-13T06:56:39.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:39.383+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.383934+0000) 2019-10-13T06:56:39.388 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:39.380+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.381521+0000) 2019-10-13T06:56:39.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:39.383+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.384347+0000) 2019-10-13T06:56:39.388 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:39.384+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.384573+0000) 2019-10-13T06:56:39.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:39.384+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.384812+0000) 2019-10-13T06:56:39.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:39.384+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.385092+0000) 2019-10-13T06:56:39.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:39.385+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.385762+0000) 2019-10-13T06:56:39.390 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:39.383+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:39.384353+0000) 2019-10-13T06:56:39.391 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:39.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110471 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:41.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:41.016+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:41.017215+0000) 2019-10-13T06:56:41.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:41.557+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:41.557305+0000) 2019-10-13T06:56:41.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:41.905+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:41.906059+0000) 2019-10-13T06:56:42.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:42.769+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:42.769837+0000) 2019-10-13T06:56:42.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:42.770+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:42.770750+0000) 2019-10-13T06:56:44.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:44.389+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.390055+0000) 2019-10-13T06:56:44.393 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:44.389+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.390199+0000) 2019-10-13T06:56:44.394 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:44.390+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.390458+0000) 2019-10-13T06:56:44.395 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:44.386+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.387666+0000) 2019-10-13T06:56:44.395 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:44.390+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.390695+0000) 2019-10-13T06:56:44.395 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:44.390+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.391112+0000) 2019-10-13T06:56:44.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:44.390+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.391276+0000) 2019-10-13T06:56:44.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:44.389+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:44.390236+0000) 2019-10-13T06:56:44.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:44.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:48.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:48.313+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:48.313602+0000) 2019-10-13T06:56:48.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:48.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:48.650700+0000) 2019-10-13T06:56:48.654 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:48.650+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:48.650721+0000) 2019-10-13T06:56:49.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:49.395+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.395706+0000) 2019-10-13T06:56:49.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:49.395+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.396030+0000) 2019-10-13T06:56:49.400 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:49.395+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.396262+0000) 2019-10-13T06:56:49.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:49.395+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.396373+0000) 2019-10-13T06:56:49.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:49.396+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.396550+0000) 2019-10-13T06:56:49.401 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:49.392+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.393974+0000) 2019-10-13T06:56:49.402 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:49.393+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.395600+0000) 2019-10-13T06:56:49.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:49.396+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:49.396783+0000) 2019-10-13T06:56:49.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:49.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110534 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:51.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:51.017+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:51.017329+0000) 2019-10-13T06:56:51.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:51.558+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:51.559777+0000) 2019-10-13T06:56:51.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:51.905+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:51.906247+0000) 2019-10-13T06:56:52.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:52.769+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:52.770024+0000) 2019-10-13T06:56:52.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:52.769+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:52.770920+0000) 2019-10-13T06:56:54.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:54.401+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.402406+0000) 2019-10-13T06:56:54.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:54.401+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.402392+0000) 2019-10-13T06:56:54.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:54.401+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.402572+0000) 2019-10-13T06:56:54.407 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:54.401+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.402746+0000) 2019-10-13T06:56:54.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:54.401+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.403195+0000) 2019-10-13T06:56:54.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:54.401+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.403323+0000) 2019-10-13T06:56:54.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:54.398+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.400307+0000) 2019-10-13T06:56:54.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:54.399+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:54.401574+0000) 2019-10-13T06:56:54.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:54.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110565 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:56:58.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:58.312+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:58.313756+0000) 2019-10-13T06:56:58.654 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:58.649+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:58.650873+0000) 2019-10-13T06:56:58.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:58.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:58.650871+0000) 2019-10-13T06:56:59.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:56:59.407+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.408626+0000) 2019-10-13T06:56:59.412 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:56:59.407+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.408926+0000) 2019-10-13T06:56:59.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:56:59.404+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.406722+0000) 2019-10-13T06:56:59.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:56:59.407+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.409102+0000) 2019-10-13T06:56:59.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:56:59.407+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.409263+0000) 2019-10-13T06:56:59.414 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:56:59.408+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.409751+0000) 2019-10-13T06:56:59.414 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:56:59.408+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.409851+0000) 2019-10-13T06:56:59.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:56:59.406+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:56:59.407766+0000) 2019-10-13T06:56:59.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:56:59.411+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:01.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:01.016+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:01.017477+0000) 2019-10-13T06:57:01.568 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:01.561+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:01.562195+0000) 2019-10-13T06:57:01.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:01.905+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:01.906436+0000) 2019-10-13T06:57:02.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:02.769+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:02.770215+0000) 2019-10-13T06:57:02.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:02.770+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:02.771103+0000) 2019-10-13T06:57:04.418 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:04.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.412356+0000) 2019-10-13T06:57:04.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:04.413+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.414872+0000) 2019-10-13T06:57:04.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:04.413+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.415230+0000) 2019-10-13T06:57:04.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:04.414+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.415555+0000) 2019-10-13T06:57:04.419 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:04.414+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.415643+0000) 2019-10-13T06:57:04.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:04.414+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.415953+0000) 2019-10-13T06:57:04.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:04.414+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.416121+0000) 2019-10-13T06:57:04.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:04.414+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:04.415321+0000) 2019-10-13T06:57:04.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:04.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110626 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:08.108 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:57:08.108 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:08.112 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:57:08.112 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:08.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:08.313+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:08.313900+0000) 2019-10-13T06:57:08.654 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:08.650+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:08.651065+0000) 2019-10-13T06:57:08.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:08.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:08.651061+0000) 2019-10-13T06:57:09.425 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:09.420+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.421449+0000) 2019-10-13T06:57:09.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:09.420+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.421664+0000) 2019-10-13T06:57:09.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:09.418+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.419461+0000) 2019-10-13T06:57:09.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:09.420+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.422014+0000) 2019-10-13T06:57:09.426 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:09.420+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.422328+0000) 2019-10-13T06:57:09.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:09.421+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.422487+0000) 2019-10-13T06:57:09.427 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:09.421+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.422861+0000) 2019-10-13T06:57:09.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:09.420+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:09.420782+0000) 2019-10-13T06:57:09.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:09.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:11.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:11.016+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:11.017633+0000) 2019-10-13T06:57:11.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:11.563+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:11.564551+0000) 2019-10-13T06:57:11.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:11.905+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:11.906648+0000) 2019-10-13T06:57:12.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:12.769+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:12.770419+0000) 2019-10-13T06:57:12.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:12.770+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:12.771264+0000) 2019-10-13T06:57:14.430 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:14.426+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.427522+0000) 2019-10-13T06:57:14.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:14.424+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.424934+0000) 2019-10-13T06:57:14.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:14.426+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.427922+0000) 2019-10-13T06:57:14.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:14.426+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.427975+0000) 2019-10-13T06:57:14.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:14.426+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.428179+0000) 2019-10-13T06:57:14.432 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:14.426+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.428341+0000) 2019-10-13T06:57:14.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:14.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.428503+0000) 2019-10-13T06:57:14.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:14.426+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:14.427227+0000) 2019-10-13T06:57:14.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:14.430+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:18.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:18.312+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:18.314102+0000) 2019-10-13T06:57:18.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:18.650+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:18.651287+0000) 2019-10-13T06:57:18.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:18.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:18.651286+0000) 2019-10-13T06:57:19.437 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:19.432+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.434060+0000) 2019-10-13T06:57:19.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:19.432+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.434266+0000) 2019-10-13T06:57:19.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:19.432+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.434350+0000) 2019-10-13T06:57:19.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:19.433+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.434550+0000) 2019-10-13T06:57:19.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:19.431+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.432147+0000) 2019-10-13T06:57:19.439 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:19.433+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.434569+0000) 2019-10-13T06:57:19.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:19.433+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.435347+0000) 2019-10-13T06:57:19.440 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:19.433+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:19.434540+0000) 2019-10-13T06:57:19.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:19.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110720 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:21.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:21.017+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:21.017810+0000) 2019-10-13T06:57:21.573 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:21.566+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:21.566818+0000) 2019-10-13T06:57:21.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:21.906+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:21.906824+0000) 2019-10-13T06:57:22.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:22.770+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:22.770605+0000) 2019-10-13T06:57:22.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:22.771+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:22.771434+0000) 2019-10-13T06:57:24.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:24.436+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.437253+0000) 2019-10-13T06:57:24.443 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:24.440+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.440466+0000) 2019-10-13T06:57:24.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:24.440+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.440604+0000) 2019-10-13T06:57:24.444 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:24.440+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.440796+0000) 2019-10-13T06:57:24.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:24.440+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.441034+0000) 2019-10-13T06:57:24.445 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:24.440+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.441086+0000) 2019-10-13T06:57:24.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:24.441+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.441579+0000) 2019-10-13T06:57:24.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:24.439+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:24.440337+0000) 2019-10-13T06:57:24.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:24.444+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:28.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:28.313+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:28.314258+0000) 2019-10-13T06:57:28.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:28.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:28.651427+0000) 2019-10-13T06:57:28.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:28.651+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:28.651435+0000) 2019-10-13T06:57:29.449 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:29.442+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.443710+0000) 2019-10-13T06:57:29.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:29.446+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.446944+0000) 2019-10-13T06:57:29.452 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:29.446+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.446791+0000) 2019-10-13T06:57:29.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:29.446+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.447168+0000) 2019-10-13T06:57:29.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:29.446+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.447328+0000) 2019-10-13T06:57:29.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:29.447+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.447448+0000) 2019-10-13T06:57:29.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:29.447+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.447759+0000) 2019-10-13T06:57:29.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:29.445+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:29.445820+0000) 2019-10-13T06:57:29.455 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:29.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:31.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:31.017+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:31.018017+0000) 2019-10-13T06:57:31.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:31.569+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:31.569296+0000) 2019-10-13T06:57:31.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:31.906+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:31.907012+0000) 2019-10-13T06:57:32.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:32.770+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:32.770762+0000) 2019-10-13T06:57:32.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:32.771+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:32.771638+0000) 2019-10-13T06:57:34.455 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:34.448+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.449628+0000) 2019-10-13T06:57:34.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:34.452+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.452503+0000) 2019-10-13T06:57:34.456 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:34.452+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.452817+0000) 2019-10-13T06:57:34.456 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:34.452+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.453120+0000) 2019-10-13T06:57:34.457 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:34.452+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.453222+0000) 2019-10-13T06:57:34.457 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:34.453+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.453638+0000) 2019-10-13T06:57:34.457 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:34.453+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.453393+0000) 2019-10-13T06:57:34.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:34.451+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:34.452232+0000) 2019-10-13T06:57:34.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:34.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:38.270 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:57:38.270 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:38.274 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:57:38.274 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:38.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:38.314+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:38.314394+0000) 2019-10-13T06:57:38.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:38.651+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:38.651596+0000) 2019-10-13T06:57:38.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:38.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:38.651591+0000) 2019-10-13T06:57:39.461 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:39.457+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.458093+0000) 2019-10-13T06:57:39.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:39.458+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.458480+0000) 2019-10-13T06:57:39.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:39.458+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.458578+0000) 2019-10-13T06:57:39.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:39.454+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.455623+0000) 2019-10-13T06:57:39.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:39.458+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.459238+0000) 2019-10-13T06:57:39.463 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:39.458+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.459352+0000) 2019-10-13T06:57:39.463 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:39.459+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.459515+0000) 2019-10-13T06:57:39.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:39.458+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:39.459340+0000) 2019-10-13T06:57:39.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:39.462+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:41.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:41.017+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:41.018176+0000) 2019-10-13T06:57:41.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:41.571+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:41.572384+0000) 2019-10-13T06:57:41.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:41.905+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:41.907213+0000) 2019-10-13T06:57:42.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:42.769+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:42.770967+0000) 2019-10-13T06:57:42.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:42.770+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:42.771818+0000) 2019-10-13T06:57:44.467 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:44.462+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.463768+0000) 2019-10-13T06:57:44.467 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:44.462+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.463893+0000) 2019-10-13T06:57:44.468 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:44.460+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.462017+0000) 2019-10-13T06:57:44.468 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:44.462+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.464326+0000) 2019-10-13T06:57:44.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:44.463+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.464487+0000) 2019-10-13T06:57:44.469 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:44.463+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.464562+0000) 2019-10-13T06:57:44.469 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:44.463+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.464863+0000) 2019-10-13T06:57:44.469 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:44.461+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:44.463467+0000) 2019-10-13T06:57:44.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:44.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:48.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:48.313+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:48.314593+0000) 2019-10-13T06:57:48.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:48.650+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:48.651801+0000) 2019-10-13T06:57:48.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:48.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:48.651817+0000) 2019-10-13T06:57:49.472 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:49.467+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.469147+0000) 2019-10-13T06:57:49.473 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:49.468+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.469654+0000) 2019-10-13T06:57:49.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:49.465+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.467091+0000) 2019-10-13T06:57:49.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:49.468+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.470049+0000) 2019-10-13T06:57:49.474 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:49.468+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.470175+0000) 2019-10-13T06:57:49.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:49.468+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.469899+0000) 2019-10-13T06:57:49.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:49.469+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.470550+0000) 2019-10-13T06:57:49.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:49.467+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:49.469487+0000) 2019-10-13T06:57:49.477 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:49.472+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110910 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:51.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:51.017+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:51.018365+0000) 2019-10-13T06:57:51.580 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:51.573+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:51.574772+0000) 2019-10-13T06:57:51.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:51.906+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:51.907390+0000) 2019-10-13T06:57:52.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:52.770+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:52.771145+0000) 2019-10-13T06:57:52.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:52.771+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:52.772000+0000) 2019-10-13T06:57:54.479 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:54.474+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.476172+0000) 2019-10-13T06:57:54.480 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:54.472+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.474015+0000) 2019-10-13T06:57:54.480 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:54.475+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.476615+0000) 2019-10-13T06:57:54.481 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:54.475+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.476869+0000) 2019-10-13T06:57:54.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:54.475+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.477141+0000) 2019-10-13T06:57:54.481 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:54.475+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.477349+0000) 2019-10-13T06:57:54.481 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:54.476+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.477484+0000) 2019-10-13T06:57:54.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:54.474+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:54.476512+0000) 2019-10-13T06:57:54.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:54.479+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110942 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:57:58.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:58.314+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:58.314821+0000) 2019-10-13T06:57:58.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:58.651+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:58.651997+0000) 2019-10-13T06:57:58.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:58.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:58.652002+0000) 2019-10-13T06:57:59.485 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:57:59.481+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.482417+0000) 2019-10-13T06:57:59.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:57:59.479+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.480320+0000) 2019-10-13T06:57:59.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:57:59.481+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.483148+0000) 2019-10-13T06:57:59.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:57:59.482+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.483429+0000) 2019-10-13T06:57:59.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:57:59.481+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.483288+0000) 2019-10-13T06:57:59.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:57:59.482+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.483753+0000) 2019-10-13T06:57:59.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:57:59.482+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.483678+0000) 2019-10-13T06:57:59.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:57:59.481+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:57:59.482268+0000) 2019-10-13T06:57:59.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:57:59.485+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 110972 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:01.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:01.018+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:01.018599+0000) 2019-10-13T06:58:01.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:01.576+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:01.577192+0000) 2019-10-13T06:58:01.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:01.906+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:01.907590+0000) 2019-10-13T06:58:02.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:02.770+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:02.771327+0000) 2019-10-13T06:58:02.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:02.771+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:02.772152+0000) 2019-10-13T06:58:04.493 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:04.487+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.488618+0000) 2019-10-13T06:58:04.494 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:04.487+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.488825+0000) 2019-10-13T06:58:04.494 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:04.487+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.489083+0000) 2019-10-13T06:58:04.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:04.488+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.489500+0000) 2019-10-13T06:58:04.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:04.488+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.489848+0000) 2019-10-13T06:58:04.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:04.488+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.490154+0000) 2019-10-13T06:58:04.495 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:04.486+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.487101+0000) 2019-10-13T06:58:04.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:04.488+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:04.489440+0000) 2019-10-13T06:58:04.496 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:04.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:08.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:08.314+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:08.315011+0000) 2019-10-13T06:58:08.433 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:58:08.433 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:08.437 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:58:08.437 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:08.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:08.651+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:08.652168+0000) 2019-10-13T06:58:08.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:08.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:08.652185+0000) 2019-10-13T06:58:09.500 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:09.492+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.493051+0000) 2019-10-13T06:58:09.500 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:09.494+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.495310+0000) 2019-10-13T06:58:09.500 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:09.495+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.495816+0000) 2019-10-13T06:58:09.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:09.495+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.496053+0000) 2019-10-13T06:58:09.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:09.495+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.496192+0000) 2019-10-13T06:58:09.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:09.496+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.496534+0000) 2019-10-13T06:58:09.501 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:09.496+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.496744+0000) 2019-10-13T06:58:09.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:09.495+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:09.496418+0000) 2019-10-13T06:58:09.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:09.500+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:11.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:11.018+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:11.018821+0000) 2019-10-13T06:58:11.585 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:11.578+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:11.579603+0000) 2019-10-13T06:58:11.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:11.907+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:11.907779+0000) 2019-10-13T06:58:12.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:12.771+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:12.771496+0000) 2019-10-13T06:58:12.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:12.771+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:12.772334+0000) 2019-10-13T06:58:14.506 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:14.502+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.502407+0000) 2019-10-13T06:58:14.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:14.502+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.502525+0000) 2019-10-13T06:58:14.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:14.502+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.502822+0000) 2019-10-13T06:58:14.506 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:14.502+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.502836+0000) 2019-10-13T06:58:14.507 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:14.500+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.500998+0000) 2019-10-13T06:58:14.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:14.502+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.503234+0000) 2019-10-13T06:58:14.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:14.502+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.503312+0000) 2019-10-13T06:58:14.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:14.502+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:14.503114+0000) 2019-10-13T06:58:14.510 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:14.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:18.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:18.314+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:18.315208+0000) 2019-10-13T06:58:18.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:18.652+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:18.652341+0000) 2019-10-13T06:58:18.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:18.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:18.652321+0000) 2019-10-13T06:58:19.512 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:19.508+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.508988+0000) 2019-10-13T06:58:19.512 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:19.508+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.509120+0000) 2019-10-13T06:58:19.513 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:19.508+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.509273+0000) 2019-10-13T06:58:19.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:19.509+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.509575+0000) 2019-10-13T06:58:19.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:19.506+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.507022+0000) 2019-10-13T06:58:19.514 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:19.509+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.509757+0000) 2019-10-13T06:58:19.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:19.509+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.509969+0000) 2019-10-13T06:58:19.514 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:19.508+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:19.508839+0000) 2019-10-13T06:58:19.516 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:19.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111098 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:21.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:21.018+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:21.018986+0000) 2019-10-13T06:58:21.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:21.581+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:21.582037+0000) 2019-10-13T06:58:21.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:21.907+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:21.907970+0000) 2019-10-13T06:58:22.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:22.771+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:22.771647+0000) 2019-10-13T06:58:22.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:22.772+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:22.772539+0000) 2019-10-13T06:58:24.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:24.514+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.514589+0000) 2019-10-13T06:58:24.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:24.514+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.514803+0000) 2019-10-13T06:58:24.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:24.514+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.515077+0000) 2019-10-13T06:58:24.520 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:24.514+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.515217+0000) 2019-10-13T06:58:24.520 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:24.515+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.515680+0000) 2019-10-13T06:58:24.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:24.515+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.515762+0000) 2019-10-13T06:58:24.521 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:24.512+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.512979+0000) 2019-10-13T06:58:24.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:24.514+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:24.514800+0000) 2019-10-13T06:58:24.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:24.517+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111130 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:28.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:28.314+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:28.315413+0000) 2019-10-13T06:58:28.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:28.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:28.652494+0000) 2019-10-13T06:58:28.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:28.651+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:28.652522+0000) 2019-10-13T06:58:29.522 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:29.516+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.516957+0000) 2019-10-13T06:58:29.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:29.518+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.519896+0000) 2019-10-13T06:58:29.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:29.518+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.520118+0000) 2019-10-13T06:58:29.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:29.518+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.520152+0000) 2019-10-13T06:58:29.524 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:29.518+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.520300+0000) 2019-10-13T06:58:29.524 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:29.518+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.520360+0000) 2019-10-13T06:58:29.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:29.519+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.520548+0000) 2019-10-13T06:58:29.526 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:29.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:29.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:29.520+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:29.521067+0000) 2019-10-13T06:58:31.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:31.018+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:31.019128+0000) 2019-10-13T06:58:31.591 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:31.584+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:31.584363+0000) 2019-10-13T06:58:31.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:31.907+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:31.908143+0000) 2019-10-13T06:58:32.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:32.770+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:32.771852+0000) 2019-10-13T06:58:32.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:32.772+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:32.772711+0000) 2019-10-13T06:58:34.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:34.520+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.522456+0000) 2019-10-13T06:58:34.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:34.523+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.525312+0000) 2019-10-13T06:58:34.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:34.524+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.525590+0000) 2019-10-13T06:58:34.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:34.524+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.526188+0000) 2019-10-13T06:58:34.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:34.525+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.526601+0000) 2019-10-13T06:58:34.530 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:34.525+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.527131+0000) 2019-10-13T06:58:34.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:34.525+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.527283+0000) 2019-10-13T06:58:34.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:34.524+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:34.525968+0000) 2019-10-13T06:58:34.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:34.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111194 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:38.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:38.314+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:38.315591+0000) 2019-10-13T06:58:38.537 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:58:38.537 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:38.541 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:58:38.541 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:38.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:38.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:38.652624+0000) 2019-10-13T06:58:38.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:38.651+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:38.652733+0000) 2019-10-13T06:58:39.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:39.530+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.531828+0000) 2019-10-13T06:58:39.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:39.530+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.532009+0000) 2019-10-13T06:58:39.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:39.530+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.532118+0000) 2019-10-13T06:58:39.536 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:39.527+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.529687+0000) 2019-10-13T06:58:39.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:39.531+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.532506+0000) 2019-10-13T06:58:39.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:39.531+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.532392+0000) 2019-10-13T06:58:39.537 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:39.531+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.533055+0000) 2019-10-13T06:58:39.538 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:39.530+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:39.532302+0000) 2019-10-13T06:58:39.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:39.535+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:41.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:41.018+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:41.019301+0000) 2019-10-13T06:58:41.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:41.585+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:41.586702+0000) 2019-10-13T06:58:41.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:41.907+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:41.908317+0000) 2019-10-13T06:58:42.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:42.771+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:42.772006+0000) 2019-10-13T06:58:42.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:42.772+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:42.772924+0000) 2019-10-13T06:58:44.541 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:44.536+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.538283+0000) 2019-10-13T06:58:44.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:44.537+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.538593+0000) 2019-10-13T06:58:44.543 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:44.534+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.536499+0000) 2019-10-13T06:58:44.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:44.537+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.538692+0000) 2019-10-13T06:58:44.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:44.537+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.539227+0000) 2019-10-13T06:58:44.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:44.538+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.539549+0000) 2019-10-13T06:58:44.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:44.538+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.539801+0000) 2019-10-13T06:58:44.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:44.536+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:44.538003+0000) 2019-10-13T06:58:44.546 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:44.541+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111256 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:48.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:48.314+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:48.315787+0000) 2019-10-13T06:58:48.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:48.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:48.652754+0000) 2019-10-13T06:58:48.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:48.652+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:48.652839+0000) 2019-10-13T06:58:49.549 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:49.541+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.542529+0000) 2019-10-13T06:58:49.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:49.542+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.544301+0000) 2019-10-13T06:58:49.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:49.542+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.544378+0000) 2019-10-13T06:58:49.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:49.543+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.544942+0000) 2019-10-13T06:58:49.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:49.543+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.545171+0000) 2019-10-13T06:58:49.551 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:49.543+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.545320+0000) 2019-10-13T06:58:49.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:49.544+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.545891+0000) 2019-10-13T06:58:49.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:49.544+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:49.544962+0000) 2019-10-13T06:58:49.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:49.547+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:51.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:51.018+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:51.019460+0000) 2019-10-13T06:58:51.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:51.588+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:51.589056+0000) 2019-10-13T06:58:51.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:51.908+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:51.908480+0000) 2019-10-13T06:58:52.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:52.771+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:52.772180+0000) 2019-10-13T06:58:52.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:52.772+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:52.773113+0000) 2019-10-13T06:58:54.553 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:54.546+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.547349+0000) 2019-10-13T06:58:54.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:54.549+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.550239+0000) 2019-10-13T06:58:54.554 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:54.550+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.550501+0000) 2019-10-13T06:58:54.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:54.550+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.551048+0000) 2019-10-13T06:58:54.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:54.551+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.551462+0000) 2019-10-13T06:58:54.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:54.551+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.551580+0000) 2019-10-13T06:58:54.555 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:54.551+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.551780+0000) 2019-10-13T06:58:54.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:54.549+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:54.550691+0000) 2019-10-13T06:58:54.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:54.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111320 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:58:58.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:58.315+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:58.315932+0000) 2019-10-13T06:58:58.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:58.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:58.652926+0000) 2019-10-13T06:58:58.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:58.652+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:58.652983+0000) 2019-10-13T06:58:59.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:58:59.553+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.554114+0000) 2019-10-13T06:58:59.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:58:59.556+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.557632+0000) 2019-10-13T06:58:59.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:58:59.556+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.556954+0000) 2019-10-13T06:58:59.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:58:59.557+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.557431+0000) 2019-10-13T06:58:59.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:58:59.557+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.557642+0000) 2019-10-13T06:58:59.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:58:59.557+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.557798+0000) 2019-10-13T06:58:59.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:58:59.557+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.557844+0000) 2019-10-13T06:58:59.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:58:59.558+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:58:59.558393+0000) 2019-10-13T06:58:59.802 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:58:59.561+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111350 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:01.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:01.019+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:01.019625+0000) 2019-10-13T06:59:01.597 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:01.590+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:01.591388+0000) 2019-10-13T06:59:01.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:01.908+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:01.908666+0000) 2019-10-13T06:59:02.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:02.772+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:02.772345+0000) 2019-10-13T06:59:02.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:02.772+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:02.773255+0000) 2019-10-13T06:59:04.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:04.562+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.563091+0000) 2019-10-13T06:59:04.568 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:04.562+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.563298+0000) 2019-10-13T06:59:04.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:04.563+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.563738+0000) 2019-10-13T06:59:04.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:04.563+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.563866+0000) 2019-10-13T06:59:04.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:04.563+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.564152+0000) 2019-10-13T06:59:04.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:04.563+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.564279+0000) 2019-10-13T06:59:04.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:04.560+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.560839+0000) 2019-10-13T06:59:04.571 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:04.562+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:04.563157+0000) 2019-10-13T06:59:04.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:04.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:08.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:08.315+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:08.316114+0000) 2019-10-13T06:59:08.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:08.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:08.653106+0000) 2019-10-13T06:59:08.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:08.652+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:08.653135+0000) 2019-10-13T06:59:08.682 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:59:08.682 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:08.686 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:59:08.686 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:09.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:09.569+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.570177+0000) 2019-10-13T06:59:09.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:09.569+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.570368+0000) 2019-10-13T06:59:09.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:09.570+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.570556+0000) 2019-10-13T06:59:09.575 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:09.570+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.570809+0000) 2019-10-13T06:59:09.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:09.570+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.571277+0000) 2019-10-13T06:59:09.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:09.571+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.571647+0000) 2019-10-13T06:59:09.576 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:09.567+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.568537+0000) 2019-10-13T06:59:09.577 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:09.569+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:09.570543+0000) 2019-10-13T06:59:09.579 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:09.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:11.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:11.019+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:11.019809+0000) 2019-10-13T06:59:11.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:11.593+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:11.593952+0000) 2019-10-13T06:59:11.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:11.908+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:11.908797+0000) 2019-10-13T06:59:12.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:12.771+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:12.772503+0000) 2019-10-13T06:59:12.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:12.773+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:12.773421+0000) 2019-10-13T06:59:14.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:14.574+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.576380+0000) 2019-10-13T06:59:14.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:14.575+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.576513+0000) 2019-10-13T06:59:14.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:14.575+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.576637+0000) 2019-10-13T06:59:14.581 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:14.574+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.575075+0000) 2019-10-13T06:59:14.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:14.576+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.577507+0000) 2019-10-13T06:59:14.581 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:14.576+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.577818+0000) 2019-10-13T06:59:14.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:14.576+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.578052+0000) 2019-10-13T06:59:14.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:14.576+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:14.577703+0000) 2019-10-13T06:59:14.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:14.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:18.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:18.315+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:18.316304+0000) 2019-10-13T06:59:18.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:18.652+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:18.653291+0000) 2019-10-13T06:59:18.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:18.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:18.653290+0000) 2019-10-13T06:59:19.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:19.579+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.580572+0000) 2019-10-13T06:59:19.587 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:19.581+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.583343+0000) 2019-10-13T06:59:19.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:19.582+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.583487+0000) 2019-10-13T06:59:19.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:19.582+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.583752+0000) 2019-10-13T06:59:19.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:19.582+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.584188+0000) 2019-10-13T06:59:19.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:19.582+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.584117+0000) 2019-10-13T06:59:19.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:19.583+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.584407+0000) 2019-10-13T06:59:19.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:19.582+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:19.583220+0000) 2019-10-13T06:59:19.591 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:19.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111476 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:21.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:21.019+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:21.019976+0000) 2019-10-13T06:59:21.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:21.596+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:21.596352+0000) 2019-10-13T06:59:21.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:21.908+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:21.908985+0000) 2019-10-13T06:59:22.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:22.771+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:22.772662+0000) 2019-10-13T06:59:22.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:22.773+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:22.773573+0000) 2019-10-13T06:59:24.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:24.586+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.587132+0000) 2019-10-13T06:59:24.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:24.588+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.589868+0000) 2019-10-13T06:59:24.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:24.588+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.590155+0000) 2019-10-13T06:59:24.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:24.588+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.590354+0000) 2019-10-13T06:59:24.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:24.589+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.590594+0000) 2019-10-13T06:59:24.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:24.589+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.590426+0000) 2019-10-13T06:59:24.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:24.589+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.591113+0000) 2019-10-13T06:59:24.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:24.593+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111508 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:24.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:24.588+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:24.589084+0000) 2019-10-13T06:59:28.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:28.315+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:28.316478+0000) 2019-10-13T06:59:28.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:28.652+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:28.653474+0000) 2019-10-13T06:59:28.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:28.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:28.653467+0000) 2019-10-13T06:59:29.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:29.592+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.594240+0000) 2019-10-13T06:59:29.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:29.595+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.596974+0000) 2019-10-13T06:59:29.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:29.595+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.597089+0000) 2019-10-13T06:59:29.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:29.595+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.597204+0000) 2019-10-13T06:59:29.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:29.595+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.597297+0000) 2019-10-13T06:59:29.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:29.596+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.598054+0000) 2019-10-13T06:59:29.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:29.596+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.598200+0000) 2019-10-13T06:59:29.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:29.599+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:29.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:29.597+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:29.598758+0000) 2019-10-13T06:59:31.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:31.019+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:31.020139+0000) 2019-10-13T06:59:31.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:31.597+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:31.598610+0000) 2019-10-13T06:59:31.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:31.908+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:31.909183+0000) 2019-10-13T06:59:32.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:32.772+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:32.772829+0000) 2019-10-13T06:59:32.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:32.772+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:32.773761+0000) 2019-10-13T06:59:34.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:34.601+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.602594+0000) 2019-10-13T06:59:34.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:34.601+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.603336+0000) 2019-10-13T06:59:34.607 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:34.602+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.603544+0000) 2019-10-13T06:59:34.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:34.602+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.603993+0000) 2019-10-13T06:59:34.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:34.602+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.604096+0000) 2019-10-13T06:59:34.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:34.599+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.601251+0000) 2019-10-13T06:59:34.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:34.601+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.603227+0000) 2019-10-13T06:59:34.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:34.603+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:34.604770+0000) 2019-10-13T06:59:34.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:34.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111572 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:38.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:38.316+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:38.316657+0000) 2019-10-13T06:59:38.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:38.653+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:38.653660+0000) 2019-10-13T06:59:38.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:38.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:38.653660+0000) 2019-10-13T06:59:38.787 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T06:59:38.787 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:38.793 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T06:59:38.793 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:39.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:39.608+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.609979+0000) 2019-10-13T06:59:39.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:39.608+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.610214+0000) 2019-10-13T06:59:39.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:39.606+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.607760+0000) 2019-10-13T06:59:39.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:39.609+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.611068+0000) 2019-10-13T06:59:39.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:39.609+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.611290+0000) 2019-10-13T06:59:39.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:39.609+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.610978+0000) 2019-10-13T06:59:39.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:39.610+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.611471+0000) 2019-10-13T06:59:39.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:39.609+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:39.611693+0000) 2019-10-13T06:59:39.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:39.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111602 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:41.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:41.019+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:41.020333+0000) 2019-10-13T06:59:41.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:41.600+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:41.600912+0000) 2019-10-13T06:59:41.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:41.908+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:41.909373+0000) 2019-10-13T06:59:42.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:42.772+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:42.773039+0000) 2019-10-13T06:59:42.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:42.773+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:42.773960+0000) 2019-10-13T06:59:44.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:44.615+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.615728+0000) 2019-10-13T06:59:44.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:44.615+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.615807+0000) 2019-10-13T06:59:44.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:44.615+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.616117+0000) 2019-10-13T06:59:44.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:44.615+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.616281+0000) 2019-10-13T06:59:44.620 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:44.616+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.616417+0000) 2019-10-13T06:59:44.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:44.613+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.614018+0000) 2019-10-13T06:59:44.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:44.616+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.617255+0000) 2019-10-13T06:59:44.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:44.619+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:44.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:44.616+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:44.616930+0000) 2019-10-13T06:59:48.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:48.316+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:48.316843+0000) 2019-10-13T06:59:48.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:48.653+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:48.653858+0000) 2019-10-13T06:59:48.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:48.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:48.653885+0000) 2019-10-13T06:59:49.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:49.620+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.620941+0000) 2019-10-13T06:59:49.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:49.620+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.621090+0000) 2019-10-13T06:59:49.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:49.620+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.621362+0000) 2019-10-13T06:59:49.625 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:49.618+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.619558+0000) 2019-10-13T06:59:49.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:49.621+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.621549+0000) 2019-10-13T06:59:49.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:49.621+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.621623+0000) 2019-10-13T06:59:49.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:49.621+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.621989+0000) 2019-10-13T06:59:49.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:49.620+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:49.621422+0000) 2019-10-13T06:59:49.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:49.624+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:51.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:51.020+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:51.020521+0000) 2019-10-13T06:59:51.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:51.602+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:51.603403+0000) 2019-10-13T06:59:51.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:51.909+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:51.909583+0000) 2019-10-13T06:59:52.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:52.772+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:52.773241+0000) 2019-10-13T06:59:52.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:52.773+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:52.774141+0000) 2019-10-13T06:59:54.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:54.625+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.626195+0000) 2019-10-13T06:59:54.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:54.625+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.626387+0000) 2019-10-13T06:59:54.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:54.626+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.626534+0000) 2019-10-13T06:59:54.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:54.626+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.626607+0000) 2019-10-13T06:59:54.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:54.626+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.626725+0000) 2019-10-13T06:59:54.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:54.623+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.624338+0000) 2019-10-13T06:59:54.631 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:54.626+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.627039+0000) 2019-10-13T06:59:54.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:54.625+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:54.626494+0000) 2019-10-13T06:59:54.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:54.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T06:59:58.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:58.316+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:58.317044+0000) 2019-10-13T06:59:58.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:58.653+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:58.654027+0000) 2019-10-13T06:59:58.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:58.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:58.654028+0000) 2019-10-13T06:59:59.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T06:59:59.629+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.631034+0000) 2019-10-13T06:59:59.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T06:59:59.629+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.631349+0000) 2019-10-13T06:59:59.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T06:59:59.628+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.629579+0000) 2019-10-13T06:59:59.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T06:59:59.630+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.631768+0000) 2019-10-13T06:59:59.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T06:59:59.630+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.631973+0000) 2019-10-13T06:59:59.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T06:59:59.630+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.632292+0000) 2019-10-13T06:59:59.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T06:59:59.630+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.631848+0000) 2019-10-13T06:59:59.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T06:59:59.631+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T05:59:59.632248+0000) 2019-10-13T06:59:59.639 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T06:59:59.634+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:01.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:01.019+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:01.020675+0000) 2019-10-13T07:00:01.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:01.605+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:01.605707+0000) 2019-10-13T07:00:01.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:01.908+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:01.909814+0000) 2019-10-13T07:00:02.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:02.772+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:02.773416+0000) 2019-10-13T07:00:02.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:02.773+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:02.774322+0000) 2019-10-13T07:00:04.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:04.636+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.638093+0000) 2019-10-13T07:00:04.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:04.637+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.638491+0000) 2019-10-13T07:00:04.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:04.637+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.638963+0000) 2019-10-13T07:00:04.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:04.637+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.639146+0000) 2019-10-13T07:00:04.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:04.636+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.637184+0000) 2019-10-13T07:00:04.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:04.637+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.639336+0000) 2019-10-13T07:00:04.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:04.638+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.639507+0000) 2019-10-13T07:00:04.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:04.638+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:04.639075+0000) 2019-10-13T07:00:04.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:04.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:08.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:08.316+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:08.317235+0000) 2019-10-13T07:00:08.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:08.653+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:08.654207+0000) 2019-10-13T07:00:08.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:08.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:08.654210+0000) 2019-10-13T07:00:08.959 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:00:08.959 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:08.964 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:00:08.964 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:09.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:09.643+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.645328+0000) 2019-10-13T07:00:09.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:09.642+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.643340+0000) 2019-10-13T07:00:09.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:09.644+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.645725+0000) 2019-10-13T07:00:09.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:09.644+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.646104+0000) 2019-10-13T07:00:09.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:09.645+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.646451+0000) 2019-10-13T07:00:09.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:09.645+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.646786+0000) 2019-10-13T07:00:09.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:09.643+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.644693+0000) 2019-10-13T07:00:09.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:09.645+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:09.647056+0000) 2019-10-13T07:00:09.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:09.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:11.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:11.020+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:11.020867+0000) 2019-10-13T07:00:11.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:11.607+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:11.608083+0000) 2019-10-13T07:00:11.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:11.909+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:11.910001+0000) 2019-10-13T07:00:12.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:12.772+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:12.773605+0000) 2019-10-13T07:00:12.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:12.774+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:12.774509+0000) 2019-10-13T07:00:14.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:14.649+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.651308+0000) 2019-10-13T07:00:14.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:14.650+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.651631+0000) 2019-10-13T07:00:14.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:14.648+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.649224+0000) 2019-10-13T07:00:14.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:14.650+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.652148+0000) 2019-10-13T07:00:14.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:14.651+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.652550+0000) 2019-10-13T07:00:14.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:14.651+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.652764+0000) 2019-10-13T07:00:14.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:14.651+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.652979+0000) 2019-10-13T07:00:14.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:14.651+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:14.651877+0000) 2019-10-13T07:00:14.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:14.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:18.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:18.316+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:18.317414+0000) 2019-10-13T07:00:18.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:18.653+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:18.654421+0000) 2019-10-13T07:00:18.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:18.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:18.654411+0000) 2019-10-13T07:00:19.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:19.654+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.655371+0000) 2019-10-13T07:00:19.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:19.656+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.658378+0000) 2019-10-13T07:00:19.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:19.657+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.658498+0000) 2019-10-13T07:00:19.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:19.657+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.658841+0000) 2019-10-13T07:00:19.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:19.657+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.658985+0000) 2019-10-13T07:00:19.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:19.657+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.659350+0000) 2019-10-13T07:00:19.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:19.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.659453+0000) 2019-10-13T07:00:19.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:19.657+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:19.657726+0000) 2019-10-13T07:00:19.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:19.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:21.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:21.020+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:21.021079+0000) 2019-10-13T07:00:21.616 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:21.608+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:21.610486+0000) 2019-10-13T07:00:21.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:21.909+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:21.910191+0000) 2019-10-13T07:00:22.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:22.773+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:22.773762+0000) 2019-10-13T07:00:22.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:22.773+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:22.774659+0000) 2019-10-13T07:00:24.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:24.662+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.664339+0000) 2019-10-13T07:00:24.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:24.663+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.664657+0000) 2019-10-13T07:00:24.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:24.663+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.664750+0000) 2019-10-13T07:00:24.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:24.663+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.664872+0000) 2019-10-13T07:00:24.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:24.660+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.662053+0000) 2019-10-13T07:00:24.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:24.664+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.665518+0000) 2019-10-13T07:00:24.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:24.664+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.665552+0000) 2019-10-13T07:00:24.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:24.663+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:24.664883+0000) 2019-10-13T07:00:24.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:24.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:28.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:28.317+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:28.317600+0000) 2019-10-13T07:00:28.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:28.654+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:28.654629+0000) 2019-10-13T07:00:28.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:28.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:28.654607+0000) 2019-10-13T07:00:29.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:29.669+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.669825+0000) 2019-10-13T07:00:29.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:29.669+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.670162+0000) 2019-10-13T07:00:29.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:29.669+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.670300+0000) 2019-10-13T07:00:29.674 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:29.670+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.670472+0000) 2019-10-13T07:00:29.674 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:29.670+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.670774+0000) 2019-10-13T07:00:29.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:29.670+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.670975+0000) 2019-10-13T07:00:29.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:29.666+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.668159+0000) 2019-10-13T07:00:29.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:29.667+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:29.669249+0000) 2019-10-13T07:00:29.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:29.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:31.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:31.020+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:31.021269+0000) 2019-10-13T07:00:31.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:31.611+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:31.612829+0000) 2019-10-13T07:00:31.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:31.910+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:31.910382+0000) 2019-10-13T07:00:32.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:32.773+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:32.773910+0000) 2019-10-13T07:00:32.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:32.774+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:32.774836+0000) 2019-10-13T07:00:34.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:34.672+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.674695+0000) 2019-10-13T07:00:34.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:34.676+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.677330+0000) 2019-10-13T07:00:34.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:34.677+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.677687+0000) 2019-10-13T07:00:34.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:34.677+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.677529+0000) 2019-10-13T07:00:34.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:34.677+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.677890+0000) 2019-10-13T07:00:34.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:34.677+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.678262+0000) 2019-10-13T07:00:34.682 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:34.678+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.678445+0000) 2019-10-13T07:00:34.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:34.677+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:34.678501+0000) 2019-10-13T07:00:34.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:34.681+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:38.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:38.317+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:38.317788+0000) 2019-10-13T07:00:38.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:38.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:38.654798+0000) 2019-10-13T07:00:38.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:38.654+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:38.654800+0000) 2019-10-13T07:00:39.074 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:00:39.074 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:39.079 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:00:39.079 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:39.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:39.683+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.684359+0000) 2019-10-13T07:00:39.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:39.684+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.684547+0000) 2019-10-13T07:00:39.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:39.682+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.682878+0000) 2019-10-13T07:00:39.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:39.684+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.684832+0000) 2019-10-13T07:00:39.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:39.684+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.685140+0000) 2019-10-13T07:00:39.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:39.685+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.685514+0000) 2019-10-13T07:00:39.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:39.685+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.685717+0000) 2019-10-13T07:00:39.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:39.683+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:39.684155+0000) 2019-10-13T07:00:39.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:39.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 111980 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:41.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:41.021+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:41.021446+0000) 2019-10-13T07:00:41.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:41.614+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:41.615207+0000) 2019-10-13T07:00:41.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:41.910+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:41.910548+0000) 2019-10-13T07:00:42.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:42.774+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:42.774097+0000) 2019-10-13T07:00:42.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:42.774+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:42.775029+0000) 2019-10-13T07:00:44.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:44.688+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.689591+0000) 2019-10-13T07:00:44.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:44.688+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.689733+0000) 2019-10-13T07:00:44.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:44.688+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.689909+0000) 2019-10-13T07:00:44.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:44.688+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.690155+0000) 2019-10-13T07:00:44.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:44.687+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.688386+0000) 2019-10-13T07:00:44.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:44.689+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.690728+0000) 2019-10-13T07:00:44.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:44.689+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.690756+0000) 2019-10-13T07:00:44.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:44.689+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:44.689810+0000) 2019-10-13T07:00:44.698 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:44.693+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:48.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:48.316+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:48.317955+0000) 2019-10-13T07:00:48.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:48.653+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:48.654970+0000) 2019-10-13T07:00:48.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:48.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:48.654971+0000) 2019-10-13T07:00:49.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:49.693+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.693881+0000) 2019-10-13T07:00:49.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:49.695+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.696593+0000) 2019-10-13T07:00:49.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:49.695+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.696943+0000) 2019-10-13T07:00:49.701 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:49.695+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.696980+0000) 2019-10-13T07:00:49.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:49.695+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.697064+0000) 2019-10-13T07:00:49.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:49.695+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.697232+0000) 2019-10-13T07:00:49.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:49.696+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.697423+0000) 2019-10-13T07:00:49.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:49.694+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:49.695623+0000) 2019-10-13T07:00:49.704 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:49.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:51.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:51.020+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:51.021619+0000) 2019-10-13T07:00:51.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:51.617+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:51.617544+0000) 2019-10-13T07:00:51.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:51.909+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:51.910715+0000) 2019-10-13T07:00:52.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:52.772+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:52.774293+0000) 2019-10-13T07:00:52.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:52.774+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:52.775186+0000) 2019-10-13T07:00:54.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:54.700+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.702354+0000) 2019-10-13T07:00:54.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:54.701+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.702681+0000) 2019-10-13T07:00:54.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:54.699+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.700565+0000) 2019-10-13T07:00:54.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:54.702+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.703853+0000) 2019-10-13T07:00:54.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:54.702+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.703996+0000) 2019-10-13T07:00:54.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:54.702+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.704149+0000) 2019-10-13T07:00:54.707 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:54.702+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.703595+0000) 2019-10-13T07:00:54.708 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:54.701+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:54.702032+0000) 2019-10-13T07:00:54.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:54.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:00:58.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:58.317+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:58.318151+0000) 2019-10-13T07:00:58.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:58.654+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:58.655152+0000) 2019-10-13T07:00:58.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:58.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:58.655151+0000) 2019-10-13T07:00:59.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:00:59.707+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.708888+0000) 2019-10-13T07:00:59.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:00:59.706+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.707180+0000) 2019-10-13T07:00:59.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:00:59.708+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.709540+0000) 2019-10-13T07:00:59.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:00:59.708+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.709867+0000) 2019-10-13T07:00:59.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:00:59.708+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.709951+0000) 2019-10-13T07:00:59.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:00:59.708+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.710247+0000) 2019-10-13T07:00:59.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:00:59.709+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.710428+0000) 2019-10-13T07:00:59.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:00:59.709+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:00:59.710463+0000) 2019-10-13T07:00:59.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:00:59.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:01.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:01.021+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:01.021810+0000) 2019-10-13T07:01:01.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:01.619+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:01.619956+0000) 2019-10-13T07:01:01.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:01.910+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:01.910904+0000) 2019-10-13T07:01:02.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:02.773+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:02.774462+0000) 2019-10-13T07:01:02.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:02.775+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:02.775310+0000) 2019-10-13T07:01:04.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:04.714+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.715539+0000) 2019-10-13T07:01:04.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:04.714+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.715699+0000) 2019-10-13T07:01:04.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:04.715+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.716420+0000) 2019-10-13T07:01:04.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:04.715+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.716541+0000) 2019-10-13T07:01:04.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:04.715+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.716721+0000) 2019-10-13T07:01:04.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:04.715+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.717043+0000) 2019-10-13T07:01:04.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:04.713+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.714506+0000) 2019-10-13T07:01:04.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:04.718+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:04.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:04.716+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:04.716984+0000) 2019-10-13T07:01:08.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:08.317+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:08.318381+0000) 2019-10-13T07:01:08.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:08.654+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:08.655368+0000) 2019-10-13T07:01:08.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:08.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:08.655356+0000) 2019-10-13T07:01:09.251 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:01:09.251 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:09.256 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:01:09.256 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:09.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:09.720+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.721521+0000) 2019-10-13T07:01:09.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:09.720+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.721794+0000) 2019-10-13T07:01:09.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:09.718+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.719131+0000) 2019-10-13T07:01:09.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:09.720+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.721985+0000) 2019-10-13T07:01:09.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:09.720+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.722186+0000) 2019-10-13T07:01:09.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:09.721+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.722536+0000) 2019-10-13T07:01:09.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:09.721+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.722608+0000) 2019-10-13T07:01:09.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:09.721+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:09.722360+0000) 2019-10-13T07:01:09.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:09.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:11.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:11.021+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:11.021992+0000) 2019-10-13T07:01:11.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:11.622+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:11.622217+0000) 2019-10-13T07:01:11.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:11.910+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:11.911077+0000) 2019-10-13T07:01:12.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:12.774+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:12.774650+0000) 2019-10-13T07:01:12.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:12.775+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:12.775486+0000) 2019-10-13T07:01:14.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:14.723+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.725533+0000) 2019-10-13T07:01:14.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:14.727+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.727589+0000) 2019-10-13T07:01:14.733 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:14.728+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.728425+0000) 2019-10-13T07:01:14.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:14.728+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.728602+0000) 2019-10-13T07:01:14.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:14.728+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.728844+0000) 2019-10-13T07:01:14.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:14.728+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.729081+0000) 2019-10-13T07:01:14.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:14.727+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.728082+0000) 2019-10-13T07:01:14.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:14.726+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:14.727862+0000) 2019-10-13T07:01:14.736 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:14.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112202 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:18.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:18.318+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:18.318590+0000) 2019-10-13T07:01:18.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:18.655+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:18.655533+0000) 2019-10-13T07:01:18.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:18.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:18.655517+0000) 2019-10-13T07:01:19.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:19.733+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.734092+0000) 2019-10-13T07:01:19.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:19.733+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.734381+0000) 2019-10-13T07:01:19.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:19.730+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.731771+0000) 2019-10-13T07:01:19.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:19.734+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.734745+0000) 2019-10-13T07:01:19.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:19.734+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.735006+0000) 2019-10-13T07:01:19.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:19.734+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.735144+0000) 2019-10-13T07:01:19.739 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:19.734+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.735349+0000) 2019-10-13T07:01:19.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:19.732+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:19.734392+0000) 2019-10-13T07:01:19.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:19.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112232 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:21.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:21.021+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:21.022179+0000) 2019-10-13T07:01:21.631 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:21.623+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:21.624752+0000) 2019-10-13T07:01:21.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:21.910+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:21.911282+0000) 2019-10-13T07:01:22.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:22.774+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:22.774800+0000) 2019-10-13T07:01:22.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:22.773+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:22.775629+0000) 2019-10-13T07:01:24.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:24.739+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.739971+0000) 2019-10-13T07:01:24.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:24.739+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.740233+0000) 2019-10-13T07:01:24.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:24.740+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.740421+0000) 2019-10-13T07:01:24.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:24.736+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.737789+0000) 2019-10-13T07:01:24.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:24.740+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.740563+0000) 2019-10-13T07:01:24.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:24.740+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.740775+0000) 2019-10-13T07:01:24.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:24.740+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.741176+0000) 2019-10-13T07:01:24.747 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:24.738+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:24.740423+0000) 2019-10-13T07:01:24.747 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:24.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112266 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:28.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:28.318+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:28.318746+0000) 2019-10-13T07:01:28.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:28.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:28.655687+0000) 2019-10-13T07:01:28.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:28.655+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:28.655721+0000) 2019-10-13T07:01:29.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:29.744+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.744911+0000) 2019-10-13T07:01:29.748 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:29.744+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.745047+0000) 2019-10-13T07:01:29.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:29.744+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.745207+0000) 2019-10-13T07:01:29.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:29.742+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.742861+0000) 2019-10-13T07:01:29.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:29.745+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.745450+0000) 2019-10-13T07:01:29.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:29.745+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.745609+0000) 2019-10-13T07:01:29.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:29.745+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.746032+0000) 2019-10-13T07:01:29.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:29.744+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:29.745298+0000) 2019-10-13T07:01:29.752 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:29.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112296 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:31.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:31.021+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:31.022392+0000) 2019-10-13T07:01:31.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:31.626+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:31.627214+0000) 2019-10-13T07:01:31.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:31.910+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:31.911473+0000) 2019-10-13T07:01:32.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:32.773+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:32.774910+0000) 2019-10-13T07:01:32.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:32.775+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:32.775831+0000) 2019-10-13T07:01:34.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:34.749+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.751205+0000) 2019-10-13T07:01:34.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:34.748+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.748905+0000) 2019-10-13T07:01:34.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:34.750+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.751896+0000) 2019-10-13T07:01:34.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:34.750+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.751992+0000) 2019-10-13T07:01:34.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:34.750+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.752142+0000) 2019-10-13T07:01:34.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:34.751+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.752403+0000) 2019-10-13T07:01:34.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:34.751+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.752531+0000) 2019-10-13T07:01:34.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:34.750+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:34.751391+0000) 2019-10-13T07:01:34.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:34.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:38.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:38.318+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:38.318934+0000) 2019-10-13T07:01:38.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:38.655+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:38.655876+0000) 2019-10-13T07:01:38.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:38.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:38.655866+0000) 2019-10-13T07:01:39.422 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:01:39.422 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:39.426 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:01:39.426 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:39.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:39.753+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.754658+0000) 2019-10-13T07:01:39.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:39.755+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.757303+0000) 2019-10-13T07:01:39.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:39.756+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.757463+0000) 2019-10-13T07:01:39.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:39.756+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.757735+0000) 2019-10-13T07:01:39.762 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:39.756+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.757926+0000) 2019-10-13T07:01:39.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:39.756+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.758351+0000) 2019-10-13T07:01:39.762 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:39.757+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.758620+0000) 2019-10-13T07:01:39.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:39.757+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:39.757781+0000) 2019-10-13T07:01:39.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:39.760+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112358 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:41.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:41.021+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:41.022604+0000) 2019-10-13T07:01:41.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:41.628+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:41.629528+0000) 2019-10-13T07:01:41.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:41.910+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:41.911696+0000) 2019-10-13T07:01:42.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:42.774+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:42.775057+0000) 2019-10-13T07:01:42.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:42.775+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:42.776013+0000) 2019-10-13T07:01:44.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:44.762+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.763597+0000) 2019-10-13T07:01:44.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:44.760+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.761625+0000) 2019-10-13T07:01:44.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:44.762+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.764212+0000) 2019-10-13T07:01:44.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:44.762+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.764374+0000) 2019-10-13T07:01:44.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:44.763+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.764543+0000) 2019-10-13T07:01:44.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:44.763+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.764712+0000) 2019-10-13T07:01:44.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:44.763+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.765107+0000) 2019-10-13T07:01:44.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:44.762+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:44.763416+0000) 2019-10-13T07:01:44.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:44.766+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112392 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:48.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:48.318+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:48.319117+0000) 2019-10-13T07:01:48.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:48.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:48.656045+0000) 2019-10-13T07:01:48.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:48.655+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:48.656078+0000) 2019-10-13T07:01:49.772 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:49.767+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.769296+0000) 2019-10-13T07:01:49.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:49.768+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.769475+0000) 2019-10-13T07:01:49.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:49.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.767216+0000) 2019-10-13T07:01:49.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:49.768+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.769750+0000) 2019-10-13T07:01:49.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:49.768+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.770234+0000) 2019-10-13T07:01:49.774 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:49.769+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.770506+0000) 2019-10-13T07:01:49.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:49.769+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.770974+0000) 2019-10-13T07:01:49.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:49.769+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:49.770264+0000) 2019-10-13T07:01:49.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:49.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:51.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:51.022+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:51.022816+0000) 2019-10-13T07:01:51.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:51.631+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:51.632001+0000) 2019-10-13T07:01:51.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:51.911+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:51.911863+0000) 2019-10-13T07:01:52.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:52.774+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:52.775225+0000) 2019-10-13T07:01:52.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:52.775+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:52.776199+0000) 2019-10-13T07:01:54.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:54.774+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.775770+0000) 2019-10-13T07:01:54.779 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:54.774+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.776031+0000) 2019-10-13T07:01:54.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:54.773+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.773956+0000) 2019-10-13T07:01:54.780 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:54.775+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.776630+0000) 2019-10-13T07:01:54.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:54.775+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.776702+0000) 2019-10-13T07:01:54.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:54.775+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.777043+0000) 2019-10-13T07:01:54.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:54.775+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.777371+0000) 2019-10-13T07:01:54.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:54.776+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:54.777139+0000) 2019-10-13T07:01:54.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:54.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:01:58.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:58.318+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:58.319299+0000) 2019-10-13T07:01:58.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:58.655+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:58.656237+0000) 2019-10-13T07:01:58.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:58.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:58.656229+0000) 2019-10-13T07:01:59.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:01:59.781+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.782355+0000) 2019-10-13T07:01:59.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:01:59.782+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.782854+0000) 2019-10-13T07:01:59.786 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:01:59.782+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.782929+0000) 2019-10-13T07:01:59.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:01:59.782+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.783148+0000) 2019-10-13T07:01:59.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:01:59.782+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.783270+0000) 2019-10-13T07:01:59.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:01:59.783+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.783518+0000) 2019-10-13T07:01:59.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:01:59.780+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.780745+0000) 2019-10-13T07:01:59.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:01:59.783+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:01:59.784089+0000) 2019-10-13T07:01:59.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:01:59.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:01.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:01.022+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:01.022994+0000) 2019-10-13T07:02:01.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:01.634+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:01.634381+0000) 2019-10-13T07:02:01.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:01.911+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:01.912057+0000) 2019-10-13T07:02:02.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:02.775+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:02.775373+0000) 2019-10-13T07:02:02.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:02.776+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:02.776378+0000) 2019-10-13T07:02:04.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:04.785+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.786196+0000) 2019-10-13T07:02:04.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:04.788+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.789129+0000) 2019-10-13T07:02:04.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:04.788+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.789035+0000) 2019-10-13T07:02:04.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:04.789+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.789543+0000) 2019-10-13T07:02:04.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:04.789+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.789394+0000) 2019-10-13T07:02:04.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:04.789+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.789696+0000) 2019-10-13T07:02:04.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:04.789+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.789747+0000) 2019-10-13T07:02:04.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:04.787+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:04.788227+0000) 2019-10-13T07:02:04.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:04.793+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:08.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:08.319+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:08.319497+0000) 2019-10-13T07:02:08.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:08.656+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:08.656439+0000) 2019-10-13T07:02:08.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:08.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:08.656442+0000) 2019-10-13T07:02:09.601 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:02:09.601 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:09.605 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:02:09.605 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:09.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:09.794+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.795335+0000) 2019-10-13T07:02:09.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:09.795+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.795618+0000) 2019-10-13T07:02:09.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:09.795+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.795822+0000) 2019-10-13T07:02:09.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:09.791+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.793140+0000) 2019-10-13T07:02:09.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:09.795+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.796128+0000) 2019-10-13T07:02:09.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:09.795+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.796348+0000) 2019-10-13T07:02:09.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:09.796+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.796579+0000) 2019-10-13T07:02:09.801 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:09.793+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:09.795578+0000) 2019-10-13T07:02:09.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:09.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:11.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:11.022+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:11.023191+0000) 2019-10-13T07:02:11.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:11.635+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:11.636920+0000) 2019-10-13T07:02:11.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:11.912+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:11.912249+0000) 2019-10-13T07:02:12.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:12.775+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:12.775528+0000) 2019-10-13T07:02:12.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:12.775+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:12.776494+0000) 2019-10-13T07:02:14.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:14.801+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.801515+0000) 2019-10-13T07:02:14.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:14.801+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.801602+0000) 2019-10-13T07:02:14.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:14.801+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.801752+0000) 2019-10-13T07:02:14.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:14.797+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.799583+0000) 2019-10-13T07:02:14.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:14.802+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.802409+0000) 2019-10-13T07:02:14.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:14.802+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.802800+0000) 2019-10-13T07:02:14.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:14.802+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.802868+0000) 2019-10-13T07:02:14.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:14.799+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:14.801259+0000) 2019-10-13T07:02:14.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:14.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:18.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:18.318+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:18.319691+0000) 2019-10-13T07:02:18.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:18.655+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:18.656651+0000) 2019-10-13T07:02:18.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:18.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:18.656645+0000) 2019-10-13T07:02:19.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:19.806+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.807514+0000) 2019-10-13T07:02:19.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:19.806+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.807929+0000) 2019-10-13T07:02:19.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:19.806+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.808154+0000) 2019-10-13T07:02:19.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:19.806+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.808302+0000) 2019-10-13T07:02:19.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:19.804+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.806241+0000) 2019-10-13T07:02:19.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:19.807+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.808820+0000) 2019-10-13T07:02:19.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:19.807+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.809092+0000) 2019-10-13T07:02:19.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:19.806+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:19.808330+0000) 2019-10-13T07:02:19.815 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:19.810+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112610 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:21.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:21.022+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:21.023377+0000) 2019-10-13T07:02:21.645 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:21.638+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:21.639237+0000) 2019-10-13T07:02:21.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:21.911+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:21.912469+0000) 2019-10-13T07:02:22.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:22.774+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:22.775690+0000) 2019-10-13T07:02:22.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:22.775+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:22.776675+0000) 2019-10-13T07:02:24.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:24.813+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.814588+0000) 2019-10-13T07:02:24.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:24.813+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.814916+0000) 2019-10-13T07:02:24.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:24.813+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.815019+0000) 2019-10-13T07:02:24.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:24.813+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.815235+0000) 2019-10-13T07:02:24.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:24.811+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.812371+0000) 2019-10-13T07:02:24.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:24.814+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.815480+0000) 2019-10-13T07:02:24.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:24.814+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.816098+0000) 2019-10-13T07:02:24.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:24.815+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:24.816140+0000) 2019-10-13T07:02:24.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:24.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:28.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:28.319+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:28.319882+0000) 2019-10-13T07:02:28.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:28.656+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:28.656853+0000) 2019-10-13T07:02:28.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:28.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:28.656843+0000) 2019-10-13T07:02:29.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:29.816+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.817705+0000) 2019-10-13T07:02:29.824 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:29.818+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.820288+0000) 2019-10-13T07:02:29.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:29.819+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.820590+0000) 2019-10-13T07:02:29.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:29.819+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.820954+0000) 2019-10-13T07:02:29.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:29.819+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.821077+0000) 2019-10-13T07:02:29.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:29.820+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.821461+0000) 2019-10-13T07:02:29.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:29.820+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.821616+0000) 2019-10-13T07:02:29.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:29.820+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:29.821009+0000) 2019-10-13T07:02:29.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:29.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:31.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:31.022+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:31.023590+0000) 2019-10-13T07:02:31.647 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:31.641+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:31.641637+0000) 2019-10-13T07:02:31.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:31.911+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:31.912673+0000) 2019-10-13T07:02:32.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:32.775+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:32.775892+0000) 2019-10-13T07:02:32.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:32.776+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:32.776842+0000) 2019-10-13T07:02:34.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:34.824+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.826323+0000) 2019-10-13T07:02:34.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:34.823+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.824064+0000) 2019-10-13T07:02:34.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:34.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.826847+0000) 2019-10-13T07:02:34.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:34.825+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.826992+0000) 2019-10-13T07:02:34.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:34.825+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.827378+0000) 2019-10-13T07:02:34.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:34.826+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.827479+0000) 2019-10-13T07:02:34.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:34.826+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.827750+0000) 2019-10-13T07:02:34.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:34.825+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:34.826340+0000) 2019-10-13T07:02:34.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:34.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:38.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:38.319+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:38.320101+0000) 2019-10-13T07:02:38.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:38.656+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:38.657048+0000) 2019-10-13T07:02:38.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:38.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:38.657039+0000) 2019-10-13T07:02:39.764 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:02:39.764 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:39.769 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:02:39.769 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:39.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:39.831+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.832702+0000) 2019-10-13T07:02:39.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:39.831+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.832941+0000) 2019-10-13T07:02:39.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:39.831+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.833067+0000) 2019-10-13T07:02:39.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:39.831+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.833233+0000) 2019-10-13T07:02:39.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:39.831+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.833350+0000) 2019-10-13T07:02:39.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:39.831+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.831832+0000) 2019-10-13T07:02:39.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:39.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.833875+0000) 2019-10-13T07:02:39.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:39.832+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:39.833222+0000) 2019-10-13T07:02:39.839 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:39.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112736 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:41.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:41.023+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:41.023776+0000) 2019-10-13T07:02:41.649 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:41.643+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:41.644062+0000) 2019-10-13T07:02:41.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:41.912+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:41.912866+0000) 2019-10-13T07:02:42.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:42.775+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:42.776068+0000) 2019-10-13T07:02:42.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:42.776+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:42.776961+0000) 2019-10-13T07:02:44.841 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:44.837+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.838252+0000) 2019-10-13T07:02:44.841 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:44.838+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.838411+0000) 2019-10-13T07:02:44.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:44.838+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.838678+0000) 2019-10-13T07:02:44.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:44.835+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.836421+0000) 2019-10-13T07:02:44.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:44.838+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.839188+0000) 2019-10-13T07:02:44.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:44.838+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.839335+0000) 2019-10-13T07:02:44.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:44.839+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.839570+0000) 2019-10-13T07:02:44.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:44.838+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:44.838762+0000) 2019-10-13T07:02:44.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:44.843+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112770 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:48.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:48.319+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:48.320332+0000) 2019-10-13T07:02:48.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:48.656+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:48.657272+0000) 2019-10-13T07:02:48.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:48.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:48.657271+0000) 2019-10-13T07:02:49.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:49.845+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.845457+0000) 2019-10-13T07:02:49.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:49.845+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.845668+0000) 2019-10-13T07:02:49.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:49.845+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.845788+0000) 2019-10-13T07:02:49.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:49.843+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.844051+0000) 2019-10-13T07:02:49.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:49.845+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.846302+0000) 2019-10-13T07:02:49.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:49.846+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.846444+0000) 2019-10-13T07:02:49.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:49.846+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.846679+0000) 2019-10-13T07:02:49.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:49.844+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:49.845712+0000) 2019-10-13T07:02:49.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:49.849+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:51.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:51.023+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:51.023983+0000) 2019-10-13T07:02:51.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:51.646+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:51.646396+0000) 2019-10-13T07:02:51.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:51.912+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:51.913066+0000) 2019-10-13T07:02:52.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:52.776+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:52.776260+0000) 2019-10-13T07:02:52.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:52.776+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:52.777121+0000) 2019-10-13T07:02:54.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:54.851+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.851846+0000) 2019-10-13T07:02:54.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:54.851+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.852087+0000) 2019-10-13T07:02:54.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:54.851+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.852241+0000) 2019-10-13T07:02:54.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:54.852+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.852480+0000) 2019-10-13T07:02:54.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:54.852+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.852757+0000) 2019-10-13T07:02:54.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:54.852+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.853026+0000) 2019-10-13T07:02:54.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:54.849+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.850482+0000) 2019-10-13T07:02:54.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:54.851+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:54.852095+0000) 2019-10-13T07:02:54.859 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:54.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112832 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:02:58.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:58.320+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:58.320520+0000) 2019-10-13T07:02:58.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:58.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:58.657461+0000) 2019-10-13T07:02:58.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:58.657+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:58.657478+0000) 2019-10-13T07:02:59.862 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:02:59.858+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.858946+0000) 2019-10-13T07:02:59.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:02:59.858+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.858851+0000) 2019-10-13T07:02:59.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:02:59.858+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.858599+0000) 2019-10-13T07:02:59.863 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:02:59.858+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.859141+0000) 2019-10-13T07:02:59.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:02:59.859+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.859585+0000) 2019-10-13T07:02:59.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:02:59.859+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.859841+0000) 2019-10-13T07:02:59.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:02:59.856+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.856946+0000) 2019-10-13T07:02:59.864 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:02:59.858+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:02:59.858881+0000) 2019-10-13T07:02:59.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:02:59.862+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:01.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:01.024+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:01.024176+0000) 2019-10-13T07:03:01.654 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:01.647+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:01.648753+0000) 2019-10-13T07:03:01.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:01.913+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:01.913238+0000) 2019-10-13T07:03:02.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:02.775+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:02.776420+0000) 2019-10-13T07:03:02.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:02.776+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:02.777311+0000) 2019-10-13T07:03:04.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:04.863+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.864420+0000) 2019-10-13T07:03:04.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:04.863+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.865137+0000) 2019-10-13T07:03:04.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:04.864+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.865429+0000) 2019-10-13T07:03:04.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:04.864+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.865508+0000) 2019-10-13T07:03:04.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:04.864+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.865599+0000) 2019-10-13T07:03:04.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:04.861+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.863501+0000) 2019-10-13T07:03:04.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:04.865+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.866612+0000) 2019-10-13T07:03:04.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:04.863+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:04.865501+0000) 2019-10-13T07:03:04.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:04.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112896 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:08.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:08.319+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:08.320693+0000) 2019-10-13T07:03:08.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:08.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:08.657606+0000) 2019-10-13T07:03:08.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:08.656+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:08.657670+0000) 2019-10-13T07:03:09.860 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:03:09.860 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:09.870 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:03:09.870 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:09.874 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:09.869+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.871147+0000) 2019-10-13T07:03:09.874 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:09.870+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.871733+0000) 2019-10-13T07:03:09.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:09.868+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.869796+0000) 2019-10-13T07:03:09.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:09.869+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.871373+0000) 2019-10-13T07:03:09.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:09.870+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.872201+0000) 2019-10-13T07:03:09.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:09.870+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.872362+0000) 2019-10-13T07:03:09.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:09.870+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.871882+0000) 2019-10-13T07:03:09.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:09.871+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:09.872492+0000) 2019-10-13T07:03:09.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:09.874+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:11.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:11.023+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:11.024362+0000) 2019-10-13T07:03:11.656 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:11.650+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:11.651033+0000) 2019-10-13T07:03:11.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:11.912+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:11.913424+0000) 2019-10-13T07:03:12.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:12.775+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:12.776587+0000) 2019-10-13T07:03:12.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:12.776+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:12.777495+0000) 2019-10-13T07:03:14.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:14.876+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.878194+0000) 2019-10-13T07:03:14.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:14.877+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.878490+0000) 2019-10-13T07:03:14.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:14.877+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.878611+0000) 2019-10-13T07:03:14.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:14.875+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.875914+0000) 2019-10-13T07:03:14.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:14.877+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.878905+0000) 2019-10-13T07:03:14.883 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:14.877+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.878916+0000) 2019-10-13T07:03:14.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:14.877+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.879090+0000) 2019-10-13T07:03:14.884 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:14.876+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:14.877705+0000) 2019-10-13T07:03:14.885 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:14.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:18.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:18.320+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:18.320882+0000) 2019-10-13T07:03:18.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:18.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:18.657802+0000) 2019-10-13T07:03:18.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:18.657+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:18.657860+0000) 2019-10-13T07:03:19.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:19.882+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.884384+0000) 2019-10-13T07:03:19.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:19.883+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.884583+0000) 2019-10-13T07:03:19.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:19.883+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.884764+0000) 2019-10-13T07:03:19.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:19.882+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.883081+0000) 2019-10-13T07:03:19.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:19.883+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.885126+0000) 2019-10-13T07:03:19.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:19.884+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.885692+0000) 2019-10-13T07:03:19.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:19.884+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.885907+0000) 2019-10-13T07:03:19.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:19.885+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:19.886091+0000) 2019-10-13T07:03:19.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:19.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 112988 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:21.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:21.023+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:21.024569+0000) 2019-10-13T07:03:21.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:21.652+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:21.653448+0000) 2019-10-13T07:03:21.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:21.913+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:21.913636+0000) 2019-10-13T07:03:22.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:22.776+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:22.776750+0000) 2019-10-13T07:03:22.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:22.777+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:22.777700+0000) 2019-10-13T07:03:24.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:24.887+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.888430+0000) 2019-10-13T07:03:24.894 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:24.889+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.890961+0000) 2019-10-13T07:03:24.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:24.889+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.891134+0000) 2019-10-13T07:03:24.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:24.889+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.891277+0000) 2019-10-13T07:03:24.895 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:24.890+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.891527+0000) 2019-10-13T07:03:24.895 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:24.890+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.891773+0000) 2019-10-13T07:03:24.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:24.890+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.892136+0000) 2019-10-13T07:03:24.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:24.890+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:24.891091+0000) 2019-10-13T07:03:24.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:24.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113023 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:28.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:28.320+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:28.321096+0000) 2019-10-13T07:03:28.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:28.656+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:28.658037+0000) 2019-10-13T07:03:28.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:28.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:28.658031+0000) 2019-10-13T07:03:29.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:29.895+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.896948+0000) 2019-10-13T07:03:29.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:29.895+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.897055+0000) 2019-10-13T07:03:29.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:29.895+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.897128+0000) 2019-10-13T07:03:29.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:29.894+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.894866+0000) 2019-10-13T07:03:29.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:29.896+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.897648+0000) 2019-10-13T07:03:29.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:29.896+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.897596+0000) 2019-10-13T07:03:29.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:29.896+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.897722+0000) 2019-10-13T07:03:29.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:29.896+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:29.897493+0000) 2019-10-13T07:03:29.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:29.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:31.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:31.024+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:31.024745+0000) 2019-10-13T07:03:31.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:31.655+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:31.655881+0000) 2019-10-13T07:03:31.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:31.913+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:31.913852+0000) 2019-10-13T07:03:32.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:32.776+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:32.776918+0000) 2019-10-13T07:03:32.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:32.777+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:32.777903+0000) 2019-10-13T07:03:34.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:34.897+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.898123+0000) 2019-10-13T07:03:34.904 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:34.900+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.900817+0000) 2019-10-13T07:03:34.904 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:34.900+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.901012+0000) 2019-10-13T07:03:34.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:34.900+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.900982+0000) 2019-10-13T07:03:34.905 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:34.900+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.901286+0000) 2019-10-13T07:03:34.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:34.900+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.901157+0000) 2019-10-13T07:03:34.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:34.901+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.901467+0000) 2019-10-13T07:03:34.906 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:34.899+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:34.900112+0000) 2019-10-13T07:03:34.907 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:34.904+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:38.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:38.320+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:38.321308+0000) 2019-10-13T07:03:38.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:38.657+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:38.658242+0000) 2019-10-13T07:03:38.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:38.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:38.658245+0000) 2019-10-13T07:03:39.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:39.903+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.904000+0000) 2019-10-13T07:03:39.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:39.906+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.906542+0000) 2019-10-13T07:03:39.911 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:39.906+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.906969+0000) 2019-10-13T07:03:39.911 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:39.906+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.906744+0000) 2019-10-13T07:03:39.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:39.906+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.907184+0000) 2019-10-13T07:03:39.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:39.906+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.907342+0000) 2019-10-13T07:03:39.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:39.907+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.907832+0000) 2019-10-13T07:03:39.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:39.906+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:39.907261+0000) 2019-10-13T07:03:39.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:39.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:39.980 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:03:39.980 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:39.985 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:03:39.985 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:41.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:41.024+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:41.024921+0000) 2019-10-13T07:03:41.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:41.657+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:41.658202+0000) 2019-10-13T07:03:41.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:41.913+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:41.914024+0000) 2019-10-13T07:03:42.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:42.776+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:42.777091+0000) 2019-10-13T07:03:42.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:42.777+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:42.778085+0000) 2019-10-13T07:03:44.916 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:44.912+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.912487+0000) 2019-10-13T07:03:44.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:44.912+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.912583+0000) 2019-10-13T07:03:44.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:44.912+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.912877+0000) 2019-10-13T07:03:44.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:44.912+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.913116+0000) 2019-10-13T07:03:44.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:44.910+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.910942+0000) 2019-10-13T07:03:44.917 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:44.912+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.913370+0000) 2019-10-13T07:03:44.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:44.913+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.913486+0000) 2019-10-13T07:03:44.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:44.915+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:44.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:44.912+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:44.913467+0000) 2019-10-13T07:03:48.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:48.321+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:48.321491+0000) 2019-10-13T07:03:48.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:48.657+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:48.658437+0000) 2019-10-13T07:03:48.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:48.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:48.658428+0000) 2019-10-13T07:03:49.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:49.915+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.917312+0000) 2019-10-13T07:03:49.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:49.916+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.917610+0000) 2019-10-13T07:03:49.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:49.916+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.917609+0000) 2019-10-13T07:03:49.923 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:49.916+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.917972+0000) 2019-10-13T07:03:49.923 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:49.916+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.918101+0000) 2019-10-13T07:03:49.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:49.915+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.915853+0000) 2019-10-13T07:03:49.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:49.917+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.918099+0000) 2019-10-13T07:03:49.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:49.917+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:49.918556+0000) 2019-10-13T07:03:49.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:49.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:51.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:51.024+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:51.025118+0000) 2019-10-13T07:03:51.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:51.660+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:51.660571+0000) 2019-10-13T07:03:51.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:51.913+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:51.914219+0000) 2019-10-13T07:03:52.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:52.776+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:52.777271+0000) 2019-10-13T07:03:52.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:52.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:52.778304+0000) 2019-10-13T07:03:54.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:54.918+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.919953+0000) 2019-10-13T07:03:54.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:54.921+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.922859+0000) 2019-10-13T07:03:54.926 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:54.921+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.923017+0000) 2019-10-13T07:03:54.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:54.921+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.923229+0000) 2019-10-13T07:03:54.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:54.921+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.923249+0000) 2019-10-13T07:03:54.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:54.921+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.923385+0000) 2019-10-13T07:03:54.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:54.922+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.924089+0000) 2019-10-13T07:03:54.928 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:54.921+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:54.923227+0000) 2019-10-13T07:03:54.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:54.925+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:03:58.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:58.320+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:58.321689+0000) 2019-10-13T07:03:58.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:58.657+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:58.658637+0000) 2019-10-13T07:03:58.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:58.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:58.658630+0000) 2019-10-13T07:03:59.933 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:03:59.927+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.928988+0000) 2019-10-13T07:03:59.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:03:59.927+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.929348+0000) 2019-10-13T07:03:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:03:59.928+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.929529+0000) 2019-10-13T07:03:59.934 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:03:59.928+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.929655+0000) 2019-10-13T07:03:59.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:03:59.928+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.929827+0000) 2019-10-13T07:03:59.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:03:59.928+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.930109+0000) 2019-10-13T07:03:59.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:03:59.924+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.926704+0000) 2019-10-13T07:03:59.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:03:59.926+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:03:59.928564+0000) 2019-10-13T07:03:59.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:03:59.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113240 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:01.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:01.024+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:01.025315+0000) 2019-10-13T07:04:01.668 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:01.662+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:01.662911+0000) 2019-10-13T07:04:01.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:01.913+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:01.914432+0000) 2019-10-13T07:04:02.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:02.776+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:02.777457+0000) 2019-10-13T07:04:02.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:02.776+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:02.778508+0000) 2019-10-13T07:04:04.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:04.931+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.932965+0000) 2019-10-13T07:04:04.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:04.934+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.935502+0000) 2019-10-13T07:04:04.939 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:04.934+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.936207+0000) 2019-10-13T07:04:04.939 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:04.934+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.936244+0000) 2019-10-13T07:04:04.940 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:04.935+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.936531+0000) 2019-10-13T07:04:04.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:04.935+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.936794+0000) 2019-10-13T07:04:04.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:04.935+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.936962+0000) 2019-10-13T07:04:04.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:04.935+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:04.936954+0000) 2019-10-13T07:04:04.943 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:04.938+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:08.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:08.321+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:08.321848+0000) 2019-10-13T07:04:08.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:08.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:08.658834+0000) 2019-10-13T07:04:08.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:08.657+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:08.658837+0000) 2019-10-13T07:04:09.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:09.940+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.941987+0000) 2019-10-13T07:04:09.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:09.940+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.942189+0000) 2019-10-13T07:04:09.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:09.941+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.942394+0000) 2019-10-13T07:04:09.946 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:09.941+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.942666+0000) 2019-10-13T07:04:09.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:09.939+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.940538+0000) 2019-10-13T07:04:09.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:09.941+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.942981+0000) 2019-10-13T07:04:09.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:09.941+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.943075+0000) 2019-10-13T07:04:09.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:09.942+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:09.943181+0000) 2019-10-13T07:04:09.949 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:09.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:10.121 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:04:10.121 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:10.126 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:04:10.126 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:11.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:11.024+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:11.025489+0000) 2019-10-13T07:04:11.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:11.664+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:11.665278+0000) 2019-10-13T07:04:11.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:11.914+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:11.914619+0000) 2019-10-13T07:04:12.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:12.777+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:12.777610+0000) 2019-10-13T07:04:12.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:12.777+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:12.778708+0000) 2019-10-13T07:04:14.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:14.944+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.945580+0000) 2019-10-13T07:04:14.952 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:14.946+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.948302+0000) 2019-10-13T07:04:14.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:14.947+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.948457+0000) 2019-10-13T07:04:14.952 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:14.947+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.948622+0000) 2019-10-13T07:04:14.952 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:14.947+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.948675+0000) 2019-10-13T07:04:14.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:14.947+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.949107+0000) 2019-10-13T07:04:14.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:14.948+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.949392+0000) 2019-10-13T07:04:14.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:14.947+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:14.948691+0000) 2019-10-13T07:04:14.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:14.950+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:18.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:18.321+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:18.322024+0000) 2019-10-13T07:04:18.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:18.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:18.659047+0000) 2019-10-13T07:04:18.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:18.658+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:18.659046+0000) 2019-10-13T07:04:19.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:19.954+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.954778+0000) 2019-10-13T07:04:19.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:19.954+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.955035+0000) 2019-10-13T07:04:19.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:19.954+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.955137+0000) 2019-10-13T07:04:19.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:19.952+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.953374+0000) 2019-10-13T07:04:19.959 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:19.955+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.955424+0000) 2019-10-13T07:04:19.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:19.955+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.955497+0000) 2019-10-13T07:04:19.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:19.955+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.955921+0000) 2019-10-13T07:04:19.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:19.953+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:19.954371+0000) 2019-10-13T07:04:19.962 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:19.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:21.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:21.025+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:21.025659+0000) 2019-10-13T07:04:21.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:21.667+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:21.667564+0000) 2019-10-13T07:04:21.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:21.914+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:21.914825+0000) 2019-10-13T07:04:22.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:22.777+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:22.777757+0000) 2019-10-13T07:04:22.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:22.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:22.778927+0000) 2019-10-13T07:04:24.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:24.960+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.960657+0000) 2019-10-13T07:04:24.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:24.960+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.960835+0000) 2019-10-13T07:04:24.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:24.960+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.961118+0000) 2019-10-13T07:04:24.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:24.958+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.959358+0000) 2019-10-13T07:04:24.965 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:24.960+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.961288+0000) 2019-10-13T07:04:24.965 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:24.961+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.961514+0000) 2019-10-13T07:04:24.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:24.961+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.961979+0000) 2019-10-13T07:04:24.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:24.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113401 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:24.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:24.960+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:24.961497+0000) 2019-10-13T07:04:28.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:28.322+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:28.322228+0000) 2019-10-13T07:04:28.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:28.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:28.659226+0000) 2019-10-13T07:04:28.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:28.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:28.659229+0000) 2019-10-13T07:04:29.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:29.965+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.966493+0000) 2019-10-13T07:04:29.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:29.967+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.968385+0000) 2019-10-13T07:04:29.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:29.966+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.967271+0000) 2019-10-13T07:04:29.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:29.967+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.967688+0000) 2019-10-13T07:04:29.980 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:29.967+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.968005+0000) 2019-10-13T07:04:29.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:29.967+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.968212+0000) 2019-10-13T07:04:29.980 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:29.967+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.968283+0000) 2019-10-13T07:04:29.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:29.968+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:29.968481+0000) 2019-10-13T07:04:29.981 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:29.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:31.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:31.025+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.025872+0000) 2019-10-13T07:04:31.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:31.669+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.669836+0000) 2019-10-13T07:04:31.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:31.914+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.915015+0000) 2019-10-13T07:04:32.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:32.777+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:32.777870+0000) 2019-10-13T07:04:32.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:32.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:32.779074+0000) 2019-10-13T07:04:34.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:34.971+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.971970+0000) 2019-10-13T07:04:34.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:34.973+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.974552+0000) 2019-10-13T07:04:34.981 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:34.972+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.974108+0000) 2019-10-13T07:04:34.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:34.973+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.974669+0000) 2019-10-13T07:04:34.981 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:34.973+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.974992+0000) 2019-10-13T07:04:34.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:34.973+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.975122+0000) 2019-10-13T07:04:34.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:34.973+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.975292+0000) 2019-10-13T07:04:34.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:34.974+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:34.975506+0000) 2019-10-13T07:04:34.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:34.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:38.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:38.321+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:38.322436+0000) 2019-10-13T07:04:38.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:38.658+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:38.659358+0000) 2019-10-13T07:04:38.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:38.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:38.659432+0000) 2019-10-13T07:04:39.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:39.978+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.980205+0000) 2019-10-13T07:04:39.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:39.979+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.980442+0000) 2019-10-13T07:04:39.984 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:39.979+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.980707+0000) 2019-10-13T07:04:39.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:39.979+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.980912+0000) 2019-10-13T07:04:39.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:39.977+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.978442+0000) 2019-10-13T07:04:39.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:39.979+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.981086+0000) 2019-10-13T07:04:39.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:39.979+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.981289+0000) 2019-10-13T07:04:39.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:39.980+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:39.980870+0000) 2019-10-13T07:04:39.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:39.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:40.289 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:04:40.289 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:40.294 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:04:40.294 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:41.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:41.025+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.026017+0000) 2019-10-13T07:04:41.677 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:41.672+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.672228+0000) 2019-10-13T07:04:41.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:41.914+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.915205+0000) 2019-10-13T07:04:42.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:42.777+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:42.778014+0000) 2019-10-13T07:04:42.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:42.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:42.779277+0000) 2019-10-13T07:04:44.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:44.984+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.984854+0000) 2019-10-13T07:04:44.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:44.986+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.987495+0000) 2019-10-13T07:04:44.991 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:44.986+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.987482+0000) 2019-10-13T07:04:44.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:44.986+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.987780+0000) 2019-10-13T07:04:44.992 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:44.986+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.987880+0000) 2019-10-13T07:04:44.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:44.986+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.988031+0000) 2019-10-13T07:04:44.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:44.987+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.988451+0000) 2019-10-13T07:04:44.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:44.987+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:44.988047+0000) 2019-10-13T07:04:44.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:44.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113527 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:48.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:48.321+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:48.322632+0000) 2019-10-13T07:04:48.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:48.658+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:48.659530+0000) 2019-10-13T07:04:48.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:48.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:48.659544+0000) 2019-10-13T07:04:49.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:49.991+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.993239+0000) 2019-10-13T07:04:49.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:49.992+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.993463+0000) 2019-10-13T07:04:49.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:49.989+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.991693+0000) 2019-10-13T07:04:49.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:49.992+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.993859+0000) 2019-10-13T07:04:49.998 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:49.992+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.993746+0000) 2019-10-13T07:04:49.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:49.992+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.994115+0000) 2019-10-13T07:04:49.998 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:49.992+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.994280+0000) 2019-10-13T07:04:49.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:49.991+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:49.993418+0000) 2019-10-13T07:04:50.001 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:49.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113557 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:51.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:51.025+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.026197+0000) 2019-10-13T07:04:51.680 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:51.673+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.674671+0000) 2019-10-13T07:04:51.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:51.913+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.915374+0000) 2019-10-13T07:04:52.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:52.777+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:52.778205+0000) 2019-10-13T07:04:52.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:52.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:52.779400+0000) 2019-10-13T07:04:55.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:54.998+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:55.000080+0000) 2019-10-13T07:04:55.004 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:04:54.999+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:55.001212+0000) 2019-10-13T07:04:55.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:54.999+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:55.001363+0000) 2019-10-13T07:04:55.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:04:54.997+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:54.999448+0000) 2019-10-13T07:04:55.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:55.000+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:55.001638+0000) 2019-10-13T07:04:55.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:04:55.000+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:55.002088+0000) 2019-10-13T07:04:55.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:04:55.000+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:55.002299+0000) 2019-10-13T07:04:55.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:04:54.999+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:55.001378+0000) 2019-10-13T07:04:55.008 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:04:55.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:04:58.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:04:58.322+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:58.322837+0000) 2019-10-13T07:04:58.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:04:58.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:58.659664+0000) 2019-10-13T07:04:58.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:04:58.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:58.659681+0000) 2019-10-13T07:05:00.010 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:00.005+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.006950+0000) 2019-10-13T07:05:00.010 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:00.005+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.007193+0000) 2019-10-13T07:05:00.011 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:00.006+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.007498+0000) 2019-10-13T07:05:00.011 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:00.006+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.007690+0000) 2019-10-13T07:05:00.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:00.006+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.007967+0000) 2019-10-13T07:05:00.011 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:00.006+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.008259+0000) 2019-10-13T07:05:00.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:00.004+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.005848+0000) 2019-10-13T07:05:00.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:00.006+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.008295+0000) 2019-10-13T07:05:00.014 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:00.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:01.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:01.025+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.026415+0000) 2019-10-13T07:05:01.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:01.677+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.677807+0000) 2019-10-13T07:05:01.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:01.915+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.915538+0000) 2019-10-13T07:05:02.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:02.778+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:02.778411+0000) 2019-10-13T07:05:02.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:02.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:02.779498+0000) 2019-10-13T07:05:05.016 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:05.013+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.013553+0000) 2019-10-13T07:05:05.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:05.013+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.013944+0000) 2019-10-13T07:05:05.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:05.013+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.014168+0000) 2019-10-13T07:05:05.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:05.011+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.011965+0000) 2019-10-13T07:05:05.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:05.014+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.014763+0000) 2019-10-13T07:05:05.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:05.014+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.014968+0000) 2019-10-13T07:05:05.018 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:05.014+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.015244+0000) 2019-10-13T07:05:05.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:05.013+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:05.014370+0000) 2019-10-13T07:05:05.022 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:05.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:08.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:08.322+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:08.322999+0000) 2019-10-13T07:05:08.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:08.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:08.659856+0000) 2019-10-13T07:05:08.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:08.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:08.659861+0000) 2019-10-13T07:05:10.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:10.020+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.020595+0000) 2019-10-13T07:05:10.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:10.020+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.021120+0000) 2019-10-13T07:05:10.024 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:10.020+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.021263+0000) 2019-10-13T07:05:10.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:10.021+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.021514+0000) 2019-10-13T07:05:10.025 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:10.021+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.021605+0000) 2019-10-13T07:05:10.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:10.019+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.020043+0000) 2019-10-13T07:05:10.026 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:10.021+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.022199+0000) 2019-10-13T07:05:10.027 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:10.024+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:10.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:10.021+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.022542+0000) 2019-10-13T07:05:10.444 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:05:10.444 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:10.449 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:05:10.449 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:11.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:11.026+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.026603+0000) 2019-10-13T07:05:11.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:11.679+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.680041+0000) 2019-10-13T07:05:11.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:11.915+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.915747+0000) 2019-10-13T07:05:12.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:12.778+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:12.778566+0000) 2019-10-13T07:05:12.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:12.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:12.779706+0000) 2019-10-13T07:05:15.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:15.023+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.024352+0000) 2019-10-13T07:05:15.031 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:15.026+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.026884+0000) 2019-10-13T07:05:15.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:15.026+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.027184+0000) 2019-10-13T07:05:15.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:15.027+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.027623+0000) 2019-10-13T07:05:15.032 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:15.027+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.027737+0000) 2019-10-13T07:05:15.032 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:15.027+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.027936+0000) 2019-10-13T07:05:15.032 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:15.027+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.028166+0000) 2019-10-13T07:05:15.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:15.026+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:15.026796+0000) 2019-10-13T07:05:15.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:15.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113716 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:18.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:18.322+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:18.323200+0000) 2019-10-13T07:05:18.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:18.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:18.660053+0000) 2019-10-13T07:05:18.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:18.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:18.660063+0000) 2019-10-13T07:05:20.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:20.032+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.033151+0000) 2019-10-13T07:05:20.036 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:20.033+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.033423+0000) 2019-10-13T07:05:20.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:20.030+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.031445+0000) 2019-10-13T07:05:20.038 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:20.033+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.034047+0000) 2019-10-13T07:05:20.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:20.033+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.033965+0000) 2019-10-13T07:05:20.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:20.033+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.034222+0000) 2019-10-13T07:05:20.038 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:20.034+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.034434+0000) 2019-10-13T07:05:20.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:20.032+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.032794+0000) 2019-10-13T07:05:20.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:20.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:21.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:21.025+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.026774+0000) 2019-10-13T07:05:21.688 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:21.681+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.682310+0000) 2019-10-13T07:05:21.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:21.914+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.915938+0000) 2019-10-13T07:05:22.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:22.777+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:22.778729+0000) 2019-10-13T07:05:22.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:22.779+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:22.779936+0000) 2019-10-13T07:05:25.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:25.038+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.040267+0000) 2019-10-13T07:05:25.044 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:25.038+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.040066+0000) 2019-10-13T07:05:25.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:25.038+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.040335+0000) 2019-10-13T07:05:25.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:25.039+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.040571+0000) 2019-10-13T07:05:25.044 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:25.039+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.040866+0000) 2019-10-13T07:05:25.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:25.039+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.041071+0000) 2019-10-13T07:05:25.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:25.037+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.038388+0000) 2019-10-13T07:05:25.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:25.041+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:25.041904+0000) 2019-10-13T07:05:25.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:25.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:28.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:28.322+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:28.323423+0000) 2019-10-13T07:05:28.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:28.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:28.660230+0000) 2019-10-13T07:05:28.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:28.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:28.660207+0000) 2019-10-13T07:05:30.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:30.045+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.046674+0000) 2019-10-13T07:05:30.050 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:30.045+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.046995+0000) 2019-10-13T07:05:30.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:30.044+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.044894+0000) 2019-10-13T07:05:30.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:30.045+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.047301+0000) 2019-10-13T07:05:30.051 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:30.046+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.047566+0000) 2019-10-13T07:05:30.051 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:30.046+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.047838+0000) 2019-10-13T07:05:30.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:30.046+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.047953+0000) 2019-10-13T07:05:30.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:30.046+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.047423+0000) 2019-10-13T07:05:30.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:30.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:31.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:31.025+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.026932+0000) 2019-10-13T07:05:31.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:31.684+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.684818+0000) 2019-10-13T07:05:31.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:31.915+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.916097+0000) 2019-10-13T07:05:32.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:32.778+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:32.778906+0000) 2019-10-13T07:05:32.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:32.779+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:32.780116+0000) 2019-10-13T07:05:35.057 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:35.052+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.053587+0000) 2019-10-13T07:05:35.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:35.052+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.053478+0000) 2019-10-13T07:05:35.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:35.052+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.053718+0000) 2019-10-13T07:05:35.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:35.052+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.054184+0000) 2019-10-13T07:05:35.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:35.052+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.054322+0000) 2019-10-13T07:05:35.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:35.053+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.054496+0000) 2019-10-13T07:05:35.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:35.051+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.052466+0000) 2019-10-13T07:05:35.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:35.052+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:35.053690+0000) 2019-10-13T07:05:35.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:35.056+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113842 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:38.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:38.322+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:38.323617+0000) 2019-10-13T07:05:38.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:38.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:38.660376+0000) 2019-10-13T07:05:38.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:38.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:38.660380+0000) 2019-10-13T07:05:40.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:40.057+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.059378+0000) 2019-10-13T07:05:40.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:40.058+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.059799+0000) 2019-10-13T07:05:40.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:40.058+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.059674+0000) 2019-10-13T07:05:40.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:40.056+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.057208+0000) 2019-10-13T07:05:40.064 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:40.058+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.060072+0000) 2019-10-13T07:05:40.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:40.058+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.060184+0000) 2019-10-13T07:05:40.064 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:40.059+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.060544+0000) 2019-10-13T07:05:40.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:40.059+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.060585+0000) 2019-10-13T07:05:40.067 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:40.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:40.615 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:05:40.615 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:40.619 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:05:40.619 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:41.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:41.026+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.027085+0000) 2019-10-13T07:05:41.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:41.685+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.687058+0000) 2019-10-13T07:05:41.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:41.915+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.916303+0000) 2019-10-13T07:05:42.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:42.777+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:42.779051+0000) 2019-10-13T07:05:42.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:42.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:42.780326+0000) 2019-10-13T07:05:45.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:45.061+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.063632+0000) 2019-10-13T07:05:45.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:45.064+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.066102+0000) 2019-10-13T07:05:45.070 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:45.065+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.066533+0000) 2019-10-13T07:05:45.070 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:45.065+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.066735+0000) 2019-10-13T07:05:45.070 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:45.065+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.066854+0000) 2019-10-13T07:05:45.070 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:45.065+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.067049+0000) 2019-10-13T07:05:45.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:45.066+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.067522+0000) 2019-10-13T07:05:45.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:45.064+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:45.066063+0000) 2019-10-13T07:05:45.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:45.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113906 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:48.327 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:48.323+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:48.323783+0000) 2019-10-13T07:05:48.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:48.660+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:48.660591+0000) 2019-10-13T07:05:48.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:48.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:48.660610+0000) 2019-10-13T07:05:50.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:50.068+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.069932+0000) 2019-10-13T07:05:50.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:50.072+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.072707+0000) 2019-10-13T07:05:50.076 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:50.072+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.072821+0000) 2019-10-13T07:05:50.076 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:50.072+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.073164+0000) 2019-10-13T07:05:50.077 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:50.072+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.073360+0000) 2019-10-13T07:05:50.077 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:50.073+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.073614+0000) 2019-10-13T07:05:50.077 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:50.073+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.073784+0000) 2019-10-13T07:05:50.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:50.071+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.073483+0000) 2019-10-13T07:05:50.080 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:50.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:51.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:51.026+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.027257+0000) 2019-10-13T07:05:51.694 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:51.688+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.689345+0000) 2019-10-13T07:05:51.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:51.916+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.916541+0000) 2019-10-13T07:05:52.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:52.778+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:52.779240+0000) 2019-10-13T07:05:52.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:52.778+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:52.780512+0000) 2019-10-13T07:05:55.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:05:55.078+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.079352+0000) 2019-10-13T07:05:55.083 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:55.079+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.079452+0000) 2019-10-13T07:05:55.083 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:05:55.079+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.079675+0000) 2019-10-13T07:05:55.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:05:55.076+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.077659+0000) 2019-10-13T07:05:55.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:05:55.079+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.080237+0000) 2019-10-13T07:05:55.084 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:55.079+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.080341+0000) 2019-10-13T07:05:55.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:55.080+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.080490+0000) 2019-10-13T07:05:55.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:05:55.078+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:55.079148+0000) 2019-10-13T07:05:55.087 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:05:55.083+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:05:58.327 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:05:58.323+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:58.323953+0000) 2019-10-13T07:05:58.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:05:58.660+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:58.660778+0000) 2019-10-13T07:05:58.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:05:58.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:58.660847+0000) 2019-10-13T07:06:00.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:00.085+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.085957+0000) 2019-10-13T07:06:00.091 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:00.085+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.086232+0000) 2019-10-13T07:06:00.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:00.085+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.085845+0000) 2019-10-13T07:06:00.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:00.086+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.086495+0000) 2019-10-13T07:06:00.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:00.086+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.086601+0000) 2019-10-13T07:06:00.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:00.086+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.086750+0000) 2019-10-13T07:06:00.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:00.082+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.083428+0000) 2019-10-13T07:06:00.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:00.085+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.085979+0000) 2019-10-13T07:06:00.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:00.089+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 113997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:01.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:01.027+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.027392+0000) 2019-10-13T07:06:01.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:01.691+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.691759+0000) 2019-10-13T07:06:01.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:01.916+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.916737+0000) 2019-10-13T07:06:02.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:02.779+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:02.779424+0000) 2019-10-13T07:06:02.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:02.780+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:02.780696+0000) 2019-10-13T07:06:05.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:05.091+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.091470+0000) 2019-10-13T07:06:05.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:05.091+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.091662+0000) 2019-10-13T07:06:05.095 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:05.091+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.091834+0000) 2019-10-13T07:06:05.095 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:05.091+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.091986+0000) 2019-10-13T07:06:05.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:05.089+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.089825+0000) 2019-10-13T07:06:05.096 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:05.092+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.092420+0000) 2019-10-13T07:06:05.096 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:05.092+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.092731+0000) 2019-10-13T07:06:05.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:05.091+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:05.092098+0000) 2019-10-13T07:06:05.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:05.096+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114032 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:08.327 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:08.323+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:08.324129+0000) 2019-10-13T07:06:08.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:08.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:08.660926+0000) 2019-10-13T07:06:08.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:08.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:08.661037+0000) 2019-10-13T07:06:10.104 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:10.097+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.098639+0000) 2019-10-13T07:06:10.105 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:10.097+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.098768+0000) 2019-10-13T07:06:10.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:10.097+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.098984+0000) 2019-10-13T07:06:10.105 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:10.097+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.099289+0000) 2019-10-13T07:06:10.105 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:10.098+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.099575+0000) 2019-10-13T07:06:10.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:10.098+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.099776+0000) 2019-10-13T07:06:10.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:10.095+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.096075+0000) 2019-10-13T07:06:10.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:10.098+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.098908+0000) 2019-10-13T07:06:10.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:10.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114061 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:10.752 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:06:10.752 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:10.757 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:06:10.757 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:11.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:11.026+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.027516+0000) 2019-10-13T07:06:11.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:11.693+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.694312+0000) 2019-10-13T07:06:11.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:11.916+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.916925+0000) 2019-10-13T07:06:12.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:12.780+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:12.780865+0000) 2019-10-13T07:06:12.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:12.778+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:12.779554+0000) 2019-10-13T07:06:15.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:15.103+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.105188+0000) 2019-10-13T07:06:15.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:15.104+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.105501+0000) 2019-10-13T07:06:15.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:15.103+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.103866+0000) 2019-10-13T07:06:15.109 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:15.104+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.105716+0000) 2019-10-13T07:06:15.110 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:15.104+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.105897+0000) 2019-10-13T07:06:15.110 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:15.105+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.106567+0000) 2019-10-13T07:06:15.110 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:15.105+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.106809+0000) 2019-10-13T07:06:15.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:15.105+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:15.106138+0000) 2019-10-13T07:06:15.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:15.108+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114094 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:18.327 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:18.323+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:18.324316+0000) 2019-10-13T07:06:18.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:18.659+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:18.661135+0000) 2019-10-13T07:06:18.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:18.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:18.661247+0000) 2019-10-13T07:06:20.119 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:20.110+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.110858+0000) 2019-10-13T07:06:20.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:20.111+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.111856+0000) 2019-10-13T07:06:20.120 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:20.110+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.111893+0000) 2019-10-13T07:06:20.120 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:20.110+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.112383+0000) 2019-10-13T07:06:20.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:20.111+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.112640+0000) 2019-10-13T07:06:20.121 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:20.111+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.112753+0000) 2019-10-13T07:06:20.121 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:20.111+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.112967+0000) 2019-10-13T07:06:20.121 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:20.111+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.113327+0000) 2019-10-13T07:06:20.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:20.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:21.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:21.026+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.027663+0000) 2019-10-13T07:06:21.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:21.695+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.696648+0000) 2019-10-13T07:06:21.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:21.916+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.917097+0000) 2019-10-13T07:06:22.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:22.778+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:22.779685+0000) 2019-10-13T07:06:22.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:22.780+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:22.780993+0000) 2019-10-13T07:06:25.121 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:25.116+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.118364+0000) 2019-10-13T07:06:25.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:25.117+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.118461+0000) 2019-10-13T07:06:25.122 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:25.117+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.118742+0000) 2019-10-13T07:06:25.123 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:25.116+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.116999+0000) 2019-10-13T07:06:25.123 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:25.117+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.119190+0000) 2019-10-13T07:06:25.123 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:25.118+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.119510+0000) 2019-10-13T07:06:25.123 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:25.118+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.119693+0000) 2019-10-13T07:06:25.124 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:25.118+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:25.118827+0000) 2019-10-13T07:06:25.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:25.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:28.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:28.323+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:28.324515+0000) 2019-10-13T07:06:28.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:28.660+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:28.661350+0000) 2019-10-13T07:06:28.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:28.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:28.661413+0000) 2019-10-13T07:06:30.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:30.123+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.124591+0000) 2019-10-13T07:06:30.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:30.123+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.124835+0000) 2019-10-13T07:06:30.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:30.124+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.125513+0000) 2019-10-13T07:06:30.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:30.124+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.125600+0000) 2019-10-13T07:06:30.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:30.124+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.125738+0000) 2019-10-13T07:06:30.131 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:30.124+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.125851+0000) 2019-10-13T07:06:30.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:30.122+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.123108+0000) 2019-10-13T07:06:30.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:30.124+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.124847+0000) 2019-10-13T07:06:30.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:30.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:31.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:31.027+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.027797+0000) 2019-10-13T07:06:31.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:31.698+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.698882+0000) 2019-10-13T07:06:31.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:31.916+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.917284+0000) 2019-10-13T07:06:32.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:32.779+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:32.779825+0000) 2019-10-13T07:06:32.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:32.781+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:32.781166+0000) 2019-10-13T07:06:35.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:35.131+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.131988+0000) 2019-10-13T07:06:35.136 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:35.131+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.132243+0000) 2019-10-13T07:06:35.136 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:35.132+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.132426+0000) 2019-10-13T07:06:35.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:35.132+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.132664+0000) 2019-10-13T07:06:35.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:35.133+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.134034+0000) 2019-10-13T07:06:35.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:35.133+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.134201+0000) 2019-10-13T07:06:35.138 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:35.130+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.132165+0000) 2019-10-13T07:06:35.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:35.131+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:35.133343+0000) 2019-10-13T07:06:35.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:35.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114220 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:38.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:38.324+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:38.324669+0000) 2019-10-13T07:06:38.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:38.661+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:38.661518+0000) 2019-10-13T07:06:38.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:38.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:38.661569+0000) 2019-10-13T07:06:40.142 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:40.138+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.138892+0000) 2019-10-13T07:06:40.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:40.138+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.139105+0000) 2019-10-13T07:06:40.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:40.139+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.139479+0000) 2019-10-13T07:06:40.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:40.139+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.139629+0000) 2019-10-13T07:06:40.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:40.139+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.139803+0000) 2019-10-13T07:06:40.144 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:40.136+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.138055+0000) 2019-10-13T07:06:40.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:40.140+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.140524+0000) 2019-10-13T07:06:40.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:40.137+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.139253+0000) 2019-10-13T07:06:40.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:40.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:40.898 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:06:40.898 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:40.905 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:06:40.905 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:41.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:41.027+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.027935+0000) 2019-10-13T07:06:41.706 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:41.700+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.701249+0000) 2019-10-13T07:06:41.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:41.917+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.917474+0000) 2019-10-13T07:06:42.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:42.779+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:42.779969+0000) 2019-10-13T07:06:42.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:42.779+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:42.781344+0000) 2019-10-13T07:06:45.149 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:45.145+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.145825+0000) 2019-10-13T07:06:45.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:45.145+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.145957+0000) 2019-10-13T07:06:45.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:45.145+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.146291+0000) 2019-10-13T07:06:45.150 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:45.142+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.144211+0000) 2019-10-13T07:06:45.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:45.146+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.146726+0000) 2019-10-13T07:06:45.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:45.146+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.147020+0000) 2019-10-13T07:06:45.151 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:45.146+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.147284+0000) 2019-10-13T07:06:45.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:45.144+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:45.146670+0000) 2019-10-13T07:06:45.153 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:45.150+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:48.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:48.324+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:48.324860+0000) 2019-10-13T07:06:48.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:48.661+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:48.661703+0000) 2019-10-13T07:06:48.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:48.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:48.661763+0000) 2019-10-13T07:06:50.155 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:50.149+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.150078+0000) 2019-10-13T07:06:50.156 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:50.151+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.152220+0000) 2019-10-13T07:06:50.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:50.152+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.152520+0000) 2019-10-13T07:06:50.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:50.152+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.152823+0000) 2019-10-13T07:06:50.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:50.152+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.153191+0000) 2019-10-13T07:06:50.157 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:50.152+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.153309+0000) 2019-10-13T07:06:50.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:50.153+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.153573+0000) 2019-10-13T07:06:50.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:50.151+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.152444+0000) 2019-10-13T07:06:50.160 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:50.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:51.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:51.028+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.028083+0000) 2019-10-13T07:06:51.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:51.702+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.703518+0000) 2019-10-13T07:06:51.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:51.917+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.917656+0000) 2019-10-13T07:06:52.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:52.779+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:52.780097+0000) 2019-10-13T07:06:52.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:52.780+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:52.781529+0000) 2019-10-13T07:06:55.161 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:55.157+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.158511+0000) 2019-10-13T07:06:55.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:06:55.157+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.158765+0000) 2019-10-13T07:06:55.162 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:06:55.155+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.156532+0000) 2019-10-13T07:06:55.163 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:06:55.157+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.159223+0000) 2019-10-13T07:06:55.163 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:06:55.158+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.159459+0000) 2019-10-13T07:06:55.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:55.158+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.159566+0000) 2019-10-13T07:06:55.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:55.158+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.159659+0000) 2019-10-13T07:06:55.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:06:55.158+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:55.159135+0000) 2019-10-13T07:06:55.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:06:55.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114346 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:06:58.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:06:58.324+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:58.325036+0000) 2019-10-13T07:06:58.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:06:58.660+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:58.661876+0000) 2019-10-13T07:06:58.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:06:58.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:58.661936+0000) 2019-10-13T07:07:00.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:00.162+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.164331+0000) 2019-10-13T07:07:00.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:00.163+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.164537+0000) 2019-10-13T07:07:00.168 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:00.161+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.162373+0000) 2019-10-13T07:07:00.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:00.163+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.164842+0000) 2019-10-13T07:07:00.169 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:00.163+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.164978+0000) 2019-10-13T07:07:00.169 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:00.163+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.165190+0000) 2019-10-13T07:07:00.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:00.163+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.165364+0000) 2019-10-13T07:07:00.170 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:00.163+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.164043+0000) 2019-10-13T07:07:00.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:00.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114375 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:01.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:01.026+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.028269+0000) 2019-10-13T07:07:01.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:01.705+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.705748+0000) 2019-10-13T07:07:01.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:01.916+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.917803+0000) 2019-10-13T07:07:02.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:02.779+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:02.780270+0000) 2019-10-13T07:07:02.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:02.781+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:02.781653+0000) 2019-10-13T07:07:05.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:05.167+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.168662+0000) 2019-10-13T07:07:05.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:05.169+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.171105+0000) 2019-10-13T07:07:05.175 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:05.169+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.171186+0000) 2019-10-13T07:07:05.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:05.170+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.171585+0000) 2019-10-13T07:07:05.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:05.170+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.172018+0000) 2019-10-13T07:07:05.175 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:05.170+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.172020+0000) 2019-10-13T07:07:05.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:05.170+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.172348+0000) 2019-10-13T07:07:05.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:05.171+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:05.171986+0000) 2019-10-13T07:07:05.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:05.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:08.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:08.324+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:08.325229+0000) 2019-10-13T07:07:08.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:08.661+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:08.662020+0000) 2019-10-13T07:07:08.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:08.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:08.662091+0000) 2019-10-13T07:07:10.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:10.174+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.175424+0000) 2019-10-13T07:07:10.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:10.176+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.178097+0000) 2019-10-13T07:07:10.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:10.176+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.178197+0000) 2019-10-13T07:07:10.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:10.177+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.178544+0000) 2019-10-13T07:07:10.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:10.177+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.178732+0000) 2019-10-13T07:07:10.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:10.177+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.179007+0000) 2019-10-13T07:07:10.183 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:10.177+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.179157+0000) 2019-10-13T07:07:10.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:10.177+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.178037+0000) 2019-10-13T07:07:10.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:10.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114439 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:11.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:11.027+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.028470+0000) 2019-10-13T07:07:11.061 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:07:11.062 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:11.066 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:07:11.066 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:11.713 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:11.707+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.708050+0000) 2019-10-13T07:07:11.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:11.917+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.917999+0000) 2019-10-13T07:07:12.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:12.779+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:12.780477+0000) 2019-10-13T07:07:12.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:12.781+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:12.781858+0000) 2019-10-13T07:07:15.188 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:15.181+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.182331+0000) 2019-10-13T07:07:15.190 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:15.183+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.184788+0000) 2019-10-13T07:07:15.190 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:15.183+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.184994+0000) 2019-10-13T07:07:15.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:15.183+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.185133+0000) 2019-10-13T07:07:15.192 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:15.183+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.185360+0000) 2019-10-13T07:07:15.192 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:15.183+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.185383+0000) 2019-10-13T07:07:15.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:15.184+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.185705+0000) 2019-10-13T07:07:15.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:15.183+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:15.184383+0000) 2019-10-13T07:07:15.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:15.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114472 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:18.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:18.324+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:18.325427+0000) 2019-10-13T07:07:18.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:18.661+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:18.662197+0000) 2019-10-13T07:07:18.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:18.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:18.662256+0000) 2019-10-13T07:07:20.196 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:20.191+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.191652+0000) 2019-10-13T07:07:20.196 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:20.191+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.191827+0000) 2019-10-13T07:07:20.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:20.191+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.192048+0000) 2019-10-13T07:07:20.197 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:20.191+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.192032+0000) 2019-10-13T07:07:20.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:20.191+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.192211+0000) 2019-10-13T07:07:20.197 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:20.192+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.192555+0000) 2019-10-13T07:07:20.198 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:20.189+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.190037+0000) 2019-10-13T07:07:20.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:20.190+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.190924+0000) 2019-10-13T07:07:20.199 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:20.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114501 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:21.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:21.028+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.028630+0000) 2019-10-13T07:07:21.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:21.710+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.710391+0000) 2019-10-13T07:07:21.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:21.917+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.918224+0000) 2019-10-13T07:07:22.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:22.780+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:22.780655+0000) 2019-10-13T07:07:22.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:22.781+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:22.782049+0000) 2019-10-13T07:07:25.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:25.197+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.198204+0000) 2019-10-13T07:07:25.202 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:25.198+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.198546+0000) 2019-10-13T07:07:25.202 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:25.195+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.196138+0000) 2019-10-13T07:07:25.203 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:25.198+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.198616+0000) 2019-10-13T07:07:25.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:25.198+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.198830+0000) 2019-10-13T07:07:25.203 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:25.198+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.198916+0000) 2019-10-13T07:07:25.203 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:25.198+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.199244+0000) 2019-10-13T07:07:25.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:25.198+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:25.199494+0000) 2019-10-13T07:07:25.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:25.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:28.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:28.325+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:28.325618+0000) 2019-10-13T07:07:28.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:28.661+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:28.662349+0000) 2019-10-13T07:07:28.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:28.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:28.662377+0000) 2019-10-13T07:07:30.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:30.203+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.204280+0000) 2019-10-13T07:07:30.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:30.204+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.204566+0000) 2019-10-13T07:07:30.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:30.201+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.203067+0000) 2019-10-13T07:07:30.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:30.204+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.204842+0000) 2019-10-13T07:07:30.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:30.204+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.204945+0000) 2019-10-13T07:07:30.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:30.205+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.205474+0000) 2019-10-13T07:07:30.209 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:30.205+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.205762+0000) 2019-10-13T07:07:30.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:30.203+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.204905+0000) 2019-10-13T07:07:30.212 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:30.208+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:31.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:31.028+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.028770+0000) 2019-10-13T07:07:31.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:31.711+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.712634+0000) 2019-10-13T07:07:31.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:31.918+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.918427+0000) 2019-10-13T07:07:32.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:32.780+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:32.780877+0000) 2019-10-13T07:07:32.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:32.781+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:32.782233+0000) 2019-10-13T07:07:35.214 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:35.210+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.210724+0000) 2019-10-13T07:07:35.214 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:35.210+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.210877+0000) 2019-10-13T07:07:35.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:35.207+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.208742+0000) 2019-10-13T07:07:35.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:35.210+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.211289+0000) 2019-10-13T07:07:35.215 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:35.211+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.211615+0000) 2019-10-13T07:07:35.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:35.211+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.211809+0000) 2019-10-13T07:07:35.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:35.211+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.212120+0000) 2019-10-13T07:07:35.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:35.209+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:35.210998+0000) 2019-10-13T07:07:35.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:35.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:38.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:38.325+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:38.325835+0000) 2019-10-13T07:07:38.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:38.661+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:38.662498+0000) 2019-10-13T07:07:38.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:38.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:38.662530+0000) 2019-10-13T07:07:40.220 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:40.215+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.217275+0000) 2019-10-13T07:07:40.221 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:40.213+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.215318+0000) 2019-10-13T07:07:40.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:40.216+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.217778+0000) 2019-10-13T07:07:40.222 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:40.216+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.218000+0000) 2019-10-13T07:07:40.222 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:40.216+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.218134+0000) 2019-10-13T07:07:40.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:40.216+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.218269+0000) 2019-10-13T07:07:40.222 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:40.217+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.218435+0000) 2019-10-13T07:07:40.223 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:40.215+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.217098+0000) 2019-10-13T07:07:40.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:40.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:41.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:41.027+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.028946+0000) 2019-10-13T07:07:41.215 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:07:41.215 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:41.219 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:07:41.219 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:41.720 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:41.714+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.714997+0000) 2019-10-13T07:07:41.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:41.917+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.918628+0000) 2019-10-13T07:07:42.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:42.779+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:42.781075+0000) 2019-10-13T07:07:42.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:42.781+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:42.782435+0000) 2019-10-13T07:07:45.228 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:45.221+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.222154+0000) 2019-10-13T07:07:45.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:45.222+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.224226+0000) 2019-10-13T07:07:45.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:45.223+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.224587+0000) 2019-10-13T07:07:45.228 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:45.223+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.224880+0000) 2019-10-13T07:07:45.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:45.223+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.225010+0000) 2019-10-13T07:07:45.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:45.223+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.225194+0000) 2019-10-13T07:07:45.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:45.224+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.225404+0000) 2019-10-13T07:07:45.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:45.223+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:45.224684+0000) 2019-10-13T07:07:45.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:45.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:48.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:48.324+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:48.326008+0000) 2019-10-13T07:07:48.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:48.661+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:48.662732+0000) 2019-10-13T07:07:48.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:48.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:48.662742+0000) 2019-10-13T07:07:50.233 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:50.227+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.228493+0000) 2019-10-13T07:07:50.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:50.229+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.230967+0000) 2019-10-13T07:07:50.235 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:50.229+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.231135+0000) 2019-10-13T07:07:50.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:50.229+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.231340+0000) 2019-10-13T07:07:50.235 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:50.230+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.231739+0000) 2019-10-13T07:07:50.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:50.230+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.231910+0000) 2019-10-13T07:07:50.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:50.230+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.232034+0000) 2019-10-13T07:07:50.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:50.230+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.231398+0000) 2019-10-13T07:07:50.238 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:50.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:51.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:51.028+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.029123+0000) 2019-10-13T07:07:51.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:51.716+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.717252+0000) 2019-10-13T07:07:51.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:51.918+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.918863+0000) 2019-10-13T07:07:52.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:52.780+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:52.781230+0000) 2019-10-13T07:07:52.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:52.782+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:52.782579+0000) 2019-10-13T07:07:55.240 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:55.235+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.237221+0000) 2019-10-13T07:07:55.241 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:07:55.234+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.235625+0000) 2019-10-13T07:07:55.241 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:07:55.236+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.237767+0000) 2019-10-13T07:07:55.242 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:07:55.236+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.237757+0000) 2019-10-13T07:07:55.242 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:55.236+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.238166+0000) 2019-10-13T07:07:55.242 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:07:55.236+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.238246+0000) 2019-10-13T07:07:55.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:55.237+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.238645+0000) 2019-10-13T07:07:55.243 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:07:55.237+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:55.237885+0000) 2019-10-13T07:07:55.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:07:55.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114720 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:07:58.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:07:58.325+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:58.326252+0000) 2019-10-13T07:07:58.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:07:58.662+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:58.662934+0000) 2019-10-13T07:07:58.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:07:58.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:58.662937+0000) 2019-10-13T07:08:00.248 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:00.242+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.243835+0000) 2019-10-13T07:08:00.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:00.242+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.244288+0000) 2019-10-13T07:08:00.248 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:00.243+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.244563+0000) 2019-10-13T07:08:00.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:00.243+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.244742+0000) 2019-10-13T07:08:00.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:00.243+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.244986+0000) 2019-10-13T07:08:00.249 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:00.242+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.242813+0000) 2019-10-13T07:08:00.249 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:00.243+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.245184+0000) 2019-10-13T07:08:00.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:00.244+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.245152+0000) 2019-10-13T07:08:00.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:00.247+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114749 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:01.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:01.028+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.029282+0000) 2019-10-13T07:08:01.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:01.719+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.719553+0000) 2019-10-13T07:08:01.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:01.918+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.919057+0000) 2019-10-13T07:08:02.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:02.780+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:02.781378+0000) 2019-10-13T07:08:02.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:02.782+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:02.782766+0000) 2019-10-13T07:08:05.253 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:05.248+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.250277+0000) 2019-10-13T07:08:05.254 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:05.249+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.250504+0000) 2019-10-13T07:08:05.254 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:05.249+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.250865+0000) 2019-10-13T07:08:05.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:05.247+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.248550+0000) 2019-10-13T07:08:05.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:05.249+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.251294+0000) 2019-10-13T07:08:05.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:05.250+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.251675+0000) 2019-10-13T07:08:05.255 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:05.250+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.251796+0000) 2019-10-13T07:08:05.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:05.251+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:05.251962+0000) 2019-10-13T07:08:05.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:05.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:08.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:08.326+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:08.326475+0000) 2019-10-13T07:08:08.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:08.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:08.663121+0000) 2019-10-13T07:08:08.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:08.662+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:08.663123+0000) 2019-10-13T07:08:10.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:10.256+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.256491+0000) 2019-10-13T07:08:10.261 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:10.253+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.254604+0000) 2019-10-13T07:08:10.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:10.256+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.257157+0000) 2019-10-13T07:08:10.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:10.257+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.257454+0000) 2019-10-13T07:08:10.261 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:10.257+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.257485+0000) 2019-10-13T07:08:10.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:10.257+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.257624+0000) 2019-10-13T07:08:10.262 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:10.257+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.257837+0000) 2019-10-13T07:08:10.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:10.256+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.256906+0000) 2019-10-13T07:08:10.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:10.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:11.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:11.029+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.029460+0000) 2019-10-13T07:08:11.378 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:08:11.378 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:11.384 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:08:11.384 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:11.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:11.721+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.721906+0000) 2019-10-13T07:08:11.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:11.918+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.919248+0000) 2019-10-13T07:08:12.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:12.781+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:12.781468+0000) 2019-10-13T07:08:12.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:12.782+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:12.782895+0000) 2019-10-13T07:08:15.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:15.262+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.262964+0000) 2019-10-13T07:08:15.267 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:15.263+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.263476+0000) 2019-10-13T07:08:15.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:15.262+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.263367+0000) 2019-10-13T07:08:15.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:15.263+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.263574+0000) 2019-10-13T07:08:15.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:15.263+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.263713+0000) 2019-10-13T07:08:15.268 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:15.260+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.261580+0000) 2019-10-13T07:08:15.268 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:15.263+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.264130+0000) 2019-10-13T07:08:15.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:15.262+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:15.263225+0000) 2019-10-13T07:08:15.270 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:15.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:18.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:18.326+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:18.326651+0000) 2019-10-13T07:08:18.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:18.663+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:18.663307+0000) 2019-10-13T07:08:18.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:18.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:18.663328+0000) 2019-10-13T07:08:20.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:20.266+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.268143+0000) 2019-10-13T07:08:20.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:20.270+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.270560+0000) 2019-10-13T07:08:20.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:20.270+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.270715+0000) 2019-10-13T07:08:20.274 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:20.270+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.270930+0000) 2019-10-13T07:08:20.275 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:20.270+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.271262+0000) 2019-10-13T07:08:20.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:20.271+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.271489+0000) 2019-10-13T07:08:20.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:20.268+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.270121+0000) 2019-10-13T07:08:20.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:20.271+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.271779+0000) 2019-10-13T07:08:20.277 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:20.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114873 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:21.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:21.029+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.029597+0000) 2019-10-13T07:08:21.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:21.722+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.724240+0000) 2019-10-13T07:08:21.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:21.919+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.919394+0000) 2019-10-13T07:08:22.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:22.781+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:22.781637+0000) 2019-10-13T07:08:22.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:22.782+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:22.783058+0000) 2019-10-13T07:08:25.279 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:25.274+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.276359+0000) 2019-10-13T07:08:25.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:25.275+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.276392+0000) 2019-10-13T07:08:25.280 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:25.275+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.276600+0000) 2019-10-13T07:08:25.281 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:25.273+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.274861+0000) 2019-10-13T07:08:25.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:25.275+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.277273+0000) 2019-10-13T07:08:25.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:25.276+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.277528+0000) 2019-10-13T07:08:25.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:25.276+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.277916+0000) 2019-10-13T07:08:25.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:25.275+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:25.277101+0000) 2019-10-13T07:08:25.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:25.279+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:28.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:28.325+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:28.326831+0000) 2019-10-13T07:08:28.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:28.662+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:28.663537+0000) 2019-10-13T07:08:28.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:28.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:28.663541+0000) 2019-10-13T07:08:30.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:30.281+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.283150+0000) 2019-10-13T07:08:30.287 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:30.282+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.283636+0000) 2019-10-13T07:08:30.287 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:30.282+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.284002+0000) 2019-10-13T07:08:30.287 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:30.282+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.284154+0000) 2019-10-13T07:08:30.288 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:30.283+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.284430+0000) 2019-10-13T07:08:30.288 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:30.280+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.281749+0000) 2019-10-13T07:08:30.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:30.283+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.285106+0000) 2019-10-13T07:08:30.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:30.282+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.283771+0000) 2019-10-13T07:08:30.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:30.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114936 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:31.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:31.028+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.029763+0000) 2019-10-13T07:08:31.732 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:31.725+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.726719+0000) 2019-10-13T07:08:31.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:31.918+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.919594+0000) 2019-10-13T07:08:32.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:32.780+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.781840+0000) 2019-10-13T07:08:32.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:32.782+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.783247+0000) 2019-10-13T07:08:35.293 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:35.287+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.288086+0000) 2019-10-13T07:08:35.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:35.289+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.290449+0000) 2019-10-13T07:08:35.295 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:35.289+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.290607+0000) 2019-10-13T07:08:35.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:35.289+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.290973+0000) 2019-10-13T07:08:35.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:35.290+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.291416+0000) 2019-10-13T07:08:35.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:35.290+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.291536+0000) 2019-10-13T07:08:35.296 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:35.290+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.291950+0000) 2019-10-13T07:08:35.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:35.289+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:35.290691+0000) 2019-10-13T07:08:35.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:35.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:38.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:38.325+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:38.327044+0000) 2019-10-13T07:08:38.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:38.663+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:38.663729+0000) 2019-10-13T07:08:38.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:38.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:38.663751+0000) 2019-10-13T07:08:40.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:40.295+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.296708+0000) 2019-10-13T07:08:40.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:40.295+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.296907+0000) 2019-10-13T07:08:40.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:40.295+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.297217+0000) 2019-10-13T07:08:40.301 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:40.295+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.297343+0000) 2019-10-13T07:08:40.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:40.294+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.295416+0000) 2019-10-13T07:08:40.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:40.296+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.297584+0000) 2019-10-13T07:08:40.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:40.296+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.297758+0000) 2019-10-13T07:08:40.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:40.297+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.298398+0000) 2019-10-13T07:08:40.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:40.300+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 114997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:41.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:41.029+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:41.029906+0000) 2019-10-13T07:08:41.540 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:08:41.540 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:41.545 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:08:41.545 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:41.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:41.728+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:41.729092+0000) 2019-10-13T07:08:41.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:41.919+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:41.919802+0000) 2019-10-13T07:08:42.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:42.780+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.782037+0000) 2019-10-13T07:08:42.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:42.782+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.783403+0000) 2019-10-13T07:08:45.306 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:45.301+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.303315+0000) 2019-10-13T07:08:45.307 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:45.302+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.303580+0000) 2019-10-13T07:08:45.307 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:45.302+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.303714+0000) 2019-10-13T07:08:45.307 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:45.302+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.304040+0000) 2019-10-13T07:08:45.308 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:45.301+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.302447+0000) 2019-10-13T07:08:45.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:45.303+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.304684+0000) 2019-10-13T07:08:45.308 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:45.303+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.304859+0000) 2019-10-13T07:08:45.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:45.303+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:45.304485+0000) 2019-10-13T07:08:45.311 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:45.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115031 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:48.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:48.326+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:48.327181+0000) 2019-10-13T07:08:48.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:48.663+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:48.663910+0000) 2019-10-13T07:08:48.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:48.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:48.663910+0000) 2019-10-13T07:08:50.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:50.308+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.309581+0000) 2019-10-13T07:08:50.313 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:50.308+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.309873+0000) 2019-10-13T07:08:50.314 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:50.307+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.308032+0000) 2019-10-13T07:08:50.314 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:50.309+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.310423+0000) 2019-10-13T07:08:50.314 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:50.309+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.310738+0000) 2019-10-13T07:08:50.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:50.309+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.310829+0000) 2019-10-13T07:08:50.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:50.309+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.311067+0000) 2019-10-13T07:08:50.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:50.309+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.310050+0000) 2019-10-13T07:08:50.317 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:50.312+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:51.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:51.029+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:51.030025+0000) 2019-10-13T07:08:51.737 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:51.730+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:51.731400+0000) 2019-10-13T07:08:51.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:51.919+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:51.920008+0000) 2019-10-13T07:08:52.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:52.781+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.782222+0000) 2019-10-13T07:08:52.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:52.783+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.783562+0000) 2019-10-13T07:08:55.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:55.315+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.316001+0000) 2019-10-13T07:08:55.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:08:55.313+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.313959+0000) 2019-10-13T07:08:55.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:08:55.316+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.316844+0000) 2019-10-13T07:08:55.320 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:08:55.315+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.316347+0000) 2019-10-13T07:08:55.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:08:55.316+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.316977+0000) 2019-10-13T07:08:55.321 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:55.316+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.317204+0000) 2019-10-13T07:08:55.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:55.316+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.317350+0000) 2019-10-13T07:08:55.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:08:55.316+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:55.316910+0000) 2019-10-13T07:08:55.324 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:08:55.320+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115092 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:08:58.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:08:58.326+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:58.327361+0000) 2019-10-13T07:08:58.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:08:58.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:58.664090+0000) 2019-10-13T07:08:58.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:08:58.663+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:58.664091+0000) 2019-10-13T07:09:00.325 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:00.319+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.320538+0000) 2019-10-13T07:09:00.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:00.322+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.322885+0000) 2019-10-13T07:09:00.326 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:00.322+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.323089+0000) 2019-10-13T07:09:00.327 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:00.322+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.323306+0000) 2019-10-13T07:09:00.327 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:00.323+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.323571+0000) 2019-10-13T07:09:00.327 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:00.323+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.323785+0000) 2019-10-13T07:09:00.328 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:00.323+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.324085+0000) 2019-10-13T07:09:00.328 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:00.322+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.323362+0000) 2019-10-13T07:09:00.330 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:00.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115121 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:01.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:01.029+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:01.030196+0000) 2019-10-13T07:09:01.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:01.733+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:01.733770+0000) 2019-10-13T07:09:01.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:01.919+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:01.920215+0000) 2019-10-13T07:09:02.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:02.782+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.782425+0000) 2019-10-13T07:09:02.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:02.783+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.783749+0000) 2019-10-13T07:09:05.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:05.329+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.329495+0000) 2019-10-13T07:09:05.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:05.329+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.329638+0000) 2019-10-13T07:09:05.334 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:05.326+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.327618+0000) 2019-10-13T07:09:05.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:05.329+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.330380+0000) 2019-10-13T07:09:05.334 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:05.330+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.330486+0000) 2019-10-13T07:09:05.334 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:05.330+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.330553+0000) 2019-10-13T07:09:05.335 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:05.330+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.330955+0000) 2019-10-13T07:09:05.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:05.329+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:05.330116+0000) 2019-10-13T07:09:05.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:05.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:08.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:08.327+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:08.327532+0000) 2019-10-13T07:09:08.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:08.664+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:08.664274+0000) 2019-10-13T07:09:08.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:08.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:08.664271+0000) 2019-10-13T07:09:10.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:10.335+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.336014+0000) 2019-10-13T07:09:10.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:10.335+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.336315+0000) 2019-10-13T07:09:10.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:10.335+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.336226+0000) 2019-10-13T07:09:10.340 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:10.333+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.334219+0000) 2019-10-13T07:09:10.340 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:10.336+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.336803+0000) 2019-10-13T07:09:10.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:10.336+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.337141+0000) 2019-10-13T07:09:10.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:10.336+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.337144+0000) 2019-10-13T07:09:10.341 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:10.335+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.336379+0000) 2019-10-13T07:09:10.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:10.340+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:11.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:11.029+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:11.030358+0000) 2019-10-13T07:09:11.690 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:09:11.691 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:11.695 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:09:11.695 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:11.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:11.735+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:11.736073+0000) 2019-10-13T07:09:11.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:11.919+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:11.920421+0000) 2019-10-13T07:09:12.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:12.781+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.782570+0000) 2019-10-13T07:09:12.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:12.783+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.783940+0000) 2019-10-13T07:09:15.345 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:15.338+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.340120+0000) 2019-10-13T07:09:15.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:15.341+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.342508+0000) 2019-10-13T07:09:15.346 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:15.341+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.342670+0000) 2019-10-13T07:09:15.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:15.341+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.342826+0000) 2019-10-13T07:09:15.347 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:15.341+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.343008+0000) 2019-10-13T07:09:15.347 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:15.341+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.343229+0000) 2019-10-13T07:09:15.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:15.342+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.343442+0000) 2019-10-13T07:09:15.348 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:15.341+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:15.343371+0000) 2019-10-13T07:09:15.350 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:15.345+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115216 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:18.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:18.326+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:18.327726+0000) 2019-10-13T07:09:18.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:18.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:18.664485+0000) 2019-10-13T07:09:18.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:18.663+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:18.664486+0000) 2019-10-13T07:09:20.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:20.347+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.349054+0000) 2019-10-13T07:09:20.352 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:20.347+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.349253+0000) 2019-10-13T07:09:20.353 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:20.345+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.347293+0000) 2019-10-13T07:09:20.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:20.348+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.349640+0000) 2019-10-13T07:09:20.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:20.348+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.350117+0000) 2019-10-13T07:09:20.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:20.348+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.350189+0000) 2019-10-13T07:09:20.354 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:20.348+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.350331+0000) 2019-10-13T07:09:20.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:20.348+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.349897+0000) 2019-10-13T07:09:20.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:20.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:21.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:21.029+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:21.030514+0000) 2019-10-13T07:09:21.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:21.737+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:21.738306+0000) 2019-10-13T07:09:21.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:21.919+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:21.920576+0000) 2019-10-13T07:09:22.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:22.782+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.782778+0000) 2019-10-13T07:09:22.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:22.783+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.784124+0000) 2019-10-13T07:09:25.359 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:25.354+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.355989+0000) 2019-10-13T07:09:25.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:25.354+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.356132+0000) 2019-10-13T07:09:25.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:25.354+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.356344+0000) 2019-10-13T07:09:25.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:25.355+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.356437+0000) 2019-10-13T07:09:25.360 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:25.355+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.356583+0000) 2019-10-13T07:09:25.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:25.353+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.355105+0000) 2019-10-13T07:09:25.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:25.353+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.355491+0000) 2019-10-13T07:09:25.361 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:25.355+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:25.357172+0000) 2019-10-13T07:09:25.362 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:25.358+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115279 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:28.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:28.326+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:28.327955+0000) 2019-10-13T07:09:28.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:28.664+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:28.664668+0000) 2019-10-13T07:09:28.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:28.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:28.664666+0000) 2019-10-13T07:09:30.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:30.360+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.361421+0000) 2019-10-13T07:09:30.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:30.360+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.361658+0000) 2019-10-13T07:09:30.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:30.360+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.361761+0000) 2019-10-13T07:09:30.365 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:30.360+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.361902+0000) 2019-10-13T07:09:30.366 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:30.359+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.360547+0000) 2019-10-13T07:09:30.366 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:30.361+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.362540+0000) 2019-10-13T07:09:30.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:30.361+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.362843+0000) 2019-10-13T07:09:30.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:30.361+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.362150+0000) 2019-10-13T07:09:30.369 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:30.364+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:31.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:31.030+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:31.030657+0000) 2019-10-13T07:09:31.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:31.740+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:31.740750+0000) 2019-10-13T07:09:31.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:31.920+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:31.920760+0000) 2019-10-13T07:09:32.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:32.782+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.782969+0000) 2019-10-13T07:09:32.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:32.783+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.784316+0000) 2019-10-13T07:09:35.371 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:35.365+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.365803+0000) 2019-10-13T07:09:35.372 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:35.366+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.368135+0000) 2019-10-13T07:09:35.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:35.366+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.368285+0000) 2019-10-13T07:09:35.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:35.367+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.368732+0000) 2019-10-13T07:09:35.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:35.367+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.368904+0000) 2019-10-13T07:09:35.373 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:35.367+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.369208+0000) 2019-10-13T07:09:35.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:35.368+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.369488+0000) 2019-10-13T07:09:35.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:35.368+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:35.369051+0000) 2019-10-13T07:09:35.376 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:35.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:38.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:38.327+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:38.328145+0000) 2019-10-13T07:09:38.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:38.664+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:38.664859+0000) 2019-10-13T07:09:38.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:38.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:38.664859+0000) 2019-10-13T07:09:40.377 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:40.374+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.374611+0000) 2019-10-13T07:09:40.378 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:40.374+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.375349+0000) 2019-10-13T07:09:40.379 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:40.375+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.375615+0000) 2019-10-13T07:09:40.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:40.375+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.375751+0000) 2019-10-13T07:09:40.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:40.375+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.375896+0000) 2019-10-13T07:09:40.379 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:40.375+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.375956+0000) 2019-10-13T07:09:40.380 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:40.372+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.373373+0000) 2019-10-13T07:09:40.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:40.374+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.374883+0000) 2019-10-13T07:09:40.381 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:40.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:41.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:41.030+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:41.030805+0000) 2019-10-13T07:09:41.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:41.742+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:41.743230+0000) 2019-10-13T07:09:41.833 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:09:41.833 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:41.838 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:09:41.838 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:41.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:41.920+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:41.920936+0000) 2019-10-13T07:09:42.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:42.782+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.783174+0000) 2019-10-13T07:09:42.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:42.784+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.784492+0000) 2019-10-13T07:09:45.383 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:45.379+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.380326+0000) 2019-10-13T07:09:45.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:45.380+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.380684+0000) 2019-10-13T07:09:45.384 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:45.380+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.380939+0000) 2019-10-13T07:09:45.385 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:45.378+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.379200+0000) 2019-10-13T07:09:45.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:45.381+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.381663+0000) 2019-10-13T07:09:45.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:45.381+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.381783+0000) 2019-10-13T07:09:45.385 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:45.381+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.382139+0000) 2019-10-13T07:09:45.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:45.380+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:45.381506+0000) 2019-10-13T07:09:45.388 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:45.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:48.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:48.328+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:48.328340+0000) 2019-10-13T07:09:48.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:48.664+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:48.665063+0000) 2019-10-13T07:09:48.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:48.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:48.665061+0000) 2019-10-13T07:09:50.390 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:50.386+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.386860+0000) 2019-10-13T07:09:50.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:50.386+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.386984+0000) 2019-10-13T07:09:50.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:50.384+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.384998+0000) 2019-10-13T07:09:50.392 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:50.387+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.387654+0000) 2019-10-13T07:09:50.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:50.387+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.387879+0000) 2019-10-13T07:09:50.392 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:50.387+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.388004+0000) 2019-10-13T07:09:50.392 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:50.388+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.388421+0000) 2019-10-13T07:09:50.393 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:50.386+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.387444+0000) 2019-10-13T07:09:50.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:50.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115432 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:51.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:51.030+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:51.030970+0000) 2019-10-13T07:09:51.751 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:51.745+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:51.745652+0000) 2019-10-13T07:09:51.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:51.920+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:51.921109+0000) 2019-10-13T07:09:52.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:52.783+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.783364+0000) 2019-10-13T07:09:52.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:52.784+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.784683+0000) 2019-10-13T07:09:55.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:55.392+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.393328+0000) 2019-10-13T07:09:55.397 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:55.393+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.393690+0000) 2019-10-13T07:09:55.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:09:55.393+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.393842+0000) 2019-10-13T07:09:55.398 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:09:55.391+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.391802+0000) 2019-10-13T07:09:55.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:09:55.394+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.394522+0000) 2019-10-13T07:09:55.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:09:55.394+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.394655+0000) 2019-10-13T07:09:55.398 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:55.394+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.394760+0000) 2019-10-13T07:09:55.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:09:55.393+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:55.394051+0000) 2019-10-13T07:09:55.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:09:55.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115464 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:09:58.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:09:58.327+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:58.328545+0000) 2019-10-13T07:09:58.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:09:58.664+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:58.665289+0000) 2019-10-13T07:09:58.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:09:58.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:58.665289+0000) 2019-10-13T07:10:00.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:00.398+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.400150+0000) 2019-10-13T07:10:00.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:00.397+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.398374+0000) 2019-10-13T07:10:00.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:00.399+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.400619+0000) 2019-10-13T07:10:00.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:00.399+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.400958+0000) 2019-10-13T07:10:00.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:00.399+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.401194+0000) 2019-10-13T07:10:00.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:00.399+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.401345+0000) 2019-10-13T07:10:00.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:00.400+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.401551+0000) 2019-10-13T07:10:00.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:00.400+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.400776+0000) 2019-10-13T07:10:00.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:00.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115493 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:01.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:01.030+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:01.031138+0000) 2019-10-13T07:10:01.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:01.748+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:01.748758+0000) 2019-10-13T07:10:01.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:01.919+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:01.921304+0000) 2019-10-13T07:10:02.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:02.782+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.783510+0000) 2019-10-13T07:10:02.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:02.784+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.784895+0000) 2019-10-13T07:10:05.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:05.405+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.407111+0000) 2019-10-13T07:10:05.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:05.405+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.407317+0000) 2019-10-13T07:10:05.411 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:05.404+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.405077+0000) 2019-10-13T07:10:05.411 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:05.406+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.407519+0000) 2019-10-13T07:10:05.411 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:05.406+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.407631+0000) 2019-10-13T07:10:05.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:05.406+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.407760+0000) 2019-10-13T07:10:05.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:05.406+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.408177+0000) 2019-10-13T07:10:05.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:05.405+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:05.406600+0000) 2019-10-13T07:10:05.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:05.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115527 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:08.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:08.327+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:08.328734+0000) 2019-10-13T07:10:08.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:08.664+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:08.665454+0000) 2019-10-13T07:10:08.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:08.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:08.665472+0000) 2019-10-13T07:10:10.417 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:10.412+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.413432+0000) 2019-10-13T07:10:10.418 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:10.412+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.413871+0000) 2019-10-13T07:10:10.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:10.412+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.413962+0000) 2019-10-13T07:10:10.419 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:10.412+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.414114+0000) 2019-10-13T07:10:10.419 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:10.412+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.414299+0000) 2019-10-13T07:10:10.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:10.413+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.414546+0000) 2019-10-13T07:10:10.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:10.409+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.411458+0000) 2019-10-13T07:10:10.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:10.411+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.413667+0000) 2019-10-13T07:10:10.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:10.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:11.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:11.030+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:11.031307+0000) 2019-10-13T07:10:11.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:11.750+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:11.751046+0000) 2019-10-13T07:10:11.916 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:10:11.917 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:11.921 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:10:11.921 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:11.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:11.920+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:11.921440+0000) 2019-10-13T07:10:12.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:12.782+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.783686+0000) 2019-10-13T07:10:12.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:12.784+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.785069+0000) 2019-10-13T07:10:15.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:15.418+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.420113+0000) 2019-10-13T07:10:15.424 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:15.419+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.420513+0000) 2019-10-13T07:10:15.424 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:15.419+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.420698+0000) 2019-10-13T07:10:15.425 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:15.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.418906+0000) 2019-10-13T07:10:15.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:15.419+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.420960+0000) 2019-10-13T07:10:15.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:15.419+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.421182+0000) 2019-10-13T07:10:15.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:15.419+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.421338+0000) 2019-10-13T07:10:15.426 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:15.418+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:15.420497+0000) 2019-10-13T07:10:15.428 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:15.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115588 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:18.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:18.328+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:18.328908+0000) 2019-10-13T07:10:18.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:18.665+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:18.665662+0000) 2019-10-13T07:10:18.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:18.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:18.665664+0000) 2019-10-13T07:10:20.430 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:20.425+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.426698+0000) 2019-10-13T07:10:20.430 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:20.425+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.426876+0000) 2019-10-13T07:10:20.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:20.424+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.425124+0000) 2019-10-13T07:10:20.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:20.426+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.427459+0000) 2019-10-13T07:10:20.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:20.426+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.427644+0000) 2019-10-13T07:10:20.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:20.426+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.427829+0000) 2019-10-13T07:10:20.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:20.426+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.428152+0000) 2019-10-13T07:10:20.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:20.427+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.427947+0000) 2019-10-13T07:10:20.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:20.430+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:21.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:21.030+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:21.031477+0000) 2019-10-13T07:10:21.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:21.752+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:21.753435+0000) 2019-10-13T07:10:21.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:21.921+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:21.921620+0000) 2019-10-13T07:10:22.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:22.783+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.783859+0000) 2019-10-13T07:10:22.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:22.784+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.785251+0000) 2019-10-13T07:10:25.437 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:25.433+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.433545+0000) 2019-10-13T07:10:25.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:25.433+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.433824+0000) 2019-10-13T07:10:25.437 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:25.433+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.434046+0000) 2019-10-13T07:10:25.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:25.433+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.434197+0000) 2019-10-13T07:10:25.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:25.430+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.431527+0000) 2019-10-13T07:10:25.439 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:25.434+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.434882+0000) 2019-10-13T07:10:25.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:25.434+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.435092+0000) 2019-10-13T07:10:25.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:25.433+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:25.434696+0000) 2019-10-13T07:10:25.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:25.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:28.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:28.328+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:28.329063+0000) 2019-10-13T07:10:28.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:28.665+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:28.665848+0000) 2019-10-13T07:10:28.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:28.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:28.665848+0000) 2019-10-13T07:10:30.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:30.439+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.439584+0000) 2019-10-13T07:10:30.443 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:30.439+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.440077+0000) 2019-10-13T07:10:30.444 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:30.436+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.437582+0000) 2019-10-13T07:10:30.444 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:30.439+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.440215+0000) 2019-10-13T07:10:30.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:30.440+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.440428+0000) 2019-10-13T07:10:30.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:30.440+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.440795+0000) 2019-10-13T07:10:30.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:30.439+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.439950+0000) 2019-10-13T07:10:30.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:30.439+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.439877+0000) 2019-10-13T07:10:30.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:30.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:31.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:31.031+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:31.031646+0000) 2019-10-13T07:10:31.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:31.755+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:31.755821+0000) 2019-10-13T07:10:31.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:31.921+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:31.921785+0000) 2019-10-13T07:10:32.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:32.783+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.784010+0000) 2019-10-13T07:10:32.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:32.784+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.785405+0000) 2019-10-13T07:10:35.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:35.445+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.446224+0000) 2019-10-13T07:10:35.449 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:35.446+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.446577+0000) 2019-10-13T07:10:35.450 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:35.443+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.444591+0000) 2019-10-13T07:10:35.450 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:35.446+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.446878+0000) 2019-10-13T07:10:35.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:35.446+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.447247+0000) 2019-10-13T07:10:35.451 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:35.447+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.447434+0000) 2019-10-13T07:10:35.451 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:35.447+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.447515+0000) 2019-10-13T07:10:35.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:35.445+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:35.446590+0000) 2019-10-13T07:10:35.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:35.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115712 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:38.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:38.329+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:38.329263+0000) 2019-10-13T07:10:38.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:38.665+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:38.666058+0000) 2019-10-13T07:10:38.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:38.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:38.666040+0000) 2019-10-13T07:10:40.455 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:40.449+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.450090+0000) 2019-10-13T07:10:40.456 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:40.452+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.452484+0000) 2019-10-13T07:10:40.456 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:40.452+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.453188+0000) 2019-10-13T07:10:40.457 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:40.453+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.453512+0000) 2019-10-13T07:10:40.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:40.453+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.453754+0000) 2019-10-13T07:10:40.458 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:40.453+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.453647+0000) 2019-10-13T07:10:40.458 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:40.453+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.454007+0000) 2019-10-13T07:10:40.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:40.452+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.452862+0000) 2019-10-13T07:10:40.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:40.457+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:41.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:41.031+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:41.031806+0000) 2019-10-13T07:10:41.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:41.757+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:41.758277+0000) 2019-10-13T07:10:41.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:41.921+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:41.921989+0000) 2019-10-13T07:10:42.033 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:10:42.033 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:42.037 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:10:42.037 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:42.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:42.782+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.784224+0000) 2019-10-13T07:10:42.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:42.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.785537+0000) 2019-10-13T07:10:45.462 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:45.457+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.459135+0000) 2019-10-13T07:10:45.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:45.457+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.459207+0000) 2019-10-13T07:10:45.463 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:45.458+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.459434+0000) 2019-10-13T07:10:45.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:45.458+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.459683+0000) 2019-10-13T07:10:45.464 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:45.457+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.458222+0000) 2019-10-13T07:10:45.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:45.458+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.459888+0000) 2019-10-13T07:10:45.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:45.458+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.460106+0000) 2019-10-13T07:10:45.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:45.458+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:45.459115+0000) 2019-10-13T07:10:45.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:45.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115775 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:48.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:48.328+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:48.329454+0000) 2019-10-13T07:10:48.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:48.665+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:48.666206+0000) 2019-10-13T07:10:48.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:48.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:48.666224+0000) 2019-10-13T07:10:50.468 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:50.463+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.464646+0000) 2019-10-13T07:10:50.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:50.463+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.464899+0000) 2019-10-13T07:10:50.468 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:50.463+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.465087+0000) 2019-10-13T07:10:50.469 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:50.463+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.465356+0000) 2019-10-13T07:10:50.469 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:50.463+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.463956+0000) 2019-10-13T07:10:50.469 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:50.464+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.465473+0000) 2019-10-13T07:10:50.470 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:50.464+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.466378+0000) 2019-10-13T07:10:50.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:50.465+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.465746+0000) 2019-10-13T07:10:50.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:50.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:51.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:51.031+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:51.031962+0000) 2019-10-13T07:10:51.766 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:51.760+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:51.760560+0000) 2019-10-13T07:10:51.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:51.921+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:51.922177+0000) 2019-10-13T07:10:52.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:52.783+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.784391+0000) 2019-10-13T07:10:52.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:52.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.785774+0000) 2019-10-13T07:10:55.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:10:55.468+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.469432+0000) 2019-10-13T07:10:55.475 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:55.470+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.472075+0000) 2019-10-13T07:10:55.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:10:55.470+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.472304+0000) 2019-10-13T07:10:55.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:10:55.471+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.472500+0000) 2019-10-13T07:10:55.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:55.471+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.472671+0000) 2019-10-13T07:10:55.477 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:10:55.471+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.472820+0000) 2019-10-13T07:10:55.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:55.471+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.473109+0000) 2019-10-13T07:10:55.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:10:55.472+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:55.473314+0000) 2019-10-13T07:10:55.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:10:55.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:10:58.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:10:58.328+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:58.329622+0000) 2019-10-13T07:10:58.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:10:58.665+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:58.666432+0000) 2019-10-13T07:10:58.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:10:58.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:58.666449+0000) 2019-10-13T07:11:00.481 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:00.476+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.478071+0000) 2019-10-13T07:11:00.481 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:00.474+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.476433+0000) 2019-10-13T07:11:00.483 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:00.477+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.478734+0000) 2019-10-13T07:11:00.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:00.477+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.479180+0000) 2019-10-13T07:11:00.483 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:00.476+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.478238+0000) 2019-10-13T07:11:00.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:00.477+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.479359+0000) 2019-10-13T07:11:00.484 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:00.477+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.478891+0000) 2019-10-13T07:11:00.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:00.477+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.478980+0000) 2019-10-13T07:11:00.485 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:00.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115865 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:01.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:01.031+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:01.032107+0000) 2019-10-13T07:11:01.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:01.761+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:01.762943+0000) 2019-10-13T07:11:01.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:01.921+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:01.922378+0000) 2019-10-13T07:11:02.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:02.783+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.784532+0000) 2019-10-13T07:11:02.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:02.784+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.785936+0000) 2019-10-13T07:11:05.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:05.483+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.484759+0000) 2019-10-13T07:11:05.488 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:05.483+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.485086+0000) 2019-10-13T07:11:05.489 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:05.481+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.483354+0000) 2019-10-13T07:11:05.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:05.484+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.485749+0000) 2019-10-13T07:11:05.489 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:05.484+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.485908+0000) 2019-10-13T07:11:05.490 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:05.484+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.486067+0000) 2019-10-13T07:11:05.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:05.484+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.486168+0000) 2019-10-13T07:11:05.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:05.484+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:05.486385+0000) 2019-10-13T07:11:05.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:05.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115899 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:08.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:08.329+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:08.329802+0000) 2019-10-13T07:11:08.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:08.666+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:08.666580+0000) 2019-10-13T07:11:08.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:08.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:08.666645+0000) 2019-10-13T07:11:10.494 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:10.487+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.489057+0000) 2019-10-13T07:11:10.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:10.490+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.491304+0000) 2019-10-13T07:11:10.495 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:10.491+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.491752+0000) 2019-10-13T07:11:10.495 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:10.491+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.492010+0000) 2019-10-13T07:11:10.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:10.491+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.492139+0000) 2019-10-13T07:11:10.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:10.491+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.492318+0000) 2019-10-13T07:11:10.496 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:10.492+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.492547+0000) 2019-10-13T07:11:10.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:10.490+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.492711+0000) 2019-10-13T07:11:10.499 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:10.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:11.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:11.031+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:11.032276+0000) 2019-10-13T07:11:11.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:11.764+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:11.765401+0000) 2019-10-13T07:11:11.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:11.922+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:11.922569+0000) 2019-10-13T07:11:12.120 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:11:12.120 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:12.124 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:11:12.125 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:12.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:12.784+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.784640+0000) 2019-10-13T07:11:12.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:12.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.786108+0000) 2019-10-13T07:11:15.501 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:15.497+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.498322+0000) 2019-10-13T07:11:15.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:15.498+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.498690+0000) 2019-10-13T07:11:15.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:15.496+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.496838+0000) 2019-10-13T07:11:15.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:15.498+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.498919+0000) 2019-10-13T07:11:15.503 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:15.498+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.499133+0000) 2019-10-13T07:11:15.503 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:15.498+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.499223+0000) 2019-10-13T07:11:15.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:15.499+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.499715+0000) 2019-10-13T07:11:15.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:15.498+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:15.499326+0000) 2019-10-13T07:11:15.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:15.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115960 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:18.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:18.329+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:18.329967+0000) 2019-10-13T07:11:18.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:18.666+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:18.666786+0000) 2019-10-13T07:11:18.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:18.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:18.666834+0000) 2019-10-13T07:11:20.508 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:20.502+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.503131+0000) 2019-10-13T07:11:20.509 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:20.505+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.505437+0000) 2019-10-13T07:11:20.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:20.505+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.505628+0000) 2019-10-13T07:11:20.510 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:20.505+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.505731+0000) 2019-10-13T07:11:20.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:20.505+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.506230+0000) 2019-10-13T07:11:20.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:20.506+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.506424+0000) 2019-10-13T07:11:20.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:20.506+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.506500+0000) 2019-10-13T07:11:20.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:20.505+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.506085+0000) 2019-10-13T07:11:20.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:20.509+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 115989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:21.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:21.032+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:21.032443+0000) 2019-10-13T07:11:21.773 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:21.767+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:21.767707+0000) 2019-10-13T07:11:21.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:21.922+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:21.922781+0000) 2019-10-13T07:11:22.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:22.784+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.784800+0000) 2019-10-13T07:11:22.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:22.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.786262+0000) 2019-10-13T07:11:25.515 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:25.511+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.511592+0000) 2019-10-13T07:11:25.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:25.511+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.511779+0000) 2019-10-13T07:11:25.515 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:25.511+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.512096+0000) 2019-10-13T07:11:25.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:25.511+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.512036+0000) 2019-10-13T07:11:25.516 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:25.509+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.510618+0000) 2019-10-13T07:11:25.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:25.511+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.512364+0000) 2019-10-13T07:11:25.517 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:25.512+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.512519+0000) 2019-10-13T07:11:25.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:25.512+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:25.512942+0000) 2019-10-13T07:11:25.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:25.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116023 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:28.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:28.329+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:28.330175+0000) 2019-10-13T07:11:28.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:28.665+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:28.667002+0000) 2019-10-13T07:11:28.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:28.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:28.666999+0000) 2019-10-13T07:11:30.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:30.514+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.515458+0000) 2019-10-13T07:11:30.521 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:30.516+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.518088+0000) 2019-10-13T07:11:30.522 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:30.517+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.518460+0000) 2019-10-13T07:11:30.522 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:30.517+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.518659+0000) 2019-10-13T07:11:30.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:30.517+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.518757+0000) 2019-10-13T07:11:30.522 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:30.517+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.519125+0000) 2019-10-13T07:11:30.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:30.517+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.519233+0000) 2019-10-13T07:11:30.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:30.518+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.518799+0000) 2019-10-13T07:11:30.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:30.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:31.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:31.031+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:31.032617+0000) 2019-10-13T07:11:31.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:31.769+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:31.769810+0000) 2019-10-13T07:11:31.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:31.922+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:31.922967+0000) 2019-10-13T07:11:32.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:32.783+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.785018+0000) 2019-10-13T07:11:32.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:32.786+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.786403+0000) 2019-10-13T07:11:35.527 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:35.521+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.521935+0000) 2019-10-13T07:11:35.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:35.522+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.524260+0000) 2019-10-13T07:11:35.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:35.523+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.524782+0000) 2019-10-13T07:11:35.529 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:35.523+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.524895+0000) 2019-10-13T07:11:35.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:35.523+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.525204+0000) 2019-10-13T07:11:35.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:35.523+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.525167+0000) 2019-10-13T07:11:35.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:35.524+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.525476+0000) 2019-10-13T07:11:35.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:35.523+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:35.524570+0000) 2019-10-13T07:11:35.532 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:35.527+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116084 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:38.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:38.329+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:38.330373+0000) 2019-10-13T07:11:38.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:38.666+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:38.667187+0000) 2019-10-13T07:11:38.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:38.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:38.667189+0000) 2019-10-13T07:11:40.533 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:40.527+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.528518+0000) 2019-10-13T07:11:40.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:40.529+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.531151+0000) 2019-10-13T07:11:40.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:40.529+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.531290+0000) 2019-10-13T07:11:40.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:40.530+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.531811+0000) 2019-10-13T07:11:40.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:40.530+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.532007+0000) 2019-10-13T07:11:40.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:40.530+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.532161+0000) 2019-10-13T07:11:40.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:40.531+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.532438+0000) 2019-10-13T07:11:40.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:40.530+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.531134+0000) 2019-10-13T07:11:40.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:40.534+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:41.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:41.031+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:41.032725+0000) 2019-10-13T07:11:41.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:41.771+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:41.772101+0000) 2019-10-13T07:11:41.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:41.921+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:41.923082+0000) 2019-10-13T07:11:42.242 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:11:42.243 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:42.247 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:11:42.247 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:42.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:42.784+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.785157+0000) 2019-10-13T07:11:42.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:42.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.786623+0000) 2019-10-13T07:11:45.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:45.536+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.537629+0000) 2019-10-13T07:11:45.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:45.536+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.537971+0000) 2019-10-13T07:11:45.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:45.535+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.535949+0000) 2019-10-13T07:11:45.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:45.536+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.538210+0000) 2019-10-13T07:11:45.542 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:45.536+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.538370+0000) 2019-10-13T07:11:45.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:45.537+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.538668+0000) 2019-10-13T07:11:45.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:45.537+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.538845+0000) 2019-10-13T07:11:45.543 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:45.537+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:45.538376+0000) 2019-10-13T07:11:45.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:45.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:48.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:48.329+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:48.330542+0000) 2019-10-13T07:11:48.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:48.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:48.667417+0000) 2019-10-13T07:11:48.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:48.666+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:48.667408+0000) 2019-10-13T07:11:50.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:50.542+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.543914+0000) 2019-10-13T07:11:50.547 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:50.542+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.544082+0000) 2019-10-13T07:11:50.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:50.542+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.543743+0000) 2019-10-13T07:11:50.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:50.543+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.544547+0000) 2019-10-13T07:11:50.548 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:50.542+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.542779+0000) 2019-10-13T07:11:50.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:50.543+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.544775+0000) 2019-10-13T07:11:50.549 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:50.543+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.545123+0000) 2019-10-13T07:11:50.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:50.544+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.544905+0000) 2019-10-13T07:11:50.551 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:50.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116176 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:51.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:51.032+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:51.032880+0000) 2019-10-13T07:11:51.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:51.774+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:51.774444+0000) 2019-10-13T07:11:51.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:51.922+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:51.923269+0000) 2019-10-13T07:11:52.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:52.783+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.785296+0000) 2019-10-13T07:11:52.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:52.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.786773+0000) 2019-10-13T07:11:55.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:55.548+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.550079+0000) 2019-10-13T07:11:55.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:55.548+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.550225+0000) 2019-10-13T07:11:55.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:55.549+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.550432+0000) 2019-10-13T07:11:55.554 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:11:55.547+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.549267+0000) 2019-10-13T07:11:55.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:11:55.549+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.550939+0000) 2019-10-13T07:11:55.555 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:11:55.549+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.551038+0000) 2019-10-13T07:11:55.555 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:11:55.550+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.551671+0000) 2019-10-13T07:11:55.555 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:11:55.548+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:55.550617+0000) 2019-10-13T07:11:55.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:11:55.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:11:58.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:11:58.330+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:58.330700+0000) 2019-10-13T07:11:58.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:11:58.667+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:58.667612+0000) 2019-10-13T07:11:58.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:11:58.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:58.667618+0000) 2019-10-13T07:12:00.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:00.556+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.556516+0000) 2019-10-13T07:12:00.560 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:00.556+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.556662+0000) 2019-10-13T07:12:00.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:00.552+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.554673+0000) 2019-10-13T07:12:00.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:00.556+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.557131+0000) 2019-10-13T07:12:00.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:00.556+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.557251+0000) 2019-10-13T07:12:00.561 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:00.557+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.557514+0000) 2019-10-13T07:12:00.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:00.557+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.557670+0000) 2019-10-13T07:12:00.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:00.554+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.556222+0000) 2019-10-13T07:12:00.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:00.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:01.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:01.032+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:01.033059+0000) 2019-10-13T07:12:01.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:01.775+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:01.776823+0000) 2019-10-13T07:12:01.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:01.923+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:01.923415+0000) 2019-10-13T07:12:02.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:02.785+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.785471+0000) 2019-10-13T07:12:02.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:02.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.786875+0000) 2019-10-13T07:12:05.566 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:05.558+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.560596+0000) 2019-10-13T07:12:05.566 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:05.562+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.562550+0000) 2019-10-13T07:12:05.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:05.563+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.563489+0000) 2019-10-13T07:12:05.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:05.563+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.563871+0000) 2019-10-13T07:12:05.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:05.563+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.564030+0000) 2019-10-13T07:12:05.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:05.563+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.563828+0000) 2019-10-13T07:12:05.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:05.563+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.563944+0000) 2019-10-13T07:12:05.568 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:05.561+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:05.563224+0000) 2019-10-13T07:12:05.571 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:05.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116271 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:08.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:08.330+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:08.330892+0000) 2019-10-13T07:12:08.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:08.667+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:08.667800+0000) 2019-10-13T07:12:08.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:08.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:08.667802+0000) 2019-10-13T07:12:10.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:10.569+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.569880+0000) 2019-10-13T07:12:10.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:10.569+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.570210+0000) 2019-10-13T07:12:10.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:10.567+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.567886+0000) 2019-10-13T07:12:10.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:10.570+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.570617+0000) 2019-10-13T07:12:10.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:10.570+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.570742+0000) 2019-10-13T07:12:10.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:10.570+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.571053+0000) 2019-10-13T07:12:10.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:10.570+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.570889+0000) 2019-10-13T07:12:10.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:10.569+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.569917+0000) 2019-10-13T07:12:10.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:10.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:11.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:11.033+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:11.033247+0000) 2019-10-13T07:12:11.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:11.778+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:11.779287+0000) 2019-10-13T07:12:11.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:11.923+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:11.923580+0000) 2019-10-13T07:12:12.331 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:12:12.331 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:12.336 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:12:12.336 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:12.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:12.785+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.785591+0000) 2019-10-13T07:12:12.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:12.786+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.787037+0000) 2019-10-13T07:12:15.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:15.575+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.576658+0000) 2019-10-13T07:12:15.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:15.575+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.576931+0000) 2019-10-13T07:12:15.581 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:15.574+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.575154+0000) 2019-10-13T07:12:15.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:15.576+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.577444+0000) 2019-10-13T07:12:15.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:15.576+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.577617+0000) 2019-10-13T07:12:15.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:15.576+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.577837+0000) 2019-10-13T07:12:15.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:15.576+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.578337+0000) 2019-10-13T07:12:15.582 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:15.576+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:15.577067+0000) 2019-10-13T07:12:15.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:15.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:18.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:18.329+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:18.331081+0000) 2019-10-13T07:12:18.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:18.667+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:18.668005+0000) 2019-10-13T07:12:18.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:18.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:18.668001+0000) 2019-10-13T07:12:20.586 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:20.581+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.582893+0000) 2019-10-13T07:12:20.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:20.581+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.583069+0000) 2019-10-13T07:12:20.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:20.580+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.581509+0000) 2019-10-13T07:12:20.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:20.582+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.583707+0000) 2019-10-13T07:12:20.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:20.582+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.584002+0000) 2019-10-13T07:12:20.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:20.582+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.584143+0000) 2019-10-13T07:12:20.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:20.582+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.584362+0000) 2019-10-13T07:12:20.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:20.582+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.583612+0000) 2019-10-13T07:12:20.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:20.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:21.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:21.032+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:21.033446+0000) 2019-10-13T07:12:21.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:21.780+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:21.781583+0000) 2019-10-13T07:12:21.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:21.922+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:21.923753+0000) 2019-10-13T07:12:22.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:22.784+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.785736+0000) 2019-10-13T07:12:22.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:22.786+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.787193+0000) 2019-10-13T07:12:25.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:25.588+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.590293+0000) 2019-10-13T07:12:25.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:25.589+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.590517+0000) 2019-10-13T07:12:25.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:25.587+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.588571+0000) 2019-10-13T07:12:25.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:25.589+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.590813+0000) 2019-10-13T07:12:25.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:25.589+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.591186+0000) 2019-10-13T07:12:25.595 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:25.589+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.591210+0000) 2019-10-13T07:12:25.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:25.590+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.591559+0000) 2019-10-13T07:12:25.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:25.589+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:25.590183+0000) 2019-10-13T07:12:25.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:25.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:28.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:28.330+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:28.331244+0000) 2019-10-13T07:12:28.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:28.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:28.668168+0000) 2019-10-13T07:12:28.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:28.667+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:28.668216+0000) 2019-10-13T07:12:30.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:30.594+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.595176+0000) 2019-10-13T07:12:30.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:30.596+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.597650+0000) 2019-10-13T07:12:30.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:30.596+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.597782+0000) 2019-10-13T07:12:30.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:30.596+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.598129+0000) 2019-10-13T07:12:30.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:30.596+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.598157+0000) 2019-10-13T07:12:30.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:30.597+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.598738+0000) 2019-10-13T07:12:30.602 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:30.597+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.598851+0000) 2019-10-13T07:12:30.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:30.597+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.598052+0000) 2019-10-13T07:12:30.606 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:30.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:31.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:31.032+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:31.033573+0000) 2019-10-13T07:12:31.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:31.782+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:31.783684+0000) 2019-10-13T07:12:31.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:31.923+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:31.923918+0000) 2019-10-13T07:12:32.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:32.784+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.785851+0000) 2019-10-13T07:12:32.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:32.786+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.787335+0000) 2019-10-13T07:12:35.607 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:35.602+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.602744+0000) 2019-10-13T07:12:35.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:35.603+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.605027+0000) 2019-10-13T07:12:35.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:35.604+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.605412+0000) 2019-10-13T07:12:35.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:35.604+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.605601+0000) 2019-10-13T07:12:35.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:35.604+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.605825+0000) 2019-10-13T07:12:35.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:35.604+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.606164+0000) 2019-10-13T07:12:35.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:35.604+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.606067+0000) 2019-10-13T07:12:35.610 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:35.604+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:35.604952+0000) 2019-10-13T07:12:35.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:35.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:38.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:38.330+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:38.331434+0000) 2019-10-13T07:12:38.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:38.667+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:38.668381+0000) 2019-10-13T07:12:38.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:38.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:38.668353+0000) 2019-10-13T07:12:40.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:40.610+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.611422+0000) 2019-10-13T07:12:40.615 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:40.610+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.611633+0000) 2019-10-13T07:12:40.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:40.610+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.611766+0000) 2019-10-13T07:12:40.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:40.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.609885+0000) 2019-10-13T07:12:40.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:40.611+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.612409+0000) 2019-10-13T07:12:40.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:40.611+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.612577+0000) 2019-10-13T07:12:40.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:40.611+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.612911+0000) 2019-10-13T07:12:40.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:40.611+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.612234+0000) 2019-10-13T07:12:40.619 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:40.615+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:41.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:41.033+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:41.033720+0000) 2019-10-13T07:12:41.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:41.785+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:41.785877+0000) 2019-10-13T07:12:41.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:41.923+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:41.924033+0000) 2019-10-13T07:12:42.462 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:12:42.462 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:42.467 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:12:42.467 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:42.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:42.785+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.785978+0000) 2019-10-13T07:12:42.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:42.786+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.787509+0000) 2019-10-13T07:12:45.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:45.614+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.615943+0000) 2019-10-13T07:12:45.622 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:45.617+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.618271+0000) 2019-10-13T07:12:45.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:45.618+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.618491+0000) 2019-10-13T07:12:45.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:45.618+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.618592+0000) 2019-10-13T07:12:45.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:45.618+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.618919+0000) 2019-10-13T07:12:45.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:45.618+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.618977+0000) 2019-10-13T07:12:45.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:45.616+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.618186+0000) 2019-10-13T07:12:45.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:45.618+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:45.619331+0000) 2019-10-13T07:12:45.625 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:45.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116519 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:48.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:48.331+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:48.331596+0000) 2019-10-13T07:12:48.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:48.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:48.668575+0000) 2019-10-13T07:12:48.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:48.668+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:48.668618+0000) 2019-10-13T07:12:50.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:50.623+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.624363+0000) 2019-10-13T07:12:50.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:50.624+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.624746+0000) 2019-10-13T07:12:50.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:50.621+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.622864+0000) 2019-10-13T07:12:50.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:50.624+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.625146+0000) 2019-10-13T07:12:50.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:50.624+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.625229+0000) 2019-10-13T07:12:50.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:50.625+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.625609+0000) 2019-10-13T07:12:50.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:50.625+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.625791+0000) 2019-10-13T07:12:50.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:50.623+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.625313+0000) 2019-10-13T07:12:50.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:50.629+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:51.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:51.033+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:51.033887+0000) 2019-10-13T07:12:51.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:51.786+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:51.788160+0000) 2019-10-13T07:12:51.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:51.923+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:51.924207+0000) 2019-10-13T07:12:52.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:52.785+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.786137+0000) 2019-10-13T07:12:52.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:52.785+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.787662+0000) 2019-10-13T07:12:55.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:55.631+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.631616+0000) 2019-10-13T07:12:55.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:12:55.631+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.631850+0000) 2019-10-13T07:12:55.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:55.631+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.632120+0000) 2019-10-13T07:12:55.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:55.631+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.632265+0000) 2019-10-13T07:12:55.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:12:55.632+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.632563+0000) 2019-10-13T07:12:55.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:12:55.632+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.633238+0000) 2019-10-13T07:12:55.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:12:55.629+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.631049+0000) 2019-10-13T07:12:55.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:12:55.631+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:55.633056+0000) 2019-10-13T07:12:55.639 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:12:55.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:12:58.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:12:58.331+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:58.331774+0000) 2019-10-13T07:12:58.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:12:58.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:58.668757+0000) 2019-10-13T07:12:58.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:12:58.668+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:58.668758+0000) 2019-10-13T07:13:00.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:00.635+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.636087+0000) 2019-10-13T07:13:00.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:00.636+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.638286+0000) 2019-10-13T07:13:00.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:00.637+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.638442+0000) 2019-10-13T07:13:00.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:00.637+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.638651+0000) 2019-10-13T07:13:00.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:00.637+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.639082+0000) 2019-10-13T07:13:00.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:00.637+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.638954+0000) 2019-10-13T07:13:00.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:00.637+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.639285+0000) 2019-10-13T07:13:00.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:00.638+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.639175+0000) 2019-10-13T07:13:00.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:00.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116610 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:01.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:01.033+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:01.034061+0000) 2019-10-13T07:13:01.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:01.789+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:01.790177+0000) 2019-10-13T07:13:01.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:01.923+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:01.924410+0000) 2019-10-13T07:13:02.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:02.785+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.786295+0000) 2019-10-13T07:13:02.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:02.787+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.787744+0000) 2019-10-13T07:13:05.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:05.643+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.645265+0000) 2019-10-13T07:13:05.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:05.642+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.643482+0000) 2019-10-13T07:13:05.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:05.644+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.645674+0000) 2019-10-13T07:13:05.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:05.644+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.646172+0000) 2019-10-13T07:13:05.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:05.645+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.646468+0000) 2019-10-13T07:13:05.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:05.644+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.646244+0000) 2019-10-13T07:13:05.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:05.645+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.646793+0000) 2019-10-13T07:13:05.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:05.644+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:05.645534+0000) 2019-10-13T07:13:05.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:05.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:08.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:08.331+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:08.331970+0000) 2019-10-13T07:13:08.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:08.668+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:08.668973+0000) 2019-10-13T07:13:08.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:08.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:08.668976+0000) 2019-10-13T07:13:10.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:10.649+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.651329+0000) 2019-10-13T07:13:10.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:10.650+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.651469+0000) 2019-10-13T07:13:10.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:10.650+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.651864+0000) 2019-10-13T07:13:10.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:10.649+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.650328+0000) 2019-10-13T07:13:10.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:10.650+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.652157+0000) 2019-10-13T07:13:10.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:10.651+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.652526+0000) 2019-10-13T07:13:10.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:10.651+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.652686+0000) 2019-10-13T07:13:10.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:10.652+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.652892+0000) 2019-10-13T07:13:10.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:10.655+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:11.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:11.032+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:11.034264+0000) 2019-10-13T07:13:11.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:11.791+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:11.792486+0000) 2019-10-13T07:13:11.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:11.923+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:11.924606+0000) 2019-10-13T07:13:12.684 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:13:12.684 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:12.689 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:13:12.689 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:12.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:12.785+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.786475+0000) 2019-10-13T07:13:12.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:12.787+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.787918+0000) 2019-10-13T07:13:15.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:15.656+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.658132+0000) 2019-10-13T07:13:15.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:15.655+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.656379+0000) 2019-10-13T07:13:15.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:15.657+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.658748+0000) 2019-10-13T07:13:15.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:15.657+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.658608+0000) 2019-10-13T07:13:15.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:15.657+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.658642+0000) 2019-10-13T07:13:15.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:15.657+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.658920+0000) 2019-10-13T07:13:15.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:15.657+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.659376+0000) 2019-10-13T07:13:15.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:15.657+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:15.658396+0000) 2019-10-13T07:13:15.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:15.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:18.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:18.331+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:18.332194+0000) 2019-10-13T07:13:18.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:18.667+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:18.669182+0000) 2019-10-13T07:13:18.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:18.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:18.669204+0000) 2019-10-13T07:13:20.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:20.662+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.664177+0000) 2019-10-13T07:13:20.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:20.662+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.664302+0000) 2019-10-13T07:13:20.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:20.663+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.664449+0000) 2019-10-13T07:13:20.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:20.663+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.664560+0000) 2019-10-13T07:13:20.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:20.663+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.665361+0000) 2019-10-13T07:13:20.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:20.664+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.665508+0000) 2019-10-13T07:13:20.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:20.662+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.663000+0000) 2019-10-13T07:13:20.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:20.663+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.664597+0000) 2019-10-13T07:13:20.673 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:20.667+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:21.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:21.033+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:21.034440+0000) 2019-10-13T07:13:21.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:21.794+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:21.795005+0000) 2019-10-13T07:13:21.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:21.924+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:21.924801+0000) 2019-10-13T07:13:22.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:22.785+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.786676+0000) 2019-10-13T07:13:22.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:22.787+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.788136+0000) 2019-10-13T07:13:25.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:25.669+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.670807+0000) 2019-10-13T07:13:25.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:25.668+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.668926+0000) 2019-10-13T07:13:25.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:25.669+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.671214+0000) 2019-10-13T07:13:25.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:25.670+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.671455+0000) 2019-10-13T07:13:25.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:25.670+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.671674+0000) 2019-10-13T07:13:25.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:25.670+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.671861+0000) 2019-10-13T07:13:25.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:25.670+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.672379+0000) 2019-10-13T07:13:25.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:25.670+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:25.671641+0000) 2019-10-13T07:13:25.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:25.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116767 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:28.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:28.332+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:28.332428+0000) 2019-10-13T07:13:28.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:28.668+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:28.669385+0000) 2019-10-13T07:13:28.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:28.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:28.669381+0000) 2019-10-13T07:13:30.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:30.674+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.675353+0000) 2019-10-13T07:13:30.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:30.677+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.677711+0000) 2019-10-13T07:13:30.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:30.677+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.677813+0000) 2019-10-13T07:13:30.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:30.677+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.678231+0000) 2019-10-13T07:13:30.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:30.678+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.678412+0000) 2019-10-13T07:13:30.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:30.678+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.678528+0000) 2019-10-13T07:13:30.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:30.678+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.678691+0000) 2019-10-13T07:13:30.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:30.678+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.679105+0000) 2019-10-13T07:13:30.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:30.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:31.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:31.034+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:31.034577+0000) 2019-10-13T07:13:31.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:31.796+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:31.797419+0000) 2019-10-13T07:13:31.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:31.924+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:31.924990+0000) 2019-10-13T07:13:32.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:32.786+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.786847+0000) 2019-10-13T07:13:32.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:32.788+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.788363+0000) 2019-10-13T07:13:35.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:35.684+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.684719+0000) 2019-10-13T07:13:35.688 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:35.684+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.685267+0000) 2019-10-13T07:13:35.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:35.685+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.685531+0000) 2019-10-13T07:13:35.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:35.685+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.685763+0000) 2019-10-13T07:13:35.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:35.685+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.685879+0000) 2019-10-13T07:13:35.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:35.682+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.683575+0000) 2019-10-13T07:13:35.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:35.685+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.686215+0000) 2019-10-13T07:13:35.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:35.684+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:35.685071+0000) 2019-10-13T07:13:35.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:35.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:38.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:38.332+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:38.332632+0000) 2019-10-13T07:13:38.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:38.669+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:38.669574+0000) 2019-10-13T07:13:38.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:38.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:38.669581+0000) 2019-10-13T07:13:40.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:40.687+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.689604+0000) 2019-10-13T07:13:40.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:40.691+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.691817+0000) 2019-10-13T07:13:40.695 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:40.691+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.692273+0000) 2019-10-13T07:13:40.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:40.692+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.692591+0000) 2019-10-13T07:13:40.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:40.692+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.692729+0000) 2019-10-13T07:13:40.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:40.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.692899+0000) 2019-10-13T07:13:40.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:40.692+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.693140+0000) 2019-10-13T07:13:40.698 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:40.690+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.692581+0000) 2019-10-13T07:13:40.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:40.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:41.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:41.034+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:41.034753+0000) 2019-10-13T07:13:41.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:41.798+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:41.799786+0000) 2019-10-13T07:13:41.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:41.925+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:41.925151+0000) 2019-10-13T07:13:42.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:42.786+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.786976+0000) 2019-10-13T07:13:42.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:42.787+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.788546+0000) 2019-10-13T07:13:42.906 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:13:42.907 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:42.911 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:13:42.911 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:45.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:45.694+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.696440+0000) 2019-10-13T07:13:45.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:45.697+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.698383+0000) 2019-10-13T07:13:45.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:45.698+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.698420+0000) 2019-10-13T07:13:45.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:45.698+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.698741+0000) 2019-10-13T07:13:45.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:45.698+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.699023+0000) 2019-10-13T07:13:45.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:45.698+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.699207+0000) 2019-10-13T07:13:45.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:45.698+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.699337+0000) 2019-10-13T07:13:45.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:45.697+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:45.699001+0000) 2019-10-13T07:13:45.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:45.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116891 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:48.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:48.331+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:48.332822+0000) 2019-10-13T07:13:48.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:48.668+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:48.669772+0000) 2019-10-13T07:13:48.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:48.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:48.669779+0000) 2019-10-13T07:13:50.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:50.702+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.704036+0000) 2019-10-13T07:13:50.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:50.703+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.704463+0000) 2019-10-13T07:13:50.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:50.700+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.702301+0000) 2019-10-13T07:13:50.708 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:50.703+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.704669+0000) 2019-10-13T07:13:50.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:50.703+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.704908+0000) 2019-10-13T07:13:50.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:50.703+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.705250+0000) 2019-10-13T07:13:50.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:50.704+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.705787+0000) 2019-10-13T07:13:50.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:50.702+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.704358+0000) 2019-10-13T07:13:50.712 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:50.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116920 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:51.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:51.033+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:51.034934+0000) 2019-10-13T07:13:51.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:51.801+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:51.802372+0000) 2019-10-13T07:13:51.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:51.924+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:51.925315+0000) 2019-10-13T07:13:52.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:52.786+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.787156+0000) 2019-10-13T07:13:52.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:52.788+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.788744+0000) 2019-10-13T07:13:55.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:13:55.708+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.709158+0000) 2019-10-13T07:13:55.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:13:55.709+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.711355+0000) 2019-10-13T07:13:55.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:55.710+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.711658+0000) 2019-10-13T07:13:55.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:13:55.710+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.711947+0000) 2019-10-13T07:13:55.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:55.710+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.712079+0000) 2019-10-13T07:13:55.716 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:55.710+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.712277+0000) 2019-10-13T07:13:55.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:13:55.711+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.712604+0000) 2019-10-13T07:13:55.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:13:55.712+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:55.713017+0000) 2019-10-13T07:13:55.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:13:55.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116952 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:13:58.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:13:58.332+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:58.333030+0000) 2019-10-13T07:13:58.674 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:13:58.669+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:58.669939+0000) 2019-10-13T07:13:58.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:13:58.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:58.669987+0000) 2019-10-13T07:14:00.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:00.715+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.716188+0000) 2019-10-13T07:14:00.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:00.716+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.718042+0000) 2019-10-13T07:14:00.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:00.716+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.718317+0000) 2019-10-13T07:14:00.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:00.717+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.718678+0000) 2019-10-13T07:14:00.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:00.717+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.718884+0000) 2019-10-13T07:14:00.723 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:00.717+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.718794+0000) 2019-10-13T07:14:00.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:00.717+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.719203+0000) 2019-10-13T07:14:00.724 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:00.718+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.719295+0000) 2019-10-13T07:14:00.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:00.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 116983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:01.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:01.034+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:01.035107+0000) 2019-10-13T07:14:01.810 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:01.804+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:01.805412+0000) 2019-10-13T07:14:01.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:01.924+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:01.925492+0000) 2019-10-13T07:14:02.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:02.786+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.787281+0000) 2019-10-13T07:14:02.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:02.788+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.788913+0000) 2019-10-13T07:14:05.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:05.720+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.721523+0000) 2019-10-13T07:14:05.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:05.722+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.723724+0000) 2019-10-13T07:14:05.728 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:05.722+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.724210+0000) 2019-10-13T07:14:05.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:05.723+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.724407+0000) 2019-10-13T07:14:05.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:05.723+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.724533+0000) 2019-10-13T07:14:05.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:05.723+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.724715+0000) 2019-10-13T07:14:05.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:05.723+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.725053+0000) 2019-10-13T07:14:05.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:05.723+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:05.723885+0000) 2019-10-13T07:14:05.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:05.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:08.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:08.332+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:08.333255+0000) 2019-10-13T07:14:08.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:08.669+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:08.670102+0000) 2019-10-13T07:14:08.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:08.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:08.670143+0000) 2019-10-13T07:14:10.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:10.727+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.728537+0000) 2019-10-13T07:14:10.735 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:10.729+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.730851+0000) 2019-10-13T07:14:10.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:10.729+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.730949+0000) 2019-10-13T07:14:10.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:10.729+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.731137+0000) 2019-10-13T07:14:10.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:10.730+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.731391+0000) 2019-10-13T07:14:10.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:10.730+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.731590+0000) 2019-10-13T07:14:10.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:10.730+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.732097+0000) 2019-10-13T07:14:10.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:10.730+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.731615+0000) 2019-10-13T07:14:10.740 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:10.734+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:11.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:11.034+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:11.035253+0000) 2019-10-13T07:14:11.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:11.807+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:11.807826+0000) 2019-10-13T07:14:11.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:11.925+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:11.925714+0000) 2019-10-13T07:14:12.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:12.787+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.787477+0000) 2019-10-13T07:14:12.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:12.788+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.789130+0000) 2019-10-13T07:14:13.111 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:14:13.112 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:13.116 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:14:13.116 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:15.741 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:15.737+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.737592+0000) 2019-10-13T07:14:15.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:15.737+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.737839+0000) 2019-10-13T07:14:15.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:15.737+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.737892+0000) 2019-10-13T07:14:15.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:15.737+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.738051+0000) 2019-10-13T07:14:15.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:15.735+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.736423+0000) 2019-10-13T07:14:15.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:15.738+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.738668+0000) 2019-10-13T07:14:15.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:15.738+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.738976+0000) 2019-10-13T07:14:15.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:15.738+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:15.739527+0000) 2019-10-13T07:14:15.746 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:15.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:18.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:18.333+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:18.333486+0000) 2019-10-13T07:14:18.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:18.669+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:18.670279+0000) 2019-10-13T07:14:18.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:18.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:18.670303+0000) 2019-10-13T07:14:20.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:20.744+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.745141+0000) 2019-10-13T07:14:20.749 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:20.744+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.745360+0000) 2019-10-13T07:14:20.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:20.745+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.745590+0000) 2019-10-13T07:14:20.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:20.745+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.745555+0000) 2019-10-13T07:14:20.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:20.743+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.743753+0000) 2019-10-13T07:14:20.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:20.745+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.746059+0000) 2019-10-13T07:14:20.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:20.745+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.746230+0000) 2019-10-13T07:14:20.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:20.744+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.745316+0000) 2019-10-13T07:14:20.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:20.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:21.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:21.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:21.035419+0000) 2019-10-13T07:14:21.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:21.809+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:21.810170+0000) 2019-10-13T07:14:21.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:21.925+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:21.925909+0000) 2019-10-13T07:14:22.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:22.787+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.787656+0000) 2019-10-13T07:14:22.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:22.789+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.789309+0000) 2019-10-13T07:14:25.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:25.751+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.751661+0000) 2019-10-13T07:14:25.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:25.751+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.751858+0000) 2019-10-13T07:14:25.755 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:25.751+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.751889+0000) 2019-10-13T07:14:25.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:25.751+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.752115+0000) 2019-10-13T07:14:25.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:25.751+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.752371+0000) 2019-10-13T07:14:25.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:25.752+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.752435+0000) 2019-10-13T07:14:25.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:25.749+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.750723+0000) 2019-10-13T07:14:25.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:25.751+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:25.752392+0000) 2019-10-13T07:14:25.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:25.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:28.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:28.333+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:28.333682+0000) 2019-10-13T07:14:28.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:28.670+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:28.670499+0000) 2019-10-13T07:14:28.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:28.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:28.670509+0000) 2019-10-13T07:14:30.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:30.754+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.754816+0000) 2019-10-13T07:14:30.760 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:30.756+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.757337+0000) 2019-10-13T07:14:30.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:30.757+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.757475+0000) 2019-10-13T07:14:30.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:30.757+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.757618+0000) 2019-10-13T07:14:30.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:30.757+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.757932+0000) 2019-10-13T07:14:30.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:30.757+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.758068+0000) 2019-10-13T07:14:30.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:30.756+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.757146+0000) 2019-10-13T07:14:30.762 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:30.757+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.758345+0000) 2019-10-13T07:14:30.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:30.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:31.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:31.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:31.035590+0000) 2019-10-13T07:14:31.817 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:31.810+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:31.812517+0000) 2019-10-13T07:14:31.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:31.926+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:31.926098+0000) 2019-10-13T07:14:32.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:32.786+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.787849+0000) 2019-10-13T07:14:32.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:32.788+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.789471+0000) 2019-10-13T07:14:35.766 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:35.762+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.763556+0000) 2019-10-13T07:14:35.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:35.760+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.761828+0000) 2019-10-13T07:14:35.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:35.762+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.764188+0000) 2019-10-13T07:14:35.768 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:35.762+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.764012+0000) 2019-10-13T07:14:35.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:35.763+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.764697+0000) 2019-10-13T07:14:35.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:35.762+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.764343+0000) 2019-10-13T07:14:35.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:35.763+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.765037+0000) 2019-10-13T07:14:35.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:35.762+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:35.764697+0000) 2019-10-13T07:14:35.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:35.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:38.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:38.332+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:38.333846+0000) 2019-10-13T07:14:38.674 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:38.669+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:38.670735+0000) 2019-10-13T07:14:38.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:38.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:38.670732+0000) 2019-10-13T07:14:40.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:40.768+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.770032+0000) 2019-10-13T07:14:40.773 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:40.768+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.770384+0000) 2019-10-13T07:14:40.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:40.769+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.770563+0000) 2019-10-13T07:14:40.774 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:40.769+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.770751+0000) 2019-10-13T07:14:40.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:40.769+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.770839+0000) 2019-10-13T07:14:40.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:40.770+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.771481+0000) 2019-10-13T07:14:40.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:40.767+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.769291+0000) 2019-10-13T07:14:40.776 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:40.769+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.770796+0000) 2019-10-13T07:14:40.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:40.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:41.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:41.034+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:41.035753+0000) 2019-10-13T07:14:41.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:41.814+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:41.815096+0000) 2019-10-13T07:14:41.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:41.925+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:41.926301+0000) 2019-10-13T07:14:42.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:42.787+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.788034+0000) 2019-10-13T07:14:42.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:42.788+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.789659+0000) 2019-10-13T07:14:43.317 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:14:43.317 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:43.322 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:14:43.322 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:45.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:45.774+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.775726+0000) 2019-10-13T07:14:45.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:45.774+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.775860+0000) 2019-10-13T07:14:45.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:45.774+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.776017+0000) 2019-10-13T07:14:45.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:45.773+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.774100+0000) 2019-10-13T07:14:45.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:45.774+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.776320+0000) 2019-10-13T07:14:45.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:45.775+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.776498+0000) 2019-10-13T07:14:45.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:45.775+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.777074+0000) 2019-10-13T07:14:45.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:45.776+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:45.776754+0000) 2019-10-13T07:14:45.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:45.778+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117263 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:48.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:48.333+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:48.334060+0000) 2019-10-13T07:14:48.674 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:48.670+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:48.670931+0000) 2019-10-13T07:14:48.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:48.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:48.670929+0000) 2019-10-13T07:14:50.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:50.778+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.779710+0000) 2019-10-13T07:14:50.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:50.780+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.782030+0000) 2019-10-13T07:14:50.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:50.780+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.782254+0000) 2019-10-13T07:14:50.786 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:50.781+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.782484+0000) 2019-10-13T07:14:50.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:50.781+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.782736+0000) 2019-10-13T07:14:50.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:50.781+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.782902+0000) 2019-10-13T07:14:50.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:50.781+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.783153+0000) 2019-10-13T07:14:50.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:50.782+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.783034+0000) 2019-10-13T07:14:50.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:50.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117292 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:51.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:51.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:51.035895+0000) 2019-10-13T07:14:51.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:51.816+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:51.817563+0000) 2019-10-13T07:14:51.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:51.925+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:51.926447+0000) 2019-10-13T07:14:52.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:52.787+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.788230+0000) 2019-10-13T07:14:52.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:52.789+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.789883+0000) 2019-10-13T07:14:55.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:14:55.785+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.786425+0000) 2019-10-13T07:14:55.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:14:55.787+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.788808+0000) 2019-10-13T07:14:55.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:55.787+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.788967+0000) 2019-10-13T07:14:55.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:14:55.787+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.789076+0000) 2019-10-13T07:14:55.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:14:55.787+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.789284+0000) 2019-10-13T07:14:55.793 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:55.788+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.789459+0000) 2019-10-13T07:14:55.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:55.788+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.789652+0000) 2019-10-13T07:14:55.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:14:55.789+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:55.789964+0000) 2019-10-13T07:14:55.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:14:55.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:14:58.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:14:58.333+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:58.334279+0000) 2019-10-13T07:14:58.674 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:14:58.670+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:58.671127+0000) 2019-10-13T07:14:58.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:14:58.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:58.671125+0000) 2019-10-13T07:15:00.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:00.792+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.793236+0000) 2019-10-13T07:15:00.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:00.794+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.794936+0000) 2019-10-13T07:15:00.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:00.794+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.795131+0000) 2019-10-13T07:15:00.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:00.795+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.795531+0000) 2019-10-13T07:15:00.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:00.795+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.795749+0000) 2019-10-13T07:15:00.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:00.795+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.795867+0000) 2019-10-13T07:15:00.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:00.795+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.796255+0000) 2019-10-13T07:15:00.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:00.795+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.796334+0000) 2019-10-13T07:15:00.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:00.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:01.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:01.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:01.036029+0000) 2019-10-13T07:15:01.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:01.819+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:01.819872+0000) 2019-10-13T07:15:01.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:01.926+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:01.926621+0000) 2019-10-13T07:15:02.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:02.788+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.788426+0000) 2019-10-13T07:15:02.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:02.789+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.790067+0000) 2019-10-13T07:15:05.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:05.800+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.800486+0000) 2019-10-13T07:15:05.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:05.800+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.800584+0000) 2019-10-13T07:15:05.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:05.800+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.800884+0000) 2019-10-13T07:15:05.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:05.800+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.801090+0000) 2019-10-13T07:15:05.805 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:05.800+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.801259+0000) 2019-10-13T07:15:05.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:05.801+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.801678+0000) 2019-10-13T07:15:05.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:05.799+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.799889+0000) 2019-10-13T07:15:05.806 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:05.800+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:05.800909+0000) 2019-10-13T07:15:05.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:05.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:08.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:08.334+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:08.334446+0000) 2019-10-13T07:15:08.674 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:08.671+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:08.671347+0000) 2019-10-13T07:15:08.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:08.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:08.671324+0000) 2019-10-13T07:15:10.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:10.806+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.807045+0000) 2019-10-13T07:15:10.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:10.806+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.807388+0000) 2019-10-13T07:15:10.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:10.807+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.807548+0000) 2019-10-13T07:15:10.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:10.807+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.807694+0000) 2019-10-13T07:15:10.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:10.805+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.805866+0000) 2019-10-13T07:15:10.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:10.807+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.807957+0000) 2019-10-13T07:15:10.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:10.807+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.808152+0000) 2019-10-13T07:15:10.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:10.806+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.807012+0000) 2019-10-13T07:15:10.814 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:10.811+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117416 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:11.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:11.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:11.036177+0000) 2019-10-13T07:15:11.827 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:11.821+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:11.822407+0000) 2019-10-13T07:15:11.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:11.926+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:11.926812+0000) 2019-10-13T07:15:12.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:12.788+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.788631+0000) 2019-10-13T07:15:12.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:12.789+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.790214+0000) 2019-10-13T07:15:13.522 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:15:13.523 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:13.527 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:15:13.527 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:15.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:15.812+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.812942+0000) 2019-10-13T07:15:15.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:15.812+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.813234+0000) 2019-10-13T07:15:15.817 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:15.812+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.813279+0000) 2019-10-13T07:15:15.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:15.811+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.811968+0000) 2019-10-13T07:15:15.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:15.813+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.813633+0000) 2019-10-13T07:15:15.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:15.813+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.813855+0000) 2019-10-13T07:15:15.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:15.813+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.814001+0000) 2019-10-13T07:15:15.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:15.813+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:15.814163+0000) 2019-10-13T07:15:15.820 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:15.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:18.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:18.333+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:18.334622+0000) 2019-10-13T07:15:18.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:18.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:18.671527+0000) 2019-10-13T07:15:18.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:18.670+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:18.671528+0000) 2019-10-13T07:15:20.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:20.816+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.816758+0000) 2019-10-13T07:15:20.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:20.817+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.819011+0000) 2019-10-13T07:15:20.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:20.817+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.819311+0000) 2019-10-13T07:15:20.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:20.818+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.819664+0000) 2019-10-13T07:15:20.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:20.818+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.819863+0000) 2019-10-13T07:15:20.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:20.818+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.820137+0000) 2019-10-13T07:15:20.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:20.819+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.820558+0000) 2019-10-13T07:15:20.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:20.819+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.820280+0000) 2019-10-13T07:15:20.826 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:20.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117479 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:21.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:21.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:21.036340+0000) 2019-10-13T07:15:21.830 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:21.824+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:21.824891+0000) 2019-10-13T07:15:21.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:21.926+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:21.927013+0000) 2019-10-13T07:15:22.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:22.787+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.788836+0000) 2019-10-13T07:15:22.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:22.790+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.790363+0000) 2019-10-13T07:15:25.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:25.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.823024+0000) 2019-10-13T07:15:25.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:25.823+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.825209+0000) 2019-10-13T07:15:25.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:25.824+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.825590+0000) 2019-10-13T07:15:25.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:25.824+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.825850+0000) 2019-10-13T07:15:25.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:25.824+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.825916+0000) 2019-10-13T07:15:25.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:25.824+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.826275+0000) 2019-10-13T07:15:25.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:25.825+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.826617+0000) 2019-10-13T07:15:25.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:25.825+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:25.826762+0000) 2019-10-13T07:15:25.833 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:25.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117511 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:28.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:28.333+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:28.334779+0000) 2019-10-13T07:15:28.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:28.670+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:28.671697+0000) 2019-10-13T07:15:28.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:28.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:28.671697+0000) 2019-10-13T07:15:30.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:30.828+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.829841+0000) 2019-10-13T07:15:30.835 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:30.830+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.832051+0000) 2019-10-13T07:15:30.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:30.831+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.832632+0000) 2019-10-13T07:15:30.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:30.831+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.832790+0000) 2019-10-13T07:15:30.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:30.831+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.833048+0000) 2019-10-13T07:15:30.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:30.830+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.832362+0000) 2019-10-13T07:15:30.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:30.831+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.833339+0000) 2019-10-13T07:15:30.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:30.832+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.833583+0000) 2019-10-13T07:15:30.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:30.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:31.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:31.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:31.036547+0000) 2019-10-13T07:15:31.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:31.826+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:31.827043+0000) 2019-10-13T07:15:31.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:31.925+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:31.927212+0000) 2019-10-13T07:15:32.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:32.788+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.789058+0000) 2019-10-13T07:15:32.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:32.789+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.790550+0000) 2019-10-13T07:15:35.842 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:35.835+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.836977+0000) 2019-10-13T07:15:35.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:35.837+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.839103+0000) 2019-10-13T07:15:35.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:35.837+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.839318+0000) 2019-10-13T07:15:35.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:35.838+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.839545+0000) 2019-10-13T07:15:35.844 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:35.838+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.839697+0000) 2019-10-13T07:15:35.844 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:35.838+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.840165+0000) 2019-10-13T07:15:35.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:35.839+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.840528+0000) 2019-10-13T07:15:35.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:35.838+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:35.840115+0000) 2019-10-13T07:15:35.847 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:35.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117572 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:38.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:38.333+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:38.334952+0000) 2019-10-13T07:15:38.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:38.671+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:38.671874+0000) 2019-10-13T07:15:38.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:38.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:38.671881+0000) 2019-10-13T07:15:40.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:40.843+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.843896+0000) 2019-10-13T07:15:40.850 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:40.844+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.846020+0000) 2019-10-13T07:15:40.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:40.844+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.846358+0000) 2019-10-13T07:15:40.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:40.845+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.846418+0000) 2019-10-13T07:15:40.851 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:40.845+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.846561+0000) 2019-10-13T07:15:40.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:40.845+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.847016+0000) 2019-10-13T07:15:40.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:40.845+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.847188+0000) 2019-10-13T07:15:40.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:40.846+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.846912+0000) 2019-10-13T07:15:40.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:40.849+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:41.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:41.036+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:41.036668+0000) 2019-10-13T07:15:41.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:41.828+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:41.829474+0000) 2019-10-13T07:15:41.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:41.926+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:41.927424+0000) 2019-10-13T07:15:42.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:42.788+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.789196+0000) 2019-10-13T07:15:42.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:42.790+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.790752+0000) 2019-10-13T07:15:43.637 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:15:43.637 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:43.642 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:15:43.642 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:45.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:45.850+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.852342+0000) 2019-10-13T07:15:45.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:45.851+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.852525+0000) 2019-10-13T07:15:45.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:45.851+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.853018+0000) 2019-10-13T07:15:45.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:45.851+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.853091+0000) 2019-10-13T07:15:45.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:45.851+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.853163+0000) 2019-10-13T07:15:45.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:45.852+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.853638+0000) 2019-10-13T07:15:45.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:45.851+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.851869+0000) 2019-10-13T07:15:45.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:45.853+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:45.854131+0000) 2019-10-13T07:15:45.860 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:45.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:48.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:48.334+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:48.335111+0000) 2019-10-13T07:15:48.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:48.671+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:48.672079+0000) 2019-10-13T07:15:48.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:48.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:48.672081+0000) 2019-10-13T07:15:50.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:50.859+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.859578+0000) 2019-10-13T07:15:50.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:50.859+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.859856+0000) 2019-10-13T07:15:50.864 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:50.859+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.860184+0000) 2019-10-13T07:15:50.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:50.859+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.860285+0000) 2019-10-13T07:15:50.864 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:50.860+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.860441+0000) 2019-10-13T07:15:50.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:50.857+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.858168+0000) 2019-10-13T07:15:50.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:50.860+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.860974+0000) 2019-10-13T07:15:50.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:50.859+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.860675+0000) 2019-10-13T07:15:50.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:50.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:51.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:51.036+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:51.036816+0000) 2019-10-13T07:15:51.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:51.831+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:51.831810+0000) 2019-10-13T07:15:51.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:51.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:51.927611+0000) 2019-10-13T07:15:52.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:52.788+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.789361+0000) 2019-10-13T07:15:52.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:52.790+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.790893+0000) 2019-10-13T07:15:55.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:55.866+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.867047+0000) 2019-10-13T07:15:55.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:15:55.864+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.865388+0000) 2019-10-13T07:15:55.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:15:55.867+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.867507+0000) 2019-10-13T07:15:55.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:55.867+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.867629+0000) 2019-10-13T07:15:55.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:15:55.867+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.867853+0000) 2019-10-13T07:15:55.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:55.867+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.867991+0000) 2019-10-13T07:15:55.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:15:55.867+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.868166+0000) 2019-10-13T07:15:55.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:15:55.869+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:15:55.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:15:55.867+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:55.868599+0000) 2019-10-13T07:15:58.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:15:58.335+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:58.335271+0000) 2019-10-13T07:15:58.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:15:58.672+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:58.672252+0000) 2019-10-13T07:15:58.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:15:58.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:58.672254+0000) 2019-10-13T07:16:00.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:00.871+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.871612+0000) 2019-10-13T07:16:00.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:00.871+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.871954+0000) 2019-10-13T07:16:00.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:00.869+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.870500+0000) 2019-10-13T07:16:00.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:00.871+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.872329+0000) 2019-10-13T07:16:00.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:00.872+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.872730+0000) 2019-10-13T07:16:00.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:00.872+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.872831+0000) 2019-10-13T07:16:00.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:00.872+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.873168+0000) 2019-10-13T07:16:00.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:00.872+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.873205+0000) 2019-10-13T07:16:00.879 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:00.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:01.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:01.036+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:01.036937+0000) 2019-10-13T07:16:01.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:01.833+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:01.834108+0000) 2019-10-13T07:16:01.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:01.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:01.927778+0000) 2019-10-13T07:16:02.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:02.789+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.789583+0000) 2019-10-13T07:16:02.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:02.790+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.791080+0000) 2019-10-13T07:16:05.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:05.876+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.877331+0000) 2019-10-13T07:16:05.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:05.878+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.879602+0000) 2019-10-13T07:16:05.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:05.878+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.879858+0000) 2019-10-13T07:16:05.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:05.878+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.880040+0000) 2019-10-13T07:16:05.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:05.879+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.880463+0000) 2019-10-13T07:16:05.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:05.879+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.880606+0000) 2019-10-13T07:16:05.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:05.879+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.880729+0000) 2019-10-13T07:16:05.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:05.879+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:05.879814+0000) 2019-10-13T07:16:05.886 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:05.882+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117759 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:08.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:08.334+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:08.335458+0000) 2019-10-13T07:16:08.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:08.671+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:08.672431+0000) 2019-10-13T07:16:08.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:08.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:08.672432+0000) 2019-10-13T07:16:10.888 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:10.883+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.885309+0000) 2019-10-13T07:16:10.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:10.884+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.885579+0000) 2019-10-13T07:16:10.889 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:10.884+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.885732+0000) 2019-10-13T07:16:10.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:10.883+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.883920+0000) 2019-10-13T07:16:10.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:10.885+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.886506+0000) 2019-10-13T07:16:10.890 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:10.885+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.886767+0000) 2019-10-13T07:16:10.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:10.885+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.886955+0000) 2019-10-13T07:16:10.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:10.885+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.886701+0000) 2019-10-13T07:16:10.893 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:10.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:11.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:11.036+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:11.037092+0000) 2019-10-13T07:16:11.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:11.836+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:11.836575+0000) 2019-10-13T07:16:11.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:11.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:11.927925+0000) 2019-10-13T07:16:12.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:12.788+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.789786+0000) 2019-10-13T07:16:12.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:12.790+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.791280+0000) 2019-10-13T07:16:13.742 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:16:13.742 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:13.746 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:16:13.747 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:15.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:15.890+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.892172+0000) 2019-10-13T07:16:15.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:15.889+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.890527+0000) 2019-10-13T07:16:15.896 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:15.891+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.892848+0000) 2019-10-13T07:16:15.896 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:15.891+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.893041+0000) 2019-10-13T07:16:15.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:15.891+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.893125+0000) 2019-10-13T07:16:15.897 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:15.891+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.893278+0000) 2019-10-13T07:16:15.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:15.892+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.893461+0000) 2019-10-13T07:16:15.898 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:15.892+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:15.892784+0000) 2019-10-13T07:16:15.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:15.895+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:18.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:18.334+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:18.335654+0000) 2019-10-13T07:16:18.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:18.671+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:18.672604+0000) 2019-10-13T07:16:18.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:18.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:18.672621+0000) 2019-10-13T07:16:20.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:20.897+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.898689+0000) 2019-10-13T07:16:20.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:20.897+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.898896+0000) 2019-10-13T07:16:20.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:20.897+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.898931+0000) 2019-10-13T07:16:20.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:20.896+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.897941+0000) 2019-10-13T07:16:20.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:20.898+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.899430+0000) 2019-10-13T07:16:20.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:20.898+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.899605+0000) 2019-10-13T07:16:20.904 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:20.898+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.899746+0000) 2019-10-13T07:16:20.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:20.898+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.900254+0000) 2019-10-13T07:16:20.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:20.901+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:21.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:21.035+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:21.037284+0000) 2019-10-13T07:16:21.843 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:21.837+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:21.838719+0000) 2019-10-13T07:16:21.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:21.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:21.928090+0000) 2019-10-13T07:16:22.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:22.789+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.789970+0000) 2019-10-13T07:16:22.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:22.790+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.791464+0000) 2019-10-13T07:16:25.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:25.903+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.905079+0000) 2019-10-13T07:16:25.909 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:25.903+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.905243+0000) 2019-10-13T07:16:25.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:25.904+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.905487+0000) 2019-10-13T07:16:25.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:25.904+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.905832+0000) 2019-10-13T07:16:25.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:25.903+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.904882+0000) 2019-10-13T07:16:25.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:25.904+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.906182+0000) 2019-10-13T07:16:25.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:25.905+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.906531+0000) 2019-10-13T07:16:25.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:25.904+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:25.905945+0000) 2019-10-13T07:16:25.913 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:25.908+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117883 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:28.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:28.335+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:28.335827+0000) 2019-10-13T07:16:28.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:28.672+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:28.672799+0000) 2019-10-13T07:16:28.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:28.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:28.672821+0000) 2019-10-13T07:16:30.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:30.910+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.911467+0000) 2019-10-13T07:16:30.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:30.910+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.911763+0000) 2019-10-13T07:16:30.915 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:30.910+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.911876+0000) 2019-10-13T07:16:30.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:30.908+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.910071+0000) 2019-10-13T07:16:30.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:30.910+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.912279+0000) 2019-10-13T07:16:30.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:30.911+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.912431+0000) 2019-10-13T07:16:30.916 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:30.911+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.912607+0000) 2019-10-13T07:16:30.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:30.910+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.912280+0000) 2019-10-13T07:16:30.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:30.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:31.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:31.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:31.037464+0000) 2019-10-13T07:16:31.846 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:31.840+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:31.841127+0000) 2019-10-13T07:16:31.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:31.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:31.928208+0000) 2019-10-13T07:16:32.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:32.789+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.790130+0000) 2019-10-13T07:16:32.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:32.790+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.791695+0000) 2019-10-13T07:16:35.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:35.914+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.915054+0000) 2019-10-13T07:16:35.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:35.916+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.917073+0000) 2019-10-13T07:16:35.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:35.916+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.917175+0000) 2019-10-13T07:16:35.921 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:35.917+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.917504+0000) 2019-10-13T07:16:35.921 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:35.917+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.917819+0000) 2019-10-13T07:16:35.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:35.917+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.918106+0000) 2019-10-13T07:16:35.922 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:35.917+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.918205+0000) 2019-10-13T07:16:35.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:35.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:35.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:35.917+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:35.918529+0000) 2019-10-13T07:16:38.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:38.335+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:38.336000+0000) 2019-10-13T07:16:38.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:38.672+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:38.672978+0000) 2019-10-13T07:16:38.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:38.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:38.673000+0000) 2019-10-13T07:16:40.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:40.923+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.923761+0000) 2019-10-13T07:16:40.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:40.923+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.923895+0000) 2019-10-13T07:16:40.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:40.923+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.924287+0000) 2019-10-13T07:16:40.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:40.921+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.922087+0000) 2019-10-13T07:16:40.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:40.924+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.924487+0000) 2019-10-13T07:16:40.928 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:40.924+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.924785+0000) 2019-10-13T07:16:40.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:40.924+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.925022+0000) 2019-10-13T07:16:40.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:40.924+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.924811+0000) 2019-10-13T07:16:40.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:40.928+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 117975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:41.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:41.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:41.037618+0000) 2019-10-13T07:16:41.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:41.842+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:41.843617+0000) 2019-10-13T07:16:41.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:41.928+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:41.928377+0000) 2019-10-13T07:16:42.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:42.790+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.790331+0000) 2019-10-13T07:16:42.796 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:42.791+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.791928+0000) 2019-10-13T07:16:43.839 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:16:43.845 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:43.850 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:16:43.850 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:45.934 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:45.930+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.930547+0000) 2019-10-13T07:16:45.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:45.930+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.930735+0000) 2019-10-13T07:16:45.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:45.930+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.930785+0000) 2019-10-13T07:16:45.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:45.930+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.930925+0000) 2019-10-13T07:16:45.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:45.930+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.931137+0000) 2019-10-13T07:16:45.935 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:45.931+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.931470+0000) 2019-10-13T07:16:45.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:45.928+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.929352+0000) 2019-10-13T07:16:45.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:45.931+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:45.931874+0000) 2019-10-13T07:16:45.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:45.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118007 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:48.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:48.336+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:48.336200+0000) 2019-10-13T07:16:48.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:48.673+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:48.673162+0000) 2019-10-13T07:16:48.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:48.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:48.673194+0000) 2019-10-13T07:16:50.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:50.934+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.936118+0000) 2019-10-13T07:16:50.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:50.935+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.936461+0000) 2019-10-13T07:16:50.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:50.933+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.934629+0000) 2019-10-13T07:16:50.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:50.934+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.936344+0000) 2019-10-13T07:16:50.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:50.935+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.936936+0000) 2019-10-13T07:16:50.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:50.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.936811+0000) 2019-10-13T07:16:50.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:50.935+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.937145+0000) 2019-10-13T07:16:50.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:50.936+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.937435+0000) 2019-10-13T07:16:50.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:50.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:51.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:51.036+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:51.037792+0000) 2019-10-13T07:16:51.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:51.845+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:51.846064+0000) 2019-10-13T07:16:51.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:51.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:51.928586+0000) 2019-10-13T07:16:52.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:52.789+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.790525+0000) 2019-10-13T07:16:52.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:52.791+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.792112+0000) 2019-10-13T07:16:55.946 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:16:55.941+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.943352+0000) 2019-10-13T07:16:55.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:16:55.942+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.943465+0000) 2019-10-13T07:16:55.947 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:55.942+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.943761+0000) 2019-10-13T07:16:55.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:16:55.940+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.941682+0000) 2019-10-13T07:16:55.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:55.942+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.944028+0000) 2019-10-13T07:16:55.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:55.942+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.944362+0000) 2019-10-13T07:16:55.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:16:55.942+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.943864+0000) 2019-10-13T07:16:55.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:16:55.943+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:55.944680+0000) 2019-10-13T07:16:55.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:16:55.947+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:16:58.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:16:58.335+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:58.336431+0000) 2019-10-13T07:16:58.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:16:58.672+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:58.673344+0000) 2019-10-13T07:16:58.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:16:58.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:58.673393+0000) 2019-10-13T07:17:00.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:00.948+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.950185+0000) 2019-10-13T07:17:00.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:00.949+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.950644+0000) 2019-10-13T07:17:00.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:00.949+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.950810+0000) 2019-10-13T07:17:00.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:00.948+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.949211+0000) 2019-10-13T07:17:00.955 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:00.949+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.951011+0000) 2019-10-13T07:17:00.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:00.949+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.951072+0000) 2019-10-13T07:17:00.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:00.950+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.951886+0000) 2019-10-13T07:17:00.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:00.949+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.950532+0000) 2019-10-13T07:17:00.957 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:00.953+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:01.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:01.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:01.037935+0000) 2019-10-13T07:17:01.853 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:01.848+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:01.848423+0000) 2019-10-13T07:17:01.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:01.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:01.928755+0000) 2019-10-13T07:17:02.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:02.789+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.790695+0000) 2019-10-13T07:17:02.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:02.792+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.792294+0000) 2019-10-13T07:17:05.960 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:05.955+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.956898+0000) 2019-10-13T07:17:05.961 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:05.955+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.957139+0000) 2019-10-13T07:17:05.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:05.956+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.957423+0000) 2019-10-13T07:17:05.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:05.956+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.957571+0000) 2019-10-13T07:17:05.961 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:05.956+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.957734+0000) 2019-10-13T07:17:05.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:05.954+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.954795+0000) 2019-10-13T07:17:05.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:05.956+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.957989+0000) 2019-10-13T07:17:05.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:05.957+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:05.958103+0000) 2019-10-13T07:17:05.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:05.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:08.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:08.335+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:08.336629+0000) 2019-10-13T07:17:08.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:08.672+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:08.673553+0000) 2019-10-13T07:17:08.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:08.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:08.673562+0000) 2019-10-13T07:17:10.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:10.959+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.961353+0000) 2019-10-13T07:17:10.967 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:10.962+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.963590+0000) 2019-10-13T07:17:10.967 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:10.962+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.963472+0000) 2019-10-13T07:17:10.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:10.962+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.963756+0000) 2019-10-13T07:17:10.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:10.962+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.963849+0000) 2019-10-13T07:17:10.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:10.962+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.963963+0000) 2019-10-13T07:17:10.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:10.962+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.964104+0000) 2019-10-13T07:17:10.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:10.961+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.963190+0000) 2019-10-13T07:17:10.970 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:10.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118160 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:11.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:11.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:11.038049+0000) 2019-10-13T07:17:11.855 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:11.849+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:11.850807+0000) 2019-10-13T07:17:11.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:11.927+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:11.928955+0000) 2019-10-13T07:17:12.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:12.790+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.790887+0000) 2019-10-13T07:17:12.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:12.791+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.792448+0000) 2019-10-13T07:17:13.936 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:17:13.936 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:13.941 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:17:13.941 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:15.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:15.967+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.968841+0000) 2019-10-13T07:17:15.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:15.967+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.969145+0000) 2019-10-13T07:17:15.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:15.967+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.969312+0000) 2019-10-13T07:17:15.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:15.965+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.967511+0000) 2019-10-13T07:17:15.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:15.968+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.969770+0000) 2019-10-13T07:17:15.974 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:15.968+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.969601+0000) 2019-10-13T07:17:15.974 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:15.969+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.970390+0000) 2019-10-13T07:17:15.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:15.969+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:15.971220+0000) 2019-10-13T07:17:15.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:15.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:18.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:18.336+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:18.336847+0000) 2019-10-13T07:17:18.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:18.673+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:18.673756+0000) 2019-10-13T07:17:18.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:18.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:18.673762+0000) 2019-10-13T07:17:20.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:20.971+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.973723+0000) 2019-10-13T07:17:20.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:20.975+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.975804+0000) 2019-10-13T07:17:20.980 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:20.975+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.976014+0000) 2019-10-13T07:17:20.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:20.975+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.976119+0000) 2019-10-13T07:17:20.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:20.975+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.976314+0000) 2019-10-13T07:17:20.980 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:20.976+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.976646+0000) 2019-10-13T07:17:20.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:20.976+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.977029+0000) 2019-10-13T07:17:20.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:20.973+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.975709+0000) 2019-10-13T07:17:20.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:20.979+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:21.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:21.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:21.038201+0000) 2019-10-13T07:17:21.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:21.852+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:21.852991+0000) 2019-10-13T07:17:21.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:21.928+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:21.929123+0000) 2019-10-13T07:17:22.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:22.790+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.791038+0000) 2019-10-13T07:17:22.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:22.791+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.792592+0000) 2019-10-13T07:17:25.989 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:25.981+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.982265+0000) 2019-10-13T07:17:25.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:25.982+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.982680+0000) 2019-10-13T07:17:25.990 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:25.982+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.982888+0000) 2019-10-13T07:17:25.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:25.982+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.983014+0000) 2019-10-13T07:17:25.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:25.982+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.983150+0000) 2019-10-13T07:17:25.990 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:25.982+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.983334+0000) 2019-10-13T07:17:25.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:25.979+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.980120+0000) 2019-10-13T07:17:25.991 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:25.981+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:25.982635+0000) 2019-10-13T07:17:25.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:25.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:28.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:28.336+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:28.336999+0000) 2019-10-13T07:17:28.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:28.673+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:28.673954+0000) 2019-10-13T07:17:28.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:28.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:28.673961+0000) 2019-10-13T07:17:30.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:30.988+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.988712+0000) 2019-10-13T07:17:30.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:30.988+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.988808+0000) 2019-10-13T07:17:30.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:30.988+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.988968+0000) 2019-10-13T07:17:30.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:30.988+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.989305+0000) 2019-10-13T07:17:30.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:30.986+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.987164+0000) 2019-10-13T07:17:30.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:30.989+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.989750+0000) 2019-10-13T07:17:30.993 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:30.989+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.989842+0000) 2019-10-13T07:17:30.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:30.989+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.990072+0000) 2019-10-13T07:17:30.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:30.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:31.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:31.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:31.038378+0000) 2019-10-13T07:17:31.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:31.854+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:31.855359+0000) 2019-10-13T07:17:31.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:31.929+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:31.929297+0000) 2019-10-13T07:17:32.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:32.790+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.791244+0000) 2019-10-13T07:17:32.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:32.792+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.792806+0000) 2019-10-13T07:17:35.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:35.995+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.996123+0000) 2019-10-13T07:17:35.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:35.995+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.996348+0000) 2019-10-13T07:17:36.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:35.994+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.995149+0000) 2019-10-13T07:17:36.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:35.996+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.996786+0000) 2019-10-13T07:17:36.001 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:35.996+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.997193+0000) 2019-10-13T07:17:36.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:35.996+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.997319+0000) 2019-10-13T07:17:36.001 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:35.997+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.997470+0000) 2019-10-13T07:17:36.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:35.996+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:35.997433+0000) 2019-10-13T07:17:36.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:35.999+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:38.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:38.336+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:38.337162+0000) 2019-10-13T07:17:38.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:38.673+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:38.674122+0000) 2019-10-13T07:17:38.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:38.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:38.674158+0000) 2019-10-13T07:17:41.005 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:41.000+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.001800+0000) 2019-10-13T07:17:41.005 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:41.000+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.001919+0000) 2019-10-13T07:17:41.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:41.000+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.002092+0000) 2019-10-13T07:17:41.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:41.000+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.002166+0000) 2019-10-13T07:17:41.006 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:41.001+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.002498+0000) 2019-10-13T07:17:41.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:41.000+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.000977+0000) 2019-10-13T07:17:41.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:41.001+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.002720+0000) 2019-10-13T07:17:41.008 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:41.002+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.002937+0000) 2019-10-13T07:17:41.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:41.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118346 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:41.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:41.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.038565+0000) 2019-10-13T07:17:41.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:41.857+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.857767+0000) 2019-10-13T07:17:41.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:41.928+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.929487+0000) 2019-10-13T07:17:42.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:42.790+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.791450+0000) 2019-10-13T07:17:42.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:42.792+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.792938+0000) 2019-10-13T07:17:44.043 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:17:44.043 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:44.048 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:17:44.048 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:46.011 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:46.006+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.007770+0000) 2019-10-13T07:17:46.011 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:46.006+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.007974+0000) 2019-10-13T07:17:46.011 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:46.006+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.008219+0000) 2019-10-13T07:17:46.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:46.006+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.006918+0000) 2019-10-13T07:17:46.013 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:46.007+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.008459+0000) 2019-10-13T07:17:46.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:46.007+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.008628+0000) 2019-10-13T07:17:46.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:46.007+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.008546+0000) 2019-10-13T07:17:46.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:46.008+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:46.009174+0000) 2019-10-13T07:17:46.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:46.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118377 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:48.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:48.336+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:48.337333+0000) 2019-10-13T07:17:48.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:48.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:48.674321+0000) 2019-10-13T07:17:48.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:48.673+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:48.674323+0000) 2019-10-13T07:17:51.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:51.012+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.014383+0000) 2019-10-13T07:17:51.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:51.013+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.014598+0000) 2019-10-13T07:17:51.018 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:51.013+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.014683+0000) 2019-10-13T07:17:51.018 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:51.013+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.014880+0000) 2019-10-13T07:17:51.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:51.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.013102+0000) 2019-10-13T07:17:51.019 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:51.014+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.015517+0000) 2019-10-13T07:17:51.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:51.014+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.015822+0000) 2019-10-13T07:17:51.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:51.014+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.015449+0000) 2019-10-13T07:17:51.023 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:51.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:51.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:51.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.038676+0000) 2019-10-13T07:17:51.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:51.859+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.860063+0000) 2019-10-13T07:17:51.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:51.928+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.929640+0000) 2019-10-13T07:17:52.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:52.790+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.791636+0000) 2019-10-13T07:17:52.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:52.792+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.793065+0000) 2019-10-13T07:17:56.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:17:56.019+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.021174+0000) 2019-10-13T07:17:56.025 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:56.020+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.021623+0000) 2019-10-13T07:17:56.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:17:56.020+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.021826+0000) 2019-10-13T07:17:56.025 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:56.020+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.022149+0000) 2019-10-13T07:17:56.025 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:56.020+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.022260+0000) 2019-10-13T07:17:56.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:17:56.019+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.020430+0000) 2019-10-13T07:17:56.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:17:56.021+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.022567+0000) 2019-10-13T07:17:56.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:17:56.021+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:56.022188+0000) 2019-10-13T07:17:56.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:17:56.024+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118437 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:17:58.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:17:58.336+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:58.337507+0000) 2019-10-13T07:17:58.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:17:58.673+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:58.674535+0000) 2019-10-13T07:17:58.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:17:58.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:58.674557+0000) 2019-10-13T07:18:01.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:01.025+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.027133+0000) 2019-10-13T07:18:01.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:01.025+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.027363+0000) 2019-10-13T07:18:01.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:01.026+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.027603+0000) 2019-10-13T07:18:01.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:01.026+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.027738+0000) 2019-10-13T07:18:01.031 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:01.026+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.027822+0000) 2019-10-13T07:18:01.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:01.025+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.026188+0000) 2019-10-13T07:18:01.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:01.026+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.028172+0000) 2019-10-13T07:18:01.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:01.027+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.028683+0000) 2019-10-13T07:18:01.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:01.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118468 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:01.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:01.037+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.038783+0000) 2019-10-13T07:18:01.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:01.862+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.862527+0000) 2019-10-13T07:18:01.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:01.929+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.929806+0000) 2019-10-13T07:18:02.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:02.791+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.791824+0000) 2019-10-13T07:18:02.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:02.792+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.793286+0000) 2019-10-13T07:18:06.035 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:06.032+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.032743+0000) 2019-10-13T07:18:06.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:06.032+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.033064+0000) 2019-10-13T07:18:06.036 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:06.032+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.033278+0000) 2019-10-13T07:18:06.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:06.033+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.033877+0000) 2019-10-13T07:18:06.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:06.033+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.033927+0000) 2019-10-13T07:18:06.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:06.030+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.032136+0000) 2019-10-13T07:18:06.038 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:06.033+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.034230+0000) 2019-10-13T07:18:06.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:06.031+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:06.033609+0000) 2019-10-13T07:18:06.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:06.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118499 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:08.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:08.337+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:08.337715+0000) 2019-10-13T07:18:08.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:08.674+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:08.674720+0000) 2019-10-13T07:18:08.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:08.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:08.674721+0000) 2019-10-13T07:18:11.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:11.036+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.037856+0000) 2019-10-13T07:18:11.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:11.039+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.039999+0000) 2019-10-13T07:18:11.043 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:11.039+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.040118+0000) 2019-10-13T07:18:11.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:11.040+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.040585+0000) 2019-10-13T07:18:11.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:11.040+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.040487+0000) 2019-10-13T07:18:11.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:11.040+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.041102+0000) 2019-10-13T07:18:11.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:11.040+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.041353+0000) 2019-10-13T07:18:11.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:11.038+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.040428+0000) 2019-10-13T07:18:11.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:11.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:11.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:11.863+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.864968+0000) 2019-10-13T07:18:11.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:11.929+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.929995+0000) 2019-10-13T07:18:12.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:12.791+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.792014+0000) 2019-10-13T07:18:12.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:12.792+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.793453+0000) 2019-10-13T07:18:14.123 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:18:14.123 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:14.127 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:18:14.128 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:16.050 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:16.046+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.047038+0000) 2019-10-13T07:18:16.050 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:16.047+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.047438+0000) 2019-10-13T07:18:16.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:16.043+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.045484+0000) 2019-10-13T07:18:16.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:16.047+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.047776+0000) 2019-10-13T07:18:16.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:16.047+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.047949+0000) 2019-10-13T07:18:16.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:16.047+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.048263+0000) 2019-10-13T07:18:16.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:16.047+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.048365+0000) 2019-10-13T07:18:16.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:16.046+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:16.047738+0000) 2019-10-13T07:18:16.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:16.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118559 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:17.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:17.789+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.789513+0000) 2019-10-13T07:18:18.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:18.337+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:18.337920+0000) 2019-10-13T07:18:18.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:18.674+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:18.674904+0000) 2019-10-13T07:18:18.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:18.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:18.674909+0000) 2019-10-13T07:18:21.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:21.049+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.050694+0000) 2019-10-13T07:18:21.056 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:21.052+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.052866+0000) 2019-10-13T07:18:21.056 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:21.052+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.053130+0000) 2019-10-13T07:18:21.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:21.052+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.053086+0000) 2019-10-13T07:18:21.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:21.053+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.053391+0000) 2019-10-13T07:18:21.057 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:21.053+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.053705+0000) 2019-10-13T07:18:21.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:21.053+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.054143+0000) 2019-10-13T07:18:21.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:21.053+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.054192+0000) 2019-10-13T07:18:21.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:21.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118590 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:21.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:21.866+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.867294+0000) 2019-10-13T07:18:21.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:21.930+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.930116+0000) 2019-10-13T07:18:22.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:22.790+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.792214+0000) 2019-10-13T07:18:22.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:22.793+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.793668+0000) 2019-10-13T07:18:26.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:26.058+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.059869+0000) 2019-10-13T07:18:26.063 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:26.058+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.060093+0000) 2019-10-13T07:18:26.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:26.058+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.060278+0000) 2019-10-13T07:18:26.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:26.057+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.058685+0000) 2019-10-13T07:18:26.064 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:26.059+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.060454+0000) 2019-10-13T07:18:26.064 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:26.059+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.061218+0000) 2019-10-13T07:18:26.065 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:26.059+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.061385+0000) 2019-10-13T07:18:26.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:26.060+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:26.061007+0000) 2019-10-13T07:18:26.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:26.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:27.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:27.788+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.789697+0000) 2019-10-13T07:18:28.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:28.337+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:28.338109+0000) 2019-10-13T07:18:28.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:28.674+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:28.675096+0000) 2019-10-13T07:18:28.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:28.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:28.675098+0000) 2019-10-13T07:18:31.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:31.064+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.064982+0000) 2019-10-13T07:18:31.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:31.065+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.067165+0000) 2019-10-13T07:18:31.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:31.065+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.067293+0000) 2019-10-13T07:18:31.071 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:31.066+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.067524+0000) 2019-10-13T07:18:31.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:31.066+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.067590+0000) 2019-10-13T07:18:31.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:31.066+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.067730+0000) 2019-10-13T07:18:31.072 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:31.066+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.067896+0000) 2019-10-13T07:18:31.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:31.067+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.068082+0000) 2019-10-13T07:18:31.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:31.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:31.874 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:31.869+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.869631+0000) 2019-10-13T07:18:31.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:31.929+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.930285+0000) 2019-10-13T07:18:32.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:32.791+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.792379+0000) 2019-10-13T07:18:32.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:32.793+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.793861+0000) 2019-10-13T07:18:36.075 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:36.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.072483+0000) 2019-10-13T07:18:36.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:36.071+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.072629+0000) 2019-10-13T07:18:36.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:36.070+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.070863+0000) 2019-10-13T07:18:36.077 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:36.071+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.073114+0000) 2019-10-13T07:18:36.077 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:36.072+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.073408+0000) 2019-10-13T07:18:36.078 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:36.072+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.073440+0000) 2019-10-13T07:18:36.078 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:36.072+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.073772+0000) 2019-10-13T07:18:36.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:36.073+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:36.073857+0000) 2019-10-13T07:18:36.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:36.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118681 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:37.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:37.789+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.789869+0000) 2019-10-13T07:18:38.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:38.337+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:38.338275+0000) 2019-10-13T07:18:38.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:38.674+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:38.675298+0000) 2019-10-13T07:18:38.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:38.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:38.675302+0000) 2019-10-13T07:18:41.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:41.076+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.077599+0000) 2019-10-13T07:18:41.083 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:41.078+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.079665+0000) 2019-10-13T07:18:41.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:41.078+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.079822+0000) 2019-10-13T07:18:41.084 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:41.078+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.079902+0000) 2019-10-13T07:18:41.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:41.079+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.080424+0000) 2019-10-13T07:18:41.084 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:41.079+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.080621+0000) 2019-10-13T07:18:41.084 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:41.079+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.080716+0000) 2019-10-13T07:18:41.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:41.079+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.080189+0000) 2019-10-13T07:18:41.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:41.081+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118712 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:41.876 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:41.871+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.871835+0000) 2019-10-13T07:18:41.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:41.929+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.930445+0000) 2019-10-13T07:18:42.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:42.791+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.792565+0000) 2019-10-13T07:18:42.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:42.793+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.794045+0000) 2019-10-13T07:18:44.272 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:18:44.272 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:44.277 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:18:44.277 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:46.087 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:46.083+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.084684+0000) 2019-10-13T07:18:46.088 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:46.083+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.085073+0000) 2019-10-13T07:18:46.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:46.083+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.085386+0000) 2019-10-13T07:18:46.089 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:46.083+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.085359+0000) 2019-10-13T07:18:46.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:46.082+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.083654+0000) 2019-10-13T07:18:46.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:46.084+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.085932+0000) 2019-10-13T07:18:46.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:46.084+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.086171+0000) 2019-10-13T07:18:46.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:46.085+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:46.086681+0000) 2019-10-13T07:18:46.092 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:46.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:47.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:47.789+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.790061+0000) 2019-10-13T07:18:48.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:48.337+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:48.338438+0000) 2019-10-13T07:18:48.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:48.675+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:48.675482+0000) 2019-10-13T07:18:48.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:48.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:48.675466+0000) 2019-10-13T07:18:51.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:51.088+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.089619+0000) 2019-10-13T07:18:51.095 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:51.091+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.091789+0000) 2019-10-13T07:18:51.095 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:51.091+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.091904+0000) 2019-10-13T07:18:51.095 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:51.091+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.092245+0000) 2019-10-13T07:18:51.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:51.091+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.092344+0000) 2019-10-13T07:18:51.096 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:51.092+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.092518+0000) 2019-10-13T07:18:51.096 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:51.092+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.093004+0000) 2019-10-13T07:18:51.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:51.092+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.092981+0000) 2019-10-13T07:18:51.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:51.096+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:51.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:51.873+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.873925+0000) 2019-10-13T07:18:51.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:51.930+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.930605+0000) 2019-10-13T07:18:52.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:52.792+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.792736+0000) 2019-10-13T07:18:52.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:52.794+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.794186+0000) 2019-10-13T07:18:56.101 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:18:56.098+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.098601+0000) 2019-10-13T07:18:56.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:56.098+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.098952+0000) 2019-10-13T07:18:56.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:18:56.096+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.097841+0000) 2019-10-13T07:18:56.103 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:56.098+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.099290+0000) 2019-10-13T07:18:56.103 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:56.098+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.099348+0000) 2019-10-13T07:18:56.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:18:56.099+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.099494+0000) 2019-10-13T07:18:56.103 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:56.099+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.099891+0000) 2019-10-13T07:18:56.104 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:18:56.098+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:56.099775+0000) 2019-10-13T07:18:56.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:18:56.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:18:57.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:18:57.789+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.790251+0000) 2019-10-13T07:18:58.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:18:58.338+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:58.338626+0000) 2019-10-13T07:18:58.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:18:58.675+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:58.675674+0000) 2019-10-13T07:18:58.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:18:58.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:58.675675+0000) 2019-10-13T07:19:01.109 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:01.104+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.105243+0000) 2019-10-13T07:19:01.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:01.105+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.105459+0000) 2019-10-13T07:19:01.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:01.105+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.105777+0000) 2019-10-13T07:19:01.110 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:01.105+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.106009+0000) 2019-10-13T07:19:01.110 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:01.105+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.106181+0000) 2019-10-13T07:19:01.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:01.105+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.106270+0000) 2019-10-13T07:19:01.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:01.101+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.103665+0000) 2019-10-13T07:19:01.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:01.104+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.105857+0000) 2019-10-13T07:19:01.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:01.108+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118834 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:01.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:01.874+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.876223+0000) 2019-10-13T07:19:01.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:01.930+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.930773+0000) 2019-10-13T07:19:02.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:02.792+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.792927+0000) 2019-10-13T07:19:02.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:02.793+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.794404+0000) 2019-10-13T07:19:06.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:06.110+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.110660+0000) 2019-10-13T07:19:06.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:06.110+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.110991+0000) 2019-10-13T07:19:06.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:06.110+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.110966+0000) 2019-10-13T07:19:06.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:06.108+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.109860+0000) 2019-10-13T07:19:06.115 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:06.111+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.111539+0000) 2019-10-13T07:19:06.116 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:06.111+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.111908+0000) 2019-10-13T07:19:06.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:06.111+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.111970+0000) 2019-10-13T07:19:06.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:06.110+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:06.112461+0000) 2019-10-13T07:19:06.119 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:06.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:07.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:07.790+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.790446+0000) 2019-10-13T07:19:08.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:08.338+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:08.338820+0000) 2019-10-13T07:19:08.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:08.674+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:08.675872+0000) 2019-10-13T07:19:08.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:08.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:08.675879+0000) 2019-10-13T07:19:11.121 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:11.115+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.116002+0000) 2019-10-13T07:19:11.122 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:11.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.118270+0000) 2019-10-13T07:19:11.122 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:11.116+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.118341+0000) 2019-10-13T07:19:11.122 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:11.117+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.118788+0000) 2019-10-13T07:19:11.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:11.117+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.118937+0000) 2019-10-13T07:19:11.122 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:11.117+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.119088+0000) 2019-10-13T07:19:11.123 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:11.117+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.119268+0000) 2019-10-13T07:19:11.124 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:11.119+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.119900+0000) 2019-10-13T07:19:11.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:11.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118893 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:11.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:11.878+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.878782+0000) 2019-10-13T07:19:11.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:11.930+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.930950+0000) 2019-10-13T07:19:12.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:12.792+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.793133+0000) 2019-10-13T07:19:12.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:12.793+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.794590+0000) 2019-10-13T07:19:14.435 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:19:14.435 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:14.440 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:19:14.440 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:16.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:16.122+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.123077+0000) 2019-10-13T07:19:16.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:16.123+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.125284+0000) 2019-10-13T07:19:16.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:16.124+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.125500+0000) 2019-10-13T07:19:16.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:16.124+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.125693+0000) 2019-10-13T07:19:16.129 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:16.124+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.125866+0000) 2019-10-13T07:19:16.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:16.124+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.125984+0000) 2019-10-13T07:19:16.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:16.124+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.126172+0000) 2019-10-13T07:19:16.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:16.125+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:16.126068+0000) 2019-10-13T07:19:16.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:16.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118922 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:17.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:17.789+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.790634+0000) 2019-10-13T07:19:18.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:18.337+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:18.339019+0000) 2019-10-13T07:19:18.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:18.675+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:18.676036+0000) 2019-10-13T07:19:18.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:18.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:18.676058+0000) 2019-10-13T07:19:21.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:21.130+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.131848+0000) 2019-10-13T07:19:21.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:21.131+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.132434+0000) 2019-10-13T07:19:21.136 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:21.131+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.132651+0000) 2019-10-13T07:19:21.136 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:21.131+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.132743+0000) 2019-10-13T07:19:21.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:21.131+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.133119+0000) 2019-10-13T07:19:21.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:21.130+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.131062+0000) 2019-10-13T07:19:21.137 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:21.132+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.133614+0000) 2019-10-13T07:19:21.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:21.131+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.132484+0000) 2019-10-13T07:19:21.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:21.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:21.886 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:21.880+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.881277+0000) 2019-10-13T07:19:21.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:21.930+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.931098+0000) 2019-10-13T07:19:22.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:22.792+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.793269+0000) 2019-10-13T07:19:22.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:22.794+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.794795+0000) 2019-10-13T07:19:26.141 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:26.137+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.138733+0000) 2019-10-13T07:19:26.142 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:26.137+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.139128+0000) 2019-10-13T07:19:26.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:26.138+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.139441+0000) 2019-10-13T07:19:26.143 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:26.138+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.139774+0000) 2019-10-13T07:19:26.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:26.138+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.139913+0000) 2019-10-13T07:19:26.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:26.138+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.140143+0000) 2019-10-13T07:19:26.144 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:26.137+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.138454+0000) 2019-10-13T07:19:26.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:26.139+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:26.140387+0000) 2019-10-13T07:19:26.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:26.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 118983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:27.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:27.790+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.790816+0000) 2019-10-13T07:19:28.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:28.338+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:28.339194+0000) 2019-10-13T07:19:28.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:28.674+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:28.676243+0000) 2019-10-13T07:19:28.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:28.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:28.676242+0000) 2019-10-13T07:19:31.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:31.145+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.146540+0000) 2019-10-13T07:19:31.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:31.145+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.146706+0000) 2019-10-13T07:19:31.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:31.145+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.146910+0000) 2019-10-13T07:19:31.151 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:31.146+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.147550+0000) 2019-10-13T07:19:31.151 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:31.146+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.147736+0000) 2019-10-13T07:19:31.151 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:31.146+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.147998+0000) 2019-10-13T07:19:31.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:31.145+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.146336+0000) 2019-10-13T07:19:31.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:31.146+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.147415+0000) 2019-10-13T07:19:31.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:31.150+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:31.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:31.883+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.883759+0000) 2019-10-13T07:19:31.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:31.930+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.931185+0000) 2019-10-13T07:19:32.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:32.792+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.793443+0000) 2019-10-13T07:19:32.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:32.794+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.794956+0000) 2019-10-13T07:19:36.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:36.152+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.153612+0000) 2019-10-13T07:19:36.157 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:36.152+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.153717+0000) 2019-10-13T07:19:36.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:36.152+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.153867+0000) 2019-10-13T07:19:36.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:36.151+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.152687+0000) 2019-10-13T07:19:36.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:36.152+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.154361+0000) 2019-10-13T07:19:36.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:36.153+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.154689+0000) 2019-10-13T07:19:36.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:36.153+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.154781+0000) 2019-10-13T07:19:36.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:36.153+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:36.154501+0000) 2019-10-13T07:19:36.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:36.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119042 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:37.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:37.790+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.790995+0000) 2019-10-13T07:19:38.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:38.339+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:38.339411+0000) 2019-10-13T07:19:38.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:38.676+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:38.676450+0000) 2019-10-13T07:19:38.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:38.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:38.676448+0000) 2019-10-13T07:19:41.163 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:41.159+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.159877+0000) 2019-10-13T07:19:41.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:41.159+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.160148+0000) 2019-10-13T07:19:41.164 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:41.160+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.160448+0000) 2019-10-13T07:19:41.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:41.160+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.160742+0000) 2019-10-13T07:19:41.164 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:41.160+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.160852+0000) 2019-10-13T07:19:41.164 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:41.160+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.161136+0000) 2019-10-13T07:19:41.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:41.158+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.159072+0000) 2019-10-13T07:19:41.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:41.159+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.160219+0000) 2019-10-13T07:19:41.167 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:41.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119074 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:41.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:41.885+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.886096+0000) 2019-10-13T07:19:41.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:41.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.931334+0000) 2019-10-13T07:19:42.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:42.793+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.793645+0000) 2019-10-13T07:19:42.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:42.794+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.795125+0000) 2019-10-13T07:19:44.597 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:19:44.597 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:44.602 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:19:44.603 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:46.169 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:46.163+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.164376+0000) 2019-10-13T07:19:46.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:46.165+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.166348+0000) 2019-10-13T07:19:46.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:46.166+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.166517+0000) 2019-10-13T07:19:46.170 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:46.166+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.166617+0000) 2019-10-13T07:19:46.171 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:46.166+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.167143+0000) 2019-10-13T07:19:46.171 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:46.166+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.167167+0000) 2019-10-13T07:19:46.171 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:46.166+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.167324+0000) 2019-10-13T07:19:46.172 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:46.166+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:46.167513+0000) 2019-10-13T07:19:46.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:46.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:47.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:47.790+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.791185+0000) 2019-10-13T07:19:48.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:48.339+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:48.339571+0000) 2019-10-13T07:19:48.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:48.676+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:48.676632+0000) 2019-10-13T07:19:48.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:48.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:48.676634+0000) 2019-10-13T07:19:51.175 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:51.168+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.170321+0000) 2019-10-13T07:19:51.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:51.171+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.172294+0000) 2019-10-13T07:19:51.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:51.172+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.172560+0000) 2019-10-13T07:19:51.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:51.172+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.172689+0000) 2019-10-13T07:19:51.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:51.172+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.172922+0000) 2019-10-13T07:19:51.177 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:51.172+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.173011+0000) 2019-10-13T07:19:51.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:51.172+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.173363+0000) 2019-10-13T07:19:51.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:51.171+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.173477+0000) 2019-10-13T07:19:51.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:51.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119133 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:51.893 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:51.887+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.888169+0000) 2019-10-13T07:19:51.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:51.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.931486+0000) 2019-10-13T07:19:52.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:52.793+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.793810+0000) 2019-10-13T07:19:52.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:52.793+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.795313+0000) 2019-10-13T07:19:56.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:19:56.175+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.177028+0000) 2019-10-13T07:19:56.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:19:56.177+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.179077+0000) 2019-10-13T07:19:56.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:56.177+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.179295+0000) 2019-10-13T07:19:56.183 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:56.178+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.179587+0000) 2019-10-13T07:19:56.183 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:56.178+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.179795+0000) 2019-10-13T07:19:56.183 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:56.178+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.179950+0000) 2019-10-13T07:19:56.184 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:19:56.178+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.180169+0000) 2019-10-13T07:19:56.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:19:56.178+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:56.180211+0000) 2019-10-13T07:19:56.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:19:56.182+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:19:57.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:19:57.790+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.791378+0000) 2019-10-13T07:19:58.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:19:58.338+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:58.339712+0000) 2019-10-13T07:19:58.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:19:58.675+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:58.676850+0000) 2019-10-13T07:19:58.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:19:58.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:58.676849+0000) 2019-10-13T07:20:01.188 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:01.184+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.185448+0000) 2019-10-13T07:20:01.189 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:01.184+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.185604+0000) 2019-10-13T07:20:01.189 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:01.183+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.183871+0000) 2019-10-13T07:20:01.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:01.184+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.185964+0000) 2019-10-13T07:20:01.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:01.185+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.186392+0000) 2019-10-13T07:20:01.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:01.185+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.186537+0000) 2019-10-13T07:20:01.190 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:01.185+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.186814+0000) 2019-10-13T07:20:01.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:01.185+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.187410+0000) 2019-10-13T07:20:01.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:01.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119194 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:01.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:01.889+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.890767+0000) 2019-10-13T07:20:01.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:01.930+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.931592+0000) 2019-10-13T07:20:02.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:02.793+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.794017+0000) 2019-10-13T07:20:02.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:02.794+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.795512+0000) 2019-10-13T07:20:06.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:06.190+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.192165+0000) 2019-10-13T07:20:06.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:06.190+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.192323+0000) 2019-10-13T07:20:06.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:06.191+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.192475+0000) 2019-10-13T07:20:06.196 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:06.191+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.192740+0000) 2019-10-13T07:20:06.197 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:06.190+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.191638+0000) 2019-10-13T07:20:06.197 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:06.191+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.193378+0000) 2019-10-13T07:20:06.197 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:06.192+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.193742+0000) 2019-10-13T07:20:06.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:06.193+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:06.193860+0000) 2019-10-13T07:20:06.199 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:06.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:07.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:07.790+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.791614+0000) 2019-10-13T07:20:08.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:08.339+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:08.339918+0000) 2019-10-13T07:20:08.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:08.676+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:08.677036+0000) 2019-10-13T07:20:08.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:08.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:08.677038+0000) 2019-10-13T07:20:11.201 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:11.197+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.198549+0000) 2019-10-13T07:20:11.202 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:11.197+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.198674+0000) 2019-10-13T07:20:11.203 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:11.196+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.197032+0000) 2019-10-13T07:20:11.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:11.198+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.199483+0000) 2019-10-13T07:20:11.203 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:11.198+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.199678+0000) 2019-10-13T07:20:11.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:11.198+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.199829+0000) 2019-10-13T07:20:11.204 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:11.198+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.200006+0000) 2019-10-13T07:20:11.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:11.198+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.199502+0000) 2019-10-13T07:20:11.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:11.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:11.898 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:11.892+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.893180+0000) 2019-10-13T07:20:11.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:11.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.931823+0000) 2019-10-13T07:20:12.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:12.793+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.794210+0000) 2019-10-13T07:20:12.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:12.794+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.795716+0000) 2019-10-13T07:20:14.760 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:20:14.760 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:14.765 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:20:14.765 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:16.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:16.202+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.203243+0000) 2019-10-13T07:20:16.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:16.204+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.205477+0000) 2019-10-13T07:20:16.209 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:16.204+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.205831+0000) 2019-10-13T07:20:16.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:16.205+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.206444+0000) 2019-10-13T07:20:16.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:16.205+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.206555+0000) 2019-10-13T07:20:16.210 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:16.205+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.206614+0000) 2019-10-13T07:20:16.210 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:16.205+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.206976+0000) 2019-10-13T07:20:16.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:16.205+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:16.206002+0000) 2019-10-13T07:20:16.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:16.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:17.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:17.791+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.791825+0000) 2019-10-13T07:20:18.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:18.339+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:18.340109+0000) 2019-10-13T07:20:18.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:18.676+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:18.677211+0000) 2019-10-13T07:20:18.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:18.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:18.677248+0000) 2019-10-13T07:20:21.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:21.210+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.212079+0000) 2019-10-13T07:20:21.215 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:21.210+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.212369+0000) 2019-10-13T07:20:21.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:21.209+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.210322+0000) 2019-10-13T07:20:21.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:21.211+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.212882+0000) 2019-10-13T07:20:21.216 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:21.211+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.212638+0000) 2019-10-13T07:20:21.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:21.212+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.213426+0000) 2019-10-13T07:20:21.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:21.212+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.213616+0000) 2019-10-13T07:20:21.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:21.212+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.213413+0000) 2019-10-13T07:20:21.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:21.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:21.900 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:21.894+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.895658+0000) 2019-10-13T07:20:21.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:21.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.931965+0000) 2019-10-13T07:20:22.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:22.793+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.794384+0000) 2019-10-13T07:20:22.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:22.795+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.795917+0000) 2019-10-13T07:20:26.221 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:26.218+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.218776+0000) 2019-10-13T07:20:26.223 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:26.216+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.216994+0000) 2019-10-13T07:20:26.223 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:26.218+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.219225+0000) 2019-10-13T07:20:26.223 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:26.218+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.219309+0000) 2019-10-13T07:20:26.223 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:26.219+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.219580+0000) 2019-10-13T07:20:26.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:26.219+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.219658+0000) 2019-10-13T07:20:26.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:26.219+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.219956+0000) 2019-10-13T07:20:26.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:26.219+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:26.220705+0000) 2019-10-13T07:20:26.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:26.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119344 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:27.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:27.791+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.792049+0000) 2019-10-13T07:20:28.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:28.340+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:28.340320+0000) 2019-10-13T07:20:28.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:28.677+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:28.677411+0000) 2019-10-13T07:20:28.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:28.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:28.677421+0000) 2019-10-13T07:20:31.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:31.225+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.225500+0000) 2019-10-13T07:20:31.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:31.223+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.224001+0000) 2019-10-13T07:20:31.229 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:31.225+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.226076+0000) 2019-10-13T07:20:31.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:31.225+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.226288+0000) 2019-10-13T07:20:31.230 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:31.225+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.226352+0000) 2019-10-13T07:20:31.230 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:31.226+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.226897+0000) 2019-10-13T07:20:31.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:31.226+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.227101+0000) 2019-10-13T07:20:31.231 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:31.226+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.226763+0000) 2019-10-13T07:20:31.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:31.230+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:31.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:31.897+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.897947+0000) 2019-10-13T07:20:31.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:31.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.932112+0000) 2019-10-13T07:20:32.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:32.794+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.794587+0000) 2019-10-13T07:20:32.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:32.795+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.796094+0000) 2019-10-13T07:20:36.235 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:36.232+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.232777+0000) 2019-10-13T07:20:36.236 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:36.230+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.231256+0000) 2019-10-13T07:20:36.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:36.232+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.233083+0000) 2019-10-13T07:20:36.237 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:36.233+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.233685+0000) 2019-10-13T07:20:36.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:36.233+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.234050+0000) 2019-10-13T07:20:36.237 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:36.233+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.233971+0000) 2019-10-13T07:20:36.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:36.233+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.234159+0000) 2019-10-13T07:20:36.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:36.232+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:36.233238+0000) 2019-10-13T07:20:36.240 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:36.237+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:37.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:37.792+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.792239+0000) 2019-10-13T07:20:38.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:38.340+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:38.340545+0000) 2019-10-13T07:20:38.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:38.677+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:38.677629+0000) 2019-10-13T07:20:38.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:38.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:38.677632+0000) 2019-10-13T07:20:41.242 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:41.239+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.239757+0000) 2019-10-13T07:20:41.243 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:41.239+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.240028+0000) 2019-10-13T07:20:41.243 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:41.239+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.240253+0000) 2019-10-13T07:20:41.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:41.240+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.240897+0000) 2019-10-13T07:20:41.244 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:41.237+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.239263+0000) 2019-10-13T07:20:41.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:41.237+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.239684+0000) 2019-10-13T07:20:41.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:41.240+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.241281+0000) 2019-10-13T07:20:41.245 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:41.241+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.241508+0000) 2019-10-13T07:20:41.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:41.244+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:41.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:41.898+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.900001+0000) 2019-10-13T07:20:41.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:41.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.932258+0000) 2019-10-13T07:20:42.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:42.793+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.794759+0000) 2019-10-13T07:20:42.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:42.794+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.796297+0000) 2019-10-13T07:20:44.948 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:20:44.948 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:44.952 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:20:44.952 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:46.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:46.245+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.246940+0000) 2019-10-13T07:20:46.250 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:46.245+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.247087+0000) 2019-10-13T07:20:46.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:46.245+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.247286+0000) 2019-10-13T07:20:46.251 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:46.243+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.245721+0000) 2019-10-13T07:20:46.252 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:46.246+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.247810+0000) 2019-10-13T07:20:46.252 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:46.246+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.248080+0000) 2019-10-13T07:20:46.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:46.246+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.248273+0000) 2019-10-13T07:20:46.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:46.246+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:46.248583+0000) 2019-10-13T07:20:46.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:46.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119466 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:47.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:47.791+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.792423+0000) 2019-10-13T07:20:48.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:48.339+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:48.340753+0000) 2019-10-13T07:20:48.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:48.676+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:48.677766+0000) 2019-10-13T07:20:48.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:48.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:48.677829+0000) 2019-10-13T07:20:51.256 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:51.251+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.253337+0000) 2019-10-13T07:20:51.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:51.252+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.253532+0000) 2019-10-13T07:20:51.258 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:51.252+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.253665+0000) 2019-10-13T07:20:51.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:51.252+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.253848+0000) 2019-10-13T07:20:51.259 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:51.250+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.252598+0000) 2019-10-13T07:20:51.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:51.252+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.254272+0000) 2019-10-13T07:20:51.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:51.253+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.254519+0000) 2019-10-13T07:20:51.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:51.252+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.254253+0000) 2019-10-13T07:20:51.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:51.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119498 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:51.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:51.901+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.902298+0000) 2019-10-13T07:20:51.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:51.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.932422+0000) 2019-10-13T07:20:52.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:52.794+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.794937+0000) 2019-10-13T07:20:52.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:52.795+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.796445+0000) 2019-10-13T07:20:56.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:56.259+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.260679+0000) 2019-10-13T07:20:56.264 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:20:56.259+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.260956+0000) 2019-10-13T07:20:56.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:56.259+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.261144+0000) 2019-10-13T07:20:56.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:20:56.259+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.261245+0000) 2019-10-13T07:20:56.265 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:56.260+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.261496+0000) 2019-10-13T07:20:56.265 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:56.260+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.261866+0000) 2019-10-13T07:20:56.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:20:56.259+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.260262+0000) 2019-10-13T07:20:56.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:20:56.261+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:56.262434+0000) 2019-10-13T07:20:56.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:20:56.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119527 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:20:57.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:20:57.791+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.792603+0000) 2019-10-13T07:20:58.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:20:58.340+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:58.340967+0000) 2019-10-13T07:20:58.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:20:58.677+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:58.677955+0000) 2019-10-13T07:20:58.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:20:58.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:58.678026+0000) 2019-10-13T07:21:01.271 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:01.266+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.267784+0000) 2019-10-13T07:21:01.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:01.266+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.268039+0000) 2019-10-13T07:21:01.272 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:01.266+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.266978+0000) 2019-10-13T07:21:01.272 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:01.266+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.268211+0000) 2019-10-13T07:21:01.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:01.267+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.268489+0000) 2019-10-13T07:21:01.272 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:01.267+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.268632+0000) 2019-10-13T07:21:01.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:01.267+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.268979+0000) 2019-10-13T07:21:01.273 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:01.267+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.268371+0000) 2019-10-13T07:21:01.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:01.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:01.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:01.903+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.904425+0000) 2019-10-13T07:21:01.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:01.931+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.932575+0000) 2019-10-13T07:21:02.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:02.794+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.795154+0000) 2019-10-13T07:21:02.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:02.795+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.796585+0000) 2019-10-13T07:21:06.277 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:06.272+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.274236+0000) 2019-10-13T07:21:06.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:06.271+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.272562+0000) 2019-10-13T07:21:06.278 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:06.273+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.274595+0000) 2019-10-13T07:21:06.278 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:06.273+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.274779+0000) 2019-10-13T07:21:06.278 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:06.273+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.274941+0000) 2019-10-13T07:21:06.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:06.274+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.275479+0000) 2019-10-13T07:21:06.279 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:06.274+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.275648+0000) 2019-10-13T07:21:06.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:06.275+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:06.276372+0000) 2019-10-13T07:21:06.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:06.277+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:07.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:07.792+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.792799+0000) 2019-10-13T07:21:08.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:08.340+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:08.341182+0000) 2019-10-13T07:21:08.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:08.677+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:08.678177+0000) 2019-10-13T07:21:08.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:08.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:08.678235+0000) 2019-10-13T07:21:11.283 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:11.280+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.280651+0000) 2019-10-13T07:21:11.284 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:11.280+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.280947+0000) 2019-10-13T07:21:11.285 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:11.278+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.278936+0000) 2019-10-13T07:21:11.285 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:11.280+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.281184+0000) 2019-10-13T07:21:11.285 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:11.281+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.281768+0000) 2019-10-13T07:21:11.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:11.281+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.282088+0000) 2019-10-13T07:21:11.286 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:11.282+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.282473+0000) 2019-10-13T07:21:11.286 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:11.281+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.282164+0000) 2019-10-13T07:21:11.289 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:11.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:11.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:11.906+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.906814+0000) 2019-10-13T07:21:11.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:11.932+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.932750+0000) 2019-10-13T07:21:12.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:12.794+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.795308+0000) 2019-10-13T07:21:12.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:12.796+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.796790+0000) 2019-10-13T07:21:15.102 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:21:15.102 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:15.107 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:21:15.107 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:16.291 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:16.287+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.288295+0000) 2019-10-13T07:21:16.292 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:16.288+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.288501+0000) 2019-10-13T07:21:16.292 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:16.288+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.288680+0000) 2019-10-13T07:21:16.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:16.288+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.288923+0000) 2019-10-13T07:21:16.293 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:16.286+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.287238+0000) 2019-10-13T07:21:16.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:16.287+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.288316+0000) 2019-10-13T07:21:16.293 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:16.288+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.289319+0000) 2019-10-13T07:21:16.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:16.289+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:16.289443+0000) 2019-10-13T07:21:16.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:16.292+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119649 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:17.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:17.792+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.792962+0000) 2019-10-13T07:21:18.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:18.340+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:18.341388+0000) 2019-10-13T07:21:18.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:18.678+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:18.678361+0000) 2019-10-13T07:21:18.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:18.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:18.678427+0000) 2019-10-13T07:21:21.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:21.293+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.294289+0000) 2019-10-13T07:21:21.297 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:21.294+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.294455+0000) 2019-10-13T07:21:21.298 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:21.292+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.293579+0000) 2019-10-13T07:21:21.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:21.294+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.294896+0000) 2019-10-13T07:21:21.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:21.294+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.294722+0000) 2019-10-13T07:21:21.299 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:21.294+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.295147+0000) 2019-10-13T07:21:21.299 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:21.295+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.295577+0000) 2019-10-13T07:21:21.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:21.294+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.295487+0000) 2019-10-13T07:21:21.302 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:21.298+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:21.914 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:21.909+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.909372+0000) 2019-10-13T07:21:21.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:21.932+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.932909+0000) 2019-10-13T07:21:22.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:22.795+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.795482+0000) 2019-10-13T07:21:22.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:22.796+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.796962+0000) 2019-10-13T07:21:26.304 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:26.298+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.298954+0000) 2019-10-13T07:21:26.305 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:26.300+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.300962+0000) 2019-10-13T07:21:26.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:26.300+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.301189+0000) 2019-10-13T07:21:26.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:26.300+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.301324+0000) 2019-10-13T07:21:26.305 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:26.301+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.301560+0000) 2019-10-13T07:21:26.305 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:26.301+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.301748+0000) 2019-10-13T07:21:26.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:26.301+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.301835+0000) 2019-10-13T07:21:26.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:26.302+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:26.302932+0000) 2019-10-13T07:21:26.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:26.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119711 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:27.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:27.792+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.793162+0000) 2019-10-13T07:21:28.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:28.340+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:28.341513+0000) 2019-10-13T07:21:28.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:28.677+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:28.678540+0000) 2019-10-13T07:21:28.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:28.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:28.678582+0000) 2019-10-13T07:21:31.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:31.305+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.307267+0000) 2019-10-13T07:21:31.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:31.306+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.307581+0000) 2019-10-13T07:21:31.311 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:31.305+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.306077+0000) 2019-10-13T07:21:31.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:31.306+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.307772+0000) 2019-10-13T07:21:31.312 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:31.306+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.307518+0000) 2019-10-13T07:21:31.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:31.306+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.308149+0000) 2019-10-13T07:21:31.312 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:31.306+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.308309+0000) 2019-10-13T07:21:31.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:31.308+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.308898+0000) 2019-10-13T07:21:31.314 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:31.309+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:31.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:31.911+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.911850+0000) 2019-10-13T07:21:31.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:31.932+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.933081+0000) 2019-10-13T07:21:32.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:32.794+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.795676+0000) 2019-10-13T07:21:32.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:32.796+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.797156+0000) 2019-10-13T07:21:36.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:36.311+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.312665+0000) 2019-10-13T07:21:36.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:36.311+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.312915+0000) 2019-10-13T07:21:36.317 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:36.311+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.313078+0000) 2019-10-13T07:21:36.317 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:36.311+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.313244+0000) 2019-10-13T07:21:36.318 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:36.311+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.313351+0000) 2019-10-13T07:21:36.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:36.309+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.311718+0000) 2019-10-13T07:21:36.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:36.312+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.314054+0000) 2019-10-13T07:21:36.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:36.312+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:36.314210+0000) 2019-10-13T07:21:36.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:36.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:37.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:37.792+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.793347+0000) 2019-10-13T07:21:38.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:38.340+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:38.341696+0000) 2019-10-13T07:21:38.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:38.677+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:38.678721+0000) 2019-10-13T07:21:38.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:38.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:38.678738+0000) 2019-10-13T07:21:41.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:41.317+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.319299+0000) 2019-10-13T07:21:41.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:41.318+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.319686+0000) 2019-10-13T07:21:41.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:41.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.317719+0000) 2019-10-13T07:21:41.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:41.318+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.319956+0000) 2019-10-13T07:21:41.324 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:41.318+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.320237+0000) 2019-10-13T07:21:41.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:41.319+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.320818+0000) 2019-10-13T07:21:41.324 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:41.318+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.319770+0000) 2019-10-13T07:21:41.325 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:41.319+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.321057+0000) 2019-10-13T07:21:41.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:41.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:41.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:41.913+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.914350+0000) 2019-10-13T07:21:41.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:41.932+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.933253+0000) 2019-10-13T07:21:42.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:42.795+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.795839+0000) 2019-10-13T07:21:42.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:42.795+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.797387+0000) 2019-10-13T07:21:45.265 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:21:45.265 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:45.270 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:21:45.270 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:46.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:46.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.326002+0000) 2019-10-13T07:21:46.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:46.323+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.324805+0000) 2019-10-13T07:21:46.330 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:46.324+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.326369+0000) 2019-10-13T07:21:46.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:46.325+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.327036+0000) 2019-10-13T07:21:46.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:46.325+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.327175+0000) 2019-10-13T07:21:46.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:46.325+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.327336+0000) 2019-10-13T07:21:46.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:46.326+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.327631+0000) 2019-10-13T07:21:46.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:46.325+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:46.327170+0000) 2019-10-13T07:21:46.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:46.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:47.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:47.792+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.793553+0000) 2019-10-13T07:21:48.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:48.341+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:48.341905+0000) 2019-10-13T07:21:48.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:48.678+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:48.678925+0000) 2019-10-13T07:21:48.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:48.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:48.678928+0000) 2019-10-13T07:21:51.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:51.331+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.333102+0000) 2019-10-13T07:21:51.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:51.330+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.331583+0000) 2019-10-13T07:21:51.337 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:51.332+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.333587+0000) 2019-10-13T07:21:51.337 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:51.332+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.333966+0000) 2019-10-13T07:21:51.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:51.332+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.334126+0000) 2019-10-13T07:21:51.338 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:51.332+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.334246+0000) 2019-10-13T07:21:51.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:51.332+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.333172+0000) 2019-10-13T07:21:51.338 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:51.333+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.334616+0000) 2019-10-13T07:21:51.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:51.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:51.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:51.915+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.916571+0000) 2019-10-13T07:21:51.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:51.932+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.933402+0000) 2019-10-13T07:21:52.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:52.795+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.796036+0000) 2019-10-13T07:21:52.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:52.796+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.797569+0000) 2019-10-13T07:21:56.342 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:56.338+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.339093+0000) 2019-10-13T07:21:56.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:56.338+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.338910+0000) 2019-10-13T07:21:56.343 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:21:56.336+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.337550+0000) 2019-10-13T07:21:56.343 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:21:56.339+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.339724+0000) 2019-10-13T07:21:56.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:56.339+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.339860+0000) 2019-10-13T07:21:56.344 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:21:56.339+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.339606+0000) 2019-10-13T07:21:56.344 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:56.339+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.339983+0000) 2019-10-13T07:21:56.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:21:56.338+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:56.339620+0000) 2019-10-13T07:21:56.345 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:21:56.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119893 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:21:57.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:21:57.793+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.793736+0000) 2019-10-13T07:21:58.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:21:58.341+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:58.342106+0000) 2019-10-13T07:21:58.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:21:58.678+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:58.679129+0000) 2019-10-13T07:21:58.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:21:58.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:58.679122+0000) 2019-10-13T07:22:01.348 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:01.344+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.344784+0000) 2019-10-13T07:22:01.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:01.344+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.344960+0000) 2019-10-13T07:22:01.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:01.344+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.345266+0000) 2019-10-13T07:22:01.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:01.342+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.343272+0000) 2019-10-13T07:22:01.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:01.345+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.345533+0000) 2019-10-13T07:22:01.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:01.345+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.345838+0000) 2019-10-13T07:22:01.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:01.345+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.346028+0000) 2019-10-13T07:22:01.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:01.344+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.345540+0000) 2019-10-13T07:22:01.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:01.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:01.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:01.918+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.918850+0000) 2019-10-13T07:22:01.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:01.933+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.933578+0000) 2019-10-13T07:22:02.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:02.795+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.796199+0000) 2019-10-13T07:22:02.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:02.797+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.797776+0000) 2019-10-13T07:22:06.357 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:06.349+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.349852+0000) 2019-10-13T07:22:06.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:06.351+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.351476+0000) 2019-10-13T07:22:06.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:06.351+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.351552+0000) 2019-10-13T07:22:06.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:06.351+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.352060+0000) 2019-10-13T07:22:06.358 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:06.351+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.352201+0000) 2019-10-13T07:22:06.358 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:06.352+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.352481+0000) 2019-10-13T07:22:06.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:06.352+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.352879+0000) 2019-10-13T07:22:06.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:06.352+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:06.352914+0000) 2019-10-13T07:22:06.359 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:06.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119955 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:07.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:07.793+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.793947+0000) 2019-10-13T07:22:08.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:08.342+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:08.342298+0000) 2019-10-13T07:22:08.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:08.678+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:08.679341+0000) 2019-10-13T07:22:08.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:08.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:08.679340+0000) 2019-10-13T07:22:11.362 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:11.358+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.358581+0000) 2019-10-13T07:22:11.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:11.358+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.359114+0000) 2019-10-13T07:22:11.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:11.356+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.357329+0000) 2019-10-13T07:22:11.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:11.358+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.359382+0000) 2019-10-13T07:22:11.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:11.359+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.359550+0000) 2019-10-13T07:22:11.365 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:11.359+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.359754+0000) 2019-10-13T07:22:11.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:11.359+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.360124+0000) 2019-10-13T07:22:11.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:11.359+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.360111+0000) 2019-10-13T07:22:11.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:11.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 119986 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:11.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:11.920+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.920925+0000) 2019-10-13T07:22:11.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:11.933+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.933690+0000) 2019-10-13T07:22:12.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:12.796+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.796336+0000) 2019-10-13T07:22:12.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:12.797+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.797994+0000) 2019-10-13T07:22:15.377 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:22:15.378 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:15.382 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:22:15.382 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:16.368 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:16.363+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.365115+0000) 2019-10-13T07:22:16.368 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:16.363+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.365224+0000) 2019-10-13T07:22:16.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:16.364+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.366082+0000) 2019-10-13T07:22:16.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:16.364+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.366141+0000) 2019-10-13T07:22:16.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:16.364+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.366301+0000) 2019-10-13T07:22:16.370 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:16.363+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.364608+0000) 2019-10-13T07:22:16.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:16.365+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.366600+0000) 2019-10-13T07:22:16.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:16.365+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:16.366005+0000) 2019-10-13T07:22:16.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:16.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:17.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:17.793+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.794152+0000) 2019-10-13T07:22:18.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:18.341+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:18.342498+0000) 2019-10-13T07:22:18.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:18.678+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:18.679572+0000) 2019-10-13T07:22:18.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:18.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:18.679577+0000) 2019-10-13T07:22:21.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:21.370+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.372019+0000) 2019-10-13T07:22:21.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:21.370+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.372086+0000) 2019-10-13T07:22:21.376 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:21.369+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.370706+0000) 2019-10-13T07:22:21.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:21.370+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.372157+0000) 2019-10-13T07:22:21.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:21.371+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.372545+0000) 2019-10-13T07:22:21.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:21.371+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.372907+0000) 2019-10-13T07:22:21.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:21.371+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.373082+0000) 2019-10-13T07:22:21.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:21.372+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.372821+0000) 2019-10-13T07:22:21.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:21.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:21.928 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:21.922+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.923252+0000) 2019-10-13T07:22:21.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:21.932+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.933856+0000) 2019-10-13T07:22:22.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:22.795+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.796517+0000) 2019-10-13T07:22:22.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:22.797+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.798184+0000) 2019-10-13T07:22:26.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:26.376+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.378188+0000) 2019-10-13T07:22:26.382 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:26.377+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.378522+0000) 2019-10-13T07:22:26.382 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:26.377+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.378852+0000) 2019-10-13T07:22:26.383 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:26.375+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.376827+0000) 2019-10-13T07:22:26.383 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:26.377+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.379251+0000) 2019-10-13T07:22:26.383 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:26.377+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.379374+0000) 2019-10-13T07:22:26.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:26.378+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.379551+0000) 2019-10-13T07:22:26.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:26.377+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:26.379285+0000) 2019-10-13T07:22:26.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:26.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:27.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:27.793+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.794358+0000) 2019-10-13T07:22:28.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:28.341+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:28.342652+0000) 2019-10-13T07:22:28.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:28.679+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:28.679765+0000) 2019-10-13T07:22:28.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:28.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:28.679770+0000) 2019-10-13T07:22:31.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:31.383+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.385362+0000) 2019-10-13T07:22:31.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:31.384+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.385657+0000) 2019-10-13T07:22:31.389 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:31.384+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.385846+0000) 2019-10-13T07:22:31.389 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:31.382+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.384138+0000) 2019-10-13T07:22:31.390 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:31.384+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.386228+0000) 2019-10-13T07:22:31.390 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:31.385+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.386606+0000) 2019-10-13T07:22:31.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:31.385+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.386426+0000) 2019-10-13T07:22:31.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:31.385+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.387069+0000) 2019-10-13T07:22:31.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:31.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:31.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:31.924+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.925359+0000) 2019-10-13T07:22:31.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:31.932+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.934059+0000) 2019-10-13T07:22:32.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:32.795+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.796644+0000) 2019-10-13T07:22:32.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:32.796+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.798408+0000) 2019-10-13T07:22:36.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:36.390+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.391826+0000) 2019-10-13T07:22:36.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:36.390+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.392313+0000) 2019-10-13T07:22:36.395 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:36.391+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.392416+0000) 2019-10-13T07:22:36.395 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:36.391+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.392567+0000) 2019-10-13T07:22:36.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:36.389+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.391059+0000) 2019-10-13T07:22:36.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:36.391+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.392767+0000) 2019-10-13T07:22:36.396 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:36.391+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.393199+0000) 2019-10-13T07:22:36.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:36.390+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:36.392319+0000) 2019-10-13T07:22:36.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:36.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:37.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:37.794+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.794566+0000) 2019-10-13T07:22:38.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:38.342+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:38.342847+0000) 2019-10-13T07:22:38.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:38.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:38.679961+0000) 2019-10-13T07:22:38.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:38.679+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:38.679961+0000) 2019-10-13T07:22:41.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:41.397+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.398940+0000) 2019-10-13T07:22:41.402 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:41.397+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.399056+0000) 2019-10-13T07:22:41.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:41.397+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.399345+0000) 2019-10-13T07:22:41.403 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:41.398+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.399506+0000) 2019-10-13T07:22:41.403 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:41.396+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.397389+0000) 2019-10-13T07:22:41.403 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:41.398+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.399901+0000) 2019-10-13T07:22:41.404 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:41.398+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.400064+0000) 2019-10-13T07:22:41.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:41.400+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.401482+0000) 2019-10-13T07:22:41.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:41.402+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:41.932 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:41.926+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.927575+0000) 2019-10-13T07:22:41.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:41.933+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.934213+0000) 2019-10-13T07:22:42.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:42.796+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.796804+0000) 2019-10-13T07:22:42.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:42.797+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.798597+0000) 2019-10-13T07:22:45.516 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:22:45.516 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:45.520 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:22:45.521 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:46.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:46.402+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.403623+0000) 2019-10-13T07:22:46.409 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:46.405+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.405767+0000) 2019-10-13T07:22:46.409 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:46.405+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.406176+0000) 2019-10-13T07:22:46.409 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:46.406+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.406719+0000) 2019-10-13T07:22:46.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:46.406+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.406853+0000) 2019-10-13T07:22:46.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:46.406+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.406997+0000) 2019-10-13T07:22:46.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:46.406+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.407311+0000) 2019-10-13T07:22:46.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:46.405+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:46.406322+0000) 2019-10-13T07:22:46.413 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:46.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:47.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:47.794+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.794722+0000) 2019-10-13T07:22:48.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:48.342+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:48.343015+0000) 2019-10-13T07:22:48.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:48.679+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:48.680136+0000) 2019-10-13T07:22:48.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:48.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:48.680136+0000) 2019-10-13T07:22:51.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:51.412+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.412524+0000) 2019-10-13T07:22:51.416 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:51.410+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.411110+0000) 2019-10-13T07:22:51.417 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:51.412+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.413270+0000) 2019-10-13T07:22:51.417 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:51.412+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.413142+0000) 2019-10-13T07:22:51.417 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:51.413+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.413480+0000) 2019-10-13T07:22:51.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:51.413+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.413597+0000) 2019-10-13T07:22:51.417 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:51.413+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.413832+0000) 2019-10-13T07:22:51.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:51.413+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.413968+0000) 2019-10-13T07:22:51.420 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:51.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:51.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:51.929+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.929663+0000) 2019-10-13T07:22:51.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:51.934+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.934342+0000) 2019-10-13T07:22:52.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:52.796+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.796979+0000) 2019-10-13T07:22:52.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:52.798+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.798754+0000) 2019-10-13T07:22:56.423 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:56.419+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.419797+0000) 2019-10-13T07:22:56.423 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:56.419+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.419972+0000) 2019-10-13T07:22:56.423 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:22:56.419+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.420179+0000) 2019-10-13T07:22:56.424 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:22:56.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.418505+0000) 2019-10-13T07:22:56.424 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:22:56.420+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.420802+0000) 2019-10-13T07:22:56.424 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:56.420+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.421019+0000) 2019-10-13T07:22:56.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:56.420+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.421082+0000) 2019-10-13T07:22:56.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:22:56.420+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:56.420860+0000) 2019-10-13T07:22:56.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:22:56.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120259 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:22:57.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:22:57.794+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.794911+0000) 2019-10-13T07:22:58.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:22:58.343+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:58.343195+0000) 2019-10-13T07:22:58.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:22:58.679+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:58.680275+0000) 2019-10-13T07:22:58.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:22:58.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:58.680270+0000) 2019-10-13T07:23:01.430 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:01.425+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.426873+0000) 2019-10-13T07:23:01.430 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:01.425+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.427122+0000) 2019-10-13T07:23:01.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:01.425+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.426172+0000) 2019-10-13T07:23:01.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:01.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.427593+0000) 2019-10-13T07:23:01.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:01.426+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.427958+0000) 2019-10-13T07:23:01.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:01.426+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.428172+0000) 2019-10-13T07:23:01.432 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:01.426+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.427827+0000) 2019-10-13T07:23:01.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:01.426+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.427512+0000) 2019-10-13T07:23:01.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:01.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120292 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:01.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:01.931+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.932029+0000) 2019-10-13T07:23:01.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:01.933+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.934497+0000) 2019-10-13T07:23:02.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:02.796+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.797185+0000) 2019-10-13T07:23:02.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:02.798+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.798936+0000) 2019-10-13T07:23:06.438 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:06.433+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.435180+0000) 2019-10-13T07:23:06.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:06.433+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.435366+0000) 2019-10-13T07:23:06.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:06.434+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.435505+0000) 2019-10-13T07:23:06.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:06.434+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.435669+0000) 2019-10-13T07:23:06.439 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:06.434+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.435867+0000) 2019-10-13T07:23:06.440 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:06.433+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.434315+0000) 2019-10-13T07:23:06.440 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:06.435+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.436438+0000) 2019-10-13T07:23:06.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:06.436+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:06.436902+0000) 2019-10-13T07:23:06.442 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:06.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:07.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:07.794+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.795082+0000) 2019-10-13T07:23:08.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:08.342+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:08.343377+0000) 2019-10-13T07:23:08.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:08.679+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:08.680441+0000) 2019-10-13T07:23:08.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:08.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:08.680444+0000) 2019-10-13T07:23:11.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:11.440+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.441456+0000) 2019-10-13T07:23:11.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:11.439+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.440062+0000) 2019-10-13T07:23:11.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:11.440+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.442071+0000) 2019-10-13T07:23:11.446 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:11.440+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.442262+0000) 2019-10-13T07:23:11.446 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:11.440+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.442365+0000) 2019-10-13T07:23:11.446 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:11.441+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.442480+0000) 2019-10-13T07:23:11.446 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:11.441+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.442614+0000) 2019-10-13T07:23:11.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:11.441+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.442082+0000) 2019-10-13T07:23:11.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:11.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120352 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:11.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:11.933+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.934644+0000) 2019-10-13T07:23:11.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:11.934+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.934414+0000) 2019-10-13T07:23:12.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:12.796+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.797378+0000) 2019-10-13T07:23:12.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:12.798+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.799087+0000) 2019-10-13T07:23:15.679 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:23:15.679 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:15.683 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:23:15.683 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:16.451 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:16.447+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.448435+0000) 2019-10-13T07:23:16.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:16.447+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.448667+0000) 2019-10-13T07:23:16.452 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:16.447+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.448756+0000) 2019-10-13T07:23:16.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:16.447+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.449117+0000) 2019-10-13T07:23:16.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:16.448+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.449542+0000) 2019-10-13T07:23:16.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:16.448+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.449699+0000) 2019-10-13T07:23:16.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:16.446+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.447386+0000) 2019-10-13T07:23:16.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:16.449+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:16.450433+0000) 2019-10-13T07:23:16.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:16.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120381 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:17.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:17.794+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.795259+0000) 2019-10-13T07:23:18.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:18.342+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:18.343514+0000) 2019-10-13T07:23:18.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:18.680+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:18.680643+0000) 2019-10-13T07:23:18.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:18.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:18.680664+0000) 2019-10-13T07:23:21.458 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:21.454+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.455531+0000) 2019-10-13T07:23:21.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:21.452+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.453822+0000) 2019-10-13T07:23:21.459 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:21.454+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.455955+0000) 2019-10-13T07:23:21.460 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:21.454+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.456278+0000) 2019-10-13T07:23:21.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:21.454+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.456165+0000) 2019-10-13T07:23:21.460 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:21.455+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.456480+0000) 2019-10-13T07:23:21.460 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:21.455+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.456979+0000) 2019-10-13T07:23:21.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:21.455+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.457075+0000) 2019-10-13T07:23:21.464 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:21.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:21.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:21.933+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.934766+0000) 2019-10-13T07:23:21.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:21.935+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.936765+0000) 2019-10-13T07:23:22.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:22.796+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.797530+0000) 2019-10-13T07:23:22.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:22.797+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.799280+0000) 2019-10-13T07:23:26.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:26.461+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.462761+0000) 2019-10-13T07:23:26.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:26.461+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.462947+0000) 2019-10-13T07:23:26.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:26.461+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.463178+0000) 2019-10-13T07:23:26.467 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:26.460+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.461881+0000) 2019-10-13T07:23:26.467 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:26.462+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.463461+0000) 2019-10-13T07:23:26.467 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:26.462+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.464045+0000) 2019-10-13T07:23:26.468 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:26.462+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.464219+0000) 2019-10-13T07:23:26.468 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:26.462+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:26.463981+0000) 2019-10-13T07:23:26.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:26.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120443 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:27.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:27.795+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.795478+0000) 2019-10-13T07:23:28.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:28.342+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:28.343632+0000) 2019-10-13T07:23:28.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:28.680+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:28.680828+0000) 2019-10-13T07:23:28.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:28.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:28.680794+0000) 2019-10-13T07:23:31.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:31.468+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.469273+0000) 2019-10-13T07:23:31.472 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:31.469+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.469594+0000) 2019-10-13T07:23:31.473 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:31.469+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.469785+0000) 2019-10-13T07:23:31.473 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:31.466+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.467840+0000) 2019-10-13T07:23:31.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:31.469+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.470166+0000) 2019-10-13T07:23:31.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:31.469+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.470269+0000) 2019-10-13T07:23:31.474 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:31.470+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.470500+0000) 2019-10-13T07:23:31.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:31.469+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.470851+0000) 2019-10-13T07:23:31.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:31.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120474 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:31.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:31.934+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.934949+0000) 2019-10-13T07:23:31.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:31.938+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.939010+0000) 2019-10-13T07:23:32.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:32.797+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.797703+0000) 2019-10-13T07:23:32.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:32.798+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.799446+0000) 2019-10-13T07:23:36.478 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:36.475+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.475583+0000) 2019-10-13T07:23:36.478 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:36.475+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.475657+0000) 2019-10-13T07:23:36.479 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:36.473+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.474372+0000) 2019-10-13T07:23:36.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:36.476+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.476483+0000) 2019-10-13T07:23:36.480 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:36.476+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.477136+0000) 2019-10-13T07:23:36.480 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:36.476+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.477223+0000) 2019-10-13T07:23:36.480 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:36.477+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.477575+0000) 2019-10-13T07:23:36.481 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:36.475+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:36.476509+0000) 2019-10-13T07:23:36.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:36.480+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:37.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:37.795+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.795654+0000) 2019-10-13T07:23:38.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:38.343+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:38.343800+0000) 2019-10-13T07:23:38.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:38.680+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:38.680982+0000) 2019-10-13T07:23:38.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:38.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:38.680972+0000) 2019-10-13T07:23:41.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:41.482+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.482642+0000) 2019-10-13T07:23:41.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:41.482+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.482822+0000) 2019-10-13T07:23:41.486 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:41.482+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.483138+0000) 2019-10-13T07:23:41.486 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:41.482+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.483213+0000) 2019-10-13T07:23:41.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:41.480+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.481672+0000) 2019-10-13T07:23:41.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:41.483+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.483509+0000) 2019-10-13T07:23:41.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:41.483+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.483550+0000) 2019-10-13T07:23:41.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:41.483+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.484011+0000) 2019-10-13T07:23:41.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:41.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120536 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:41.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:41.934+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.935084+0000) 2019-10-13T07:23:41.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:41.940+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.941102+0000) 2019-10-13T07:23:42.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:42.797+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.797890+0000) 2019-10-13T07:23:42.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:42.799+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.799609+0000) 2019-10-13T07:23:45.833 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:23:45.834 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:45.838 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:23:45.838 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:46.491 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:46.485+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.486328+0000) 2019-10-13T07:23:46.491 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:46.488+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.488496+0000) 2019-10-13T07:23:46.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:46.488+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.488751+0000) 2019-10-13T07:23:46.492 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:46.488+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.489214+0000) 2019-10-13T07:23:46.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:46.488+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.489101+0000) 2019-10-13T07:23:46.493 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:46.489+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.489542+0000) 2019-10-13T07:23:46.493 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:46.489+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.489753+0000) 2019-10-13T07:23:46.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:46.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:46.489622+0000) 2019-10-13T07:23:46.496 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:46.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120565 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:47.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:47.794+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.795824+0000) 2019-10-13T07:23:48.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:48.342+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:48.343998+0000) 2019-10-13T07:23:48.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:48.680+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:48.681177+0000) 2019-10-13T07:23:48.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:48.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:48.681171+0000) 2019-10-13T07:23:51.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:51.493+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.494973+0000) 2019-10-13T07:23:51.499 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:51.492+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.493420+0000) 2019-10-13T07:23:51.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:51.494+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.495644+0000) 2019-10-13T07:23:51.499 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:51.494+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.495811+0000) 2019-10-13T07:23:51.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:51.494+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.495940+0000) 2019-10-13T07:23:51.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:51.494+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.496295+0000) 2019-10-13T07:23:51.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:51.495+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.496510+0000) 2019-10-13T07:23:51.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:51.495+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.496448+0000) 2019-10-13T07:23:51.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:51.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:51.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:51.933+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.935294+0000) 2019-10-13T07:23:51.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:51.942+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.943437+0000) 2019-10-13T07:23:52.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:52.797+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.798110+0000) 2019-10-13T07:23:52.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:52.799+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.799739+0000) 2019-10-13T07:23:56.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:56.500+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.502029+0000) 2019-10-13T07:23:56.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:56.500+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.502179+0000) 2019-10-13T07:23:56.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:23:56.500+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.502362+0000) 2019-10-13T07:23:56.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:23:56.500+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.501282+0000) 2019-10-13T07:23:56.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:56.501+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.502529+0000) 2019-10-13T07:23:56.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:23:56.501+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.502757+0000) 2019-10-13T07:23:56.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:56.502+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.503520+0000) 2019-10-13T07:23:56.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:23:56.502+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:56.503504+0000) 2019-10-13T07:23:56.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:23:56.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:23:57.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:23:57.795+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.796035+0000) 2019-10-13T07:23:58.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:23:58.343+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:58.344173+0000) 2019-10-13T07:23:58.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:23:58.680+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:58.681370+0000) 2019-10-13T07:23:58.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:23:58.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:58.681406+0000) 2019-10-13T07:24:01.511 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:01.505+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.506317+0000) 2019-10-13T07:24:01.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:01.507+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.508457+0000) 2019-10-13T07:24:01.512 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:01.507+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.508545+0000) 2019-10-13T07:24:01.512 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:01.507+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.508708+0000) 2019-10-13T07:24:01.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:01.507+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.509287+0000) 2019-10-13T07:24:01.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:01.508+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.509466+0000) 2019-10-13T07:24:01.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:01.508+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.509662+0000) 2019-10-13T07:24:01.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:01.508+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.509139+0000) 2019-10-13T07:24:01.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:01.510+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:01.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:01.934+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.935429+0000) 2019-10-13T07:24:01.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:01.945+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.945484+0000) 2019-10-13T07:24:02.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:02.797+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.798313+0000) 2019-10-13T07:24:02.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:02.799+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.799936+0000) 2019-10-13T07:24:06.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:06.512+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.514114+0000) 2019-10-13T07:24:06.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:06.512+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.512876+0000) 2019-10-13T07:24:06.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:06.513+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.514510+0000) 2019-10-13T07:24:06.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:06.513+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.514928+0000) 2019-10-13T07:24:06.518 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:06.513+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.515086+0000) 2019-10-13T07:24:06.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:06.513+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.515245+0000) 2019-10-13T07:24:06.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:06.514+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.515420+0000) 2019-10-13T07:24:06.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:06.513+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:06.514330+0000) 2019-10-13T07:24:06.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:06.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120687 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:07.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:07.795+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.796230+0000) 2019-10-13T07:24:08.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:08.343+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:08.344306+0000) 2019-10-13T07:24:08.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:08.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:08.681505+0000) 2019-10-13T07:24:08.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:08.680+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:08.681527+0000) 2019-10-13T07:24:11.524 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:11.518+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.519589+0000) 2019-10-13T07:24:11.525 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:11.520+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.521958+0000) 2019-10-13T07:24:11.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:11.520+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.522132+0000) 2019-10-13T07:24:11.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:11.521+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.522425+0000) 2019-10-13T07:24:11.526 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:11.521+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.522536+0000) 2019-10-13T07:24:11.527 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:11.521+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.522635+0000) 2019-10-13T07:24:11.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:11.521+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.522823+0000) 2019-10-13T07:24:11.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:11.522+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.523254+0000) 2019-10-13T07:24:11.529 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:11.525+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:11.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:11.934+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.935555+0000) 2019-10-13T07:24:11.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:11.946+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.947659+0000) 2019-10-13T07:24:12.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:12.798+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.798505+0000) 2019-10-13T07:24:12.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:12.799+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.800114+0000) 2019-10-13T07:24:15.914 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:24:15.915 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:15.919 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:24:15.919 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:16.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:16.527+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.528056+0000) 2019-10-13T07:24:16.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:16.528+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.528422+0000) 2019-10-13T07:24:16.532 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:16.528+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.528705+0000) 2019-10-13T07:24:16.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:16.528+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.528839+0000) 2019-10-13T07:24:16.532 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:16.528+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.528990+0000) 2019-10-13T07:24:16.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:16.528+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.529109+0000) 2019-10-13T07:24:16.533 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:16.524+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.526664+0000) 2019-10-13T07:24:16.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:16.531+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:16.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:16.527+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:16.529623+0000) 2019-10-13T07:24:17.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:17.795+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.796365+0000) 2019-10-13T07:24:18.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:18.344+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:18.344499+0000) 2019-10-13T07:24:18.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:18.681+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:18.681696+0000) 2019-10-13T07:24:18.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:18.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:18.681678+0000) 2019-10-13T07:24:21.537 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:21.530+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.532349+0000) 2019-10-13T07:24:21.538 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:21.532+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.533252+0000) 2019-10-13T07:24:21.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:21.533+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.533734+0000) 2019-10-13T07:24:21.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:21.533+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.533999+0000) 2019-10-13T07:24:21.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:21.533+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.534023+0000) 2019-10-13T07:24:21.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:21.533+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.534382+0000) 2019-10-13T07:24:21.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:21.534+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.534680+0000) 2019-10-13T07:24:21.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:21.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:21.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:21.533+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.535569+0000) 2019-10-13T07:24:21.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:21.935+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.935683+0000) 2019-10-13T07:24:21.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:21.949+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.949924+0000) 2019-10-13T07:24:22.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:22.798+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.798707+0000) 2019-10-13T07:24:22.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:22.799+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.800279+0000) 2019-10-13T07:24:26.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:26.539+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.539596+0000) 2019-10-13T07:24:26.543 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:26.537+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.538162+0000) 2019-10-13T07:24:26.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:26.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.539900+0000) 2019-10-13T07:24:26.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:26.540+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.540413+0000) 2019-10-13T07:24:26.544 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:26.540+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.540662+0000) 2019-10-13T07:24:26.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:26.540+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.540905+0000) 2019-10-13T07:24:26.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:26.540+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.541097+0000) 2019-10-13T07:24:26.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:26.540+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:26.541031+0000) 2019-10-13T07:24:26.546 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:26.543+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:27.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:27.796+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.796503+0000) 2019-10-13T07:24:28.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:28.344+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:28.344638+0000) 2019-10-13T07:24:28.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:28.681+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:28.681905+0000) 2019-10-13T07:24:28.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:28.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:28.681899+0000) 2019-10-13T07:24:31.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:31.545+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.545824+0000) 2019-10-13T07:24:31.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:31.545+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.546008+0000) 2019-10-13T07:24:31.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:31.543+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.544413+0000) 2019-10-13T07:24:31.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:31.546+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.546447+0000) 2019-10-13T07:24:31.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:31.546+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.546794+0000) 2019-10-13T07:24:31.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:31.546+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.547036+0000) 2019-10-13T07:24:31.551 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:31.546+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.547189+0000) 2019-10-13T07:24:31.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:31.546+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.546761+0000) 2019-10-13T07:24:31.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:31.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120840 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:31.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:31.935+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.935794+0000) 2019-10-13T07:24:31.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:31.951+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.952250+0000) 2019-10-13T07:24:32.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:32.798+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.798879+0000) 2019-10-13T07:24:32.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:32.799+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.800463+0000) 2019-10-13T07:24:36.555 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:36.551+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.552431+0000) 2019-10-13T07:24:36.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:36.551+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.552777+0000) 2019-10-13T07:24:36.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:36.551+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.553066+0000) 2019-10-13T07:24:36.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:36.551+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.553206+0000) 2019-10-13T07:24:36.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:36.551+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.553366+0000) 2019-10-13T07:24:36.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:36.552+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.553917+0000) 2019-10-13T07:24:36.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:36.551+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.551885+0000) 2019-10-13T07:24:36.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:36.552+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:36.552858+0000) 2019-10-13T07:24:36.560 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:36.556+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:37.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:37.795+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.796685+0000) 2019-10-13T07:24:38.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:38.343+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:38.344802+0000) 2019-10-13T07:24:38.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:38.680+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:38.682065+0000) 2019-10-13T07:24:38.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:38.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:38.682087+0000) 2019-10-13T07:24:41.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:41.558+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.559426+0000) 2019-10-13T07:24:41.562 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:41.558+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.559651+0000) 2019-10-13T07:24:41.563 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:41.557+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.558591+0000) 2019-10-13T07:24:41.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:41.558+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.560002+0000) 2019-10-13T07:24:41.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:41.558+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.560268+0000) 2019-10-13T07:24:41.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:41.559+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.560728+0000) 2019-10-13T07:24:41.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:41.559+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.560916+0000) 2019-10-13T07:24:41.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:41.560+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.560901+0000) 2019-10-13T07:24:41.567 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:41.563+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120902 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:41.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:41.934+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.935964+0000) 2019-10-13T07:24:41.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:41.954+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.954639+0000) 2019-10-13T07:24:42.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:42.797+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.799031+0000) 2019-10-13T07:24:42.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:42.800+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.800578+0000) 2019-10-13T07:24:46.058 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:24:46.058 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:46.063 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:24:46.063 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:46.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:46.565+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.566444+0000) 2019-10-13T07:24:46.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:46.565+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.566866+0000) 2019-10-13T07:24:46.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:46.565+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.566972+0000) 2019-10-13T07:24:46.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:46.564+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.564748+0000) 2019-10-13T07:24:46.571 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:46.565+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.567230+0000) 2019-10-13T07:24:46.571 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:46.566+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.567650+0000) 2019-10-13T07:24:46.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:46.566+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.567979+0000) 2019-10-13T07:24:46.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:46.565+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:46.566609+0000) 2019-10-13T07:24:46.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:46.569+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:47.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:47.796+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.796877+0000) 2019-10-13T07:24:48.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:48.344+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:48.344995+0000) 2019-10-13T07:24:48.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:48.681+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:48.682242+0000) 2019-10-13T07:24:48.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:48.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:48.682246+0000) 2019-10-13T07:24:51.575 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:51.570+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.571172+0000) 2019-10-13T07:24:51.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:51.572+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.573453+0000) 2019-10-13T07:24:51.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:51.572+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.573661+0000) 2019-10-13T07:24:51.577 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:51.572+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.573847+0000) 2019-10-13T07:24:51.577 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:51.572+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.573968+0000) 2019-10-13T07:24:51.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:51.572+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.574176+0000) 2019-10-13T07:24:51.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:51.572+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.573618+0000) 2019-10-13T07:24:51.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:51.573+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.574645+0000) 2019-10-13T07:24:51.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:51.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:51.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:51.934+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.936160+0000) 2019-10-13T07:24:51.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:51.956+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.956835+0000) 2019-10-13T07:24:52.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:52.798+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.799190+0000) 2019-10-13T07:24:52.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:52.800+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.800755+0000) 2019-10-13T07:24:56.583 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:24:56.578+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.580156+0000) 2019-10-13T07:24:56.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:56.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.580491+0000) 2019-10-13T07:24:56.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:56.578+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.580387+0000) 2019-10-13T07:24:56.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:24:56.579+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.580701+0000) 2019-10-13T07:24:56.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:24:56.578+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.579498+0000) 2019-10-13T07:24:56.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:56.579+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.580991+0000) 2019-10-13T07:24:56.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:56.579+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.581144+0000) 2019-10-13T07:24:56.585 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:24:56.580+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:56.580888+0000) 2019-10-13T07:24:56.587 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:24:56.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 120991 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:24:57.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:24:57.796+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.797031+0000) 2019-10-13T07:24:58.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:24:58.344+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:58.345184+0000) 2019-10-13T07:24:58.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:24:58.681+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:58.682436+0000) 2019-10-13T07:24:58.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:24:58.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:58.682460+0000) 2019-10-13T07:25:01.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:01.584+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.585823+0000) 2019-10-13T07:25:01.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:01.584+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.586158+0000) 2019-10-13T07:25:01.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:01.584+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.586356+0000) 2019-10-13T07:25:01.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:01.583+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.584129+0000) 2019-10-13T07:25:01.590 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:01.586+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.586848+0000) 2019-10-13T07:25:01.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:01.586+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.587114+0000) 2019-10-13T07:25:01.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:01.586+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.587301+0000) 2019-10-13T07:25:01.591 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:01.586+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.587060+0000) 2019-10-13T07:25:01.594 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:01.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121024 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:01.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:01.935+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.936343+0000) 2019-10-13T07:25:01.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:01.958+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.958914+0000) 2019-10-13T07:25:02.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:02.798+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.799340+0000) 2019-10-13T07:25:02.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:02.800+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.800898+0000) 2019-10-13T07:25:06.595 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:06.592+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.592994+0000) 2019-10-13T07:25:06.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:06.593+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.593415+0000) 2019-10-13T07:25:06.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:06.593+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.593527+0000) 2019-10-13T07:25:06.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:06.593+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.593665+0000) 2019-10-13T07:25:06.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:06.593+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.593854+0000) 2019-10-13T07:25:06.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:06.590+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.592180+0000) 2019-10-13T07:25:06.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:06.594+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.594584+0000) 2019-10-13T07:25:06.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:06.592+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:06.593985+0000) 2019-10-13T07:25:06.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:06.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:07.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:07.796+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.797223+0000) 2019-10-13T07:25:08.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:08.345+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:08.345372+0000) 2019-10-13T07:25:08.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:08.682+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:08.682622+0000) 2019-10-13T07:25:08.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:08.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:08.682622+0000) 2019-10-13T07:25:11.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:11.598+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.598683+0000) 2019-10-13T07:25:11.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:11.595+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.597604+0000) 2019-10-13T07:25:11.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:11.598+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.599296+0000) 2019-10-13T07:25:11.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:11.599+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.599686+0000) 2019-10-13T07:25:11.604 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:11.599+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.599913+0000) 2019-10-13T07:25:11.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:11.599+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.600196+0000) 2019-10-13T07:25:11.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:11.599+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.600060+0000) 2019-10-13T07:25:11.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:11.597+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.599359+0000) 2019-10-13T07:25:11.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:11.603+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121084 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:11.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:11.936+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.936498+0000) 2019-10-13T07:25:11.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:11.959+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.960830+0000) 2019-10-13T07:25:12.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:12.799+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.799480+0000) 2019-10-13T07:25:12.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:12.800+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.801025+0000) 2019-10-13T07:25:16.152 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:25:16.152 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:16.156 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:25:16.156 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:16.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:16.605+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.605511+0000) 2019-10-13T07:25:16.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:16.605+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.605820+0000) 2019-10-13T07:25:16.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:16.605+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.606150+0000) 2019-10-13T07:25:16.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:16.605+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.606234+0000) 2019-10-13T07:25:16.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:16.606+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.606610+0000) 2019-10-13T07:25:16.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:16.606+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.606793+0000) 2019-10-13T07:25:16.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:16.602+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.604684+0000) 2019-10-13T07:25:16.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:16.605+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:16.606886+0000) 2019-10-13T07:25:16.614 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:16.610+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:17.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:17.797+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.797430+0000) 2019-10-13T07:25:18.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:18.345+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:18.345578+0000) 2019-10-13T07:25:18.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:18.682+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:18.682812+0000) 2019-10-13T07:25:18.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:18.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:18.682809+0000) 2019-10-13T07:25:21.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:21.611+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.612446+0000) 2019-10-13T07:25:21.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:21.611+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.612819+0000) 2019-10-13T07:25:21.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:21.611+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.613025+0000) 2019-10-13T07:25:21.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:21.611+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.613297+0000) 2019-10-13T07:25:21.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:21.611+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.613217+0000) 2019-10-13T07:25:21.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:21.610+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.611654+0000) 2019-10-13T07:25:21.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:21.612+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.613158+0000) 2019-10-13T07:25:21.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:21.612+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.613695+0000) 2019-10-13T07:25:21.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:21.615+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:21.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:21.935+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.936621+0000) 2019-10-13T07:25:21.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:21.962+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.963303+0000) 2019-10-13T07:25:22.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:22.798+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.799651+0000) 2019-10-13T07:25:22.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:22.800+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.801150+0000) 2019-10-13T07:25:26.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:26.616+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.617262+0000) 2019-10-13T07:25:26.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:26.617+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.619316+0000) 2019-10-13T07:25:26.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:26.618+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.619439+0000) 2019-10-13T07:25:26.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:26.618+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.620218+0000) 2019-10-13T07:25:26.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:26.619+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.620449+0000) 2019-10-13T07:25:26.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:26.619+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.620578+0000) 2019-10-13T07:25:26.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:26.619+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.620791+0000) 2019-10-13T07:25:26.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:26.618+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:26.619719+0000) 2019-10-13T07:25:26.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:26.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121175 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:27.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:27.796+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.797599+0000) 2019-10-13T07:25:28.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:28.344+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:28.345726+0000) 2019-10-13T07:25:28.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:28.682+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:28.683007+0000) 2019-10-13T07:25:28.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:28.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:28.683008+0000) 2019-10-13T07:25:31.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:31.624+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.626084+0000) 2019-10-13T07:25:31.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:31.624+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.626303+0000) 2019-10-13T07:25:31.630 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:31.624+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.624727+0000) 2019-10-13T07:25:31.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:31.625+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.626658+0000) 2019-10-13T07:25:31.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:31.625+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.627022+0000) 2019-10-13T07:25:31.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:31.625+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.627165+0000) 2019-10-13T07:25:31.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:31.626+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.627514+0000) 2019-10-13T07:25:31.631 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:31.625+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.626411+0000) 2019-10-13T07:25:31.634 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:31.629+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:31.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:31.935+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.936768+0000) 2019-10-13T07:25:31.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:31.964+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.965428+0000) 2019-10-13T07:25:32.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:32.799+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.799844+0000) 2019-10-13T07:25:32.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:32.800+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.801343+0000) 2019-10-13T07:25:36.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:36.631+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.633220+0000) 2019-10-13T07:25:36.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:36.631+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.633373+0000) 2019-10-13T07:25:36.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:36.632+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.633470+0000) 2019-10-13T07:25:36.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:36.632+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.633581+0000) 2019-10-13T07:25:36.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:36.632+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.633759+0000) 2019-10-13T07:25:36.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:36.632+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.633798+0000) 2019-10-13T07:25:36.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:36.630+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.630997+0000) 2019-10-13T07:25:36.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:36.632+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:36.633280+0000) 2019-10-13T07:25:36.639 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:36.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:37.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:37.797+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.797746+0000) 2019-10-13T07:25:38.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:38.345+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:38.345903+0000) 2019-10-13T07:25:38.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:38.682+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:38.683197+0000) 2019-10-13T07:25:38.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:38.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:38.683204+0000) 2019-10-13T07:25:41.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:41.637+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.638866+0000) 2019-10-13T07:25:41.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:41.637+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.639214+0000) 2019-10-13T07:25:41.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:41.636+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.637263+0000) 2019-10-13T07:25:41.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:41.638+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.639736+0000) 2019-10-13T07:25:41.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:41.638+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.639890+0000) 2019-10-13T07:25:41.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:41.638+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.640189+0000) 2019-10-13T07:25:41.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:41.639+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.640390+0000) 2019-10-13T07:25:41.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:41.638+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.639560+0000) 2019-10-13T07:25:41.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:41.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121268 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:41.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:41.935+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.936928+0000) 2019-10-13T07:25:41.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:41.966+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.967711+0000) 2019-10-13T07:25:42.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:42.799+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.800027+0000) 2019-10-13T07:25:42.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:42.800+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.801522+0000) 2019-10-13T07:25:46.317 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:25:46.317 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:46.322 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:25:46.322 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:46.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:46.644+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.645732+0000) 2019-10-13T07:25:46.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:46.644+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.645840+0000) 2019-10-13T07:25:46.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:46.644+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.646208+0000) 2019-10-13T07:25:46.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:46.643+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.644719+0000) 2019-10-13T07:25:46.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:46.645+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.646392+0000) 2019-10-13T07:25:46.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:46.645+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.646981+0000) 2019-10-13T07:25:46.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:46.645+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.647155+0000) 2019-10-13T07:25:46.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:46.646+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:46.647117+0000) 2019-10-13T07:25:46.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:46.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:47.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:47.797+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.797887+0000) 2019-10-13T07:25:48.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:48.345+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:48.346113+0000) 2019-10-13T07:25:48.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:48.683+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:48.683428+0000) 2019-10-13T07:25:48.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:48.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:48.683423+0000) 2019-10-13T07:25:51.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:51.651+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.652035+0000) 2019-10-13T07:25:51.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:51.651+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.652234+0000) 2019-10-13T07:25:51.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:51.651+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.652358+0000) 2019-10-13T07:25:51.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:51.650+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.651601+0000) 2019-10-13T07:25:51.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:51.652+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.652912+0000) 2019-10-13T07:25:51.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:51.652+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.653280+0000) 2019-10-13T07:25:51.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:51.653+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.653641+0000) 2019-10-13T07:25:51.658 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:51.652+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.653607+0000) 2019-10-13T07:25:51.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:51.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:51.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:51.936+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.937110+0000) 2019-10-13T07:25:51.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:51.969+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.969958+0000) 2019-10-13T07:25:52.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:52.799+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.800218+0000) 2019-10-13T07:25:52.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:52.801+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.801712+0000) 2019-10-13T07:25:56.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:56.658+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.658837+0000) 2019-10-13T07:25:56.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:56.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.659258+0000) 2019-10-13T07:25:56.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:25:56.659+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.659581+0000) 2019-10-13T07:25:56.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:56.659+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.659698+0000) 2019-10-13T07:25:56.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:56.659+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.659893+0000) 2019-10-13T07:25:56.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:25:56.659+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.660058+0000) 2019-10-13T07:25:56.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:25:56.657+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.658465+0000) 2019-10-13T07:25:56.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:25:56.658+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:56.659391+0000) 2019-10-13T07:25:56.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:25:56.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:25:57.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:25:57.797+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.798041+0000) 2019-10-13T07:25:58.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:25:58.345+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:58.346267+0000) 2019-10-13T07:25:58.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:25:58.683+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:58.683624+0000) 2019-10-13T07:25:58.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:25:58.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:58.683645+0000) 2019-10-13T07:26:01.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:01.665+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.666033+0000) 2019-10-13T07:26:01.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:01.665+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.666246+0000) 2019-10-13T07:26:01.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:01.665+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.666372+0000) 2019-10-13T07:26:01.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:01.666+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.666548+0000) 2019-10-13T07:26:01.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:01.664+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.665871+0000) 2019-10-13T07:26:01.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:01.666+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.667148+0000) 2019-10-13T07:26:01.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:01.667+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.667428+0000) 2019-10-13T07:26:01.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:01.665+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.667364+0000) 2019-10-13T07:26:01.673 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:01.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:01.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:01.936+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.937278+0000) 2019-10-13T07:26:01.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:01.971+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.972342+0000) 2019-10-13T07:26:02.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:02.801+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.801958+0000) 2019-10-13T07:26:02.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:02.800+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.800438+0000) 2019-10-13T07:26:06.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:06.671+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.673171+0000) 2019-10-13T07:26:06.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:06.671+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.673336+0000) 2019-10-13T07:26:06.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:06.672+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.673685+0000) 2019-10-13T07:26:06.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:06.672+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.673674+0000) 2019-10-13T07:26:06.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:06.670+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.672213+0000) 2019-10-13T07:26:06.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:06.672+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.674135+0000) 2019-10-13T07:26:06.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:06.672+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.674297+0000) 2019-10-13T07:26:06.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:06.672+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:06.674295+0000) 2019-10-13T07:26:06.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:06.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:07.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:07.797+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.798221+0000) 2019-10-13T07:26:08.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:08.345+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:08.346475+0000) 2019-10-13T07:26:08.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:08.682+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:08.683780+0000) 2019-10-13T07:26:08.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:08.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:08.683781+0000) 2019-10-13T07:26:11.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:11.678+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.680347+0000) 2019-10-13T07:26:11.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:11.679+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.680461+0000) 2019-10-13T07:26:11.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:11.679+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.680722+0000) 2019-10-13T07:26:11.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:11.679+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.680526+0000) 2019-10-13T07:26:11.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:11.679+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.680871+0000) 2019-10-13T07:26:11.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:11.678+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.679341+0000) 2019-10-13T07:26:11.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:11.679+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.681384+0000) 2019-10-13T07:26:11.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:11.681+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.681883+0000) 2019-10-13T07:26:11.686 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:11.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:11.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:11.936+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.937467+0000) 2019-10-13T07:26:11.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:11.973+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.974401+0000) 2019-10-13T07:26:12.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:12.799+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.800629+0000) 2019-10-13T07:26:12.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:12.801+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.802100+0000) 2019-10-13T07:26:16.480 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:26:16.480 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:16.486 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:26:16.486 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:16.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:16.683+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.685318+0000) 2019-10-13T07:26:16.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:16.682+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.683626+0000) 2019-10-13T07:26:16.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:16.684+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.685804+0000) 2019-10-13T07:26:16.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:16.684+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.685855+0000) 2019-10-13T07:26:16.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:16.684+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.686029+0000) 2019-10-13T07:26:16.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:16.684+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.686212+0000) 2019-10-13T07:26:16.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:16.685+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.686689+0000) 2019-10-13T07:26:16.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:16.684+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:16.685345+0000) 2019-10-13T07:26:16.693 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:16.688+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:17.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:17.797+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.798430+0000) 2019-10-13T07:26:18.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:18.345+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:18.346687+0000) 2019-10-13T07:26:18.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:18.683+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:18.683932+0000) 2019-10-13T07:26:18.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:18.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:18.683931+0000) 2019-10-13T07:26:21.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:21.688+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.689339+0000) 2019-10-13T07:26:21.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:21.690+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.691465+0000) 2019-10-13T07:26:21.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:21.690+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.691568+0000) 2019-10-13T07:26:21.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:21.690+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.691902+0000) 2019-10-13T07:26:21.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:21.690+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.692179+0000) 2019-10-13T07:26:21.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:21.690+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.692283+0000) 2019-10-13T07:26:21.697 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:21.691+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.692500+0000) 2019-10-13T07:26:21.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:21.692+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.693241+0000) 2019-10-13T07:26:21.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:21.695+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121512 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:21.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:21.936+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.937577+0000) 2019-10-13T07:26:21.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:21.975+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.976511+0000) 2019-10-13T07:26:22.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:22.800+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.800813+0000) 2019-10-13T07:26:22.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:22.801+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.802251+0000) 2019-10-13T07:26:26.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:26.696+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.697998+0000) 2019-10-13T07:26:26.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:26.696+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.698267+0000) 2019-10-13T07:26:26.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:26.697+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.698580+0000) 2019-10-13T07:26:26.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:26.697+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.698675+0000) 2019-10-13T07:26:26.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:26.697+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.698911+0000) 2019-10-13T07:26:26.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:26.698+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.699408+0000) 2019-10-13T07:26:26.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:26.696+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.697363+0000) 2019-10-13T07:26:26.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:26.698+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:26.699134+0000) 2019-10-13T07:26:26.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:26.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121541 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:27.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:27.798+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.798621+0000) 2019-10-13T07:26:28.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:28.346+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:28.346875+0000) 2019-10-13T07:26:28.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:28.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:28.684115+0000) 2019-10-13T07:26:28.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:28.683+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:28.684113+0000) 2019-10-13T07:26:31.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:31.703+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.704703+0000) 2019-10-13T07:26:31.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:31.703+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.704783+0000) 2019-10-13T07:26:31.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:31.703+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.705145+0000) 2019-10-13T07:26:31.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:31.702+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.703470+0000) 2019-10-13T07:26:31.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:31.704+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.705733+0000) 2019-10-13T07:26:31.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:31.704+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.705909+0000) 2019-10-13T07:26:31.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:31.704+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.706231+0000) 2019-10-13T07:26:31.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:31.705+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.705871+0000) 2019-10-13T07:26:31.712 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:31.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121572 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:31.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:31.936+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.937712+0000) 2019-10-13T07:26:31.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:31.978+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.978866+0000) 2019-10-13T07:26:32.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:32.800+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.801039+0000) 2019-10-13T07:26:32.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:32.802+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.802433+0000) 2019-10-13T07:26:36.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:36.711+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.711444+0000) 2019-10-13T07:26:36.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:36.711+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.711811+0000) 2019-10-13T07:26:36.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:36.711+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.711976+0000) 2019-10-13T07:26:36.718 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:36.709+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.710611+0000) 2019-10-13T07:26:36.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:36.712+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.713234+0000) 2019-10-13T07:26:36.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:36.712+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.712541+0000) 2019-10-13T07:26:36.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:36.712+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.712931+0000) 2019-10-13T07:26:36.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:36.712+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:36.713176+0000) 2019-10-13T07:26:36.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:36.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:37.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:37.798+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.798801+0000) 2019-10-13T07:26:38.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:38.346+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:38.347082+0000) 2019-10-13T07:26:38.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:38.683+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:38.684310+0000) 2019-10-13T07:26:38.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:38.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:38.684316+0000) 2019-10-13T07:26:41.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:41.718+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.718577+0000) 2019-10-13T07:26:41.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:41.718+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.718749+0000) 2019-10-13T07:26:41.722 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:41.718+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.718858+0000) 2019-10-13T07:26:41.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:41.718+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.719007+0000) 2019-10-13T07:26:41.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:41.718+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.719276+0000) 2019-10-13T07:26:41.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:41.716+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.717332+0000) 2019-10-13T07:26:41.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:41.717+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.718495+0000) 2019-10-13T07:26:41.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:41.719+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.719863+0000) 2019-10-13T07:26:41.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:41.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:41.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:41.937+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.937857+0000) 2019-10-13T07:26:41.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:41.981+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.981207+0000) 2019-10-13T07:26:42.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:42.800+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.801249+0000) 2019-10-13T07:26:42.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:42.802+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.802614+0000) 2019-10-13T07:26:46.587 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:26:46.587 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:46.593 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:26:46.593 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:46.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:46.724+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.724634+0000) 2019-10-13T07:26:46.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:46.722+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.723214+0000) 2019-10-13T07:26:46.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:46.724+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.725301+0000) 2019-10-13T07:26:46.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:46.725+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.725427+0000) 2019-10-13T07:26:46.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:46.725+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.725761+0000) 2019-10-13T07:26:46.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:46.725+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.725964+0000) 2019-10-13T07:26:46.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:46.725+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.726112+0000) 2019-10-13T07:26:46.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:46.723+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:46.724694+0000) 2019-10-13T07:26:46.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:46.728+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:47.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:47.798+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.798995+0000) 2019-10-13T07:26:48.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:48.347+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:48.347258+0000) 2019-10-13T07:26:48.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:48.684+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:48.684500+0000) 2019-10-13T07:26:48.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:48.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:48.684501+0000) 2019-10-13T07:26:51.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:51.730+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.730633+0000) 2019-10-13T07:26:51.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:51.730+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.731038+0000) 2019-10-13T07:26:51.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:51.730+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.731287+0000) 2019-10-13T07:26:51.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:51.731+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.731464+0000) 2019-10-13T07:26:51.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:51.728+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.729851+0000) 2019-10-13T07:26:51.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:51.731+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.731891+0000) 2019-10-13T07:26:51.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:51.731+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.731507+0000) 2019-10-13T07:26:51.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:51.730+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.731900+0000) 2019-10-13T07:26:51.737 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:51.734+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121694 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:51.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:51.937+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.938049+0000) 2019-10-13T07:26:51.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:51.982+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.983677+0000) 2019-10-13T07:26:52.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:52.801+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.801438+0000) 2019-10-13T07:26:52.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:52.801+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.802765+0000) 2019-10-13T07:26:56.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:56.734+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.735866+0000) 2019-10-13T07:26:56.739 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:56.734+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.736012+0000) 2019-10-13T07:26:56.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:56.734+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.736140+0000) 2019-10-13T07:26:56.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:26:56.735+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.736423+0000) 2019-10-13T07:26:56.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:26:56.735+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.736529+0000) 2019-10-13T07:26:56.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:26:56.733+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.735330+0000) 2019-10-13T07:26:56.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:56.735+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.736803+0000) 2019-10-13T07:26:56.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:26:56.734+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:56.736639+0000) 2019-10-13T07:26:56.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:26:56.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:26:57.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:26:57.798+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.799236+0000) 2019-10-13T07:26:58.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:26:58.346+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:58.347455+0000) 2019-10-13T07:26:58.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:26:58.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:58.684655+0000) 2019-10-13T07:26:58.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:26:58.683+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:58.684687+0000) 2019-10-13T07:27:01.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:01.739+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.741315+0000) 2019-10-13T07:27:01.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:01.740+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.741761+0000) 2019-10-13T07:27:01.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:01.738+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.740500+0000) 2019-10-13T07:27:01.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:01.740+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.741933+0000) 2019-10-13T07:27:01.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:01.740+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.742121+0000) 2019-10-13T07:27:01.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:01.741+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.742781+0000) 2019-10-13T07:27:01.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:01.741+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.742861+0000) 2019-10-13T07:27:01.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:01.741+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.743720+0000) 2019-10-13T07:27:01.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:01.744+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:01.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:01.936+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.938218+0000) 2019-10-13T07:27:01.991 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:01.985+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.986679+0000) 2019-10-13T07:27:02.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:02.800+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.801594+0000) 2019-10-13T07:27:02.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:02.802+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.802935+0000) 2019-10-13T07:27:06.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:06.745+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.746120+0000) 2019-10-13T07:27:06.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:06.746+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.748378+0000) 2019-10-13T07:27:06.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:06.747+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.748483+0000) 2019-10-13T07:27:06.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:06.747+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.748774+0000) 2019-10-13T07:27:06.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:06.747+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.748653+0000) 2019-10-13T07:27:06.752 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:06.747+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.749190+0000) 2019-10-13T07:27:06.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:06.748+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.749426+0000) 2019-10-13T07:27:06.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:06.748+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:06.749531+0000) 2019-10-13T07:27:06.756 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:06.752+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:07.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:07.798+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.799456+0000) 2019-10-13T07:27:08.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:08.346+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:08.347645+0000) 2019-10-13T07:27:08.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:08.683+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:08.684822+0000) 2019-10-13T07:27:08.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:08.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:08.684846+0000) 2019-10-13T07:27:11.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:11.753+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.755100+0000) 2019-10-13T07:27:11.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:11.753+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.755239+0000) 2019-10-13T07:27:11.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:11.752+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.753615+0000) 2019-10-13T07:27:11.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:11.754+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.755718+0000) 2019-10-13T07:27:11.759 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:11.754+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.755922+0000) 2019-10-13T07:27:11.760 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:11.754+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.756104+0000) 2019-10-13T07:27:11.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:11.754+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.756320+0000) 2019-10-13T07:27:11.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:11.755+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.755885+0000) 2019-10-13T07:27:11.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:11.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:11.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:11.937+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.938416+0000) 2019-10-13T07:27:11.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:11.988+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.988790+0000) 2019-10-13T07:27:12.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:12.801+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.801738+0000) 2019-10-13T07:27:12.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:12.802+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.803158+0000) 2019-10-13T07:27:16.746 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:27:16.746 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:16.750 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:27:16.751 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:16.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:16.759+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.760121+0000) 2019-10-13T07:27:16.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:16.760+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.761716+0000) 2019-10-13T07:27:16.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:16.760+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.762210+0000) 2019-10-13T07:27:16.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:16.761+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.762509+0000) 2019-10-13T07:27:16.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:16.761+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.762793+0000) 2019-10-13T07:27:16.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:16.761+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.762869+0000) 2019-10-13T07:27:16.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:16.761+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.763091+0000) 2019-10-13T07:27:16.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:16.761+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:16.762589+0000) 2019-10-13T07:27:16.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:16.766+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:17.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:17.799+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:17.799672+0000) 2019-10-13T07:27:18.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:18.346+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.347863+0000) 2019-10-13T07:27:18.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:18.684+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.685032+0000) 2019-10-13T07:27:18.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:18.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.685027+0000) 2019-10-13T07:27:21.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:21.768+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.768674+0000) 2019-10-13T07:27:21.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:21.768+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.768805+0000) 2019-10-13T07:27:21.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:21.768+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.769206+0000) 2019-10-13T07:27:21.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:21.766+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.767674+0000) 2019-10-13T07:27:21.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:21.769+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.769471+0000) 2019-10-13T07:27:21.773 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:21.769+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.769695+0000) 2019-10-13T07:27:21.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:21.769+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.769841+0000) 2019-10-13T07:27:21.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:21.769+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.769939+0000) 2019-10-13T07:27:21.775 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:21.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:21.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:21.938+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.938562+0000) 2019-10-13T07:27:21.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:21.990+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.991157+0000) 2019-10-13T07:27:22.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:22.801+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:22.801907+0000) 2019-10-13T07:27:22.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:22.802+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:22.803310+0000) 2019-10-13T07:27:26.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:26.774+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.774761+0000) 2019-10-13T07:27:26.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:26.772+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.773300+0000) 2019-10-13T07:27:26.778 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:26.775+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.775454+0000) 2019-10-13T07:27:26.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:26.775+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.775648+0000) 2019-10-13T07:27:26.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:26.775+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.776011+0000) 2019-10-13T07:27:26.779 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:26.775+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.775836+0000) 2019-10-13T07:27:26.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:26.775+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.776167+0000) 2019-10-13T07:27:26.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:26.774+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:26.775359+0000) 2019-10-13T07:27:26.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:26.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:27.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:27.799+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:27.799885+0000) 2019-10-13T07:27:28.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:28.347+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.348057+0000) 2019-10-13T07:27:28.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:28.684+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.685256+0000) 2019-10-13T07:27:28.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:28.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.685274+0000) 2019-10-13T07:27:31.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:31.779+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.779967+0000) 2019-10-13T07:27:31.785 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:31.781+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.782066+0000) 2019-10-13T07:27:31.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:31.781+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.782244+0000) 2019-10-13T07:27:31.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:31.782+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.782451+0000) 2019-10-13T07:27:31.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:31.782+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.782661+0000) 2019-10-13T07:27:31.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:31.782+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.782943+0000) 2019-10-13T07:27:31.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:31.782+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.783010+0000) 2019-10-13T07:27:31.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:31.782+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.783029+0000) 2019-10-13T07:27:31.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:31.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121938 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:31.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:31.938+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.938708+0000) 2019-10-13T07:27:31.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:31.993+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.993534+0000) 2019-10-13T07:27:32.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:32.801+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:32.802079+0000) 2019-10-13T07:27:32.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:32.802+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:32.803486+0000) 2019-10-13T07:27:36.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:36.789+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.789418+0000) 2019-10-13T07:27:36.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:36.789+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.789660+0000) 2019-10-13T07:27:36.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:36.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.787964+0000) 2019-10-13T07:27:36.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:36.789+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.789920+0000) 2019-10-13T07:27:36.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:36.789+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.790270+0000) 2019-10-13T07:27:36.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:36.789+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.790205+0000) 2019-10-13T07:27:36.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:36.790+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.790408+0000) 2019-10-13T07:27:36.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:36.788+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:36.789633+0000) 2019-10-13T07:27:36.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:36.793+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 121969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:37.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:37.800+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:37.800080+0000) 2019-10-13T07:27:38.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:38.348+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.348247+0000) 2019-10-13T07:27:38.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:38.685+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.685459+0000) 2019-10-13T07:27:38.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:38.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.685461+0000) 2019-10-13T07:27:41.798 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:41.793+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.795158+0000) 2019-10-13T07:27:41.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:41.793+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.795379+0000) 2019-10-13T07:27:41.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:41.794+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.795505+0000) 2019-10-13T07:27:41.799 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:41.794+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.795715+0000) 2019-10-13T07:27:41.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:41.794+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.795787+0000) 2019-10-13T07:27:41.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:41.793+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.794677+0000) 2019-10-13T07:27:41.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:41.794+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.796142+0000) 2019-10-13T07:27:41.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:41.795+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.796312+0000) 2019-10-13T07:27:41.802 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:41.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:41.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:41.937+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.938873+0000) 2019-10-13T07:27:42.000 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:41.995+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.995603+0000) 2019-10-13T07:27:42.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:42.801+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:42.802264+0000) 2019-10-13T07:27:42.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:42.803+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:42.803658+0000) 2019-10-13T07:27:46.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:46.799+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.801305+0000) 2019-10-13T07:27:46.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:46.800+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.801616+0000) 2019-10-13T07:27:46.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:46.800+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.801860+0000) 2019-10-13T07:27:46.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:46.800+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.802025+0000) 2019-10-13T07:27:46.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:46.801+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.802525+0000) 2019-10-13T07:27:46.805 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:46.800+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.802336+0000) 2019-10-13T07:27:46.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:46.799+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.800968+0000) 2019-10-13T07:27:46.806 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:46.800+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:46.801918+0000) 2019-10-13T07:27:46.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:46.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:46.885 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:27:46.885 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:46.891 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:27:46.891 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:47.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:47.799+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:47.800275+0000) 2019-10-13T07:27:48.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:48.347+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.348431+0000) 2019-10-13T07:27:48.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:48.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.685665+0000) 2019-10-13T07:27:48.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:48.684+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.685688+0000) 2019-10-13T07:27:51.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:51.806+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.807894+0000) 2019-10-13T07:27:51.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:51.806+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.808061+0000) 2019-10-13T07:27:51.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:51.806+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.808306+0000) 2019-10-13T07:27:51.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:51.807+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.808460+0000) 2019-10-13T07:27:51.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:51.807+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.808621+0000) 2019-10-13T07:27:51.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:51.805+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.806985+0000) 2019-10-13T07:27:51.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:51.807+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.809082+0000) 2019-10-13T07:27:51.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:51.807+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.809279+0000) 2019-10-13T07:27:51.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:51.811+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:51.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:51.938+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.939063+0000) 2019-10-13T07:27:52.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:51.996+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.998073+0000) 2019-10-13T07:27:52.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:52.801+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:52.802428+0000) 2019-10-13T07:27:52.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:52.802+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:52.803812+0000) 2019-10-13T07:27:56.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:27:56.813+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.814911+0000) 2019-10-13T07:27:56.818 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:56.813+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.815082+0000) 2019-10-13T07:27:56.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:27:56.812+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.814014+0000) 2019-10-13T07:27:56.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:56.814+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.815622+0000) 2019-10-13T07:27:56.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:56.814+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.815837+0000) 2019-10-13T07:27:56.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:56.814+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.815941+0000) 2019-10-13T07:27:56.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:27:56.814+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.816288+0000) 2019-10-13T07:27:56.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:27:56.814+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:56.815836+0000) 2019-10-13T07:27:56.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:27:56.818+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:27:57.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:27:57.799+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:57.800445+0000) 2019-10-13T07:27:58.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:27:58.347+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.348628+0000) 2019-10-13T07:27:58.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:27:58.685+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.685856+0000) 2019-10-13T07:27:58.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:27:58.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.685852+0000) 2019-10-13T07:28:01.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:01.820+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.821980+0000) 2019-10-13T07:28:01.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:01.819+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.820697+0000) 2019-10-13T07:28:01.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:01.821+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.822602+0000) 2019-10-13T07:28:01.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:01.821+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.822954+0000) 2019-10-13T07:28:01.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:01.821+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.822848+0000) 2019-10-13T07:28:01.826 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:01.821+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.823071+0000) 2019-10-13T07:28:01.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:01.821+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.823241+0000) 2019-10-13T07:28:01.827 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:01.821+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.822703+0000) 2019-10-13T07:28:01.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:01.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122122 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:01.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:01.937+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.939194+0000) 2019-10-13T07:28:02.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:01.999+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:02.000367+0000) 2019-10-13T07:28:02.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:02.802+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:02.802641+0000) 2019-10-13T07:28:02.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:02.803+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:02.803945+0000) 2019-10-13T07:28:06.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:06.827+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.828528+0000) 2019-10-13T07:28:06.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:06.827+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.828737+0000) 2019-10-13T07:28:06.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:06.827+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.827893+0000) 2019-10-13T07:28:06.833 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:06.827+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.829116+0000) 2019-10-13T07:28:06.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:06.828+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.829440+0000) 2019-10-13T07:28:06.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:06.828+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.829651+0000) 2019-10-13T07:28:06.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:06.828+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.829901+0000) 2019-10-13T07:28:06.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:06.829+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:06.829827+0000) 2019-10-13T07:28:06.836 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:06.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122151 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:07.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:07.800+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:07.800603+0000) 2019-10-13T07:28:08.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:08.348+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.348821+0000) 2019-10-13T07:28:08.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:08.685+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.686064+0000) 2019-10-13T07:28:08.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:08.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.686067+0000) 2019-10-13T07:28:11.839 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:11.835+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.835810+0000) 2019-10-13T07:28:11.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:11.835+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.836045+0000) 2019-10-13T07:28:11.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:11.835+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.836242+0000) 2019-10-13T07:28:11.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:11.836+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.836403+0000) 2019-10-13T07:28:11.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:11.834+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.834788+0000) 2019-10-13T07:28:11.840 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:11.836+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.837045+0000) 2019-10-13T07:28:11.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:11.837+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.837421+0000) 2019-10-13T07:28:11.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:11.836+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.837278+0000) 2019-10-13T07:28:11.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:11.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122182 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:11.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:11.938+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.939330+0000) 2019-10-13T07:28:12.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:12.002+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:12.002874+0000) 2019-10-13T07:28:12.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:12.802+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:12.802800+0000) 2019-10-13T07:28:12.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:12.803+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:12.804137+0000) 2019-10-13T07:28:16.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:16.842+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.842434+0000) 2019-10-13T07:28:16.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:16.842+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.842785+0000) 2019-10-13T07:28:16.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:16.840+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.841330+0000) 2019-10-13T07:28:16.846 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:16.842+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.843224+0000) 2019-10-13T07:28:16.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:16.842+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.843356+0000) 2019-10-13T07:28:16.847 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:16.843+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.843473+0000) 2019-10-13T07:28:16.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:16.843+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.843648+0000) 2019-10-13T07:28:16.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:16.842+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:16.843424+0000) 2019-10-13T07:28:16.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:16.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122213 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:17.069 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:28:17.069 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:17.073 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:28:17.073 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:17.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:17.800+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:17.800796+0000) 2019-10-13T07:28:18.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:18.348+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.349050+0000) 2019-10-13T07:28:18.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:18.685+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.686214+0000) 2019-10-13T07:28:18.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:18.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.686257+0000) 2019-10-13T07:28:21.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:21.845+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.846037+0000) 2019-10-13T07:28:21.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:21.847+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.848170+0000) 2019-10-13T07:28:21.852 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:21.847+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.848216+0000) 2019-10-13T07:28:21.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:21.848+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.848499+0000) 2019-10-13T07:28:21.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:21.848+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.848757+0000) 2019-10-13T07:28:21.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:21.848+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.849117+0000) 2019-10-13T07:28:21.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:21.848+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.849225+0000) 2019-10-13T07:28:21.853 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:21.847+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.848177+0000) 2019-10-13T07:28:21.855 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:21.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:21.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:21.939+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.939489+0000) 2019-10-13T07:28:22.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:22.004+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:22.005332+0000) 2019-10-13T07:28:22.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:22.802+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:22.802971+0000) 2019-10-13T07:28:22.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:22.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:22.804306+0000) 2019-10-13T07:28:26.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:26.852+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.853021+0000) 2019-10-13T07:28:26.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:26.853+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.855017+0000) 2019-10-13T07:28:26.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:26.853+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.855128+0000) 2019-10-13T07:28:26.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:26.853+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.855283+0000) 2019-10-13T07:28:26.859 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:26.854+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.855852+0000) 2019-10-13T07:28:26.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:26.854+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.855936+0000) 2019-10-13T07:28:26.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:26.854+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.856311+0000) 2019-10-13T07:28:26.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:26.855+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:26.855789+0000) 2019-10-13T07:28:26.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:26.858+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:27.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:27.799+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:27.800962+0000) 2019-10-13T07:28:28.352 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:28.348+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.349248+0000) 2019-10-13T07:28:28.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:28.685+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.686428+0000) 2019-10-13T07:28:28.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:28.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.686446+0000) 2019-10-13T07:28:31.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:31.858+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.859527+0000) 2019-10-13T07:28:31.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:31.860+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.861760+0000) 2019-10-13T07:28:31.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:31.860+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.861855+0000) 2019-10-13T07:28:31.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:31.860+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.862171+0000) 2019-10-13T07:28:31.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:31.861+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.862408+0000) 2019-10-13T07:28:31.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:31.861+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.862549+0000) 2019-10-13T07:28:31.866 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:31.861+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.862675+0000) 2019-10-13T07:28:31.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:31.861+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.862306+0000) 2019-10-13T07:28:31.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:31.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:31.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:31.938+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.939687+0000) 2019-10-13T07:28:32.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:32.007+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:32.007265+0000) 2019-10-13T07:28:32.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:32.802+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:32.803149+0000) 2019-10-13T07:28:32.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:32.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:32.804459+0000) 2019-10-13T07:28:36.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:36.864+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.865310+0000) 2019-10-13T07:28:36.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:36.866+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.867410+0000) 2019-10-13T07:28:36.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:36.866+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.867509+0000) 2019-10-13T07:28:36.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:36.866+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.867734+0000) 2019-10-13T07:28:36.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:36.867+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.868476+0000) 2019-10-13T07:28:36.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:36.867+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.868645+0000) 2019-10-13T07:28:36.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:36.867+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.868746+0000) 2019-10-13T07:28:36.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:36.868+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:36.869143+0000) 2019-10-13T07:28:36.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:36.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:37.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:37.800+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:37.801125+0000) 2019-10-13T07:28:38.352 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:38.348+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.349466+0000) 2019-10-13T07:28:38.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:38.685+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.686638+0000) 2019-10-13T07:28:38.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:38.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.686642+0000) 2019-10-13T07:28:41.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:41.873+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.874660+0000) 2019-10-13T07:28:41.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:41.873+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.874773+0000) 2019-10-13T07:28:41.878 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:41.873+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.874968+0000) 2019-10-13T07:28:41.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:41.871+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.873591+0000) 2019-10-13T07:28:41.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:41.874+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.875571+0000) 2019-10-13T07:28:41.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:41.874+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.875708+0000) 2019-10-13T07:28:41.879 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:41.874+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.875590+0000) 2019-10-13T07:28:41.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:41.873+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.874968+0000) 2019-10-13T07:28:41.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:41.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:41.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:41.939+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.939878+0000) 2019-10-13T07:28:42.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:42.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:42.009451+0000) 2019-10-13T07:28:42.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:42.802+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:42.803332+0000) 2019-10-13T07:28:42.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:42.803+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:42.804606+0000) 2019-10-13T07:28:46.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:46.878+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.880169+0000) 2019-10-13T07:28:46.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:46.880+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.882051+0000) 2019-10-13T07:28:46.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:46.881+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.882408+0000) 2019-10-13T07:28:46.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:46.881+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.882641+0000) 2019-10-13T07:28:46.886 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:46.881+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.882782+0000) 2019-10-13T07:28:46.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:46.881+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.882996+0000) 2019-10-13T07:28:46.886 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:46.882+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.883418+0000) 2019-10-13T07:28:46.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:46.881+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:46.883005+0000) 2019-10-13T07:28:46.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:46.886+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:47.221 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:28:47.222 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:47.226 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:28:47.226 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:47.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:47.799+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:47.801304+0000) 2019-10-13T07:28:48.352 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:48.349+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.349639+0000) 2019-10-13T07:28:48.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:48.686+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.686843+0000) 2019-10-13T07:28:48.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:48.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.686841+0000) 2019-10-13T07:28:51.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:51.887+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.889180+0000) 2019-10-13T07:28:51.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:51.886+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.888155+0000) 2019-10-13T07:28:51.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:51.887+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.889332+0000) 2019-10-13T07:28:51.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:51.888+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.889520+0000) 2019-10-13T07:28:51.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:51.888+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.889717+0000) 2019-10-13T07:28:51.894 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:51.888+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.890257+0000) 2019-10-13T07:28:51.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:51.888+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.890104+0000) 2019-10-13T07:28:51.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:51.888+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.890466+0000) 2019-10-13T07:28:51.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:51.892+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:51.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:51.939+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.940076+0000) 2019-10-13T07:28:52.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:52.010+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:52.011576+0000) 2019-10-13T07:28:52.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:52.802+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:52.803496+0000) 2019-10-13T07:28:52.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:52.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:52.804766+0000) 2019-10-13T07:28:56.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:28:56.895+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.895841+0000) 2019-10-13T07:28:56.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:28:56.895+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.895989+0000) 2019-10-13T07:28:56.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:28:56.893+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.894548+0000) 2019-10-13T07:28:56.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:56.896+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.896693+0000) 2019-10-13T07:28:56.900 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:56.896+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.896785+0000) 2019-10-13T07:28:56.900 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:56.896+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.896967+0000) 2019-10-13T07:28:56.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:56.896+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.897087+0000) 2019-10-13T07:28:56.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:28:56.896+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:56.897707+0000) 2019-10-13T07:28:56.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:28:56.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:28:57.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:28:57.801+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:57.801502+0000) 2019-10-13T07:28:58.352 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:28:58.349+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.349814+0000) 2019-10-13T07:28:58.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:28:58.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.686974+0000) 2019-10-13T07:28:58.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:28:58.686+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.687038+0000) 2019-10-13T07:29:01.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:01.901+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.902153+0000) 2019-10-13T07:29:01.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:01.901+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.902353+0000) 2019-10-13T07:29:01.905 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:01.902+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.902690+0000) 2019-10-13T07:29:01.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:01.900+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.901125+0000) 2019-10-13T07:29:01.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:01.902+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.903066+0000) 2019-10-13T07:29:01.907 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:01.903+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.903391+0000) 2019-10-13T07:29:01.907 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:01.903+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.903662+0000) 2019-10-13T07:29:01.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:01.902+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.903628+0000) 2019-10-13T07:29:01.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:01.907+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122488 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:01.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:01.939+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.940218+0000) 2019-10-13T07:29:02.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:02.013+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:02.013974+0000) 2019-10-13T07:29:02.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:02.803+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:02.803671+0000) 2019-10-13T07:29:02.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:02.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:02.804927+0000) 2019-10-13T07:29:06.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:06.907+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.908050+0000) 2019-10-13T07:29:06.913 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:06.909+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.909892+0000) 2019-10-13T07:29:06.914 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:06.909+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.910213+0000) 2019-10-13T07:29:06.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:06.910+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.910497+0000) 2019-10-13T07:29:06.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:06.909+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.910365+0000) 2019-10-13T07:29:06.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:06.910+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.910641+0000) 2019-10-13T07:29:06.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:06.910+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.910875+0000) 2019-10-13T07:29:06.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:06.910+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:06.911680+0000) 2019-10-13T07:29:06.917 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:06.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:07.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:07.801+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:07.801641+0000) 2019-10-13T07:29:08.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:08.349+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.350028+0000) 2019-10-13T07:29:08.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:08.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.687149+0000) 2019-10-13T07:29:08.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:08.687+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.687190+0000) 2019-10-13T07:29:11.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:11.914+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.914820+0000) 2019-10-13T07:29:11.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:11.916+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.916691+0000) 2019-10-13T07:29:11.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:11.916+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.917004+0000) 2019-10-13T07:29:11.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:11.916+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.917104+0000) 2019-10-13T07:29:11.922 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:11.916+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.917280+0000) 2019-10-13T07:29:11.922 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:11.917+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.917561+0000) 2019-10-13T07:29:11.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:11.917+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.917841+0000) 2019-10-13T07:29:11.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:11.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:11.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:11.918+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.919302+0000) 2019-10-13T07:29:11.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:11.940+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.940364+0000) 2019-10-13T07:29:12.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:12.015+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:12.016088+0000) 2019-10-13T07:29:12.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:12.802+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:12.803818+0000) 2019-10-13T07:29:12.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:12.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:12.805075+0000) 2019-10-13T07:29:16.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:16.921+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.922684+0000) 2019-10-13T07:29:16.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:16.921+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.922839+0000) 2019-10-13T07:29:16.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:16.921+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.921783+0000) 2019-10-13T07:29:16.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:16.921+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.923359+0000) 2019-10-13T07:29:16.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:16.922+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.923831+0000) 2019-10-13T07:29:16.927 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:16.922+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.923964+0000) 2019-10-13T07:29:16.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:16.922+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.924125+0000) 2019-10-13T07:29:16.928 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:16.923+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:16.923799+0000) 2019-10-13T07:29:16.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:16.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:17.375 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:29:17.375 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:17.380 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:29:17.380 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:17.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:17.800+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:17.801821+0000) 2019-10-13T07:29:18.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:18.349+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.350221+0000) 2019-10-13T07:29:18.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:18.686+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.687363+0000) 2019-10-13T07:29:18.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:18.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.687341+0000) 2019-10-13T07:29:21.932 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:21.928+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.929745+0000) 2019-10-13T07:29:21.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:21.927+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.928677+0000) 2019-10-13T07:29:21.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:21.929+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.930482+0000) 2019-10-13T07:29:21.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:21.929+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.930604+0000) 2019-10-13T07:29:21.934 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:21.929+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.930950+0000) 2019-10-13T07:29:21.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:21.929+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.931076+0000) 2019-10-13T07:29:21.934 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:21.929+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.931179+0000) 2019-10-13T07:29:21.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:21.930+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.931335+0000) 2019-10-13T07:29:21.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:21.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122610 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:21.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:21.939+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.940471+0000) 2019-10-13T07:29:22.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:22.018+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:22.018229+0000) 2019-10-13T07:29:22.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:22.803+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:22.803907+0000) 2019-10-13T07:29:22.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:22.805+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:22.805229+0000) 2019-10-13T07:29:26.938 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:26.934+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.936132+0000) 2019-10-13T07:29:26.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:26.934+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.934850+0000) 2019-10-13T07:29:26.940 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:26.935+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.936725+0000) 2019-10-13T07:29:26.940 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:26.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.936932+0000) 2019-10-13T07:29:26.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:26.935+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.937172+0000) 2019-10-13T07:29:26.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:26.935+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.937347+0000) 2019-10-13T07:29:26.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:26.935+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.936578+0000) 2019-10-13T07:29:26.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:26.936+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:26.937551+0000) 2019-10-13T07:29:26.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:26.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:27.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:27.801+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:27.801998+0000) 2019-10-13T07:29:28.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:28.349+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.350464+0000) 2019-10-13T07:29:28.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:28.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.687562+0000) 2019-10-13T07:29:28.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:28.686+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.687563+0000) 2019-10-13T07:29:31.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:31.940+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.940600+0000) 2019-10-13T07:29:31.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:31.941+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.942953+0000) 2019-10-13T07:29:31.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:31.941+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.943178+0000) 2019-10-13T07:29:31.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:31.941+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.943324+0000) 2019-10-13T07:29:31.947 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:31.942+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.943554+0000) 2019-10-13T07:29:31.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:31.942+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.943801+0000) 2019-10-13T07:29:31.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:31.940+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.941743+0000) 2019-10-13T07:29:31.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:31.942+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.944051+0000) 2019-10-13T07:29:31.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:31.942+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.944608+0000) 2019-10-13T07:29:31.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:31.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:32.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:32.019+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:32.020264+0000) 2019-10-13T07:29:32.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:32.803+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:32.804098+0000) 2019-10-13T07:29:32.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:32.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:32.805405+0000) 2019-10-13T07:29:36.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:36.947+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.948889+0000) 2019-10-13T07:29:36.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:36.947+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.949083+0000) 2019-10-13T07:29:36.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:36.945+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.947595+0000) 2019-10-13T07:29:36.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:36.948+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.949413+0000) 2019-10-13T07:29:36.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:36.948+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.949620+0000) 2019-10-13T07:29:36.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:36.948+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.949982+0000) 2019-10-13T07:29:36.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:36.948+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.950363+0000) 2019-10-13T07:29:36.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:36.948+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:36.950117+0000) 2019-10-13T07:29:36.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:36.952+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:37.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:37.801+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:37.802165+0000) 2019-10-13T07:29:38.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:38.350+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.350650+0000) 2019-10-13T07:29:38.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:38.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.687748+0000) 2019-10-13T07:29:38.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:38.687+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.687776+0000) 2019-10-13T07:29:41.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:41.940+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.940748+0000) 2019-10-13T07:29:41.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:41.955+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.956033+0000) 2019-10-13T07:29:41.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:41.955+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.956199+0000) 2019-10-13T07:29:41.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:41.953+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.955150+0000) 2019-10-13T07:29:41.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:41.956+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.956662+0000) 2019-10-13T07:29:41.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:41.956+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.956997+0000) 2019-10-13T07:29:41.960 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:41.956+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.957378+0000) 2019-10-13T07:29:41.960 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:41.957+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.957580+0000) 2019-10-13T07:29:41.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:41.954+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.956710+0000) 2019-10-13T07:29:41.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:41.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122732 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:42.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:42.021+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:42.022564+0000) 2019-10-13T07:29:42.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:42.803+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:42.804293+0000) 2019-10-13T07:29:42.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:42.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:42.805591+0000) 2019-10-13T07:29:46.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:46.960+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.961201+0000) 2019-10-13T07:29:46.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:46.962+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.963186+0000) 2019-10-13T07:29:46.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:46.962+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.963353+0000) 2019-10-13T07:29:46.967 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:46.963+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.963616+0000) 2019-10-13T07:29:46.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:46.963+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.963800+0000) 2019-10-13T07:29:46.967 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:46.963+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.964025+0000) 2019-10-13T07:29:46.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:46.964+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.964527+0000) 2019-10-13T07:29:46.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:46.964+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:46.964895+0000) 2019-10-13T07:29:46.970 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:46.967+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:47.530 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:29:47.530 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:47.534 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:29:47.534 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:47.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:47.802+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:47.802316+0000) 2019-10-13T07:29:48.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:48.350+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.350849+0000) 2019-10-13T07:29:48.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:48.687+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.687990+0000) 2019-10-13T07:29:48.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:48.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.687961+0000) 2019-10-13T07:29:51.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:51.940+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.940927+0000) 2019-10-13T07:29:51.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:51.969+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.969756+0000) 2019-10-13T07:29:51.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:51.967+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.968672+0000) 2019-10-13T07:29:51.973 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:51.969+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.970288+0000) 2019-10-13T07:29:51.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:51.970+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.970456+0000) 2019-10-13T07:29:51.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:51.970+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.970566+0000) 2019-10-13T07:29:51.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:51.970+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.970816+0000) 2019-10-13T07:29:51.974 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:51.970+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.971074+0000) 2019-10-13T07:29:51.975 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:51.969+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.970307+0000) 2019-10-13T07:29:51.977 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:51.974+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122791 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:52.029 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:52.024+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:52.024767+0000) 2019-10-13T07:29:52.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:52.804+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:52.804467+0000) 2019-10-13T07:29:52.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:52.804+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:52.805726+0000) 2019-10-13T07:29:56.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:56.976+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.976536+0000) 2019-10-13T07:29:56.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:56.976+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.976662+0000) 2019-10-13T07:29:56.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:56.976+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.977116+0000) 2019-10-13T07:29:56.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:29:56.974+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.975661+0000) 2019-10-13T07:29:56.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:29:56.977+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.977635+0000) 2019-10-13T07:29:56.981 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:56.977+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.978112+0000) 2019-10-13T07:29:56.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:29:56.977+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.978273+0000) 2019-10-13T07:29:56.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:29:56.977+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:56.978003+0000) 2019-10-13T07:29:56.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:29:56.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:29:57.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:29:57.802+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:57.802485+0000) 2019-10-13T07:29:58.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:29:58.350+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.351012+0000) 2019-10-13T07:29:58.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:29:58.687+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.688178+0000) 2019-10-13T07:29:58.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:29:58.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.688156+0000) 2019-10-13T07:30:01.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:01.940+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.941126+0000) 2019-10-13T07:30:01.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:01.981+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.982032+0000) 2019-10-13T07:30:01.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:01.982+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.983910+0000) 2019-10-13T07:30:01.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:01.982+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.984390+0000) 2019-10-13T07:30:01.987 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:01.983+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.984702+0000) 2019-10-13T07:30:01.988 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:01.983+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.984882+0000) 2019-10-13T07:30:01.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:01.983+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.985114+0000) 2019-10-13T07:30:01.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:01.983+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.985360+0000) 2019-10-13T07:30:01.989 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:01.984+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.985136+0000) 2019-10-13T07:30:01.992 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:01.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122850 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:02.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:02.026+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:02.027018+0000) 2019-10-13T07:30:02.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:02.803+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:02.804616+0000) 2019-10-13T07:30:02.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:02.805+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:02.805859+0000) 2019-10-13T07:30:06.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:06.988+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.989691+0000) 2019-10-13T07:30:06.995 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:06.990+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.991541+0000) 2019-10-13T07:30:06.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:06.990+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.991675+0000) 2019-10-13T07:30:06.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:06.990+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.991842+0000) 2019-10-13T07:30:06.995 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:06.990+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.992126+0000) 2019-10-13T07:30:06.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:06.990+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.992262+0000) 2019-10-13T07:30:06.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:06.991+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.992545+0000) 2019-10-13T07:30:06.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:06.992+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:06.992907+0000) 2019-10-13T07:30:06.999 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:06.995+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122877 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:07.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:07.801+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:07.802624+0000) 2019-10-13T07:30:08.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:08.350+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.351186+0000) 2019-10-13T07:30:08.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:08.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.688347+0000) 2019-10-13T07:30:08.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:08.686+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.688351+0000) 2019-10-13T07:30:11.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:11.940+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.941316+0000) 2019-10-13T07:30:12.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:11.997+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.999023+0000) 2019-10-13T07:30:12.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:11.997+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.999194+0000) 2019-10-13T07:30:12.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:11.997+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.999345+0000) 2019-10-13T07:30:12.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:11.998+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.999487+0000) 2019-10-13T07:30:12.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:11.998+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.999815+0000) 2019-10-13T07:30:12.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:11.998+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.999625+0000) 2019-10-13T07:30:12.004 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:11.997+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.998353+0000) 2019-10-13T07:30:12.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:11.999+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.999833+0000) 2019-10-13T07:30:12.006 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:12.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:12.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:12.028+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:12.029281+0000) 2019-10-13T07:30:12.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:12.804+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:12.804784+0000) 2019-10-13T07:30:12.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:12.805+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:12.806010+0000) 2019-10-13T07:30:17.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:17.003+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.004202+0000) 2019-10-13T07:30:17.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:17.004+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.006059+0000) 2019-10-13T07:30:17.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:17.004+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.006202+0000) 2019-10-13T07:30:17.010 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:17.004+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.006389+0000) 2019-10-13T07:30:17.010 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:17.005+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.006530+0000) 2019-10-13T07:30:17.010 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:17.005+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.006774+0000) 2019-10-13T07:30:17.010 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:17.005+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.006701+0000) 2019-10-13T07:30:17.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:17.007+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.007805+0000) 2019-10-13T07:30:17.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:17.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122936 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:17.626 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:30:17.627 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:17.631 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:30:17.632 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:17.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:17.802+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.802776+0000) 2019-10-13T07:30:18.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:18.350+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.351377+0000) 2019-10-13T07:30:18.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:18.687+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.688564+0000) 2019-10-13T07:30:18.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:18.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.688564+0000) 2019-10-13T07:30:21.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:21.940+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:21.941513+0000) 2019-10-13T07:30:22.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:22.010+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.011460+0000) 2019-10-13T07:30:22.016 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:22.011+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.013202+0000) 2019-10-13T07:30:22.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:22.011+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.013364+0000) 2019-10-13T07:30:22.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:22.012+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.013746+0000) 2019-10-13T07:30:22.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:22.012+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.013836+0000) 2019-10-13T07:30:22.017 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:22.012+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.014204+0000) 2019-10-13T07:30:22.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:22.012+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.014381+0000) 2019-10-13T07:30:22.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:22.013+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.014486+0000) 2019-10-13T07:30:22.021 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:22.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:22.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:22.031+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.031534+0000) 2019-10-13T07:30:22.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:22.804+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.804947+0000) 2019-10-13T07:30:22.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:22.806+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.806118+0000) 2019-10-13T07:30:27.023 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:27.019+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.020201+0000) 2019-10-13T07:30:27.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:27.020+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.020583+0000) 2019-10-13T07:30:27.023 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:27.020+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.020685+0000) 2019-10-13T07:30:27.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:27.017+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.018742+0000) 2019-10-13T07:30:27.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:27.020+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.020959+0000) 2019-10-13T07:30:27.025 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:27.020+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.021035+0000) 2019-10-13T07:30:27.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:27.020+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.021245+0000) 2019-10-13T07:30:27.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:27.019+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.021104+0000) 2019-10-13T07:30:27.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:27.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 122993 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:27.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:27.802+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.802942+0000) 2019-10-13T07:30:28.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:28.351+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.351563+0000) 2019-10-13T07:30:28.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:28.688+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.688784+0000) 2019-10-13T07:30:28.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:28.688+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.688807+0000) 2019-10-13T07:30:31.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:31.941+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:31.941675+0000) 2019-10-13T07:30:32.030 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:32.026+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.027287+0000) 2019-10-13T07:30:32.030 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:32.027+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.027406+0000) 2019-10-13T07:30:32.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:32.024+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.025980+0000) 2019-10-13T07:30:32.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:32.027+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.027817+0000) 2019-10-13T07:30:32.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:32.027+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.028218+0000) 2019-10-13T07:30:32.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:32.028+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.028764+0000) 2019-10-13T07:30:32.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:32.028+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.028922+0000) 2019-10-13T07:30:32.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:32.027+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.029039+0000) 2019-10-13T07:30:32.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:32.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123024 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:32.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:32.032+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.033749+0000) 2019-10-13T07:30:32.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:32.804+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.805112+0000) 2019-10-13T07:30:32.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:32.805+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.806222+0000) 2019-10-13T07:30:37.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:37.031+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.032858+0000) 2019-10-13T07:30:37.038 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:37.034+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.034745+0000) 2019-10-13T07:30:37.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:37.034+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.035216+0000) 2019-10-13T07:30:37.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:37.034+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.035349+0000) 2019-10-13T07:30:37.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:37.035+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.035652+0000) 2019-10-13T07:30:37.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:37.035+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.035816+0000) 2019-10-13T07:30:37.040 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:37.035+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.035964+0000) 2019-10-13T07:30:37.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:37.034+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.036181+0000) 2019-10-13T07:30:37.042 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:37.039+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:37.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:37.802+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.803129+0000) 2019-10-13T07:30:38.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:38.351+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.351738+0000) 2019-10-13T07:30:38.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:38.688+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.688947+0000) 2019-10-13T07:30:38.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:38.688+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.688965+0000) 2019-10-13T07:30:41.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:41.941+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:41.941867+0000) 2019-10-13T07:30:42.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:42.035+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.036083+0000) 2019-10-13T07:30:42.044 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:42.041+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.041682+0000) 2019-10-13T07:30:42.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:42.039+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.040229+0000) 2019-10-13T07:30:42.045 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:42.042+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.042418+0000) 2019-10-13T07:30:42.045 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:42.042+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.042627+0000) 2019-10-13T07:30:42.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:42.042+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.042798+0000) 2019-10-13T07:30:42.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:42.042+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.042946+0000) 2019-10-13T07:30:42.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:42.042+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.043319+0000) 2019-10-13T07:30:42.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:42.041+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.042643+0000) 2019-10-13T07:30:42.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:42.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:42.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:42.805+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.805283+0000) 2019-10-13T07:30:42.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:42.805+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.806364+0000) 2019-10-13T07:30:47.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:47.047+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.049101+0000) 2019-10-13T07:30:47.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:47.047+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.049331+0000) 2019-10-13T07:30:47.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:47.047+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.047808+0000) 2019-10-13T07:30:47.054 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:47.048+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.049643+0000) 2019-10-13T07:30:47.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:47.048+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.049802+0000) 2019-10-13T07:30:47.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:47.048+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.049859+0000) 2019-10-13T07:30:47.054 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:47.048+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.050236+0000) 2019-10-13T07:30:47.055 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:47.048+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.049099+0000) 2019-10-13T07:30:47.056 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:47.052+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:47.716 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:30:47.716 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:47.720 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:30:47.721 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:47.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:47.801+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.803327+0000) 2019-10-13T07:30:48.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:48.350+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.351929+0000) 2019-10-13T07:30:48.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:48.687+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.689162+0000) 2019-10-13T07:30:48.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:48.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.689165+0000) 2019-10-13T07:30:51.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:51.941+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:51.942054+0000) 2019-10-13T07:30:52.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:52.037+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.038455+0000) 2019-10-13T07:30:52.058 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:52.054+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.055472+0000) 2019-10-13T07:30:52.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:52.054+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.055861+0000) 2019-10-13T07:30:52.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:52.054+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.056254+0000) 2019-10-13T07:30:52.059 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:52.055+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.056637+0000) 2019-10-13T07:30:52.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:52.054+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.055501+0000) 2019-10-13T07:30:52.060 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:52.055+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.057360+0000) 2019-10-13T07:30:52.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:52.056+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.057593+0000) 2019-10-13T07:30:52.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:52.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:52.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:52.058+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.058840+0000) 2019-10-13T07:30:52.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:52.804+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.805421+0000) 2019-10-13T07:30:52.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:52.805+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.806545+0000) 2019-10-13T07:30:57.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:30:57.061+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.062015+0000) 2019-10-13T07:30:57.067 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:57.062+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.063853+0000) 2019-10-13T07:30:57.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:30:57.062+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.064036+0000) 2019-10-13T07:30:57.067 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:57.062+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.064346+0000) 2019-10-13T07:30:57.067 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:57.063+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.064604+0000) 2019-10-13T07:30:57.068 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:57.063+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.064771+0000) 2019-10-13T07:30:57.068 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:30:57.063+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.064824+0000) 2019-10-13T07:30:57.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:30:57.063+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.064343+0000) 2019-10-13T07:30:57.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:30:57.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:30:57.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:30:57.802+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.803499+0000) 2019-10-13T07:30:58.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:30:58.351+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.352131+0000) 2019-10-13T07:30:58.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:30:58.688+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.689355+0000) 2019-10-13T07:30:58.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:30:58.688+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.689365+0000) 2019-10-13T07:31:01.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:01.941+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:01.942258+0000) 2019-10-13T07:31:02.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:02.040+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.040865+0000) 2019-10-13T07:31:02.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:02.069+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.070471+0000) 2019-10-13T07:31:02.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:02.069+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.070759+0000) 2019-10-13T07:31:02.074 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:02.069+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.071032+0000) 2019-10-13T07:31:02.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:02.069+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.071227+0000) 2019-10-13T07:31:02.074 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:02.070+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.071450+0000) 2019-10-13T07:31:02.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:02.069+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.070049+0000) 2019-10-13T07:31:02.075 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:02.070+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.072039+0000) 2019-10-13T07:31:02.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:02.070+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.071319+0000) 2019-10-13T07:31:02.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:02.074+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:02.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:02.804+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.805572+0000) 2019-10-13T07:31:02.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:02.806+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.806719+0000) 2019-10-13T07:31:07.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:07.076+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.078319+0000) 2019-10-13T07:31:07.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:07.077+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.078393+0000) 2019-10-13T07:31:07.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:07.077+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.078622+0000) 2019-10-13T07:31:07.082 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:07.077+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.078798+0000) 2019-10-13T07:31:07.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:07.077+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.078983+0000) 2019-10-13T07:31:07.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:07.076+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.077692+0000) 2019-10-13T07:31:07.083 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:07.078+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.079802+0000) 2019-10-13T07:31:07.084 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:07.079+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.080108+0000) 2019-10-13T07:31:07.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:07.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:07.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:07.803+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.803699+0000) 2019-10-13T07:31:08.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:08.351+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.352297+0000) 2019-10-13T07:31:08.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:08.689+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.689579+0000) 2019-10-13T07:31:08.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:08.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.689584+0000) 2019-10-13T07:31:11.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:11.941+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:11.942393+0000) 2019-10-13T07:31:12.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:12.042+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.043094+0000) 2019-10-13T07:31:12.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:12.083+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.083845+0000) 2019-10-13T07:31:12.088 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:12.084+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.085798+0000) 2019-10-13T07:31:12.089 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:12.084+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.086136+0000) 2019-10-13T07:31:12.089 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:12.084+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.086183+0000) 2019-10-13T07:31:12.089 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:12.084+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.086370+0000) 2019-10-13T07:31:12.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:12.085+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.085887+0000) 2019-10-13T07:31:12.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:12.085+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.086894+0000) 2019-10-13T07:31:12.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:12.085+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.086531+0000) 2019-10-13T07:31:12.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:12.089+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123256 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:12.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:12.805+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.805798+0000) 2019-10-13T07:31:12.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:12.806+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.806885+0000) 2019-10-13T07:31:17.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:17.092+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.092744+0000) 2019-10-13T07:31:17.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:17.092+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.093078+0000) 2019-10-13T07:31:17.096 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:17.092+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.093176+0000) 2019-10-13T07:31:17.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:17.090+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.091259+0000) 2019-10-13T07:31:17.097 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:17.093+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.093471+0000) 2019-10-13T07:31:17.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:17.093+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.093649+0000) 2019-10-13T07:31:17.098 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:17.093+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.093922+0000) 2019-10-13T07:31:17.099 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:17.093+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.094603+0000) 2019-10-13T07:31:17.100 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:17.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:17.804 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:31:17.804 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:17.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:17.803+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.803870+0000) 2019-10-13T07:31:17.808 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:31:17.809 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:18.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:18.352+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.352461+0000) 2019-10-13T07:31:18.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:18.689+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.689786+0000) 2019-10-13T07:31:18.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:18.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.689787+0000) 2019-10-13T07:31:21.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:21.942+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:21.942584+0000) 2019-10-13T07:31:22.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:22.044+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.045259+0000) 2019-10-13T07:31:22.103 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:22.099+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.100042+0000) 2019-10-13T07:31:22.103 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:22.100+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.100471+0000) 2019-10-13T07:31:22.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:22.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.100610+0000) 2019-10-13T07:31:22.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:22.096+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.098695+0000) 2019-10-13T07:31:22.104 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:22.100+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.100862+0000) 2019-10-13T07:31:22.105 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:22.100+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.100695+0000) 2019-10-13T07:31:22.105 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:22.101+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.101549+0000) 2019-10-13T07:31:22.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:22.098+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.100502+0000) 2019-10-13T07:31:22.108 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:22.105+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:22.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:22.805+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.805955+0000) 2019-10-13T07:31:22.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:22.806+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.807057+0000) 2019-10-13T07:31:27.110 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:27.106+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.107313+0000) 2019-10-13T07:31:27.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:27.104+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.105941+0000) 2019-10-13T07:31:27.111 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:27.107+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.107863+0000) 2019-10-13T07:31:27.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:27.107+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.107626+0000) 2019-10-13T07:31:27.112 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:27.107+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.108005+0000) 2019-10-13T07:31:27.112 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:27.107+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.108325+0000) 2019-10-13T07:31:27.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:27.107+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.108379+0000) 2019-10-13T07:31:27.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:27.106+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.108627+0000) 2019-10-13T07:31:27.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:27.111+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123341 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:27.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:27.803+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.804006+0000) 2019-10-13T07:31:28.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:28.352+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.352634+0000) 2019-10-13T07:31:28.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:28.689+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.689991+0000) 2019-10-13T07:31:28.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:28.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.689992+0000) 2019-10-13T07:31:31.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:31.941+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:31.942780+0000) 2019-10-13T07:31:32.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:32.046+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.047665+0000) 2019-10-13T07:31:32.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:32.112+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.113582+0000) 2019-10-13T07:31:32.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:32.112+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.114076+0000) 2019-10-13T07:31:32.118 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:32.112+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.114282+0000) 2019-10-13T07:31:32.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:32.113+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.114464+0000) 2019-10-13T07:31:32.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:32.113+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.114619+0000) 2019-10-13T07:31:32.119 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:32.111+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.113489+0000) 2019-10-13T07:31:32.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:32.114+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.115817+0000) 2019-10-13T07:31:32.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:32.114+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.116120+0000) 2019-10-13T07:31:32.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:32.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123373 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:32.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:32.805+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.806121+0000) 2019-10-13T07:31:32.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:32.805+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.807235+0000) 2019-10-13T07:31:37.123 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:37.119+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.120686+0000) 2019-10-13T07:31:37.124 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:37.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.119749+0000) 2019-10-13T07:31:37.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:37.119+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.121187+0000) 2019-10-13T07:31:37.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:37.120+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.121450+0000) 2019-10-13T07:31:37.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:37.120+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.121613+0000) 2019-10-13T07:31:37.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:37.120+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.121830+0000) 2019-10-13T07:31:37.125 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:37.120+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.122015+0000) 2019-10-13T07:31:37.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:37.121+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.122235+0000) 2019-10-13T07:31:37.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:37.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123401 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:37.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:37.803+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.804154+0000) 2019-10-13T07:31:38.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:38.351+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.352776+0000) 2019-10-13T07:31:38.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:38.689+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.690171+0000) 2019-10-13T07:31:38.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:38.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.690174+0000) 2019-10-13T07:31:41.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:41.942+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:41.942972+0000) 2019-10-13T07:31:42.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:42.049+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.050105+0000) 2019-10-13T07:31:42.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:42.124+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.125151+0000) 2019-10-13T07:31:42.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:42.125+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.126972+0000) 2019-10-13T07:31:42.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:42.125+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.127125+0000) 2019-10-13T07:31:42.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:42.126+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.127435+0000) 2019-10-13T07:31:42.131 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:42.126+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.127602+0000) 2019-10-13T07:31:42.131 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:42.126+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.127803+0000) 2019-10-13T07:31:42.131 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:42.126+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.128026+0000) 2019-10-13T07:31:42.132 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:42.127+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.128438+0000) 2019-10-13T07:31:42.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:42.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123433 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:42.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:42.805+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.806282+0000) 2019-10-13T07:31:42.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:42.806+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.807393+0000) 2019-10-13T07:31:47.136 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:47.132+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.133833+0000) 2019-10-13T07:31:47.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:47.131+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.132580+0000) 2019-10-13T07:31:47.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:47.132+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.134289+0000) 2019-10-13T07:31:47.137 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:47.133+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.134490+0000) 2019-10-13T07:31:47.138 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:47.133+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.134726+0000) 2019-10-13T07:31:47.138 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:47.133+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.134994+0000) 2019-10-13T07:31:47.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:47.133+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.134319+0000) 2019-10-13T07:31:47.139 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:47.133+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.135316+0000) 2019-10-13T07:31:47.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:47.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:47.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:47.802+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.804337+0000) 2019-10-13T07:31:47.910 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:31:47.910 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:47.914 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:31:47.914 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:48.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:48.352+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.352948+0000) 2019-10-13T07:31:48.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:48.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.690359+0000) 2019-10-13T07:31:48.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:48.689+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.690360+0000) 2019-10-13T07:31:51.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:51.942+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:51.943164+0000) 2019-10-13T07:31:52.057 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:52.052+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.052608+0000) 2019-10-13T07:31:52.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:52.138+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.139061+0000) 2019-10-13T07:31:52.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:52.139+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.140752+0000) 2019-10-13T07:31:52.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:52.139+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.140969+0000) 2019-10-13T07:31:52.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:52.139+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.141044+0000) 2019-10-13T07:31:52.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:52.139+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.141280+0000) 2019-10-13T07:31:52.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:52.139+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.141382+0000) 2019-10-13T07:31:52.145 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:52.140+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.141756+0000) 2019-10-13T07:31:52.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:52.140+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.140771+0000) 2019-10-13T07:31:52.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:52.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:52.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:52.805+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.806444+0000) 2019-10-13T07:31:52.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:52.807+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.807583+0000) 2019-10-13T07:31:57.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:57.145+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.147383+0000) 2019-10-13T07:31:57.150 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:31:57.146+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.147471+0000) 2019-10-13T07:31:57.151 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:31:57.146+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.147625+0000) 2019-10-13T07:31:57.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:31:57.145+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.146335+0000) 2019-10-13T07:31:57.152 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:57.146+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.148064+0000) 2019-10-13T07:31:57.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:57.146+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.148194+0000) 2019-10-13T07:31:57.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:57.147+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.148680+0000) 2019-10-13T07:31:57.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:31:57.148+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.149689+0000) 2019-10-13T07:31:57.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:31:57.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123519 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:31:57.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:31:57.804+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.804512+0000) 2019-10-13T07:31:58.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:31:58.352+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.353167+0000) 2019-10-13T07:31:58.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:31:58.690+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.690553+0000) 2019-10-13T07:31:58.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:31:58.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.690484+0000) 2019-10-13T07:32:01.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:01.942+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:01.943323+0000) 2019-10-13T07:32:02.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:02.054+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.055142+0000) 2019-10-13T07:32:02.157 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:02.152+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.152841+0000) 2019-10-13T07:32:02.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:02.154+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.154908+0000) 2019-10-13T07:32:02.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:02.154+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.154975+0000) 2019-10-13T07:32:02.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:02.154+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.155052+0000) 2019-10-13T07:32:02.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:02.155+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.155405+0000) 2019-10-13T07:32:02.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:02.154+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.155362+0000) 2019-10-13T07:32:02.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:02.155+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.155636+0000) 2019-10-13T07:32:02.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:02.154+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.155098+0000) 2019-10-13T07:32:02.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:02.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:02.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:02.806+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.806630+0000) 2019-10-13T07:32:02.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:02.807+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.807756+0000) 2019-10-13T07:32:07.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:07.159+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.160095+0000) 2019-10-13T07:32:07.164 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:07.161+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.161952+0000) 2019-10-13T07:32:07.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:07.162+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.162518+0000) 2019-10-13T07:32:07.166 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:07.162+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.162392+0000) 2019-10-13T07:32:07.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:07.162+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.162605+0000) 2019-10-13T07:32:07.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:07.162+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.162908+0000) 2019-10-13T07:32:07.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:07.162+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.163038+0000) 2019-10-13T07:32:07.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:07.162+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.162820+0000) 2019-10-13T07:32:07.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:07.166+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:07.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:07.804+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.804698+0000) 2019-10-13T07:32:08.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:08.353+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.353379+0000) 2019-10-13T07:32:08.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:08.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.690668+0000) 2019-10-13T07:32:08.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:08.690+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.690714+0000) 2019-10-13T07:32:11.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:11.943+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:11.943479+0000) 2019-10-13T07:32:12.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:12.057+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.057434+0000) 2019-10-13T07:32:12.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:12.168+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.168788+0000) 2019-10-13T07:32:12.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:12.168+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.169183+0000) 2019-10-13T07:32:12.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:12.168+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.169321+0000) 2019-10-13T07:32:12.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:12.169+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.169650+0000) 2019-10-13T07:32:12.173 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:12.169+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.169514+0000) 2019-10-13T07:32:12.173 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:12.169+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.169799+0000) 2019-10-13T07:32:12.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:12.166+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.167452+0000) 2019-10-13T07:32:12.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:12.168+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.169033+0000) 2019-10-13T07:32:12.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:12.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:12.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:12.806+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.806778+0000) 2019-10-13T07:32:12.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:12.807+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.807872+0000) 2019-10-13T07:32:17.178 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:17.175+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.175565+0000) 2019-10-13T07:32:17.179 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:17.172+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.174379+0000) 2019-10-13T07:32:17.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:17.175+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.176191+0000) 2019-10-13T07:32:17.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:17.175+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.176383+0000) 2019-10-13T07:32:17.180 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:17.176+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.176659+0000) 2019-10-13T07:32:17.180 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:17.176+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.176836+0000) 2019-10-13T07:32:17.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:17.174+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.176277+0000) 2019-10-13T07:32:17.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:17.176+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.177103+0000) 2019-10-13T07:32:17.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:17.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:17.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:17.803+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.804875+0000) 2019-10-13T07:32:18.060 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:32:18.061 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:18.065 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:32:18.065 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:18.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:18.352+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.353568+0000) 2019-10-13T07:32:18.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:18.689+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.690873+0000) 2019-10-13T07:32:18.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:18.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.690871+0000) 2019-10-13T07:32:21.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:21.942+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:21.943608+0000) 2019-10-13T07:32:22.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:22.058+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.059712+0000) 2019-10-13T07:32:22.185 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:22.181+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.182569+0000) 2019-10-13T07:32:22.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:22.179+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.180776+0000) 2019-10-13T07:32:22.186 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:22.181+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.182655+0000) 2019-10-13T07:32:22.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:22.181+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.182796+0000) 2019-10-13T07:32:22.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:22.181+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.183164+0000) 2019-10-13T07:32:22.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:22.182+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.183419+0000) 2019-10-13T07:32:22.187 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:22.182+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.183574+0000) 2019-10-13T07:32:22.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:22.181+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.183650+0000) 2019-10-13T07:32:22.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:22.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:22.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:22.805+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.806917+0000) 2019-10-13T07:32:22.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:22.807+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.808033+0000) 2019-10-13T07:32:27.191 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:27.186+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.188079+0000) 2019-10-13T07:32:27.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:27.186+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.188215+0000) 2019-10-13T07:32:27.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:27.187+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.188402+0000) 2019-10-13T07:32:27.192 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:27.185+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.187078+0000) 2019-10-13T07:32:27.192 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:27.187+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.188981+0000) 2019-10-13T07:32:27.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:27.187+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.189150+0000) 2019-10-13T07:32:27.192 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:27.188+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.189553+0000) 2019-10-13T07:32:27.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:27.188+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.190044+0000) 2019-10-13T07:32:27.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:27.192+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:27.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:27.804+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.805048+0000) 2019-10-13T07:32:28.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:28.352+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.353715+0000) 2019-10-13T07:32:28.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:28.690+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.691086+0000) 2019-10-13T07:32:28.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:28.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.691080+0000) 2019-10-13T07:32:31.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:31.942+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:31.943741+0000) 2019-10-13T07:32:32.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:32.061+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.062104+0000) 2019-10-13T07:32:32.198 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:32.193+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.194973+0000) 2019-10-13T07:32:32.198 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:32.193+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.194816+0000) 2019-10-13T07:32:32.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:32.193+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.195231+0000) 2019-10-13T07:32:32.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:32.194+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.195655+0000) 2019-10-13T07:32:32.199 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:32.193+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.193738+0000) 2019-10-13T07:32:32.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:32.194+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.195879+0000) 2019-10-13T07:32:32.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:32.194+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.196053+0000) 2019-10-13T07:32:32.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:32.194+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.195732+0000) 2019-10-13T07:32:32.201 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:32.197+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:32.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:32.806+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.807094+0000) 2019-10-13T07:32:32.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:32.807+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.808184+0000) 2019-10-13T07:32:37.203 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:37.199+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.200595+0000) 2019-10-13T07:32:37.204 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:37.199+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.200893+0000) 2019-10-13T07:32:37.204 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:37.199+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.201119+0000) 2019-10-13T07:32:37.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:37.199+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.201233+0000) 2019-10-13T07:32:37.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:37.199+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.199868+0000) 2019-10-13T07:32:37.205 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:37.200+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.200952+0000) 2019-10-13T07:32:37.205 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:37.200+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.201652+0000) 2019-10-13T07:32:37.206 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:37.200+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.201852+0000) 2019-10-13T07:32:37.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:37.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123755 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:37.808 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:37.804+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.805190+0000) 2019-10-13T07:32:38.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:38.353+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.353866+0000) 2019-10-13T07:32:38.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:38.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.691298+0000) 2019-10-13T07:32:38.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:38.690+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.691319+0000) 2019-10-13T07:32:41.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:41.942+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:41.943924+0000) 2019-10-13T07:32:42.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:42.063+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.064516+0000) 2019-10-13T07:32:42.210 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:42.205+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.205916+0000) 2019-10-13T07:32:42.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:42.206+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.207657+0000) 2019-10-13T07:32:42.211 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:42.206+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.207817+0000) 2019-10-13T07:32:42.211 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:42.206+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.208238+0000) 2019-10-13T07:32:42.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:42.207+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.208415+0000) 2019-10-13T07:32:42.212 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:42.207+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.208550+0000) 2019-10-13T07:32:42.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:42.207+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.208761+0000) 2019-10-13T07:32:42.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:42.208+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.209257+0000) 2019-10-13T07:32:42.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:42.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123787 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:42.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:42.806+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.807264+0000) 2019-10-13T07:32:42.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:42.807+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.808312+0000) 2019-10-13T07:32:47.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:47.212+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.212934+0000) 2019-10-13T07:32:47.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:47.214+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.214584+0000) 2019-10-13T07:32:47.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:47.214+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.214826+0000) 2019-10-13T07:32:47.218 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:47.214+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.214998+0000) 2019-10-13T07:32:47.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:47.214+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.215060+0000) 2019-10-13T07:32:47.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:47.215+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.215672+0000) 2019-10-13T07:32:47.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:47.215+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.215766+0000) 2019-10-13T07:32:47.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:47.215+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.216067+0000) 2019-10-13T07:32:47.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:47.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123813 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:47.808 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:47.804+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.805326+0000) 2019-10-13T07:32:48.157 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:32:48.157 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:48.162 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:32:48.162 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:48.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:48.353+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.354068+0000) 2019-10-13T07:32:48.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:48.691+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.691493+0000) 2019-10-13T07:32:48.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:48.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.691482+0000) 2019-10-13T07:32:51.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:51.943+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:51.944141+0000) 2019-10-13T07:32:52.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:52.066+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.066989+0000) 2019-10-13T07:32:52.223 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:52.218+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.219124+0000) 2019-10-13T07:32:52.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:52.220+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.221062+0000) 2019-10-13T07:32:52.224 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:52.220+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.221146+0000) 2019-10-13T07:32:52.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:52.220+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.221301+0000) 2019-10-13T07:32:52.225 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:52.221+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.221454+0000) 2019-10-13T07:32:52.225 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:52.221+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.221547+0000) 2019-10-13T07:32:52.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:52.221+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.221809+0000) 2019-10-13T07:32:52.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:52.220+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.221117+0000) 2019-10-13T07:32:52.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:52.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:52.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:52.807+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.807409+0000) 2019-10-13T07:32:52.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:52.808+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.808461+0000) 2019-10-13T07:32:57.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:57.225+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.226275+0000) 2019-10-13T07:32:57.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:32:57.226+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.226501+0000) 2019-10-13T07:32:57.230 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:32:57.225+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.225773+0000) 2019-10-13T07:32:57.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:57.226+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.227136+0000) 2019-10-13T07:32:57.230 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:32:57.227+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.227439+0000) 2019-10-13T07:32:57.230 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:57.227+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.227706+0000) 2019-10-13T07:32:57.231 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:57.227+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.227855+0000) 2019-10-13T07:32:57.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:32:57.227+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.228143+0000) 2019-10-13T07:32:57.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:32:57.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123874 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:32:57.808 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:32:57.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.805478+0000) 2019-10-13T07:32:58.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:32:58.354+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.354265+0000) 2019-10-13T07:32:58.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:32:58.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.691691+0000) 2019-10-13T07:32:58.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:32:58.691+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.691693+0000) 2019-10-13T07:33:01.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:01.943+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:01.944332+0000) 2019-10-13T07:33:02.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:02.069+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.069504+0000) 2019-10-13T07:33:02.237 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:02.233+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.234224+0000) 2019-10-13T07:33:02.237 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:02.233+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.234388+0000) 2019-10-13T07:33:02.238 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:02.232+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.233691+0000) 2019-10-13T07:33:02.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:02.234+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.234801+0000) 2019-10-13T07:33:02.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:02.234+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.235084+0000) 2019-10-13T07:33:02.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:02.234+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.235385+0000) 2019-10-13T07:33:02.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:02.235+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.235530+0000) 2019-10-13T07:33:02.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:02.235+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.236269+0000) 2019-10-13T07:33:02.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:02.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123907 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:02.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:02.807+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.807525+0000) 2019-10-13T07:33:02.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:02.808+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.808596+0000) 2019-10-13T07:33:07.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:07.240+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.241937+0000) 2019-10-13T07:33:07.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:07.240+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.242354+0000) 2019-10-13T07:33:07.246 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:07.240+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.241014+0000) 2019-10-13T07:33:07.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:07.241+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.242669+0000) 2019-10-13T07:33:07.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:07.241+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.242990+0000) 2019-10-13T07:33:07.246 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:07.241+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.243147+0000) 2019-10-13T07:33:07.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:07.241+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.243371+0000) 2019-10-13T07:33:07.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:07.241+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.242675+0000) 2019-10-13T07:33:07.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:07.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123933 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:07.808 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:07.804+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.805648+0000) 2019-10-13T07:33:08.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:08.353+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.354461+0000) 2019-10-13T07:33:08.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:08.690+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.691878+0000) 2019-10-13T07:33:08.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:08.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.691873+0000) 2019-10-13T07:33:11.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:11.943+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:11.944543+0000) 2019-10-13T07:33:12.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:12.070+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.071914+0000) 2019-10-13T07:33:12.252 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:12.248+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.249689+0000) 2019-10-13T07:33:12.253 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:12.248+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.249857+0000) 2019-10-13T07:33:12.253 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:12.248+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.250099+0000) 2019-10-13T07:33:12.254 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:12.247+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.248826+0000) 2019-10-13T07:33:12.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:12.249+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.250849+0000) 2019-10-13T07:33:12.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:12.249+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.250902+0000) 2019-10-13T07:33:12.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:12.249+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.251250+0000) 2019-10-13T07:33:12.255 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:12.249+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.250826+0000) 2019-10-13T07:33:12.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:12.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:12.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:12.806+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.807705+0000) 2019-10-13T07:33:12.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:12.807+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.808724+0000) 2019-10-13T07:33:17.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:17.255+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.257250+0000) 2019-10-13T07:33:17.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:17.256+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.257457+0000) 2019-10-13T07:33:17.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:17.256+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.257790+0000) 2019-10-13T07:33:17.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:17.256+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.258127+0000) 2019-10-13T07:33:17.261 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:17.256+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.258296+0000) 2019-10-13T07:33:17.261 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:17.257+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.258723+0000) 2019-10-13T07:33:17.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:17.255+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.257357+0000) 2019-10-13T07:33:17.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:17.257+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.259266+0000) 2019-10-13T07:33:17.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:17.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 123994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:17.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:17.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.805817+0000) 2019-10-13T07:33:18.286 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:33:18.286 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:18.291 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:33:18.291 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:18.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:18.353+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.354612+0000) 2019-10-13T07:33:18.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:18.690+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.692111+0000) 2019-10-13T07:33:18.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:18.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.692090+0000) 2019-10-13T07:33:21.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:21.943+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:21.944744+0000) 2019-10-13T07:33:22.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:22.072+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.074419+0000) 2019-10-13T07:33:22.267 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:22.262+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.263932+0000) 2019-10-13T07:33:22.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:22.262+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.264034+0000) 2019-10-13T07:33:22.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:22.262+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.264367+0000) 2019-10-13T07:33:22.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:22.263+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.264546+0000) 2019-10-13T07:33:22.268 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:22.261+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.263570+0000) 2019-10-13T07:33:22.268 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:22.263+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.264808+0000) 2019-10-13T07:33:22.268 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:22.263+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.264946+0000) 2019-10-13T07:33:22.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:22.264+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.265850+0000) 2019-10-13T07:33:22.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:22.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124027 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:22.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:22.806+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.807836+0000) 2019-10-13T07:33:22.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:22.808+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.808856+0000) 2019-10-13T07:33:27.273 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:27.269+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.271218+0000) 2019-10-13T07:33:27.274 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:27.270+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.271616+0000) 2019-10-13T07:33:27.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:27.270+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.271747+0000) 2019-10-13T07:33:27.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:27.270+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.271864+0000) 2019-10-13T07:33:27.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:27.270+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.271985+0000) 2019-10-13T07:33:27.275 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:27.270+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.272356+0000) 2019-10-13T07:33:27.275 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:27.269+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.270669+0000) 2019-10-13T07:33:27.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:27.271+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.272698+0000) 2019-10-13T07:33:27.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:27.275+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:27.808 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:27.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.805985+0000) 2019-10-13T07:33:28.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:28.353+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.354735+0000) 2019-10-13T07:33:28.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:28.691+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.692290+0000) 2019-10-13T07:33:28.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:28.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.692274+0000) 2019-10-13T07:33:31.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:31.944+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:31.944930+0000) 2019-10-13T07:33:32.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:32.076+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.076764+0000) 2019-10-13T07:33:32.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:32.278+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.278391+0000) 2019-10-13T07:33:32.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:32.278+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.278585+0000) 2019-10-13T07:33:32.281 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:32.278+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.278927+0000) 2019-10-13T07:33:32.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:32.277+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.277796+0000) 2019-10-13T07:33:32.283 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:32.279+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.279431+0000) 2019-10-13T07:33:32.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:32.279+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.279576+0000) 2019-10-13T07:33:32.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:32.279+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.279699+0000) 2019-10-13T07:33:32.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:32.278+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.279577+0000) 2019-10-13T07:33:32.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:32.283+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:32.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:32.807+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.807999+0000) 2019-10-13T07:33:32.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:32.808+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.809015+0000) 2019-10-13T07:33:37.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:37.284+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.285359+0000) 2019-10-13T07:33:37.288 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:37.285+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.285532+0000) 2019-10-13T07:33:37.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:37.283+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.284669+0000) 2019-10-13T07:33:37.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:37.285+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.285925+0000) 2019-10-13T07:33:37.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:37.286+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.286480+0000) 2019-10-13T07:33:37.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:37.286+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.286687+0000) 2019-10-13T07:33:37.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:37.286+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.286893+0000) 2019-10-13T07:33:37.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:37.286+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.287005+0000) 2019-10-13T07:33:37.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:37.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:37.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:37.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.806158+0000) 2019-10-13T07:33:38.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:38.354+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.354927+0000) 2019-10-13T07:33:38.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:38.692+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.692487+0000) 2019-10-13T07:33:38.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:38.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.692480+0000) 2019-10-13T07:33:41.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:41.944+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:41.945141+0000) 2019-10-13T07:33:42.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:42.078+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.079184+0000) 2019-10-13T07:33:42.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:42.290+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.290967+0000) 2019-10-13T07:33:42.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:42.292+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.292549+0000) 2019-10-13T07:33:42.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:42.292+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.292652+0000) 2019-10-13T07:33:42.296 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:42.292+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.293206+0000) 2019-10-13T07:33:42.297 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:42.292+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.293329+0000) 2019-10-13T07:33:42.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:42.293+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.293528+0000) 2019-10-13T07:33:42.297 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:42.293+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.293751+0000) 2019-10-13T07:33:42.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:42.293+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.293807+0000) 2019-10-13T07:33:42.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:42.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:42.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:42.807+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.808136+0000) 2019-10-13T07:33:42.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:42.808+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.809171+0000) 2019-10-13T07:33:47.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:47.295+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.296555+0000) 2019-10-13T07:33:47.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:47.298+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.298506+0000) 2019-10-13T07:33:47.302 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:47.298+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.298723+0000) 2019-10-13T07:33:47.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:47.298+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.298885+0000) 2019-10-13T07:33:47.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:47.298+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.299028+0000) 2019-10-13T07:33:47.303 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:47.299+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.299654+0000) 2019-10-13T07:33:47.303 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:47.299+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.299819+0000) 2019-10-13T07:33:47.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:47.299+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.299893+0000) 2019-10-13T07:33:47.305 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:47.302+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124174 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:47.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:47.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.806327+0000) 2019-10-13T07:33:48.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:48.355+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.355142+0000) 2019-10-13T07:33:48.401 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:33:48.401 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:48.412 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:33:48.412 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:48.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:48.692+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.692703+0000) 2019-10-13T07:33:48.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:48.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.692702+0000) 2019-10-13T07:33:51.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:51.944+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:51.945331+0000) 2019-10-13T07:33:52.086 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:52.081+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.081494+0000) 2019-10-13T07:33:52.308 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:52.303+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.304899+0000) 2019-10-13T07:33:52.308 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:52.303+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.305236+0000) 2019-10-13T07:33:52.309 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:52.304+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.305475+0000) 2019-10-13T07:33:52.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:52.304+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.305604+0000) 2019-10-13T07:33:52.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:52.303+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.304519+0000) 2019-10-13T07:33:52.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:52.305+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.306489+0000) 2019-10-13T07:33:52.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:52.305+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.306626+0000) 2019-10-13T07:33:52.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:52.305+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.306529+0000) 2019-10-13T07:33:52.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:52.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:52.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:52.807+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.808313+0000) 2019-10-13T07:33:52.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:52.809+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.809331+0000) 2019-10-13T07:33:57.314 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:33:57.309+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.310341+0000) 2019-10-13T07:33:57.315 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:57.310+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.312046+0000) 2019-10-13T07:33:57.315 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:33:57.310+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.312234+0000) 2019-10-13T07:33:57.316 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:57.311+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.312423+0000) 2019-10-13T07:33:57.316 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:33:57.311+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.312743+0000) 2019-10-13T07:33:57.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:57.311+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.312928+0000) 2019-10-13T07:33:57.317 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:57.311+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.313369+0000) 2019-10-13T07:33:57.317 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:33:57.312+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.313254+0000) 2019-10-13T07:33:57.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:33:57.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:33:57.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:33:57.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.806484+0000) 2019-10-13T07:33:58.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:33:58.354+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.355338+0000) 2019-10-13T07:33:58.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:33:58.692+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.692898+0000) 2019-10-13T07:33:58.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:33:58.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.692896+0000) 2019-10-13T07:34:01.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:01.944+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:01.945547+0000) 2019-10-13T07:34:02.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:02.083+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.083951+0000) 2019-10-13T07:34:02.321 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:02.316+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.318371+0000) 2019-10-13T07:34:02.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:02.317+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.318802+0000) 2019-10-13T07:34:02.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:02.317+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.318958+0000) 2019-10-13T07:34:02.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:02.317+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.319110+0000) 2019-10-13T07:34:02.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:02.317+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.317867+0000) 2019-10-13T07:34:02.323 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:02.318+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.319548+0000) 2019-10-13T07:34:02.323 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:02.318+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.319668+0000) 2019-10-13T07:34:02.324 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:02.320+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.320746+0000) 2019-10-13T07:34:02.325 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:02.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:02.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:02.807+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.808479+0000) 2019-10-13T07:34:02.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:02.809+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.809499+0000) 2019-10-13T07:34:07.328 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:07.323+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.325223+0000) 2019-10-13T07:34:07.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:07.323+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.325376+0000) 2019-10-13T07:34:07.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:07.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.325650+0000) 2019-10-13T07:34:07.329 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:07.324+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.325672+0000) 2019-10-13T07:34:07.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:07.324+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.325793+0000) 2019-10-13T07:34:07.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:07.322+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.324617+0000) 2019-10-13T07:34:07.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:07.324+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.326151+0000) 2019-10-13T07:34:07.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:07.325+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.326915+0000) 2019-10-13T07:34:07.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:07.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:07.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:07.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.806632+0000) 2019-10-13T07:34:08.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:08.354+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.355519+0000) 2019-10-13T07:34:08.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:08.691+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.693111+0000) 2019-10-13T07:34:08.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:08.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.693109+0000) 2019-10-13T07:34:11.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:11.945+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:11.945719+0000) 2019-10-13T07:34:12.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:12.085+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.086244+0000) 2019-10-13T07:34:12.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:12.329+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.330926+0000) 2019-10-13T07:34:12.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:12.329+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.331225+0000) 2019-10-13T07:34:12.334 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:12.329+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.331278+0000) 2019-10-13T07:34:12.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:12.330+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.331630+0000) 2019-10-13T07:34:12.335 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:12.328+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.330678+0000) 2019-10-13T07:34:12.335 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:12.330+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.331778+0000) 2019-10-13T07:34:12.335 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:12.330+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.332359+0000) 2019-10-13T07:34:12.336 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:12.331+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.332943+0000) 2019-10-13T07:34:12.339 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:12.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:12.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:12.808+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.809657+0000) 2019-10-13T07:34:12.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:12.808+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.808629+0000) 2019-10-13T07:34:17.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:17.336+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.338077+0000) 2019-10-13T07:34:17.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:17.336+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.338151+0000) 2019-10-13T07:34:17.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:17.336+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.338338+0000) 2019-10-13T07:34:17.342 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:17.337+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.338472+0000) 2019-10-13T07:34:17.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:17.337+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.338750+0000) 2019-10-13T07:34:17.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:17.335+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.337196+0000) 2019-10-13T07:34:17.343 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:17.337+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.339324+0000) 2019-10-13T07:34:17.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:17.338+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.340382+0000) 2019-10-13T07:34:17.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:17.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:17.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:17.806+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.806795+0000) 2019-10-13T07:34:18.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:18.355+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.355631+0000) 2019-10-13T07:34:18.599 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:34:18.599 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:18.603 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:34:18.603 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:18.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:18.692+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.693336+0000) 2019-10-13T07:34:18.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:18.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.693313+0000) 2019-10-13T07:34:21.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:21.945+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:21.945904+0000) 2019-10-13T07:34:22.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:22.088+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.088715+0000) 2019-10-13T07:34:22.348 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:22.343+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.343760+0000) 2019-10-13T07:34:22.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:22.344+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.345323+0000) 2019-10-13T07:34:22.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:22.345+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.345455+0000) 2019-10-13T07:34:22.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:22.345+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.345655+0000) 2019-10-13T07:34:22.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:22.345+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.345796+0000) 2019-10-13T07:34:22.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:22.345+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.346004+0000) 2019-10-13T07:34:22.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:22.345+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.346034+0000) 2019-10-13T07:34:22.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:22.344+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.345723+0000) 2019-10-13T07:34:22.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:22.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:22.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:22.808+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.808804+0000) 2019-10-13T07:34:22.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:22.809+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.809818+0000) 2019-10-13T07:34:27.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:27.349+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.349910+0000) 2019-10-13T07:34:27.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:27.351+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.351572+0000) 2019-10-13T07:34:27.355 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:27.351+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.351711+0000) 2019-10-13T07:34:27.355 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:27.351+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.351826+0000) 2019-10-13T07:34:27.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:27.351+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.352202+0000) 2019-10-13T07:34:27.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:27.352+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.352552+0000) 2019-10-13T07:34:27.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:27.352+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.352693+0000) 2019-10-13T07:34:27.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:27.352+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.353504+0000) 2019-10-13T07:34:27.359 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:27.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:27.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:27.806+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.806945+0000) 2019-10-13T07:34:28.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:28.355+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.355787+0000) 2019-10-13T07:34:28.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:28.693+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.693506+0000) 2019-10-13T07:34:28.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:28.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.693503+0000) 2019-10-13T07:34:31.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:31.945+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:31.946104+0000) 2019-10-13T07:34:32.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:32.090+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.091272+0000) 2019-10-13T07:34:32.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:32.356+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.356895+0000) 2019-10-13T07:34:32.362 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:32.358+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.358485+0000) 2019-10-13T07:34:32.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:32.358+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.358878+0000) 2019-10-13T07:34:32.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:32.358+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.359106+0000) 2019-10-13T07:34:32.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:32.358+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.359236+0000) 2019-10-13T07:34:32.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:32.358+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.358772+0000) 2019-10-13T07:34:32.363 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:32.359+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.359493+0000) 2019-10-13T07:34:32.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:32.358+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.359420+0000) 2019-10-13T07:34:32.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:32.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:32.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:32.808+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.808966+0000) 2019-10-13T07:34:32.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:32.809+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.809951+0000) 2019-10-13T07:34:37.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:37.363+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.363795+0000) 2019-10-13T07:34:37.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:37.363+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.365375+0000) 2019-10-13T07:34:37.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:37.364+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.365536+0000) 2019-10-13T07:34:37.369 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:37.364+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.365860+0000) 2019-10-13T07:34:37.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:37.364+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.366293+0000) 2019-10-13T07:34:37.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:37.364+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.366361+0000) 2019-10-13T07:34:37.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:37.365+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.366582+0000) 2019-10-13T07:34:37.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:37.365+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.366524+0000) 2019-10-13T07:34:37.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:37.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124475 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:37.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:37.805+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.807135+0000) 2019-10-13T07:34:38.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:38.354+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.355981+0000) 2019-10-13T07:34:38.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:38.692+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.693701+0000) 2019-10-13T07:34:38.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:38.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.693700+0000) 2019-10-13T07:34:41.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:41.945+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:41.946305+0000) 2019-10-13T07:34:42.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:42.093+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.093771+0000) 2019-10-13T07:34:42.374 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:42.369+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.370646+0000) 2019-10-13T07:34:42.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:42.370+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.372343+0000) 2019-10-13T07:34:42.376 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:42.371+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.372631+0000) 2019-10-13T07:34:42.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:42.371+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.372570+0000) 2019-10-13T07:34:42.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:42.371+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.372826+0000) 2019-10-13T07:34:42.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:42.371+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.373017+0000) 2019-10-13T07:34:42.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:42.372+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.373445+0000) 2019-10-13T07:34:42.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:42.372+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.373343+0000) 2019-10-13T07:34:42.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:42.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:42.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:42.807+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.809177+0000) 2019-10-13T07:34:42.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:42.809+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.810101+0000) 2019-10-13T07:34:47.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:47.378+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.379796+0000) 2019-10-13T07:34:47.383 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:47.378+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.379868+0000) 2019-10-13T07:34:47.384 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:47.378+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.380046+0000) 2019-10-13T07:34:47.384 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:47.379+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.380554+0000) 2019-10-13T07:34:47.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:47.379+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.380684+0000) 2019-10-13T07:34:47.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:47.379+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.380827+0000) 2019-10-13T07:34:47.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:47.378+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.378726+0000) 2019-10-13T07:34:47.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:47.381+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.381933+0000) 2019-10-13T07:34:47.395 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:47.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124534 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:47.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:47.806+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.807309+0000) 2019-10-13T07:34:48.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:48.355+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.356146+0000) 2019-10-13T07:34:48.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:48.693+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.693911+0000) 2019-10-13T07:34:48.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:48.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.693913+0000) 2019-10-13T07:34:48.700 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:34:48.700 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:48.704 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:34:48.705 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:51.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:51.945+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:51.946484+0000) 2019-10-13T07:34:52.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:52.095+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.096084+0000) 2019-10-13T07:34:52.398 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:52.384+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.385607+0000) 2019-10-13T07:34:52.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:52.387+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.388646+0000) 2019-10-13T07:34:52.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:52.385+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.386811+0000) 2019-10-13T07:34:52.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:52.385+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.387190+0000) 2019-10-13T07:34:52.399 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:52.386+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.387513+0000) 2019-10-13T07:34:52.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:52.386+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.387755+0000) 2019-10-13T07:34:52.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:52.386+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.387617+0000) 2019-10-13T07:34:52.400 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:52.386+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.388168+0000) 2019-10-13T07:34:52.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:52.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:52.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:52.808+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.809346+0000) 2019-10-13T07:34:52.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:52.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.810221+0000) 2019-10-13T07:34:57.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:34:57.390+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.392555+0000) 2019-10-13T07:34:57.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:57.392+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.394225+0000) 2019-10-13T07:34:57.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:57.393+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.394485+0000) 2019-10-13T07:34:57.398 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:34:57.393+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.394678+0000) 2019-10-13T07:34:57.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:57.393+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.395024+0000) 2019-10-13T07:34:57.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:34:57.394+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.395475+0000) 2019-10-13T07:34:57.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:57.394+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.395795+0000) 2019-10-13T07:34:57.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:34:57.394+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.395889+0000) 2019-10-13T07:34:57.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:34:57.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:34:57.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:34:57.806+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.807491+0000) 2019-10-13T07:34:58.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:34:58.355+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.356321+0000) 2019-10-13T07:34:58.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:34:58.693+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.694087+0000) 2019-10-13T07:34:58.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:34:58.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.694104+0000) 2019-10-13T07:35:01.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:01.945+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:01.946637+0000) 2019-10-13T07:35:02.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:02.096+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.098535+0000) 2019-10-13T07:35:02.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:02.399+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.400980+0000) 2019-10-13T07:35:02.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:02.399+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.401119+0000) 2019-10-13T07:35:02.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:02.398+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.400427+0000) 2019-10-13T07:35:02.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:02.400+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.402023+0000) 2019-10-13T07:35:02.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:02.400+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.402177+0000) 2019-10-13T07:35:02.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:02.400+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.402323+0000) 2019-10-13T07:35:02.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:02.401+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.402497+0000) 2019-10-13T07:35:02.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:02.400+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.402363+0000) 2019-10-13T07:35:02.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:02.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:02.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:02.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.809516+0000) 2019-10-13T07:35:02.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:02.809+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.810325+0000) 2019-10-13T07:35:07.411 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:07.407+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.408115+0000) 2019-10-13T07:35:07.411 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:07.407+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.408172+0000) 2019-10-13T07:35:07.412 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:07.406+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.407768+0000) 2019-10-13T07:35:07.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:07.408+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.409022+0000) 2019-10-13T07:35:07.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:07.408+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.409280+0000) 2019-10-13T07:35:07.412 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:07.409+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.409507+0000) 2019-10-13T07:35:07.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:07.409+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.409698+0000) 2019-10-13T07:35:07.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:07.408+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.410608+0000) 2019-10-13T07:35:07.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:07.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124654 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:07.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:07.807+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.807685+0000) 2019-10-13T07:35:08.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:08.356+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.356488+0000) 2019-10-13T07:35:08.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:08.693+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.694314+0000) 2019-10-13T07:35:08.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:08.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.694317+0000) 2019-10-13T07:35:11.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:11.946+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:11.946820+0000) 2019-10-13T07:35:12.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:12.100+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.100846+0000) 2019-10-13T07:35:12.417 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:12.414+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.415074+0000) 2019-10-13T07:35:12.418 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:12.413+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.414031+0000) 2019-10-13T07:35:12.419 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:12.415+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.415797+0000) 2019-10-13T07:35:12.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:12.415+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.415971+0000) 2019-10-13T07:35:12.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:12.415+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.416136+0000) 2019-10-13T07:35:12.419 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:12.415+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.416386+0000) 2019-10-13T07:35:12.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:12.415+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.415931+0000) 2019-10-13T07:35:12.420 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:12.416+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.416626+0000) 2019-10-13T07:35:12.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:12.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124686 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:12.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:12.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.809671+0000) 2019-10-13T07:35:12.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:12.809+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.810474+0000) 2019-10-13T07:35:17.424 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:17.419+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.420494+0000) 2019-10-13T07:35:17.426 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:17.421+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.422152+0000) 2019-10-13T07:35:17.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:17.421+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.422214+0000) 2019-10-13T07:35:17.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:17.421+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.422375+0000) 2019-10-13T07:35:17.427 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:17.422+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.422595+0000) 2019-10-13T07:35:17.427 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:17.422+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.422736+0000) 2019-10-13T07:35:17.427 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:17.422+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.422877+0000) 2019-10-13T07:35:17.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:17.423+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.423863+0000) 2019-10-13T07:35:17.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:17.426+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124716 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:17.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:17.807+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.807895+0000) 2019-10-13T07:35:18.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:18.356+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.356659+0000) 2019-10-13T07:35:18.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:18.694+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.694509+0000) 2019-10-13T07:35:18.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:18.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.694520+0000) 2019-10-13T07:35:18.844 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:35:18.844 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:18.849 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:35:18.849 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:21.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:21.946+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:21.947023+0000) 2019-10-13T07:35:22.107 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:22.102+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.103149+0000) 2019-10-13T07:35:22.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:22.426+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.427377+0000) 2019-10-13T07:35:22.438 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:22.428+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.428958+0000) 2019-10-13T07:35:22.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:22.428+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.429121+0000) 2019-10-13T07:35:22.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:22.428+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.429251+0000) 2019-10-13T07:35:22.439 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:22.429+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.429466+0000) 2019-10-13T07:35:22.439 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:22.428+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.429357+0000) 2019-10-13T07:35:22.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:22.429+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.429537+0000) 2019-10-13T07:35:22.440 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:22.428+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.429709+0000) 2019-10-13T07:35:22.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:22.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:22.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:22.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.809841+0000) 2019-10-13T07:35:22.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:22.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.810630+0000) 2019-10-13T07:35:27.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:27.432+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.434248+0000) 2019-10-13T07:35:27.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:27.433+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.434689+0000) 2019-10-13T07:35:27.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:27.432+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.432998+0000) 2019-10-13T07:35:27.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:27.433+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.434853+0000) 2019-10-13T07:35:27.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:27.433+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.435013+0000) 2019-10-13T07:35:27.439 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:27.433+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.435101+0000) 2019-10-13T07:35:27.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:27.434+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.435526+0000) 2019-10-13T07:35:27.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:27.435+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.435858+0000) 2019-10-13T07:35:27.442 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:27.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124775 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:27.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:27.807+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.808075+0000) 2019-10-13T07:35:28.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:28.355+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.356822+0000) 2019-10-13T07:35:28.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:28.693+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.694643+0000) 2019-10-13T07:35:28.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:28.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.694697+0000) 2019-10-13T07:35:31.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:31.946+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:31.947237+0000) 2019-10-13T07:35:32.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:32.105+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.105476+0000) 2019-10-13T07:35:32.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:32.439+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.441069+0000) 2019-10-13T07:35:32.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:32.439+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.441175+0000) 2019-10-13T07:35:32.444 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:32.439+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.441335+0000) 2019-10-13T07:35:32.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:32.440+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.441438+0000) 2019-10-13T07:35:32.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:32.440+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.441952+0000) 2019-10-13T07:35:32.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:32.439+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.440474+0000) 2019-10-13T07:35:32.446 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:32.440+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.442042+0000) 2019-10-13T07:35:32.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:32.441+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.442405+0000) 2019-10-13T07:35:32.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:32.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:32.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:32.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.810017+0000) 2019-10-13T07:35:32.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:32.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.810771+0000) 2019-10-13T07:35:37.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:37.445+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.446791+0000) 2019-10-13T07:35:37.450 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:37.445+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.447186+0000) 2019-10-13T07:35:37.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:37.445+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.447326+0000) 2019-10-13T07:35:37.451 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:37.446+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.447459+0000) 2019-10-13T07:35:37.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:37.446+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.447589+0000) 2019-10-13T07:35:37.451 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:37.446+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.447923+0000) 2019-10-13T07:35:37.452 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:37.446+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.446877+0000) 2019-10-13T07:35:37.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:37.448+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.449174+0000) 2019-10-13T07:35:37.454 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:37.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:37.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:37.807+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.808243+0000) 2019-10-13T07:35:38.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:38.356+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.357012+0000) 2019-10-13T07:35:38.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:38.694+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.694777+0000) 2019-10-13T07:35:38.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:38.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.694827+0000) 2019-10-13T07:35:41.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:41.946+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:41.947424+0000) 2019-10-13T07:35:42.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:42.107+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.107786+0000) 2019-10-13T07:35:42.456 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:42.452+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.453437+0000) 2019-10-13T07:35:42.457 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:42.452+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.453521+0000) 2019-10-13T07:35:42.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:42.452+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.453839+0000) 2019-10-13T07:35:42.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:42.452+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.454152+0000) 2019-10-13T07:35:42.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:42.452+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.454213+0000) 2019-10-13T07:35:42.458 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:42.451+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.452700+0000) 2019-10-13T07:35:42.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:42.454+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.454822+0000) 2019-10-13T07:35:42.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:42.453+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.454532+0000) 2019-10-13T07:35:42.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:42.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:42.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:42.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.810154+0000) 2019-10-13T07:35:42.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:42.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.810887+0000) 2019-10-13T07:35:47.461 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:47.456+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.457634+0000) 2019-10-13T07:35:47.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:47.457+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.459120+0000) 2019-10-13T07:35:47.462 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:47.458+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.459639+0000) 2019-10-13T07:35:47.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:47.458+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.459752+0000) 2019-10-13T07:35:47.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:47.458+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.459855+0000) 2019-10-13T07:35:47.463 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:47.458+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.460033+0000) 2019-10-13T07:35:47.463 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:47.458+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.460148+0000) 2019-10-13T07:35:47.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:47.459+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.459993+0000) 2019-10-13T07:35:47.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:47.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:47.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:47.807+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.808430+0000) 2019-10-13T07:35:48.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:48.356+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.357205+0000) 2019-10-13T07:35:48.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:48.694+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.694951+0000) 2019-10-13T07:35:48.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:48.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.695017+0000) 2019-10-13T07:35:48.949 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:35:48.949 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:48.953 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:35:48.954 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:51.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:51.947+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:51.947581+0000) 2019-10-13T07:35:52.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:52.109+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.110241+0000) 2019-10-13T07:35:52.469 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:52.466+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.466392+0000) 2019-10-13T07:35:52.470 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:52.464+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.465725+0000) 2019-10-13T07:35:52.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:52.466+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.467013+0000) 2019-10-13T07:35:52.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:52.466+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.466869+0000) 2019-10-13T07:35:52.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:52.466+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.467079+0000) 2019-10-13T07:35:52.471 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:52.466+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.467243+0000) 2019-10-13T07:35:52.472 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:52.467+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.467446+0000) 2019-10-13T07:35:52.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:52.466+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.468576+0000) 2019-10-13T07:35:52.474 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:52.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:52.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:52.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.810267+0000) 2019-10-13T07:35:52.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:52.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.811024+0000) 2019-10-13T07:35:57.477 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:35:57.470+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.471998+0000) 2019-10-13T07:35:57.478 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:57.472+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.473232+0000) 2019-10-13T07:35:57.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:57.473+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.473873+0000) 2019-10-13T07:35:57.478 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:57.473+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.474371+0000) 2019-10-13T07:35:57.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:35:57.474+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.474520+0000) 2019-10-13T07:35:57.479 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:35:57.473+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.474202+0000) 2019-10-13T07:35:57.479 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:57.474+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.474737+0000) 2019-10-13T07:35:57.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:35:57.472+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.474296+0000) 2019-10-13T07:35:57.481 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:35:57.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:35:57.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:35:57.808+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.808578+0000) 2019-10-13T07:35:58.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:35:58.357+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.357355+0000) 2019-10-13T07:35:58.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:35:58.694+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.695140+0000) 2019-10-13T07:35:58.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:35:58.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.695219+0000) 2019-10-13T07:36:01.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:01.947+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:01.947759+0000) 2019-10-13T07:36:02.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:02.111+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.112553+0000) 2019-10-13T07:36:02.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:02.477+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.478839+0000) 2019-10-13T07:36:02.483 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:02.480+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.480602+0000) 2019-10-13T07:36:02.484 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:02.480+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.480725+0000) 2019-10-13T07:36:02.484 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:02.480+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.480946+0000) 2019-10-13T07:36:02.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:02.480+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.480640+0000) 2019-10-13T07:36:02.484 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:02.480+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.481381+0000) 2019-10-13T07:36:02.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:02.481+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.481574+0000) 2019-10-13T07:36:02.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:02.480+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.481814+0000) 2019-10-13T07:36:02.488 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:02.485+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 124987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:02.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:02.810+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.810422+0000) 2019-10-13T07:36:02.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:02.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.811164+0000) 2019-10-13T07:36:07.490 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:07.487+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.487447+0000) 2019-10-13T07:36:07.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:07.487+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.487888+0000) 2019-10-13T07:36:07.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:07.486+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.486845+0000) 2019-10-13T07:36:07.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:07.487+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.488285+0000) 2019-10-13T07:36:07.497 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:07.487+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.488073+0000) 2019-10-13T07:36:07.497 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:07.488+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.488754+0000) 2019-10-13T07:36:07.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:07.488+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.488409+0000) 2019-10-13T07:36:07.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:07.488+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.488949+0000) 2019-10-13T07:36:07.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:07.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:07.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:07.808+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.808750+0000) 2019-10-13T07:36:08.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:08.357+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.357520+0000) 2019-10-13T07:36:08.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:08.695+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.695324+0000) 2019-10-13T07:36:08.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:08.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.695417+0000) 2019-10-13T07:36:11.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:11.946+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:11.947953+0000) 2019-10-13T07:36:12.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:12.114+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.114851+0000) 2019-10-13T07:36:12.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:12.493+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.494573+0000) 2019-10-13T07:36:12.498 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:12.493+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.493737+0000) 2019-10-13T07:36:12.498 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:12.493+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.494999+0000) 2019-10-13T07:36:12.498 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:12.494+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.495408+0000) 2019-10-13T07:36:12.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:12.494+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.495521+0000) 2019-10-13T07:36:12.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:12.494+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.495829+0000) 2019-10-13T07:36:12.499 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:12.494+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.495996+0000) 2019-10-13T07:36:12.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:12.494+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.495524+0000) 2019-10-13T07:36:12.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:12.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:12.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:12.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.810546+0000) 2019-10-13T07:36:12.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:12.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.811276+0000) 2019-10-13T07:36:17.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:17.500+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.501701+0000) 2019-10-13T07:36:17.505 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:17.499+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.500506+0000) 2019-10-13T07:36:17.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:17.500+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.502384+0000) 2019-10-13T07:36:17.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:17.501+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.502603+0000) 2019-10-13T07:36:17.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:17.501+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.502689+0000) 2019-10-13T07:36:17.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:17.501+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.502800+0000) 2019-10-13T07:36:17.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:17.501+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.502905+0000) 2019-10-13T07:36:17.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:17.502+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.502931+0000) 2019-10-13T07:36:17.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:17.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:17.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:17.808+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.808918+0000) 2019-10-13T07:36:18.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:18.356+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.357674+0000) 2019-10-13T07:36:18.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:18.694+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.695493+0000) 2019-10-13T07:36:18.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:18.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.695566+0000) 2019-10-13T07:36:19.063 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:36:19.063 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:19.068 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:36:19.068 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:21.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:21.947+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:21.948169+0000) 2019-10-13T07:36:22.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:22.116+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.117247+0000) 2019-10-13T07:36:22.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:22.506+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.508075+0000) 2019-10-13T07:36:22.511 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:22.506+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.508178+0000) 2019-10-13T07:36:22.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:22.506+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.508241+0000) 2019-10-13T07:36:22.511 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:22.507+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.508816+0000) 2019-10-13T07:36:22.512 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:22.506+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.507211+0000) 2019-10-13T07:36:22.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:22.507+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.508920+0000) 2019-10-13T07:36:22.513 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:22.507+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.509126+0000) 2019-10-13T07:36:22.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:22.508+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.509280+0000) 2019-10-13T07:36:22.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:22.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:22.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:22.809+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.810708+0000) 2019-10-13T07:36:22.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:22.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.811413+0000) 2019-10-13T07:36:27.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:27.513+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.514831+0000) 2019-10-13T07:36:27.518 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:27.513+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.515243+0000) 2019-10-13T07:36:27.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:27.513+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.514269+0000) 2019-10-13T07:36:27.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:27.514+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.515438+0000) 2019-10-13T07:36:27.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:27.514+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.516165+0000) 2019-10-13T07:36:27.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:27.514+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.516385+0000) 2019-10-13T07:36:27.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:27.514+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.515702+0000) 2019-10-13T07:36:27.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:27.515+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.516636+0000) 2019-10-13T07:36:27.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:27.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:27.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:27.808+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.809092+0000) 2019-10-13T07:36:28.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:28.357+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.357842+0000) 2019-10-13T07:36:28.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:28.695+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.695695+0000) 2019-10-13T07:36:28.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:28.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.695765+0000) 2019-10-13T07:36:31.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:31.947+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:31.948365+0000) 2019-10-13T07:36:32.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:32.119+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.119518+0000) 2019-10-13T07:36:32.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:32.520+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.521901+0000) 2019-10-13T07:36:32.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:32.520+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.522037+0000) 2019-10-13T07:36:32.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:32.520+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.522222+0000) 2019-10-13T07:36:32.525 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:32.521+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.522596+0000) 2019-10-13T07:36:32.526 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:32.520+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.521060+0000) 2019-10-13T07:36:32.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:32.521+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.522731+0000) 2019-10-13T07:36:32.526 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:32.522+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.523413+0000) 2019-10-13T07:36:32.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:32.523+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.523910+0000) 2019-10-13T07:36:32.529 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:32.525+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:32.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:32.810+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.810878+0000) 2019-10-13T07:36:32.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:32.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.811544+0000) 2019-10-13T07:36:37.531 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:37.526+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.527640+0000) 2019-10-13T07:36:37.532 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:37.527+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.529188+0000) 2019-10-13T07:36:37.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:37.528+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.529405+0000) 2019-10-13T07:36:37.533 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:37.528+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.529617+0000) 2019-10-13T07:36:37.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:37.528+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.529817+0000) 2019-10-13T07:36:37.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:37.528+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.530146+0000) 2019-10-13T07:36:37.533 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:37.529+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.530630+0000) 2019-10-13T07:36:37.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:37.530+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.531143+0000) 2019-10-13T07:36:37.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:37.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:37.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:37.808+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.809282+0000) 2019-10-13T07:36:38.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:38.357+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.358007+0000) 2019-10-13T07:36:38.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:38.695+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.695884+0000) 2019-10-13T07:36:38.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:38.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.695959+0000) 2019-10-13T07:36:41.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:41.948+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:41.948538+0000) 2019-10-13T07:36:42.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:42.121+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.121919+0000) 2019-10-13T07:36:42.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:42.535+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.536070+0000) 2019-10-13T07:36:42.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:42.535+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.536199+0000) 2019-10-13T07:36:42.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:42.536+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.536521+0000) 2019-10-13T07:36:42.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:42.536+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.536778+0000) 2019-10-13T07:36:42.540 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:42.536+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.536798+0000) 2019-10-13T07:36:42.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:42.534+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.535078+0000) 2019-10-13T07:36:42.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:42.537+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.537824+0000) 2019-10-13T07:36:42.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:42.536+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.537281+0000) 2019-10-13T07:36:42.543 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:42.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:42.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:42.810+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.811020+0000) 2019-10-13T07:36:42.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:42.811+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.811692+0000) 2019-10-13T07:36:47.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:47.541+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.542274+0000) 2019-10-13T07:36:47.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:47.542+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.542444+0000) 2019-10-13T07:36:47.546 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:47.539+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.541290+0000) 2019-10-13T07:36:47.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:47.542+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.542866+0000) 2019-10-13T07:36:47.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:47.542+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.542999+0000) 2019-10-13T07:36:47.546 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:47.542+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.543221+0000) 2019-10-13T07:36:47.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:47.543+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.543794+0000) 2019-10-13T07:36:47.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:47.542+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.544317+0000) 2019-10-13T07:36:47.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:47.547+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:47.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:47.809+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.809455+0000) 2019-10-13T07:36:48.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:48.357+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.358168+0000) 2019-10-13T07:36:48.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:48.695+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.696081+0000) 2019-10-13T07:36:48.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:48.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.696145+0000) 2019-10-13T07:36:49.234 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:36:49.235 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:49.239 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:36:49.239 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:51.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:51.948+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:51.948706+0000) 2019-10-13T07:36:52.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:52.122+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.124199+0000) 2019-10-13T07:36:52.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:52.548+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.549083+0000) 2019-10-13T07:36:52.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:52.546+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.547973+0000) 2019-10-13T07:36:52.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:52.549+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.549935+0000) 2019-10-13T07:36:52.553 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:52.549+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.550114+0000) 2019-10-13T07:36:52.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:52.550+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.550394+0000) 2019-10-13T07:36:52.553 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:52.550+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.550471+0000) 2019-10-13T07:36:52.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:52.550+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.550703+0000) 2019-10-13T07:36:52.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:52.549+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.550966+0000) 2019-10-13T07:36:52.555 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:52.552+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:52.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:52.811+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.811182+0000) 2019-10-13T07:36:52.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:52.811+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.811903+0000) 2019-10-13T07:36:57.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:57.553+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.555011+0000) 2019-10-13T07:36:57.558 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:36:57.552+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.553859+0000) 2019-10-13T07:36:57.558 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:36:57.554+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.555625+0000) 2019-10-13T07:36:57.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:57.554+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.555818+0000) 2019-10-13T07:36:57.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:57.554+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.556024+0000) 2019-10-13T07:36:57.559 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:36:57.554+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.556365+0000) 2019-10-13T07:36:57.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:36:57.554+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.556178+0000) 2019-10-13T07:36:57.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:57.555+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.556592+0000) 2019-10-13T07:36:57.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:36:57.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:36:57.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:36:57.808+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.809593+0000) 2019-10-13T07:36:58.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:36:58.357+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.358331+0000) 2019-10-13T07:36:58.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:36:58.694+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.696287+0000) 2019-10-13T07:36:58.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:36:58.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.696313+0000) 2019-10-13T07:37:01.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:01.947+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:01.948891+0000) 2019-10-13T07:37:02.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:02.126+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.126722+0000) 2019-10-13T07:37:02.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:02.561+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.562542+0000) 2019-10-13T07:37:02.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:02.561+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.562747+0000) 2019-10-13T07:37:02.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:02.561+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.562790+0000) 2019-10-13T07:37:02.567 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:02.561+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.563129+0000) 2019-10-13T07:37:02.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:02.562+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.563464+0000) 2019-10-13T07:37:02.568 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:02.562+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.563764+0000) 2019-10-13T07:37:02.568 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:02.560+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.560874+0000) 2019-10-13T07:37:02.568 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:02.562+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.563675+0000) 2019-10-13T07:37:02.570 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:02.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:02.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:02.810+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.811343+0000) 2019-10-13T07:37:02.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:02.811+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.812023+0000) 2019-10-13T07:37:07.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:07.568+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.569646+0000) 2019-10-13T07:37:07.573 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:07.568+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.570102+0000) 2019-10-13T07:37:07.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:07.568+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.570286+0000) 2019-10-13T07:37:07.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:07.569+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.570700+0000) 2019-10-13T07:37:07.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:07.569+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.570870+0000) 2019-10-13T07:37:07.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:07.568+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.569180+0000) 2019-10-13T07:37:07.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:07.569+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.570719+0000) 2019-10-13T07:37:07.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:07.569+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.571074+0000) 2019-10-13T07:37:07.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:07.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:07.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:07.808+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.809792+0000) 2019-10-13T07:37:08.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:08.357+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.358492+0000) 2019-10-13T07:37:08.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:08.695+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.696477+0000) 2019-10-13T07:37:08.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:08.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.696470+0000) 2019-10-13T07:37:11.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:11.947+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:11.949083+0000) 2019-10-13T07:37:12.132 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:12.128+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.128928+0000) 2019-10-13T07:37:12.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:12.574+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.576296+0000) 2019-10-13T07:37:12.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:12.575+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.576450+0000) 2019-10-13T07:37:12.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:12.575+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.576572+0000) 2019-10-13T07:37:12.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:12.575+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.576948+0000) 2019-10-13T07:37:12.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:12.575+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.577087+0000) 2019-10-13T07:37:12.582 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:12.574+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.575490+0000) 2019-10-13T07:37:12.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:12.576+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.577677+0000) 2019-10-13T07:37:12.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:12.578+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.578935+0000) 2019-10-13T07:37:12.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:12.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:12.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:12.810+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.811484+0000) 2019-10-13T07:37:12.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:12.811+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.812163+0000) 2019-10-13T07:37:17.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:17.580+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.581972+0000) 2019-10-13T07:37:17.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:17.580+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.582362+0000) 2019-10-13T07:37:17.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:17.580+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.580844+0000) 2019-10-13T07:37:17.586 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:17.581+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.583093+0000) 2019-10-13T07:37:17.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:17.581+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.583338+0000) 2019-10-13T07:37:17.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:17.582+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.583459+0000) 2019-10-13T07:37:17.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:17.582+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.583955+0000) 2019-10-13T07:37:17.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:17.583+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.584566+0000) 2019-10-13T07:37:17.589 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:17.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125432 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:17.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:17.809+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.809992+0000) 2019-10-13T07:37:18.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:18.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.358658+0000) 2019-10-13T07:37:18.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:18.696+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.696689+0000) 2019-10-13T07:37:18.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:18.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.696678+0000) 2019-10-13T07:37:19.397 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:37:19.397 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:19.402 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:37:19.402 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:21.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:21.948+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:21.949294+0000) 2019-10-13T07:37:22.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:22.130+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.131343+0000) 2019-10-13T07:37:22.591 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:22.587+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.588781+0000) 2019-10-13T07:37:22.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:22.587+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.587920+0000) 2019-10-13T07:37:22.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:22.587+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.589387+0000) 2019-10-13T07:37:22.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:22.588+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.589747+0000) 2019-10-13T07:37:22.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:22.588+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.589954+0000) 2019-10-13T07:37:22.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:22.588+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.590116+0000) 2019-10-13T07:37:22.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:22.588+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.590241+0000) 2019-10-13T07:37:22.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:22.589+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.590631+0000) 2019-10-13T07:37:22.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:22.593+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:22.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:22.811+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.811651+0000) 2019-10-13T07:37:22.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:22.811+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.812326+0000) 2019-10-13T07:37:27.599 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:27.596+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.596649+0000) 2019-10-13T07:37:27.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:27.594+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.595543+0000) 2019-10-13T07:37:27.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:27.596+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.597322+0000) 2019-10-13T07:37:27.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:27.596+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.597069+0000) 2019-10-13T07:37:27.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:27.597+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.597505+0000) 2019-10-13T07:37:27.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:27.597+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.597710+0000) 2019-10-13T07:37:27.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:27.597+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.598010+0000) 2019-10-13T07:37:27.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:27.597+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.598511+0000) 2019-10-13T07:37:27.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:27.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125490 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:27.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:27.809+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.810143+0000) 2019-10-13T07:37:28.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:28.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.358872+0000) 2019-10-13T07:37:28.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:28.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.696910+0000) 2019-10-13T07:37:28.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:28.696+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.696912+0000) 2019-10-13T07:37:31.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:31.949+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:31.949489+0000) 2019-10-13T07:37:32.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:32.133+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.133755+0000) 2019-10-13T07:37:32.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:32.602+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.603339+0000) 2019-10-13T07:37:32.606 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:32.603+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.603422+0000) 2019-10-13T07:37:32.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:32.603+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.603674+0000) 2019-10-13T07:37:32.607 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:32.602+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.603372+0000) 2019-10-13T07:37:32.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:32.603+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.604104+0000) 2019-10-13T07:37:32.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:32.604+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.604466+0000) 2019-10-13T07:37:32.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:32.604+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.604767+0000) 2019-10-13T07:37:32.608 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:32.604+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.604931+0000) 2019-10-13T07:37:32.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:32.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:32.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:32.811+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.811794+0000) 2019-10-13T07:37:32.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:32.812+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.812486+0000) 2019-10-13T07:37:37.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:37.609+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.610361+0000) 2019-10-13T07:37:37.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:37.610+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.610638+0000) 2019-10-13T07:37:37.614 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:37.610+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.610659+0000) 2019-10-13T07:37:37.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:37.610+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.610830+0000) 2019-10-13T07:37:37.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:37.610+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.610946+0000) 2019-10-13T07:37:37.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:37.610+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.611137+0000) 2019-10-13T07:37:37.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:37.609+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.609944+0000) 2019-10-13T07:37:37.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:37.610+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.611553+0000) 2019-10-13T07:37:37.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:37.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125550 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:37.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:37.810+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.810275+0000) 2019-10-13T07:37:38.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:38.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.359012+0000) 2019-10-13T07:37:38.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:38.696+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.697091+0000) 2019-10-13T07:37:38.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:38.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.697092+0000) 2019-10-13T07:37:41.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:41.949+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:41.949663+0000) 2019-10-13T07:37:42.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:42.135+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.136059+0000) 2019-10-13T07:37:42.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:42.612+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.614074+0000) 2019-10-13T07:37:42.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:42.614+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.615733+0000) 2019-10-13T07:37:42.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:42.614+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.615807+0000) 2019-10-13T07:37:42.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:42.614+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.616006+0000) 2019-10-13T07:37:42.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:42.615+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.616457+0000) 2019-10-13T07:37:42.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:42.615+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.616700+0000) 2019-10-13T07:37:42.621 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:42.615+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.616736+0000) 2019-10-13T07:37:42.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:42.615+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.617028+0000) 2019-10-13T07:37:42.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:42.619+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:42.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:42.810+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.811927+0000) 2019-10-13T07:37:42.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:42.810+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.812666+0000) 2019-10-13T07:37:47.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:47.619+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.620940+0000) 2019-10-13T07:37:47.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:47.620+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.622227+0000) 2019-10-13T07:37:47.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:47.620+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.622349+0000) 2019-10-13T07:37:47.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:47.621+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.622768+0000) 2019-10-13T07:37:47.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:47.621+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.622920+0000) 2019-10-13T07:37:47.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:47.621+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.623061+0000) 2019-10-13T07:37:47.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:47.621+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.623266+0000) 2019-10-13T07:37:47.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:47.621+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.623450+0000) 2019-10-13T07:37:47.629 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:47.625+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:47.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:47.809+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.810434+0000) 2019-10-13T07:37:48.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:48.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.359172+0000) 2019-10-13T07:37:48.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:48.696+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.697300+0000) 2019-10-13T07:37:48.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:48.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.697298+0000) 2019-10-13T07:37:49.559 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:37:49.560 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:49.564 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:37:49.564 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:51.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:51.949+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:51.949873+0000) 2019-10-13T07:37:52.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:52.137+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.138384+0000) 2019-10-13T07:37:52.632 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:52.627+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.629004+0000) 2019-10-13T07:37:52.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:52.627+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.628904+0000) 2019-10-13T07:37:52.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:52.628+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.629489+0000) 2019-10-13T07:37:52.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:52.628+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.629701+0000) 2019-10-13T07:37:52.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:52.628+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.629896+0000) 2019-10-13T07:37:52.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:52.626+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.628015+0000) 2019-10-13T07:37:52.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:52.628+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.630095+0000) 2019-10-13T07:37:52.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:52.628+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.630285+0000) 2019-10-13T07:37:52.636 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:52.632+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:52.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:52.811+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.812078+0000) 2019-10-13T07:37:52.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:52.812+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.812826+0000) 2019-10-13T07:37:57.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:37:57.633+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.634438+0000) 2019-10-13T07:37:57.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:37:57.634+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.635874+0000) 2019-10-13T07:37:57.639 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:57.634+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.636006+0000) 2019-10-13T07:37:57.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:57.634+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.636361+0000) 2019-10-13T07:37:57.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:57.635+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.636451+0000) 2019-10-13T07:37:57.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:37:57.635+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.636890+0000) 2019-10-13T07:37:57.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:57.635+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.637031+0000) 2019-10-13T07:37:57.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:37:57.636+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.636774+0000) 2019-10-13T07:37:57.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:37:57.639+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:37:57.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:37:57.809+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.810604+0000) 2019-10-13T07:37:58.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:37:58.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.359305+0000) 2019-10-13T07:37:58.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:37:58.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.697477+0000) 2019-10-13T07:37:58.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:37:58.696+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.697484+0000) 2019-10-13T07:38:01.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:01.949+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:01.950063+0000) 2019-10-13T07:38:02.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:02.140+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.140787+0000) 2019-10-13T07:38:02.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:02.641+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.642422+0000) 2019-10-13T07:38:02.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:02.641+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.642586+0000) 2019-10-13T07:38:02.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:02.641+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.641786+0000) 2019-10-13T07:38:02.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:02.642+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.643537+0000) 2019-10-13T07:38:02.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:02.642+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.643695+0000) 2019-10-13T07:38:02.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:02.642+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.643980+0000) 2019-10-13T07:38:02.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:02.642+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.644219+0000) 2019-10-13T07:38:02.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:02.643+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.644650+0000) 2019-10-13T07:38:02.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:02.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:02.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:02.811+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.812237+0000) 2019-10-13T07:38:02.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:02.812+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.812973+0000) 2019-10-13T07:38:07.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:07.648+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.649833+0000) 2019-10-13T07:38:07.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:07.648+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.650063+0000) 2019-10-13T07:38:07.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:07.648+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.650117+0000) 2019-10-13T07:38:07.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:07.648+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.649070+0000) 2019-10-13T07:38:07.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:07.649+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.650783+0000) 2019-10-13T07:38:07.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:07.649+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.651042+0000) 2019-10-13T07:38:07.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:07.649+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.651265+0000) 2019-10-13T07:38:07.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:07.651+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.651935+0000) 2019-10-13T07:38:07.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:07.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125726 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:07.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:07.810+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.810733+0000) 2019-10-13T07:38:08.362 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:08.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.359432+0000) 2019-10-13T07:38:08.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:08.696+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.697636+0000) 2019-10-13T07:38:08.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:08.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.697631+0000) 2019-10-13T07:38:11.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:11.949+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:11.950178+0000) 2019-10-13T07:38:12.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:12.142+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.143093+0000) 2019-10-13T07:38:12.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:12.656+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.657015+0000) 2019-10-13T07:38:12.660 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:12.656+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.657171+0000) 2019-10-13T07:38:12.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:12.656+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.657311+0000) 2019-10-13T07:38:12.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:12.657+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.657601+0000) 2019-10-13T07:38:12.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:12.655+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.656216+0000) 2019-10-13T07:38:12.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:12.657+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.657860+0000) 2019-10-13T07:38:12.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:12.657+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.658305+0000) 2019-10-13T07:38:12.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:12.657+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.658282+0000) 2019-10-13T07:38:12.663 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:12.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:12.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:12.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.812409+0000) 2019-10-13T07:38:12.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:12.812+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.813095+0000) 2019-10-13T07:38:17.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:17.662+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.663363+0000) 2019-10-13T07:38:17.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:17.663+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.663545+0000) 2019-10-13T07:38:17.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:17.662+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.662799+0000) 2019-10-13T07:38:17.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:17.663+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.664192+0000) 2019-10-13T07:38:17.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:17.663+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.664364+0000) 2019-10-13T07:38:17.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:17.664+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.664540+0000) 2019-10-13T07:38:17.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:17.664+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.664835+0000) 2019-10-13T07:38:17.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:17.665+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.666690+0000) 2019-10-13T07:38:17.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:17.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:17.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:17.810+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.810930+0000) 2019-10-13T07:38:18.362 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:18.359+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.359575+0000) 2019-10-13T07:38:18.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:18.697+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.697863+0000) 2019-10-13T07:38:18.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:18.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.697861+0000) 2019-10-13T07:38:19.715 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:38:19.715 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:19.720 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:38:19.720 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:21.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:21.950+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:21.950361+0000) 2019-10-13T07:38:22.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:22.145+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.145357+0000) 2019-10-13T07:38:22.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:22.670+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.670514+0000) 2019-10-13T07:38:22.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:22.670+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.670963+0000) 2019-10-13T07:38:22.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:22.670+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.671035+0000) 2019-10-13T07:38:22.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:22.670+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.671180+0000) 2019-10-13T07:38:22.678 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:22.670+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.671270+0000) 2019-10-13T07:38:22.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:22.671+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.671414+0000) 2019-10-13T07:38:22.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:22.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.669075+0000) 2019-10-13T07:38:22.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:22.670+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.670895+0000) 2019-10-13T07:38:22.679 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:22.675+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125817 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:22.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:22.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.812523+0000) 2019-10-13T07:38:22.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:22.813+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.813250+0000) 2019-10-13T07:38:27.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:27.677+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.677597+0000) 2019-10-13T07:38:27.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:27.676+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.676924+0000) 2019-10-13T07:38:27.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:27.677+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.677994+0000) 2019-10-13T07:38:27.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:27.678+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.678457+0000) 2019-10-13T07:38:27.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:27.678+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.678643+0000) 2019-10-13T07:38:27.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:27.678+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.678961+0000) 2019-10-13T07:38:27.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:27.678+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.679156+0000) 2019-10-13T07:38:27.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:27.679+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.679836+0000) 2019-10-13T07:38:27.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:27.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:27.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:27.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.811092+0000) 2019-10-13T07:38:28.362 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:28.359+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.359760+0000) 2019-10-13T07:38:28.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:28.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.698017+0000) 2019-10-13T07:38:28.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:28.696+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.698015+0000) 2019-10-13T07:38:31.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:31.949+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:31.950585+0000) 2019-10-13T07:38:32.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:32.147+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.147724+0000) 2019-10-13T07:38:32.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:32.682+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.683503+0000) 2019-10-13T07:38:32.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:32.683+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.685018+0000) 2019-10-13T07:38:32.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:32.684+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.685421+0000) 2019-10-13T07:38:32.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:32.684+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.685577+0000) 2019-10-13T07:38:32.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:32.684+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.685704+0000) 2019-10-13T07:38:32.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:32.684+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.686002+0000) 2019-10-13T07:38:32.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:32.685+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.686461+0000) 2019-10-13T07:38:32.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:32.685+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.685999+0000) 2019-10-13T07:38:32.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:32.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:32.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:32.811+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.812687+0000) 2019-10-13T07:38:32.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:32.813+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.813426+0000) 2019-10-13T07:38:37.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:37.690+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.692368+0000) 2019-10-13T07:38:37.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:37.691+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.692450+0000) 2019-10-13T07:38:37.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:37.691+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.692641+0000) 2019-10-13T07:38:37.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:37.691+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.693052+0000) 2019-10-13T07:38:37.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:37.689+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.691597+0000) 2019-10-13T07:38:37.697 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:37.692+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.693479+0000) 2019-10-13T07:38:37.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:37.692+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.693902+0000) 2019-10-13T07:38:37.698 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:37.693+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.694975+0000) 2019-10-13T07:38:37.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:37.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125904 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:37.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:37.810+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.811200+0000) 2019-10-13T07:38:38.362 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:38.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.359956+0000) 2019-10-13T07:38:38.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:38.697+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.698208+0000) 2019-10-13T07:38:38.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:38.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.698206+0000) 2019-10-13T07:38:41.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:41.949+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:41.950747+0000) 2019-10-13T07:38:42.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:42.149+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.150180+0000) 2019-10-13T07:38:42.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:42.697+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.699289+0000) 2019-10-13T07:38:42.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:42.698+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.699707+0000) 2019-10-13T07:38:42.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:42.698+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.699797+0000) 2019-10-13T07:38:42.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:42.697+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.698877+0000) 2019-10-13T07:38:42.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:42.699+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.700535+0000) 2019-10-13T07:38:42.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:42.699+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.700671+0000) 2019-10-13T07:38:42.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:42.699+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.700853+0000) 2019-10-13T07:38:42.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:42.698+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.700363+0000) 2019-10-13T07:38:42.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:42.703+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:42.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:42.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.812817+0000) 2019-10-13T07:38:42.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:42.811+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.813603+0000) 2019-10-13T07:38:47.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:47.705+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.706615+0000) 2019-10-13T07:38:47.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:47.705+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.706891+0000) 2019-10-13T07:38:47.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:47.705+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.707070+0000) 2019-10-13T07:38:47.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:47.705+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.707268+0000) 2019-10-13T07:38:47.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:47.704+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.706160+0000) 2019-10-13T07:38:47.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:47.705+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.707280+0000) 2019-10-13T07:38:47.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:47.706+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.707482+0000) 2019-10-13T07:38:47.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:47.706+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.707800+0000) 2019-10-13T07:38:47.714 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:47.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:47.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:47.810+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.811363+0000) 2019-10-13T07:38:48.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:48.358+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.360130+0000) 2019-10-13T07:38:48.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:48.697+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.698365+0000) 2019-10-13T07:38:48.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:48.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.698374+0000) 2019-10-13T07:38:49.846 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:38:49.846 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:49.850 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:38:49.850 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:51.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:51.950+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:51.950924+0000) 2019-10-13T07:38:52.156 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:52.151+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.152468+0000) 2019-10-13T07:38:52.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:52.711+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.712079+0000) 2019-10-13T07:38:52.717 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:52.712+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.713536+0000) 2019-10-13T07:38:52.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:52.712+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.713865+0000) 2019-10-13T07:38:52.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:52.712+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.714202+0000) 2019-10-13T07:38:52.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:52.712+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.714361+0000) 2019-10-13T07:38:52.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:52.713+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.714542+0000) 2019-10-13T07:38:52.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:52.713+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.714752+0000) 2019-10-13T07:38:52.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:52.713+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.714591+0000) 2019-10-13T07:38:52.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:52.717+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 125995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:52.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:52.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.812957+0000) 2019-10-13T07:38:52.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:52.813+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.813766+0000) 2019-10-13T07:38:57.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:57.720+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.720727+0000) 2019-10-13T07:38:57.723 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:57.720+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.720835+0000) 2019-10-13T07:38:57.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:57.720+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.721054+0000) 2019-10-13T07:38:57.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:57.720+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.721363+0000) 2019-10-13T07:38:57.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:38:57.719+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.720010+0000) 2019-10-13T07:38:57.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:38:57.720+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.721142+0000) 2019-10-13T07:38:57.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:38:57.721+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.721490+0000) 2019-10-13T07:38:57.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:38:57.721+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.721655+0000) 2019-10-13T07:38:57.728 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:38:57.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:38:57.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:38:57.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.811487+0000) 2019-10-13T07:38:58.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:38:58.359+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.360295+0000) 2019-10-13T07:38:58.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:38:58.698+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.698540+0000) 2019-10-13T07:38:58.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:38:58.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.698546+0000) 2019-10-13T07:39:01.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:01.950+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:01.951111+0000) 2019-10-13T07:39:02.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:02.153+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.154645+0000) 2019-10-13T07:39:02.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:02.727+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.727727+0000) 2019-10-13T07:39:02.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:02.727+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.727857+0000) 2019-10-13T07:39:02.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:02.727+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.728222+0000) 2019-10-13T07:39:02.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:02.727+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.728312+0000) 2019-10-13T07:39:02.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:02.728+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.728528+0000) 2019-10-13T07:39:02.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:02.726+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.726769+0000) 2019-10-13T07:39:02.732 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:02.728+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.728953+0000) 2019-10-13T07:39:02.733 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:02.728+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.729308+0000) 2019-10-13T07:39:02.736 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:02.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:02.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:02.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.813118+0000) 2019-10-13T07:39:02.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:02.813+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.813954+0000) 2019-10-13T07:39:07.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:07.735+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.735511+0000) 2019-10-13T07:39:07.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:07.735+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.735696+0000) 2019-10-13T07:39:07.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:07.735+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.735821+0000) 2019-10-13T07:39:07.739 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:07.734+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.734802+0000) 2019-10-13T07:39:07.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:07.735+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.736163+0000) 2019-10-13T07:39:07.740 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:07.736+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.736584+0000) 2019-10-13T07:39:07.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:07.736+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.737046+0000) 2019-10-13T07:39:07.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:07.735+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.736147+0000) 2019-10-13T07:39:07.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:07.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:07.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:07.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.811636+0000) 2019-10-13T07:39:08.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:08.360+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.360442+0000) 2019-10-13T07:39:08.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:08.698+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.698710+0000) 2019-10-13T07:39:08.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:08.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.698714+0000) 2019-10-13T07:39:11.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:11.951+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:11.951297+0000) 2019-10-13T07:39:12.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:12.156+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.157011+0000) 2019-10-13T07:39:12.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:12.741+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.742104+0000) 2019-10-13T07:39:12.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:12.741+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.741996+0000) 2019-10-13T07:39:12.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:12.741+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.742243+0000) 2019-10-13T07:39:12.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:12.742+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.742402+0000) 2019-10-13T07:39:12.746 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:12.742+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.742569+0000) 2019-10-13T07:39:12.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:12.742+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.742864+0000) 2019-10-13T07:39:12.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:12.740+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.741111+0000) 2019-10-13T07:39:12.747 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:12.742+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.743590+0000) 2019-10-13T07:39:12.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:12.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:12.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:12.813+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.813247+0000) 2019-10-13T07:39:12.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:12.813+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:12.814090+0000) 2019-10-13T07:39:17.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:17.746+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.747813+0000) 2019-10-13T07:39:17.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:17.746+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.748087+0000) 2019-10-13T07:39:17.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:17.746+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.748198+0000) 2019-10-13T07:39:17.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:17.747+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.747777+0000) 2019-10-13T07:39:17.752 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:17.747+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.748505+0000) 2019-10-13T07:39:17.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:17.747+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.748641+0000) 2019-10-13T07:39:17.753 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:17.747+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.749095+0000) 2019-10-13T07:39:17.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:17.748+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.749671+0000) 2019-10-13T07:39:17.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:17.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:17.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:17.810+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.811805+0000) 2019-10-13T07:39:18.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:18.359+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.360632+0000) 2019-10-13T07:39:18.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:18.697+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.698819+0000) 2019-10-13T07:39:18.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:18.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.698881+0000) 2019-10-13T07:39:19.985 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:39:19.986 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:19.990 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:39:19.990 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:21.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:21.950+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:21.951420+0000) 2019-10-13T07:39:22.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:22.159+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.159284+0000) 2019-10-13T07:39:22.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:22.752+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.754371+0000) 2019-10-13T07:39:22.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:22.753+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.754584+0000) 2019-10-13T07:39:22.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:22.753+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.754889+0000) 2019-10-13T07:39:22.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:22.753+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.755101+0000) 2019-10-13T07:39:22.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:22.753+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.755181+0000) 2019-10-13T07:39:22.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:22.753+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.754279+0000) 2019-10-13T07:39:22.759 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:22.754+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.755686+0000) 2019-10-13T07:39:22.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:22.755+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.756526+0000) 2019-10-13T07:39:22.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:22.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:22.816 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:22.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.813404+0000) 2019-10-13T07:39:22.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:22.814+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:22.814228+0000) 2019-10-13T07:39:27.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:27.760+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.762040+0000) 2019-10-13T07:39:27.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:27.760+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.762231+0000) 2019-10-13T07:39:27.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:27.760+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.762270+0000) 2019-10-13T07:39:27.765 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:27.761+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.762517+0000) 2019-10-13T07:39:27.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:27.759+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.761196+0000) 2019-10-13T07:39:27.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:27.761+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.762870+0000) 2019-10-13T07:39:27.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:27.761+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.762982+0000) 2019-10-13T07:39:27.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:27.761+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.763623+0000) 2019-10-13T07:39:27.767 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:27.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126198 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:27.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:27.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.811974+0000) 2019-10-13T07:39:28.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:28.360+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.360797+0000) 2019-10-13T07:39:28.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:28.698+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.698960+0000) 2019-10-13T07:39:28.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:28.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.699024+0000) 2019-10-13T07:39:31.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:31.950+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:31.951573+0000) 2019-10-13T07:39:32.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:32.160+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.161600+0000) 2019-10-13T07:39:32.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:32.764+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.766345+0000) 2019-10-13T07:39:32.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:32.767+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.768793+0000) 2019-10-13T07:39:32.774 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:32.765+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.767109+0000) 2019-10-13T07:39:32.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:32.765+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.767176+0000) 2019-10-13T07:39:32.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:32.766+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.767970+0000) 2019-10-13T07:39:32.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:32.766+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.768302+0000) 2019-10-13T07:39:32.775 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:32.767+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.768667+0000) 2019-10-13T07:39:32.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:32.767+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.768887+0000) 2019-10-13T07:39:32.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:32.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:32.816 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:32.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.813519+0000) 2019-10-13T07:39:32.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:32.812+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:32.814391+0000) 2019-10-13T07:39:37.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:37.772+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.773923+0000) 2019-10-13T07:39:37.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:37.771+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.773099+0000) 2019-10-13T07:39:37.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:37.773+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.774501+0000) 2019-10-13T07:39:37.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:37.773+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.774904+0000) 2019-10-13T07:39:37.778 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:37.773+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.774989+0000) 2019-10-13T07:39:37.778 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:37.773+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.775150+0000) 2019-10-13T07:39:37.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:37.774+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.775497+0000) 2019-10-13T07:39:37.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:37.774+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.776300+0000) 2019-10-13T07:39:37.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:37.778+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:37.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:37.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.812103+0000) 2019-10-13T07:39:38.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:38.360+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.360941+0000) 2019-10-13T07:39:38.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:38.698+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.699129+0000) 2019-10-13T07:39:38.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:38.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.699193+0000) 2019-10-13T07:39:41.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:41.951+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:41.951779+0000) 2019-10-13T07:39:42.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:42.163+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.164044+0000) 2019-10-13T07:39:42.784 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:42.779+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.781149+0000) 2019-10-13T07:39:42.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:42.779+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.781353+0000) 2019-10-13T07:39:42.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:42.780+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.781661+0000) 2019-10-13T07:39:42.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:42.780+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.781738+0000) 2019-10-13T07:39:42.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:42.780+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.781904+0000) 2019-10-13T07:39:42.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:42.780+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.780965+0000) 2019-10-13T07:39:42.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:42.780+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.782329+0000) 2019-10-13T07:39:42.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:42.782+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.783198+0000) 2019-10-13T07:39:42.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:42.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126289 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:42.816 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:42.813+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.813678+0000) 2019-10-13T07:39:42.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:42.813+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:42.814523+0000) 2019-10-13T07:39:47.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:47.786+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.787057+0000) 2019-10-13T07:39:47.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:47.788+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.788721+0000) 2019-10-13T07:39:47.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:47.788+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.788887+0000) 2019-10-13T07:39:47.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:47.788+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.789039+0000) 2019-10-13T07:39:47.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:47.788+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.789297+0000) 2019-10-13T07:39:47.792 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:47.789+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.789482+0000) 2019-10-13T07:39:47.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:47.789+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.789693+0000) 2019-10-13T07:39:47.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:47.788+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.789608+0000) 2019-10-13T07:39:47.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:47.793+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:47.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:47.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.812223+0000) 2019-10-13T07:39:48.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:48.360+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.361084+0000) 2019-10-13T07:39:48.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:48.698+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.699292+0000) 2019-10-13T07:39:48.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:48.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.699383+0000) 2019-10-13T07:39:50.074 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:39:50.074 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:50.080 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:39:50.080 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:51.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:51.951+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:51.951990+0000) 2019-10-13T07:39:52.170 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:52.165+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.166460+0000) 2019-10-13T07:39:52.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:52.795+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.796084+0000) 2019-10-13T07:39:52.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:52.794+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.795196+0000) 2019-10-13T07:39:52.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:52.796+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.796612+0000) 2019-10-13T07:39:52.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:52.796+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.796960+0000) 2019-10-13T07:39:52.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:52.796+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.796648+0000) 2019-10-13T07:39:52.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:52.796+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.797104+0000) 2019-10-13T07:39:52.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:52.796+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.797214+0000) 2019-10-13T07:39:52.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:52.797+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.798532+0000) 2019-10-13T07:39:52.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:52.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:52.816 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:52.813+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.813804+0000) 2019-10-13T07:39:52.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:52.813+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:52.814629+0000) 2019-10-13T07:39:57.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:57.802+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.803061+0000) 2019-10-13T07:39:57.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:57.803+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.803425+0000) 2019-10-13T07:39:57.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:57.802+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.803299+0000) 2019-10-13T07:39:57.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:39:57.803+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.803576+0000) 2019-10-13T07:39:57.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:57.803+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.803869+0000) 2019-10-13T07:39:57.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:39:57.801+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.802666+0000) 2019-10-13T07:39:57.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:39:57.803+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.804031+0000) 2019-10-13T07:39:57.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:39:57.804+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.805060+0000) 2019-10-13T07:39:57.810 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:39:57.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:39:57.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:39:57.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.812348+0000) 2019-10-13T07:39:58.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:39:58.361+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.361226+0000) 2019-10-13T07:39:58.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:39:58.699+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.699479+0000) 2019-10-13T07:39:58.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:39:58.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.699580+0000) 2019-10-13T07:40:01.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:01.951+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:01.952201+0000) 2019-10-13T07:40:02.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:02.168+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.168898+0000) 2019-10-13T07:40:02.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:02.808+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.809514+0000) 2019-10-13T07:40:02.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:02.808+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.809711+0000) 2019-10-13T07:40:02.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:02.807+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.808677+0000) 2019-10-13T07:40:02.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:02.808+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.810108+0000) 2019-10-13T07:40:02.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:02.808+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.810260+0000) 2019-10-13T07:40:02.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:02.809+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.810424+0000) 2019-10-13T07:40:02.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:02.809+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.810541+0000) 2019-10-13T07:40:02.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:02.809+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.810317+0000) 2019-10-13T07:40:02.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:02.814+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.814744+0000) 2019-10-13T07:40:02.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:02.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:02.813938+0000) 2019-10-13T07:40:02.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:02.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:07.815 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:07.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.812473+0000) 2019-10-13T07:40:07.818 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:07.813+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.814590+0000) 2019-10-13T07:40:07.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:07.814+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.816092+0000) 2019-10-13T07:40:07.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:07.814+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.816340+0000) 2019-10-13T07:40:07.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:07.815+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.816402+0000) 2019-10-13T07:40:07.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:07.815+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.816513+0000) 2019-10-13T07:40:07.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:07.815+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.816648+0000) 2019-10-13T07:40:07.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:07.815+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.816742+0000) 2019-10-13T07:40:07.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:07.817+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.817970+0000) 2019-10-13T07:40:07.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:07.820+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126434 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:08.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:08.360+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.361390+0000) 2019-10-13T07:40:08.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:08.698+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.699649+0000) 2019-10-13T07:40:08.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:08.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.699740+0000) 2019-10-13T07:40:11.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:11.951+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:11.952428+0000) 2019-10-13T07:40:12.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:12.171+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.171365+0000) 2019-10-13T07:40:12.816 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:12.812+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.814128+0000) 2019-10-13T07:40:12.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:12.814+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.815465+0000) 2019-10-13T07:40:12.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:12.821+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.823284+0000) 2019-10-13T07:40:12.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:12.822+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.823630+0000) 2019-10-13T07:40:12.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:12.822+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.823537+0000) 2019-10-13T07:40:12.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:12.822+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.822914+0000) 2019-10-13T07:40:12.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:12.823+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.824637+0000) 2019-10-13T07:40:12.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:12.823+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.824478+0000) 2019-10-13T07:40:12.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:12.823+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.824903+0000) 2019-10-13T07:40:12.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:12.825+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:12.826009+0000) 2019-10-13T07:40:12.831 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:12.827+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:17.815 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:17.811+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.812651+0000) 2019-10-13T07:40:17.832 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:17.829+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.830492+0000) 2019-10-13T07:40:17.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:17.829+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.830795+0000) 2019-10-13T07:40:17.833 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:17.829+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.831001+0000) 2019-10-13T07:40:17.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:17.829+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.831103+0000) 2019-10-13T07:40:17.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:17.829+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.830444+0000) 2019-10-13T07:40:17.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:17.829+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.831378+0000) 2019-10-13T07:40:17.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:17.830+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.831465+0000) 2019-10-13T07:40:17.836 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:17.831+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.832630+0000) 2019-10-13T07:40:17.838 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:17.834+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:18.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:18.360+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.361577+0000) 2019-10-13T07:40:18.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:18.699+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.699844+0000) 2019-10-13T07:40:18.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:18.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.699888+0000) 2019-10-13T07:40:20.290 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:40:20.290 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:20.295 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:40:20.295 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:21.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:21.952+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:21.952622+0000) 2019-10-13T07:40:22.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:22.172+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.173775+0000) 2019-10-13T07:40:22.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:22.813+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.814321+0000) 2019-10-13T07:40:22.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:22.814+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.815648+0000) 2019-10-13T07:40:22.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:22.836+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.838161+0000) 2019-10-13T07:40:22.842 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:22.837+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.838750+0000) 2019-10-13T07:40:22.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:22.837+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.838903+0000) 2019-10-13T07:40:22.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:22.837+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.839193+0000) 2019-10-13T07:40:22.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:22.838+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.839424+0000) 2019-10-13T07:40:22.844 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:22.838+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.839761+0000) 2019-10-13T07:40:22.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:22.837+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.838793+0000) 2019-10-13T07:40:22.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:22.838+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:22.840277+0000) 2019-10-13T07:40:22.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:22.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:27.815 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:27.812+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.812841+0000) 2019-10-13T07:40:27.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:27.843+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.844784+0000) 2019-10-13T07:40:27.847 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:27.843+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.844915+0000) 2019-10-13T07:40:27.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:27.843+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.845045+0000) 2019-10-13T07:40:27.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:27.843+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.845279+0000) 2019-10-13T07:40:27.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:27.842+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.844465+0000) 2019-10-13T07:40:27.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:27.844+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.845781+0000) 2019-10-13T07:40:27.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:27.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.846044+0000) 2019-10-13T07:40:27.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:27.844+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.846174+0000) 2019-10-13T07:40:27.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:27.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:28.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:28.361+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.361731+0000) 2019-10-13T07:40:28.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:28.699+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.700052+0000) 2019-10-13T07:40:28.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:28.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.700077+0000) 2019-10-13T07:40:31.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:31.952+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:31.952797+0000) 2019-10-13T07:40:32.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:32.175+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.176250+0000) 2019-10-13T07:40:32.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:32.814+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.814504+0000) 2019-10-13T07:40:32.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:32.814+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.815791+0000) 2019-10-13T07:40:32.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:32.848+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.850651+0000) 2019-10-13T07:40:32.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:32.850+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.851070+0000) 2019-10-13T07:40:32.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:32.850+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.851187+0000) 2019-10-13T07:40:32.855 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:32.850+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.851280+0000) 2019-10-13T07:40:32.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:32.850+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.851381+0000) 2019-10-13T07:40:32.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:32.851+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.851516+0000) 2019-10-13T07:40:32.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:32.851+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.851776+0000) 2019-10-13T07:40:32.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:32.851+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:32.852800+0000) 2019-10-13T07:40:32.857 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:32.854+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:37.815 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:37.812+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.813034+0000) 2019-10-13T07:40:37.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:37.856+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.856632+0000) 2019-10-13T07:40:37.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:37.856+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.856816+0000) 2019-10-13T07:40:37.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:37.855+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.856535+0000) 2019-10-13T07:40:37.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:37.856+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.857176+0000) 2019-10-13T07:40:37.860 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:37.856+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.857309+0000) 2019-10-13T07:40:37.861 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:37.857+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.857504+0000) 2019-10-13T07:40:37.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:37.857+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.857905+0000) 2019-10-13T07:40:37.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:37.856+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.857588+0000) 2019-10-13T07:40:37.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:37.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:38.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:38.361+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.361893+0000) 2019-10-13T07:40:38.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:38.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.700206+0000) 2019-10-13T07:40:38.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:38.699+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.700198+0000) 2019-10-13T07:40:41.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:41.952+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:41.953002+0000) 2019-10-13T07:40:42.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:42.178+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.178682+0000) 2019-10-13T07:40:42.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:42.814+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.814710+0000) 2019-10-13T07:40:42.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:42.815+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.815974+0000) 2019-10-13T07:40:42.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:42.861+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.862399+0000) 2019-10-13T07:40:42.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:42.863+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.863535+0000) 2019-10-13T07:40:42.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:42.863+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.863425+0000) 2019-10-13T07:40:42.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:42.863+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.863865+0000) 2019-10-13T07:40:42.867 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:42.863+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.864062+0000) 2019-10-13T07:40:42.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:42.864+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.864408+0000) 2019-10-13T07:40:42.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:42.864+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.864739+0000) 2019-10-13T07:40:42.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:42.864+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:42.864912+0000) 2019-10-13T07:40:42.871 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:42.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:47.815 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:47.812+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.813256+0000) 2019-10-13T07:40:47.873 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:47.869+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.870509+0000) 2019-10-13T07:40:47.873 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:47.869+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.870738+0000) 2019-10-13T07:40:47.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:47.869+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.870827+0000) 2019-10-13T07:40:47.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:47.869+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.870617+0000) 2019-10-13T07:40:47.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:47.869+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.871340+0000) 2019-10-13T07:40:47.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:47.870+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.871550+0000) 2019-10-13T07:40:47.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:47.870+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.872121+0000) 2019-10-13T07:40:47.876 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:47.872+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.872778+0000) 2019-10-13T07:40:47.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:47.874+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:48.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:48.361+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.362056+0000) 2019-10-13T07:40:48.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:48.699+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.700311+0000) 2019-10-13T07:40:48.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:48.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.700346+0000) 2019-10-13T07:40:50.512 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:40:50.512 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:50.520 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:40:50.521 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:51.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:51.952+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:51.953161+0000) 2019-10-13T07:40:52.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:52.180+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.180960+0000) 2019-10-13T07:40:52.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:52.813+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.814926+0000) 2019-10-13T07:40:52.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:52.815+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.816125+0000) 2019-10-13T07:40:52.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:52.876+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.877757+0000) 2019-10-13T07:40:52.880 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:52.876+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.878071+0000) 2019-10-13T07:40:52.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:52.876+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.878156+0000) 2019-10-13T07:40:52.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:52.876+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.877444+0000) 2019-10-13T07:40:52.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:52.877+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.878486+0000) 2019-10-13T07:40:52.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:52.877+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.879121+0000) 2019-10-13T07:40:52.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:52.877+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.879125+0000) 2019-10-13T07:40:52.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:52.878+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:52.879064+0000) 2019-10-13T07:40:52.885 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:52.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:57.816 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:57.812+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.813452+0000) 2019-10-13T07:40:57.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:40:57.883+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.884729+0000) 2019-10-13T07:40:57.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:57.883+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.885023+0000) 2019-10-13T07:40:57.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:40:57.882+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.883715+0000) 2019-10-13T07:40:57.888 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:57.883+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.885354+0000) 2019-10-13T07:40:57.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:57.883+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.885249+0000) 2019-10-13T07:40:57.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:40:57.884+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.886040+0000) 2019-10-13T07:40:57.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:40:57.884+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.886253+0000) 2019-10-13T07:40:57.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:40:57.886+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.886775+0000) 2019-10-13T07:40:57.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:40:57.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:40:58.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:40:58.361+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.362202+0000) 2019-10-13T07:40:58.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:40:58.699+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.700486+0000) 2019-10-13T07:40:58.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:40:58.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.700492+0000) 2019-10-13T07:41:01.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:01.952+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:01.953340+0000) 2019-10-13T07:41:02.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:02.183+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.183956+0000) 2019-10-13T07:41:02.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:02.814+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.815087+0000) 2019-10-13T07:41:02.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:02.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.816269+0000) 2019-10-13T07:41:02.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:02.890+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.891482+0000) 2019-10-13T07:41:02.894 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:02.890+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.891673+0000) 2019-10-13T07:41:02.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:02.890+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.891866+0000) 2019-10-13T07:41:02.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:02.890+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.891997+0000) 2019-10-13T07:41:02.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:02.890+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.891014+0000) 2019-10-13T07:41:02.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:02.890+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.892264+0000) 2019-10-13T07:41:02.896 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:02.891+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.892673+0000) 2019-10-13T07:41:02.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:02.892+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:02.893340+0000) 2019-10-13T07:41:02.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:02.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:07.816 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:07.813+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.813624+0000) 2019-10-13T07:41:07.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:07.895+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.897071+0000) 2019-10-13T07:41:07.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:07.895+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.897211+0000) 2019-10-13T07:41:07.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:07.895+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.897350+0000) 2019-10-13T07:41:07.900 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:07.896+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.897523+0000) 2019-10-13T07:41:07.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:07.896+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.897102+0000) 2019-10-13T07:41:07.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:07.896+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.897730+0000) 2019-10-13T07:41:07.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:07.896+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.898383+0000) 2019-10-13T07:41:07.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:07.898+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.898820+0000) 2019-10-13T07:41:07.904 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:07.901+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:08.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:08.361+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.362388+0000) 2019-10-13T07:41:08.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:08.700+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.700650+0000) 2019-10-13T07:41:08.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:08.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.700670+0000) 2019-10-13T07:41:11.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:11.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:11.953522+0000) 2019-10-13T07:41:12.190 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:12.186+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.186495+0000) 2019-10-13T07:41:12.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:12.814+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.815245+0000) 2019-10-13T07:41:12.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:12.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.816418+0000) 2019-10-13T07:41:12.906 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:12.902+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.904149+0000) 2019-10-13T07:41:12.907 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:12.903+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.904473+0000) 2019-10-13T07:41:12.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:12.903+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.904625+0000) 2019-10-13T07:41:12.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:12.903+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.904774+0000) 2019-10-13T07:41:12.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:12.903+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.904871+0000) 2019-10-13T07:41:12.908 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:12.904+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.905579+0000) 2019-10-13T07:41:12.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:12.903+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.904165+0000) 2019-10-13T07:41:12.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:12.906+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:12.907126+0000) 2019-10-13T07:41:12.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:12.908+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:17.816 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:17.813+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.813799+0000) 2019-10-13T07:41:17.913 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:17.910+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.910901+0000) 2019-10-13T07:41:17.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:17.908+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.909906+0000) 2019-10-13T07:41:17.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:17.910+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.911290+0000) 2019-10-13T07:41:17.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:17.910+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.911329+0000) 2019-10-13T07:41:17.915 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:17.911+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.911580+0000) 2019-10-13T07:41:17.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:17.911+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.911787+0000) 2019-10-13T07:41:17.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:17.911+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.911749+0000) 2019-10-13T07:41:17.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:17.910+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.912163+0000) 2019-10-13T07:41:17.917 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:17.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126842 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:18.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:18.362+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.362565+0000) 2019-10-13T07:41:18.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:18.700+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.700831+0000) 2019-10-13T07:41:18.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:18.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.700835+0000) 2019-10-13T07:41:20.643 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:41:20.644 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:20.649 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:41:20.649 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:21.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:21.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:21.953692+0000) 2019-10-13T07:41:22.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:22.188+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.189027+0000) 2019-10-13T07:41:22.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:22.815+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.815445+0000) 2019-10-13T07:41:22.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:22.815+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.816568+0000) 2019-10-13T07:41:22.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:22.916+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.916626+0000) 2019-10-13T07:41:22.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:22.914+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.915956+0000) 2019-10-13T07:41:22.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:22.916+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.916772+0000) 2019-10-13T07:41:22.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:22.917+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.917420+0000) 2019-10-13T07:41:22.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:22.916+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.917179+0000) 2019-10-13T07:41:22.921 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:22.917+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.917588+0000) 2019-10-13T07:41:22.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:22.917+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.917886+0000) 2019-10-13T07:41:22.922 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:22.916+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:22.918641+0000) 2019-10-13T07:41:22.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:22.922+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:27.816 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:27.813+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.813928+0000) 2019-10-13T07:41:27.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:27.923+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.924359+0000) 2019-10-13T07:41:27.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:27.922+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.924071+0000) 2019-10-13T07:41:27.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:27.924+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.924642+0000) 2019-10-13T07:41:27.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:27.924+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.924844+0000) 2019-10-13T07:41:27.928 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:27.924+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.925019+0000) 2019-10-13T07:41:27.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:27.925+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.925561+0000) 2019-10-13T07:41:27.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:27.925+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.925680+0000) 2019-10-13T07:41:27.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:27.924+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.926445+0000) 2019-10-13T07:41:27.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:27.929+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:28.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:28.362+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.362733+0000) 2019-10-13T07:41:28.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:28.700+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.701008+0000) 2019-10-13T07:41:28.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:28.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.701009+0000) 2019-10-13T07:41:31.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:31.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:31.953906+0000) 2019-10-13T07:41:32.195 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:32.190+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.191394+0000) 2019-10-13T07:41:32.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:32.815+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.815650+0000) 2019-10-13T07:41:32.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:32.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.816738+0000) 2019-10-13T07:41:32.937 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:32.931+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.931583+0000) 2019-10-13T07:41:32.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:32.931+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.931710+0000) 2019-10-13T07:41:32.938 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:32.931+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.931883+0000) 2019-10-13T07:41:32.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:32.931+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.932025+0000) 2019-10-13T07:41:32.939 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:32.931+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.932240+0000) 2019-10-13T07:41:32.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:32.932+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.932785+0000) 2019-10-13T07:41:32.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:32.929+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.930240+0000) 2019-10-13T07:41:32.940 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:32.933+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:32.933753+0000) 2019-10-13T07:41:32.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:32.936+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126929 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:37.816 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:37.813+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.814064+0000) 2019-10-13T07:41:37.942 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:37.937+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.939093+0000) 2019-10-13T07:41:37.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:37.937+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.939233+0000) 2019-10-13T07:41:37.943 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:37.938+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.939405+0000) 2019-10-13T07:41:37.943 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:37.938+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.939482+0000) 2019-10-13T07:41:37.943 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:37.938+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.939609+0000) 2019-10-13T07:41:37.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:37.938+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.939718+0000) 2019-10-13T07:41:37.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:37.936+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.937374+0000) 2019-10-13T07:41:37.944 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:37.939+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.939847+0000) 2019-10-13T07:41:37.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:37.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:38.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:38.361+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.362887+0000) 2019-10-13T07:41:38.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:38.700+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.701188+0000) 2019-10-13T07:41:38.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:38.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.701185+0000) 2019-10-13T07:41:41.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:41.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:41.954096+0000) 2019-10-13T07:41:42.197 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:42.193+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.193762+0000) 2019-10-13T07:41:42.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:42.814+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.815873+0000) 2019-10-13T07:41:42.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:42.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.816879+0000) 2019-10-13T07:41:42.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:42.943+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.944777+0000) 2019-10-13T07:41:42.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:42.943+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.945221+0000) 2019-10-13T07:41:42.948 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:42.944+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.945465+0000) 2019-10-13T07:41:42.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:42.944+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.945518+0000) 2019-10-13T07:41:42.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:42.944+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.944748+0000) 2019-10-13T07:41:42.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:42.944+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.946041+0000) 2019-10-13T07:41:42.949 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:42.944+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.946350+0000) 2019-10-13T07:41:42.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:42.945+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:42.946633+0000) 2019-10-13T07:41:42.952 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:42.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 126985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:47.816 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:47.813+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.814245+0000) 2019-10-13T07:41:47.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:47.950+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.951372+0000) 2019-10-13T07:41:47.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:47.950+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.952266+0000) 2019-10-13T07:41:47.956 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:47.951+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.952737+0000) 2019-10-13T07:41:47.956 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:47.951+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.953010+0000) 2019-10-13T07:41:47.956 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:47.951+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.953263+0000) 2019-10-13T07:41:47.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:47.952+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.953496+0000) 2019-10-13T07:41:47.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:47.952+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.953438+0000) 2019-10-13T07:41:47.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:47.953+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.953823+0000) 2019-10-13T07:41:47.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:47.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127012 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:48.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:48.362+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.363054+0000) 2019-10-13T07:41:48.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:48.700+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.701381+0000) 2019-10-13T07:41:48.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:48.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.701382+0000) 2019-10-13T07:41:50.757 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:41:50.757 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:50.762 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:41:50.762 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:51.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:51.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:51.954264+0000) 2019-10-13T07:41:52.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:52.195+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.196125+0000) 2019-10-13T07:41:52.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:52.815+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.816051+0000) 2019-10-13T07:41:52.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:52.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.816972+0000) 2019-10-13T07:41:52.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:52.957+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.957999+0000) 2019-10-13T07:41:52.962 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:52.957+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.959271+0000) 2019-10-13T07:41:52.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:52.958+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.959433+0000) 2019-10-13T07:41:52.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:52.958+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.959494+0000) 2019-10-13T07:41:52.963 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:52.958+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.959884+0000) 2019-10-13T07:41:52.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:52.958+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.959963+0000) 2019-10-13T07:41:52.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:52.958+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.960206+0000) 2019-10-13T07:41:52.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:52.959+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:52.960434+0000) 2019-10-13T07:41:52.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:52.961+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127043 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:57.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:57.813+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.814437+0000) 2019-10-13T07:41:57.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:41:57.963+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.963984+0000) 2019-10-13T07:41:57.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:41:57.963+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.965365+0000) 2019-10-13T07:41:57.968 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:57.964+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.965766+0000) 2019-10-13T07:41:57.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:57.964+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.966085+0000) 2019-10-13T07:41:57.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:41:57.964+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.966130+0000) 2019-10-13T07:41:57.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:57.964+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.966210+0000) 2019-10-13T07:41:57.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:41:57.965+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.966507+0000) 2019-10-13T07:41:57.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:41:57.966+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.967238+0000) 2019-10-13T07:41:57.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:41:57.969+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:41:58.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:41:58.362+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.363203+0000) 2019-10-13T07:41:58.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:41:58.701+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.701586+0000) 2019-10-13T07:41:58.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:41:58.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.701588+0000) 2019-10-13T07:42:01.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:01.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:01.954415+0000) 2019-10-13T07:42:02.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:02.197+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.198567+0000) 2019-10-13T07:42:02.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:02.815+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.816222+0000) 2019-10-13T07:42:02.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:02.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.817109+0000) 2019-10-13T07:42:02.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:02.972+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.972613+0000) 2019-10-13T07:42:02.975 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:02.972+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.972789+0000) 2019-10-13T07:42:02.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:02.971+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.971939+0000) 2019-10-13T07:42:02.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:02.972+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.973334+0000) 2019-10-13T07:42:02.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:02.973+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.973525+0000) 2019-10-13T07:42:02.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:02.973+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.973770+0000) 2019-10-13T07:42:02.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:02.973+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.974051+0000) 2019-10-13T07:42:02.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:02.973+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:02.973852+0000) 2019-10-13T07:42:02.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:02.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:07.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:07.814+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.814662+0000) 2019-10-13T07:42:07.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:07.978+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.978989+0000) 2019-10-13T07:42:07.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:07.979+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.980277+0000) 2019-10-13T07:42:07.983 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:07.980+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.980509+0000) 2019-10-13T07:42:07.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:07.980+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.980562+0000) 2019-10-13T07:42:07.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:07.980+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.980944+0000) 2019-10-13T07:42:07.984 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:07.980+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.981187+0000) 2019-10-13T07:42:07.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:07.981+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.981406+0000) 2019-10-13T07:42:07.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:07.980+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.980984+0000) 2019-10-13T07:42:07.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:07.985+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:08.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:08.363+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.363316+0000) 2019-10-13T07:42:08.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:08.701+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.701727+0000) 2019-10-13T07:42:08.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:08.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.701728+0000) 2019-10-13T07:42:11.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:11.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:11.954590+0000) 2019-10-13T07:42:12.205 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:12.200+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.201754+0000) 2019-10-13T07:42:12.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:12.816+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.816390+0000) 2019-10-13T07:42:12.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:12.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.817251+0000) 2019-10-13T07:42:12.990 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:12.987+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.987684+0000) 2019-10-13T07:42:12.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:12.987+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.988248+0000) 2019-10-13T07:42:12.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:12.988+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.988448+0000) 2019-10-13T07:42:12.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:12.988+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.988641+0000) 2019-10-13T07:42:12.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:12.988+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.988870+0000) 2019-10-13T07:42:12.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:12.985+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.987625+0000) 2019-10-13T07:42:12.992 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:12.988+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.989038+0000) 2019-10-13T07:42:12.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:12.987+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:12.989054+0000) 2019-10-13T07:42:12.995 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:12.992+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127160 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:17.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:17.814+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.814841+0000) 2019-10-13T07:42:17.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:17.994+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.995075+0000) 2019-10-13T07:42:17.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:17.992+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.994195+0000) 2019-10-13T07:42:17.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:17.995+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.995531+0000) 2019-10-13T07:42:17.999 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:17.995+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.995785+0000) 2019-10-13T07:42:17.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:17.995+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.995990+0000) 2019-10-13T07:42:17.999 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:17.995+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.996385+0000) 2019-10-13T07:42:17.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:17.996+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.996593+0000) 2019-10-13T07:42:18.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:17.996+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.997796+0000) 2019-10-13T07:42:18.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:18.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:18.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:18.363+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.363434+0000) 2019-10-13T07:42:18.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:18.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.701917+0000) 2019-10-13T07:42:18.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:18.701+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.701916+0000) 2019-10-13T07:42:20.858 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:42:20.859 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:20.864 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:42:20.864 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:21.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:21.953+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:21.954756+0000) 2019-10-13T07:42:22.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:22.203+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:22.204699+0000) 2019-10-13T07:42:22.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:22.815+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:22.816539+0000) 2019-10-13T07:42:22.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:22.816+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:22.817411+0000) 2019-10-13T07:42:23.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:23.000+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.002331+0000) 2019-10-13T07:42:23.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:23.001+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.002578+0000) 2019-10-13T07:42:23.005 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:23.001+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.003262+0000) 2019-10-13T07:42:23.005 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:23.001+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.003370+0000) 2019-10-13T07:42:23.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:23.000+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.002592+0000) 2019-10-13T07:42:23.006 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:23.002+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.003753+0000) 2019-10-13T07:42:23.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:23.002+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.003893+0000) 2019-10-13T07:42:23.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:23.002+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.004528+0000) 2019-10-13T07:42:23.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:23.006+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:27.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:27.814+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:27.815003+0000) 2019-10-13T07:42:28.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:28.007+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.009199+0000) 2019-10-13T07:42:28.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:28.008+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.009656+0000) 2019-10-13T07:42:28.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:28.007+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.008368+0000) 2019-10-13T07:42:28.014 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:28.008+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.009985+0000) 2019-10-13T07:42:28.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:28.008+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.010142+0000) 2019-10-13T07:42:28.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:28.009+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.010435+0000) 2019-10-13T07:42:28.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:28.009+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.010576+0000) 2019-10-13T07:42:28.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:28.010+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.011255+0000) 2019-10-13T07:42:28.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:28.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:28.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:28.362+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.363621+0000) 2019-10-13T07:42:28.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:28.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.702043+0000) 2019-10-13T07:42:28.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:28.700+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.702082+0000) 2019-10-13T07:42:31.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:31.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:31.954935+0000) 2019-10-13T07:42:32.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:32.206+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:32.207019+0000) 2019-10-13T07:42:32.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:32.815+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:32.816686+0000) 2019-10-13T07:42:32.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:32.817+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:32.817628+0000) 2019-10-13T07:42:33.019 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:33.015+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.016518+0000) 2019-10-13T07:42:33.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:33.015+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.016792+0000) 2019-10-13T07:42:33.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:33.015+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.016293+0000) 2019-10-13T07:42:33.020 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:33.015+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.017211+0000) 2019-10-13T07:42:33.020 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:33.016+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.017576+0000) 2019-10-13T07:42:33.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:33.016+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.017834+0000) 2019-10-13T07:42:33.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:33.016+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.018138+0000) 2019-10-13T07:42:33.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:33.017+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.018028+0000) 2019-10-13T07:42:33.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:33.020+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127280 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:37.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:37.814+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:37.815160+0000) 2019-10-13T07:42:38.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:38.021+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.021907+0000) 2019-10-13T07:42:38.026 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:38.022+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.023438+0000) 2019-10-13T07:42:38.026 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:38.022+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.023798+0000) 2019-10-13T07:42:38.026 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:38.022+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.023658+0000) 2019-10-13T07:42:38.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:38.022+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.023550+0000) 2019-10-13T07:42:38.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:38.022+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.024285+0000) 2019-10-13T07:42:38.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:38.023+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.024710+0000) 2019-10-13T07:42:38.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:38.024+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.025454+0000) 2019-10-13T07:42:38.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:38.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:38.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:38.363+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.363788+0000) 2019-10-13T07:42:38.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:38.701+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.702244+0000) 2019-10-13T07:42:38.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:38.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.702226+0000) 2019-10-13T07:42:41.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:41.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:41.955131+0000) 2019-10-13T07:42:42.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:42.208+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:42.209511+0000) 2019-10-13T07:42:42.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:42.816+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:42.816842+0000) 2019-10-13T07:42:42.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:42.817+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:42.817763+0000) 2019-10-13T07:42:43.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:43.028+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.029198+0000) 2019-10-13T07:42:43.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:43.028+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.030379+0000) 2019-10-13T07:42:43.033 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:43.029+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.030668+0000) 2019-10-13T07:42:43.034 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:43.029+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.030783+0000) 2019-10-13T07:42:43.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:43.029+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.031039+0000) 2019-10-13T07:42:43.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:43.030+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.031408+0000) 2019-10-13T07:42:43.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:43.030+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.031584+0000) 2019-10-13T07:42:43.036 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:43.032+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.032812+0000) 2019-10-13T07:42:43.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:43.034+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127339 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:47.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:47.814+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:47.815301+0000) 2019-10-13T07:42:48.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:48.035+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.035770+0000) 2019-10-13T07:42:48.040 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:48.035+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.036966+0000) 2019-10-13T07:42:48.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:48.035+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.037132+0000) 2019-10-13T07:42:48.040 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:48.035+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.037320+0000) 2019-10-13T07:42:48.040 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:48.035+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.037382+0000) 2019-10-13T07:42:48.041 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:48.036+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.037629+0000) 2019-10-13T07:42:48.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:48.036+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.037836+0000) 2019-10-13T07:42:48.041 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:48.037+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.038591+0000) 2019-10-13T07:42:48.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:48.039+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:48.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:48.363+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.363955+0000) 2019-10-13T07:42:48.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:48.702+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.702427+0000) 2019-10-13T07:42:48.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:48.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.702421+0000) 2019-10-13T07:42:51.011 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:42:51.012 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:51.016 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:42:51.016 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:51.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:51.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:51.955319+0000) 2019-10-13T07:42:52.215 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:52.211+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:52.211852+0000) 2019-10-13T07:42:52.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:52.816+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:52.817012+0000) 2019-10-13T07:42:52.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:52.817+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:52.817966+0000) 2019-10-13T07:42:53.045 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:53.042+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.042719+0000) 2019-10-13T07:42:53.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:53.041+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.042155+0000) 2019-10-13T07:42:53.046 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:53.042+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.043335+0000) 2019-10-13T07:42:53.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:53.043+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.043495+0000) 2019-10-13T07:42:53.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:53.043+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.043568+0000) 2019-10-13T07:42:53.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:53.043+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.043867+0000) 2019-10-13T07:42:53.047 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:53.043+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.044329+0000) 2019-10-13T07:42:53.048 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:53.044+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.044828+0000) 2019-10-13T07:42:53.050 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:53.047+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:57.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:57.815+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:57.815411+0000) 2019-10-13T07:42:58.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:42:58.047+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.048572+0000) 2019-10-13T07:42:58.053 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:58.049+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.049985+0000) 2019-10-13T07:42:58.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:42:58.049+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.050252+0000) 2019-10-13T07:42:58.053 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:42:58.049+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.050328+0000) 2019-10-13T07:42:58.053 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:58.050+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.050507+0000) 2019-10-13T07:42:58.054 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:58.050+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.050714+0000) 2019-10-13T07:42:58.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:42:58.050+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.050830+0000) 2019-10-13T07:42:58.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:42:58.050+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.051105+0000) 2019-10-13T07:42:58.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:42:58.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:42:58.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:42:58.363+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.364143+0000) 2019-10-13T07:42:58.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:42:58.702+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.702598+0000) 2019-10-13T07:42:58.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:42:58.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.702593+0000) 2019-10-13T07:43:01.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:01.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:01.955490+0000) 2019-10-13T07:43:02.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:02.214+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:02.214227+0000) 2019-10-13T07:43:02.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:02.817+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:02.817165+0000) 2019-10-13T07:43:02.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:02.818+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:02.818147+0000) 2019-10-13T07:43:03.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:03.054+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.055136+0000) 2019-10-13T07:43:03.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:03.056+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.056732+0000) 2019-10-13T07:43:03.059 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:03.056+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.056941+0000) 2019-10-13T07:43:03.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:03.056+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.057143+0000) 2019-10-13T07:43:03.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:03.056+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.057245+0000) 2019-10-13T07:43:03.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:03.056+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.057386+0000) 2019-10-13T07:43:03.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:03.057+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.057839+0000) 2019-10-13T07:43:03.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:03.057+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.058395+0000) 2019-10-13T07:43:03.064 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:03.061+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:07.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:07.814+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:07.815599+0000) 2019-10-13T07:43:08.067 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:08.062+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.063849+0000) 2019-10-13T07:43:08.067 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:08.062+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.064067+0000) 2019-10-13T07:43:08.067 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:08.062+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.064348+0000) 2019-10-13T07:43:08.067 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:08.063+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.064517+0000) 2019-10-13T07:43:08.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:08.060+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.062667+0000) 2019-10-13T07:43:08.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:08.063+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.065021+0000) 2019-10-13T07:43:08.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:08.063+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.065202+0000) 2019-10-13T07:43:08.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:08.064+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.066707+0000) 2019-10-13T07:43:08.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:08.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:08.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:08.362+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.364309+0000) 2019-10-13T07:43:08.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:08.701+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.702745+0000) 2019-10-13T07:43:08.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:08.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.702766+0000) 2019-10-13T07:43:11.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:11.954+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:11.955649+0000) 2019-10-13T07:43:12.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:12.215+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:12.216585+0000) 2019-10-13T07:43:12.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:12.816+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:12.817344+0000) 2019-10-13T07:43:12.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:12.817+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:12.818302+0000) 2019-10-13T07:43:13.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:13.069+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.070426+0000) 2019-10-13T07:43:13.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:13.069+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.070618+0000) 2019-10-13T07:43:13.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:13.069+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.070850+0000) 2019-10-13T07:43:13.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:13.068+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.069934+0000) 2019-10-13T07:43:13.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:13.069+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.071327+0000) 2019-10-13T07:43:13.075 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:13.070+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.071532+0000) 2019-10-13T07:43:13.075 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:13.070+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.071652+0000) 2019-10-13T07:43:13.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:13.070+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.072595+0000) 2019-10-13T07:43:13.076 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:13.073+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127520 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:17.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:17.815+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:17.815811+0000) 2019-10-13T07:43:18.078 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:18.074+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.076116+0000) 2019-10-13T07:43:18.079 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:18.075+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.076534+0000) 2019-10-13T07:43:18.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:18.074+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.075313+0000) 2019-10-13T07:43:18.080 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:18.075+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.076850+0000) 2019-10-13T07:43:18.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:18.075+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.077011+0000) 2019-10-13T07:43:18.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:18.075+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.077320+0000) 2019-10-13T07:43:18.080 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:18.076+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.077539+0000) 2019-10-13T07:43:18.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:18.078+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.078911+0000) 2019-10-13T07:43:18.083 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:18.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:18.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:18.363+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.364448+0000) 2019-10-13T07:43:18.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:18.702+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.702909+0000) 2019-10-13T07:43:18.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:18.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.702928+0000) 2019-10-13T07:43:21.100 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:43:21.100 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:21.104 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:43:21.104 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:21.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:21.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:21.955841+0000) 2019-10-13T07:43:22.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:22.218+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:22.218859+0000) 2019-10-13T07:43:22.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:22.816+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:22.817499+0000) 2019-10-13T07:43:22.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:22.817+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:22.818456+0000) 2019-10-13T07:43:23.086 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:23.082+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.083540+0000) 2019-10-13T07:43:23.086 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:23.082+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.083706+0000) 2019-10-13T07:43:23.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:23.082+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.083964+0000) 2019-10-13T07:43:23.087 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:23.082+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.084160+0000) 2019-10-13T07:43:23.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:23.082+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.082803+0000) 2019-10-13T07:43:23.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:23.083+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.084455+0000) 2019-10-13T07:43:23.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:23.083+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.084587+0000) 2019-10-13T07:43:23.088 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:23.083+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.084454+0000) 2019-10-13T07:43:23.091 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:23.087+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:27.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:27.815+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:27.816013+0000) 2019-10-13T07:43:28.093 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:28.089+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.090421+0000) 2019-10-13T07:43:28.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:28.089+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.090617+0000) 2019-10-13T07:43:28.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:28.088+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.089679+0000) 2019-10-13T07:43:28.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:28.089+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.091097+0000) 2019-10-13T07:43:28.094 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:28.089+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.091221+0000) 2019-10-13T07:43:28.095 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:28.090+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.091499+0000) 2019-10-13T07:43:28.095 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:28.090+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.091835+0000) 2019-10-13T07:43:28.096 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:28.092+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.092805+0000) 2019-10-13T07:43:28.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:28.094+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:28.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:28.364+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.364594+0000) 2019-10-13T07:43:28.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:28.702+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.703120+0000) 2019-10-13T07:43:28.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:28.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.703099+0000) 2019-10-13T07:43:31.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:31.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:31.956060+0000) 2019-10-13T07:43:32.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:32.220+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:32.221232+0000) 2019-10-13T07:43:32.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:32.817+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:32.817671+0000) 2019-10-13T07:43:32.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:32.817+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:32.818663+0000) 2019-10-13T07:43:33.100 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:33.096+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.098037+0000) 2019-10-13T07:43:33.101 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:33.096+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.098149+0000) 2019-10-13T07:43:33.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:33.096+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.098305+0000) 2019-10-13T07:43:33.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:33.097+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.098650+0000) 2019-10-13T07:43:33.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:33.097+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.097762+0000) 2019-10-13T07:43:33.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:33.098+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.098893+0000) 2019-10-13T07:43:33.102 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:33.097+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.098927+0000) 2019-10-13T07:43:33.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:33.097+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.099245+0000) 2019-10-13T07:43:33.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:33.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:37.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:37.815+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:37.816203+0000) 2019-10-13T07:43:38.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:38.104+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.105050+0000) 2019-10-13T07:43:38.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:38.104+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.105332+0000) 2019-10-13T07:43:38.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:38.105+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.105738+0000) 2019-10-13T07:43:38.109 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:38.105+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.105740+0000) 2019-10-13T07:43:38.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:38.105+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.105909+0000) 2019-10-13T07:43:38.109 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:38.105+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.106079+0000) 2019-10-13T07:43:38.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:38.104+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.105262+0000) 2019-10-13T07:43:38.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:38.105+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.106557+0000) 2019-10-13T07:43:38.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:38.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:38.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:38.364+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.364771+0000) 2019-10-13T07:43:38.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:38.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.703264+0000) 2019-10-13T07:43:38.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:38.702+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.703263+0000) 2019-10-13T07:43:41.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:41.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:41.956272+0000) 2019-10-13T07:43:42.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:42.223+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:42.223601+0000) 2019-10-13T07:43:42.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:42.817+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:42.817826+0000) 2019-10-13T07:43:42.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:42.818+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:42.818844+0000) 2019-10-13T07:43:43.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:43.111+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.112104+0000) 2019-10-13T07:43:43.115 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:43.111+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.112310+0000) 2019-10-13T07:43:43.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:43.112+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.112442+0000) 2019-10-13T07:43:43.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:43.112+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.112632+0000) 2019-10-13T07:43:43.116 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:43.112+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.112806+0000) 2019-10-13T07:43:43.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:43.111+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.111910+0000) 2019-10-13T07:43:43.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:43.113+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.113852+0000) 2019-10-13T07:43:43.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:43.113+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.114512+0000) 2019-10-13T07:43:43.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:43.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:47.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:47.816+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:47.816389+0000) 2019-10-13T07:43:48.122 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:48.118+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.119372+0000) 2019-10-13T07:43:48.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:48.119+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.119472+0000) 2019-10-13T07:43:48.122 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:48.119+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.119711+0000) 2019-10-13T07:43:48.122 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:48.119+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.119874+0000) 2019-10-13T07:43:48.123 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:48.118+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.119431+0000) 2019-10-13T07:43:48.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:48.120+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.120647+0000) 2019-10-13T07:43:48.124 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:48.120+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.120776+0000) 2019-10-13T07:43:48.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:48.121+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.122523+0000) 2019-10-13T07:43:48.127 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:48.124+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:48.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:48.364+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.364921+0000) 2019-10-13T07:43:48.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:48.703+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.703466+0000) 2019-10-13T07:43:48.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:48.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.703463+0000) 2019-10-13T07:43:51.181 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:43:51.182 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:51.186 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:43:51.186 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:51.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:51.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:51.956489+0000) 2019-10-13T07:43:52.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:52.225+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:52.226028+0000) 2019-10-13T07:43:52.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:52.817+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:52.817996+0000) 2019-10-13T07:43:52.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:52.818+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:52.819029+0000) 2019-10-13T07:43:53.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:53.124+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.125527+0000) 2019-10-13T07:43:53.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:53.125+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.126770+0000) 2019-10-13T07:43:53.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:53.125+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.127091+0000) 2019-10-13T07:43:53.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:53.125+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.127268+0000) 2019-10-13T07:43:53.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:53.125+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.127353+0000) 2019-10-13T07:43:53.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:53.126+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.127456+0000) 2019-10-13T07:43:53.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:53.126+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.127553+0000) 2019-10-13T07:43:53.132 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:53.128+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.128810+0000) 2019-10-13T07:43:53.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:53.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127760 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:57.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:57.815+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:57.816577+0000) 2019-10-13T07:43:58.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:58.131+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.133144+0000) 2019-10-13T07:43:58.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:58.132+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.133478+0000) 2019-10-13T07:43:58.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:43:58.132+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.133622+0000) 2019-10-13T07:43:58.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:43:58.131+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.133002+0000) 2019-10-13T07:43:58.137 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:43:58.132+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.134172+0000) 2019-10-13T07:43:58.137 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:43:58.132+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.134372+0000) 2019-10-13T07:43:58.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:58.133+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.134537+0000) 2019-10-13T07:43:58.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:43:58.133+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.135388+0000) 2019-10-13T07:43:58.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:43:58.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:43:58.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:43:58.364+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.365068+0000) 2019-10-13T07:43:58.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:43:58.702+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.703646+0000) 2019-10-13T07:43:58.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:43:58.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:58.703645+0000) 2019-10-13T07:44:01.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:01.955+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:01.956702+0000) 2019-10-13T07:44:02.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:02.227+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:02.228382+0000) 2019-10-13T07:44:02.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:02.817+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:02.818181+0000) 2019-10-13T07:44:02.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:02.818+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:02.819211+0000) 2019-10-13T07:44:03.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:03.139+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.140796+0000) 2019-10-13T07:44:03.143 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:03.139+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.141123+0000) 2019-10-13T07:44:03.144 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:03.138+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.139943+0000) 2019-10-13T07:44:03.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:03.140+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.141504+0000) 2019-10-13T07:44:03.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:03.140+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.141606+0000) 2019-10-13T07:44:03.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:03.140+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.141885+0000) 2019-10-13T07:44:03.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:03.140+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.141993+0000) 2019-10-13T07:44:03.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:03.140+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.142284+0000) 2019-10-13T07:44:03.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:03.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127819 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:07.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:07.816+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:07.816767+0000) 2019-10-13T07:44:08.151 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:08.147+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.148450+0000) 2019-10-13T07:44:08.151 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:08.147+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.148609+0000) 2019-10-13T07:44:08.152 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:08.146+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.147924+0000) 2019-10-13T07:44:08.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:08.147+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.149100+0000) 2019-10-13T07:44:08.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:08.147+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.149378+0000) 2019-10-13T07:44:08.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:08.148+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.149503+0000) 2019-10-13T07:44:08.153 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:08.148+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.149689+0000) 2019-10-13T07:44:08.155 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:08.149+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.150923+0000) 2019-10-13T07:44:08.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:08.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:08.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:08.363+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.365265+0000) 2019-10-13T07:44:08.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:08.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.703823+0000) 2019-10-13T07:44:08.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:08.703+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:08.703825+0000) 2019-10-13T07:44:11.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:11.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:11.956894+0000) 2019-10-13T07:44:12.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:12.229+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:12.230705+0000) 2019-10-13T07:44:12.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:12.817+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:12.818407+0000) 2019-10-13T07:44:12.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:12.818+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:12.819394+0000) 2019-10-13T07:44:13.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:13.154+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.156026+0000) 2019-10-13T07:44:13.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:13.154+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.155534+0000) 2019-10-13T07:44:13.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:13.155+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.156572+0000) 2019-10-13T07:44:13.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:13.155+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.157168+0000) 2019-10-13T07:44:13.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:13.155+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.157339+0000) 2019-10-13T07:44:13.160 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:13.156+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.157483+0000) 2019-10-13T07:44:13.160 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:13.156+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.157626+0000) 2019-10-13T07:44:13.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:13.157+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.157888+0000) 2019-10-13T07:44:13.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:13.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127880 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:17.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:17.816+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:17.816962+0000) 2019-10-13T07:44:18.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:18.162+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.163493+0000) 2019-10-13T07:44:18.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:18.162+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.163747+0000) 2019-10-13T07:44:18.167 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:18.162+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.163500+0000) 2019-10-13T07:44:18.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:18.162+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.164280+0000) 2019-10-13T07:44:18.167 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:18.163+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.164687+0000) 2019-10-13T07:44:18.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:18.163+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.164929+0000) 2019-10-13T07:44:18.168 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:18.163+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.165289+0000) 2019-10-13T07:44:18.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:18.165+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.166488+0000) 2019-10-13T07:44:18.171 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:18.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127908 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:18.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:18.364+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.365392+0000) 2019-10-13T07:44:18.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:18.703+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.703979+0000) 2019-10-13T07:44:18.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:18.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:18.703960+0000) 2019-10-13T07:44:21.358 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:44:21.359 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:21.363 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:44:21.363 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:21.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:21.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:21.957097+0000) 2019-10-13T07:44:22.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:22.232+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:22.233205+0000) 2019-10-13T07:44:22.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:22.818+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:22.818520+0000) 2019-10-13T07:44:22.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:22.819+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:22.819569+0000) 2019-10-13T07:44:23.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:23.170+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.170498+0000) 2019-10-13T07:44:23.173 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:23.170+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.170629+0000) 2019-10-13T07:44:23.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:23.169+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.169879+0000) 2019-10-13T07:44:23.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:23.170+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.171168+0000) 2019-10-13T07:44:23.174 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:23.170+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.171214+0000) 2019-10-13T07:44:23.174 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:23.170+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.171336+0000) 2019-10-13T07:44:23.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:23.171+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.171455+0000) 2019-10-13T07:44:23.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:23.171+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.171855+0000) 2019-10-13T07:44:23.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:23.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:27.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:27.816+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:27.817158+0000) 2019-10-13T07:44:28.178 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:28.175+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.176029+0000) 2019-10-13T07:44:28.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:28.175+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.176110+0000) 2019-10-13T07:44:28.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:28.175+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.176325+0000) 2019-10-13T07:44:28.179 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:28.176+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.176418+0000) 2019-10-13T07:44:28.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:28.176+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.176543+0000) 2019-10-13T07:44:28.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:28.175+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.176292+0000) 2019-10-13T07:44:28.180 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:28.176+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.177014+0000) 2019-10-13T07:44:28.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:28.177+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.178476+0000) 2019-10-13T07:44:28.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:28.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 127967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:28.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:28.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.365549+0000) 2019-10-13T07:44:28.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:28.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.704146+0000) 2019-10-13T07:44:28.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:28.703+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:28.704149+0000) 2019-10-13T07:44:31.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:31.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:31.957321+0000) 2019-10-13T07:44:32.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:32.235+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:32.235791+0000) 2019-10-13T07:44:32.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:32.818+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:32.818628+0000) 2019-10-13T07:44:32.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:32.819+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:32.819683+0000) 2019-10-13T07:44:33.184 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:33.179+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.180639+0000) 2019-10-13T07:44:33.185 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:33.181+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.181874+0000) 2019-10-13T07:44:33.185 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:33.181+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.182071+0000) 2019-10-13T07:44:33.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:33.181+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.182293+0000) 2019-10-13T07:44:33.185 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:33.182+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.182415+0000) 2019-10-13T07:44:33.185 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:33.182+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.182789+0000) 2019-10-13T07:44:33.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:33.182+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.183082+0000) 2019-10-13T07:44:33.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:33.183+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.183958+0000) 2019-10-13T07:44:33.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:33.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:37.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:37.817+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:37.817352+0000) 2019-10-13T07:44:38.192 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:38.188+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.188774+0000) 2019-10-13T07:44:38.193 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:38.189+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.190094+0000) 2019-10-13T07:44:38.193 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:38.190+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.190448+0000) 2019-10-13T07:44:38.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:38.190+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.190491+0000) 2019-10-13T07:44:38.193 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:38.190+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.190631+0000) 2019-10-13T07:44:38.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:38.190+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.190884+0000) 2019-10-13T07:44:38.194 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:38.190+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.190968+0000) 2019-10-13T07:44:38.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:38.190+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.191034+0000) 2019-10-13T07:44:38.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:38.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:38.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:38.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.365691+0000) 2019-10-13T07:44:38.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:38.704+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.704306+0000) 2019-10-13T07:44:38.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:38.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:38.704305+0000) 2019-10-13T07:44:41.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:41.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:41.957500+0000) 2019-10-13T07:44:42.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:42.237+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:42.238081+0000) 2019-10-13T07:44:42.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:42.817+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:42.818779+0000) 2019-10-13T07:44:42.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:42.819+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:42.819844+0000) 2019-10-13T07:44:43.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:43.194+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.196347+0000) 2019-10-13T07:44:43.199 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:43.195+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.196535+0000) 2019-10-13T07:44:43.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:43.195+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.196680+0000) 2019-10-13T07:44:43.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:43.195+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.196586+0000) 2019-10-13T07:44:43.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:43.195+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.197375+0000) 2019-10-13T07:44:43.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:43.196+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.197629+0000) 2019-10-13T07:44:43.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:43.196+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.198124+0000) 2019-10-13T07:44:43.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:43.197+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.198715+0000) 2019-10-13T07:44:43.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:43.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128059 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:47.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:47.816+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:47.817575+0000) 2019-10-13T07:44:48.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:48.202+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.204292+0000) 2019-10-13T07:44:48.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:48.203+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.204433+0000) 2019-10-13T07:44:48.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:48.203+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.204580+0000) 2019-10-13T07:44:48.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:48.203+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.204797+0000) 2019-10-13T07:44:48.208 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:48.203+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.204966+0000) 2019-10-13T07:44:48.209 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:48.203+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.204322+0000) 2019-10-13T07:44:48.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:48.204+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.205309+0000) 2019-10-13T07:44:48.209 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:48.204+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.205429+0000) 2019-10-13T07:44:48.212 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:48.208+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128087 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:48.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:48.364+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.365848+0000) 2019-10-13T07:44:48.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:48.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.704443+0000) 2019-10-13T07:44:48.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:48.703+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:48.704445+0000) 2019-10-13T07:44:51.543 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:44:51.544 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:51.548 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:44:51.548 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:51.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:51.956+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:51.957680+0000) 2019-10-13T07:44:52.243 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:52.239+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:52.240390+0000) 2019-10-13T07:44:52.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:52.818+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:52.818964+0000) 2019-10-13T07:44:52.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:52.819+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:52.820024+0000) 2019-10-13T07:44:53.214 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:53.210+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.211705+0000) 2019-10-13T07:44:53.214 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:53.210+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.211823+0000) 2019-10-13T07:44:53.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:53.209+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.211062+0000) 2019-10-13T07:44:53.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:53.210+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.212263+0000) 2019-10-13T07:44:53.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:53.211+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.212570+0000) 2019-10-13T07:44:53.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:53.211+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.212821+0000) 2019-10-13T07:44:53.216 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:53.211+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.213126+0000) 2019-10-13T07:44:53.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:53.211+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.213523+0000) 2019-10-13T07:44:53.219 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:53.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128120 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:57.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:57.817+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:57.817765+0000) 2019-10-13T07:44:58.221 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:44:58.216+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.218053+0000) 2019-10-13T07:44:58.222 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:58.217+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.219331+0000) 2019-10-13T07:44:58.222 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:58.218+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.219732+0000) 2019-10-13T07:44:58.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:44:58.218+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.219621+0000) 2019-10-13T07:44:58.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:58.218+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.220112+0000) 2019-10-13T07:44:58.223 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:44:58.218+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.220284+0000) 2019-10-13T07:44:58.223 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:44:58.219+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.220488+0000) 2019-10-13T07:44:58.224 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:44:58.219+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.221106+0000) 2019-10-13T07:44:58.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:44:58.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128148 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:44:58.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:44:58.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.366035+0000) 2019-10-13T07:44:58.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:44:58.703+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.704588+0000) 2019-10-13T07:44:58.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:44:58.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:58.704589+0000) 2019-10-13T07:45:01.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:01.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:01.957858+0000) 2019-10-13T07:45:02.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:02.241+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:02.242762+0000) 2019-10-13T07:45:02.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:02.818+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:02.819136+0000) 2019-10-13T07:45:02.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:02.819+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:02.820229+0000) 2019-10-13T07:45:03.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:03.224+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.226188+0000) 2019-10-13T07:45:03.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:03.225+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.226518+0000) 2019-10-13T07:45:03.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:03.223+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.225571+0000) 2019-10-13T07:45:03.230 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:03.225+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.226693+0000) 2019-10-13T07:45:03.230 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:03.225+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.226912+0000) 2019-10-13T07:45:03.230 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:03.225+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.227163+0000) 2019-10-13T07:45:03.230 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:03.226+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.227517+0000) 2019-10-13T07:45:03.231 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:03.226+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.228704+0000) 2019-10-13T07:45:03.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:03.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:07.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:07.817+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:07.817947+0000) 2019-10-13T07:45:08.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:08.231+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.232389+0000) 2019-10-13T07:45:08.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:08.233+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.233580+0000) 2019-10-13T07:45:08.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:08.233+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.233976+0000) 2019-10-13T07:45:08.237 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:08.233+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.234235+0000) 2019-10-13T07:45:08.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:08.233+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.234242+0000) 2019-10-13T07:45:08.237 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:08.234+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.234440+0000) 2019-10-13T07:45:08.237 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:08.234+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.234559+0000) 2019-10-13T07:45:08.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:08.234+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.234892+0000) 2019-10-13T07:45:08.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:08.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128207 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:08.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:08.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.366191+0000) 2019-10-13T07:45:08.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:08.704+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.704758+0000) 2019-10-13T07:45:08.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:08.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:08.704758+0000) 2019-10-13T07:45:11.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:11.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:11.958047+0000) 2019-10-13T07:45:12.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:12.244+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:12.245100+0000) 2019-10-13T07:45:12.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:12.819+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:12.819346+0000) 2019-10-13T07:45:12.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:12.819+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:12.820413+0000) 2019-10-13T07:45:13.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:13.241+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.241543+0000) 2019-10-13T07:45:13.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:13.240+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.241336+0000) 2019-10-13T07:45:13.245 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:13.241+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.241623+0000) 2019-10-13T07:45:13.245 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:13.241+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.241834+0000) 2019-10-13T07:45:13.246 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:13.241+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.242326+0000) 2019-10-13T07:45:13.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:13.242+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.242561+0000) 2019-10-13T07:45:13.246 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:13.239+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.240236+0000) 2019-10-13T07:45:13.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:13.241+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.242012+0000) 2019-10-13T07:45:13.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:13.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128240 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:17.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:17.817+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:17.818160+0000) 2019-10-13T07:45:18.250 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:18.246+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.247358+0000) 2019-10-13T07:45:18.251 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:18.248+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.248654+0000) 2019-10-13T07:45:18.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:18.248+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.248896+0000) 2019-10-13T07:45:18.252 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:18.248+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.248954+0000) 2019-10-13T07:45:18.252 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:18.248+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.249096+0000) 2019-10-13T07:45:18.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:18.249+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.249449+0000) 2019-10-13T07:45:18.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:18.248+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.249438+0000) 2019-10-13T07:45:18.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:18.249+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.249738+0000) 2019-10-13T07:45:18.256 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:18.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128268 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:18.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:18.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.366337+0000) 2019-10-13T07:45:18.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:18.704+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.704931+0000) 2019-10-13T07:45:18.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:18.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:18.704964+0000) 2019-10-13T07:45:21.707 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:45:21.708 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:21.712 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:45:21.712 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:21.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:21.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:21.958181+0000) 2019-10-13T07:45:22.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:22.246+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:22.247409+0000) 2019-10-13T07:45:22.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:22.819+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:22.819498+0000) 2019-10-13T07:45:22.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:22.819+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:22.820550+0000) 2019-10-13T07:45:23.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:23.253+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.254648+0000) 2019-10-13T07:45:23.259 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:23.255+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.255866+0000) 2019-10-13T07:45:23.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:23.255+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.256081+0000) 2019-10-13T07:45:23.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:23.255+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.256242+0000) 2019-10-13T07:45:23.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:23.256+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.256473+0000) 2019-10-13T07:45:23.259 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:23.256+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.256644+0000) 2019-10-13T07:45:23.260 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:23.256+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.256746+0000) 2019-10-13T07:45:23.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:23.257+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.258078+0000) 2019-10-13T07:45:23.263 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:23.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:27.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:27.817+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:27.818349+0000) 2019-10-13T07:45:28.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:28.261+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.262981+0000) 2019-10-13T07:45:28.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:28.261+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.262202+0000) 2019-10-13T07:45:28.266 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:28.262+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.263488+0000) 2019-10-13T07:45:28.266 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:28.262+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.263620+0000) 2019-10-13T07:45:28.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:28.262+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.263858+0000) 2019-10-13T07:45:28.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:28.262+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.264100+0000) 2019-10-13T07:45:28.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:28.262+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.264320+0000) 2019-10-13T07:45:28.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:28.264+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.265422+0000) 2019-10-13T07:45:28.270 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:28.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128327 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:28.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:28.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.366496+0000) 2019-10-13T07:45:28.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:28.704+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.705149+0000) 2019-10-13T07:45:28.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:28.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:28.705159+0000) 2019-10-13T07:45:31.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:31.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:31.958314+0000) 2019-10-13T07:45:32.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:32.249+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:32.249936+0000) 2019-10-13T07:45:32.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:32.818+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:32.819672+0000) 2019-10-13T07:45:32.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:32.820+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:32.820772+0000) 2019-10-13T07:45:33.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:33.268+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.269552+0000) 2019-10-13T07:45:33.273 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:33.268+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.270268+0000) 2019-10-13T07:45:33.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:33.269+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.270447+0000) 2019-10-13T07:45:33.274 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:33.269+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.270775+0000) 2019-10-13T07:45:33.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:33.269+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.270725+0000) 2019-10-13T07:45:33.274 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:33.269+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.270918+0000) 2019-10-13T07:45:33.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:33.269+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.271001+0000) 2019-10-13T07:45:33.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:33.271+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.272224+0000) 2019-10-13T07:45:33.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:33.273+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128360 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:37.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:37.817+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:37.818552+0000) 2019-10-13T07:45:38.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:38.274+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.275244+0000) 2019-10-13T07:45:38.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:38.275+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.276657+0000) 2019-10-13T07:45:38.279 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:38.275+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.276859+0000) 2019-10-13T07:45:38.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:38.275+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.276930+0000) 2019-10-13T07:45:38.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:38.275+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.277057+0000) 2019-10-13T07:45:38.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:38.275+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.277376+0000) 2019-10-13T07:45:38.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:38.276+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.277434+0000) 2019-10-13T07:45:38.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:38.276+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.277868+0000) 2019-10-13T07:45:38.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:38.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128388 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:38.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:38.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.366589+0000) 2019-10-13T07:45:38.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:38.704+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.705319+0000) 2019-10-13T07:45:38.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:38.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:38.705324+0000) 2019-10-13T07:45:41.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:41.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:41.958539+0000) 2019-10-13T07:45:42.255 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:42.252+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:42.252265+0000) 2019-10-13T07:45:42.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:42.818+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:42.819877+0000) 2019-10-13T07:45:42.824 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:42.820+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:42.820915+0000) 2019-10-13T07:45:43.285 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:43.281+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.282365+0000) 2019-10-13T07:45:43.287 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:43.282+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.283845+0000) 2019-10-13T07:45:43.287 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:43.282+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.284160+0000) 2019-10-13T07:45:43.287 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:43.282+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.284238+0000) 2019-10-13T07:45:43.288 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:43.283+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.284601+0000) 2019-10-13T07:45:43.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:43.283+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.284691+0000) 2019-10-13T07:45:43.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:43.283+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.285006+0000) 2019-10-13T07:45:43.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:43.284+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.284780+0000) 2019-10-13T07:45:43.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:43.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:47.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:47.818+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:47.818684+0000) 2019-10-13T07:45:48.293 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:48.289+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.290727+0000) 2019-10-13T07:45:48.293 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:48.288+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.290006+0000) 2019-10-13T07:45:48.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:48.289+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.291277+0000) 2019-10-13T07:45:48.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:48.290+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.291836+0000) 2019-10-13T07:45:48.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:48.290+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.291993+0000) 2019-10-13T07:45:48.295 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:48.290+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.292152+0000) 2019-10-13T07:45:48.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:48.290+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.292337+0000) 2019-10-13T07:45:48.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:48.290+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.292316+0000) 2019-10-13T07:45:48.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:48.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128447 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:48.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:48.366+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.366754+0000) 2019-10-13T07:45:48.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:48.704+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.705459+0000) 2019-10-13T07:45:48.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:48.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:48.705459+0000) 2019-10-13T07:45:51.804 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:45:51.805 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:51.809 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:45:51.809 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:51.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:51.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:51.958719+0000) 2019-10-13T07:45:52.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:52.253+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:52.254680+0000) 2019-10-13T07:45:52.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:52.819+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:52.820037+0000) 2019-10-13T07:45:52.824 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:52.820+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:52.821119+0000) 2019-10-13T07:45:53.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:53.295+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.297339+0000) 2019-10-13T07:45:53.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:53.296+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.297881+0000) 2019-10-13T07:45:53.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:53.296+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.298116+0000) 2019-10-13T07:45:53.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:53.295+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.297477+0000) 2019-10-13T07:45:53.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:53.296+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.298302+0000) 2019-10-13T07:45:53.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:53.296+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.298345+0000) 2019-10-13T07:45:53.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:53.297+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.298592+0000) 2019-10-13T07:45:53.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:53.296+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.298542+0000) 2019-10-13T07:45:53.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:53.300+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128480 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:57.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:57.818+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:57.818797+0000) 2019-10-13T07:45:58.305 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:45:58.300+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.302226+0000) 2019-10-13T07:45:58.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:45:58.303+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.303438+0000) 2019-10-13T07:45:58.307 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:58.303+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.303626+0000) 2019-10-13T07:45:58.307 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:45:58.303+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.303935+0000) 2019-10-13T07:45:58.307 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:58.303+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.304230+0000) 2019-10-13T07:45:58.307 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:45:58.303+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.304098+0000) 2019-10-13T07:45:58.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:58.304+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.304475+0000) 2019-10-13T07:45:58.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:45:58.304+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.305478+0000) 2019-10-13T07:45:58.310 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:45:58.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128508 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:45:58.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:45:58.366+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.366902+0000) 2019-10-13T07:45:58.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:45:58.705+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.705652+0000) 2019-10-13T07:45:58.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:45:58.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:58.705651+0000) 2019-10-13T07:46:01.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:01.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:01.958914+0000) 2019-10-13T07:46:02.260 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:02.256+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:02.257257+0000) 2019-10-13T07:46:02.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:02.819+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:02.820184+0000) 2019-10-13T07:46:02.824 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:02.820+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:02.821317+0000) 2019-10-13T07:46:03.312 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:03.308+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.309183+0000) 2019-10-13T07:46:03.313 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:03.309+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.310256+0000) 2019-10-13T07:46:03.313 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:03.310+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.310417+0000) 2019-10-13T07:46:03.313 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:03.310+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.310818+0000) 2019-10-13T07:46:03.314 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:03.310+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.310934+0000) 2019-10-13T07:46:03.314 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:03.310+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.311257+0000) 2019-10-13T07:46:03.314 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:03.311+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.311436+0000) 2019-10-13T07:46:03.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:03.311+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.312221+0000) 2019-10-13T07:46:03.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:03.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:07.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:07.818+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:07.818960+0000) 2019-10-13T07:46:08.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:08.316+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.316946+0000) 2019-10-13T07:46:08.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:08.317+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.317997+0000) 2019-10-13T07:46:08.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:08.317+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.318160+0000) 2019-10-13T07:46:08.321 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:08.318+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.318668+0000) 2019-10-13T07:46:08.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:08.318+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.318985+0000) 2019-10-13T07:46:08.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:08.318+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.319031+0000) 2019-10-13T07:46:08.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:08.318+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.319192+0000) 2019-10-13T07:46:08.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:08.319+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.319923+0000) 2019-10-13T07:46:08.325 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:08.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128567 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:08.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:08.366+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.367073+0000) 2019-10-13T07:46:08.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:08.705+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.705829+0000) 2019-10-13T07:46:08.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:08.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:08.705833+0000) 2019-10-13T07:46:11.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:11.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:11.959114+0000) 2019-10-13T07:46:12.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:12.259+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:12.259777+0000) 2019-10-13T07:46:12.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:12.819+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:12.820383+0000) 2019-10-13T07:46:12.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:12.821+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:12.821507+0000) 2019-10-13T07:46:13.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:13.323+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.325101+0000) 2019-10-13T07:46:13.328 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:13.323+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.325306+0000) 2019-10-13T07:46:13.328 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:13.323+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.324590+0000) 2019-10-13T07:46:13.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:13.324+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.325676+0000) 2019-10-13T07:46:13.329 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:13.324+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.326146+0000) 2019-10-13T07:46:13.329 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:13.324+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.326253+0000) 2019-10-13T07:46:13.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:13.325+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.326461+0000) 2019-10-13T07:46:13.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:13.326+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.326987+0000) 2019-10-13T07:46:13.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:13.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128600 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:17.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:17.818+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:17.819135+0000) 2019-10-13T07:46:18.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:18.330+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.331646+0000) 2019-10-13T07:46:18.335 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:18.330+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.330922+0000) 2019-10-13T07:46:18.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:18.330+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.332094+0000) 2019-10-13T07:46:18.335 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:18.330+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.332163+0000) 2019-10-13T07:46:18.335 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:18.331+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.332523+0000) 2019-10-13T07:46:18.335 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:18.331+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.332630+0000) 2019-10-13T07:46:18.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:18.331+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.332725+0000) 2019-10-13T07:46:18.339 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:18.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128628 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:18.339 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:18.332+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.333485+0000) 2019-10-13T07:46:18.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:18.365+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.367304+0000) 2019-10-13T07:46:18.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:18.705+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.705978+0000) 2019-10-13T07:46:18.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:18.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:18.705998+0000) 2019-10-13T07:46:21.911 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:46:21.911 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:21.915 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:46:21.916 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:21.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:21.957+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:21.959253+0000) 2019-10-13T07:46:22.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:22.261+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:22.262161+0000) 2019-10-13T07:46:22.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:22.819+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:22.820590+0000) 2019-10-13T07:46:22.824 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:22.821+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:22.821653+0000) 2019-10-13T07:46:23.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:23.337+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.338498+0000) 2019-10-13T07:46:23.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:23.337+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.339026+0000) 2019-10-13T07:46:23.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:23.337+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.339228+0000) 2019-10-13T07:46:23.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:23.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.338713+0000) 2019-10-13T07:46:23.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:23.338+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.339404+0000) 2019-10-13T07:46:23.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:23.338+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.339487+0000) 2019-10-13T07:46:23.343 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:23.338+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.339803+0000) 2019-10-13T07:46:23.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:23.339+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.340005+0000) 2019-10-13T07:46:23.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:23.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:27.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:27.818+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:27.819329+0000) 2019-10-13T07:46:28.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:28.344+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.346113+0000) 2019-10-13T07:46:28.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:28.344+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.346194+0000) 2019-10-13T07:46:28.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:28.345+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.346653+0000) 2019-10-13T07:46:28.350 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:28.345+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.346888+0000) 2019-10-13T07:46:28.351 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:28.345+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.347054+0000) 2019-10-13T07:46:28.351 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:28.345+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.347168+0000) 2019-10-13T07:46:28.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:28.344+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.344953+0000) 2019-10-13T07:46:28.352 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:28.347+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.348550+0000) 2019-10-13T07:46:28.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:28.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128687 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:28.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:28.366+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.367483+0000) 2019-10-13T07:46:28.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:28.705+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.706140+0000) 2019-10-13T07:46:28.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:28.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:28.706161+0000) 2019-10-13T07:46:31.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:31.958+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:31.959386+0000) 2019-10-13T07:46:32.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:32.264+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:32.264631+0000) 2019-10-13T07:46:32.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:32.820+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:32.820773+0000) 2019-10-13T07:46:32.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:32.821+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:32.821837+0000) 2019-10-13T07:46:33.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:33.351+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.352849+0000) 2019-10-13T07:46:33.356 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:33.351+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.352082+0000) 2019-10-13T07:46:33.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:33.352+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.353421+0000) 2019-10-13T07:46:33.356 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:33.352+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.353668+0000) 2019-10-13T07:46:33.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:33.352+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.353770+0000) 2019-10-13T07:46:33.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:33.352+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.353840+0000) 2019-10-13T07:46:33.357 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:33.352+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.354028+0000) 2019-10-13T07:46:33.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:33.353+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.354398+0000) 2019-10-13T07:46:33.359 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:33.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128720 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:37.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:37.819+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:37.819519+0000) 2019-10-13T07:46:38.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:38.358+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.359453+0000) 2019-10-13T07:46:38.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:38.358+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.359625+0000) 2019-10-13T07:46:38.363 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:38.357+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.358920+0000) 2019-10-13T07:46:38.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:38.358+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.360213+0000) 2019-10-13T07:46:38.363 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:38.358+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.360295+0000) 2019-10-13T07:46:38.364 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:38.359+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.360488+0000) 2019-10-13T07:46:38.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:38.359+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.360809+0000) 2019-10-13T07:46:38.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:38.360+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.361872+0000) 2019-10-13T07:46:38.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:38.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128748 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:38.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:38.366+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.367948+0000) 2019-10-13T07:46:38.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:38.705+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.706325+0000) 2019-10-13T07:46:38.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:38.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:38.706331+0000) 2019-10-13T07:46:41.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:41.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:41.959617+0000) 2019-10-13T07:46:42.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:42.266+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:42.267062+0000) 2019-10-13T07:46:42.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:42.820+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:42.820979+0000) 2019-10-13T07:46:42.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:42.821+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:42.822016+0000) 2019-10-13T07:46:43.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:43.366+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.366599+0000) 2019-10-13T07:46:43.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:43.366+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.367137+0000) 2019-10-13T07:46:43.370 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:43.366+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.367160+0000) 2019-10-13T07:46:43.370 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:43.366+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.367147+0000) 2019-10-13T07:46:43.370 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:43.367+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.367544+0000) 2019-10-13T07:46:43.371 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:43.365+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.366866+0000) 2019-10-13T07:46:43.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:43.367+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.367856+0000) 2019-10-13T07:46:43.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:43.366+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.368531+0000) 2019-10-13T07:46:43.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:43.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128779 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:47.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:47.819+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:47.819710+0000) 2019-10-13T07:46:48.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:48.368+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.368604+0000) 2019-10-13T07:46:48.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:48.370+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.372069+0000) 2019-10-13T07:46:48.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:48.372+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.373198+0000) 2019-10-13T07:46:48.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:48.373+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.373570+0000) 2019-10-13T07:46:48.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:48.373+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.373954+0000) 2019-10-13T07:46:48.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:48.373+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.374222+0000) 2019-10-13T07:46:48.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:48.373+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.374312+0000) 2019-10-13T07:46:48.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:48.374+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.374463+0000) 2019-10-13T07:46:48.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:48.372+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.374544+0000) 2019-10-13T07:46:48.381 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:48.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128807 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:48.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:48.706+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.706505+0000) 2019-10-13T07:46:48.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:48.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:48.706518+0000) 2019-10-13T07:46:51.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:51.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:51.959831+0000) 2019-10-13T07:46:51.999 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:46:52.000 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:52.009 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:46:52.009 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:52.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:52.268+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:52.269337+0000) 2019-10-13T07:46:52.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:52.821+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:52.821183+0000) 2019-10-13T07:46:52.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:52.821+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:52.822217+0000) 2019-10-13T07:46:53.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:53.378+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.379519+0000) 2019-10-13T07:46:53.383 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:53.380+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.380608+0000) 2019-10-13T07:46:53.383 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:53.380+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.380897+0000) 2019-10-13T07:46:53.384 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:53.380+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.381081+0000) 2019-10-13T07:46:53.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:53.381+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.381431+0000) 2019-10-13T07:46:53.384 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:53.381+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.381670+0000) 2019-10-13T07:46:53.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:53.381+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.381802+0000) 2019-10-13T07:46:53.386 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:53.382+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.382980+0000) 2019-10-13T07:46:53.388 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:53.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128840 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:57.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:57.819+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:57.819901+0000) 2019-10-13T07:46:58.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:58.367+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.368831+0000) 2019-10-13T07:46:58.390 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:46:58.386+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.387980+0000) 2019-10-13T07:46:58.390 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:58.386+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.388216+0000) 2019-10-13T07:46:58.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:46:58.386+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.387688+0000) 2019-10-13T07:46:58.391 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:46:58.386+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.388375+0000) 2019-10-13T07:46:58.391 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:46:58.387+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.388580+0000) 2019-10-13T07:46:58.392 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:58.387+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.388707+0000) 2019-10-13T07:46:58.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:46:58.387+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.389233+0000) 2019-10-13T07:46:58.393 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:46:58.389+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.390335+0000) 2019-10-13T07:46:58.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:46:58.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128868 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:46:58.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:46:58.705+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.706658+0000) 2019-10-13T07:46:58.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:46:58.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:58.706667+0000) 2019-10-13T07:47:01.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:01.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:01.960019+0000) 2019-10-13T07:47:02.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:02.271+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:02.271770+0000) 2019-10-13T07:47:02.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:02.820+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:02.821362+0000) 2019-10-13T07:47:02.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:02.821+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:02.822398+0000) 2019-10-13T07:47:03.399 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:03.393+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.394405+0000) 2019-10-13T07:47:03.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:03.393+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.394564+0000) 2019-10-13T07:47:03.400 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:03.393+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.394503+0000) 2019-10-13T07:47:03.401 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:03.393+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.394865+0000) 2019-10-13T07:47:03.402 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:03.395+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.396206+0000) 2019-10-13T07:47:03.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:03.393+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.395130+0000) 2019-10-13T07:47:03.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:03.393+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.395230+0000) 2019-10-13T07:47:03.403 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:03.394+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.395538+0000) 2019-10-13T07:47:03.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:03.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128899 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:07.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:07.819+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:07.820111+0000) 2019-10-13T07:47:08.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:08.368+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.369027+0000) 2019-10-13T07:47:08.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:08.399+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.401283+0000) 2019-10-13T07:47:08.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:08.400+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.401849+0000) 2019-10-13T07:47:08.404 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:08.400+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.401920+0000) 2019-10-13T07:47:08.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:08.400+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.401213+0000) 2019-10-13T07:47:08.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:08.401+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.402543+0000) 2019-10-13T07:47:08.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:08.401+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.402391+0000) 2019-10-13T07:47:08.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:08.400+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.402262+0000) 2019-10-13T07:47:08.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:08.403+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.403889+0000) 2019-10-13T07:47:08.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:08.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:08.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:08.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.706750+0000) 2019-10-13T07:47:08.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:08.705+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:08.706767+0000) 2019-10-13T07:47:11.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:11.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:11.960211+0000) 2019-10-13T07:47:12.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:12.273+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:12.274155+0000) 2019-10-13T07:47:12.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:12.820+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:12.821559+0000) 2019-10-13T07:47:12.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:12.822+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:12.822629+0000) 2019-10-13T07:47:13.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:13.407+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.408768+0000) 2019-10-13T07:47:13.411 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:13.407+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.409107+0000) 2019-10-13T07:47:13.412 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:13.408+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.409565+0000) 2019-10-13T07:47:13.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:13.408+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.409698+0000) 2019-10-13T07:47:13.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:13.408+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.410270+0000) 2019-10-13T07:47:13.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:13.409+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.410500+0000) 2019-10-13T07:47:13.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:13.408+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.409281+0000) 2019-10-13T07:47:13.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:13.409+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.410655+0000) 2019-10-13T07:47:13.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:13.413+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128960 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:17.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:17.819+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:17.820323+0000) 2019-10-13T07:47:18.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:18.368+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.369191+0000) 2019-10-13T07:47:18.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:18.414+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.416268+0000) 2019-10-13T07:47:18.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:18.415+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.416661+0000) 2019-10-13T07:47:18.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:18.415+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.416684+0000) 2019-10-13T07:47:18.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:18.415+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.417037+0000) 2019-10-13T07:47:18.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:18.415+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.416635+0000) 2019-10-13T07:47:18.420 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:18.416+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.417731+0000) 2019-10-13T07:47:18.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:18.416+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.417836+0000) 2019-10-13T07:47:18.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:18.417+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.418044+0000) 2019-10-13T07:47:18.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:18.420+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 128988 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:18.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:18.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.706926+0000) 2019-10-13T07:47:18.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:18.706+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:18.706927+0000) 2019-10-13T07:47:21.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:21.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:21.960374+0000) 2019-10-13T07:47:22.107 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:47:22.107 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:22.112 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:47:22.112 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:22.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:22.276+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:22.276490+0000) 2019-10-13T07:47:22.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:22.821+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:22.821728+0000) 2019-10-13T07:47:22.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:22.822+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:22.822806+0000) 2019-10-13T07:47:23.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:23.422+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.423554+0000) 2019-10-13T07:47:23.426 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:23.422+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.423772+0000) 2019-10-13T07:47:23.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:23.422+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.423932+0000) 2019-10-13T07:47:23.427 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:23.422+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.424128+0000) 2019-10-13T07:47:23.427 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:23.422+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.423410+0000) 2019-10-13T07:47:23.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:23.422+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.424383+0000) 2019-10-13T07:47:23.428 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:23.423+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.424713+0000) 2019-10-13T07:47:23.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:23.425+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.426055+0000) 2019-10-13T07:47:23.430 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:23.426+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:27.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:27.820+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:27.820569+0000) 2019-10-13T07:47:28.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:28.369+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.369431+0000) 2019-10-13T07:47:28.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:28.429+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.429545+0000) 2019-10-13T07:47:28.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:28.429+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.429743+0000) 2019-10-13T07:47:28.433 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:28.428+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.429009+0000) 2019-10-13T07:47:28.433 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:28.429+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.430191+0000) 2019-10-13T07:47:28.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:28.429+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.430367+0000) 2019-10-13T07:47:28.452 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:28.430+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.430642+0000) 2019-10-13T07:47:28.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:28.430+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.430893+0000) 2019-10-13T07:47:28.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:28.431+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.432165+0000) 2019-10-13T07:47:28.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:28.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:28.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:28.706+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.707098+0000) 2019-10-13T07:47:28.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:28.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:28.707127+0000) 2019-10-13T07:47:31.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:31.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:31.960516+0000) 2019-10-13T07:47:32.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:32.277+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:32.278749+0000) 2019-10-13T07:47:32.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:32.821+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:32.821924+0000) 2019-10-13T07:47:32.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:32.821+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:32.822971+0000) 2019-10-13T07:47:33.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:33.434+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.435788+0000) 2019-10-13T07:47:33.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:33.437+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.438863+0000) 2019-10-13T07:47:33.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:33.435+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.436253+0000) 2019-10-13T07:47:33.445 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:33.436+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.436466+0000) 2019-10-13T07:47:33.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:33.436+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.437055+0000) 2019-10-13T07:47:33.446 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:33.436+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.436874+0000) 2019-10-13T07:47:33.446 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:33.436+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.437144+0000) 2019-10-13T07:47:33.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:33.436+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.437359+0000) 2019-10-13T07:47:33.446 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:33.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:37.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:37.820+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:37.820773+0000) 2019-10-13T07:47:38.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:38.369+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.369644+0000) 2019-10-13T07:47:38.446 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:38.441+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.443172+0000) 2019-10-13T07:47:38.447 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:38.443+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.444375+0000) 2019-10-13T07:47:38.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:38.443+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.444301+0000) 2019-10-13T07:47:38.447 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:38.444+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.444879+0000) 2019-10-13T07:47:38.448 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:38.444+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.444988+0000) 2019-10-13T07:47:38.448 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:38.444+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.445179+0000) 2019-10-13T07:47:38.448 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:38.444+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.445360+0000) 2019-10-13T07:47:38.448 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:38.444+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.445760+0000) 2019-10-13T07:47:38.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:38.448+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:38.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:38.707+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.707263+0000) 2019-10-13T07:47:38.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:38.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:38.707331+0000) 2019-10-13T07:47:41.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:41.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:41.960712+0000) 2019-10-13T07:47:42.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:42.280+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:42.281081+0000) 2019-10-13T07:47:42.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:42.822+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:42.822078+0000) 2019-10-13T07:47:42.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:42.822+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:42.823151+0000) 2019-10-13T07:47:43.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:43.450+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.451158+0000) 2019-10-13T07:47:43.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:43.450+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.451346+0000) 2019-10-13T07:47:43.455 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:43.451+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.451500+0000) 2019-10-13T07:47:43.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:43.451+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.451830+0000) 2019-10-13T07:47:43.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:43.451+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.451932+0000) 2019-10-13T07:47:43.455 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:43.451+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.452363+0000) 2019-10-13T07:47:43.456 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:43.449+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.451303+0000) 2019-10-13T07:47:43.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:43.451+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.453333+0000) 2019-10-13T07:47:43.458 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:43.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:47.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:47.819+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:47.820941+0000) 2019-10-13T07:47:48.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:48.368+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.369842+0000) 2019-10-13T07:47:48.460 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:48.456+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.457736+0000) 2019-10-13T07:47:48.460 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:48.456+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.458023+0000) 2019-10-13T07:47:48.461 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:48.456+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.457006+0000) 2019-10-13T07:47:48.462 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:48.457+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.458560+0000) 2019-10-13T07:47:48.462 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:48.457+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.458653+0000) 2019-10-13T07:47:48.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:48.457+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.458863+0000) 2019-10-13T07:47:48.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:48.457+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.458953+0000) 2019-10-13T07:47:48.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:48.459+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.460603+0000) 2019-10-13T07:47:48.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:48.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:48.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:48.706+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.707452+0000) 2019-10-13T07:47:48.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:48.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:48.707498+0000) 2019-10-13T07:47:51.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:51.959+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:51.960808+0000) 2019-10-13T07:47:52.211 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:47:52.211 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:52.215 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:47:52.215 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:52.286 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:52.282+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:52.283460+0000) 2019-10-13T07:47:52.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:52.820+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:52.822271+0000) 2019-10-13T07:47:52.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:52.822+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:52.823331+0000) 2019-10-13T07:47:53.466 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:53.463+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.463774+0000) 2019-10-13T07:47:53.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:53.463+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.464834+0000) 2019-10-13T07:47:53.467 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:53.463+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.464962+0000) 2019-10-13T07:47:53.468 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:53.463+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.465377+0000) 2019-10-13T07:47:53.468 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:53.464+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.465679+0000) 2019-10-13T07:47:53.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:53.464+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.465842+0000) 2019-10-13T07:47:53.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:53.464+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.465746+0000) 2019-10-13T07:47:53.469 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:53.465+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.466259+0000) 2019-10-13T07:47:53.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:53.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:57.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:57.819+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:57.821100+0000) 2019-10-13T07:47:58.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:58.368+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.370030+0000) 2019-10-13T07:47:58.473 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:47:58.469+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.470468+0000) 2019-10-13T07:47:58.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:47:58.470+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.471530+0000) 2019-10-13T07:47:58.474 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:58.470+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.471800+0000) 2019-10-13T07:47:58.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:47:58.470+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.471949+0000) 2019-10-13T07:47:58.475 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:58.470+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.472077+0000) 2019-10-13T07:47:58.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:47:58.470+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.472303+0000) 2019-10-13T07:47:58.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:47:58.471+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.472448+0000) 2019-10-13T07:47:58.480 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:47:58.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129228 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:47:58.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:47:58.473+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.474233+0000) 2019-10-13T07:47:58.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:47:58.706+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.707619+0000) 2019-10-13T07:47:58.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:47:58.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:58.707649+0000) 2019-10-13T07:48:01.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:01.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:01.960962+0000) 2019-10-13T07:48:02.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:02.285+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:02.285782+0000) 2019-10-13T07:48:02.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:02.821+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:02.822452+0000) 2019-10-13T07:48:02.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:02.823+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:02.823520+0000) 2019-10-13T07:48:03.481 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:03.477+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.478392+0000) 2019-10-13T07:48:03.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:03.478+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.479442+0000) 2019-10-13T07:48:03.483 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:03.478+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.479579+0000) 2019-10-13T07:48:03.483 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:03.478+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.479828+0000) 2019-10-13T07:48:03.484 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:03.478+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.480260+0000) 2019-10-13T07:48:03.484 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:03.479+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.480494+0000) 2019-10-13T07:48:03.484 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:03.478+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.480145+0000) 2019-10-13T07:48:03.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:03.479+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.480311+0000) 2019-10-13T07:48:03.486 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:03.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129259 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:07.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:07.820+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:07.821280+0000) 2019-10-13T07:48:08.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:08.369+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.370183+0000) 2019-10-13T07:48:08.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:08.484+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.486301+0000) 2019-10-13T07:48:08.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:08.485+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.486596+0000) 2019-10-13T07:48:08.490 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:08.484+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.485582+0000) 2019-10-13T07:48:08.490 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:08.485+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.487134+0000) 2019-10-13T07:48:08.490 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:08.485+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.487257+0000) 2019-10-13T07:48:08.490 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:08.486+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.487429+0000) 2019-10-13T07:48:08.490 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:08.486+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.487522+0000) 2019-10-13T07:48:08.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:08.487+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.488154+0000) 2019-10-13T07:48:08.494 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:08.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129287 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:08.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:08.707+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.707766+0000) 2019-10-13T07:48:08.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:08.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:08.707818+0000) 2019-10-13T07:48:11.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:11.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:11.961152+0000) 2019-10-13T07:48:12.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:12.287+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:12.288151+0000) 2019-10-13T07:48:12.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:12.822+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:12.822680+0000) 2019-10-13T07:48:12.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:12.823+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:12.823727+0000) 2019-10-13T07:48:13.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:13.493+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.493779+0000) 2019-10-13T07:48:13.499 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:13.493+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.494114+0000) 2019-10-13T07:48:13.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:13.494+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.494439+0000) 2019-10-13T07:48:13.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:13.494+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.494686+0000) 2019-10-13T07:48:13.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:13.494+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.494733+0000) 2019-10-13T07:48:13.500 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:13.494+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.494996+0000) 2019-10-13T07:48:13.500 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:13.491+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.492599+0000) 2019-10-13T07:48:13.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:13.494+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.495539+0000) 2019-10-13T07:48:13.501 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:13.499+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:17.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:17.821+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:17.821497+0000) 2019-10-13T07:48:18.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:18.369+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.370370+0000) 2019-10-13T07:48:18.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:18.500+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.501284+0000) 2019-10-13T07:48:18.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:18.501+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.501614+0000) 2019-10-13T07:48:18.506 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:18.501+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.502058+0000) 2019-10-13T07:48:18.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:18.501+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.502255+0000) 2019-10-13T07:48:18.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:18.501+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.502309+0000) 2019-10-13T07:48:18.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:18.502+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.502450+0000) 2019-10-13T07:48:18.507 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:18.499+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.500149+0000) 2019-10-13T07:48:18.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:18.501+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.502715+0000) 2019-10-13T07:48:18.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:18.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129348 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:18.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:18.707+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.707999+0000) 2019-10-13T07:48:18.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:18.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:18.707996+0000) 2019-10-13T07:48:21.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:21.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:21.961340+0000) 2019-10-13T07:48:22.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:22.290+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:22.290577+0000) 2019-10-13T07:48:22.308 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:48:22.308 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:22.312 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:48:22.312 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:22.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:22.823+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:22.823906+0000) 2019-10-13T07:48:22.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:22.822+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:22.822812+0000) 2019-10-13T07:48:23.510 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:23.507+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.507963+0000) 2019-10-13T07:48:23.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:23.507+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.508265+0000) 2019-10-13T07:48:23.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:23.508+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.509018+0000) 2019-10-13T07:48:23.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:23.508+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.509119+0000) 2019-10-13T07:48:23.511 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:23.509+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.509426+0000) 2019-10-13T07:48:23.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:23.509+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.509784+0000) 2019-10-13T07:48:23.512 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:23.508+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.509004+0000) 2019-10-13T07:48:23.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:23.509+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.510553+0000) 2019-10-13T07:48:23.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:23.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129380 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:27.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:27.821+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:27.821734+0000) 2019-10-13T07:48:28.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:28.370+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.370538+0000) 2019-10-13T07:48:28.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:28.513+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.513663+0000) 2019-10-13T07:48:28.517 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:28.511+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.513014+0000) 2019-10-13T07:48:28.517 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:28.513+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.514259+0000) 2019-10-13T07:48:28.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:28.513+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.514370+0000) 2019-10-13T07:48:28.517 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:28.514+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.514697+0000) 2019-10-13T07:48:28.518 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:28.514+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.514797+0000) 2019-10-13T07:48:28.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:28.514+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.515023+0000) 2019-10-13T07:48:28.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:28.513+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.515671+0000) 2019-10-13T07:48:28.521 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:28.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:28.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:28.708+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.708178+0000) 2019-10-13T07:48:28.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:28.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:28.708177+0000) 2019-10-13T07:48:31.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:31.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:31.961555+0000) 2019-10-13T07:48:32.296 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:32.292+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:32.293198+0000) 2019-10-13T07:48:32.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:32.821+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:32.822975+0000) 2019-10-13T07:48:32.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:32.823+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:32.824077+0000) 2019-10-13T07:48:33.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:33.519+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.520546+0000) 2019-10-13T07:48:33.523 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:33.518+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.519842+0000) 2019-10-13T07:48:33.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:33.519+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.521046+0000) 2019-10-13T07:48:33.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:33.519+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.521201+0000) 2019-10-13T07:48:33.524 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:33.520+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.521628+0000) 2019-10-13T07:48:33.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:33.520+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.521724+0000) 2019-10-13T07:48:33.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:33.520+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.521976+0000) 2019-10-13T07:48:33.525 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:33.520+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.522527+0000) 2019-10-13T07:48:33.527 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:33.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:37.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:37.821+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:37.821924+0000) 2019-10-13T07:48:38.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:38.369+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.370746+0000) 2019-10-13T07:48:38.529 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:38.525+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.525924+0000) 2019-10-13T07:48:38.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:38.525+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.526988+0000) 2019-10-13T07:48:38.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:38.525+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.527128+0000) 2019-10-13T07:48:38.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:38.526+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.527417+0000) 2019-10-13T07:48:38.530 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:38.526+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.527474+0000) 2019-10-13T07:48:38.530 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:38.526+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.527878+0000) 2019-10-13T07:48:38.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:38.526+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.528044+0000) 2019-10-13T07:48:38.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:38.528+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.528920+0000) 2019-10-13T07:48:38.534 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:38.531+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129468 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:38.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:38.707+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.708346+0000) 2019-10-13T07:48:38.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:38.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:38.708342+0000) 2019-10-13T07:48:41.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:41.960+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:41.961749+0000) 2019-10-13T07:48:42.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:42.294+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:42.295506+0000) 2019-10-13T07:48:42.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:42.822+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:42.823161+0000) 2019-10-13T07:48:42.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:42.823+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:42.824227+0000) 2019-10-13T07:48:43.536 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:43.532+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.534271+0000) 2019-10-13T07:48:43.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:43.533+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.534616+0000) 2019-10-13T07:48:43.537 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:43.533+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.534429+0000) 2019-10-13T07:48:43.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:43.533+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.534931+0000) 2019-10-13T07:48:43.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:43.534+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.535566+0000) 2019-10-13T07:48:43.538 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:43.534+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.535693+0000) 2019-10-13T07:48:43.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:43.534+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.535683+0000) 2019-10-13T07:48:43.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:43.534+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.535999+0000) 2019-10-13T07:48:43.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:43.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:47.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:47.821+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:47.822103+0000) 2019-10-13T07:48:48.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:48.370+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.370909+0000) 2019-10-13T07:48:48.544 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:48.540+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.541181+0000) 2019-10-13T07:48:48.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:48.540+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.542347+0000) 2019-10-13T07:48:48.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:48.541+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.542428+0000) 2019-10-13T07:48:48.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:48.541+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.542702+0000) 2019-10-13T07:48:48.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:48.541+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.542714+0000) 2019-10-13T07:48:48.546 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:48.541+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.543064+0000) 2019-10-13T07:48:48.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:48.542+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.543465+0000) 2019-10-13T07:48:48.547 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:48.543+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.544172+0000) 2019-10-13T07:48:48.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:48.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129528 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:48.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:48.707+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.708510+0000) 2019-10-13T07:48:48.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:48.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:48.708510+0000) 2019-10-13T07:48:51.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:51.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:51.961935+0000) 2019-10-13T07:48:52.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:52.297+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:52.297985+0000) 2019-10-13T07:48:52.456 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:48:52.456 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:52.461 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:48:52.461 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:52.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:52.821+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:52.823377+0000) 2019-10-13T07:48:52.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:52.823+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:52.824426+0000) 2019-10-13T07:48:53.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:53.547+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.549337+0000) 2019-10-13T07:48:53.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:53.548+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.548796+0000) 2019-10-13T07:48:53.552 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:53.548+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.549771+0000) 2019-10-13T07:48:53.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:53.548+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.549872+0000) 2019-10-13T07:48:53.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:53.548+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.550345+0000) 2019-10-13T07:48:53.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:53.549+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.550684+0000) 2019-10-13T07:48:53.553 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:53.549+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.550845+0000) 2019-10-13T07:48:53.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:53.550+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.550741+0000) 2019-10-13T07:48:53.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:53.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:57.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:57.821+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:57.822267+0000) 2019-10-13T07:48:58.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:58.370+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.371052+0000) 2019-10-13T07:48:58.559 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:48:58.555+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.557064+0000) 2019-10-13T07:48:58.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:58.555+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.557240+0000) 2019-10-13T07:48:58.560 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:48:58.556+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.557600+0000) 2019-10-13T07:48:58.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:48:58.555+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.556512+0000) 2019-10-13T07:48:58.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:48:58.556+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.558059+0000) 2019-10-13T07:48:58.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:58.556+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.558221+0000) 2019-10-13T07:48:58.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:48:58.557+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.558432+0000) 2019-10-13T07:48:58.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:48:58.558+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.559075+0000) 2019-10-13T07:48:58.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:48:58.561+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129590 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:48:58.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:48:58.708+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.708690+0000) 2019-10-13T07:48:58.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:48:58.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:58.708704+0000) 2019-10-13T07:49:01.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:01.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:01.962093+0000) 2019-10-13T07:49:02.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:02.300+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:02.300450+0000) 2019-10-13T07:49:02.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:02.823+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:02.823590+0000) 2019-10-13T07:49:02.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:02.824+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:02.824617+0000) 2019-10-13T07:49:03.566 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:03.562+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.563470+0000) 2019-10-13T07:49:03.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:03.564+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.564616+0000) 2019-10-13T07:49:03.568 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:03.564+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.565149+0000) 2019-10-13T07:49:03.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:03.564+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.565319+0000) 2019-10-13T07:49:03.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:03.565+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.565521+0000) 2019-10-13T07:49:03.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:03.565+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.565685+0000) 2019-10-13T07:49:03.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:03.565+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.565919+0000) 2019-10-13T07:49:03.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:03.565+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.565995+0000) 2019-10-13T07:49:03.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:03.569+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129622 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:07.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:07.822+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:07.822471+0000) 2019-10-13T07:49:08.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:08.371+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.371215+0000) 2019-10-13T07:49:08.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:08.570+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.570808+0000) 2019-10-13T07:49:08.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:08.571+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.572235+0000) 2019-10-13T07:49:08.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:08.571+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.572333+0000) 2019-10-13T07:49:08.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:08.572+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.572537+0000) 2019-10-13T07:49:08.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:08.572+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.572750+0000) 2019-10-13T07:49:08.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:08.572+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.572955+0000) 2019-10-13T07:49:08.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:08.572+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.573242+0000) 2019-10-13T07:49:08.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:08.572+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.573325+0000) 2019-10-13T07:49:08.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:08.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:08.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:08.708+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.708909+0000) 2019-10-13T07:49:08.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:08.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:08.708908+0000) 2019-10-13T07:49:11.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:11.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:11.962266+0000) 2019-10-13T07:49:12.306 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:12.302+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:12.302992+0000) 2019-10-13T07:49:12.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:12.823+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:12.823770+0000) 2019-10-13T07:49:12.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:12.824+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:12.824769+0000) 2019-10-13T07:49:13.581 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:13.578+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.578859+0000) 2019-10-13T07:49:13.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:13.579+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.579890+0000) 2019-10-13T07:49:13.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:13.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.580020+0000) 2019-10-13T07:49:13.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:13.579+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.580359+0000) 2019-10-13T07:49:13.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:13.580+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.580716+0000) 2019-10-13T07:49:13.584 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:13.580+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.580958+0000) 2019-10-13T07:49:13.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:13.580+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.581138+0000) 2019-10-13T07:49:13.584 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:13.580+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.581701+0000) 2019-10-13T07:49:13.587 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:13.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:17.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:17.821+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:17.822676+0000) 2019-10-13T07:49:18.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:18.370+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.371375+0000) 2019-10-13T07:49:18.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:18.585+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.587264+0000) 2019-10-13T07:49:18.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:18.586+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.587607+0000) 2019-10-13T07:49:18.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:18.586+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.587752+0000) 2019-10-13T07:49:18.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:18.585+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.587215+0000) 2019-10-13T07:49:18.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:18.586+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.588284+0000) 2019-10-13T07:49:18.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:18.587+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.588480+0000) 2019-10-13T07:49:18.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:18.586+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.588103+0000) 2019-10-13T07:49:18.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:18.588+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.589937+0000) 2019-10-13T07:49:18.594 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:18.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129712 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:18.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:18.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.709078+0000) 2019-10-13T07:49:18.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:18.708+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:18.709078+0000) 2019-10-13T07:49:21.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:21.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:21.962468+0000) 2019-10-13T07:49:22.316 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:22.304+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:22.305285+0000) 2019-10-13T07:49:22.544 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:49:22.545 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:22.572 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:49:22.572 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:22.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:22.823+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:22.823934+0000) 2019-10-13T07:49:22.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:22.823+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:22.824897+0000) 2019-10-13T07:49:23.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:23.593+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.594746+0000) 2019-10-13T07:49:23.598 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:23.593+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.594962+0000) 2019-10-13T07:49:23.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:23.593+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.595155+0000) 2019-10-13T07:49:23.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:23.593+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.595237+0000) 2019-10-13T07:49:23.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:23.593+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.595361+0000) 2019-10-13T07:49:23.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:23.594+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.595581+0000) 2019-10-13T07:49:23.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:23.593+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.594958+0000) 2019-10-13T07:49:23.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:23.595+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.597527+0000) 2019-10-13T07:49:23.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:23.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:27.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:27.822+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:27.822861+0000) 2019-10-13T07:49:28.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:28.370+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.371602+0000) 2019-10-13T07:49:28.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:28.599+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.601319+0000) 2019-10-13T07:49:28.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:28.599+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.600842+0000) 2019-10-13T07:49:28.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:28.600+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.602074+0000) 2019-10-13T07:49:28.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:28.600+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.602283+0000) 2019-10-13T07:49:28.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:28.600+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.602169+0000) 2019-10-13T07:49:28.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:28.601+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.602534+0000) 2019-10-13T07:49:28.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:28.601+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.602636+0000) 2019-10-13T07:49:28.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:28.601+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.602821+0000) 2019-10-13T07:49:28.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:28.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129772 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:28.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:28.708+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.709242+0000) 2019-10-13T07:49:28.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:28.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:28.709240+0000) 2019-10-13T07:49:31.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:31.961+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:31.962659+0000) 2019-10-13T07:49:32.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:32.306+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:32.307557+0000) 2019-10-13T07:49:32.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:32.823+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:32.824148+0000) 2019-10-13T07:49:32.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:32.824+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:32.825105+0000) 2019-10-13T07:49:33.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:33.606+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.607884+0000) 2019-10-13T07:49:33.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:33.606+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.608066+0000) 2019-10-13T07:49:33.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:33.606+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.607993+0000) 2019-10-13T07:49:33.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:33.606+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.607402+0000) 2019-10-13T07:49:33.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:33.607+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.608495+0000) 2019-10-13T07:49:33.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:33.607+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.609158+0000) 2019-10-13T07:49:33.612 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:33.607+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.609300+0000) 2019-10-13T07:49:33.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:33.609+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.610220+0000) 2019-10-13T07:49:33.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:33.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129807 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:37.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:37.822+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:37.823027+0000) 2019-10-13T07:49:38.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:38.371+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.371831+0000) 2019-10-13T07:49:38.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:38.613+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.615014+0000) 2019-10-13T07:49:38.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:38.613+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.615244+0000) 2019-10-13T07:49:38.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:38.614+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.615069+0000) 2019-10-13T07:49:38.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:38.614+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.615695+0000) 2019-10-13T07:49:38.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:38.614+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.616027+0000) 2019-10-13T07:49:38.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:38.614+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.616149+0000) 2019-10-13T07:49:38.619 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:38.614+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.616368+0000) 2019-10-13T07:49:38.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:38.616+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.616834+0000) 2019-10-13T07:49:38.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:38.619+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129834 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:38.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:38.708+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.709427+0000) 2019-10-13T07:49:38.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:38.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:38.709426+0000) 2019-10-13T07:49:41.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:41.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:41.962847+0000) 2019-10-13T07:49:42.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:42.308+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:42.309710+0000) 2019-10-13T07:49:42.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:42.823+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:42.824336+0000) 2019-10-13T07:49:42.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:42.824+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:42.825320+0000) 2019-10-13T07:49:43.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:43.620+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.622333+0000) 2019-10-13T07:49:43.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:43.621+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.623305+0000) 2019-10-13T07:49:43.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:43.622+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.623494+0000) 2019-10-13T07:49:43.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:43.622+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.623724+0000) 2019-10-13T07:49:43.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:43.622+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.624033+0000) 2019-10-13T07:49:43.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:43.622+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.623336+0000) 2019-10-13T07:49:43.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:43.623+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.624645+0000) 2019-10-13T07:49:43.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:43.623+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.624353+0000) 2019-10-13T07:49:43.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:43.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:47.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:47.822+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:47.823206+0000) 2019-10-13T07:49:48.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:48.371+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.372010+0000) 2019-10-13T07:49:48.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:48.629+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.630146+0000) 2019-10-13T07:49:48.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:48.629+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.630304+0000) 2019-10-13T07:49:48.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:48.630+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.630457+0000) 2019-10-13T07:49:48.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:48.630+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.630706+0000) 2019-10-13T07:49:48.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:48.630+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.630871+0000) 2019-10-13T07:49:48.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:48.629+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.630314+0000) 2019-10-13T07:49:48.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:48.631+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.631391+0000) 2019-10-13T07:49:48.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:48.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129894 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:48.645 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:48.631+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.632304+0000) 2019-10-13T07:49:48.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:48.709+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.709581+0000) 2019-10-13T07:49:48.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:48.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:48.709585+0000) 2019-10-13T07:49:51.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:51.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:51.963030+0000) 2019-10-13T07:49:52.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:52.311+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:52.311981+0000) 2019-10-13T07:49:52.689 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:49:52.689 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:52.694 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:49:52.694 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:52.826 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:52.824+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:52.824546+0000) 2019-10-13T07:49:52.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:52.825+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:52.825484+0000) 2019-10-13T07:49:53.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:53.636+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.637289+0000) 2019-10-13T07:49:53.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:53.636+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.636962+0000) 2019-10-13T07:49:53.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:53.637+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.637742+0000) 2019-10-13T07:49:53.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:53.637+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.637990+0000) 2019-10-13T07:49:53.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:53.638+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.638416+0000) 2019-10-13T07:49:53.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:53.638+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.638717+0000) 2019-10-13T07:49:53.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:53.638+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.638828+0000) 2019-10-13T07:49:53.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:53.639+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.640659+0000) 2019-10-13T07:49:53.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:53.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129929 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:57.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:57.823+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:57.823417+0000) 2019-10-13T07:49:58.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:58.372+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.372192+0000) 2019-10-13T07:49:58.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:49:58.644+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.644528+0000) 2019-10-13T07:49:58.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:49:58.644+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.644831+0000) 2019-10-13T07:49:58.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:49:58.643+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.644030+0000) 2019-10-13T07:49:58.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:58.644+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.645370+0000) 2019-10-13T07:49:58.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:49:58.645+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.645538+0000) 2019-10-13T07:49:58.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:49:58.645+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.645716+0000) 2019-10-13T07:49:58.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:58.645+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.645925+0000) 2019-10-13T07:49:58.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:49:58.645+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.646380+0000) 2019-10-13T07:49:58.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:49:58.650+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:49:58.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:49:58.709+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.709722+0000) 2019-10-13T07:49:58.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:49:58.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:58.709718+0000) 2019-10-13T07:50:01.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:01.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:01.963234+0000) 2019-10-13T07:50:02.317 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:02.314+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:02.314597+0000) 2019-10-13T07:50:02.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:02.824+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:02.824757+0000) 2019-10-13T07:50:02.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:02.825+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:02.825672+0000) 2019-10-13T07:50:03.655 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:03.651+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.652294+0000) 2019-10-13T07:50:03.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:03.652+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.653471+0000) 2019-10-13T07:50:03.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:03.652+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.653869+0000) 2019-10-13T07:50:03.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:03.652+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.653853+0000) 2019-10-13T07:50:03.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:03.652+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.653958+0000) 2019-10-13T07:50:03.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:03.652+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.654282+0000) 2019-10-13T07:50:03.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:03.653+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.654475+0000) 2019-10-13T07:50:03.658 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:03.654+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.655450+0000) 2019-10-13T07:50:03.660 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:03.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 129989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:07.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:07.822+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:07.823599+0000) 2019-10-13T07:50:08.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:08.371+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.372368+0000) 2019-10-13T07:50:08.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:08.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.659922+0000) 2019-10-13T07:50:08.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:08.658+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.660239+0000) 2019-10-13T07:50:08.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:08.658+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.659275+0000) 2019-10-13T07:50:08.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:08.659+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.660607+0000) 2019-10-13T07:50:08.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:08.659+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.660805+0000) 2019-10-13T07:50:08.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:08.659+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.660938+0000) 2019-10-13T07:50:08.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:08.659+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.661291+0000) 2019-10-13T07:50:08.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:08.660+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.661626+0000) 2019-10-13T07:50:08.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:08.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130016 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:08.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:08.708+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.709821+0000) 2019-10-13T07:50:08.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:08.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:08.709885+0000) 2019-10-13T07:50:11.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:11.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:11.963443+0000) 2019-10-13T07:50:12.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:12.316+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:12.317011+0000) 2019-10-13T07:50:12.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:12.824+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:12.824929+0000) 2019-10-13T07:50:12.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:12.824+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:12.825844+0000) 2019-10-13T07:50:13.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:13.664+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.665591+0000) 2019-10-13T07:50:13.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:13.664+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.665781+0000) 2019-10-13T07:50:13.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:13.664+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.666126+0000) 2019-10-13T07:50:13.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:13.664+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.666298+0000) 2019-10-13T07:50:13.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:13.663+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.665637+0000) 2019-10-13T07:50:13.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:13.665+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.666829+0000) 2019-10-13T07:50:13.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:13.664+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.666174+0000) 2019-10-13T07:50:13.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:13.666+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.668410+0000) 2019-10-13T07:50:13.672 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:13.669+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:17.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:17.823+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:17.823769+0000) 2019-10-13T07:50:18.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:18.371+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.372534+0000) 2019-10-13T07:50:18.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:18.671+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.672493+0000) 2019-10-13T07:50:18.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:18.671+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.672650+0000) 2019-10-13T07:50:18.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:18.671+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.672782+0000) 2019-10-13T07:50:18.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:18.670+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.672165+0000) 2019-10-13T07:50:18.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:18.671+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.673170+0000) 2019-10-13T07:50:18.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:18.671+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.673202+0000) 2019-10-13T07:50:18.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:18.672+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.674034+0000) 2019-10-13T07:50:18.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:18.674+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.675805+0000) 2019-10-13T07:50:18.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:18.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:18.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:18.709+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.709984+0000) 2019-10-13T07:50:18.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:18.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:18.710018+0000) 2019-10-13T07:50:21.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:21.962+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:21.963673+0000) 2019-10-13T07:50:22.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:22.318+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:22.319315+0000) 2019-10-13T07:50:22.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:22.824+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:22.825081+0000) 2019-10-13T07:50:22.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:22.825+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:22.826057+0000) 2019-10-13T07:50:22.836 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:50:22.836 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:22.840 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:50:22.840 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:23.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:23.677+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.679255+0000) 2019-10-13T07:50:23.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:23.678+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.680255+0000) 2019-10-13T07:50:23.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:23.679+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.680458+0000) 2019-10-13T07:50:23.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:23.679+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.680767+0000) 2019-10-13T07:50:23.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:23.679+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.680781+0000) 2019-10-13T07:50:23.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:23.679+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.680910+0000) 2019-10-13T07:50:23.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:23.679+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.681239+0000) 2019-10-13T07:50:23.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:23.680+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.682583+0000) 2019-10-13T07:50:23.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:23.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:27.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:27.823+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:27.823907+0000) 2019-10-13T07:50:28.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:28.372+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.372736+0000) 2019-10-13T07:50:28.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:28.686+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.687902+0000) 2019-10-13T07:50:28.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:28.686+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.688131+0000) 2019-10-13T07:50:28.690 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:28.686+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.688329+0000) 2019-10-13T07:50:28.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:28.687+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.687930+0000) 2019-10-13T07:50:28.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:28.687+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.688538+0000) 2019-10-13T07:50:28.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:28.687+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.688585+0000) 2019-10-13T07:50:28.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:28.687+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.688885+0000) 2019-10-13T07:50:28.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:28.688+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.689090+0000) 2019-10-13T07:50:28.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:28.691+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130138 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:28.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:28.709+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.710163+0000) 2019-10-13T07:50:28.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:28.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:28.710171+0000) 2019-10-13T07:50:31.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:31.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:31.963883+0000) 2019-10-13T07:50:32.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:32.321+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:32.322345+0000) 2019-10-13T07:50:32.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:32.824+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:32.825238+0000) 2019-10-13T07:50:32.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:32.825+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:32.826257+0000) 2019-10-13T07:50:33.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:33.693+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.694510+0000) 2019-10-13T07:50:33.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:33.695+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.695408+0000) 2019-10-13T07:50:33.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:33.695+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.695739+0000) 2019-10-13T07:50:33.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:33.695+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.695842+0000) 2019-10-13T07:50:33.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:33.695+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.696017+0000) 2019-10-13T07:50:33.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:33.695+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.696181+0000) 2019-10-13T07:50:33.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:33.695+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.696479+0000) 2019-10-13T07:50:33.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:33.696+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.696743+0000) 2019-10-13T07:50:33.702 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:33.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130173 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:37.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:37.823+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:37.824075+0000) 2019-10-13T07:50:38.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:38.372+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.372944+0000) 2019-10-13T07:50:38.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:38.701+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.701973+0000) 2019-10-13T07:50:38.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:38.701+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.701854+0000) 2019-10-13T07:50:38.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:38.702+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.702821+0000) 2019-10-13T07:50:38.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:38.702+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.703023+0000) 2019-10-13T07:50:38.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:38.702+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.703153+0000) 2019-10-13T07:50:38.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:38.702+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.703339+0000) 2019-10-13T07:50:38.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:38.703+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.703502+0000) 2019-10-13T07:50:38.707 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:38.703+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.703816+0000) 2019-10-13T07:50:38.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:38.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:38.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:38.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.710384+0000) 2019-10-13T07:50:38.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:38.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:38.710550+0000) 2019-10-13T07:50:41.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:41.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:41.964069+0000) 2019-10-13T07:50:42.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:42.324+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:42.324691+0000) 2019-10-13T07:50:42.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:42.825+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:42.825384+0000) 2019-10-13T07:50:42.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:42.826+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:42.826450+0000) 2019-10-13T07:50:43.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:43.708+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.709093+0000) 2019-10-13T07:50:43.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:43.709+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.710177+0000) 2019-10-13T07:50:43.713 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:43.709+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.710333+0000) 2019-10-13T07:50:43.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:43.710+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.710605+0000) 2019-10-13T07:50:43.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:43.710+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.710768+0000) 2019-10-13T07:50:43.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:43.710+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.710917+0000) 2019-10-13T07:50:43.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:43.710+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.711177+0000) 2019-10-13T07:50:43.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:43.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:43.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:43.712+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.713295+0000) 2019-10-13T07:50:47.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:47.824+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:47.824263+0000) 2019-10-13T07:50:48.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:48.373+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.373142+0000) 2019-10-13T07:50:48.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:48.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.710503+0000) 2019-10-13T07:50:48.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:48.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.710637+0000) 2019-10-13T07:50:48.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:48.716+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.716857+0000) 2019-10-13T07:50:48.719 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:48.716+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.717201+0000) 2019-10-13T07:50:48.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:48.717+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.717429+0000) 2019-10-13T07:50:48.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:48.717+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.717805+0000) 2019-10-13T07:50:48.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:48.717+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.717875+0000) 2019-10-13T07:50:48.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:48.717+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.717656+0000) 2019-10-13T07:50:48.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:48.716+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.717303+0000) 2019-10-13T07:50:48.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:48.718+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:48.718851+0000) 2019-10-13T07:50:48.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:48.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130260 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:51.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:51.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:51.964282+0000) 2019-10-13T07:50:52.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:52.326+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:52.327034+0000) 2019-10-13T07:50:52.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:52.824+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:52.825567+0000) 2019-10-13T07:50:52.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:52.826+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:52.826601+0000) 2019-10-13T07:50:52.932 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:50:52.932 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:52.936 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:50:52.936 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:53.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:53.721+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.722900+0000) 2019-10-13T07:50:53.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:53.721+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.722745+0000) 2019-10-13T07:50:53.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:53.721+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.722314+0000) 2019-10-13T07:50:53.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:53.722+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.723660+0000) 2019-10-13T07:50:53.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:53.722+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.723742+0000) 2019-10-13T07:50:53.727 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:53.722+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.724035+0000) 2019-10-13T07:50:53.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:53.722+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.724179+0000) 2019-10-13T07:50:53.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:53.723+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.724710+0000) 2019-10-13T07:50:53.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:53.726+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130295 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:50:57.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:57.823+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:57.824455+0000) 2019-10-13T07:50:58.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:58.372+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.373342+0000) 2019-10-13T07:50:58.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:58.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.710664+0000) 2019-10-13T07:50:58.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:58.709+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.710809+0000) 2019-10-13T07:50:58.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:50:58.728+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.729853+0000) 2019-10-13T07:50:58.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:50:58.728+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.730043+0000) 2019-10-13T07:50:58.732 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:50:58.728+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.730330+0000) 2019-10-13T07:50:58.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:50:58.729+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.729827+0000) 2019-10-13T07:50:58.733 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:50:58.729+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.730485+0000) 2019-10-13T07:50:58.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:50:58.729+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.730788+0000) 2019-10-13T07:50:58.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:50:58.729+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.731107+0000) 2019-10-13T07:50:58.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:50:58.731+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:58.732655+0000) 2019-10-13T07:50:58.737 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:50:58.734+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130322 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:01.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:01.963+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:01.964472+0000) 2019-10-13T07:51:02.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:02.330+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:02.330331+0000) 2019-10-13T07:51:02.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:02.824+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:02.825755+0000) 2019-10-13T07:51:02.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:02.825+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:02.826709+0000) 2019-10-13T07:51:03.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:03.736+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.737476+0000) 2019-10-13T07:51:03.740 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:03.736+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.737882+0000) 2019-10-13T07:51:03.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:03.736+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.738164+0000) 2019-10-13T07:51:03.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:03.735+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.737419+0000) 2019-10-13T07:51:03.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:03.737+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.738390+0000) 2019-10-13T07:51:03.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:03.737+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.738558+0000) 2019-10-13T07:51:03.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:03.737+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.739074+0000) 2019-10-13T07:51:03.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:03.737+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.739128+0000) 2019-10-13T07:51:03.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:03.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:07.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:07.824+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:07.824667+0000) 2019-10-13T07:51:08.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:08.372+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.373554+0000) 2019-10-13T07:51:08.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:08.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.710859+0000) 2019-10-13T07:51:08.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:08.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.710984+0000) 2019-10-13T07:51:08.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:08.742+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.744251+0000) 2019-10-13T07:51:08.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:08.743+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.745224+0000) 2019-10-13T07:51:08.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:08.744+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.745646+0000) 2019-10-13T07:51:08.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:08.744+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.745906+0000) 2019-10-13T07:51:08.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:08.744+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.746092+0000) 2019-10-13T07:51:08.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:08.744+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.746142+0000) 2019-10-13T07:51:08.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:08.744+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.746618+0000) 2019-10-13T07:51:08.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:08.745+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:08.747058+0000) 2019-10-13T07:51:08.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:08.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:11.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:11.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:11.964631+0000) 2019-10-13T07:51:12.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:12.331+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:12.332809+0000) 2019-10-13T07:51:12.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:12.825+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:12.825959+0000) 2019-10-13T07:51:12.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:12.826+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:12.826875+0000) 2019-10-13T07:51:13.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:13.750+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.751922+0000) 2019-10-13T07:51:13.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:13.751+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.752952+0000) 2019-10-13T07:51:13.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:13.751+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.753082+0000) 2019-10-13T07:51:13.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:13.752+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.753721+0000) 2019-10-13T07:51:13.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:13.752+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.753867+0000) 2019-10-13T07:51:13.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:13.752+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.754284+0000) 2019-10-13T07:51:13.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:13.753+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.754633+0000) 2019-10-13T07:51:13.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:13.753+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.755503+0000) 2019-10-13T07:51:13.760 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:13.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130417 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:17.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:17.824+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:17.824866+0000) 2019-10-13T07:51:18.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:18.373+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.373731+0000) 2019-10-13T07:51:18.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:18.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.711023+0000) 2019-10-13T07:51:18.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:18.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.711146+0000) 2019-10-13T07:51:18.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:18.758+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.759387+0000) 2019-10-13T07:51:18.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:18.759+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.760291+0000) 2019-10-13T07:51:18.763 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:18.760+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.760479+0000) 2019-10-13T07:51:18.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:18.760+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.760796+0000) 2019-10-13T07:51:18.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:18.760+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.760958+0000) 2019-10-13T07:51:18.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:18.760+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.761056+0000) 2019-10-13T07:51:18.764 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:18.761+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.761554+0000) 2019-10-13T07:51:18.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:18.761+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:18.762555+0000) 2019-10-13T07:51:18.767 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:18.765+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130444 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:21.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:21.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:21.964854+0000) 2019-10-13T07:51:22.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:22.334+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:22.335292+0000) 2019-10-13T07:51:22.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:22.825+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:22.826157+0000) 2019-10-13T07:51:22.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:22.826+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:22.827103+0000) 2019-10-13T07:51:23.126 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:51:23.127 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:23.131 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:51:23.131 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:23.769 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:23.767+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.767518+0000) 2019-10-13T07:51:23.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:23.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.766908+0000) 2019-10-13T07:51:23.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:23.767+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.767994+0000) 2019-10-13T07:51:23.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:23.768+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.768489+0000) 2019-10-13T07:51:23.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:23.767+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.768181+0000) 2019-10-13T07:51:23.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:23.768+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.768733+0000) 2019-10-13T07:51:23.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:23.768+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.769035+0000) 2019-10-13T07:51:23.773 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:23.769+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.770467+0000) 2019-10-13T07:51:23.775 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:23.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:27.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:27.824+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:27.825058+0000) 2019-10-13T07:51:28.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:28.373+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.373917+0000) 2019-10-13T07:51:28.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:28.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.711208+0000) 2019-10-13T07:51:28.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:28.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.711290+0000) 2019-10-13T07:51:28.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:28.774+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.775018+0000) 2019-10-13T07:51:28.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:28.774+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.775174+0000) 2019-10-13T07:51:28.777 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:28.774+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.775372+0000) 2019-10-13T07:51:28.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:28.775+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.775416+0000) 2019-10-13T07:51:28.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:28.774+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.774819+0000) 2019-10-13T07:51:28.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:28.775+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.775723+0000) 2019-10-13T07:51:28.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:28.775+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.776021+0000) 2019-10-13T07:51:28.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:28.776+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:28.777430+0000) 2019-10-13T07:51:28.780 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:28.778+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:31.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:31.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:31.965046+0000) 2019-10-13T07:51:32.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:32.337+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:32.337649+0000) 2019-10-13T07:51:32.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:32.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:32.826348+0000) 2019-10-13T07:51:32.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:32.826+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:32.827225+0000) 2019-10-13T07:51:33.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:33.780+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.780744+0000) 2019-10-13T07:51:33.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:33.779+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.780696+0000) 2019-10-13T07:51:33.784 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:33.780+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.781254+0000) 2019-10-13T07:51:33.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:33.780+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.781339+0000) 2019-10-13T07:51:33.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:33.781+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.781775+0000) 2019-10-13T07:51:33.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:33.781+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.781647+0000) 2019-10-13T07:51:33.785 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:33.781+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.782208+0000) 2019-10-13T07:51:33.785 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:33.782+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.783255+0000) 2019-10-13T07:51:33.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:33.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:37.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:37.824+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:37.825263+0000) 2019-10-13T07:51:38.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:38.373+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.374079+0000) 2019-10-13T07:51:38.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:38.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.711394+0000) 2019-10-13T07:51:38.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:38.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.711481+0000) 2019-10-13T07:51:38.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:38.786+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.788075+0000) 2019-10-13T07:51:38.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:38.787+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.787861+0000) 2019-10-13T07:51:38.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:38.787+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.788533+0000) 2019-10-13T07:51:38.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:38.787+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.788794+0000) 2019-10-13T07:51:38.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:38.787+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.788935+0000) 2019-10-13T07:51:38.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:38.787+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.789282+0000) 2019-10-13T07:51:38.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:38.788+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.789639+0000) 2019-10-13T07:51:38.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:38.788+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:38.789544+0000) 2019-10-13T07:51:38.795 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:38.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130567 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:41.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:41.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:41.965238+0000) 2019-10-13T07:51:42.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:42.339+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:42.340007+0000) 2019-10-13T07:51:42.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:42.825+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:42.826545+0000) 2019-10-13T07:51:42.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:42.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:42.827409+0000) 2019-10-13T07:51:43.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:43.794+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.794925+0000) 2019-10-13T07:51:43.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:43.794+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.796257+0000) 2019-10-13T07:51:43.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:43.794+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.796375+0000) 2019-10-13T07:51:43.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:43.795+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.796539+0000) 2019-10-13T07:51:43.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:43.795+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.796844+0000) 2019-10-13T07:51:43.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:43.795+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.797244+0000) 2019-10-13T07:51:43.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:43.795+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.796672+0000) 2019-10-13T07:51:43.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:43.796+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.797321+0000) 2019-10-13T07:51:43.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:43.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130599 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:47.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:47.824+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:47.825470+0000) 2019-10-13T07:51:48.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:48.373+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.374239+0000) 2019-10-13T07:51:48.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:48.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.711557+0000) 2019-10-13T07:51:48.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:48.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.711636+0000) 2019-10-13T07:51:48.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:48.801+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.803109+0000) 2019-10-13T07:51:48.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:48.801+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.802702+0000) 2019-10-13T07:51:48.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:48.802+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.804147+0000) 2019-10-13T07:51:48.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:48.803+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.804460+0000) 2019-10-13T07:51:48.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:48.803+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.804648+0000) 2019-10-13T07:51:48.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:48.803+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.804847+0000) 2019-10-13T07:51:48.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:48.803+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.805024+0000) 2019-10-13T07:51:48.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:48.804+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:48.805614+0000) 2019-10-13T07:51:48.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:48.808+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130629 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:51.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:51.964+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:51.965439+0000) 2019-10-13T07:51:52.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:52.342+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:52.342487+0000) 2019-10-13T07:51:52.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:52.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:52.826706+0000) 2019-10-13T07:51:52.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:52.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:52.827572+0000) 2019-10-13T07:51:53.283 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:51:53.283 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:53.287 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:51:53.288 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:53.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:53.809+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.811336+0000) 2019-10-13T07:51:53.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:53.810+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.810744+0000) 2019-10-13T07:51:53.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:53.810+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.812029+0000) 2019-10-13T07:51:53.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:53.810+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.812050+0000) 2019-10-13T07:51:53.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:53.810+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.812153+0000) 2019-10-13T07:51:53.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:53.810+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.812293+0000) 2019-10-13T07:51:53.815 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:53.811+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.812491+0000) 2019-10-13T07:51:53.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:53.812+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.813233+0000) 2019-10-13T07:51:53.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:53.814+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130661 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:51:57.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:57.825+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:57.825646+0000) 2019-10-13T07:51:58.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:58.373+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.374429+0000) 2019-10-13T07:51:58.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:58.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.711735+0000) 2019-10-13T07:51:58.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:58.710+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.711798+0000) 2019-10-13T07:51:58.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:51:58.815+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.817128+0000) 2019-10-13T07:51:58.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:51:58.815+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.816958+0000) 2019-10-13T07:51:58.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:51:58.816+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.817602+0000) 2019-10-13T07:51:58.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:51:58.816+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.817628+0000) 2019-10-13T07:51:58.821 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:51:58.816+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.818051+0000) 2019-10-13T07:51:58.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:51:58.816+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.818250+0000) 2019-10-13T07:51:58.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:51:58.817+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.818434+0000) 2019-10-13T07:51:58.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:51:58.817+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:58.819247+0000) 2019-10-13T07:51:58.824 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:51:58.820+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130689 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:01.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:01.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:01.965602+0000) 2019-10-13T07:52:02.347 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:02.343+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:02.344728+0000) 2019-10-13T07:52:02.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:02.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:02.826860+0000) 2019-10-13T07:52:02.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:02.826+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:02.827768+0000) 2019-10-13T07:52:03.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:03.822+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.823858+0000) 2019-10-13T07:52:03.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:03.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.823400+0000) 2019-10-13T07:52:03.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:03.823+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.824561+0000) 2019-10-13T07:52:03.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:03.823+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.824732+0000) 2019-10-13T07:52:03.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:03.823+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.824845+0000) 2019-10-13T07:52:03.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:03.823+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.824957+0000) 2019-10-13T07:52:03.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:03.823+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.825120+0000) 2019-10-13T07:52:03.828 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:03.823+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.825677+0000) 2019-10-13T07:52:03.831 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:03.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:07.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:07.825+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:07.825824+0000) 2019-10-13T07:52:08.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:08.374+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.374627+0000) 2019-10-13T07:52:08.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:08.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.711923+0000) 2019-10-13T07:52:08.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:08.711+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.712002+0000) 2019-10-13T07:52:08.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:08.828+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.830650+0000) 2019-10-13T07:52:08.834 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:08.831+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.831555+0000) 2019-10-13T07:52:08.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:08.831+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.831920+0000) 2019-10-13T07:52:08.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:08.831+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.832086+0000) 2019-10-13T07:52:08.835 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:08.831+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.832249+0000) 2019-10-13T07:52:08.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:08.832+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.832590+0000) 2019-10-13T07:52:08.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:08.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.832780+0000) 2019-10-13T07:52:08.836 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:08.831+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:08.833690+0000) 2019-10-13T07:52:08.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:08.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130751 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:11.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:11.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:11.965796+0000) 2019-10-13T07:52:12.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:12.346+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:12.347254+0000) 2019-10-13T07:52:12.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:12.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:12.827018+0000) 2019-10-13T07:52:12.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:12.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:12.827925+0000) 2019-10-13T07:52:13.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:13.836+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.837507+0000) 2019-10-13T07:52:13.841 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:13.838+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.838668+0000) 2019-10-13T07:52:13.841 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:13.838+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.838552+0000) 2019-10-13T07:52:13.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:13.838+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.838790+0000) 2019-10-13T07:52:13.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:13.838+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.839099+0000) 2019-10-13T07:52:13.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:13.838+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.839287+0000) 2019-10-13T07:52:13.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:13.839+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.839446+0000) 2019-10-13T07:52:13.843 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:13.840+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.841051+0000) 2019-10-13T07:52:13.850 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:13.843+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:17.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:17.825+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:17.826039+0000) 2019-10-13T07:52:18.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:18.374+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.374805+0000) 2019-10-13T07:52:18.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:18.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.712082+0000) 2019-10-13T07:52:18.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:18.711+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.712157+0000) 2019-10-13T07:52:18.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:18.845+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.845735+0000) 2019-10-13T07:52:18.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:18.845+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.845835+0000) 2019-10-13T07:52:18.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:18.845+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.846146+0000) 2019-10-13T07:52:18.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:18.844+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.845267+0000) 2019-10-13T07:52:18.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:18.845+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.846309+0000) 2019-10-13T07:52:18.849 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:18.846+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.846516+0000) 2019-10-13T07:52:18.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:18.846+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.847018+0000) 2019-10-13T07:52:18.850 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:18.847+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:18.848234+0000) 2019-10-13T07:52:18.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:18.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:21.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:21.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:21.965962+0000) 2019-10-13T07:52:22.352 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:22.348+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:22.349691+0000) 2019-10-13T07:52:22.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:22.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:22.827187+0000) 2019-10-13T07:52:22.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:22.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:22.828112+0000) 2019-10-13T07:52:23.363 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:52:23.363 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:23.368 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:52:23.368 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:23.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:23.851+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.853003+0000) 2019-10-13T07:52:23.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:23.851+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.853366+0000) 2019-10-13T07:52:23.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:23.852+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.853515+0000) 2019-10-13T07:52:23.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:23.852+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.853717+0000) 2019-10-13T07:52:23.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:23.852+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.853936+0000) 2019-10-13T07:52:23.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:23.852+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.852877+0000) 2019-10-13T07:52:23.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:23.853+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.854240+0000) 2019-10-13T07:52:23.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:23.852+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.854313+0000) 2019-10-13T07:52:23.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:23.857+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:27.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:27.825+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:27.826233+0000) 2019-10-13T07:52:28.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:28.374+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.374973+0000) 2019-10-13T07:52:28.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:28.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.712223+0000) 2019-10-13T07:52:28.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:28.711+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.712337+0000) 2019-10-13T07:52:28.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:28.859+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.860940+0000) 2019-10-13T07:52:28.863 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:28.859+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.861052+0000) 2019-10-13T07:52:28.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:28.859+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.860568+0000) 2019-10-13T07:52:28.864 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:28.860+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.861468+0000) 2019-10-13T07:52:28.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:28.860+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.861640+0000) 2019-10-13T07:52:28.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:28.860+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.861778+0000) 2019-10-13T07:52:28.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:28.860+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.862075+0000) 2019-10-13T07:52:28.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:28.862+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:28.863572+0000) 2019-10-13T07:52:28.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:28.865+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130873 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:31.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:31.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:31.966148+0000) 2019-10-13T07:52:32.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:32.351+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:32.352222+0000) 2019-10-13T07:52:32.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:32.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:32.827371+0000) 2019-10-13T07:52:32.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:32.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:32.828244+0000) 2019-10-13T07:52:33.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:33.866+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.867604+0000) 2019-10-13T07:52:33.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:33.867+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.868562+0000) 2019-10-13T07:52:33.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:33.867+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.868711+0000) 2019-10-13T07:52:33.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:33.867+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.868918+0000) 2019-10-13T07:52:33.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:33.867+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.868857+0000) 2019-10-13T07:52:33.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:33.867+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.869137+0000) 2019-10-13T07:52:33.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:33.867+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.869262+0000) 2019-10-13T07:52:33.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:33.869+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.870716+0000) 2019-10-13T07:52:33.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:33.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:37.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:37.825+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:37.826445+0000) 2019-10-13T07:52:38.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:38.374+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.375126+0000) 2019-10-13T07:52:38.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:38.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.712437+0000) 2019-10-13T07:52:38.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:38.711+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.712538+0000) 2019-10-13T07:52:38.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:38.873+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.875172+0000) 2019-10-13T07:52:38.877 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:38.873+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.875323+0000) 2019-10-13T07:52:38.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:38.874+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.875723+0000) 2019-10-13T07:52:38.878 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:38.874+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.875614+0000) 2019-10-13T07:52:38.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:38.874+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.875149+0000) 2019-10-13T07:52:38.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:38.874+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.876261+0000) 2019-10-13T07:52:38.879 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:38.875+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.876509+0000) 2019-10-13T07:52:38.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:38.877+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:38.877843+0000) 2019-10-13T07:52:38.882 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:38.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130933 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:41.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:41.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:41.966317+0000) 2019-10-13T07:52:42.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:42.354+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:42.354626+0000) 2019-10-13T07:52:42.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:42.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:42.827555+0000) 2019-10-13T07:52:42.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:42.828+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:42.828410+0000) 2019-10-13T07:52:43.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:43.881+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.882640+0000) 2019-10-13T07:52:43.885 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:43.881+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.882811+0000) 2019-10-13T07:52:43.885 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:43.881+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.883045+0000) 2019-10-13T07:52:43.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:43.881+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.883189+0000) 2019-10-13T07:52:43.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:43.881+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.882598+0000) 2019-10-13T07:52:43.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:43.882+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.883520+0000) 2019-10-13T07:52:43.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:43.882+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.884018+0000) 2019-10-13T07:52:43.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:43.884+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.885128+0000) 2019-10-13T07:52:43.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:43.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:47.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:47.826+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:47.826611+0000) 2019-10-13T07:52:48.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:48.374+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.375308+0000) 2019-10-13T07:52:48.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:48.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.712625+0000) 2019-10-13T07:52:48.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:48.712+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.712685+0000) 2019-10-13T07:52:48.892 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:48.888+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.890000+0000) 2019-10-13T07:52:48.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:48.888+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.890182+0000) 2019-10-13T07:52:48.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:48.889+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.890005+0000) 2019-10-13T07:52:48.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:48.889+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.890567+0000) 2019-10-13T07:52:48.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:48.888+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.890390+0000) 2019-10-13T07:52:48.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:48.889+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.890854+0000) 2019-10-13T07:52:48.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:48.889+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.890744+0000) 2019-10-13T07:52:48.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:48.891+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:48.892385+0000) 2019-10-13T07:52:48.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:48.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 130997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:51.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:51.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:51.966430+0000) 2019-10-13T07:52:52.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:52.355+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:52.356897+0000) 2019-10-13T07:52:52.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:52.827+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:52.827712+0000) 2019-10-13T07:52:52.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:52.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:52.828567+0000) 2019-10-13T07:52:53.486 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:52:53.486 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:53.490 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:52:53.490 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:53.899 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:53.896+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.896517+0000) 2019-10-13T07:52:53.900 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:53.896+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.896421+0000) 2019-10-13T07:52:53.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:53.896+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.896922+0000) 2019-10-13T07:52:53.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:53.896+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.897106+0000) 2019-10-13T07:52:53.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:53.897+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.897546+0000) 2019-10-13T07:52:53.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:53.897+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.897644+0000) 2019-10-13T07:52:53.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:53.894+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.895958+0000) 2019-10-13T07:52:53.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:53.895+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.897600+0000) 2019-10-13T07:52:53.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:53.901+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:52:57.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:57.826+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:57.826774+0000) 2019-10-13T07:52:58.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:58.375+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.375480+0000) 2019-10-13T07:52:58.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:58.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.712737+0000) 2019-10-13T07:52:58.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:58.712+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.712788+0000) 2019-10-13T07:52:58.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:52:58.900+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.902561+0000) 2019-10-13T07:52:58.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:52:58.903+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.903425+0000) 2019-10-13T07:52:58.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:52:58.903+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.903690+0000) 2019-10-13T07:52:58.908 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:52:58.903+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.903812+0000) 2019-10-13T07:52:58.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:52:58.903+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.904057+0000) 2019-10-13T07:52:58.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:52:58.903+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.903950+0000) 2019-10-13T07:52:58.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:52:58.904+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.904610+0000) 2019-10-13T07:52:58.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:52:58.903+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:58.904986+0000) 2019-10-13T07:52:58.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:52:58.907+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131059 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:01.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:01.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:01.966566+0000) 2019-10-13T07:53:02.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:02.358+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:02.359190+0000) 2019-10-13T07:53:02.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:02.827+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:02.827845+0000) 2019-10-13T07:53:02.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:02.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:02.828758+0000) 2019-10-13T07:53:03.911 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:03.909+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.909792+0000) 2019-10-13T07:53:03.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:03.909+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.910109+0000) 2019-10-13T07:53:03.912 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:03.910+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.910626+0000) 2019-10-13T07:53:03.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:03.908+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.910181+0000) 2019-10-13T07:53:03.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:03.910+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.910643+0000) 2019-10-13T07:53:03.913 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:03.910+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.911198+0000) 2019-10-13T07:53:03.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:03.910+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.910937+0000) 2019-10-13T07:53:03.914 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:03.909+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.911337+0000) 2019-10-13T07:53:03.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:03.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:07.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:07.826+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:07.826969+0000) 2019-10-13T07:53:08.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:08.375+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.375697+0000) 2019-10-13T07:53:08.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:08.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.712934+0000) 2019-10-13T07:53:08.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:08.711+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.712961+0000) 2019-10-13T07:53:08.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:08.915+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.916292+0000) 2019-10-13T07:53:08.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:08.915+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.917277+0000) 2019-10-13T07:53:08.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:08.915+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.917381+0000) 2019-10-13T07:53:08.920 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:08.916+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.917523+0000) 2019-10-13T07:53:08.920 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:08.916+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.917711+0000) 2019-10-13T07:53:08.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:08.916+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.918291+0000) 2019-10-13T07:53:08.921 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:08.917+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.918644+0000) 2019-10-13T07:53:08.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:08.918+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:08.919184+0000) 2019-10-13T07:53:08.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:08.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:11.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:11.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:11.966735+0000) 2019-10-13T07:53:12.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:12.360+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:12.361482+0000) 2019-10-13T07:53:12.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:12.827+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:12.828011+0000) 2019-10-13T07:53:12.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:12.828+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:12.828934+0000) 2019-10-13T07:53:13.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:13.921+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.922884+0000) 2019-10-13T07:53:13.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:13.921+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.923047+0000) 2019-10-13T07:53:13.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:13.921+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.923256+0000) 2019-10-13T07:53:13.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:13.922+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.923060+0000) 2019-10-13T07:53:13.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:13.922+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.923730+0000) 2019-10-13T07:53:13.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:13.922+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.923952+0000) 2019-10-13T07:53:13.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:13.923+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.924438+0000) 2019-10-13T07:53:13.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:13.924+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.925024+0000) 2019-10-13T07:53:13.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:13.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:17.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:17.826+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:17.827136+0000) 2019-10-13T07:53:18.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:18.375+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.375888+0000) 2019-10-13T07:53:18.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:18.712+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.713129+0000) 2019-10-13T07:53:18.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:18.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.713114+0000) 2019-10-13T07:53:18.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:18.928+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.930120+0000) 2019-10-13T07:53:18.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:18.929+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.930451+0000) 2019-10-13T07:53:18.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:18.929+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.930706+0000) 2019-10-13T07:53:18.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:18.929+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.930093+0000) 2019-10-13T07:53:18.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:18.929+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.931306+0000) 2019-10-13T07:53:18.934 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:18.930+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.931517+0000) 2019-10-13T07:53:18.934 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:18.930+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.931678+0000) 2019-10-13T07:53:18.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:18.930+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:18.931653+0000) 2019-10-13T07:53:18.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:18.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:21.968 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:21.965+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:21.966898+0000) 2019-10-13T07:53:22.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:22.363+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:22.363989+0000) 2019-10-13T07:53:22.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:22.826+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:22.828215+0000) 2019-10-13T07:53:22.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:22.828+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:22.829133+0000) 2019-10-13T07:53:23.638 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:53:23.638 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:23.642 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:53:23.642 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:23.939 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:23.936+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.937569+0000) 2019-10-13T07:53:23.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:23.936+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.937926+0000) 2019-10-13T07:53:23.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:23.936+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.937347+0000) 2019-10-13T07:53:23.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:23.937+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.938471+0000) 2019-10-13T07:53:23.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:23.937+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.938504+0000) 2019-10-13T07:53:23.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:23.937+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.938699+0000) 2019-10-13T07:53:23.941 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:23.937+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.938816+0000) 2019-10-13T07:53:23.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:23.938+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.939434+0000) 2019-10-13T07:53:23.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:23.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:27.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:27.826+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:27.827304+0000) 2019-10-13T07:53:28.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:28.375+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.376079+0000) 2019-10-13T07:53:28.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:28.711+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.713339+0000) 2019-10-13T07:53:28.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:28.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.713334+0000) 2019-10-13T07:53:28.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:28.942+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.943583+0000) 2019-10-13T07:53:28.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:28.943+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.944662+0000) 2019-10-13T07:53:28.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:28.943+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.944873+0000) 2019-10-13T07:53:28.947 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:28.943+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.944989+0000) 2019-10-13T07:53:28.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:28.943+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.945129+0000) 2019-10-13T07:53:28.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:28.943+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.945350+0000) 2019-10-13T07:53:28.948 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:28.944+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.945749+0000) 2019-10-13T07:53:28.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:28.946+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:28.947062+0000) 2019-10-13T07:53:28.952 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:28.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:31.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:31.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:31.967092+0000) 2019-10-13T07:53:32.368 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:32.366+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:32.366292+0000) 2019-10-13T07:53:32.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:32.827+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:32.828406+0000) 2019-10-13T07:53:32.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:32.829+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:32.829315+0000) 2019-10-13T07:53:33.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:33.950+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.952088+0000) 2019-10-13T07:53:33.954 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:33.950+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.952280+0000) 2019-10-13T07:53:33.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:33.951+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.951740+0000) 2019-10-13T07:53:33.955 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:33.951+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.952936+0000) 2019-10-13T07:53:33.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:33.951+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.953140+0000) 2019-10-13T07:53:33.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:33.951+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.953254+0000) 2019-10-13T07:53:33.956 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:33.952+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.953663+0000) 2019-10-13T07:53:33.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:33.954+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.954808+0000) 2019-10-13T07:53:33.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:33.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:37.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:37.827+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:37.827464+0000) 2019-10-13T07:53:38.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:38.375+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.376273+0000) 2019-10-13T07:53:38.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:38.713+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.713521+0000) 2019-10-13T07:53:38.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:38.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.713524+0000) 2019-10-13T07:53:38.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:38.957+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.958697+0000) 2019-10-13T07:53:38.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:38.959+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.959693+0000) 2019-10-13T07:53:38.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:38.959+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.960212+0000) 2019-10-13T07:53:38.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:38.960+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.960478+0000) 2019-10-13T07:53:38.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:38.959+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.960353+0000) 2019-10-13T07:53:38.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:38.960+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.960695+0000) 2019-10-13T07:53:38.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:38.960+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.960760+0000) 2019-10-13T07:53:38.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:38.960+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:38.961481+0000) 2019-10-13T07:53:38.966 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:38.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131311 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:41.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:41.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:41.967271+0000) 2019-10-13T07:53:42.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:42.368+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:42.368614+0000) 2019-10-13T07:53:42.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:42.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:42.828581+0000) 2019-10-13T07:53:42.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:42.829+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:42.829501+0000) 2019-10-13T07:53:43.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:43.966+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.966862+0000) 2019-10-13T07:53:43.969 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:43.966+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.966758+0000) 2019-10-13T07:53:43.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:43.966+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.967061+0000) 2019-10-13T07:53:43.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:43.966+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.967169+0000) 2019-10-13T07:53:43.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:43.966+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.967306+0000) 2019-10-13T07:53:43.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:43.967+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.967628+0000) 2019-10-13T07:53:43.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:43.965+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.967457+0000) 2019-10-13T07:53:43.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:43.967+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.969234+0000) 2019-10-13T07:53:43.974 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:43.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:47.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:47.827+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:47.827605+0000) 2019-10-13T07:53:48.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:48.376+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.376469+0000) 2019-10-13T07:53:48.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:48.713+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.713710+0000) 2019-10-13T07:53:48.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:48.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.713721+0000) 2019-10-13T07:53:48.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:48.971+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.973262+0000) 2019-10-13T07:53:48.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:48.973+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.974163+0000) 2019-10-13T07:53:48.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:48.973+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.974280+0000) 2019-10-13T07:53:48.977 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:48.974+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.974621+0000) 2019-10-13T07:53:48.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:48.974+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.974833+0000) 2019-10-13T07:53:48.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:48.974+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.975195+0000) 2019-10-13T07:53:48.978 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:48.974+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.975093+0000) 2019-10-13T07:53:48.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:48.974+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:48.975893+0000) 2019-10-13T07:53:48.981 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:48.979+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131375 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:51.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:51.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:51.967461+0000) 2019-10-13T07:53:52.373 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:52.370+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:52.371087+0000) 2019-10-13T07:53:52.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:52.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:52.828762+0000) 2019-10-13T07:53:52.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:52.827+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:52.829699+0000) 2019-10-13T07:53:53.808 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:53:53.809 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:53.813 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:53:53.813 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:53.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:53.980+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.981389+0000) 2019-10-13T07:53:53.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:53.981+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.981782+0000) 2019-10-13T07:53:53.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:53.981+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.981924+0000) 2019-10-13T07:53:53.985 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:53.981+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.982374+0000) 2019-10-13T07:53:53.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:53.981+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.982192+0000) 2019-10-13T07:53:53.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:53.979+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.981382+0000) 2019-10-13T07:53:53.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:53.981+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.982811+0000) 2019-10-13T07:53:53.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:53.982+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.982880+0000) 2019-10-13T07:53:53.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:53.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:53:57.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:57.826+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:57.827784+0000) 2019-10-13T07:53:58.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:58.375+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.376666+0000) 2019-10-13T07:53:58.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:58.712+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.713916+0000) 2019-10-13T07:53:58.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:58.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.713921+0000) 2019-10-13T07:53:58.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:53:58.986+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.988065+0000) 2019-10-13T07:53:58.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:53:58.986+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.988308+0000) 2019-10-13T07:53:58.990 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:53:58.987+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.988489+0000) 2019-10-13T07:53:58.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:53:58.987+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.988633+0000) 2019-10-13T07:53:58.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:53:58.987+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.988294+0000) 2019-10-13T07:53:58.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:53:58.987+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.988713+0000) 2019-10-13T07:53:58.992 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:53:58.987+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.988881+0000) 2019-10-13T07:53:58.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:53:58.990+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:58.990896+0000) 2019-10-13T07:53:58.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:53:58.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131437 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:01.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:01.966+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:01.967653+0000) 2019-10-13T07:54:02.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:02.372+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:02.373385+0000) 2019-10-13T07:54:02.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:02.827+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:02.828954+0000) 2019-10-13T07:54:02.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:02.829+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:02.829874+0000) 2019-10-13T07:54:03.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:03.994+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.996159+0000) 2019-10-13T07:54:03.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:03.994+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.996363+0000) 2019-10-13T07:54:03.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:03.995+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.996463+0000) 2019-10-13T07:54:03.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:03.995+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.996630+0000) 2019-10-13T07:54:03.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:03.995+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.996098+0000) 2019-10-13T07:54:03.999 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:03.995+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.996932+0000) 2019-10-13T07:54:04.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:03.995+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.997199+0000) 2019-10-13T07:54:04.000 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:03.997+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.997766+0000) 2019-10-13T07:54:04.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:03.999+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131469 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:07.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:07.827+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:07.827974+0000) 2019-10-13T07:54:08.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:08.376+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:08.376824+0000) 2019-10-13T07:54:08.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:08.713+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:08.714114+0000) 2019-10-13T07:54:08.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:08.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:08.714119+0000) 2019-10-13T07:54:09.004 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:09.000+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.001447+0000) 2019-10-13T07:54:09.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:09.000+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.002240+0000) 2019-10-13T07:54:09.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:09.001+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.002553+0000) 2019-10-13T07:54:09.005 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:09.001+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.002676+0000) 2019-10-13T07:54:09.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:09.001+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.003019+0000) 2019-10-13T07:54:09.005 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:09.001+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.002894+0000) 2019-10-13T07:54:09.006 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:09.001+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.003126+0000) 2019-10-13T07:54:09.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:09.003+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.003776+0000) 2019-10-13T07:54:09.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:09.006+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131501 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:11.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:11.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:11.967844+0000) 2019-10-13T07:54:12.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:12.375+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:12.375509+0000) 2019-10-13T07:54:12.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:12.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:12.829138+0000) 2019-10-13T07:54:12.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:12.829+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:12.830041+0000) 2019-10-13T07:54:14.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:14.007+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.008545+0000) 2019-10-13T07:54:14.012 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:14.008+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.009474+0000) 2019-10-13T07:54:14.012 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:14.008+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.009617+0000) 2019-10-13T07:54:14.012 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:14.008+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.009772+0000) 2019-10-13T07:54:14.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:14.008+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.010181+0000) 2019-10-13T07:54:14.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:14.008+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.010252+0000) 2019-10-13T07:54:14.013 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:14.008+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.010386+0000) 2019-10-13T07:54:14.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:14.010+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.011341+0000) 2019-10-13T07:54:14.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:14.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131533 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:17.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:17.827+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:17.828152+0000) 2019-10-13T07:54:18.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:18.376+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:18.376995+0000) 2019-10-13T07:54:18.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:18.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:18.714319+0000) 2019-10-13T07:54:18.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:18.713+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:18.714341+0000) 2019-10-13T07:54:19.018 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:19.015+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.016831+0000) 2019-10-13T07:54:19.019 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:19.015+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.017135+0000) 2019-10-13T07:54:19.019 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:19.016+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.017547+0000) 2019-10-13T07:54:19.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:19.016+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.017265+0000) 2019-10-13T07:54:19.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:19.016+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.018275+0000) 2019-10-13T07:54:19.020 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:19.016+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.018376+0000) 2019-10-13T07:54:19.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:19.017+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.018572+0000) 2019-10-13T07:54:19.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:19.018+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.019147+0000) 2019-10-13T07:54:19.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:19.021+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131563 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:21.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:21.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:21.968040+0000) 2019-10-13T07:54:22.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:22.377+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:22.377854+0000) 2019-10-13T07:54:22.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:22.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:22.829307+0000) 2019-10-13T07:54:22.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:22.829+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:22.830254+0000) 2019-10-13T07:54:24.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:24.023+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.023900+0000) 2019-10-13T07:54:24.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:24.024+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.024816+0000) 2019-10-13T07:54:24.027 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:24.024+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.024987+0000) 2019-10-13T07:54:24.027 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:24.024+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.025164+0000) 2019-10-13T07:54:24.027 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:24.025+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.025528+0000) 2019-10-13T07:54:24.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:24.025+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.025784+0000) 2019-10-13T07:54:24.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:24.026+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.026427+0000) 2019-10-13T07:54:24.029 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:24.026+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.027360+0000) 2019-10-13T07:54:24.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:24.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:24.037 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:54:24.037 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:24.041 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:54:24.041 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:27.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:27.827+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:27.828305+0000) 2019-10-13T07:54:28.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:28.376+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:28.377182+0000) 2019-10-13T07:54:28.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:28.714+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:28.714501+0000) 2019-10-13T07:54:28.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:28.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:28.714501+0000) 2019-10-13T07:54:29.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:29.031+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.031831+0000) 2019-10-13T07:54:29.034 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:29.031+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.032185+0000) 2019-10-13T07:54:29.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:29.030+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.031379+0000) 2019-10-13T07:54:29.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:29.032+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.032818+0000) 2019-10-13T07:54:29.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:29.032+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.033091+0000) 2019-10-13T07:54:29.035 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:29.032+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.033013+0000) 2019-10-13T07:54:29.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:29.032+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.033335+0000) 2019-10-13T07:54:29.036 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:29.033+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.033873+0000) 2019-10-13T07:54:29.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:29.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:31.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:31.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:31.968250+0000) 2019-10-13T07:54:32.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:32.380+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:32.380272+0000) 2019-10-13T07:54:32.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:32.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:32.829470+0000) 2019-10-13T07:54:32.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:32.829+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:32.830439+0000) 2019-10-13T07:54:34.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:34.039+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.039527+0000) 2019-10-13T07:54:34.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:34.038+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.039015+0000) 2019-10-13T07:54:34.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:34.039+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.039941+0000) 2019-10-13T07:54:34.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:34.039+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.040114+0000) 2019-10-13T07:54:34.043 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:34.039+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.040261+0000) 2019-10-13T07:54:34.043 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:34.040+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.040476+0000) 2019-10-13T07:54:34.044 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:34.040+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.040792+0000) 2019-10-13T07:54:34.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:34.040+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.040762+0000) 2019-10-13T07:54:34.046 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:34.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:37.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:37.828+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:37.828486+0000) 2019-10-13T07:54:38.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:38.377+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:38.377378+0000) 2019-10-13T07:54:38.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:38.714+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:38.714713+0000) 2019-10-13T07:54:38.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:38.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:38.714713+0000) 2019-10-13T07:54:39.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:39.046+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.046726+0000) 2019-10-13T07:54:39.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:39.046+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.046938+0000) 2019-10-13T07:54:39.049 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:39.046+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.047025+0000) 2019-10-13T07:54:39.049 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:39.047+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.047446+0000) 2019-10-13T07:54:39.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:39.044+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.046400+0000) 2019-10-13T07:54:39.050 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:39.047+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.047719+0000) 2019-10-13T07:54:39.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:39.047+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.047965+0000) 2019-10-13T07:54:39.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:39.046+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.048496+0000) 2019-10-13T07:54:39.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:39.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131689 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:41.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:41.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:41.968422+0000) 2019-10-13T07:54:42.385 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:42.381+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:42.382537+0000) 2019-10-13T07:54:42.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:42.828+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:42.829674+0000) 2019-10-13T07:54:42.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:42.829+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:42.830604+0000) 2019-10-13T07:54:44.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:44.051+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.053047+0000) 2019-10-13T07:54:44.055 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:44.051+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.053238+0000) 2019-10-13T07:54:44.055 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:44.052+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.053879+0000) 2019-10-13T07:54:44.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:44.051+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.053553+0000) 2019-10-13T07:54:44.056 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:44.052+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.054194+0000) 2019-10-13T07:54:44.056 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:44.053+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.054535+0000) 2019-10-13T07:54:44.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:44.053+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.054714+0000) 2019-10-13T07:54:44.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:44.054+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.056398+0000) 2019-10-13T07:54:44.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:44.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:47.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:47.827+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:47.828632+0000) 2019-10-13T07:54:48.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:48.376+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:48.377580+0000) 2019-10-13T07:54:48.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:48.714+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:48.714922+0000) 2019-10-13T07:54:48.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:48.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:48.714918+0000) 2019-10-13T07:54:49.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:49.059+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.060453+0000) 2019-10-13T07:54:49.062 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:49.059+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.060737+0000) 2019-10-13T07:54:49.063 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:49.059+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.061001+0000) 2019-10-13T07:54:49.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:49.058+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.060623+0000) 2019-10-13T07:54:49.064 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:49.060+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.061459+0000) 2019-10-13T07:54:49.064 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:49.060+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.061708+0000) 2019-10-13T07:54:49.064 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:49.060+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.062074+0000) 2019-10-13T07:54:49.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:49.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.062298+0000) 2019-10-13T07:54:49.067 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:49.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131753 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:51.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:51.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:51.968608+0000) 2019-10-13T07:54:52.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:52.384+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:52.384970+0000) 2019-10-13T07:54:52.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:52.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:52.829857+0000) 2019-10-13T07:54:52.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:52.830+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:52.830752+0000) 2019-10-13T07:54:54.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:54.066+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.067240+0000) 2019-10-13T07:54:54.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:54.066+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.068021+0000) 2019-10-13T07:54:54.070 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:54.066+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.068183+0000) 2019-10-13T07:54:54.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:54.067+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.068390+0000) 2019-10-13T07:54:54.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:54.067+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.068522+0000) 2019-10-13T07:54:54.071 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:54.067+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.068765+0000) 2019-10-13T07:54:54.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:54.068+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.069494+0000) 2019-10-13T07:54:54.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:54.070+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.071172+0000) 2019-10-13T07:54:54.075 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:54.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:54:54.250 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:54:54.250 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:54.254 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:54:54.254 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:57.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:57.828+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:57.828789+0000) 2019-10-13T07:54:58.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:58.377+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:58.377771+0000) 2019-10-13T07:54:58.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:58.713+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:58.715134+0000) 2019-10-13T07:54:58.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:58.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:58.715113+0000) 2019-10-13T07:54:59.077 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:54:59.073+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.075173+0000) 2019-10-13T07:54:59.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:54:59.073+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.075378+0000) 2019-10-13T07:54:59.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:54:59.074+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.074977+0000) 2019-10-13T07:54:59.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:54:59.074+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.076120+0000) 2019-10-13T07:54:59.078 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:54:59.074+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.076175+0000) 2019-10-13T07:54:59.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:54:59.075+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.076459+0000) 2019-10-13T07:54:59.079 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:54:59.075+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.076617+0000) 2019-10-13T07:54:59.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:54:59.077+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.077933+0000) 2019-10-13T07:54:59.083 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:54:59.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:01.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:01.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:01.968724+0000) 2019-10-13T07:55:02.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:02.386+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:02.387426+0000) 2019-10-13T07:55:02.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:02.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:02.830007+0000) 2019-10-13T07:55:02.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:02.830+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:02.830910+0000) 2019-10-13T07:55:04.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:04.081+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.082902+0000) 2019-10-13T07:55:04.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:04.082+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.082890+0000) 2019-10-13T07:55:04.086 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:04.082+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.083467+0000) 2019-10-13T07:55:04.086 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:04.082+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.083747+0000) 2019-10-13T07:55:04.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:04.082+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.083907+0000) 2019-10-13T07:55:04.086 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:04.082+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.084130+0000) 2019-10-13T07:55:04.086 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:04.083+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.084453+0000) 2019-10-13T07:55:04.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:04.085+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.086202+0000) 2019-10-13T07:55:04.090 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:04.087+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:07.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:07.828+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:07.828986+0000) 2019-10-13T07:55:08.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:08.377+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:08.377956+0000) 2019-10-13T07:55:08.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:08.714+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:08.715330+0000) 2019-10-13T07:55:08.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:08.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:08.715323+0000) 2019-10-13T07:55:09.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:09.089+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.090588+0000) 2019-10-13T07:55:09.094 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:09.090+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.091441+0000) 2019-10-13T07:55:09.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:09.090+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.091680+0000) 2019-10-13T07:55:09.094 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:09.090+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.091879+0000) 2019-10-13T07:55:09.094 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:09.090+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.092072+0000) 2019-10-13T07:55:09.095 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:09.090+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.092083+0000) 2019-10-13T07:55:09.095 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:09.090+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.092385+0000) 2019-10-13T07:55:09.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:09.091+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.092410+0000) 2019-10-13T07:55:09.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:09.096+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131879 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:11.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:11.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:11.968900+0000) 2019-10-13T07:55:12.392 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:12.389+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:12.389761+0000) 2019-10-13T07:55:12.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:12.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:12.830190+0000) 2019-10-13T07:55:12.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:12.830+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:12.831091+0000) 2019-10-13T07:55:14.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:14.097+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.097737+0000) 2019-10-13T07:55:14.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:14.098+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.098716+0000) 2019-10-13T07:55:14.101 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:14.098+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.098994+0000) 2019-10-13T07:55:14.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:14.098+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.099074+0000) 2019-10-13T07:55:14.101 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:14.098+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.099260+0000) 2019-10-13T07:55:14.101 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:14.098+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.099343+0000) 2019-10-13T07:55:14.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:14.099+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.099617+0000) 2019-10-13T07:55:14.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:14.100+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.100845+0000) 2019-10-13T07:55:14.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:14.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:17.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:17.828+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:17.829203+0000) 2019-10-13T07:55:18.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:18.377+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:18.378153+0000) 2019-10-13T07:55:18.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:18.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:18.715518+0000) 2019-10-13T07:55:18.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:18.715+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:18.715539+0000) 2019-10-13T07:55:19.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:19.105+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.105939+0000) 2019-10-13T07:55:19.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:19.105+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.106173+0000) 2019-10-13T07:55:19.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:19.104+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.105420+0000) 2019-10-13T07:55:19.109 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:19.106+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.106491+0000) 2019-10-13T07:55:19.109 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:19.105+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.106310+0000) 2019-10-13T07:55:19.109 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:19.106+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.106950+0000) 2019-10-13T07:55:19.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:19.106+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.107121+0000) 2019-10-13T07:55:19.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:19.107+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.108066+0000) 2019-10-13T07:55:19.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:19.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131941 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:21.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:21.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:21.969035+0000) 2019-10-13T07:55:22.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:22.391+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:22.392081+0000) 2019-10-13T07:55:22.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:22.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:22.830375+0000) 2019-10-13T07:55:22.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:22.831+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:22.831277+0000) 2019-10-13T07:55:24.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:24.111+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.112395+0000) 2019-10-13T07:55:24.115 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:24.112+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.112614+0000) 2019-10-13T07:55:24.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:24.112+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.112885+0000) 2019-10-13T07:55:24.116 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:24.112+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.113335+0000) 2019-10-13T07:55:24.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:24.113+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.113545+0000) 2019-10-13T07:55:24.116 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:24.113+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.113747+0000) 2019-10-13T07:55:24.116 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:24.113+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.114078+0000) 2019-10-13T07:55:24.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:24.114+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.115648+0000) 2019-10-13T07:55:24.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:24.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 131973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:24.464 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:55:24.465 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:24.470 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:55:24.470 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:27.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:27.828+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:27.829422+0000) 2019-10-13T07:55:28.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:28.376+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:28.378311+0000) 2019-10-13T07:55:28.717 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:28.714+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:28.715723+0000) 2019-10-13T07:55:28.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:28.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:28.715704+0000) 2019-10-13T07:55:29.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:29.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.119802+0000) 2019-10-13T07:55:29.123 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:29.119+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.120732+0000) 2019-10-13T07:55:29.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:29.119+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.121195+0000) 2019-10-13T07:55:29.124 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:29.119+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.121304+0000) 2019-10-13T07:55:29.124 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:29.120+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.121460+0000) 2019-10-13T07:55:29.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:29.120+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.121597+0000) 2019-10-13T07:55:29.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:29.120+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.122054+0000) 2019-10-13T07:55:29.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:29.121+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.122248+0000) 2019-10-13T07:55:29.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:29.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132005 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:31.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:31.967+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:31.969230+0000) 2019-10-13T07:55:32.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:32.393+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:32.394447+0000) 2019-10-13T07:55:32.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:32.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:32.830579+0000) 2019-10-13T07:55:32.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:32.830+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:32.831433+0000) 2019-10-13T07:55:34.129 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:34.126+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.127921+0000) 2019-10-13T07:55:34.130 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:34.125+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.127436+0000) 2019-10-13T07:55:34.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:34.127+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.128568+0000) 2019-10-13T07:55:34.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:34.127+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.129004+0000) 2019-10-13T07:55:34.131 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:34.127+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.129349+0000) 2019-10-13T07:55:34.131 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:34.128+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.129786+0000) 2019-10-13T07:55:34.132 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:34.128+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.129979+0000) 2019-10-13T07:55:34.132 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:34.128+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.130641+0000) 2019-10-13T07:55:34.135 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:34.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:37.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:37.828+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:37.829592+0000) 2019-10-13T07:55:38.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:38.377+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:38.378495+0000) 2019-10-13T07:55:38.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:38.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:38.715897+0000) 2019-10-13T07:55:38.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:38.715+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:38.715898+0000) 2019-10-13T07:55:39.136 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:39.132+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.134552+0000) 2019-10-13T07:55:39.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:39.134+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.135667+0000) 2019-10-13T07:55:39.137 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:39.134+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.135812+0000) 2019-10-13T07:55:39.138 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:39.134+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.136304+0000) 2019-10-13T07:55:39.139 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:39.135+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.136458+0000) 2019-10-13T07:55:39.139 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:39.135+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.136694+0000) 2019-10-13T07:55:39.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:39.135+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.136811+0000) 2019-10-13T07:55:39.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:39.135+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.137452+0000) 2019-10-13T07:55:39.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:39.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132067 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:41.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:41.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:41.969390+0000) 2019-10-13T07:55:42.400 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:42.395+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:42.396783+0000) 2019-10-13T07:55:42.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:42.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:42.830746+0000) 2019-10-13T07:55:42.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:42.830+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:42.831607+0000) 2019-10-13T07:55:44.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:44.141+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.142563+0000) 2019-10-13T07:55:44.145 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:44.141+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.142912+0000) 2019-10-13T07:55:44.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:44.141+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.143092+0000) 2019-10-13T07:55:44.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:44.141+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.143313+0000) 2019-10-13T07:55:44.146 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:44.142+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.143532+0000) 2019-10-13T07:55:44.147 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:44.141+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.142045+0000) 2019-10-13T07:55:44.147 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:44.142+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.143847+0000) 2019-10-13T07:55:44.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:44.144+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.145473+0000) 2019-10-13T07:55:44.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:44.147+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:47.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:47.829+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:47.829803+0000) 2019-10-13T07:55:48.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:48.378+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:48.378698+0000) 2019-10-13T07:55:48.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:48.715+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:48.716079+0000) 2019-10-13T07:55:48.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:48.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:48.716076+0000) 2019-10-13T07:55:49.152 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:49.148+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.150140+0000) 2019-10-13T07:55:49.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:49.149+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.150547+0000) 2019-10-13T07:55:49.153 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:49.149+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.149756+0000) 2019-10-13T07:55:49.153 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:49.149+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.150752+0000) 2019-10-13T07:55:49.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:49.149+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.150926+0000) 2019-10-13T07:55:49.153 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:49.149+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.151364+0000) 2019-10-13T07:55:49.154 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:49.149+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.151145+0000) 2019-10-13T07:55:49.156 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:49.152+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.152899+0000) 2019-10-13T07:55:49.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:49.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:51.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:51.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:51.969596+0000) 2019-10-13T07:55:52.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:52.398+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:52.399245+0000) 2019-10-13T07:55:52.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:52.829+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:52.830941+0000) 2019-10-13T07:55:52.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:52.831+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:52.831793+0000) 2019-10-13T07:55:54.159 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:54.156+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.157887+0000) 2019-10-13T07:55:54.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:54.156+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.157395+0000) 2019-10-13T07:55:54.160 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:54.157+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.158510+0000) 2019-10-13T07:55:54.161 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:54.157+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.158726+0000) 2019-10-13T07:55:54.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:54.157+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.158926+0000) 2019-10-13T07:55:54.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:54.157+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.159020+0000) 2019-10-13T07:55:54.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:54.157+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.159218+0000) 2019-10-13T07:55:54.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:54.158+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.159692+0000) 2019-10-13T07:55:54.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:54.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:55:54.655 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:55:54.655 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:54.660 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:55:54.660 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:57.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:57.829+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:57.829994+0000) 2019-10-13T07:55:58.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:58.378+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:58.378866+0000) 2019-10-13T07:55:58.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:58.715+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:58.716277+0000) 2019-10-13T07:55:58.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:58.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:58.716277+0000) 2019-10-13T07:55:59.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:55:59.164+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.165156+0000) 2019-10-13T07:55:59.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:55:59.164+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.165364+0000) 2019-10-13T07:55:59.168 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:55:59.164+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.165200+0000) 2019-10-13T07:55:59.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:55:59.165+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.165664+0000) 2019-10-13T07:55:59.169 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:55:59.165+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.166125+0000) 2019-10-13T07:55:59.169 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:55:59.165+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.166380+0000) 2019-10-13T07:55:59.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:55:59.166+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.166718+0000) 2019-10-13T07:55:59.170 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:55:59.167+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.167873+0000) 2019-10-13T07:55:59.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:55:59.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132193 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:01.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:01.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:01.969767+0000) 2019-10-13T07:56:02.410 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:02.401+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:02.401725+0000) 2019-10-13T07:56:02.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:02.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:02.831124+0000) 2019-10-13T07:56:02.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:02.831+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:02.831980+0000) 2019-10-13T07:56:04.174 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:04.171+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.172327+0000) 2019-10-13T07:56:04.174 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:04.172+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.172875+0000) 2019-10-13T07:56:04.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:04.172+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.173185+0000) 2019-10-13T07:56:04.175 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:04.172+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.173017+0000) 2019-10-13T07:56:04.175 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:04.171+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.172630+0000) 2019-10-13T07:56:04.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:04.173+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.173487+0000) 2019-10-13T07:56:04.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:04.173+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.174011+0000) 2019-10-13T07:56:04.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:04.173+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.174213+0000) 2019-10-13T07:56:04.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:04.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:07.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:07.829+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:07.830182+0000) 2019-10-13T07:56:08.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:08.378+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:08.379043+0000) 2019-10-13T07:56:08.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:08.716+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:08.716443+0000) 2019-10-13T07:56:08.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:08.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:08.716443+0000) 2019-10-13T07:56:09.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:09.179+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.179465+0000) 2019-10-13T07:56:09.182 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:09.179+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.179667+0000) 2019-10-13T07:56:09.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:09.179+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.180117+0000) 2019-10-13T07:56:09.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:09.179+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.180380+0000) 2019-10-13T07:56:09.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:09.178+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.179614+0000) 2019-10-13T07:56:09.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:09.180+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.180831+0000) 2019-10-13T07:56:09.184 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:09.180+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.180731+0000) 2019-10-13T07:56:09.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:09.181+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.182017+0000) 2019-10-13T07:56:09.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:09.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:11.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:11.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:11.969923+0000) 2019-10-13T07:56:12.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:12.403+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:12.403777+0000) 2019-10-13T07:56:12.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:12.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:12.831319+0000) 2019-10-13T07:56:12.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:12.831+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:12.832145+0000) 2019-10-13T07:56:14.188 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:14.185+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.185925+0000) 2019-10-13T07:56:14.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:14.185+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.186824+0000) 2019-10-13T07:56:14.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:14.185+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.187171+0000) 2019-10-13T07:56:14.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:14.186+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.187546+0000) 2019-10-13T07:56:14.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:14.186+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.187457+0000) 2019-10-13T07:56:14.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:14.186+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.187796+0000) 2019-10-13T07:56:14.190 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:14.186+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.188063+0000) 2019-10-13T07:56:14.190 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:14.188+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.188859+0000) 2019-10-13T07:56:14.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:14.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132289 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:17.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:17.828+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:17.830318+0000) 2019-10-13T07:56:18.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:18.377+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:18.379224+0000) 2019-10-13T07:56:18.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:18.715+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:18.716595+0000) 2019-10-13T07:56:18.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:18.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:18.716598+0000) 2019-10-13T07:56:19.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:19.192+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.194301+0000) 2019-10-13T07:56:19.196 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:19.193+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.194654+0000) 2019-10-13T07:56:19.197 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:19.193+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.194776+0000) 2019-10-13T07:56:19.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:19.193+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.195183+0000) 2019-10-13T07:56:19.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:19.193+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.195321+0000) 2019-10-13T07:56:19.197 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:19.193+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.195365+0000) 2019-10-13T07:56:19.198 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:19.194+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.195090+0000) 2019-10-13T07:56:19.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:19.196+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.197370+0000) 2019-10-13T07:56:19.201 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:19.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132319 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:21.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:21.968+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:21.970138+0000) 2019-10-13T07:56:22.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:22.406+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:22.406374+0000) 2019-10-13T07:56:22.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:22.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:22.831446+0000) 2019-10-13T07:56:22.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:22.830+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:22.832325+0000) 2019-10-13T07:56:24.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:24.200+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.201796+0000) 2019-10-13T07:56:24.204 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:24.200+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.201555+0000) 2019-10-13T07:56:24.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:24.200+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.201974+0000) 2019-10-13T07:56:24.204 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:24.200+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.202184+0000) 2019-10-13T07:56:24.204 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:24.201+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.202427+0000) 2019-10-13T07:56:24.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:24.200+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.202090+0000) 2019-10-13T07:56:24.205 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:24.201+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.202557+0000) 2019-10-13T07:56:24.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:24.202+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.204321+0000) 2019-10-13T07:56:24.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:24.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:24.755 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:56:24.755 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:24.760 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:56:24.761 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:27.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:27.829+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:27.830494+0000) 2019-10-13T07:56:28.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:28.378+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:28.379405+0000) 2019-10-13T07:56:28.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:28.716+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:28.716787+0000) 2019-10-13T07:56:28.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:28.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:28.716790+0000) 2019-10-13T07:56:29.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:29.207+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.208829+0000) 2019-10-13T07:56:29.211 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:29.207+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.209012+0000) 2019-10-13T07:56:29.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:29.207+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.208781+0000) 2019-10-13T07:56:29.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:29.207+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.209331+0000) 2019-10-13T07:56:29.211 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:29.208+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.209546+0000) 2019-10-13T07:56:29.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:29.207+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.209549+0000) 2019-10-13T07:56:29.212 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:29.208+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.209944+0000) 2019-10-13T07:56:29.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:29.208+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.210702+0000) 2019-10-13T07:56:29.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:29.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132383 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:31.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:31.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:31.970356+0000) 2019-10-13T07:56:32.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:32.407+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:32.408782+0000) 2019-10-13T07:56:32.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:32.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:32.831651+0000) 2019-10-13T07:56:32.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:32.831+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:32.832513+0000) 2019-10-13T07:56:34.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:34.214+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.216437+0000) 2019-10-13T07:56:34.219 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:34.214+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.215729+0000) 2019-10-13T07:56:34.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:34.214+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.216008+0000) 2019-10-13T07:56:34.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:34.215+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.216497+0000) 2019-10-13T07:56:34.220 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:34.215+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.216712+0000) 2019-10-13T07:56:34.220 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:34.215+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.216906+0000) 2019-10-13T07:56:34.220 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:34.215+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.217234+0000) 2019-10-13T07:56:34.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:34.215+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.217456+0000) 2019-10-13T07:56:34.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:34.219+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:37.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:37.830+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:37.830709+0000) 2019-10-13T07:56:38.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:38.379+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:38.379577+0000) 2019-10-13T07:56:38.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:38.716+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:38.717000+0000) 2019-10-13T07:56:38.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:38.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:38.717020+0000) 2019-10-13T07:56:39.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:39.221+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.222694+0000) 2019-10-13T07:56:39.224 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:39.221+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.222974+0000) 2019-10-13T07:56:39.225 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:39.221+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.222395+0000) 2019-10-13T07:56:39.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:39.221+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.223272+0000) 2019-10-13T07:56:39.226 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:39.222+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.223438+0000) 2019-10-13T07:56:39.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:39.222+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.223618+0000) 2019-10-13T07:56:39.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:39.222+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.224033+0000) 2019-10-13T07:56:39.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:39.224+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.225631+0000) 2019-10-13T07:56:39.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:39.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132445 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:41.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:41.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:41.970539+0000) 2019-10-13T07:56:42.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:42.410+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:42.411174+0000) 2019-10-13T07:56:42.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:42.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:42.831785+0000) 2019-10-13T07:56:42.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:42.831+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:42.832698+0000) 2019-10-13T07:56:44.231 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:44.228+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.229436+0000) 2019-10-13T07:56:44.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:44.229+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.230268+0000) 2019-10-13T07:56:44.232 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:44.230+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.230598+0000) 2019-10-13T07:56:44.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:44.230+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.230875+0000) 2019-10-13T07:56:44.233 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:44.230+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.231256+0000) 2019-10-13T07:56:44.233 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:44.231+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.231473+0000) 2019-10-13T07:56:44.233 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:44.231+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.231718+0000) 2019-10-13T07:56:44.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:44.232+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.232774+0000) 2019-10-13T07:56:44.237 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:44.235+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:47.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:47.830+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:47.830926+0000) 2019-10-13T07:56:48.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:48.379+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:48.379767+0000) 2019-10-13T07:56:48.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:48.716+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:48.717199+0000) 2019-10-13T07:56:48.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:48.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:48.717200+0000) 2019-10-13T07:56:49.238 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:49.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.236780+0000) 2019-10-13T07:56:49.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:49.237+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.237779+0000) 2019-10-13T07:56:49.240 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:49.237+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.238232+0000) 2019-10-13T07:56:49.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:49.238+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.238436+0000) 2019-10-13T07:56:49.240 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:49.238+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.238746+0000) 2019-10-13T07:56:49.240 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:49.238+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.238846+0000) 2019-10-13T07:56:49.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:49.238+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.239066+0000) 2019-10-13T07:56:49.241 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:49.238+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.238978+0000) 2019-10-13T07:56:49.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:49.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:51.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:51.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:51.970734+0000) 2019-10-13T07:56:52.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:52.412+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:52.413442+0000) 2019-10-13T07:56:52.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:52.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:52.831900+0000) 2019-10-13T07:56:52.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:52.832+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:52.832849+0000) 2019-10-13T07:56:54.247 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:54.244+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.245325+0000) 2019-10-13T07:56:54.247 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:54.245+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.245450+0000) 2019-10-13T07:56:54.247 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:54.245+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.245626+0000) 2019-10-13T07:56:54.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:54.245+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.245847+0000) 2019-10-13T07:56:54.248 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:54.245+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.245814+0000) 2019-10-13T07:56:54.249 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:54.245+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.246026+0000) 2019-10-13T07:56:54.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:54.246+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.246755+0000) 2019-10-13T07:56:54.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:54.247+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.247768+0000) 2019-10-13T07:56:54.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:54.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132541 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:56:54.890 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:56:54.890 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:54.894 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:56:54.895 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:57.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:57.831+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:57.831101+0000) 2019-10-13T07:56:58.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:58.379+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:58.379942+0000) 2019-10-13T07:56:58.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:58.717+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:58.717319+0000) 2019-10-13T07:56:58.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:58.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:58.717376+0000) 2019-10-13T07:56:59.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:56:59.252+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.252619+0000) 2019-10-13T07:56:59.254 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:56:59.251+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.252168+0000) 2019-10-13T07:56:59.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:56:59.252+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.253282+0000) 2019-10-13T07:56:59.255 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:56:59.253+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.253421+0000) 2019-10-13T07:56:59.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:56:59.253+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.253539+0000) 2019-10-13T07:56:59.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:56:59.253+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.254073+0000) 2019-10-13T07:56:59.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:56:59.253+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.254276+0000) 2019-10-13T07:56:59.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:56:59.253+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.254557+0000) 2019-10-13T07:56:59.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:56:59.257+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132571 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:01.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:01.969+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:01.970933+0000) 2019-10-13T07:57:02.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:02.415+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:02.415780+0000) 2019-10-13T07:57:02.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:02.830+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:02.832024+0000) 2019-10-13T07:57:02.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:02.832+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:02.833048+0000) 2019-10-13T07:57:04.261 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:04.258+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.258780+0000) 2019-10-13T07:57:04.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:04.258+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.259528+0000) 2019-10-13T07:57:04.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:04.258+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.259686+0000) 2019-10-13T07:57:04.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:04.258+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.259798+0000) 2019-10-13T07:57:04.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:04.258+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.260217+0000) 2019-10-13T07:57:04.262 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:04.259+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.260542+0000) 2019-10-13T07:57:04.262 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:04.259+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.260692+0000) 2019-10-13T07:57:04.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:04.260+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.261259+0000) 2019-10-13T07:57:04.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:04.263+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:07.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:07.829+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:07.831296+0000) 2019-10-13T07:57:08.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:08.379+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:08.380159+0000) 2019-10-13T07:57:08.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:08.716+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:08.717470+0000) 2019-10-13T07:57:08.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:08.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:08.717493+0000) 2019-10-13T07:57:09.269 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:09.265+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.266836+0000) 2019-10-13T07:57:09.269 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:09.265+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.267070+0000) 2019-10-13T07:57:09.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:09.265+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.267321+0000) 2019-10-13T07:57:09.270 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:09.265+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.266679+0000) 2019-10-13T07:57:09.271 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:09.266+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.267551+0000) 2019-10-13T07:57:09.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:09.266+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.267443+0000) 2019-10-13T07:57:09.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:09.266+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.267745+0000) 2019-10-13T07:57:09.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:09.267+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.268362+0000) 2019-10-13T07:57:09.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:09.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:11.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:11.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:11.971171+0000) 2019-10-13T07:57:12.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:12.417+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:12.418241+0000) 2019-10-13T07:57:12.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:12.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:12.832194+0000) 2019-10-13T07:57:12.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:12.833+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:12.833255+0000) 2019-10-13T07:57:14.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:14.271+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.272809+0000) 2019-10-13T07:57:14.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:14.271+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.273200+0000) 2019-10-13T07:57:14.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:14.272+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.273585+0000) 2019-10-13T07:57:14.276 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:14.272+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.273912+0000) 2019-10-13T07:57:14.276 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:14.272+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.273528+0000) 2019-10-13T07:57:14.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:14.272+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.274260+0000) 2019-10-13T07:57:14.277 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:14.273+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.274507+0000) 2019-10-13T07:57:14.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:14.275+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.275890+0000) 2019-10-13T07:57:14.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:14.277+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:17.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:17.830+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:17.831485+0000) 2019-10-13T07:57:18.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:18.378+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:18.380343+0000) 2019-10-13T07:57:18.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:18.717+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:18.717642+0000) 2019-10-13T07:57:18.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:18.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:18.717648+0000) 2019-10-13T07:57:19.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:19.278+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.280234+0000) 2019-10-13T07:57:19.282 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:19.278+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.280143+0000) 2019-10-13T07:57:19.283 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:19.278+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.280203+0000) 2019-10-13T07:57:19.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:19.279+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.280755+0000) 2019-10-13T07:57:19.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:19.279+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.280952+0000) 2019-10-13T07:57:19.283 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:19.279+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.281193+0000) 2019-10-13T07:57:19.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:19.279+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.281349+0000) 2019-10-13T07:57:19.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:19.280+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.282302+0000) 2019-10-13T07:57:19.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:19.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:21.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:21.970+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:21.971365+0000) 2019-10-13T07:57:22.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:22.419+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:22.420691+0000) 2019-10-13T07:57:22.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:22.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:22.832411+0000) 2019-10-13T07:57:22.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:22.831+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:22.833435+0000) 2019-10-13T07:57:24.289 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:24.285+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.287206+0000) 2019-10-13T07:57:24.289 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:24.286+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.287533+0000) 2019-10-13T07:57:24.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:24.286+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.287695+0000) 2019-10-13T07:57:24.290 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:24.285+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.287495+0000) 2019-10-13T07:57:24.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:24.286+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.288136+0000) 2019-10-13T07:57:24.291 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:24.287+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.288539+0000) 2019-10-13T07:57:24.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:24.287+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.288841+0000) 2019-10-13T07:57:24.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:24.288+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.289813+0000) 2019-10-13T07:57:24.294 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:24.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:25.003 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:57:25.003 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:25.008 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:57:25.008 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:27.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:27.831+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:27.831679+0000) 2019-10-13T07:57:28.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:28.380+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:28.380471+0000) 2019-10-13T07:57:28.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:28.717+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:28.717869+0000) 2019-10-13T07:57:28.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:28.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:28.717868+0000) 2019-10-13T07:57:29.296 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:29.294+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.294643+0000) 2019-10-13T07:57:29.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:29.294+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.295126+0000) 2019-10-13T07:57:29.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:29.292+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.294261+0000) 2019-10-13T07:57:29.297 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:29.295+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.295408+0000) 2019-10-13T07:57:29.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:29.295+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.295540+0000) 2019-10-13T07:57:29.298 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:29.295+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.295720+0000) 2019-10-13T07:57:29.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:29.295+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.296004+0000) 2019-10-13T07:57:29.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:29.294+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.296417+0000) 2019-10-13T07:57:29.301 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:29.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:31.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:31.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:31.971532+0000) 2019-10-13T07:57:32.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:32.422+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:32.423160+0000) 2019-10-13T07:57:32.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:32.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:32.832617+0000) 2019-10-13T07:57:32.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:32.832+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:32.833550+0000) 2019-10-13T07:57:34.304 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:34.301+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.302309+0000) 2019-10-13T07:57:34.304 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:34.302+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.302551+0000) 2019-10-13T07:57:34.304 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:34.302+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.302677+0000) 2019-10-13T07:57:34.305 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:34.301+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.302579+0000) 2019-10-13T07:57:34.305 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:34.302+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.303338+0000) 2019-10-13T07:57:34.305 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:34.303+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.303456+0000) 2019-10-13T07:57:34.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:34.303+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.303599+0000) 2019-10-13T07:57:34.306 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:34.303+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.304412+0000) 2019-10-13T07:57:34.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:34.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:37.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:37.831+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:37.831868+0000) 2019-10-13T07:57:38.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:38.380+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:38.380616+0000) 2019-10-13T07:57:38.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:38.717+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:38.718052+0000) 2019-10-13T07:57:38.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:38.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:38.718054+0000) 2019-10-13T07:57:39.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:39.308+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.309184+0000) 2019-10-13T07:57:39.311 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:39.308+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.309391+0000) 2019-10-13T07:57:39.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:39.309+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.309822+0000) 2019-10-13T07:57:39.312 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:39.309+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.310057+0000) 2019-10-13T07:57:39.312 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:39.309+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.310236+0000) 2019-10-13T07:57:39.312 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:39.309+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.310331+0000) 2019-10-13T07:57:39.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:39.310+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.310472+0000) 2019-10-13T07:57:39.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:39.310+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.311336+0000) 2019-10-13T07:57:39.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:39.314+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132823 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:41.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:41.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:41.971716+0000) 2019-10-13T07:57:42.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:42.425+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:42.425552+0000) 2019-10-13T07:57:42.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:42.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:42.832741+0000) 2019-10-13T07:57:42.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:42.832+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:42.833677+0000) 2019-10-13T07:57:44.318 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:44.315+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.316342+0000) 2019-10-13T07:57:44.318 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:44.316+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.316541+0000) 2019-10-13T07:57:44.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:44.315+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.316461+0000) 2019-10-13T07:57:44.319 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:44.316+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.316942+0000) 2019-10-13T07:57:44.319 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:44.317+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.317428+0000) 2019-10-13T07:57:44.319 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:44.317+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.317629+0000) 2019-10-13T07:57:44.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:44.317+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.317793+0000) 2019-10-13T07:57:44.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:44.318+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.319132+0000) 2019-10-13T07:57:44.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:44.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132855 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:47.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:47.831+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:47.832046+0000) 2019-10-13T07:57:48.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:48.379+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:48.380828+0000) 2019-10-13T07:57:48.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:48.717+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:48.718231+0000) 2019-10-13T07:57:48.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:48.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:48.718235+0000) 2019-10-13T07:57:49.326 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:49.322+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.323776+0000) 2019-10-13T07:57:49.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:49.322+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.324092+0000) 2019-10-13T07:57:49.326 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:49.322+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.324340+0000) 2019-10-13T07:57:49.327 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:49.323+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.324491+0000) 2019-10-13T07:57:49.327 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:49.323+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.324667+0000) 2019-10-13T07:57:49.328 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:49.323+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.324624+0000) 2019-10-13T07:57:49.329 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:49.324+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.325678+0000) 2019-10-13T07:57:49.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:49.323+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.325065+0000) 2019-10-13T07:57:49.330 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:49.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:51.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:51.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:51.971924+0000) 2019-10-13T07:57:52.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:52.427+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:52.427908+0000) 2019-10-13T07:57:52.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:52.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:52.832910+0000) 2019-10-13T07:57:52.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:52.833+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:52.833824+0000) 2019-10-13T07:57:54.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:54.329+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.331228+0000) 2019-10-13T07:57:54.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:54.330+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.331015+0000) 2019-10-13T07:57:54.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:54.330+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.331557+0000) 2019-10-13T07:57:54.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:54.330+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.331836+0000) 2019-10-13T07:57:54.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:54.330+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.332149+0000) 2019-10-13T07:57:54.334 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:54.330+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.332301+0000) 2019-10-13T07:57:54.334 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:54.331+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.332526+0000) 2019-10-13T07:57:54.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:54.332+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.333517+0000) 2019-10-13T07:57:54.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:54.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:57:55.134 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:57:55.134 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:55.138 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:57:55.138 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:57.833 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:57.830+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:57.832254+0000) 2019-10-13T07:57:58.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:58.380+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:58.381006+0000) 2019-10-13T07:57:58.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:58.717+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:58.718465+0000) 2019-10-13T07:57:58.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:58.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:58.718467+0000) 2019-10-13T07:57:59.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:57:59.336+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.338383+0000) 2019-10-13T07:57:59.340 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:57:59.337+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.338538+0000) 2019-10-13T07:57:59.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:57:59.337+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.338861+0000) 2019-10-13T07:57:59.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:57:59.337+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.339291+0000) 2019-10-13T07:57:59.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:57:59.337+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.339331+0000) 2019-10-13T07:57:59.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:57:59.338+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.339757+0000) 2019-10-13T07:57:59.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:57:59.338+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.338931+0000) 2019-10-13T07:57:59.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:57:59.340+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.341407+0000) 2019-10-13T07:57:59.345 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:57:59.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:01.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:01.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:01.972092+0000) 2019-10-13T07:58:02.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:02.430+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:02.430213+0000) 2019-10-13T07:58:02.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:02.831+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:02.833106+0000) 2019-10-13T07:58:02.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:02.833+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:02.833971+0000) 2019-10-13T07:58:04.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:04.344+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.346089+0000) 2019-10-13T07:58:04.348 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:04.344+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.346314+0000) 2019-10-13T07:58:04.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:04.344+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.346295+0000) 2019-10-13T07:58:04.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:04.345+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.346622+0000) 2019-10-13T07:58:04.348 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:04.345+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.346749+0000) 2019-10-13T07:58:04.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:04.345+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.345823+0000) 2019-10-13T07:58:04.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:04.345+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.346885+0000) 2019-10-13T07:58:04.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:04.347+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.348500+0000) 2019-10-13T07:58:04.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:04.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 132981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:07.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:07.831+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:07.832447+0000) 2019-10-13T07:58:08.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:08.380+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:08.381229+0000) 2019-10-13T07:58:08.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:08.718+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:08.718613+0000) 2019-10-13T07:58:08.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:08.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:08.718646+0000) 2019-10-13T07:58:09.352 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:09.349+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.350989+0000) 2019-10-13T07:58:09.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:09.349+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.351155+0000) 2019-10-13T07:58:09.353 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:09.349+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.351291+0000) 2019-10-13T07:58:09.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:09.350+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.351499+0000) 2019-10-13T07:58:09.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:09.349+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.351366+0000) 2019-10-13T07:58:09.354 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:09.350+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.351740+0000) 2019-10-13T07:58:09.354 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:09.350+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.352076+0000) 2019-10-13T07:58:09.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:09.352+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.353824+0000) 2019-10-13T07:58:09.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:09.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:11.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:11.971+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:11.972262+0000) 2019-10-13T07:58:12.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:12.431+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:12.432638+0000) 2019-10-13T07:58:12.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:12.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:12.833247+0000) 2019-10-13T07:58:12.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:12.833+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:12.834140+0000) 2019-10-13T07:58:14.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:14.356+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.356974+0000) 2019-10-13T07:58:14.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:14.356+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.357087+0000) 2019-10-13T07:58:14.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:14.356+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.357325+0000) 2019-10-13T07:58:14.359 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:14.357+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.357501+0000) 2019-10-13T07:58:14.360 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:14.355+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.357723+0000) 2019-10-13T07:58:14.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:14.357+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.357698+0000) 2019-10-13T07:58:14.360 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:14.357+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.358179+0000) 2019-10-13T07:58:14.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:14.357+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.359653+0000) 2019-10-13T07:58:14.363 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:14.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133045 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:17.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:17.832+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:17.832612+0000) 2019-10-13T07:58:18.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:18.381+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:18.381435+0000) 2019-10-13T07:58:18.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:18.718+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:18.718766+0000) 2019-10-13T07:58:18.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:18.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:18.718784+0000) 2019-10-13T07:58:19.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:19.363+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.363431+0000) 2019-10-13T07:58:19.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:19.363+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.363731+0000) 2019-10-13T07:58:19.366 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:19.363+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.363783+0000) 2019-10-13T07:58:19.366 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:19.363+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.364057+0000) 2019-10-13T07:58:19.367 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:19.362+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.364599+0000) 2019-10-13T07:58:19.367 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:19.364+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.364768+0000) 2019-10-13T07:58:19.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:19.364+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.364926+0000) 2019-10-13T07:58:19.368 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:19.364+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.366691+0000) 2019-10-13T07:58:19.370 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:19.368+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:21.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:21.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:21.972458+0000) 2019-10-13T07:58:22.437 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:22.434+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:22.434981+0000) 2019-10-13T07:58:22.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:22.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:22.833459+0000) 2019-10-13T07:58:22.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:22.833+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:22.834319+0000) 2019-10-13T07:58:24.372 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:24.370+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.371183+0000) 2019-10-13T07:58:24.373 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:24.371+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.371430+0000) 2019-10-13T07:58:24.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:24.371+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.371689+0000) 2019-10-13T07:58:24.374 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:24.370+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.371433+0000) 2019-10-13T07:58:24.374 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:24.371+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.371862+0000) 2019-10-13T07:58:24.374 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:24.372+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.372479+0000) 2019-10-13T07:58:24.374 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:24.371+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.372372+0000) 2019-10-13T07:58:24.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:24.372+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.373585+0000) 2019-10-13T07:58:24.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:24.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:25.273 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:58:25.273 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:25.277 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:58:25.277 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:27.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:27.832+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:27.832767+0000) 2019-10-13T07:58:28.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:28.381+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:28.381625+0000) 2019-10-13T07:58:28.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:28.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:28.718944+0000) 2019-10-13T07:58:28.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:28.718+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:28.718944+0000) 2019-10-13T07:58:29.380 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:29.378+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.378675+0000) 2019-10-13T07:58:29.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:29.378+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.378786+0000) 2019-10-13T07:58:29.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:29.378+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.379012+0000) 2019-10-13T07:58:29.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:29.377+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.378350+0000) 2019-10-13T07:58:29.382 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:29.379+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.379491+0000) 2019-10-13T07:58:29.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:29.379+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.380020+0000) 2019-10-13T07:58:29.382 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:29.379+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.380106+0000) 2019-10-13T07:58:29.383 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:29.379+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.380700+0000) 2019-10-13T07:58:29.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:29.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:31.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:31.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:31.972667+0000) 2019-10-13T07:58:32.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:32.436+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:32.437267+0000) 2019-10-13T07:58:32.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:32.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:32.833611+0000) 2019-10-13T07:58:32.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:32.833+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:32.834484+0000) 2019-10-13T07:58:34.387 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:34.384+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.385432+0000) 2019-10-13T07:58:34.388 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:34.384+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.386183+0000) 2019-10-13T07:58:34.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:34.385+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.386450+0000) 2019-10-13T07:58:34.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:34.385+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.386725+0000) 2019-10-13T07:58:34.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:34.385+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.387082+0000) 2019-10-13T07:58:34.389 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:34.385+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.387280+0000) 2019-10-13T07:58:34.389 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:34.386+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.387496+0000) 2019-10-13T07:58:34.390 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:34.387+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.388344+0000) 2019-10-13T07:58:34.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:34.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:37.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:37.832+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:37.832977+0000) 2019-10-13T07:58:38.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:38.380+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:38.381847+0000) 2019-10-13T07:58:38.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:38.718+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:38.719110+0000) 2019-10-13T07:58:38.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:38.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:38.719112+0000) 2019-10-13T07:58:39.394 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:39.391+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.392626+0000) 2019-10-13T07:58:39.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:39.392+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.393721+0000) 2019-10-13T07:58:39.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:39.392+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.393914+0000) 2019-10-13T07:58:39.396 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:39.392+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.394039+0000) 2019-10-13T07:58:39.396 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:39.392+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.394388+0000) 2019-10-13T07:58:39.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:39.392+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.394263+0000) 2019-10-13T07:58:39.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:39.393+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.395051+0000) 2019-10-13T07:58:39.398 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:39.395+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.396075+0000) 2019-10-13T07:58:39.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:39.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:41.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:41.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:41.972851+0000) 2019-10-13T07:58:42.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:42.438+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:42.439584+0000) 2019-10-13T07:58:42.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:42.832+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:42.833782+0000) 2019-10-13T07:58:42.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:42.834+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:42.834669+0000) 2019-10-13T07:58:44.401 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:44.399+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.399931+0000) 2019-10-13T07:58:44.402 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:44.399+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.400763+0000) 2019-10-13T07:58:44.403 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:44.399+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.401202+0000) 2019-10-13T07:58:44.403 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:44.399+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.401355+0000) 2019-10-13T07:58:44.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:44.400+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.401500+0000) 2019-10-13T07:58:44.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:44.400+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.401711+0000) 2019-10-13T07:58:44.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:44.400+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.402205+0000) 2019-10-13T07:58:44.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:44.401+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.402532+0000) 2019-10-13T07:58:44.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:44.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:47.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:47.832+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:47.833151+0000) 2019-10-13T07:58:48.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:48.381+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:48.382074+0000) 2019-10-13T07:58:48.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:48.718+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:48.719339+0000) 2019-10-13T07:58:48.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:48.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:48.719340+0000) 2019-10-13T07:58:49.408 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:49.405+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.406999+0000) 2019-10-13T07:58:49.409 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:49.405+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.406651+0000) 2019-10-13T07:58:49.409 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:49.406+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.407502+0000) 2019-10-13T07:58:49.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:49.406+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.407737+0000) 2019-10-13T07:58:49.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:49.406+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.407901+0000) 2019-10-13T07:58:49.410 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:49.406+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.407979+0000) 2019-10-13T07:58:49.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:49.406+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.407973+0000) 2019-10-13T07:58:49.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:49.408+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.408892+0000) 2019-10-13T07:58:49.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:49.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:51.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:51.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:51.973062+0000) 2019-10-13T07:58:52.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:52.441+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:52.441846+0000) 2019-10-13T07:58:52.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:52.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:52.833904+0000) 2019-10-13T07:58:52.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:52.834+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:52.834853+0000) 2019-10-13T07:58:54.415 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:54.411+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.413297+0000) 2019-10-13T07:58:54.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:54.411+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.413387+0000) 2019-10-13T07:58:54.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:54.412+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.413455+0000) 2019-10-13T07:58:54.416 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:54.412+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.413625+0000) 2019-10-13T07:58:54.416 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:54.412+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.413772+0000) 2019-10-13T07:58:54.416 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:54.412+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.414047+0000) 2019-10-13T07:58:54.416 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:54.413+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.413797+0000) 2019-10-13T07:58:54.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:54.414+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.415473+0000) 2019-10-13T07:58:54.420 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:54.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:58:55.444 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:58:55.444 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:55.449 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:58:55.449 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:57.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:57.832+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:57.833317+0000) 2019-10-13T07:58:58.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:58.381+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:58.382302+0000) 2019-10-13T07:58:58.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:58.719+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:58.719535+0000) 2019-10-13T07:58:58.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:58.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:58.719537+0000) 2019-10-13T07:58:59.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:58:59.418+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.420216+0000) 2019-10-13T07:58:59.422 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:58:59.419+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.420722+0000) 2019-10-13T07:58:59.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:58:59.419+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.420934+0000) 2019-10-13T07:58:59.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:58:59.419+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.421150+0000) 2019-10-13T07:58:59.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:58:59.419+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.420510+0000) 2019-10-13T07:58:59.423 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:58:59.419+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.421273+0000) 2019-10-13T07:58:59.423 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:58:59.419+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.421386+0000) 2019-10-13T07:58:59.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:58:59.421+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.421990+0000) 2019-10-13T07:58:59.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:58:59.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133327 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:01.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:01.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:01.973260+0000) 2019-10-13T07:59:02.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:02.442+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:02.444049+0000) 2019-10-13T07:59:02.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:02.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:02.834043+0000) 2019-10-13T07:59:02.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:02.833+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:02.834989+0000) 2019-10-13T07:59:04.429 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:04.427+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.427484+0000) 2019-10-13T07:59:04.429 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:04.427+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.427581+0000) 2019-10-13T07:59:04.430 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:04.427+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.427865+0000) 2019-10-13T07:59:04.430 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:04.427+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.428043+0000) 2019-10-13T07:59:04.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:04.427+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.428320+0000) 2019-10-13T07:59:04.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:04.425+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.427668+0000) 2019-10-13T07:59:04.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:04.428+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.428661+0000) 2019-10-13T07:59:04.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:04.428+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.430078+0000) 2019-10-13T07:59:04.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:04.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133359 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:07.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:07.833+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:07.833483+0000) 2019-10-13T07:59:08.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:08.382+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:08.382458+0000) 2019-10-13T07:59:08.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:08.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:08.719644+0000) 2019-10-13T07:59:08.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:08.719+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:08.719683+0000) 2019-10-13T07:59:09.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:09.431+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.433662+0000) 2019-10-13T07:59:09.436 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:09.434+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.434491+0000) 2019-10-13T07:59:09.437 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:09.434+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.434691+0000) 2019-10-13T07:59:09.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:09.434+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.434589+0000) 2019-10-13T07:59:09.437 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:09.434+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.434799+0000) 2019-10-13T07:59:09.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:09.434+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.434943+0000) 2019-10-13T07:59:09.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:09.434+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.435159+0000) 2019-10-13T07:59:09.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:09.435+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.437619+0000) 2019-10-13T07:59:09.439 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:09.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:11.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:11.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:11.973469+0000) 2019-10-13T07:59:12.448 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:12.445+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:12.446579+0000) 2019-10-13T07:59:12.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:12.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:12.834210+0000) 2019-10-13T07:59:12.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:12.834+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:12.835121+0000) 2019-10-13T07:59:14.441 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:14.439+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.440088+0000) 2019-10-13T07:59:14.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:14.439+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.440367+0000) 2019-10-13T07:59:14.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:14.437+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.439677+0000) 2019-10-13T07:59:14.443 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:14.440+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.441056+0000) 2019-10-13T07:59:14.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:14.440+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.441230+0000) 2019-10-13T07:59:14.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:14.441+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.441445+0000) 2019-10-13T07:59:14.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:14.441+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.441841+0000) 2019-10-13T07:59:14.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:14.440+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.442402+0000) 2019-10-13T07:59:14.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:14.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:17.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:17.833+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:17.833674+0000) 2019-10-13T07:59:18.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:18.382+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:18.382626+0000) 2019-10-13T07:59:18.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:18.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:18.719797+0000) 2019-10-13T07:59:18.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:18.719+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:18.719832+0000) 2019-10-13T07:59:19.449 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:19.446+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.447829+0000) 2019-10-13T07:59:19.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:19.446+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.448059+0000) 2019-10-13T07:59:19.450 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:19.446+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.448316+0000) 2019-10-13T07:59:19.450 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:19.447+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.448507+0000) 2019-10-13T07:59:19.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:19.447+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.448146+0000) 2019-10-13T07:59:19.451 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:19.448+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.449416+0000) 2019-10-13T07:59:19.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:19.448+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.449723+0000) 2019-10-13T07:59:19.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:19.449+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:19.450493+0000) 2019-10-13T07:59:19.455 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:19.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133453 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:21.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:21.972+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:21.973661+0000) 2019-10-13T07:59:22.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:22.448+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:22.449021+0000) 2019-10-13T07:59:22.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:22.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:22.834426+0000) 2019-10-13T07:59:22.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:22.834+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:22.835259+0000) 2019-10-13T07:59:24.457 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:24.454+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.455790+0000) 2019-10-13T07:59:24.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:24.454+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.455821+0000) 2019-10-13T07:59:24.458 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:24.454+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.456072+0000) 2019-10-13T07:59:24.458 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:24.454+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.455653+0000) 2019-10-13T07:59:24.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:24.454+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.456356+0000) 2019-10-13T07:59:24.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:24.455+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.456620+0000) 2019-10-13T07:59:24.459 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:24.455+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.456920+0000) 2019-10-13T07:59:24.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:24.456+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.457489+0000) 2019-10-13T07:59:24.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:24.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:25.564 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:59:25.564 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:25.568 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:59:25.569 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:27.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:27.833+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:27.833876+0000) 2019-10-13T07:59:28.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:28.381+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:28.382806+0000) 2019-10-13T07:59:28.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:28.719+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:28.719996+0000) 2019-10-13T07:59:28.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:28.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:28.719997+0000) 2019-10-13T07:59:29.464 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:29.461+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.462636+0000) 2019-10-13T07:59:29.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:29.462+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.463728+0000) 2019-10-13T07:59:29.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:29.462+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.463867+0000) 2019-10-13T07:59:29.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:29.462+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.464020+0000) 2019-10-13T07:59:29.466 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:29.462+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.464185+0000) 2019-10-13T07:59:29.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:29.463+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.464561+0000) 2019-10-13T07:59:29.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:29.463+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.464655+0000) 2019-10-13T07:59:29.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:29.464+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:29.465083+0000) 2019-10-13T07:59:29.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:29.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:31.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:31.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:31.973883+0000) 2019-10-13T07:59:32.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:32.450+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:32.451426+0000) 2019-10-13T07:59:32.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:32.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:32.834628+0000) 2019-10-13T07:59:32.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:32.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:32.835456+0000) 2019-10-13T07:59:34.472 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:34.469+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.471223+0000) 2019-10-13T07:59:34.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:34.470+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.471538+0000) 2019-10-13T07:59:34.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:34.470+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.471163+0000) 2019-10-13T07:59:34.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:34.470+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.471992+0000) 2019-10-13T07:59:34.474 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:34.470+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.472223+0000) 2019-10-13T07:59:34.474 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:34.471+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.472436+0000) 2019-10-13T07:59:34.474 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:34.471+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.472857+0000) 2019-10-13T07:59:34.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:34.472+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.473036+0000) 2019-10-13T07:59:34.478 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:34.475+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:37.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:37.833+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:37.834065+0000) 2019-10-13T07:59:38.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:38.382+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:38.382978+0000) 2019-10-13T07:59:38.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:38.718+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:38.720182+0000) 2019-10-13T07:59:38.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:38.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:38.720200+0000) 2019-10-13T07:59:39.480 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:39.477+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.478713+0000) 2019-10-13T07:59:39.480 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:39.477+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.479117+0000) 2019-10-13T07:59:39.480 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:39.477+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.479301+0000) 2019-10-13T07:59:39.481 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:39.477+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.478535+0000) 2019-10-13T07:59:39.481 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:39.478+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.479421+0000) 2019-10-13T07:59:39.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:39.478+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.479675+0000) 2019-10-13T07:59:39.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:39.478+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.479821+0000) 2019-10-13T07:59:39.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:39.480+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:39.481109+0000) 2019-10-13T07:59:39.485 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:39.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:41.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:41.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:41.974085+0000) 2019-10-13T07:59:42.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:42.453+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:42.453906+0000) 2019-10-13T07:59:42.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:42.833+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:42.834748+0000) 2019-10-13T07:59:42.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:42.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:42.835630+0000) 2019-10-13T07:59:44.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:44.484+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.486276+0000) 2019-10-13T07:59:44.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:44.485+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.486649+0000) 2019-10-13T07:59:44.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:44.485+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.486771+0000) 2019-10-13T07:59:44.488 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:44.485+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.486926+0000) 2019-10-13T07:59:44.489 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:44.485+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.486678+0000) 2019-10-13T07:59:44.489 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:44.485+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.487273+0000) 2019-10-13T07:59:44.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:44.486+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.487634+0000) 2019-10-13T07:59:44.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:44.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.489379+0000) 2019-10-13T07:59:44.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:44.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:47.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:47.833+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:47.834261+0000) 2019-10-13T07:59:48.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:48.382+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:48.383189+0000) 2019-10-13T07:59:48.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:48.720+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:48.720431+0000) 2019-10-13T07:59:48.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:48.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:48.720429+0000) 2019-10-13T07:59:49.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:49.493+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.493828+0000) 2019-10-13T07:59:49.495 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:49.493+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.494145+0000) 2019-10-13T07:59:49.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:49.493+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.494227+0000) 2019-10-13T07:59:49.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:49.493+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.494633+0000) 2019-10-13T07:59:49.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:49.494+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.494491+0000) 2019-10-13T07:59:49.497 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:49.494+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.494626+0000) 2019-10-13T07:59:49.497 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:49.494+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.495342+0000) 2019-10-13T07:59:49.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:49.495+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:49.495977+0000) 2019-10-13T07:59:49.501 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:49.499+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:51.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:51.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:51.974277+0000) 2019-10-13T07:59:52.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:52.456+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:52.456338+0000) 2019-10-13T07:59:52.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:52.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:52.834868+0000) 2019-10-13T07:59:52.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:52.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:52.835752+0000) 2019-10-13T07:59:54.503 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:54.501+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.501717+0000) 2019-10-13T07:59:54.504 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:54.500+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.501679+0000) 2019-10-13T07:59:54.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:54.501+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.502168+0000) 2019-10-13T07:59:54.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:54.502+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.502726+0000) 2019-10-13T07:59:54.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:54.502+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.502607+0000) 2019-10-13T07:59:54.504 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:54.502+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.502910+0000) 2019-10-13T07:59:54.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:54.503+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.503389+0000) 2019-10-13T07:59:54.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:54.503+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.504610+0000) 2019-10-13T07:59:54.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:54.507+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T07:59:55.869 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T07:59:55.869 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:55.873 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T07:59:55.873 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:57.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:57.834+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:57.834445+0000) 2019-10-13T07:59:58.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:58.383+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:58.383379+0000) 2019-10-13T07:59:58.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:58.720+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:58.720629+0000) 2019-10-13T07:59:58.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:58.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:58.720651+0000) 2019-10-13T07:59:59.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T07:59:59.508+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.509315+0000) 2019-10-13T07:59:59.511 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T07:59:59.509+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.509560+0000) 2019-10-13T07:59:59.512 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T07:59:59.507+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.509329+0000) 2019-10-13T07:59:59.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T07:59:59.509+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.510057+0000) 2019-10-13T07:59:59.512 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T07:59:59.510+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.510466+0000) 2019-10-13T07:59:59.512 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T07:59:59.510+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.510810+0000) 2019-10-13T07:59:59.513 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T07:59:59.510+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.511059+0000) 2019-10-13T07:59:59.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T07:59:59.510+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:59.512133+0000) 2019-10-13T07:59:59.516 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T07:59:59.514+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:01.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:01.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:01.974467+0000) 2019-10-13T08:00:02.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:02.457+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:02.458665+0000) 2019-10-13T08:00:02.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:02.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:02.835038+0000) 2019-10-13T08:00:02.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:02.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:02.835906+0000) 2019-10-13T08:00:04.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:04.516+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.517058+0000) 2019-10-13T08:00:04.518 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:04.517+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.517409+0000) 2019-10-13T08:00:04.519 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:04.515+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.516752+0000) 2019-10-13T08:00:04.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:04.517+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.517692+0000) 2019-10-13T08:00:04.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:04.517+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.517947+0000) 2019-10-13T08:00:04.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:04.518+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.518659+0000) 2019-10-13T08:00:04.520 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:04.518+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.518851+0000) 2019-10-13T08:00:04.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:04.517+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.519720+0000) 2019-10-13T08:00:04.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:04.522+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:07.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:07.833+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:07.834619+0000) 2019-10-13T08:00:08.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:08.382+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:08.383571+0000) 2019-10-13T08:00:08.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:08.719+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:08.720823+0000) 2019-10-13T08:00:08.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:08.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:08.720825+0000) 2019-10-13T08:00:09.526 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:09.523+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.524711+0000) 2019-10-13T08:00:09.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:09.523+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.525318+0000) 2019-10-13T08:00:09.528 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:09.524+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.525535+0000) 2019-10-13T08:00:09.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:09.524+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.525914+0000) 2019-10-13T08:00:09.528 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:09.524+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.525782+0000) 2019-10-13T08:00:09.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:09.524+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.526311+0000) 2019-10-13T08:00:09.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:09.525+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.526565+0000) 2019-10-13T08:00:09.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:09.526+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:09.527276+0000) 2019-10-13T08:00:09.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:09.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:11.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:11.973+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:11.974663+0000) 2019-10-13T08:00:12.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:12.460+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:12.461197+0000) 2019-10-13T08:00:12.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:12.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:12.835213+0000) 2019-10-13T08:00:12.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:12.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:12.836086+0000) 2019-10-13T08:00:14.533 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:14.531+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.531753+0000) 2019-10-13T08:00:14.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:14.531+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.532405+0000) 2019-10-13T08:00:14.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:14.531+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.532537+0000) 2019-10-13T08:00:14.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:14.531+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.532836+0000) 2019-10-13T08:00:14.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:14.531+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.532732+0000) 2019-10-13T08:00:14.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:14.531+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.533269+0000) 2019-10-13T08:00:14.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:14.532+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.533484+0000) 2019-10-13T08:00:14.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:14.533+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.534524+0000) 2019-10-13T08:00:14.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:14.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:17.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:17.834+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:17.834812+0000) 2019-10-13T08:00:18.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:18.383+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:18.383763+0000) 2019-10-13T08:00:18.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:18.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:18.720976+0000) 2019-10-13T08:00:18.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:18.720+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:18.721005+0000) 2019-10-13T08:00:19.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:19.538+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.539448+0000) 2019-10-13T08:00:19.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:19.538+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.539190+0000) 2019-10-13T08:00:19.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:19.538+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.540307+0000) 2019-10-13T08:00:19.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:19.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.540604+0000) 2019-10-13T08:00:19.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:19.539+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.540736+0000) 2019-10-13T08:00:19.542 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:19.539+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.541003+0000) 2019-10-13T08:00:19.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:19.539+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.541075+0000) 2019-10-13T08:00:19.543 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:19.541+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:19.541953+0000) 2019-10-13T08:00:19.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:19.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:21.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:21.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:21.974889+0000) 2019-10-13T08:00:22.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:22.462+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:22.463303+0000) 2019-10-13T08:00:22.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:22.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:22.835410+0000) 2019-10-13T08:00:22.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:22.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:22.836287+0000) 2019-10-13T08:00:24.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:24.545+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.547297+0000) 2019-10-13T08:00:24.549 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:24.546+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.547111+0000) 2019-10-13T08:00:24.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:24.546+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.547849+0000) 2019-10-13T08:00:24.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:24.546+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.547969+0000) 2019-10-13T08:00:24.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:24.546+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.548257+0000) 2019-10-13T08:00:24.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:24.547+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.548431+0000) 2019-10-13T08:00:24.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:24.547+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.548512+0000) 2019-10-13T08:00:24.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:24.548+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.549503+0000) 2019-10-13T08:00:24.554 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:24.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:25.957 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:00:25.957 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:25.962 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:00:25.962 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:27.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:27.834+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:27.835007+0000) 2019-10-13T08:00:28.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:28.383+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:28.383877+0000) 2019-10-13T08:00:28.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:28.720+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:28.721188+0000) 2019-10-13T08:00:28.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:28.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:28.721182+0000) 2019-10-13T08:00:29.556 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:29.553+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.554621+0000) 2019-10-13T08:00:29.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:29.554+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.555505+0000) 2019-10-13T08:00:29.557 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:29.553+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.555351+0000) 2019-10-13T08:00:29.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:29.554+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.555838+0000) 2019-10-13T08:00:29.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:29.554+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.555787+0000) 2019-10-13T08:00:29.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:29.554+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.555991+0000) 2019-10-13T08:00:29.558 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:29.555+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.556460+0000) 2019-10-13T08:00:29.559 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:29.555+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:29.556550+0000) 2019-10-13T08:00:29.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:29.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:31.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:31.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:31.975115+0000) 2019-10-13T08:00:32.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:32.465+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:32.465635+0000) 2019-10-13T08:00:32.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:32.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:32.835596+0000) 2019-10-13T08:00:32.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:32.836+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:32.836433+0000) 2019-10-13T08:00:34.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:34.562+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.562644+0000) 2019-10-13T08:00:34.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:34.562+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.562704+0000) 2019-10-13T08:00:34.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:34.562+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.563034+0000) 2019-10-13T08:00:34.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:34.561+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.562675+0000) 2019-10-13T08:00:34.566 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:34.563+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.563811+0000) 2019-10-13T08:00:34.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:34.563+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.563743+0000) 2019-10-13T08:00:34.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:34.563+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.564234+0000) 2019-10-13T08:00:34.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:34.564+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.564935+0000) 2019-10-13T08:00:34.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:34.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:37.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:37.834+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:37.835239+0000) 2019-10-13T08:00:38.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:38.383+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:38.384045+0000) 2019-10-13T08:00:38.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:38.721+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:38.721390+0000) 2019-10-13T08:00:38.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:38.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:38.721374+0000) 2019-10-13T08:00:39.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:39.568+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.569593+0000) 2019-10-13T08:00:39.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:39.570+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.570639+0000) 2019-10-13T08:00:39.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:39.570+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.570636+0000) 2019-10-13T08:00:39.573 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:39.570+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.570901+0000) 2019-10-13T08:00:39.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:39.570+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.571250+0000) 2019-10-13T08:00:39.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:39.570+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.571374+0000) 2019-10-13T08:00:39.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:39.571+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.571551+0000) 2019-10-13T08:00:39.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:39.571+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:39.571834+0000) 2019-10-13T08:00:39.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:39.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133957 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:41.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:41.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:41.975309+0000) 2019-10-13T08:00:42.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:42.468+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:42.468746+0000) 2019-10-13T08:00:42.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:42.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:42.835778+0000) 2019-10-13T08:00:42.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:42.836+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:42.836569+0000) 2019-10-13T08:00:44.579 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:44.576+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.577281+0000) 2019-10-13T08:00:44.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:44.577+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.577971+0000) 2019-10-13T08:00:44.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:44.577+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.578084+0000) 2019-10-13T08:00:44.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:44.577+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.578286+0000) 2019-10-13T08:00:44.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:44.578+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.578488+0000) 2019-10-13T08:00:44.581 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:44.578+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.578720+0000) 2019-10-13T08:00:44.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:44.578+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.579137+0000) 2019-10-13T08:00:44.582 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:44.580+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.580905+0000) 2019-10-13T08:00:44.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:44.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 133989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:47.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:47.835+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:47.835441+0000) 2019-10-13T08:00:48.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:48.384+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:48.384209+0000) 2019-10-13T08:00:48.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:48.721+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:48.721611+0000) 2019-10-13T08:00:48.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:48.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:48.721606+0000) 2019-10-13T08:00:49.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:49.584+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.584652+0000) 2019-10-13T08:00:49.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:49.584+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.584825+0000) 2019-10-13T08:00:49.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:49.582+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.584399+0000) 2019-10-13T08:00:49.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:49.584+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.585245+0000) 2019-10-13T08:00:49.588 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:49.585+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.585395+0000) 2019-10-13T08:00:49.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:49.585+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.585673+0000) 2019-10-13T08:00:49.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:49.585+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.586126+0000) 2019-10-13T08:00:49.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:49.584+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:49.586380+0000) 2019-10-13T08:00:49.592 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:49.589+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:51.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:51.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:51.975496+0000) 2019-10-13T08:00:52.473 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:52.470+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:52.471175+0000) 2019-10-13T08:00:52.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:52.834+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:52.835963+0000) 2019-10-13T08:00:52.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:52.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:52.836743+0000) 2019-10-13T08:00:54.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:54.589+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.591576+0000) 2019-10-13T08:00:54.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:54.590+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.592214+0000) 2019-10-13T08:00:54.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:54.590+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.592362+0000) 2019-10-13T08:00:54.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:54.591+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.592693+0000) 2019-10-13T08:00:54.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:54.591+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.593012+0000) 2019-10-13T08:00:54.595 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:54.591+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.593073+0000) 2019-10-13T08:00:54.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:54.592+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.593568+0000) 2019-10-13T08:00:54.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:54.592+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.594052+0000) 2019-10-13T08:00:54.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:54.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:00:56.119 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:00:56.119 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:56.124 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:00:56.124 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:57.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:57.834+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:57.835599+0000) 2019-10-13T08:00:58.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:58.383+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:58.384436+0000) 2019-10-13T08:00:58.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:58.720+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:58.721817+0000) 2019-10-13T08:00:58.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:58.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:58.721826+0000) 2019-10-13T08:00:59.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:00:59.598+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.599742+0000) 2019-10-13T08:00:59.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:00:59.598+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.599961+0000) 2019-10-13T08:00:59.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:00:59.598+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.600242+0000) 2019-10-13T08:00:59.602 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:00:59.598+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.600383+0000) 2019-10-13T08:00:59.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:00:59.597+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.599575+0000) 2019-10-13T08:00:59.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:00:59.599+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.600994+0000) 2019-10-13T08:00:59.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:00:59.599+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.601160+0000) 2019-10-13T08:00:59.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:00:59.601+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:59.602962+0000) 2019-10-13T08:00:59.606 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:00:59.603+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:01.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:01.974+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:01.975662+0000) 2019-10-13T08:01:02.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:02.472+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:02.473633+0000) 2019-10-13T08:01:02.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:02.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:02.836124+0000) 2019-10-13T08:01:02.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:02.836+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:02.836914+0000) 2019-10-13T08:01:04.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:04.605+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.606595+0000) 2019-10-13T08:01:04.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:04.606+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.607561+0000) 2019-10-13T08:01:04.610 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:04.606+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.607783+0000) 2019-10-13T08:01:04.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:04.606+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.608000+0000) 2019-10-13T08:01:04.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:04.606+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.608155+0000) 2019-10-13T08:01:04.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:04.606+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.608385+0000) 2019-10-13T08:01:04.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:04.607+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.608562+0000) 2019-10-13T08:01:04.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:04.608+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.608884+0000) 2019-10-13T08:01:04.614 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:04.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134115 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:07.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:07.835+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:07.835774+0000) 2019-10-13T08:01:08.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:08.383+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:08.384593+0000) 2019-10-13T08:01:08.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:08.721+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:08.722003+0000) 2019-10-13T08:01:08.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:08.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:08.722009+0000) 2019-10-13T08:01:09.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:09.613+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.613876+0000) 2019-10-13T08:01:09.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:09.613+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.614804+0000) 2019-10-13T08:01:09.617 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:09.613+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.614872+0000) 2019-10-13T08:01:09.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:09.613+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.615063+0000) 2019-10-13T08:01:09.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:09.613+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.615229+0000) 2019-10-13T08:01:09.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:09.614+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.615469+0000) 2019-10-13T08:01:09.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:09.614+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.615907+0000) 2019-10-13T08:01:09.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:09.616+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:09.616870+0000) 2019-10-13T08:01:09.621 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:09.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:11.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:11.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:11.975856+0000) 2019-10-13T08:01:12.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:12.475+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:12.475949+0000) 2019-10-13T08:01:12.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:12.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:12.836284+0000) 2019-10-13T08:01:12.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:12.836+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:12.837123+0000) 2019-10-13T08:01:14.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:14.620+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.621947+0000) 2019-10-13T08:01:14.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:14.620+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.622216+0000) 2019-10-13T08:01:14.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:14.621+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.621930+0000) 2019-10-13T08:01:14.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:14.621+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.622615+0000) 2019-10-13T08:01:14.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:14.621+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.622829+0000) 2019-10-13T08:01:14.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:14.621+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.623231+0000) 2019-10-13T08:01:14.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:14.621+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.623291+0000) 2019-10-13T08:01:14.626 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:14.623+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.624521+0000) 2019-10-13T08:01:14.629 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:14.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:17.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:17.835+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:17.835984+0000) 2019-10-13T08:01:18.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:18.384+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:18.384791+0000) 2019-10-13T08:01:18.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:18.721+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:18.722168+0000) 2019-10-13T08:01:18.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:18.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:18.722175+0000) 2019-10-13T08:01:19.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:19.629+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.629711+0000) 2019-10-13T08:01:19.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:19.628+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.629286+0000) 2019-10-13T08:01:19.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:19.629+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.630215+0000) 2019-10-13T08:01:19.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:19.629+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.630376+0000) 2019-10-13T08:01:19.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:19.630+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.630840+0000) 2019-10-13T08:01:19.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:19.630+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.631031+0000) 2019-10-13T08:01:19.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:19.630+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.631184+0000) 2019-10-13T08:01:19.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:19.631+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:19.632241+0000) 2019-10-13T08:01:19.637 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:19.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:21.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:21.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:21.976039+0000) 2019-10-13T08:01:22.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:22.477+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:22.478328+0000) 2019-10-13T08:01:22.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:22.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:22.836472+0000) 2019-10-13T08:01:22.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:22.836+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:22.837321+0000) 2019-10-13T08:01:24.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:24.637+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.637587+0000) 2019-10-13T08:01:24.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:24.636+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.637300+0000) 2019-10-13T08:01:24.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:24.637+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.638008+0000) 2019-10-13T08:01:24.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:24.637+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.638271+0000) 2019-10-13T08:01:24.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:24.638+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.638432+0000) 2019-10-13T08:01:24.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:24.638+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.638599+0000) 2019-10-13T08:01:24.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:24.638+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.638813+0000) 2019-10-13T08:01:24.641 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:24.638+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.639658+0000) 2019-10-13T08:01:24.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:24.643+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:26.274 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:01:26.274 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:26.279 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:01:26.279 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:27.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:27.835+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:27.836193+0000) 2019-10-13T08:01:28.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:28.384+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:28.384983+0000) 2019-10-13T08:01:28.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:28.721+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:28.722350+0000) 2019-10-13T08:01:28.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:28.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:28.722344+0000) 2019-10-13T08:01:29.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:29.645+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.645492+0000) 2019-10-13T08:01:29.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:29.645+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.645710+0000) 2019-10-13T08:01:29.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:29.645+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.645806+0000) 2019-10-13T08:01:29.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:29.645+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.646205+0000) 2019-10-13T08:01:29.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:29.646+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.646472+0000) 2019-10-13T08:01:29.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:29.645+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.646544+0000) 2019-10-13T08:01:29.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:29.646+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.647266+0000) 2019-10-13T08:01:29.650 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:29.647+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:29.648676+0000) 2019-10-13T08:01:29.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:29.651+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:31.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:31.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:31.976203+0000) 2019-10-13T08:01:32.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:32.480+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:32.480687+0000) 2019-10-13T08:01:32.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:32.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:32.836654+0000) 2019-10-13T08:01:32.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:32.837+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:32.837502+0000) 2019-10-13T08:01:34.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:34.653+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.654705+0000) 2019-10-13T08:01:34.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:34.653+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.653805+0000) 2019-10-13T08:01:34.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:34.653+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.654014+0000) 2019-10-13T08:01:34.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:34.653+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.654256+0000) 2019-10-13T08:01:34.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:34.654+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.654654+0000) 2019-10-13T08:01:34.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:34.654+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.654797+0000) 2019-10-13T08:01:34.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:34.655+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.655425+0000) 2019-10-13T08:01:34.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:34.655+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.656333+0000) 2019-10-13T08:01:34.661 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:34.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:37.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:37.835+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:37.836363+0000) 2019-10-13T08:01:38.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:38.384+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:38.385170+0000) 2019-10-13T08:01:38.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:38.721+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:38.722552+0000) 2019-10-13T08:01:38.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:38.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:38.722551+0000) 2019-10-13T08:01:39.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:39.660+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.661404+0000) 2019-10-13T08:01:39.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:39.660+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.661884+0000) 2019-10-13T08:01:39.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:39.660+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.661941+0000) 2019-10-13T08:01:39.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:39.660+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.662114+0000) 2019-10-13T08:01:39.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:39.661+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.661763+0000) 2019-10-13T08:01:39.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:39.660+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.662387+0000) 2019-10-13T08:01:39.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:39.661+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.663041+0000) 2019-10-13T08:01:39.665 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:39.663+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:39.664197+0000) 2019-10-13T08:01:39.668 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:39.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:41.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:41.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:41.976416+0000) 2019-10-13T08:01:42.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:42.482+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:42.483144+0000) 2019-10-13T08:01:42.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:42.835+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:42.836831+0000) 2019-10-13T08:01:42.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:42.835+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:42.837645+0000) 2019-10-13T08:01:44.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:44.668+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.669482+0000) 2019-10-13T08:01:44.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:44.668+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.669569+0000) 2019-10-13T08:01:44.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:44.668+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.669724+0000) 2019-10-13T08:01:44.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:44.668+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.669879+0000) 2019-10-13T08:01:44.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:44.667+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.669582+0000) 2019-10-13T08:01:44.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:44.668+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.670179+0000) 2019-10-13T08:01:44.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:44.669+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.670558+0000) 2019-10-13T08:01:44.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:44.669+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.670975+0000) 2019-10-13T08:01:44.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:44.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:47.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:47.835+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:47.836539+0000) 2019-10-13T08:01:48.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:48.384+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:48.385345+0000) 2019-10-13T08:01:48.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:48.721+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:48.722740+0000) 2019-10-13T08:01:48.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:48.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:48.722745+0000) 2019-10-13T08:01:49.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:49.675+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.676735+0000) 2019-10-13T08:01:49.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:49.675+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.677024+0000) 2019-10-13T08:01:49.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:49.675+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.677188+0000) 2019-10-13T08:01:49.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:49.676+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.677564+0000) 2019-10-13T08:01:49.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:49.675+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.677540+0000) 2019-10-13T08:01:49.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:49.676+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.678253+0000) 2019-10-13T08:01:49.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:49.677+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.678419+0000) 2019-10-13T08:01:49.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:49.678+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:49.679921+0000) 2019-10-13T08:01:49.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:49.681+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:51.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:51.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:51.976619+0000) 2019-10-13T08:01:52.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:52.484+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:52.485477+0000) 2019-10-13T08:01:52.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:52.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:52.837038+0000) 2019-10-13T08:01:52.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:52.837+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:52.837830+0000) 2019-10-13T08:01:54.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:54.683+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.684664+0000) 2019-10-13T08:01:54.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:54.683+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.684812+0000) 2019-10-13T08:01:54.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:54.683+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.685129+0000) 2019-10-13T08:01:54.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:54.683+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.685367+0000) 2019-10-13T08:01:54.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:54.683+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.684790+0000) 2019-10-13T08:01:54.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:54.684+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.685607+0000) 2019-10-13T08:01:54.688 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:54.684+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.685855+0000) 2019-10-13T08:01:54.688 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:54.684+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.686670+0000) 2019-10-13T08:01:54.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:54.688+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:01:56.432 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:01:56.433 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:56.437 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:01:56.437 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:57.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:57.836+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:57.836733+0000) 2019-10-13T08:01:58.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:58.384+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:58.385544+0000) 2019-10-13T08:01:58.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:58.722+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:58.722931+0000) 2019-10-13T08:01:58.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:58.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:58.722935+0000) 2019-10-13T08:01:59.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:01:59.689+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.691298+0000) 2019-10-13T08:01:59.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:01:59.690+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.691468+0000) 2019-10-13T08:01:59.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:01:59.690+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.691633+0000) 2019-10-13T08:01:59.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:01:59.690+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.691804+0000) 2019-10-13T08:01:59.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:01:59.690+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.691700+0000) 2019-10-13T08:01:59.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:01:59.690+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.692350+0000) 2019-10-13T08:01:59.694 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:01:59.691+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.692461+0000) 2019-10-13T08:01:59.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:01:59.692+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:59.693616+0000) 2019-10-13T08:01:59.697 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:01:59.694+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134462 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:01.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:01.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:01.976796+0000) 2019-10-13T08:02:02.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:02.487+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:02.487971+0000) 2019-10-13T08:02:02.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:02.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:02.837249+0000) 2019-10-13T08:02:02.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:02.837+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:02.838014+0000) 2019-10-13T08:02:04.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:04.696+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.697685+0000) 2019-10-13T08:02:04.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:04.696+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.697887+0000) 2019-10-13T08:02:04.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:04.696+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.698123+0000) 2019-10-13T08:02:04.700 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:04.697+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.697737+0000) 2019-10-13T08:02:04.700 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:04.697+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.698511+0000) 2019-10-13T08:02:04.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:04.697+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.698858+0000) 2019-10-13T08:02:04.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:04.697+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.699295+0000) 2019-10-13T08:02:04.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:04.699+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.700457+0000) 2019-10-13T08:02:04.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:04.702+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134493 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:07.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:07.836+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:07.836945+0000) 2019-10-13T08:02:08.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:08.385+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:08.385743+0000) 2019-10-13T08:02:08.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:08.722+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:08.723109+0000) 2019-10-13T08:02:08.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:08.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:08.723111+0000) 2019-10-13T08:02:09.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:09.704+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.705163+0000) 2019-10-13T08:02:09.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:09.704+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.705344+0000) 2019-10-13T08:02:09.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:09.705+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.705630+0000) 2019-10-13T08:02:09.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:09.704+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.705464+0000) 2019-10-13T08:02:09.708 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:09.705+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.705765+0000) 2019-10-13T08:02:09.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:09.705+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.706189+0000) 2019-10-13T08:02:09.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:09.705+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.705597+0000) 2019-10-13T08:02:09.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:09.707+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:09.708690+0000) 2019-10-13T08:02:09.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:09.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:11.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:11.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:11.976984+0000) 2019-10-13T08:02:12.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:12.489+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:12.490312+0000) 2019-10-13T08:02:12.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:12.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:12.837434+0000) 2019-10-13T08:02:12.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:12.837+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:12.838149+0000) 2019-10-13T08:02:14.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:14.710+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.711036+0000) 2019-10-13T08:02:14.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:14.710+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.711290+0000) 2019-10-13T08:02:14.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:14.710+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.711388+0000) 2019-10-13T08:02:14.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:14.711+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.711627+0000) 2019-10-13T08:02:14.713 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:14.711+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.712319+0000) 2019-10-13T08:02:14.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:14.712+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.712475+0000) 2019-10-13T08:02:14.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:14.711+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.712284+0000) 2019-10-13T08:02:14.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:14.712+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.713703+0000) 2019-10-13T08:02:14.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:14.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134557 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:17.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:17.836+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:17.837113+0000) 2019-10-13T08:02:18.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:18.385+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:18.385922+0000) 2019-10-13T08:02:18.724 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:18.722+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:18.723318+0000) 2019-10-13T08:02:18.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:18.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:18.723317+0000) 2019-10-13T08:02:19.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:19.718+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.718507+0000) 2019-10-13T08:02:19.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:19.718+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.718829+0000) 2019-10-13T08:02:19.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:19.718+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.719088+0000) 2019-10-13T08:02:19.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:19.718+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.718836+0000) 2019-10-13T08:02:19.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:19.719+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.719571+0000) 2019-10-13T08:02:19.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:19.719+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.719750+0000) 2019-10-13T08:02:19.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:19.719+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.719967+0000) 2019-10-13T08:02:19.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:19.720+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:19.721077+0000) 2019-10-13T08:02:19.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:19.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134588 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:21.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:21.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:21.977182+0000) 2019-10-13T08:02:22.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:22.493+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:22.493491+0000) 2019-10-13T08:02:22.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:22.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:22.837640+0000) 2019-10-13T08:02:22.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:22.838+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:22.838283+0000) 2019-10-13T08:02:24.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:24.725+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.727041+0000) 2019-10-13T08:02:24.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:24.726+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.727404+0000) 2019-10-13T08:02:24.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:24.726+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.727652+0000) 2019-10-13T08:02:24.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:24.726+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.727852+0000) 2019-10-13T08:02:24.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:24.726+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.727266+0000) 2019-10-13T08:02:24.730 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:24.726+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.727955+0000) 2019-10-13T08:02:24.730 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:24.726+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.728042+0000) 2019-10-13T08:02:24.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:24.728+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.728885+0000) 2019-10-13T08:02:24.733 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:24.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:26.574 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:02:26.574 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:26.579 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:02:26.580 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:27.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:27.835+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:27.837312+0000) 2019-10-13T08:02:28.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:28.385+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:28.386120+0000) 2019-10-13T08:02:28.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:28.722+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:28.723517+0000) 2019-10-13T08:02:28.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:28.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:28.723540+0000) 2019-10-13T08:02:29.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:29.731+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.733273+0000) 2019-10-13T08:02:29.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:29.732+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.733755+0000) 2019-10-13T08:02:29.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:29.732+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.734011+0000) 2019-10-13T08:02:29.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:29.732+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.733916+0000) 2019-10-13T08:02:29.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:29.732+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.734386+0000) 2019-10-13T08:02:29.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:29.733+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.734827+0000) 2019-10-13T08:02:29.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:29.733+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.734016+0000) 2019-10-13T08:02:29.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:29.735+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:29.736259+0000) 2019-10-13T08:02:29.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:29.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134652 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:31.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:31.975+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:31.977382+0000) 2019-10-13T08:02:32.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:32.496+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:32.496722+0000) 2019-10-13T08:02:32.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:32.836+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:32.837798+0000) 2019-10-13T08:02:32.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:32.838+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:32.838425+0000) 2019-10-13T08:02:34.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:34.738+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.739697+0000) 2019-10-13T08:02:34.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:34.738+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.739881+0000) 2019-10-13T08:02:34.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:34.738+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.740006+0000) 2019-10-13T08:02:34.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:34.738+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.740013+0000) 2019-10-13T08:02:34.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:34.739+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.740863+0000) 2019-10-13T08:02:34.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:34.739+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.741030+0000) 2019-10-13T08:02:34.743 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:34.738+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.740255+0000) 2019-10-13T08:02:34.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:34.740+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.741737+0000) 2019-10-13T08:02:34.746 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:34.743+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:37.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:37.836+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:37.837517+0000) 2019-10-13T08:02:38.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:38.384+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:38.386298+0000) 2019-10-13T08:02:38.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:38.722+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:38.723702+0000) 2019-10-13T08:02:38.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:38.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:38.723709+0000) 2019-10-13T08:02:39.749 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:39.746+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.747653+0000) 2019-10-13T08:02:39.750 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:39.745+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.747348+0000) 2019-10-13T08:02:39.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:39.746+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.747989+0000) 2019-10-13T08:02:39.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:39.746+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.748225+0000) 2019-10-13T08:02:39.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:39.747+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.748401+0000) 2019-10-13T08:02:39.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:39.747+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.748618+0000) 2019-10-13T08:02:39.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:39.745+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.747398+0000) 2019-10-13T08:02:39.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:39.748+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:39.750233+0000) 2019-10-13T08:02:39.764 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:39.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:41.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:41.976+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:41.977545+0000) 2019-10-13T08:02:42.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:42.498+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:42.499176+0000) 2019-10-13T08:02:42.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:42.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:42.837946+0000) 2019-10-13T08:02:42.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:42.837+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:42.838595+0000) 2019-10-13T08:02:44.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:44.752+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.754022+0000) 2019-10-13T08:02:44.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:44.752+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.754137+0000) 2019-10-13T08:02:44.755 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:44.752+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.754382+0000) 2019-10-13T08:02:44.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:44.752+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.754280+0000) 2019-10-13T08:02:44.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:44.753+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.754578+0000) 2019-10-13T08:02:44.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:44.753+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.754679+0000) 2019-10-13T08:02:44.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:44.753+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.755275+0000) 2019-10-13T08:02:44.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:44.754+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.756583+0000) 2019-10-13T08:02:44.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:44.756+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:47.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:47.837+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:47.837726+0000) 2019-10-13T08:02:48.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:48.385+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:48.386489+0000) 2019-10-13T08:02:48.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:48.723+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:48.723915+0000) 2019-10-13T08:02:48.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:48.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:48.723918+0000) 2019-10-13T08:02:49.761 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:49.758+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.759999+0000) 2019-10-13T08:02:49.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:49.758+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.760331+0000) 2019-10-13T08:02:49.762 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:49.759+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.760547+0000) 2019-10-13T08:02:49.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:49.759+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.760626+0000) 2019-10-13T08:02:49.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:49.758+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.759661+0000) 2019-10-13T08:02:49.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:49.759+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.760783+0000) 2019-10-13T08:02:49.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:49.759+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.760950+0000) 2019-10-13T08:02:49.764 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:49.761+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:49.762694+0000) 2019-10-13T08:02:49.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:49.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134778 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:51.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:51.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:51.977723+0000) 2019-10-13T08:02:52.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:52.500+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:52.501467+0000) 2019-10-13T08:02:52.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:52.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:52.838142+0000) 2019-10-13T08:02:52.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:52.838+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:52.838751+0000) 2019-10-13T08:02:54.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:54.766+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.767377+0000) 2019-10-13T08:02:54.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:54.767+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.767556+0000) 2019-10-13T08:02:54.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:54.766+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.767379+0000) 2019-10-13T08:02:54.770 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:54.767+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.767935+0000) 2019-10-13T08:02:54.770 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:54.767+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.768379+0000) 2019-10-13T08:02:54.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:54.768+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.768555+0000) 2019-10-13T08:02:54.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:54.768+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.768743+0000) 2019-10-13T08:02:54.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:54.769+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.769770+0000) 2019-10-13T08:02:54.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:54.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:02:56.687 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:02:56.687 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:56.692 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:02:56.692 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:57.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:57.837+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:57.837920+0000) 2019-10-13T08:02:58.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:58.386+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:58.386679+0000) 2019-10-13T08:02:58.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:58.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:58.724145+0000) 2019-10-13T08:02:58.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:58.723+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:58.724144+0000) 2019-10-13T08:02:59.776 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:02:59.774+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.774975+0000) 2019-10-13T08:02:59.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:02:59.774+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.775246+0000) 2019-10-13T08:02:59.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:02:59.775+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.775629+0000) 2019-10-13T08:02:59.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:02:59.775+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.775725+0000) 2019-10-13T08:02:59.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:02:59.774+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.775722+0000) 2019-10-13T08:02:59.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:02:59.776+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.776542+0000) 2019-10-13T08:02:59.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:02:59.776+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.776854+0000) 2019-10-13T08:02:59.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:02:59.777+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:59.777956+0000) 2019-10-13T08:02:59.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:02:59.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134840 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:01.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:01.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:01.977915+0000) 2019-10-13T08:03:02.505 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:02.503+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:02.503777+0000) 2019-10-13T08:03:02.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:02.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:02.838284+0000) 2019-10-13T08:03:02.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:02.838+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:02.838924+0000) 2019-10-13T08:03:04.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:04.782+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.782739+0000) 2019-10-13T08:03:04.784 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:04.782+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.783006+0000) 2019-10-13T08:03:04.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:04.782+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.782883+0000) 2019-10-13T08:03:04.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:04.782+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.782820+0000) 2019-10-13T08:03:04.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:04.783+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.783479+0000) 2019-10-13T08:03:04.786 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:04.783+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.783755+0000) 2019-10-13T08:03:04.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:04.783+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.783824+0000) 2019-10-13T08:03:04.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:04.785+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.785916+0000) 2019-10-13T08:03:04.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:04.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134873 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:07.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:07.838+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:07.838109+0000) 2019-10-13T08:03:08.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:08.386+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:08.386893+0000) 2019-10-13T08:03:08.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:08.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:08.724281+0000) 2019-10-13T08:03:08.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:08.724+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:08.724329+0000) 2019-10-13T08:03:09.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:09.788+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.788811+0000) 2019-10-13T08:03:09.790 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:09.788+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.788719+0000) 2019-10-13T08:03:09.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:09.788+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.789129+0000) 2019-10-13T08:03:09.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:09.788+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.789213+0000) 2019-10-13T08:03:09.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:09.788+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.788995+0000) 2019-10-13T08:03:09.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:09.789+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.789802+0000) 2019-10-13T08:03:09.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:09.789+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.789997+0000) 2019-10-13T08:03:09.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:09.790+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:09.791345+0000) 2019-10-13T08:03:09.795 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:09.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:11.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:11.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:11.978082+0000) 2019-10-13T08:03:12.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:12.505+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:12.506208+0000) 2019-10-13T08:03:12.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:12.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:12.838448+0000) 2019-10-13T08:03:12.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:12.838+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:12.839118+0000) 2019-10-13T08:03:14.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:14.795+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.795728+0000) 2019-10-13T08:03:14.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:14.795+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.796623+0000) 2019-10-13T08:03:14.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:14.795+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.796902+0000) 2019-10-13T08:03:14.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:14.795+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.797166+0000) 2019-10-13T08:03:14.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:14.795+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.797363+0000) 2019-10-13T08:03:14.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:14.796+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.797637+0000) 2019-10-13T08:03:14.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:14.796+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.797756+0000) 2019-10-13T08:03:14.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:14.797+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.798636+0000) 2019-10-13T08:03:14.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:14.801+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:17.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:17.837+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:17.838303+0000) 2019-10-13T08:03:18.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:18.386+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:18.387086+0000) 2019-10-13T08:03:18.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:18.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:18.724444+0000) 2019-10-13T08:03:18.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:18.723+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:18.724470+0000) 2019-10-13T08:03:19.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:19.802+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.803513+0000) 2019-10-13T08:03:19.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:19.802+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.804119+0000) 2019-10-13T08:03:19.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:19.802+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.804383+0000) 2019-10-13T08:03:19.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:19.803+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.804538+0000) 2019-10-13T08:03:19.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:19.803+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.804873+0000) 2019-10-13T08:03:19.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:19.803+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.804976+0000) 2019-10-13T08:03:19.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:19.803+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.805112+0000) 2019-10-13T08:03:19.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:19.805+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:19.805921+0000) 2019-10-13T08:03:19.810 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:19.808+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:21.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:21.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:21.978270+0000) 2019-10-13T08:03:22.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:22.507+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:22.508587+0000) 2019-10-13T08:03:22.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:22.837+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:22.838610+0000) 2019-10-13T08:03:22.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:22.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:22.839302+0000) 2019-10-13T08:03:24.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:24.810+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.811459+0000) 2019-10-13T08:03:24.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:24.810+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.811727+0000) 2019-10-13T08:03:24.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:24.810+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.811458+0000) 2019-10-13T08:03:24.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:24.810+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.812099+0000) 2019-10-13T08:03:24.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:24.810+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.812270+0000) 2019-10-13T08:03:24.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:24.811+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.812528+0000) 2019-10-13T08:03:24.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:24.811+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.812828+0000) 2019-10-13T08:03:24.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:24.813+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.814688+0000) 2019-10-13T08:03:24.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:24.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 134999 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:26.841 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:03:26.842 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:26.846 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:03:26.847 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:27.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:27.837+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:27.838430+0000) 2019-10-13T08:03:28.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:28.386+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:28.387276+0000) 2019-10-13T08:03:28.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:28.723+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:28.724616+0000) 2019-10-13T08:03:28.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:28.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:28.724573+0000) 2019-10-13T08:03:29.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:29.816+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.818251+0000) 2019-10-13T08:03:29.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:29.817+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.818668+0000) 2019-10-13T08:03:29.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:29.817+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.818891+0000) 2019-10-13T08:03:29.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:29.817+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.819081+0000) 2019-10-13T08:03:29.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:29.817+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.819291+0000) 2019-10-13T08:03:29.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:29.817+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.818941+0000) 2019-10-13T08:03:29.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:29.818+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.820070+0000) 2019-10-13T08:03:29.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:29.818+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:29.819516+0000) 2019-10-13T08:03:29.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:29.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135031 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:31.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:31.977+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:31.978489+0000) 2019-10-13T08:03:32.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:32.509+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:32.510910+0000) 2019-10-13T08:03:32.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:32.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:32.838783+0000) 2019-10-13T08:03:32.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:32.838+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:32.839483+0000) 2019-10-13T08:03:34.827 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:34.824+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.825808+0000) 2019-10-13T08:03:34.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:34.824+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.825998+0000) 2019-10-13T08:03:34.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:34.823+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.825708+0000) 2019-10-13T08:03:34.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:34.825+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.826628+0000) 2019-10-13T08:03:34.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:34.825+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.826925+0000) 2019-10-13T08:03:34.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:34.825+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.826940+0000) 2019-10-13T08:03:34.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:34.825+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.827191+0000) 2019-10-13T08:03:34.830 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:34.827+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.828850+0000) 2019-10-13T08:03:34.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:34.830+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135061 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:37.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:37.838+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:37.838598+0000) 2019-10-13T08:03:38.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:38.387+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:38.387450+0000) 2019-10-13T08:03:38.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:38.724+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:38.724780+0000) 2019-10-13T08:03:38.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:38.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:38.724783+0000) 2019-10-13T08:03:39.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:39.832+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.833325+0000) 2019-10-13T08:03:39.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:39.831+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.833102+0000) 2019-10-13T08:03:39.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:39.833+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.834104+0000) 2019-10-13T08:03:39.836 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:39.833+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.834255+0000) 2019-10-13T08:03:39.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:39.834+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.834582+0000) 2019-10-13T08:03:39.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:39.834+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.834845+0000) 2019-10-13T08:03:39.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:39.834+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.834985+0000) 2019-10-13T08:03:39.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:39.833+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:39.835328+0000) 2019-10-13T08:03:39.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:39.838+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:41.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:41.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:41.978627+0000) 2019-10-13T08:03:42.515 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:42.512+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:42.513212+0000) 2019-10-13T08:03:42.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:42.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:42.838987+0000) 2019-10-13T08:03:42.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:42.838+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:42.839649+0000) 2019-10-13T08:03:44.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:44.840+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.840736+0000) 2019-10-13T08:03:44.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:44.840+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.840980+0000) 2019-10-13T08:03:44.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:44.840+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.841246+0000) 2019-10-13T08:03:44.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:44.840+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.841355+0000) 2019-10-13T08:03:44.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:44.841+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.841630+0000) 2019-10-13T08:03:44.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:44.840+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.841372+0000) 2019-10-13T08:03:44.846 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:44.841+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.842311+0000) 2019-10-13T08:03:44.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:44.843+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.844323+0000) 2019-10-13T08:03:44.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:44.846+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:47.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:47.838+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:47.838768+0000) 2019-10-13T08:03:48.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:48.387+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:48.387640+0000) 2019-10-13T08:03:48.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:48.724+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:48.724992+0000) 2019-10-13T08:03:48.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:48.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:48.724999+0000) 2019-10-13T08:03:49.849 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:49.847+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.848099+0000) 2019-10-13T08:03:49.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:49.848+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.848581+0000) 2019-10-13T08:03:49.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:49.848+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.848748+0000) 2019-10-13T08:03:49.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:49.848+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.848959+0000) 2019-10-13T08:03:49.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:49.848+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.849116+0000) 2019-10-13T08:03:49.851 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:49.848+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.849320+0000) 2019-10-13T08:03:49.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:49.848+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.849049+0000) 2019-10-13T08:03:49.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:49.848+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:49.849510+0000) 2019-10-13T08:03:49.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:49.853+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135157 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:51.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:51.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:51.978838+0000) 2019-10-13T08:03:52.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:52.514+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:52.515503+0000) 2019-10-13T08:03:52.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:52.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:52.839178+0000) 2019-10-13T08:03:52.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:52.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:52.839821+0000) 2019-10-13T08:03:54.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:54.854+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.855018+0000) 2019-10-13T08:03:54.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:54.854+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.855159+0000) 2019-10-13T08:03:54.857 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:54.855+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.855572+0000) 2019-10-13T08:03:54.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:54.855+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.855717+0000) 2019-10-13T08:03:54.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:54.854+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.855374+0000) 2019-10-13T08:03:54.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:54.855+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.856095+0000) 2019-10-13T08:03:54.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:54.855+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.856281+0000) 2019-10-13T08:03:54.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:54.856+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.857647+0000) 2019-10-13T08:03:54.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:54.859+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:03:57.195 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:03:57.196 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:57.200 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:03:57.200 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:57.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:57.837+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:57.838970+0000) 2019-10-13T08:03:58.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:58.386+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:58.387905+0000) 2019-10-13T08:03:58.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:58.723+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:58.725162+0000) 2019-10-13T08:03:58.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:58.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:58.725154+0000) 2019-10-13T08:03:59.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:03:59.860+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.861810+0000) 2019-10-13T08:03:59.863 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:03:59.860+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.862028+0000) 2019-10-13T08:03:59.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:03:59.860+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.862280+0000) 2019-10-13T08:03:59.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:03:59.861+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.862604+0000) 2019-10-13T08:03:59.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:03:59.861+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.862008+0000) 2019-10-13T08:03:59.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:03:59.861+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.862904+0000) 2019-10-13T08:03:59.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:03:59.861+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.863215+0000) 2019-10-13T08:03:59.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:03:59.863+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:59.864469+0000) 2019-10-13T08:03:59.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:03:59.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:01.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:01.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:01.979029+0000) 2019-10-13T08:04:02.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:02.517+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:02.517974+0000) 2019-10-13T08:04:02.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:02.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:02.839375+0000) 2019-10-13T08:04:02.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:02.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:02.839946+0000) 2019-10-13T08:04:04.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:04.868+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.869669+0000) 2019-10-13T08:04:04.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:04.868+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.869822+0000) 2019-10-13T08:04:04.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:04.869+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.870230+0000) 2019-10-13T08:04:04.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:04.868+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.870041+0000) 2019-10-13T08:04:04.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:04.868+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.870225+0000) 2019-10-13T08:04:04.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:04.869+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.870391+0000) 2019-10-13T08:04:04.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:04.869+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.870466+0000) 2019-10-13T08:04:04.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:04.871+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.872024+0000) 2019-10-13T08:04:04.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:04.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:07.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:07.838+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:07.839146+0000) 2019-10-13T08:04:08.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:08.387+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:08.388069+0000) 2019-10-13T08:04:08.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:08.724+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:08.725330+0000) 2019-10-13T08:04:08.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:08.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:08.725325+0000) 2019-10-13T08:04:09.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:09.874+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.875634+0000) 2019-10-13T08:04:09.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:09.874+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.875334+0000) 2019-10-13T08:04:09.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:09.874+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.876067+0000) 2019-10-13T08:04:09.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:09.874+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.876191+0000) 2019-10-13T08:04:09.878 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:09.874+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.876349+0000) 2019-10-13T08:04:09.878 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:09.875+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.876740+0000) 2019-10-13T08:04:09.879 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:09.875+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.876886+0000) 2019-10-13T08:04:09.879 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:09.876+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:09.877086+0000) 2019-10-13T08:04:09.882 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:09.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:11.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:11.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:11.979222+0000) 2019-10-13T08:04:12.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:12.520+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:12.520368+0000) 2019-10-13T08:04:12.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:12.838+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:12.839589+0000) 2019-10-13T08:04:12.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:12.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:12.840093+0000) 2019-10-13T08:04:14.884 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:14.881+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.882595+0000) 2019-10-13T08:04:14.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:14.881+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.882775+0000) 2019-10-13T08:04:14.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:14.881+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.883074+0000) 2019-10-13T08:04:14.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:14.881+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.883220+0000) 2019-10-13T08:04:14.885 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:14.881+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.883360+0000) 2019-10-13T08:04:14.885 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:14.882+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.883444+0000) 2019-10-13T08:04:14.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:14.881+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.882466+0000) 2019-10-13T08:04:14.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:14.884+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.885579+0000) 2019-10-13T08:04:14.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:14.885+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135312 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:17.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:17.838+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:17.839350+0000) 2019-10-13T08:04:18.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:18.387+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:18.388245+0000) 2019-10-13T08:04:18.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:18.724+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:18.725547+0000) 2019-10-13T08:04:18.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:18.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:18.725499+0000) 2019-10-13T08:04:19.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:19.887+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.888968+0000) 2019-10-13T08:04:19.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:19.887+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.889139+0000) 2019-10-13T08:04:19.890 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:19.887+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.889271+0000) 2019-10-13T08:04:19.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:19.887+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.889455+0000) 2019-10-13T08:04:19.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:19.888+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.889677+0000) 2019-10-13T08:04:19.892 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:19.888+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.890346+0000) 2019-10-13T08:04:19.892 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:19.889+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.890582+0000) 2019-10-13T08:04:19.893 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:19.890+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:19.891764+0000) 2019-10-13T08:04:19.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:19.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135344 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:21.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:21.978+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:21.979428+0000) 2019-10-13T08:04:22.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:22.521+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:22.522692+0000) 2019-10-13T08:04:22.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:22.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:22.839807+0000) 2019-10-13T08:04:22.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:22.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:22.840267+0000) 2019-10-13T08:04:24.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:24.894+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.896555+0000) 2019-10-13T08:04:24.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:24.896+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.897295+0000) 2019-10-13T08:04:24.899 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:24.897+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.897626+0000) 2019-10-13T08:04:24.899 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:24.897+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.897741+0000) 2019-10-13T08:04:24.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:24.897+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.897846+0000) 2019-10-13T08:04:24.899 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:24.897+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.897981+0000) 2019-10-13T08:04:24.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:24.897+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.898348+0000) 2019-10-13T08:04:24.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:24.898+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.899810+0000) 2019-10-13T08:04:24.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:24.902+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:27.300 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:04:27.300 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:27.304 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:04:27.305 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:27.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:27.839+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:27.839569+0000) 2019-10-13T08:04:28.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:28.388+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:28.388406+0000) 2019-10-13T08:04:28.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:28.725+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:28.725704+0000) 2019-10-13T08:04:28.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:28.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:28.725676+0000) 2019-10-13T08:04:29.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:29.904+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.904916+0000) 2019-10-13T08:04:29.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:29.904+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.905057+0000) 2019-10-13T08:04:29.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:29.903+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.905215+0000) 2019-10-13T08:04:29.907 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:29.905+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.905807+0000) 2019-10-13T08:04:29.907 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:29.905+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.905900+0000) 2019-10-13T08:04:29.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:29.905+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.906138+0000) 2019-10-13T08:04:29.908 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:29.906+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.906424+0000) 2019-10-13T08:04:29.908 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:29.904+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:29.906432+0000) 2019-10-13T08:04:29.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:29.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:31.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:31.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:31.979618+0000) 2019-10-13T08:04:32.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:32.524+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:32.525034+0000) 2019-10-13T08:04:32.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:32.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:32.839962+0000) 2019-10-13T08:04:32.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:32.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:32.840390+0000) 2019-10-13T08:04:34.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:34.911+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.912168+0000) 2019-10-13T08:04:34.914 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:34.912+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.912509+0000) 2019-10-13T08:04:34.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:34.912+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.912567+0000) 2019-10-13T08:04:34.914 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:34.912+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.912708+0000) 2019-10-13T08:04:34.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:34.912+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.912884+0000) 2019-10-13T08:04:34.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:34.912+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.913215+0000) 2019-10-13T08:04:34.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:34.911+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.912633+0000) 2019-10-13T08:04:34.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:34.914+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.914989+0000) 2019-10-13T08:04:34.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:34.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:37.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:37.839+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:37.839769+0000) 2019-10-13T08:04:38.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:38.388+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:38.388602+0000) 2019-10-13T08:04:38.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:38.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:38.725822+0000) 2019-10-13T08:04:38.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:38.725+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:38.725890+0000) 2019-10-13T08:04:39.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:39.918+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.919244+0000) 2019-10-13T08:04:39.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:39.919+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.919652+0000) 2019-10-13T08:04:39.921 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:39.919+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.919763+0000) 2019-10-13T08:04:39.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:39.918+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.919212+0000) 2019-10-13T08:04:39.922 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:39.919+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.920090+0000) 2019-10-13T08:04:39.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:39.919+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.920166+0000) 2019-10-13T08:04:39.922 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:39.919+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.920372+0000) 2019-10-13T08:04:39.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:39.920+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:39.921566+0000) 2019-10-13T08:04:39.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:39.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135470 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:41.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:41.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:41.979780+0000) 2019-10-13T08:04:42.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:42.526+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:42.527458+0000) 2019-10-13T08:04:42.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:42.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:42.840122+0000) 2019-10-13T08:04:42.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:42.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:42.840530+0000) 2019-10-13T08:04:44.927 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:44.925+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.926472+0000) 2019-10-13T08:04:44.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:44.925+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.926702+0000) 2019-10-13T08:04:44.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:44.925+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.926487+0000) 2019-10-13T08:04:44.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:44.925+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.927061+0000) 2019-10-13T08:04:44.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:44.926+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.927415+0000) 2019-10-13T08:04:44.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:44.926+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.927864+0000) 2019-10-13T08:04:44.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:44.926+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.927898+0000) 2019-10-13T08:04:44.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:44.928+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.928995+0000) 2019-10-13T08:04:44.933 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:44.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:47.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:47.838+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:47.839998+0000) 2019-10-13T08:04:48.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:48.387+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:48.388788+0000) 2019-10-13T08:04:48.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:48.724+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:48.726044+0000) 2019-10-13T08:04:48.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:48.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:48.726030+0000) 2019-10-13T08:04:49.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:49.932+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.933146+0000) 2019-10-13T08:04:49.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:49.932+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.933933+0000) 2019-10-13T08:04:49.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:49.932+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.934195+0000) 2019-10-13T08:04:49.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:49.933+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.934472+0000) 2019-10-13T08:04:49.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:49.933+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.934593+0000) 2019-10-13T08:04:49.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:49.933+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.934660+0000) 2019-10-13T08:04:49.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:49.933+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.934864+0000) 2019-10-13T08:04:49.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:49.934+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:49.935509+0000) 2019-10-13T08:04:49.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:49.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135534 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:51.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:51.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:51.979939+0000) 2019-10-13T08:04:52.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:52.529+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:52.529921+0000) 2019-10-13T08:04:52.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:52.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:52.840678+0000) 2019-10-13T08:04:52.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:52.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:52.840294+0000) 2019-10-13T08:04:54.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:54.940+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.941251+0000) 2019-10-13T08:04:54.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:54.939+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.941173+0000) 2019-10-13T08:04:54.945 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:54.940+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.941482+0000) 2019-10-13T08:04:54.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:54.940+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.941492+0000) 2019-10-13T08:04:54.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:54.940+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.941999+0000) 2019-10-13T08:04:54.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:54.940+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.942268+0000) 2019-10-13T08:04:54.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:54.941+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.942516+0000) 2019-10-13T08:04:54.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:54.943+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.943797+0000) 2019-10-13T08:04:54.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:54.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:04:57.437 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:04:57.438 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:57.442 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:04:57.442 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:57.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:57.839+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:57.840211+0000) 2019-10-13T08:04:58.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:58.388+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:58.388953+0000) 2019-10-13T08:04:58.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:58.725+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:58.726225+0000) 2019-10-13T08:04:58.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:58.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:58.726221+0000) 2019-10-13T08:04:59.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:04:59.946+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.947527+0000) 2019-10-13T08:04:59.949 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:04:59.946+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.948081+0000) 2019-10-13T08:04:59.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:04:59.946+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.948191+0000) 2019-10-13T08:04:59.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:04:59.947+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.948469+0000) 2019-10-13T08:04:59.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:04:59.946+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.948373+0000) 2019-10-13T08:04:59.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:04:59.947+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.949212+0000) 2019-10-13T08:04:59.951 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:04:59.947+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.949338+0000) 2019-10-13T08:04:59.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:04:59.951+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:59.952092+0000) 2019-10-13T08:04:59.953 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:04:59.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:01.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:01.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:01.980064+0000) 2019-10-13T08:05:02.533 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:02.532+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:02.532373+0000) 2019-10-13T08:05:02.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:02.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:02.840432+0000) 2019-10-13T08:05:02.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:02.840+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:02.840793+0000) 2019-10-13T08:05:04.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:04.952+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.954313+0000) 2019-10-13T08:05:04.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:04.953+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.954458+0000) 2019-10-13T08:05:04.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:04.953+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.954137+0000) 2019-10-13T08:05:04.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:04.953+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.954815+0000) 2019-10-13T08:05:04.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:04.953+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.954858+0000) 2019-10-13T08:05:04.957 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:04.953+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.954965+0000) 2019-10-13T08:05:04.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:04.953+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.955290+0000) 2019-10-13T08:05:04.958 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:04.956+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.956886+0000) 2019-10-13T08:05:04.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:04.957+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135628 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:07.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:07.839+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:07.840412+0000) 2019-10-13T08:05:08.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:08.388+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:08.389133+0000) 2019-10-13T08:05:08.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:08.725+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:08.726445+0000) 2019-10-13T08:05:08.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:08.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:08.726451+0000) 2019-10-13T08:05:09.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:09.959+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.960619+0000) 2019-10-13T08:05:09.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:09.959+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.960947+0000) 2019-10-13T08:05:09.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:09.959+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.961151+0000) 2019-10-13T08:05:09.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:09.959+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.960376+0000) 2019-10-13T08:05:09.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:09.959+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.961375+0000) 2019-10-13T08:05:09.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:09.960+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.961775+0000) 2019-10-13T08:05:09.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:09.960+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.962186+0000) 2019-10-13T08:05:09.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:09.962+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:09.963343+0000) 2019-10-13T08:05:09.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:09.965+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135660 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:11.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:11.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:11.980273+0000) 2019-10-13T08:05:12.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:12.533+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:12.534663+0000) 2019-10-13T08:05:12.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:12.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:12.840600+0000) 2019-10-13T08:05:12.842 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:12.839+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:12.840938+0000) 2019-10-13T08:05:14.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:14.968+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.968485+0000) 2019-10-13T08:05:14.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:14.968+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.968726+0000) 2019-10-13T08:05:14.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:14.968+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.969093+0000) 2019-10-13T08:05:14.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:14.968+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.969203+0000) 2019-10-13T08:05:14.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:14.969+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.969512+0000) 2019-10-13T08:05:14.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:14.967+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.969269+0000) 2019-10-13T08:05:14.971 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:14.969+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.970044+0000) 2019-10-13T08:05:14.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:14.969+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.971479+0000) 2019-10-13T08:05:14.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:14.973+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:17.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:17.840+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:17.840643+0000) 2019-10-13T08:05:18.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:18.389+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:18.389308+0000) 2019-10-13T08:05:18.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:18.726+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:18.726654+0000) 2019-10-13T08:05:18.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:18.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:18.726651+0000) 2019-10-13T08:05:19.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:19.975+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.976300+0000) 2019-10-13T08:05:19.978 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:19.976+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.976509+0000) 2019-10-13T08:05:19.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:19.976+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.976676+0000) 2019-10-13T08:05:19.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:19.974+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.975884+0000) 2019-10-13T08:05:19.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:19.976+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.976948+0000) 2019-10-13T08:05:19.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:19.976+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.977223+0000) 2019-10-13T08:05:19.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:19.976+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.977330+0000) 2019-10-13T08:05:19.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:19.977+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:19.978812+0000) 2019-10-13T08:05:19.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:19.982+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135722 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:21.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:21.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:21.980468+0000) 2019-10-13T08:05:22.538 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:22.536+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:22.537171+0000) 2019-10-13T08:05:22.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:22.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:22.840809+0000) 2019-10-13T08:05:22.842 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:22.840+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:22.841092+0000) 2019-10-13T08:05:24.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:24.983+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.984312+0000) 2019-10-13T08:05:24.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:24.984+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.984737+0000) 2019-10-13T08:05:24.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:24.984+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.985050+0000) 2019-10-13T08:05:24.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:24.983+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.984978+0000) 2019-10-13T08:05:24.987 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:24.985+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.985519+0000) 2019-10-13T08:05:24.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:24.985+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.985704+0000) 2019-10-13T08:05:24.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:24.985+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.985921+0000) 2019-10-13T08:05:24.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:24.984+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.986351+0000) 2019-10-13T08:05:24.990 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:24.989+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135754 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:27.592 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:05:27.592 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:27.596 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:05:27.597 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:27.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:27.840+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:27.840861+0000) 2019-10-13T08:05:28.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:28.389+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:28.389496+0000) 2019-10-13T08:05:28.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:28.725+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:28.726881+0000) 2019-10-13T08:05:28.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:28.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:28.726883+0000) 2019-10-13T08:05:29.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:29.990+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.991762+0000) 2019-10-13T08:05:29.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:29.990+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.991962+0000) 2019-10-13T08:05:29.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:29.990+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.992166+0000) 2019-10-13T08:05:29.994 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:29.990+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.992301+0000) 2019-10-13T08:05:29.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:29.991+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.992462+0000) 2019-10-13T08:05:29.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:29.991+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.992256+0000) 2019-10-13T08:05:29.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:29.991+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.993176+0000) 2019-10-13T08:05:29.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:29.993+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:29.994289+0000) 2019-10-13T08:05:29.998 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:29.996+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:31.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:31.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:31.980658+0000) 2019-10-13T08:05:32.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:32.538+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:32.539542+0000) 2019-10-13T08:05:32.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:32.839+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:32.840989+0000) 2019-10-13T08:05:32.842 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:32.840+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:32.841258+0000) 2019-10-13T08:05:35.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:34.997+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:34.999099+0000) 2019-10-13T08:05:35.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:34.997+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:34.999352+0000) 2019-10-13T08:05:35.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:34.998+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:34.999522+0000) 2019-10-13T08:05:35.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:34.998+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:34.999688+0000) 2019-10-13T08:05:35.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:34.998+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:34.998817+0000) 2019-10-13T08:05:35.002 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:34.998+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.000085+0000) 2019-10-13T08:05:35.002 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:34.998+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.000291+0000) 2019-10-13T08:05:35.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:35.000+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.001573+0000) 2019-10-13T08:05:35.005 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:35.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:37.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:37.840+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:37.841021+0000) 2019-10-13T08:05:38.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:38.388+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:38.389731+0000) 2019-10-13T08:05:38.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:38.726+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:38.727073+0000) 2019-10-13T08:05:38.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:38.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:38.727100+0000) 2019-10-13T08:05:40.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:40.004+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.006125+0000) 2019-10-13T08:05:40.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:40.004+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.006371+0000) 2019-10-13T08:05:40.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:40.005+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.006283+0000) 2019-10-13T08:05:40.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:40.005+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.006876+0000) 2019-10-13T08:05:40.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:40.005+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.007363+0000) 2019-10-13T08:05:40.009 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:40.005+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.007361+0000) 2019-10-13T08:05:40.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:40.005+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.007117+0000) 2019-10-13T08:05:40.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:40.007+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.008421+0000) 2019-10-13T08:05:40.012 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:40.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135848 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:41.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:41.979+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:41.980806+0000) 2019-10-13T08:05:42.543 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:42.541+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:42.541975+0000) 2019-10-13T08:05:42.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:42.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:42.841214+0000) 2019-10-13T08:05:42.842 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:42.840+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:42.841415+0000) 2019-10-13T08:05:45.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:45.011+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.012888+0000) 2019-10-13T08:05:45.014 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:45.011+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.013249+0000) 2019-10-13T08:05:45.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:45.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.012964+0000) 2019-10-13T08:05:45.015 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:45.012+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.013413+0000) 2019-10-13T08:05:45.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:45.012+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.013534+0000) 2019-10-13T08:05:45.015 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:45.012+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.013751+0000) 2019-10-13T08:05:45.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:45.012+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.014059+0000) 2019-10-13T08:05:45.017 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:45.015+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.016330+0000) 2019-10-13T08:05:45.018 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:45.015+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135881 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:47.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:47.840+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:47.841209+0000) 2019-10-13T08:05:48.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:48.389+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:48.389906+0000) 2019-10-13T08:05:48.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:48.726+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:48.727266+0000) 2019-10-13T08:05:48.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:48.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:48.727291+0000) 2019-10-13T08:05:50.020 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:50.017+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.019385+0000) 2019-10-13T08:05:50.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:50.018+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.019691+0000) 2019-10-13T08:05:50.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:50.018+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.019748+0000) 2019-10-13T08:05:50.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:50.018+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.019547+0000) 2019-10-13T08:05:50.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:50.019+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.020491+0000) 2019-10-13T08:05:50.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:50.018+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.020107+0000) 2019-10-13T08:05:50.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:50.018+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.020284+0000) 2019-10-13T08:05:50.023 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:50.019+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.020608+0000) 2019-10-13T08:05:50.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:50.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:51.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:51.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:51.980952+0000) 2019-10-13T08:05:52.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:52.544+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:52.544326+0000) 2019-10-13T08:05:52.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:52.840+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:52.841415+0000) 2019-10-13T08:05:52.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:52.841+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:52.841559+0000) 2019-10-13T08:05:55.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:05:55.025+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.026321+0000) 2019-10-13T08:05:55.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:05:55.027+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.028139+0000) 2019-10-13T08:05:55.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:55.025+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.026558+0000) 2019-10-13T08:05:55.033 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:55.025+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.026815+0000) 2019-10-13T08:05:55.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:55.025+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.026944+0000) 2019-10-13T08:05:55.033 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:55.025+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.027224+0000) 2019-10-13T08:05:55.034 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:05:55.025+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.027373+0000) 2019-10-13T08:05:55.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:05:55.026+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.027692+0000) 2019-10-13T08:05:55.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:05:55.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:05:57.763 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:05:57.764 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:57.768 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:05:57.769 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:57.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:05:57.841+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:57.841391+0000) 2019-10-13T08:05:58.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:05:58.389+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:58.390116+0000) 2019-10-13T08:05:58.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:05:58.727+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:58.727454+0000) 2019-10-13T08:05:58.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:05:58.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:58.727495+0000) 2019-10-13T08:06:00.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:00.033+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.033748+0000) 2019-10-13T08:06:00.036 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:00.034+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.034578+0000) 2019-10-13T08:06:00.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:00.034+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.034810+0000) 2019-10-13T08:06:00.036 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:00.034+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.035074+0000) 2019-10-13T08:06:00.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:00.034+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.035237+0000) 2019-10-13T08:06:00.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:00.034+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.035376+0000) 2019-10-13T08:06:00.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:00.035+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.035487+0000) 2019-10-13T08:06:00.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:00.035+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.036088+0000) 2019-10-13T08:06:00.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:00.038+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 135974 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:01.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:01.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:01.981127+0000) 2019-10-13T08:06:02.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:02.545+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:02.546688+0000) 2019-10-13T08:06:02.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:02.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:02.841604+0000) 2019-10-13T08:06:02.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:02.841+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:02.841775+0000) 2019-10-13T08:06:05.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:05.040+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.040837+0000) 2019-10-13T08:06:05.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:05.040+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.040806+0000) 2019-10-13T08:06:05.043 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:05.041+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.041475+0000) 2019-10-13T08:06:05.043 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:05.041+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.041560+0000) 2019-10-13T08:06:05.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:05.041+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.041905+0000) 2019-10-13T08:06:05.044 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:05.041+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.042155+0000) 2019-10-13T08:06:05.044 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:05.041+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.042266+0000) 2019-10-13T08:06:05.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:05.042+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.042836+0000) 2019-10-13T08:06:05.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:05.045+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136007 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:07.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:07.841+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:07.841584+0000) 2019-10-13T08:06:08.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:08.390+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:08.390327+0000) 2019-10-13T08:06:08.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:08.727+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:08.727647+0000) 2019-10-13T08:06:08.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:08.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:08.727653+0000) 2019-10-13T08:06:10.049 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:10.047+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.048185+0000) 2019-10-13T08:06:10.050 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:10.048+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.048570+0000) 2019-10-13T08:06:10.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:10.048+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.048657+0000) 2019-10-13T08:06:10.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:10.047+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.048992+0000) 2019-10-13T08:06:10.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:10.048+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.048962+0000) 2019-10-13T08:06:10.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:10.048+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.049116+0000) 2019-10-13T08:06:10.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:10.049+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.050119+0000) 2019-10-13T08:06:10.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:10.049+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.051364+0000) 2019-10-13T08:06:10.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:10.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:11.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:11.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:11.981312+0000) 2019-10-13T08:06:12.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:12.547+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:12.548775+0000) 2019-10-13T08:06:12.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:12.841+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:12.841957+0000) 2019-10-13T08:06:12.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:12.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:12.841799+0000) 2019-10-13T08:06:15.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:15.053+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.054864+0000) 2019-10-13T08:06:15.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:15.053+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.055378+0000) 2019-10-13T08:06:15.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:15.054+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.055642+0000) 2019-10-13T08:06:15.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:15.054+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.055854+0000) 2019-10-13T08:06:15.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:15.054+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.056184+0000) 2019-10-13T08:06:15.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:15.054+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.056320+0000) 2019-10-13T08:06:15.058 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:15.055+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.056600+0000) 2019-10-13T08:06:15.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:15.055+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.057452+0000) 2019-10-13T08:06:15.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:15.058+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:17.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:17.840+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:17.841731+0000) 2019-10-13T08:06:18.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:18.389+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:18.390544+0000) 2019-10-13T08:06:18.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:18.726+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:18.727857+0000) 2019-10-13T08:06:18.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:18.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:18.727856+0000) 2019-10-13T08:06:20.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:20.060+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.061743+0000) 2019-10-13T08:06:20.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:20.060+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.061930+0000) 2019-10-13T08:06:20.064 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:20.060+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.062260+0000) 2019-10-13T08:06:20.064 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:20.061+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.062427+0000) 2019-10-13T08:06:20.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:20.061+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.062365+0000) 2019-10-13T08:06:20.065 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:20.061+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.062932+0000) 2019-10-13T08:06:20.065 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:20.061+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.063023+0000) 2019-10-13T08:06:20.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:20.063+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.063752+0000) 2019-10-13T08:06:20.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:20.066+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136100 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:21.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:21.980+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:21.981482+0000) 2019-10-13T08:06:22.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:22.550+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:22.551307+0000) 2019-10-13T08:06:22.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:22.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:22.841987+0000) 2019-10-13T08:06:22.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:22.841+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:22.842103+0000) 2019-10-13T08:06:25.070 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:25.067+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.069303+0000) 2019-10-13T08:06:25.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:25.068+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.069803+0000) 2019-10-13T08:06:25.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:25.068+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.069256+0000) 2019-10-13T08:06:25.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:25.068+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.070058+0000) 2019-10-13T08:06:25.072 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:25.068+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.069916+0000) 2019-10-13T08:06:25.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:25.068+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.070243+0000) 2019-10-13T08:06:25.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:25.069+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.070986+0000) 2019-10-13T08:06:25.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:25.070+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.071153+0000) 2019-10-13T08:06:25.076 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:25.073+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136133 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:27.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:27.841+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:27.841926+0000) 2019-10-13T08:06:28.076 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:06:28.076 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:28.081 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:06:28.081 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:28.391 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:28.389+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:28.390724+0000) 2019-10-13T08:06:28.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:28.727+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:28.728036+0000) 2019-10-13T08:06:28.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:28.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:28.728035+0000) 2019-10-13T08:06:30.077 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:30.075+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.076559+0000) 2019-10-13T08:06:30.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:30.075+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.076614+0000) 2019-10-13T08:06:30.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:30.076+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.077740+0000) 2019-10-13T08:06:30.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:30.076+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.077852+0000) 2019-10-13T08:06:30.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:30.076+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.078127+0000) 2019-10-13T08:06:30.079 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:30.077+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.078749+0000) 2019-10-13T08:06:30.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:30.077+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.079131+0000) 2019-10-13T08:06:30.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:30.079+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.079849+0000) 2019-10-13T08:06:30.083 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:30.081+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:31.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:31.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:31.981680+0000) 2019-10-13T08:06:32.555 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:32.553+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:32.553653+0000) 2019-10-13T08:06:32.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:32.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:32.842166+0000) 2019-10-13T08:06:32.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:32.841+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:32.842208+0000) 2019-10-13T08:06:35.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:35.082+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.084314+0000) 2019-10-13T08:06:35.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:35.083+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.084268+0000) 2019-10-13T08:06:35.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:35.083+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.084804+0000) 2019-10-13T08:06:35.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:35.083+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.085048+0000) 2019-10-13T08:06:35.087 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:35.084+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.085411+0000) 2019-10-13T08:06:35.087 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:35.084+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.085635+0000) 2019-10-13T08:06:35.087 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:35.084+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.085710+0000) 2019-10-13T08:06:35.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:35.086+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.087409+0000) 2019-10-13T08:06:35.091 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:35.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:37.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:37.840+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:37.842139+0000) 2019-10-13T08:06:38.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:38.390+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:38.390921+0000) 2019-10-13T08:06:38.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:38.727+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:38.728230+0000) 2019-10-13T08:06:38.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:38.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:38.728233+0000) 2019-10-13T08:06:40.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:40.090+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.091468+0000) 2019-10-13T08:06:40.093 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:40.090+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.092279+0000) 2019-10-13T08:06:40.093 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:40.091+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.092472+0000) 2019-10-13T08:06:40.094 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:40.091+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.092674+0000) 2019-10-13T08:06:40.094 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:40.091+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.092957+0000) 2019-10-13T08:06:40.095 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:40.091+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.093162+0000) 2019-10-13T08:06:40.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:40.091+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.093369+0000) 2019-10-13T08:06:40.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:40.093+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.094273+0000) 2019-10-13T08:06:40.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:40.096+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:41.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:41.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:41.981814+0000) 2019-10-13T08:06:42.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:42.555+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:42.555993+0000) 2019-10-13T08:06:42.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:42.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:42.842348+0000) 2019-10-13T08:06:42.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:42.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:42.842374+0000) 2019-10-13T08:06:45.100 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:45.099+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.099407+0000) 2019-10-13T08:06:45.100 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:45.099+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.099596+0000) 2019-10-13T08:06:45.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:45.098+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.099376+0000) 2019-10-13T08:06:45.102 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:45.099+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.099978+0000) 2019-10-13T08:06:45.102 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:45.100+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.100580+0000) 2019-10-13T08:06:45.102 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:45.100+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.100746+0000) 2019-10-13T08:06:45.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:45.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.101173+0000) 2019-10-13T08:06:45.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:45.101+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.101916+0000) 2019-10-13T08:06:45.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:45.104+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136259 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:47.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:47.841+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:47.842341+0000) 2019-10-13T08:06:48.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:48.390+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:48.391131+0000) 2019-10-13T08:06:48.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:48.728+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:48.728443+0000) 2019-10-13T08:06:48.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:48.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:48.728444+0000) 2019-10-13T08:06:50.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:50.105+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.106293+0000) 2019-10-13T08:06:50.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:50.106+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.107145+0000) 2019-10-13T08:06:50.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:50.107+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.107389+0000) 2019-10-13T08:06:50.109 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:50.107+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.107598+0000) 2019-10-13T08:06:50.109 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:50.107+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.107696+0000) 2019-10-13T08:06:50.109 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:50.107+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.107796+0000) 2019-10-13T08:06:50.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:50.107+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.108007+0000) 2019-10-13T08:06:50.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:50.107+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.108367+0000) 2019-10-13T08:06:50.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:50.111+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136290 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:51.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:51.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:51.981987+0000) 2019-10-13T08:06:52.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:52.557+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:52.558320+0000) 2019-10-13T08:06:52.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:52.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:52.842550+0000) 2019-10-13T08:06:52.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:52.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:52.842559+0000) 2019-10-13T08:06:55.115 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:06:55.113+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.113770+0000) 2019-10-13T08:06:55.115 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:55.113+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.113931+0000) 2019-10-13T08:06:55.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:55.113+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.114047+0000) 2019-10-13T08:06:55.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:06:55.113+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.113984+0000) 2019-10-13T08:06:55.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:55.114+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.114568+0000) 2019-10-13T08:06:55.117 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:06:55.114+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.114846+0000) 2019-10-13T08:06:55.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:55.114+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.115019+0000) 2019-10-13T08:06:55.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:06:55.115+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.116097+0000) 2019-10-13T08:06:55.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:06:55.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:06:57.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:06:57.842+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:57.842477+0000) 2019-10-13T08:06:58.189 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:06:58.189 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:58.193 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:06:58.194 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:58.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:06:58.391+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:58.391327+0000) 2019-10-13T08:06:58.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:06:58.728+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:58.728607+0000) 2019-10-13T08:06:58.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:06:58.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:58.728607+0000) 2019-10-13T08:07:00.122 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:00.120+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.120802+0000) 2019-10-13T08:07:00.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:00.120+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.121013+0000) 2019-10-13T08:07:00.123 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:00.120+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.121263+0000) 2019-10-13T08:07:00.123 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:00.121+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.121547+0000) 2019-10-13T08:07:00.123 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:00.121+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.121663+0000) 2019-10-13T08:07:00.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:00.121+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.122197+0000) 2019-10-13T08:07:00.124 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:00.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.121562+0000) 2019-10-13T08:07:00.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:00.121+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.123546+0000) 2019-10-13T08:07:00.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:00.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136352 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:01.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:01.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:01.982144+0000) 2019-10-13T08:07:02.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:02.559+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:02.560709+0000) 2019-10-13T08:07:02.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:02.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:02.842706+0000) 2019-10-13T08:07:02.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:02.841+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:02.842782+0000) 2019-10-13T08:07:05.128 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:05.126+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.127646+0000) 2019-10-13T08:07:05.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:05.126+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.127772+0000) 2019-10-13T08:07:05.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:05.126+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.127881+0000) 2019-10-13T08:07:05.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:05.127+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.128538+0000) 2019-10-13T08:07:05.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:05.127+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.128740+0000) 2019-10-13T08:07:05.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:05.127+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.128704+0000) 2019-10-13T08:07:05.131 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:05.127+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.129234+0000) 2019-10-13T08:07:05.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:05.128+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.129940+0000) 2019-10-13T08:07:05.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:05.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:07.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:07.841+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:07.842635+0000) 2019-10-13T08:07:08.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:08.390+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:08.391483+0000) 2019-10-13T08:07:08.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:08.727+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:08.728789+0000) 2019-10-13T08:07:08.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:08.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:08.728789+0000) 2019-10-13T08:07:10.136 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:10.133+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.135197+0000) 2019-10-13T08:07:10.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:10.134+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.135935+0000) 2019-10-13T08:07:10.137 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:10.134+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.136092+0000) 2019-10-13T08:07:10.138 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:10.134+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.136193+0000) 2019-10-13T08:07:10.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:10.135+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.136567+0000) 2019-10-13T08:07:10.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:10.135+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.136736+0000) 2019-10-13T08:07:10.138 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:10.135+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.137153+0000) 2019-10-13T08:07:10.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:10.136+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.138520+0000) 2019-10-13T08:07:10.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:10.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136416 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:11.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:11.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:11.982324+0000) 2019-10-13T08:07:12.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:12.562+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:12.563202+0000) 2019-10-13T08:07:12.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:12.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:12.842827+0000) 2019-10-13T08:07:12.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:12.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:12.842946+0000) 2019-10-13T08:07:15.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:15.141+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.143269+0000) 2019-10-13T08:07:15.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:15.142+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.143620+0000) 2019-10-13T08:07:15.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:15.142+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.143051+0000) 2019-10-13T08:07:15.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:15.142+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.143750+0000) 2019-10-13T08:07:15.146 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:15.142+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.144188+0000) 2019-10-13T08:07:15.146 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:15.142+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.144288+0000) 2019-10-13T08:07:15.146 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:15.143+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.145135+0000) 2019-10-13T08:07:15.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:15.145+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.146667+0000) 2019-10-13T08:07:15.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:15.147+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136447 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:17.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:17.842+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:17.842822+0000) 2019-10-13T08:07:18.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:18.390+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:18.391659+0000) 2019-10-13T08:07:18.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:18.728+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:18.728978+0000) 2019-10-13T08:07:18.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:18.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:18.728978+0000) 2019-10-13T08:07:20.152 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:20.149+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.151261+0000) 2019-10-13T08:07:20.153 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:20.150+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.151627+0000) 2019-10-13T08:07:20.153 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:20.150+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.151792+0000) 2019-10-13T08:07:20.154 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:20.150+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.151565+0000) 2019-10-13T08:07:20.154 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:20.150+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.152310+0000) 2019-10-13T08:07:20.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:20.151+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.152464+0000) 2019-10-13T08:07:20.154 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:20.151+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.152592+0000) 2019-10-13T08:07:20.155 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:20.153+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.153911+0000) 2019-10-13T08:07:20.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:20.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136478 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:21.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:21.981+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:21.982537+0000) 2019-10-13T08:07:22.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:22.565+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:22.565578+0000) 2019-10-13T08:07:22.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:22.841+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:22.843042+0000) 2019-10-13T08:07:22.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:22.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:22.843053+0000) 2019-10-13T08:07:25.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:25.157+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.158539+0000) 2019-10-13T08:07:25.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:25.157+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.158344+0000) 2019-10-13T08:07:25.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:25.157+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.158931+0000) 2019-10-13T08:07:25.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:25.157+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.159232+0000) 2019-10-13T08:07:25.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:25.157+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.159122+0000) 2019-10-13T08:07:25.161 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:25.157+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.159353+0000) 2019-10-13T08:07:25.162 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:25.158+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.159576+0000) 2019-10-13T08:07:25.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:25.160+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.161573+0000) 2019-10-13T08:07:25.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:25.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136511 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:27.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:27.842+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:27.843033+0000) 2019-10-13T08:07:28.322 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:07:28.323 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:28.327 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:07:28.327 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:28.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:28.391+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:28.391801+0000) 2019-10-13T08:07:28.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:28.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:28.729211+0000) 2019-10-13T08:07:28.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:28.728+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:28.729211+0000) 2019-10-13T08:07:30.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:30.165+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.165402+0000) 2019-10-13T08:07:30.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:30.165+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.165506+0000) 2019-10-13T08:07:30.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:30.165+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.165847+0000) 2019-10-13T08:07:30.168 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:30.164+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.165691+0000) 2019-10-13T08:07:30.168 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:30.165+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.166250+0000) 2019-10-13T08:07:30.168 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:30.166+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.166548+0000) 2019-10-13T08:07:30.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:30.166+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.166587+0000) 2019-10-13T08:07:30.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:30.166+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.167625+0000) 2019-10-13T08:07:30.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:30.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136542 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:31.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:31.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:31.982710+0000) 2019-10-13T08:07:32.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:32.567+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:32.567927+0000) 2019-10-13T08:07:32.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:32.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:32.843162+0000) 2019-10-13T08:07:32.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:32.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:32.843213+0000) 2019-10-13T08:07:35.173 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:35.171+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.172478+0000) 2019-10-13T08:07:35.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:35.172+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.173215+0000) 2019-10-13T08:07:35.175 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:35.172+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.173310+0000) 2019-10-13T08:07:35.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:35.173+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.173460+0000) 2019-10-13T08:07:35.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:35.173+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.173633+0000) 2019-10-13T08:07:35.175 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:35.173+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.173863+0000) 2019-10-13T08:07:35.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:35.173+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.174214+0000) 2019-10-13T08:07:35.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:35.174+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.175712+0000) 2019-10-13T08:07:35.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:35.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:37.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:37.842+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:37.843226+0000) 2019-10-13T08:07:38.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:38.391+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:38.391978+0000) 2019-10-13T08:07:38.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:38.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:38.729450+0000) 2019-10-13T08:07:38.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:38.729+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:38.729450+0000) 2019-10-13T08:07:40.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:40.180+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.180881+0000) 2019-10-13T08:07:40.182 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:40.180+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.181090+0000) 2019-10-13T08:07:40.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:40.180+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.181146+0000) 2019-10-13T08:07:40.183 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:40.181+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.181435+0000) 2019-10-13T08:07:40.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:40.181+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.181741+0000) 2019-10-13T08:07:40.183 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:40.181+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.181904+0000) 2019-10-13T08:07:40.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:40.181+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.182262+0000) 2019-10-13T08:07:40.185 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:40.183+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.184262+0000) 2019-10-13T08:07:40.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:40.186+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136604 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:41.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:41.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:41.982924+0000) 2019-10-13T08:07:42.571 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:42.570+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:42.570301+0000) 2019-10-13T08:07:42.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:42.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:42.843345+0000) 2019-10-13T08:07:42.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:42.843+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:42.843359+0000) 2019-10-13T08:07:45.189 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:45.188+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.188646+0000) 2019-10-13T08:07:45.190 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:45.188+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.189220+0000) 2019-10-13T08:07:45.190 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:45.189+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.189418+0000) 2019-10-13T08:07:45.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:45.189+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.189529+0000) 2019-10-13T08:07:45.191 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:45.188+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.189350+0000) 2019-10-13T08:07:45.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:45.189+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.189871+0000) 2019-10-13T08:07:45.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:45.189+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.190228+0000) 2019-10-13T08:07:45.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:45.189+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.190517+0000) 2019-10-13T08:07:45.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:45.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:47.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:47.842+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:47.843403+0000) 2019-10-13T08:07:48.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:48.391+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:48.392150+0000) 2019-10-13T08:07:48.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:48.728+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:48.729627+0000) 2019-10-13T08:07:48.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:48.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:48.729627+0000) 2019-10-13T08:07:50.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:50.194+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.196029+0000) 2019-10-13T08:07:50.197 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:50.194+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.196186+0000) 2019-10-13T08:07:50.197 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:50.194+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.196292+0000) 2019-10-13T08:07:50.198 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:50.195+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.196459+0000) 2019-10-13T08:07:50.198 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:50.196+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.197068+0000) 2019-10-13T08:07:50.198 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:50.196+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.197704+0000) 2019-10-13T08:07:50.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:50.196+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.197907+0000) 2019-10-13T08:07:50.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:50.198+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.198901+0000) 2019-10-13T08:07:50.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:50.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:51.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:51.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:51.983117+0000) 2019-10-13T08:07:52.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:52.571+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:52.572796+0000) 2019-10-13T08:07:52.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:52.842+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:52.843534+0000) 2019-10-13T08:07:52.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:52.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:52.843560+0000) 2019-10-13T08:07:55.205 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:55.202+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.204050+0000) 2019-10-13T08:07:55.205 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:07:55.203+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.204419+0000) 2019-10-13T08:07:55.206 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:07:55.203+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.205251+0000) 2019-10-13T08:07:55.206 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:55.203+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.205129+0000) 2019-10-13T08:07:55.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:07:55.203+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.205336+0000) 2019-10-13T08:07:55.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:55.204+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.205574+0000) 2019-10-13T08:07:55.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:55.204+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.205803+0000) 2019-10-13T08:07:55.207 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:07:55.204+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.206196+0000) 2019-10-13T08:07:55.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:07:55.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:07:57.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:07:57.842+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:57.843523+0000) 2019-10-13T08:07:58.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:07:58.391+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:58.392311+0000) 2019-10-13T08:07:58.481 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:07:58.481 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:58.486 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:07:58.486 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:58.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:07:58.729+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:58.729809+0000) 2019-10-13T08:07:58.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:07:58.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:58.729809+0000) 2019-10-13T08:08:00.213 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:00.210+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.212151+0000) 2019-10-13T08:08:00.214 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:00.210+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.212333+0000) 2019-10-13T08:08:00.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:00.211+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.212813+0000) 2019-10-13T08:08:00.214 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:00.211+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.212926+0000) 2019-10-13T08:08:00.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:00.210+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.212102+0000) 2019-10-13T08:08:00.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:00.211+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.213291+0000) 2019-10-13T08:08:00.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:00.212+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.213429+0000) 2019-10-13T08:08:00.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:00.213+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.215607+0000) 2019-10-13T08:08:00.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:00.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136730 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:01.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:01.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:01.983295+0000) 2019-10-13T08:08:02.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:02.574+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:02.575152+0000) 2019-10-13T08:08:02.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:02.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:02.843723+0000) 2019-10-13T08:08:02.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:02.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:02.843715+0000) 2019-10-13T08:08:05.220 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:05.217+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.219354+0000) 2019-10-13T08:08:05.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:05.218+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.219624+0000) 2019-10-13T08:08:05.221 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:05.218+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.219717+0000) 2019-10-13T08:08:05.222 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:05.218+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.219468+0000) 2019-10-13T08:08:05.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:05.218+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.220108+0000) 2019-10-13T08:08:05.223 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:05.219+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.220811+0000) 2019-10-13T08:08:05.223 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:05.219+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.221012+0000) 2019-10-13T08:08:05.223 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:05.221+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.222184+0000) 2019-10-13T08:08:05.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:05.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:07.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:07.843+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:07.843671+0000) 2019-10-13T08:08:08.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:08.391+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:08.392479+0000) 2019-10-13T08:08:08.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:08.729+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:08.730013+0000) 2019-10-13T08:08:08.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:08.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:08.730012+0000) 2019-10-13T08:08:10.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:10.225+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.227068+0000) 2019-10-13T08:08:10.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:10.226+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.227469+0000) 2019-10-13T08:08:10.228 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:10.226+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.227738+0000) 2019-10-13T08:08:10.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:10.226+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.227484+0000) 2019-10-13T08:08:10.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:10.226+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.228133+0000) 2019-10-13T08:08:10.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:10.226+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.228258+0000) 2019-10-13T08:08:10.230 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:10.226+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.227609+0000) 2019-10-13T08:08:10.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:10.229+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.229761+0000) 2019-10-13T08:08:10.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:10.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136794 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:11.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:11.982+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:11.983495+0000) 2019-10-13T08:08:12.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:12.576+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:12.577649+0000) 2019-10-13T08:08:12.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:12.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:12.843908+0000) 2019-10-13T08:08:12.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:12.843+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:12.843860+0000) 2019-10-13T08:08:15.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:15.234+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.234627+0000) 2019-10-13T08:08:15.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:15.234+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.234867+0000) 2019-10-13T08:08:15.236 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:15.234+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.235226+0000) 2019-10-13T08:08:15.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:15.234+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.235130+0000) 2019-10-13T08:08:15.237 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:15.235+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.235566+0000) 2019-10-13T08:08:15.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:15.235+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.235774+0000) 2019-10-13T08:08:15.237 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:15.235+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.236091+0000) 2019-10-13T08:08:15.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:15.236+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.237174+0000) 2019-10-13T08:08:15.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:15.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:17.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:17.843+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:17.843825+0000) 2019-10-13T08:08:18.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:18.392+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:18.392677+0000) 2019-10-13T08:08:18.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:18.729+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:18.730223+0000) 2019-10-13T08:08:18.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:18.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:18.730223+0000) 2019-10-13T08:08:20.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:20.241+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.241919+0000) 2019-10-13T08:08:20.243 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:20.241+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.242297+0000) 2019-10-13T08:08:20.243 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:20.241+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.242340+0000) 2019-10-13T08:08:20.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:20.242+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.242541+0000) 2019-10-13T08:08:20.244 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:20.240+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.241701+0000) 2019-10-13T08:08:20.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:20.242+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.242796+0000) 2019-10-13T08:08:20.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:20.242+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.242910+0000) 2019-10-13T08:08:20.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:20.244+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.244824+0000) 2019-10-13T08:08:20.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:20.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136856 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:21.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:21.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:21.983726+0000) 2019-10-13T08:08:22.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:22.579+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:22.580111+0000) 2019-10-13T08:08:22.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:22.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:22.844088+0000) 2019-10-13T08:08:22.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:22.843+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:22.844020+0000) 2019-10-13T08:08:25.248 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:25.246+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.247345+0000) 2019-10-13T08:08:25.249 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:25.247+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.247904+0000) 2019-10-13T08:08:25.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:25.247+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.248160+0000) 2019-10-13T08:08:25.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:25.247+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.248374+0000) 2019-10-13T08:08:25.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:25.248+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.248524+0000) 2019-10-13T08:08:25.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:25.248+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.248682+0000) 2019-10-13T08:08:25.250 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:25.248+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.249050+0000) 2019-10-13T08:08:25.251 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:25.248+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.249453+0000) 2019-10-13T08:08:25.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:25.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136889 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:27.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:27.843+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:27.843997+0000) 2019-10-13T08:08:28.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:28.392+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:28.392870+0000) 2019-10-13T08:08:28.643 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:08:28.644 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:28.648 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:08:28.648 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:28.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:28.730+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:28.730442+0000) 2019-10-13T08:08:28.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:28.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:28.730442+0000) 2019-10-13T08:08:30.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:30.255+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.255807+0000) 2019-10-13T08:08:30.257 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:30.255+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.256131+0000) 2019-10-13T08:08:30.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:30.255+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.256376+0000) 2019-10-13T08:08:30.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:30.254+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.255643+0000) 2019-10-13T08:08:30.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:30.256+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.256785+0000) 2019-10-13T08:08:30.258 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:30.256+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.256963+0000) 2019-10-13T08:08:30.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:30.256+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.257104+0000) 2019-10-13T08:08:30.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:30.256+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.257413+0000) 2019-10-13T08:08:30.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:30.261+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136920 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:31.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:31.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:31.983861+0000) 2019-10-13T08:08:32.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:32.582+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:32.582398+0000) 2019-10-13T08:08:32.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:32.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:32.844249+0000) 2019-10-13T08:08:32.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:32.843+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:32.844219+0000) 2019-10-13T08:08:35.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:35.262+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.263310+0000) 2019-10-13T08:08:35.266 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:35.261+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.263047+0000) 2019-10-13T08:08:35.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:35.261+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.263195+0000) 2019-10-13T08:08:35.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:35.261+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.263376+0000) 2019-10-13T08:08:35.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:35.262+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.263805+0000) 2019-10-13T08:08:35.267 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:35.262+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.264097+0000) 2019-10-13T08:08:35.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:35.262+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.264385+0000) 2019-10-13T08:08:35.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:35.265+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.266581+0000) 2019-10-13T08:08:35.269 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:35.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:37.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:37.842+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:37.844180+0000) 2019-10-13T08:08:38.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:38.392+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:38.393049+0000) 2019-10-13T08:08:38.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:38.729+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:38.730680+0000) 2019-10-13T08:08:38.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:38.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:38.730681+0000) 2019-10-13T08:08:40.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:40.268+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.269804+0000) 2019-10-13T08:08:40.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:40.268+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.270020+0000) 2019-10-13T08:08:40.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:40.268+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.270179+0000) 2019-10-13T08:08:40.272 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:40.269+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.269991+0000) 2019-10-13T08:08:40.272 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:40.269+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.270707+0000) 2019-10-13T08:08:40.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:40.269+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.270922+0000) 2019-10-13T08:08:40.273 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:40.269+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.271326+0000) 2019-10-13T08:08:40.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:40.272+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.273373+0000) 2019-10-13T08:08:40.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:40.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 136982 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:41.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:41.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:41.984045+0000) 2019-10-13T08:08:42.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:42.583+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:42.584688+0000) 2019-10-13T08:08:42.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:42.843+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:42.844415+0000) 2019-10-13T08:08:42.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:42.844+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:42.844372+0000) 2019-10-13T08:08:45.278 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:45.275+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.277358+0000) 2019-10-13T08:08:45.279 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:45.276+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.277533+0000) 2019-10-13T08:08:45.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:45.276+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.277678+0000) 2019-10-13T08:08:45.279 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:45.276+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.277904+0000) 2019-10-13T08:08:45.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:45.276+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.277936+0000) 2019-10-13T08:08:45.280 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:45.276+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.277739+0000) 2019-10-13T08:08:45.280 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:45.276+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.278364+0000) 2019-10-13T08:08:45.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:45.278+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.280283+0000) 2019-10-13T08:08:45.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:45.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:47.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:47.843+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:47.844380+0000) 2019-10-13T08:08:48.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:48.392+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:48.393246+0000) 2019-10-13T08:08:48.732 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:48.730+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:48.730859+0000) 2019-10-13T08:08:48.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:48.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:48.730858+0000) 2019-10-13T08:08:50.286 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:50.283+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.284840+0000) 2019-10-13T08:08:50.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:50.283+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.285150+0000) 2019-10-13T08:08:50.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:50.283+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.285285+0000) 2019-10-13T08:08:50.287 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:50.284+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.285415+0000) 2019-10-13T08:08:50.287 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:50.284+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.285576+0000) 2019-10-13T08:08:50.287 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:50.284+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.285888+0000) 2019-10-13T08:08:50.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:50.284+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.285947+0000) 2019-10-13T08:08:50.288 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:50.285+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.287571+0000) 2019-10-13T08:08:50.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:50.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:51.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:51.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:51.984225+0000) 2019-10-13T08:08:52.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:52.585+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:52.586716+0000) 2019-10-13T08:08:52.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:52.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:52.844603+0000) 2019-10-13T08:08:52.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:52.842+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:52.844559+0000) 2019-10-13T08:08:55.293 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:55.290+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.291779+0000) 2019-10-13T08:08:55.293 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:08:55.290+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.292111+0000) 2019-10-13T08:08:55.293 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:08:55.290+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.292342+0000) 2019-10-13T08:08:55.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:55.291+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.292574+0000) 2019-10-13T08:08:55.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:55.291+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.292629+0000) 2019-10-13T08:08:55.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:55.291+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.293034+0000) 2019-10-13T08:08:55.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:08:55.291+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.293041+0000) 2019-10-13T08:08:55.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:08:55.292+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.294470+0000) 2019-10-13T08:08:55.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:08:55.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:08:57.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:08:57.844+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:57.844610+0000) 2019-10-13T08:08:58.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:08:58.392+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:58.393424+0000) 2019-10-13T08:08:58.732 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:08:58.730+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:58.731027+0000) 2019-10-13T08:08:58.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:08:58.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:58.731027+0000) 2019-10-13T08:08:58.798 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:08:58.798 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:58.802 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:08:58.802 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:00.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:00.297+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.299203+0000) 2019-10-13T08:09:00.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:00.297+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.299118+0000) 2019-10-13T08:09:00.300 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:00.297+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.299350+0000) 2019-10-13T08:09:00.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:00.298+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.299429+0000) 2019-10-13T08:09:00.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:00.298+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.299923+0000) 2019-10-13T08:09:00.302 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:00.298+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.300136+0000) 2019-10-13T08:09:00.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:00.298+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.300361+0000) 2019-10-13T08:09:00.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:00.301+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.302261+0000) 2019-10-13T08:09:00.305 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:00.303+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:01.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:01.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:01.984414+0000) 2019-10-13T08:09:02.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:02.588+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:02.589011+0000) 2019-10-13T08:09:02.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:02.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:02.844737+0000) 2019-10-13T08:09:02.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:02.844+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:02.844680+0000) 2019-10-13T08:09:05.308 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:05.306+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.307033+0000) 2019-10-13T08:09:05.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:05.306+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.307126+0000) 2019-10-13T08:09:05.308 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:05.306+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.307220+0000) 2019-10-13T08:09:05.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:05.307+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.307585+0000) 2019-10-13T08:09:05.309 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:05.307+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.307719+0000) 2019-10-13T08:09:05.309 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:05.306+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.307013+0000) 2019-10-13T08:09:05.310 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:05.307+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.308547+0000) 2019-10-13T08:09:05.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:05.307+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.307929+0000) 2019-10-13T08:09:05.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:05.312+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:07.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:07.844+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:07.844786+0000) 2019-10-13T08:09:08.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:08.393+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:08.393605+0000) 2019-10-13T08:09:08.732 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:08.730+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:08.731242+0000) 2019-10-13T08:09:08.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:08.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:08.731242+0000) 2019-10-13T08:09:10.316 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:10.313+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.314292+0000) 2019-10-13T08:09:10.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:10.313+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.314134+0000) 2019-10-13T08:09:10.317 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:10.314+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.314564+0000) 2019-10-13T08:09:10.317 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:10.314+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.314605+0000) 2019-10-13T08:09:10.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:10.314+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.315070+0000) 2019-10-13T08:09:10.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:10.314+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.315266+0000) 2019-10-13T08:09:10.318 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:10.314+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.315001+0000) 2019-10-13T08:09:10.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:10.315+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.316482+0000) 2019-10-13T08:09:10.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:10.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:11.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:11.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:11.984599+0000) 2019-10-13T08:09:12.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:12.590+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:12.591403+0000) 2019-10-13T08:09:12.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:12.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:12.844926+0000) 2019-10-13T08:09:12.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:12.844+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:12.844890+0000) 2019-10-13T08:09:15.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:15.319+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.320244+0000) 2019-10-13T08:09:15.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:15.320+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.320955+0000) 2019-10-13T08:09:15.323 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:15.320+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.321214+0000) 2019-10-13T08:09:15.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:15.320+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.321315+0000) 2019-10-13T08:09:15.323 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:15.321+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.321456+0000) 2019-10-13T08:09:15.323 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:15.321+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.321793+0000) 2019-10-13T08:09:15.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:15.321+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.321662+0000) 2019-10-13T08:09:15.324 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:15.321+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.322504+0000) 2019-10-13T08:09:15.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:15.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:17.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:17.844+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:17.844979+0000) 2019-10-13T08:09:18.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:18.393+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:18.393796+0000) 2019-10-13T08:09:18.732 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:18.731+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:18.731412+0000) 2019-10-13T08:09:18.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:18.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:18.731412+0000) 2019-10-13T08:09:20.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:20.327+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.328478+0000) 2019-10-13T08:09:20.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:20.327+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.328651+0000) 2019-10-13T08:09:20.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:20.327+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.329083+0000) 2019-10-13T08:09:20.331 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:20.327+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.329251+0000) 2019-10-13T08:09:20.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:20.328+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.329446+0000) 2019-10-13T08:09:20.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:20.328+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.329460+0000) 2019-10-13T08:09:20.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:20.328+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.329859+0000) 2019-10-13T08:09:20.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:20.331+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.331926+0000) 2019-10-13T08:09:20.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:20.332+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:21.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:21.983+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:21.984794+0000) 2019-10-13T08:09:22.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:22.593+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:22.593680+0000) 2019-10-13T08:09:22.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:22.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:22.845096+0000) 2019-10-13T08:09:22.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:22.844+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:22.845093+0000) 2019-10-13T08:09:25.336 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:25.334+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.335528+0000) 2019-10-13T08:09:25.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:25.334+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.335764+0000) 2019-10-13T08:09:25.338 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:25.334+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.335637+0000) 2019-10-13T08:09:25.338 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:25.334+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.336228+0000) 2019-10-13T08:09:25.338 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:25.335+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.336430+0000) 2019-10-13T08:09:25.338 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:25.335+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.336609+0000) 2019-10-13T08:09:25.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:25.335+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.337050+0000) 2019-10-13T08:09:25.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:25.338+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.339026+0000) 2019-10-13T08:09:25.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:25.340+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:27.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:27.844+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:27.845177+0000) 2019-10-13T08:09:28.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:28.393+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:28.393992+0000) 2019-10-13T08:09:28.732 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:28.730+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:28.731601+0000) 2019-10-13T08:09:28.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:28.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:28.731601+0000) 2019-10-13T08:09:28.944 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:09:28.945 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:28.949 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:09:28.949 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:30.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:30.342+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.344107+0000) 2019-10-13T08:09:30.345 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:30.342+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.344261+0000) 2019-10-13T08:09:30.346 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:30.344+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.344770+0000) 2019-10-13T08:09:30.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:30.343+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.344832+0000) 2019-10-13T08:09:30.346 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:30.343+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.345003+0000) 2019-10-13T08:09:30.347 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:30.343+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.345214+0000) 2019-10-13T08:09:30.347 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:30.344+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.345553+0000) 2019-10-13T08:09:30.347 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:30.345+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.346668+0000) 2019-10-13T08:09:30.350 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:30.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:31.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:31.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:31.984942+0000) 2019-10-13T08:09:32.597 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:32.596+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:32.596251+0000) 2019-10-13T08:09:32.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:32.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:32.845234+0000) 2019-10-13T08:09:32.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:32.845+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:32.845292+0000) 2019-10-13T08:09:35.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:35.350+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.351560+0000) 2019-10-13T08:09:35.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:35.353+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.354131+0000) 2019-10-13T08:09:35.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:35.349+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.350810+0000) 2019-10-13T08:09:35.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:35.349+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.350995+0000) 2019-10-13T08:09:35.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:35.349+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.351158+0000) 2019-10-13T08:09:35.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:35.349+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.351339+0000) 2019-10-13T08:09:35.596 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:35.350+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.351738+0000) 2019-10-13T08:09:35.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:35.350+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.352044+0000) 2019-10-13T08:09:35.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:35.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:37.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:37.844+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:37.845307+0000) 2019-10-13T08:09:38.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:38.393+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:38.394181+0000) 2019-10-13T08:09:38.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:38.731+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:38.731830+0000) 2019-10-13T08:09:38.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:38.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:38.731830+0000) 2019-10-13T08:09:40.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:40.357+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.358894+0000) 2019-10-13T08:09:40.360 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:40.357+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.359231+0000) 2019-10-13T08:09:40.361 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:40.357+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.359242+0000) 2019-10-13T08:09:40.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:40.358+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.359436+0000) 2019-10-13T08:09:40.361 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:40.358+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.359723+0000) 2019-10-13T08:09:40.361 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:40.358+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.359876+0000) 2019-10-13T08:09:40.362 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:40.358+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.360060+0000) 2019-10-13T08:09:40.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:40.358+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.360111+0000) 2019-10-13T08:09:40.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:40.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137360 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:41.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:41.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:41.985150+0000) 2019-10-13T08:09:42.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:42.597+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:42.598601+0000) 2019-10-13T08:09:42.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:42.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:42.845402+0000) 2019-10-13T08:09:42.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:42.844+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:42.845467+0000) 2019-10-13T08:09:45.367 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:45.364+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.366131+0000) 2019-10-13T08:09:45.368 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:45.365+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.366575+0000) 2019-10-13T08:09:45.368 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:45.365+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.366653+0000) 2019-10-13T08:09:45.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:45.365+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.367124+0000) 2019-10-13T08:09:45.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:45.366+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.367411+0000) 2019-10-13T08:09:45.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:45.365+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.367050+0000) 2019-10-13T08:09:45.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:45.366+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.367675+0000) 2019-10-13T08:09:45.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:45.367+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.369252+0000) 2019-10-13T08:09:45.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:45.369+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:47.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:47.845+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:47.845474+0000) 2019-10-13T08:09:48.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:48.394+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:48.394382+0000) 2019-10-13T08:09:48.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:48.731+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:48.732065+0000) 2019-10-13T08:09:48.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:48.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:48.732065+0000) 2019-10-13T08:09:50.373 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:50.372+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.372920+0000) 2019-10-13T08:09:50.374 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:50.372+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.373190+0000) 2019-10-13T08:09:50.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:50.372+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.373236+0000) 2019-10-13T08:09:50.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:50.373+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.373412+0000) 2019-10-13T08:09:50.375 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:50.373+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.373716+0000) 2019-10-13T08:09:50.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:50.373+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.374086+0000) 2019-10-13T08:09:50.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:50.373+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.374239+0000) 2019-10-13T08:09:50.376 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:50.373+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.374497+0000) 2019-10-13T08:09:50.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:50.377+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:51.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:51.985+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:51.985348+0000) 2019-10-13T08:09:52.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:52.600+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:52.601139+0000) 2019-10-13T08:09:52.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:52.844+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:52.845633+0000) 2019-10-13T08:09:52.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:52.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:52.845614+0000) 2019-10-13T08:09:55.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:09:55.379+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.379956+0000) 2019-10-13T08:09:55.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:55.379+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.380204+0000) 2019-10-13T08:09:55.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:09:55.379+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.379992+0000) 2019-10-13T08:09:55.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:55.380+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.380523+0000) 2019-10-13T08:09:55.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:55.380+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.380616+0000) 2019-10-13T08:09:55.382 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:55.380+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.380769+0000) 2019-10-13T08:09:55.382 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:09:55.380+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.380894+0000) 2019-10-13T08:09:55.383 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:09:55.380+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.381178+0000) 2019-10-13T08:09:55.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:09:55.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137455 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:09:57.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:09:57.845+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:57.845683+0000) 2019-10-13T08:09:58.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:09:58.394+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:58.394587+0000) 2019-10-13T08:09:58.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:09:58.731+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:58.732243+0000) 2019-10-13T08:09:58.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:09:58.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:58.732243+0000) 2019-10-13T08:09:59.098 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:09:59.098 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:59.103 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:09:59.103 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:00.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:00.385+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.386288+0000) 2019-10-13T08:10:00.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:00.386+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.386650+0000) 2019-10-13T08:10:00.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:00.386+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.386826+0000) 2019-10-13T08:10:00.389 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:00.386+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.387341+0000) 2019-10-13T08:10:00.389 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:00.386+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.387116+0000) 2019-10-13T08:10:00.389 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:00.386+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.387211+0000) 2019-10-13T08:10:00.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:00.387+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.387417+0000) 2019-10-13T08:10:00.390 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:00.388+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.388773+0000) 2019-10-13T08:10:00.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:00.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:01.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:01.985+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:01.985569+0000) 2019-10-13T08:10:02.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:02.602+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:02.603527+0000) 2019-10-13T08:10:02.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:02.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:02.845735+0000) 2019-10-13T08:10:02.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:02.845+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:02.845817+0000) 2019-10-13T08:10:05.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:05.393+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.393578+0000) 2019-10-13T08:10:05.395 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:05.393+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.393803+0000) 2019-10-13T08:10:05.395 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:05.393+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.393865+0000) 2019-10-13T08:10:05.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:05.393+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.394180+0000) 2019-10-13T08:10:05.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:05.392+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.393698+0000) 2019-10-13T08:10:05.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:05.394+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.394568+0000) 2019-10-13T08:10:05.396 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:05.394+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.394962+0000) 2019-10-13T08:10:05.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:05.395+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.396625+0000) 2019-10-13T08:10:05.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:05.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137519 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:07.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:07.844+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:07.845818+0000) 2019-10-13T08:10:08.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:08.393+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:08.394777+0000) 2019-10-13T08:10:08.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:08.731+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:08.732457+0000) 2019-10-13T08:10:08.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:08.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:08.732458+0000) 2019-10-13T08:10:10.402 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:10.400+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.401115+0000) 2019-10-13T08:10:10.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:10.400+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.401786+0000) 2019-10-13T08:10:10.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:10.400+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.401837+0000) 2019-10-13T08:10:10.403 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:10.400+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.402301+0000) 2019-10-13T08:10:10.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:10.401+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.402910+0000) 2019-10-13T08:10:10.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:10.401+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.403047+0000) 2019-10-13T08:10:10.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:10.401+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.402535+0000) 2019-10-13T08:10:10.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:10.403+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.404554+0000) 2019-10-13T08:10:10.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:10.406+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137550 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:11.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:11.984+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:11.985732+0000) 2019-10-13T08:10:12.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:12.605+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:12.605899+0000) 2019-10-13T08:10:12.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:12.844+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:12.845879+0000) 2019-10-13T08:10:12.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:12.845+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:12.846027+0000) 2019-10-13T08:10:15.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:15.407+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.409336+0000) 2019-10-13T08:10:15.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:15.408+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.409457+0000) 2019-10-13T08:10:15.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:15.408+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.409782+0000) 2019-10-13T08:10:15.411 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:15.408+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.409259+0000) 2019-10-13T08:10:15.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:15.408+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.410218+0000) 2019-10-13T08:10:15.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:15.409+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.410439+0000) 2019-10-13T08:10:15.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:15.409+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.410522+0000) 2019-10-13T08:10:15.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:15.411+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.412069+0000) 2019-10-13T08:10:15.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:15.412+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:17.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:17.845+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:17.846001+0000) 2019-10-13T08:10:18.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:18.394+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:18.394984+0000) 2019-10-13T08:10:18.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:18.731+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:18.732680+0000) 2019-10-13T08:10:18.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:18.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:18.732680+0000) 2019-10-13T08:10:20.416 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:20.414+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.414807+0000) 2019-10-13T08:10:20.416 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:20.413+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.414953+0000) 2019-10-13T08:10:20.417 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:20.413+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.415343+0000) 2019-10-13T08:10:20.417 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:20.414+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.415514+0000) 2019-10-13T08:10:20.417 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:20.414+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.416035+0000) 2019-10-13T08:10:20.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:20.414+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.416206+0000) 2019-10-13T08:10:20.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:20.414+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.416347+0000) 2019-10-13T08:10:20.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:20.416+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.417454+0000) 2019-10-13T08:10:20.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:20.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:21.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:21.985+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:21.985907+0000) 2019-10-13T08:10:22.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:22.607+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:22.608400+0000) 2019-10-13T08:10:22.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:22.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:22.846014+0000) 2019-10-13T08:10:22.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:22.846+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:22.846199+0000) 2019-10-13T08:10:25.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:25.420+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.422261+0000) 2019-10-13T08:10:25.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:25.421+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.422411+0000) 2019-10-13T08:10:25.424 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:25.421+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.422717+0000) 2019-10-13T08:10:25.424 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:25.421+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.422807+0000) 2019-10-13T08:10:25.424 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:25.421+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.422209+0000) 2019-10-13T08:10:25.425 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:25.421+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.423145+0000) 2019-10-13T08:10:25.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:25.422+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.423947+0000) 2019-10-13T08:10:25.426 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:25.424+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.425566+0000) 2019-10-13T08:10:25.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:25.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:27.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:27.845+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:27.846191+0000) 2019-10-13T08:10:28.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:28.394+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:28.395196+0000) 2019-10-13T08:10:28.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:28.731+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:28.732887+0000) 2019-10-13T08:10:28.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:28.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:28.732887+0000) 2019-10-13T08:10:29.261 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:10:29.261 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:29.266 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:10:29.266 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:30.430 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:30.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.428808+0000) 2019-10-13T08:10:30.430 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:30.427+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.429176+0000) 2019-10-13T08:10:30.430 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:30.427+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.429030+0000) 2019-10-13T08:10:30.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:30.427+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.429179+0000) 2019-10-13T08:10:30.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:30.428+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.429716+0000) 2019-10-13T08:10:30.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:30.428+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.430133+0000) 2019-10-13T08:10:30.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:30.429+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.430461+0000) 2019-10-13T08:10:30.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:30.430+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.431857+0000) 2019-10-13T08:10:30.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:30.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:31.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:31.985+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:31.986094+0000) 2019-10-13T08:10:32.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:32.609+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:32.610802+0000) 2019-10-13T08:10:32.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:32.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:32.846143+0000) 2019-10-13T08:10:32.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:32.845+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:32.846395+0000) 2019-10-13T08:10:35.437 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:35.436+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.436440+0000) 2019-10-13T08:10:35.437 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:35.436+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.436572+0000) 2019-10-13T08:10:35.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:35.435+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.437124+0000) 2019-10-13T08:10:35.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:35.436+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.436926+0000) 2019-10-13T08:10:35.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:35.436+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.436976+0000) 2019-10-13T08:10:35.439 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:35.436+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.437204+0000) 2019-10-13T08:10:35.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:35.436+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.437329+0000) 2019-10-13T08:10:35.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:35.436+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.438134+0000) 2019-10-13T08:10:35.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:35.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:37.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:37.845+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:37.846341+0000) 2019-10-13T08:10:38.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:38.395+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:38.395390+0000) 2019-10-13T08:10:38.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:38.732+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:38.733069+0000) 2019-10-13T08:10:38.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:38.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:38.733069+0000) 2019-10-13T08:10:40.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:40.439+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.441698+0000) 2019-10-13T08:10:40.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:40.441+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.442216+0000) 2019-10-13T08:10:40.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:40.442+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.442391+0000) 2019-10-13T08:10:40.444 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:40.442+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.442629+0000) 2019-10-13T08:10:40.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:40.442+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.443243+0000) 2019-10-13T08:10:40.445 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:40.443+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.443428+0000) 2019-10-13T08:10:40.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:40.443+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.443586+0000) 2019-10-13T08:10:40.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:40.442+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.444567+0000) 2019-10-13T08:10:40.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:40.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:41.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:41.985+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:41.986227+0000) 2019-10-13T08:10:42.614 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:42.612+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:42.613325+0000) 2019-10-13T08:10:42.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:42.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:42.846292+0000) 2019-10-13T08:10:42.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:42.845+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:42.846542+0000) 2019-10-13T08:10:45.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:45.449+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.450437+0000) 2019-10-13T08:10:45.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:45.449+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.449758+0000) 2019-10-13T08:10:45.454 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:45.449+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.450001+0000) 2019-10-13T08:10:45.455 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:45.450+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.450391+0000) 2019-10-13T08:10:45.455 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:45.450+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.450517+0000) 2019-10-13T08:10:45.455 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:45.450+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.450785+0000) 2019-10-13T08:10:45.455 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:45.450+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.451175+0000) 2019-10-13T08:10:45.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:45.452+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.452809+0000) 2019-10-13T08:10:45.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:45.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:47.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:47.846+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:47.846541+0000) 2019-10-13T08:10:48.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:48.395+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:48.395617+0000) 2019-10-13T08:10:48.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:48.732+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:48.733235+0000) 2019-10-13T08:10:48.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:48.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:48.733233+0000) 2019-10-13T08:10:50.457 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:50.456+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.456936+0000) 2019-10-13T08:10:50.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:50.456+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.457232+0000) 2019-10-13T08:10:50.458 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:50.457+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.457420+0000) 2019-10-13T08:10:50.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:50.457+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.457801+0000) 2019-10-13T08:10:50.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:50.457+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.457907+0000) 2019-10-13T08:10:50.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:50.457+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.457792+0000) 2019-10-13T08:10:50.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:50.457+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.458347+0000) 2019-10-13T08:10:50.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:50.458+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.459174+0000) 2019-10-13T08:10:50.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:50.462+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:51.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:51.986+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:51.986449+0000) 2019-10-13T08:10:52.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:52.615+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:52.615875+0000) 2019-10-13T08:10:52.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:52.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:52.846467+0000) 2019-10-13T08:10:52.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:52.846+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:52.846681+0000) 2019-10-13T08:10:55.465 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:55.462+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.464012+0000) 2019-10-13T08:10:55.465 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:55.462+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.464238+0000) 2019-10-13T08:10:55.466 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:10:55.463+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.464666+0000) 2019-10-13T08:10:55.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:55.463+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.464635+0000) 2019-10-13T08:10:55.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:55.463+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.464564+0000) 2019-10-13T08:10:55.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:10:55.463+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.464944+0000) 2019-10-13T08:10:55.467 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:10:55.463+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.465250+0000) 2019-10-13T08:10:55.469 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:10:55.467+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.468191+0000) 2019-10-13T08:10:55.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:10:55.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:10:57.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:10:57.845+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:57.846743+0000) 2019-10-13T08:10:58.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:10:58.394+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:58.395815+0000) 2019-10-13T08:10:58.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:10:58.732+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:58.733479+0000) 2019-10-13T08:10:58.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:10:58.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:58.733472+0000) 2019-10-13T08:10:59.364 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:10:59.365 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:59.369 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:10:59.369 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:00.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:00.470+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.471945+0000) 2019-10-13T08:11:00.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:00.471+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.472048+0000) 2019-10-13T08:11:00.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:00.471+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.472428+0000) 2019-10-13T08:11:00.474 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:00.471+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.472535+0000) 2019-10-13T08:11:00.474 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:00.471+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.472994+0000) 2019-10-13T08:11:00.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:00.471+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.472892+0000) 2019-10-13T08:11:00.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:00.471+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.473293+0000) 2019-10-13T08:11:00.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:00.473+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.474158+0000) 2019-10-13T08:11:00.478 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:00.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:01.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:01.985+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:01.986649+0000) 2019-10-13T08:11:02.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:02.617+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:02.618262+0000) 2019-10-13T08:11:02.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:02.845+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:02.846634+0000) 2019-10-13T08:11:02.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:02.846+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:02.846897+0000) 2019-10-13T08:11:05.480 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:05.478+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.479453+0000) 2019-10-13T08:11:05.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:05.478+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.479594+0000) 2019-10-13T08:11:05.481 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:05.478+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.480038+0000) 2019-10-13T08:11:05.482 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:05.479+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.480481+0000) 2019-10-13T08:11:05.482 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:05.479+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.480777+0000) 2019-10-13T08:11:05.482 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:05.479+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.481007+0000) 2019-10-13T08:11:05.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:05.479+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.480592+0000) 2019-10-13T08:11:05.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:05.481+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.481940+0000) 2019-10-13T08:11:05.485 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:05.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:07.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:07.846+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:07.846934+0000) 2019-10-13T08:11:08.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:08.395+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:08.396033+0000) 2019-10-13T08:11:08.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:08.733+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:08.733683+0000) 2019-10-13T08:11:08.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:08.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:08.733683+0000) 2019-10-13T08:11:10.487 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:10.485+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.486792+0000) 2019-10-13T08:11:10.488 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:10.485+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.487001+0000) 2019-10-13T08:11:10.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:10.485+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.487107+0000) 2019-10-13T08:11:10.489 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:10.486+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.486915+0000) 2019-10-13T08:11:10.489 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:10.486+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.487444+0000) 2019-10-13T08:11:10.489 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:10.486+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.488078+0000) 2019-10-13T08:11:10.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:10.487+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.488446+0000) 2019-10-13T08:11:10.490 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:10.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.489718+0000) 2019-10-13T08:11:10.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:10.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:11.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:11.986+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:11.986814+0000) 2019-10-13T08:11:12.622 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:12.620+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:12.620703+0000) 2019-10-13T08:11:12.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:12.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:12.846817+0000) 2019-10-13T08:11:12.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:12.846+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:12.847093+0000) 2019-10-13T08:11:15.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:15.493+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.494515+0000) 2019-10-13T08:11:15.495 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:15.493+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.494786+0000) 2019-10-13T08:11:15.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:15.493+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.495360+0000) 2019-10-13T08:11:15.496 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:15.494+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.495592+0000) 2019-10-13T08:11:15.496 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:15.494+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.495623+0000) 2019-10-13T08:11:15.497 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:15.494+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.495437+0000) 2019-10-13T08:11:15.497 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:15.494+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.496028+0000) 2019-10-13T08:11:15.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:15.496+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.497613+0000) 2019-10-13T08:11:15.501 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:15.499+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:17.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:17.846+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:17.847130+0000) 2019-10-13T08:11:18.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:18.395+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:18.396223+0000) 2019-10-13T08:11:18.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:18.732+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:18.733892+0000) 2019-10-13T08:11:18.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:18.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:18.733892+0000) 2019-10-13T08:11:20.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:20.502+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.503365+0000) 2019-10-13T08:11:20.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:20.503+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.503667+0000) 2019-10-13T08:11:20.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:20.503+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.503817+0000) 2019-10-13T08:11:20.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:20.503+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.503953+0000) 2019-10-13T08:11:20.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:20.503+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.504088+0000) 2019-10-13T08:11:20.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:20.502+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.503233+0000) 2019-10-13T08:11:20.507 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:20.504+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.504620+0000) 2019-10-13T08:11:20.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:20.505+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.506258+0000) 2019-10-13T08:11:20.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:20.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 137990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:21.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:21.986+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:21.987005+0000) 2019-10-13T08:11:22.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:22.621+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:22.623272+0000) 2019-10-13T08:11:22.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:22.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:22.847019+0000) 2019-10-13T08:11:22.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:22.846+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:22.847278+0000) 2019-10-13T08:11:25.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:25.510+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.510832+0000) 2019-10-13T08:11:25.512 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:25.510+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.511003+0000) 2019-10-13T08:11:25.513 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:25.510+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.510980+0000) 2019-10-13T08:11:25.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:25.511+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.511462+0000) 2019-10-13T08:11:25.513 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:25.511+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.511809+0000) 2019-10-13T08:11:25.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:25.509+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.510844+0000) 2019-10-13T08:11:25.514 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:25.511+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.512071+0000) 2019-10-13T08:11:25.515 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:25.512+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.514157+0000) 2019-10-13T08:11:25.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:25.516+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138023 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:27.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:27.847+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:27.847322+0000) 2019-10-13T08:11:28.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:28.396+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:28.396436+0000) 2019-10-13T08:11:28.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:28.733+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:28.734068+0000) 2019-10-13T08:11:28.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:28.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:28.734068+0000) 2019-10-13T08:11:29.535 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:11:29.536 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:29.540 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:11:29.540 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:30.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:30.518+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.518555+0000) 2019-10-13T08:11:30.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:30.518+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.518699+0000) 2019-10-13T08:11:30.520 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:30.518+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.518808+0000) 2019-10-13T08:11:30.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:30.518+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.519221+0000) 2019-10-13T08:11:30.521 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:30.519+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.519489+0000) 2019-10-13T08:11:30.521 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:30.517+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.518826+0000) 2019-10-13T08:11:30.521 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:30.519+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.519682+0000) 2019-10-13T08:11:30.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:30.519+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.521268+0000) 2019-10-13T08:11:30.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:30.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138054 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:31.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:31.987+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:31.987207+0000) 2019-10-13T08:11:32.626 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:32.624+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:32.625669+0000) 2019-10-13T08:11:32.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:32.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:32.847210+0000) 2019-10-13T08:11:32.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:32.845+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:32.847482+0000) 2019-10-13T08:11:35.527 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:35.525+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.525867+0000) 2019-10-13T08:11:35.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:35.525+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.525976+0000) 2019-10-13T08:11:35.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:35.525+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.526368+0000) 2019-10-13T08:11:35.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:35.524+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.525883+0000) 2019-10-13T08:11:35.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:35.526+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.526490+0000) 2019-10-13T08:11:35.528 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:35.526+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.526752+0000) 2019-10-13T08:11:35.528 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:35.526+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.526996+0000) 2019-10-13T08:11:35.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:35.525+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.527464+0000) 2019-10-13T08:11:35.532 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:35.531+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:37.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:37.847+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:37.847463+0000) 2019-10-13T08:11:38.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:38.395+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:38.396621+0000) 2019-10-13T08:11:38.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:38.733+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:38.734239+0000) 2019-10-13T08:11:38.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:38.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:38.734241+0000) 2019-10-13T08:11:40.534 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:40.532+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.533315+0000) 2019-10-13T08:11:40.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:40.532+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.533814+0000) 2019-10-13T08:11:40.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:40.532+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.533847+0000) 2019-10-13T08:11:40.536 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:40.532+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.533996+0000) 2019-10-13T08:11:40.536 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:40.532+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.534147+0000) 2019-10-13T08:11:40.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:40.533+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.534572+0000) 2019-10-13T08:11:40.536 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:40.533+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.534787+0000) 2019-10-13T08:11:40.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:40.535+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.536239+0000) 2019-10-13T08:11:40.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:40.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:41.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:41.986+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:41.987395+0000) 2019-10-13T08:11:42.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:42.627+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:42.628077+0000) 2019-10-13T08:11:42.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:42.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:42.847371+0000) 2019-10-13T08:11:42.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:42.846+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:42.847637+0000) 2019-10-13T08:11:45.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:45.540+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.541123+0000) 2019-10-13T08:11:45.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:45.540+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.541520+0000) 2019-10-13T08:11:45.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:45.540+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.541674+0000) 2019-10-13T08:11:45.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:45.540+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.542255+0000) 2019-10-13T08:11:45.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:45.541+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.542432+0000) 2019-10-13T08:11:45.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:45.541+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.542554+0000) 2019-10-13T08:11:45.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:45.541+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.542691+0000) 2019-10-13T08:11:45.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:45.543+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.544644+0000) 2019-10-13T08:11:45.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:45.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:47.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:47.846+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:47.847654+0000) 2019-10-13T08:11:48.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:48.396+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:48.396852+0000) 2019-10-13T08:11:48.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:48.733+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:48.734403+0000) 2019-10-13T08:11:48.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:48.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:48.734403+0000) 2019-10-13T08:11:50.549 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:50.547+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.547993+0000) 2019-10-13T08:11:50.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:50.547+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.548476+0000) 2019-10-13T08:11:50.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:50.547+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.548750+0000) 2019-10-13T08:11:50.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:50.547+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.548980+0000) 2019-10-13T08:11:50.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:50.547+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.549018+0000) 2019-10-13T08:11:50.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:50.547+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.549254+0000) 2019-10-13T08:11:50.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:50.548+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.549428+0000) 2019-10-13T08:11:50.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:50.549+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.550380+0000) 2019-10-13T08:11:50.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:50.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138180 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:51.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:51.986+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:51.987553+0000) 2019-10-13T08:11:52.631 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:52.630+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:52.630566+0000) 2019-10-13T08:11:52.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:52.846+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:52.847512+0000) 2019-10-13T08:11:52.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:52.847+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:52.847841+0000) 2019-10-13T08:11:55.555 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:11:55.553+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.554444+0000) 2019-10-13T08:11:55.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:55.553+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.554748+0000) 2019-10-13T08:11:55.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:55.553+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.554873+0000) 2019-10-13T08:11:55.556 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:11:55.554+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.555480+0000) 2019-10-13T08:11:55.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:55.554+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.555466+0000) 2019-10-13T08:11:55.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:11:55.554+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.555647+0000) 2019-10-13T08:11:55.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:55.554+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.555907+0000) 2019-10-13T08:11:55.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:11:55.556+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.556824+0000) 2019-10-13T08:11:55.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:11:55.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:11:57.848 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:11:57.847+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:57.847841+0000) 2019-10-13T08:11:58.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:11:58.396+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:58.397008+0000) 2019-10-13T08:11:58.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:11:58.733+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:58.734572+0000) 2019-10-13T08:11:58.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:11:58.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:58.734572+0000) 2019-10-13T08:11:59.689 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:11:59.690 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:59.694 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:11:59.694 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:00.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:00.560+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.562254+0000) 2019-10-13T08:12:00.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:00.561+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.562568+0000) 2019-10-13T08:12:00.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:00.561+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.562662+0000) 2019-10-13T08:12:00.564 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:00.561+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.562842+0000) 2019-10-13T08:12:00.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:00.561+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.563119+0000) 2019-10-13T08:12:00.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:00.562+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.562953+0000) 2019-10-13T08:12:00.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:00.562+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.563512+0000) 2019-10-13T08:12:00.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:00.564+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.564831+0000) 2019-10-13T08:12:00.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:00.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:01.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:01.987+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:01.987755+0000) 2019-10-13T08:12:02.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:02.632+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:02.633091+0000) 2019-10-13T08:12:02.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:02.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:02.847681+0000) 2019-10-13T08:12:02.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:02.847+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:02.848012+0000) 2019-10-13T08:12:05.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:05.568+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.570018+0000) 2019-10-13T08:12:05.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:05.569+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.570459+0000) 2019-10-13T08:12:05.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:05.569+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.570892+0000) 2019-10-13T08:12:05.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:05.569+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.570942+0000) 2019-10-13T08:12:05.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:05.569+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.571053+0000) 2019-10-13T08:12:05.579 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:05.569+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.570284+0000) 2019-10-13T08:12:05.580 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:05.571+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.572622+0000) 2019-10-13T08:12:05.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:05.569+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.571378+0000) 2019-10-13T08:12:05.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:05.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:07.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:07.847+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:07.848037+0000) 2019-10-13T08:12:08.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:08.396+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:08.397217+0000) 2019-10-13T08:12:08.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:08.734+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:08.734717+0000) 2019-10-13T08:12:08.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:08.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:08.734718+0000) 2019-10-13T08:12:10.577 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:10.575+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.576437+0000) 2019-10-13T08:12:10.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:10.576+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.577123+0000) 2019-10-13T08:12:10.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:10.576+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.577026+0000) 2019-10-13T08:12:10.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:10.577+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.577741+0000) 2019-10-13T08:12:10.581 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:10.577+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.577839+0000) 2019-10-13T08:12:10.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:10.577+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.578099+0000) 2019-10-13T08:12:10.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:10.577+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.578258+0000) 2019-10-13T08:12:10.582 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:10.579+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.580033+0000) 2019-10-13T08:12:10.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:10.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:11.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:11.987+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:11.987934+0000) 2019-10-13T08:12:12.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:12.635+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:12.635527+0000) 2019-10-13T08:12:12.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:12.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:12.847898+0000) 2019-10-13T08:12:12.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:12.848+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:12.848157+0000) 2019-10-13T08:12:15.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:15.583+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.583882+0000) 2019-10-13T08:12:15.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:15.583+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.584129+0000) 2019-10-13T08:12:15.585 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:15.583+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.584219+0000) 2019-10-13T08:12:15.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:15.583+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.584388+0000) 2019-10-13T08:12:15.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:15.582+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.584180+0000) 2019-10-13T08:12:15.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:15.584+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.584834+0000) 2019-10-13T08:12:15.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:15.584+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.585258+0000) 2019-10-13T08:12:15.587 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:15.584+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.586436+0000) 2019-10-13T08:12:15.589 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:15.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:17.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:17.847+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:17.848236+0000) 2019-10-13T08:12:18.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:18.397+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:18.397429+0000) 2019-10-13T08:12:18.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:18.734+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:18.734879+0000) 2019-10-13T08:12:18.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:18.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:18.734881+0000) 2019-10-13T08:12:20.591 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:20.589+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.590161+0000) 2019-10-13T08:12:20.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:20.589+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.590258+0000) 2019-10-13T08:12:20.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:20.588+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.590164+0000) 2019-10-13T08:12:20.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:20.590+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.590599+0000) 2019-10-13T08:12:20.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:20.590+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.590780+0000) 2019-10-13T08:12:20.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:20.590+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.590911+0000) 2019-10-13T08:12:20.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:20.591+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.591420+0000) 2019-10-13T08:12:20.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:20.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:20.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:20.591+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.593267+0000) 2019-10-13T08:12:21.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:21.988+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:21.988124+0000) 2019-10-13T08:12:22.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:22.637+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:22.637990+0000) 2019-10-13T08:12:22.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:22.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:22.848055+0000) 2019-10-13T08:12:22.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:22.847+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:22.848332+0000) 2019-10-13T08:12:25.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:25.595+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.596516+0000) 2019-10-13T08:12:25.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:25.595+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.596769+0000) 2019-10-13T08:12:25.598 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:25.595+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.597316+0000) 2019-10-13T08:12:25.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:25.596+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.597527+0000) 2019-10-13T08:12:25.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:25.596+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.597886+0000) 2019-10-13T08:12:25.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:25.596+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.598178+0000) 2019-10-13T08:12:25.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:25.596+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.598343+0000) 2019-10-13T08:12:25.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:25.597+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.599489+0000) 2019-10-13T08:12:25.603 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:25.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138401 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:27.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:27.847+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:27.848489+0000) 2019-10-13T08:12:28.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:28.396+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:28.397619+0000) 2019-10-13T08:12:28.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:28.733+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:28.735070+0000) 2019-10-13T08:12:28.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:28.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:28.735070+0000) 2019-10-13T08:12:29.812 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:12:29.812 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:29.816 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:12:29.816 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:30.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:30.603+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.603844+0000) 2019-10-13T08:12:30.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:30.605+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.606693+0000) 2019-10-13T08:12:30.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:30.602+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.603914+0000) 2019-10-13T08:12:30.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:30.602+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.604236+0000) 2019-10-13T08:12:30.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:30.603+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.604538+0000) 2019-10-13T08:12:30.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:30.603+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.604749+0000) 2019-10-13T08:12:30.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:30.603+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.604931+0000) 2019-10-13T08:12:30.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:30.603+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.605065+0000) 2019-10-13T08:12:30.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:30.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138432 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:31.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:31.987+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:31.988319+0000) 2019-10-13T08:12:32.641 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:32.639+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:32.640243+0000) 2019-10-13T08:12:32.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:32.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:32.848253+0000) 2019-10-13T08:12:32.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:32.847+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:32.848538+0000) 2019-10-13T08:12:35.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:35.608+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.610113+0000) 2019-10-13T08:12:35.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:35.609+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.610426+0000) 2019-10-13T08:12:35.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:35.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.610356+0000) 2019-10-13T08:12:35.612 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:35.609+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.610838+0000) 2019-10-13T08:12:35.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:35.609+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.611125+0000) 2019-10-13T08:12:35.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:35.609+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.611205+0000) 2019-10-13T08:12:35.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:35.610+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.611654+0000) 2019-10-13T08:12:35.614 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:35.612+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.613009+0000) 2019-10-13T08:12:35.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:35.614+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:37.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:37.847+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:37.848680+0000) 2019-10-13T08:12:38.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:38.397+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:38.397810+0000) 2019-10-13T08:12:38.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:38.734+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:38.735242+0000) 2019-10-13T08:12:38.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:38.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:38.735241+0000) 2019-10-13T08:12:40.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:40.615+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.617173+0000) 2019-10-13T08:12:40.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:40.615+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.617375+0000) 2019-10-13T08:12:40.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:40.616+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.617295+0000) 2019-10-13T08:12:40.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:40.616+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.617863+0000) 2019-10-13T08:12:40.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:40.616+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.618096+0000) 2019-10-13T08:12:40.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:40.616+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.618258+0000) 2019-10-13T08:12:40.620 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:40.617+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.618538+0000) 2019-10-13T08:12:40.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:40.618+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.619514+0000) 2019-10-13T08:12:40.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:40.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:41.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:41.987+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:41.988505+0000) 2019-10-13T08:12:42.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:42.642+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:42.642801+0000) 2019-10-13T08:12:42.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:42.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:42.848432+0000) 2019-10-13T08:12:42.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:42.848+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:42.848758+0000) 2019-10-13T08:12:45.625 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:45.623+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.624622+0000) 2019-10-13T08:12:45.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:45.623+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.624866+0000) 2019-10-13T08:12:45.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:45.623+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.625142+0000) 2019-10-13T08:12:45.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:45.623+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.625181+0000) 2019-10-13T08:12:45.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:45.624+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.625048+0000) 2019-10-13T08:12:45.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:45.624+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.625490+0000) 2019-10-13T08:12:45.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:45.624+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.626002+0000) 2019-10-13T08:12:45.628 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:45.626+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.627653+0000) 2019-10-13T08:12:45.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:45.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138527 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:47.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:47.848+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:47.848880+0000) 2019-10-13T08:12:48.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:48.397+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:48.397985+0000) 2019-10-13T08:12:48.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:48.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:48.735451+0000) 2019-10-13T08:12:48.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:48.734+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:48.735452+0000) 2019-10-13T08:12:50.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:50.630+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.631549+0000) 2019-10-13T08:12:50.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:50.630+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.631904+0000) 2019-10-13T08:12:50.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:50.631+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.632496+0000) 2019-10-13T08:12:50.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:50.630+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.632270+0000) 2019-10-13T08:12:50.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:50.631+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.632415+0000) 2019-10-13T08:12:50.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:50.631+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.632585+0000) 2019-10-13T08:12:50.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:50.631+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.632906+0000) 2019-10-13T08:12:50.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:50.632+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.633540+0000) 2019-10-13T08:12:50.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:50.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:51.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:51.988+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:51.988665+0000) 2019-10-13T08:12:52.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:52.644+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:52.645049+0000) 2019-10-13T08:12:52.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:52.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:52.848635+0000) 2019-10-13T08:12:52.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:52.848+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:52.848908+0000) 2019-10-13T08:12:55.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:55.639+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.639561+0000) 2019-10-13T08:12:55.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:12:55.639+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.639797+0000) 2019-10-13T08:12:55.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:12:55.639+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.640012+0000) 2019-10-13T08:12:55.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:12:55.639+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.640501+0000) 2019-10-13T08:12:55.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:55.639+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.640294+0000) 2019-10-13T08:12:55.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:55.640+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.640469+0000) 2019-10-13T08:12:55.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:55.640+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.640972+0000) 2019-10-13T08:12:55.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:12:55.642+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.642986+0000) 2019-10-13T08:12:55.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:12:55.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:12:57.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:12:57.848+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:57.849111+0000) 2019-10-13T08:12:58.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:12:58.397+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:58.398208+0000) 2019-10-13T08:12:58.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:12:58.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:58.735674+0000) 2019-10-13T08:12:58.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:12:58.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:58.735674+0000) 2019-10-13T08:12:59.991 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:12:59.991 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:59.996 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:12:59.997 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:00.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:00.647+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.647544+0000) 2019-10-13T08:13:00.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:00.646+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.647604+0000) 2019-10-13T08:13:00.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:00.647+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.648287+0000) 2019-10-13T08:13:00.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:00.647+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.648176+0000) 2019-10-13T08:13:00.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:00.648+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.648421+0000) 2019-10-13T08:13:00.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:00.648+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.648572+0000) 2019-10-13T08:13:00.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:00.648+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.648690+0000) 2019-10-13T08:13:00.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:00.648+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.649557+0000) 2019-10-13T08:13:00.654 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:00.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:01.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:01.988+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:01.988856+0000) 2019-10-13T08:13:02.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:02.646+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:02.647074+0000) 2019-10-13T08:13:02.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:02.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:02.848797+0000) 2019-10-13T08:13:02.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:02.848+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:02.849075+0000) 2019-10-13T08:13:05.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:05.654+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.655238+0000) 2019-10-13T08:13:05.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:05.654+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.655365+0000) 2019-10-13T08:13:05.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:05.655+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.655555+0000) 2019-10-13T08:13:05.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:05.655+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.656335+0000) 2019-10-13T08:13:05.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:05.655+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.656197+0000) 2019-10-13T08:13:05.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:05.656+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.656762+0000) 2019-10-13T08:13:05.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:05.656+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.656905+0000) 2019-10-13T08:13:05.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:05.657+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.657824+0000) 2019-10-13T08:13:05.661 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:05.660+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:07.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:07.849+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:07.849304+0000) 2019-10-13T08:13:08.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:08.398+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:08.398413+0000) 2019-10-13T08:13:08.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:08.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:08.735872+0000) 2019-10-13T08:13:08.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:08.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:08.735872+0000) 2019-10-13T08:13:10.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:10.661+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.662750+0000) 2019-10-13T08:13:10.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:10.662+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.663463+0000) 2019-10-13T08:13:10.664 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:10.661+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.663545+0000) 2019-10-13T08:13:10.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:10.662+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.663718+0000) 2019-10-13T08:13:10.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:10.662+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.663930+0000) 2019-10-13T08:13:10.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:10.662+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.664119+0000) 2019-10-13T08:13:10.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:10.663+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.664935+0000) 2019-10-13T08:13:10.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:10.663+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.664592+0000) 2019-10-13T08:13:10.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:10.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:11.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:11.987+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:11.989096+0000) 2019-10-13T08:13:12.650 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:12.648+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:12.649379+0000) 2019-10-13T08:13:12.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:12.847+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:12.848990+0000) 2019-10-13T08:13:12.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:12.847+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:12.849257+0000) 2019-10-13T08:13:15.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:15.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.669944+0000) 2019-10-13T08:13:15.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:15.669+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.670577+0000) 2019-10-13T08:13:15.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:15.669+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.670725+0000) 2019-10-13T08:13:15.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:15.669+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.670903+0000) 2019-10-13T08:13:15.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:15.669+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.670962+0000) 2019-10-13T08:13:15.672 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:15.670+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.671659+0000) 2019-10-13T08:13:15.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:15.670+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.672004+0000) 2019-10-13T08:13:15.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:15.671+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.673330+0000) 2019-10-13T08:13:15.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:15.675+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:17.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:17.848+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:17.849474+0000) 2019-10-13T08:13:18.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:18.397+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:18.398570+0000) 2019-10-13T08:13:18.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:18.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:18.736085+0000) 2019-10-13T08:13:18.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:18.734+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:18.736085+0000) 2019-10-13T08:13:20.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:20.676+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.678179+0000) 2019-10-13T08:13:20.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:20.676+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.678200+0000) 2019-10-13T08:13:20.680 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:20.677+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.678820+0000) 2019-10-13T08:13:20.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:20.677+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.679030+0000) 2019-10-13T08:13:20.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:20.677+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.679348+0000) 2019-10-13T08:13:20.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:20.677+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.679228+0000) 2019-10-13T08:13:20.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:20.678+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.679543+0000) 2019-10-13T08:13:20.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:20.679+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.681309+0000) 2019-10-13T08:13:20.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:20.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:21.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:21.988+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:21.989302+0000) 2019-10-13T08:13:22.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:22.650+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:22.651707+0000) 2019-10-13T08:13:22.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:22.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:22.849171+0000) 2019-10-13T08:13:22.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:22.848+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:22.849392+0000) 2019-10-13T08:13:25.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:25.684+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.685576+0000) 2019-10-13T08:13:25.686 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:25.684+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.685938+0000) 2019-10-13T08:13:25.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:25.684+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.686145+0000) 2019-10-13T08:13:25.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:25.684+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.686367+0000) 2019-10-13T08:13:25.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:25.685+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.686585+0000) 2019-10-13T08:13:25.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:25.685+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.686981+0000) 2019-10-13T08:13:25.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:25.685+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.686637+0000) 2019-10-13T08:13:25.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:25.687+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.688357+0000) 2019-10-13T08:13:25.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:25.690+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138779 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:27.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:27.848+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:27.849668+0000) 2019-10-13T08:13:28.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:28.398+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:28.398768+0000) 2019-10-13T08:13:28.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:28.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:28.736231+0000) 2019-10-13T08:13:28.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:28.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:28.736231+0000) 2019-10-13T08:13:30.129 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:13:30.129 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:30.133 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:13:30.133 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:30.694 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:30.691+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.693142+0000) 2019-10-13T08:13:30.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:30.691+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.693337+0000) 2019-10-13T08:13:30.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:30.692+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.693548+0000) 2019-10-13T08:13:30.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:30.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.693764+0000) 2019-10-13T08:13:30.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:30.692+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.694254+0000) 2019-10-13T08:13:30.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:30.693+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.694531+0000) 2019-10-13T08:13:30.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:30.693+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.694697+0000) 2019-10-13T08:13:30.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:30.695+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.695909+0000) 2019-10-13T08:13:30.698 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:30.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:31.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:31.988+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:31.989505+0000) 2019-10-13T08:13:32.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:32.653+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:32.654114+0000) 2019-10-13T08:13:32.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:32.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:32.849327+0000) 2019-10-13T08:13:32.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:32.849+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:32.849568+0000) 2019-10-13T08:13:35.700 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:35.698+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.699840+0000) 2019-10-13T08:13:35.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:35.698+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.700005+0000) 2019-10-13T08:13:35.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:35.698+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.700222+0000) 2019-10-13T08:13:35.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:35.699+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.700276+0000) 2019-10-13T08:13:35.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:35.699+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.700441+0000) 2019-10-13T08:13:35.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:35.699+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.700862+0000) 2019-10-13T08:13:35.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:35.699+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.701218+0000) 2019-10-13T08:13:35.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:35.702+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.703043+0000) 2019-10-13T08:13:35.706 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:35.704+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:37.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:37.849+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:37.849863+0000) 2019-10-13T08:13:38.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:38.398+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:38.398957+0000) 2019-10-13T08:13:38.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:38.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:38.736429+0000) 2019-10-13T08:13:38.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:38.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:38.736429+0000) 2019-10-13T08:13:40.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:40.707+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.707627+0000) 2019-10-13T08:13:40.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:40.707+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.708008+0000) 2019-10-13T08:13:40.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:40.707+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.708282+0000) 2019-10-13T08:13:40.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:40.708+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.708461+0000) 2019-10-13T08:13:40.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:40.708+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.708711+0000) 2019-10-13T08:13:40.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:40.708+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.708648+0000) 2019-10-13T08:13:40.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:40.707+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.708640+0000) 2019-10-13T08:13:40.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:40.709+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:40.709946+0000) 2019-10-13T08:13:40.713 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:40.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:41.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:41.989+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.989667+0000) 2019-10-13T08:13:42.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:42.656+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:42.656362+0000) 2019-10-13T08:13:42.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:42.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:42.849478+0000) 2019-10-13T08:13:42.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:42.849+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:42.849775+0000) 2019-10-13T08:13:45.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:45.714+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.714590+0000) 2019-10-13T08:13:45.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:45.714+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.714958+0000) 2019-10-13T08:13:45.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:45.714+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.715333+0000) 2019-10-13T08:13:45.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:45.715+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.715446+0000) 2019-10-13T08:13:45.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:45.715+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.715667+0000) 2019-10-13T08:13:45.717 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:45.715+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.715850+0000) 2019-10-13T08:13:45.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:45.714+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.715244+0000) 2019-10-13T08:13:45.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:45.717+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:45.717953+0000) 2019-10-13T08:13:45.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:45.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:47.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:47.849+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:47.850057+0000) 2019-10-13T08:13:48.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:48.398+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:48.399111+0000) 2019-10-13T08:13:48.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:48.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:48.736614+0000) 2019-10-13T08:13:48.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:48.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:48.736615+0000) 2019-10-13T08:13:50.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:50.721+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.721951+0000) 2019-10-13T08:13:50.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:50.721+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.722182+0000) 2019-10-13T08:13:50.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:50.721+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.722253+0000) 2019-10-13T08:13:50.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:50.722+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.722468+0000) 2019-10-13T08:13:50.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:50.722+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.722744+0000) 2019-10-13T08:13:50.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:50.721+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.722699+0000) 2019-10-13T08:13:50.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:50.722+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.723166+0000) 2019-10-13T08:13:50.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:50.723+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:50.724591+0000) 2019-10-13T08:13:50.728 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:50.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138936 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:51.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:51.989+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.989806+0000) 2019-10-13T08:13:52.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:52.658+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:52.658667+0000) 2019-10-13T08:13:52.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:52.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:52.849623+0000) 2019-10-13T08:13:52.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:52.849+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:52.849955+0000) 2019-10-13T08:13:55.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:13:55.728+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.728964+0000) 2019-10-13T08:13:55.730 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:55.728+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.729379+0000) 2019-10-13T08:13:55.730 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:55.729+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.729603+0000) 2019-10-13T08:13:55.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:55.729+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.729925+0000) 2019-10-13T08:13:55.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:13:55.730+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.730419+0000) 2019-10-13T08:13:55.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:55.730+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.730548+0000) 2019-10-13T08:13:55.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:13:55.730+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.730705+0000) 2019-10-13T08:13:55.733 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:13:55.731+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:55.732376+0000) 2019-10-13T08:13:55.736 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:13:55.735+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:13:57.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:13:57.849+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:57.850253+0000) 2019-10-13T08:13:58.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:13:58.398+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:58.399289+0000) 2019-10-13T08:13:58.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:13:58.735+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:58.736827+0000) 2019-10-13T08:13:58.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:13:58.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:58.736828+0000) 2019-10-13T08:14:00.284 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:14:00.284 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:00.288 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:14:00.288 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:00.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:00.734+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.736536+0000) 2019-10-13T08:14:00.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:00.735+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.736992+0000) 2019-10-13T08:14:00.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:00.735+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.737245+0000) 2019-10-13T08:14:00.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:00.735+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.737335+0000) 2019-10-13T08:14:00.739 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:00.736+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.737415+0000) 2019-10-13T08:14:00.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:00.736+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.737516+0000) 2019-10-13T08:14:00.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:00.736+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.737689+0000) 2019-10-13T08:14:00.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:00.737+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:00.739514+0000) 2019-10-13T08:14:00.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:00.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 138998 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:01.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:01.989+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.989974+0000) 2019-10-13T08:14:02.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:02.660+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:02.661011+0000) 2019-10-13T08:14:02.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:02.848+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:02.849806+0000) 2019-10-13T08:14:02.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:02.848+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:02.850107+0000) 2019-10-13T08:14:05.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:05.741+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.743177+0000) 2019-10-13T08:14:05.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:05.741+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.743310+0000) 2019-10-13T08:14:05.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:05.742+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.743422+0000) 2019-10-13T08:14:05.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:05.742+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.743664+0000) 2019-10-13T08:14:05.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:05.741+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.743300+0000) 2019-10-13T08:14:05.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:05.742+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.744468+0000) 2019-10-13T08:14:05.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:05.742+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.744138+0000) 2019-10-13T08:14:05.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:05.743+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:05.744557+0000) 2019-10-13T08:14:05.750 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:05.748+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139031 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:07.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:07.849+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:07.850449+0000) 2019-10-13T08:14:08.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:08.398+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:08.399491+0000) 2019-10-13T08:14:08.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:08.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:08.737060+0000) 2019-10-13T08:14:08.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:08.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:08.737061+0000) 2019-10-13T08:14:10.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:10.750+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.751491+0000) 2019-10-13T08:14:10.752 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:10.750+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.751757+0000) 2019-10-13T08:14:10.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:10.749+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.751388+0000) 2019-10-13T08:14:10.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:10.750+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.751891+0000) 2019-10-13T08:14:10.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:10.750+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.752185+0000) 2019-10-13T08:14:10.754 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:10.751+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.752415+0000) 2019-10-13T08:14:10.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:10.751+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.752943+0000) 2019-10-13T08:14:10.755 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:10.752+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:10.754541+0000) 2019-10-13T08:14:10.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:10.756+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139062 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:11.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:11.988+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.990198+0000) 2019-10-13T08:14:12.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:12.662+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:12.663367+0000) 2019-10-13T08:14:12.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:12.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:12.849996+0000) 2019-10-13T08:14:12.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:12.849+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:12.850280+0000) 2019-10-13T08:14:15.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:15.757+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.759322+0000) 2019-10-13T08:14:15.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:15.758+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.759709+0000) 2019-10-13T08:14:15.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:15.758+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.759956+0000) 2019-10-13T08:14:15.761 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:15.758+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.759950+0000) 2019-10-13T08:14:15.761 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:15.758+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.760173+0000) 2019-10-13T08:14:15.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:15.759+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.760111+0000) 2019-10-13T08:14:15.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:15.760+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.761357+0000) 2019-10-13T08:14:15.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:15.759+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:15.760629+0000) 2019-10-13T08:14:15.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:15.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:17.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:17.850+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:17.850657+0000) 2019-10-13T08:14:18.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:18.399+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:18.399648+0000) 2019-10-13T08:14:18.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:18.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:18.737273+0000) 2019-10-13T08:14:18.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:18.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:18.737273+0000) 2019-10-13T08:14:20.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:20.765+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.767325+0000) 2019-10-13T08:14:20.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:20.766+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.767608+0000) 2019-10-13T08:14:20.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:20.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.767381+0000) 2019-10-13T08:14:20.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:20.766+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.768130+0000) 2019-10-13T08:14:20.769 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:20.766+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.768380+0000) 2019-10-13T08:14:20.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:20.767+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.768401+0000) 2019-10-13T08:14:20.770 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:20.767+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.768672+0000) 2019-10-13T08:14:20.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:20.768+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:20.769717+0000) 2019-10-13T08:14:20.773 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:20.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139124 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:21.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:21.989+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.990434+0000) 2019-10-13T08:14:22.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:22.665+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:22.665772+0000) 2019-10-13T08:14:22.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:22.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:22.850211+0000) 2019-10-13T08:14:22.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:22.849+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:22.850456+0000) 2019-10-13T08:14:25.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:25.774+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.774787+0000) 2019-10-13T08:14:25.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:25.774+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.774952+0000) 2019-10-13T08:14:25.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:25.774+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.775216+0000) 2019-10-13T08:14:25.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:25.775+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.775417+0000) 2019-10-13T08:14:25.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:25.775+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.775583+0000) 2019-10-13T08:14:25.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:25.775+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.775721+0000) 2019-10-13T08:14:25.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:25.775+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.775874+0000) 2019-10-13T08:14:25.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:25.777+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:25.777864+0000) 2019-10-13T08:14:25.781 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:25.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139157 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:27.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:27.850+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:27.850860+0000) 2019-10-13T08:14:28.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:28.399+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:28.399832+0000) 2019-10-13T08:14:28.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:28.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:28.737469+0000) 2019-10-13T08:14:28.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:28.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:28.737469+0000) 2019-10-13T08:14:30.388 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:14:30.388 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:30.393 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:14:30.393 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:30.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:30.782+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.782739+0000) 2019-10-13T08:14:30.784 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:30.782+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.783107+0000) 2019-10-13T08:14:30.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:30.782+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.783222+0000) 2019-10-13T08:14:30.784 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:30.783+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.783463+0000) 2019-10-13T08:14:30.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:30.783+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.783667+0000) 2019-10-13T08:14:30.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:30.782+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.783055+0000) 2019-10-13T08:14:30.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:30.783+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.783965+0000) 2019-10-13T08:14:30.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:30.784+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:30.785691+0000) 2019-10-13T08:14:30.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:30.788+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:31.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:31.990+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.990627+0000) 2019-10-13T08:14:32.669 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:32.667+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:32.668229+0000) 2019-10-13T08:14:32.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:32.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:32.850344+0000) 2019-10-13T08:14:32.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:32.850+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:32.850596+0000) 2019-10-13T08:14:35.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:35.789+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.789893+0000) 2019-10-13T08:14:35.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:35.789+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.790292+0000) 2019-10-13T08:14:35.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:35.790+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.790586+0000) 2019-10-13T08:14:35.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:35.790+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.791012+0000) 2019-10-13T08:14:35.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:35.790+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.791131+0000) 2019-10-13T08:14:35.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:35.790+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.791226+0000) 2019-10-13T08:14:35.792 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:35.791+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.791668+0000) 2019-10-13T08:14:35.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:35.792+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:35.793278+0000) 2019-10-13T08:14:35.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:35.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:37.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:37.850+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:37.851057+0000) 2019-10-13T08:14:38.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:38.399+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:38.400029+0000) 2019-10-13T08:14:38.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:38.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:38.737645+0000) 2019-10-13T08:14:38.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:38.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:38.737645+0000) 2019-10-13T08:14:40.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:40.796+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.797197+0000) 2019-10-13T08:14:40.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:40.797+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.797500+0000) 2019-10-13T08:14:40.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:40.796+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.797380+0000) 2019-10-13T08:14:40.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:40.797+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.797942+0000) 2019-10-13T08:14:40.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:40.797+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.798223+0000) 2019-10-13T08:14:40.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:40.797+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.798383+0000) 2019-10-13T08:14:40.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:40.798+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.798669+0000) 2019-10-13T08:14:40.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:40.799+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:40.799904+0000) 2019-10-13T08:14:40.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:40.802+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:41.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:41.990+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.990878+0000) 2019-10-13T08:14:42.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:42.670+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:42.670346+0000) 2019-10-13T08:14:42.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:42.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:42.850492+0000) 2019-10-13T08:14:42.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:42.850+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:42.850753+0000) 2019-10-13T08:14:45.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:45.803+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.804717+0000) 2019-10-13T08:14:45.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:45.803+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.804873+0000) 2019-10-13T08:14:45.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:45.804+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.805566+0000) 2019-10-13T08:14:45.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:45.804+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.805764+0000) 2019-10-13T08:14:45.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:45.804+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.806045+0000) 2019-10-13T08:14:45.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:45.804+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.806216+0000) 2019-10-13T08:14:45.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:45.805+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.806177+0000) 2019-10-13T08:14:45.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:45.806+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:45.807159+0000) 2019-10-13T08:14:45.810 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:45.808+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:47.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:47.850+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:47.851254+0000) 2019-10-13T08:14:48.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:48.399+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:48.400237+0000) 2019-10-13T08:14:48.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:48.736+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:48.737805+0000) 2019-10-13T08:14:48.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:48.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:48.737805+0000) 2019-10-13T08:14:50.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:50.810+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.812356+0000) 2019-10-13T08:14:50.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:50.811+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.812599+0000) 2019-10-13T08:14:50.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:50.811+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.812746+0000) 2019-10-13T08:14:50.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:50.812+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.813327+0000) 2019-10-13T08:14:50.814 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:50.811+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.813214+0000) 2019-10-13T08:14:50.815 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:50.812+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.813809+0000) 2019-10-13T08:14:50.815 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:50.812+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.814040+0000) 2019-10-13T08:14:50.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:50.814+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:50.814932+0000) 2019-10-13T08:14:50.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:50.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:51.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:51.990+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.991071+0000) 2019-10-13T08:14:52.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:52.671+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:52.672568+0000) 2019-10-13T08:14:52.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:52.849+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:52.850679+0000) 2019-10-13T08:14:52.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:52.849+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:52.850957+0000) 2019-10-13T08:14:55.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:55.818+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.820051+0000) 2019-10-13T08:14:55.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:55.819+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.820538+0000) 2019-10-13T08:14:55.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:14:55.819+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.820789+0000) 2019-10-13T08:14:55.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:14:55.819+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.821106+0000) 2019-10-13T08:14:55.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:55.819+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.821312+0000) 2019-10-13T08:14:55.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:55.819+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.820906+0000) 2019-10-13T08:14:55.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:14:55.819+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.821234+0000) 2019-10-13T08:14:55.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:14:55.820+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:55.822127+0000) 2019-10-13T08:14:55.826 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:14:55.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:14:57.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:14:57.850+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:57.851449+0000) 2019-10-13T08:14:58.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:14:58.399+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:58.400414+0000) 2019-10-13T08:14:58.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:14:58.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:58.738004+0000) 2019-10-13T08:14:58.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:14:58.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:58.738003+0000) 2019-10-13T08:15:00.558 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:15:00.558 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:00.562 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:15:00.562 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:00.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:00.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.827593+0000) 2019-10-13T08:15:00.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:00.826+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.827818+0000) 2019-10-13T08:15:00.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:00.826+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.828271+0000) 2019-10-13T08:15:00.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:00.826+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.828356+0000) 2019-10-13T08:15:00.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:00.826+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.827768+0000) 2019-10-13T08:15:00.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:00.827+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.828622+0000) 2019-10-13T08:15:00.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:00.827+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.828800+0000) 2019-10-13T08:15:00.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:00.829+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:00.830892+0000) 2019-10-13T08:15:00.833 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:00.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139378 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:01.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:01.989+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.991290+0000) 2019-10-13T08:15:02.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:02.674+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:02.674966+0000) 2019-10-13T08:15:02.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:02.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:02.850846+0000) 2019-10-13T08:15:02.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:02.850+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:02.851132+0000) 2019-10-13T08:15:05.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:05.832+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.834673+0000) 2019-10-13T08:15:05.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:05.833+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.835269+0000) 2019-10-13T08:15:05.836 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:05.833+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.835270+0000) 2019-10-13T08:15:05.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:05.834+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.835667+0000) 2019-10-13T08:15:05.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:05.834+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.836170+0000) 2019-10-13T08:15:05.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:05.834+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.836361+0000) 2019-10-13T08:15:05.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:05.835+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.836500+0000) 2019-10-13T08:15:05.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:05.836+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:05.837794+0000) 2019-10-13T08:15:05.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:05.839+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139409 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:07.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:07.851+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:07.851648+0000) 2019-10-13T08:15:08.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:08.400+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:08.400638+0000) 2019-10-13T08:15:08.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:08.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:08.738211+0000) 2019-10-13T08:15:08.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:08.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:08.738211+0000) 2019-10-13T08:15:10.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:10.841+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.842308+0000) 2019-10-13T08:15:10.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:10.842+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.842474+0000) 2019-10-13T08:15:10.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:10.842+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.842715+0000) 2019-10-13T08:15:10.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:10.843+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.843407+0000) 2019-10-13T08:15:10.846 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:10.843+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.843571+0000) 2019-10-13T08:15:10.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:10.843+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.843673+0000) 2019-10-13T08:15:10.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:10.841+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.842666+0000) 2019-10-13T08:15:10.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:10.844+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:10.845089+0000) 2019-10-13T08:15:10.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:10.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139440 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:11.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:11.991+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.991490+0000) 2019-10-13T08:15:12.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:12.676+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:12.677249+0000) 2019-10-13T08:15:12.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:12.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:12.850998+0000) 2019-10-13T08:15:12.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:12.850+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:12.851286+0000) 2019-10-13T08:15:15.851 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:15.849+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.849982+0000) 2019-10-13T08:15:15.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:15.850+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.850405+0000) 2019-10-13T08:15:15.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:15.850+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.850535+0000) 2019-10-13T08:15:15.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:15.850+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.850889+0000) 2019-10-13T08:15:15.853 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:15.850+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.851069+0000) 2019-10-13T08:15:15.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:15.850+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.851318+0000) 2019-10-13T08:15:15.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:15.851+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.851626+0000) 2019-10-13T08:15:15.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:15.851+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:15.852542+0000) 2019-10-13T08:15:15.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:15.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139471 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:17.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:17.851+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:17.851825+0000) 2019-10-13T08:15:18.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:18.400+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:18.400829+0000) 2019-10-13T08:15:18.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:18.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:18.738446+0000) 2019-10-13T08:15:18.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:18.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:18.738446+0000) 2019-10-13T08:15:20.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:20.857+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.857795+0000) 2019-10-13T08:15:20.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:20.856+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.857690+0000) 2019-10-13T08:15:20.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:20.857+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.858226+0000) 2019-10-13T08:15:20.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:20.858+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.858443+0000) 2019-10-13T08:15:20.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:20.858+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.858734+0000) 2019-10-13T08:15:20.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:20.858+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.858999+0000) 2019-10-13T08:15:20.860 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:20.858+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.859205+0000) 2019-10-13T08:15:20.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:20.860+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:20.861343+0000) 2019-10-13T08:15:20.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:20.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139504 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:21.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:21.991+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.991649+0000) 2019-10-13T08:15:22.680 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:22.679+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:22.679487+0000) 2019-10-13T08:15:22.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:22.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:22.851170+0000) 2019-10-13T08:15:22.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:22.850+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:22.851417+0000) 2019-10-13T08:15:25.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:25.865+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.865715+0000) 2019-10-13T08:15:25.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:25.865+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.865802+0000) 2019-10-13T08:15:25.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:25.865+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.866235+0000) 2019-10-13T08:15:25.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:25.865+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.866293+0000) 2019-10-13T08:15:25.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:25.866+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.866425+0000) 2019-10-13T08:15:25.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:25.866+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.866759+0000) 2019-10-13T08:15:25.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:25.866+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.866859+0000) 2019-10-13T08:15:25.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:25.867+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:25.868423+0000) 2019-10-13T08:15:25.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:25.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139536 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:27.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:27.851+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:27.851994+0000) 2019-10-13T08:15:28.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:28.400+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:28.401047+0000) 2019-10-13T08:15:28.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:28.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:28.738630+0000) 2019-10-13T08:15:28.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:28.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:28.738630+0000) 2019-10-13T08:15:30.729 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:15:30.729 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:30.734 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:15:30.734 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:30.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:30.871+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.873319+0000) 2019-10-13T08:15:30.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:30.872+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.873472+0000) 2019-10-13T08:15:30.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:30.872+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.873771+0000) 2019-10-13T08:15:30.875 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:30.872+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.873821+0000) 2019-10-13T08:15:30.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:30.872+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.874172+0000) 2019-10-13T08:15:30.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:30.872+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.874069+0000) 2019-10-13T08:15:30.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:30.872+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.873384+0000) 2019-10-13T08:15:30.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:30.874+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:30.875064+0000) 2019-10-13T08:15:30.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:30.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:31.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:31.990+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.991818+0000) 2019-10-13T08:15:32.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:32.681+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:32.681713+0000) 2019-10-13T08:15:32.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:32.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:32.851325+0000) 2019-10-13T08:15:32.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:32.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:32.851526+0000) 2019-10-13T08:15:35.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:35.878+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.880117+0000) 2019-10-13T08:15:35.881 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:35.878+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.880340+0000) 2019-10-13T08:15:35.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:35.879+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.880665+0000) 2019-10-13T08:15:35.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:35.879+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.880473+0000) 2019-10-13T08:15:35.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:35.879+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.880884+0000) 2019-10-13T08:15:35.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:35.879+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.881217+0000) 2019-10-13T08:15:35.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:35.880+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.881164+0000) 2019-10-13T08:15:35.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:35.881+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:35.881906+0000) 2019-10-13T08:15:35.886 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:35.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:37.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:37.851+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:37.852165+0000) 2019-10-13T08:15:38.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:38.400+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:38.401260+0000) 2019-10-13T08:15:38.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:38.737+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:38.738805+0000) 2019-10-13T08:15:38.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:38.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:38.738805+0000) 2019-10-13T08:15:40.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:40.886+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.887601+0000) 2019-10-13T08:15:40.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:40.886+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.888005+0000) 2019-10-13T08:15:40.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:40.886+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.887635+0000) 2019-10-13T08:15:40.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:40.886+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.888157+0000) 2019-10-13T08:15:40.890 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:40.886+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.888355+0000) 2019-10-13T08:15:40.890 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:40.887+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.888639+0000) 2019-10-13T08:15:40.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:40.887+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.888727+0000) 2019-10-13T08:15:40.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:40.888+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:40.889262+0000) 2019-10-13T08:15:40.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:40.892+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139630 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:41.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:41.991+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.991972+0000) 2019-10-13T08:15:42.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:42.684+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:42.684084+0000) 2019-10-13T08:15:42.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:42.850+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:42.851502+0000) 2019-10-13T08:15:42.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:42.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:42.851697+0000) 2019-10-13T08:15:45.896 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:45.894+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.895729+0000) 2019-10-13T08:15:45.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:45.893+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.895690+0000) 2019-10-13T08:15:45.897 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:45.894+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.896049+0000) 2019-10-13T08:15:45.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:45.894+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.896192+0000) 2019-10-13T08:15:45.900 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:45.897+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.898931+0000) 2019-10-13T08:15:45.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:45.894+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.896379+0000) 2019-10-13T08:15:45.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:45.895+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.896532+0000) 2019-10-13T08:15:45.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:45.895+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:45.896720+0000) 2019-10-13T08:15:45.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:45.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139662 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:47.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:47.851+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:47.852384+0000) 2019-10-13T08:15:48.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:48.400+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:48.401448+0000) 2019-10-13T08:15:48.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:48.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:48.738959+0000) 2019-10-13T08:15:48.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:48.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:48.738959+0000) 2019-10-13T08:15:50.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:50.900+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.901746+0000) 2019-10-13T08:15:50.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:50.900+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.901962+0000) 2019-10-13T08:15:50.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:50.900+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.902182+0000) 2019-10-13T08:15:50.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:50.900+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.902347+0000) 2019-10-13T08:15:50.904 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:50.901+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.902551+0000) 2019-10-13T08:15:50.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:50.901+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.902804+0000) 2019-10-13T08:15:50.904 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:50.901+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.902957+0000) 2019-10-13T08:15:50.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:50.901+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:50.903613+0000) 2019-10-13T08:15:50.908 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:50.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139692 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:51.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:51.991+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.992179+0000) 2019-10-13T08:15:52.687 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:52.685+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:52.686471+0000) 2019-10-13T08:15:52.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:52.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:52.851677+0000) 2019-10-13T08:15:52.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:52.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:52.851910+0000) 2019-10-13T08:15:55.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:15:55.907+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.909053+0000) 2019-10-13T08:15:55.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:55.908+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.909545+0000) 2019-10-13T08:15:55.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:15:55.908+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.909897+0000) 2019-10-13T08:15:55.911 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:55.908+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.910018+0000) 2019-10-13T08:15:55.911 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:55.909+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.910422+0000) 2019-10-13T08:15:55.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:15:55.909+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.910577+0000) 2019-10-13T08:15:55.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:55.909+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.911000+0000) 2019-10-13T08:15:55.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:15:55.910+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:55.912535+0000) 2019-10-13T08:15:55.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:15:55.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139724 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:15:57.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:15:57.852+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:57.852610+0000) 2019-10-13T08:15:58.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:15:58.401+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:58.401616+0000) 2019-10-13T08:15:58.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:15:58.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:58.739125+0000) 2019-10-13T08:15:58.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:15:58.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:58.739126+0000) 2019-10-13T08:16:00.878 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:16:00.878 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:00.883 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:16:00.883 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:00.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:00.917+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.917612+0000) 2019-10-13T08:16:00.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:00.917+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.917816+0000) 2019-10-13T08:16:00.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:00.918+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.918413+0000) 2019-10-13T08:16:00.920 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:00.918+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.918651+0000) 2019-10-13T08:16:00.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:00.918+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.918880+0000) 2019-10-13T08:16:00.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:00.918+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.919025+0000) 2019-10-13T08:16:00.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:00.918+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.919247+0000) 2019-10-13T08:16:00.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:00.919+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:00.919918+0000) 2019-10-13T08:16:00.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:00.922+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:01.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:01.991+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.992322+0000) 2019-10-13T08:16:02.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:02.688+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:02.688784+0000) 2019-10-13T08:16:02.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:02.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:02.851884+0000) 2019-10-13T08:16:02.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:02.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:02.852007+0000) 2019-10-13T08:16:05.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:05.924+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.924790+0000) 2019-10-13T08:16:05.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:05.924+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.924934+0000) 2019-10-13T08:16:05.926 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:05.924+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.925076+0000) 2019-10-13T08:16:05.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:05.924+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.925245+0000) 2019-10-13T08:16:05.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:05.925+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.925478+0000) 2019-10-13T08:16:05.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:05.924+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.925706+0000) 2019-10-13T08:16:05.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:05.925+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.926010+0000) 2019-10-13T08:16:05.928 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:05.926+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:05.927383+0000) 2019-10-13T08:16:05.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:05.929+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:07.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:07.852+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:07.852822+0000) 2019-10-13T08:16:08.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:08.401+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:08.401772+0000) 2019-10-13T08:16:08.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:08.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:08.739306+0000) 2019-10-13T08:16:08.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:08.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:08.739306+0000) 2019-10-13T08:16:10.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:10.930+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.931724+0000) 2019-10-13T08:16:10.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:10.931+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.932118+0000) 2019-10-13T08:16:10.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:10.932+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.932512+0000) 2019-10-13T08:16:10.934 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:10.932+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.932700+0000) 2019-10-13T08:16:10.934 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:10.932+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.933017+0000) 2019-10-13T08:16:10.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:10.932+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.933140+0000) 2019-10-13T08:16:10.934 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:10.932+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.933225+0000) 2019-10-13T08:16:10.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:10.934+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:10.935221+0000) 2019-10-13T08:16:10.938 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:10.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:11.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:11.992+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.992423+0000) 2019-10-13T08:16:12.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:12.690+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:12.691251+0000) 2019-10-13T08:16:12.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:12.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:12.852099+0000) 2019-10-13T08:16:12.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:12.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:12.852152+0000) 2019-10-13T08:16:15.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:15.939+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.940076+0000) 2019-10-13T08:16:15.942 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:15.939+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.940595+0000) 2019-10-13T08:16:15.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:15.939+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.940658+0000) 2019-10-13T08:16:15.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:15.939+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.941015+0000) 2019-10-13T08:16:15.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:15.939+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.941068+0000) 2019-10-13T08:16:15.942 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:15.940+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.941500+0000) 2019-10-13T08:16:15.943 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:15.940+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.941648+0000) 2019-10-13T08:16:15.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:15.942+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:15.942989+0000) 2019-10-13T08:16:15.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:15.944+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139850 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:17.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:17.852+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:17.853030+0000) 2019-10-13T08:16:18.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:18.400+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:18.401950+0000) 2019-10-13T08:16:18.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:18.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:18.739518+0000) 2019-10-13T08:16:18.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:18.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:18.739518+0000) 2019-10-13T08:16:20.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:20.946+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.947684+0000) 2019-10-13T08:16:20.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:20.947+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.947806+0000) 2019-10-13T08:16:20.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:20.946+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.948261+0000) 2019-10-13T08:16:20.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:20.947+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.948683+0000) 2019-10-13T08:16:20.950 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:20.947+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.948761+0000) 2019-10-13T08:16:20.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:20.947+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.949116+0000) 2019-10-13T08:16:20.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:20.947+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.949152+0000) 2019-10-13T08:16:20.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:20.949+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:20.950342+0000) 2019-10-13T08:16:20.955 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:20.952+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:21.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:21.991+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.992603+0000) 2019-10-13T08:16:22.694 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:22.693+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:22.693432+0000) 2019-10-13T08:16:22.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:22.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:22.852225+0000) 2019-10-13T08:16:22.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:22.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:22.852293+0000) 2019-10-13T08:16:25.956 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:25.954+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.955764+0000) 2019-10-13T08:16:25.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:25.954+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.955704+0000) 2019-10-13T08:16:25.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:25.954+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.956250+0000) 2019-10-13T08:16:25.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:25.955+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.956653+0000) 2019-10-13T08:16:25.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:25.955+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.956785+0000) 2019-10-13T08:16:25.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:25.955+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.956875+0000) 2019-10-13T08:16:25.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:25.955+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.957052+0000) 2019-10-13T08:16:25.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:25.957+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:25.958288+0000) 2019-10-13T08:16:25.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:25.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:27.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:27.852+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:27.853229+0000) 2019-10-13T08:16:28.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:28.401+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:28.402064+0000) 2019-10-13T08:16:28.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:28.738+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:28.739718+0000) 2019-10-13T08:16:28.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:28.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:28.739718+0000) 2019-10-13T08:16:30.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:30.961+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.963142+0000) 2019-10-13T08:16:30.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:30.962+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.963438+0000) 2019-10-13T08:16:30.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:30.962+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.963552+0000) 2019-10-13T08:16:30.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:30.962+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.963731+0000) 2019-10-13T08:16:30.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:30.963+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.963868+0000) 2019-10-13T08:16:30.966 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:30.962+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.963949+0000) 2019-10-13T08:16:30.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:30.962+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.964197+0000) 2019-10-13T08:16:30.966 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:30.965+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:30.966165+0000) 2019-10-13T08:16:30.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:30.967+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:31.073 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:16:31.073 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:31.078 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:16:31.078 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:31.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:31.992+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.992798+0000) 2019-10-13T08:16:32.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:32.696+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:32.696432+0000) 2019-10-13T08:16:32.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:32.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:32.852431+0000) 2019-10-13T08:16:32.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:32.852+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:32.852493+0000) 2019-10-13T08:16:35.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:35.968+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.969824+0000) 2019-10-13T08:16:35.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:35.968+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.970104+0000) 2019-10-13T08:16:35.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:35.968+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.970204+0000) 2019-10-13T08:16:35.971 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:35.968+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.970358+0000) 2019-10-13T08:16:35.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:35.969+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.970453+0000) 2019-10-13T08:16:35.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:35.969+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.970738+0000) 2019-10-13T08:16:35.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:35.969+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.970824+0000) 2019-10-13T08:16:35.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:35.972+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:35.972873+0000) 2019-10-13T08:16:35.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:35.973+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 139977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:37.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:37.852+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:37.853428+0000) 2019-10-13T08:16:38.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:38.401+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:38.402235+0000) 2019-10-13T08:16:38.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:38.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:38.739923+0000) 2019-10-13T08:16:38.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:38.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:38.739923+0000) 2019-10-13T08:16:40.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:40.975+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.977145+0000) 2019-10-13T08:16:40.978 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:40.975+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.977370+0000) 2019-10-13T08:16:40.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:40.976+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.977602+0000) 2019-10-13T08:16:40.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:40.976+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.977734+0000) 2019-10-13T08:16:40.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:40.975+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.977634+0000) 2019-10-13T08:16:40.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:40.976+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.978167+0000) 2019-10-13T08:16:40.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:40.977+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.978692+0000) 2019-10-13T08:16:40.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:40.978+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:40.980214+0000) 2019-10-13T08:16:40.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:40.980+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140008 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:41.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:41.992+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.992988+0000) 2019-10-13T08:16:42.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:42.697+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:42.698882+0000) 2019-10-13T08:16:42.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:42.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:42.852603+0000) 2019-10-13T08:16:42.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:42.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:42.852621+0000) 2019-10-13T08:16:45.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:45.983+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.983933+0000) 2019-10-13T08:16:45.985 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:45.983+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.984144+0000) 2019-10-13T08:16:45.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:45.983+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.984256+0000) 2019-10-13T08:16:45.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:45.982+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.984608+0000) 2019-10-13T08:16:45.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:45.984+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.984565+0000) 2019-10-13T08:16:45.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:45.984+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.984978+0000) 2019-10-13T08:16:45.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:45.984+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.985183+0000) 2019-10-13T08:16:45.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:45.985+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:45.987412+0000) 2019-10-13T08:16:45.990 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:45.989+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:47.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:47.853+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:47.853616+0000) 2019-10-13T08:16:48.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:48.402+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:48.402435+0000) 2019-10-13T08:16:48.740 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:48.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:48.740091+0000) 2019-10-13T08:16:48.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:48.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:48.740091+0000) 2019-10-13T08:16:50.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:50.991+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.992092+0000) 2019-10-13T08:16:50.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:50.992+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.992415+0000) 2019-10-13T08:16:50.993 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:50.992+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.992543+0000) 2019-10-13T08:16:50.993 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:50.992+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.992684+0000) 2019-10-13T08:16:50.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:50.992+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.993011+0000) 2019-10-13T08:16:50.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:50.992+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.992840+0000) 2019-10-13T08:16:50.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:50.993+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.993891+0000) 2019-10-13T08:16:50.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:50.993+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:50.995447+0000) 2019-10-13T08:16:50.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:50.996+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:51.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:51.992+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.993157+0000) 2019-10-13T08:16:52.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:52.701+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:52.701963+0000) 2019-10-13T08:16:52.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:52.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:52.852750+0000) 2019-10-13T08:16:52.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:52.851+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:52.852704+0000) 2019-10-13T08:16:55.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:16:55.997+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:55.998546+0000) 2019-10-13T08:16:56.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:55.998+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:55.999104+0000) 2019-10-13T08:16:56.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:55.998+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:55.999227+0000) 2019-10-13T08:16:56.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:55.999+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:55.999414+0000) 2019-10-13T08:16:56.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:16:55.999+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:55.999609+0000) 2019-10-13T08:16:56.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:16:55.999+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:55.999997+0000) 2019-10-13T08:16:56.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:16:56.000+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.000837+0000) 2019-10-13T08:16:56.001 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:55.999+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.000101+0000) 2019-10-13T08:16:56.005 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:16:56.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:16:57.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:16:57.853+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:57.853792+0000) 2019-10-13T08:16:58.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:16:58.402+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:58.402590+0000) 2019-10-13T08:16:58.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:16:58.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:58.740249+0000) 2019-10-13T08:16:58.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:16:58.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:58.740249+0000) 2019-10-13T08:17:01.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:01.005+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.006485+0000) 2019-10-13T08:17:01.008 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:01.006+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.006937+0000) 2019-10-13T08:17:01.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:01.006+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.007365+0000) 2019-10-13T08:17:01.009 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:01.007+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.007620+0000) 2019-10-13T08:17:01.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:01.007+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.007640+0000) 2019-10-13T08:17:01.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:01.007+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.007772+0000) 2019-10-13T08:17:01.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:01.007+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.008116+0000) 2019-10-13T08:17:01.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:01.009+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.010489+0000) 2019-10-13T08:17:01.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:01.012+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140134 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:01.218 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:17:01.218 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:01.222 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:17:01.222 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:01.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:01.992+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.993332+0000) 2019-10-13T08:17:02.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:02.704+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:02.705022+0000) 2019-10-13T08:17:02.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:02.851+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:02.852848+0000) 2019-10-13T08:17:02.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:02.852+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:02.852871+0000) 2019-10-13T08:17:06.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:06.012+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.013924+0000) 2019-10-13T08:17:06.015 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:06.012+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.014060+0000) 2019-10-13T08:17:06.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:06.013+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.014187+0000) 2019-10-13T08:17:06.016 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:06.013+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.014540+0000) 2019-10-13T08:17:06.016 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:06.013+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.014807+0000) 2019-10-13T08:17:06.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:06.013+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.015012+0000) 2019-10-13T08:17:06.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:06.013+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.015190+0000) 2019-10-13T08:17:06.017 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:06.015+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.016202+0000) 2019-10-13T08:17:06.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:06.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:07.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:07.853+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:07.853990+0000) 2019-10-13T08:17:08.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:08.401+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:08.402790+0000) 2019-10-13T08:17:08.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:08.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:08.740455+0000) 2019-10-13T08:17:08.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:08.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:08.740455+0000) 2019-10-13T08:17:11.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:11.018+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.020312+0000) 2019-10-13T08:17:11.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:11.019+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.020428+0000) 2019-10-13T08:17:11.021 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:11.019+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.020658+0000) 2019-10-13T08:17:11.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:11.019+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.020860+0000) 2019-10-13T08:17:11.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:11.020+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.020927+0000) 2019-10-13T08:17:11.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:11.019+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.021236+0000) 2019-10-13T08:17:11.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:11.020+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.021702+0000) 2019-10-13T08:17:11.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:11.021+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.022145+0000) 2019-10-13T08:17:11.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:11.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:11.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:11.992+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.993503+0000) 2019-10-13T08:17:12.708 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:12.706+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:12.707289+0000) 2019-10-13T08:17:12.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:12.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:12.853041+0000) 2019-10-13T08:17:12.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:12.852+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:12.853061+0000) 2019-10-13T08:17:16.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:16.025+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.026590+0000) 2019-10-13T08:17:16.028 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:16.025+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.027135+0000) 2019-10-13T08:17:16.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:16.026+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.027544+0000) 2019-10-13T08:17:16.028 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:16.026+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.027743+0000) 2019-10-13T08:17:16.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:16.026+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.028104+0000) 2019-10-13T08:17:16.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:16.026+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.028278+0000) 2019-10-13T08:17:16.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:16.027+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.028503+0000) 2019-10-13T08:17:16.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:16.028+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.028873+0000) 2019-10-13T08:17:16.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:16.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140229 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:17.854 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:17.853+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:17.854165+0000) 2019-10-13T08:17:18.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:18.402+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:18.403014+0000) 2019-10-13T08:17:18.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:18.739+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:18.740648+0000) 2019-10-13T08:17:18.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:18.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:18.740648+0000) 2019-10-13T08:17:21.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:21.032+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.034250+0000) 2019-10-13T08:17:21.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:21.033+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.034498+0000) 2019-10-13T08:17:21.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:21.033+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.034542+0000) 2019-10-13T08:17:21.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:21.033+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.034854+0000) 2019-10-13T08:17:21.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:21.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.034813+0000) 2019-10-13T08:17:21.037 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:21.034+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.035410+0000) 2019-10-13T08:17:21.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:21.034+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.035552+0000) 2019-10-13T08:17:21.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:21.036+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.036921+0000) 2019-10-13T08:17:21.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:21.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140260 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:21.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:21.993+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.993673+0000) 2019-10-13T08:17:22.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:22.710+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:22.710267+0000) 2019-10-13T08:17:22.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:22.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:22.853204+0000) 2019-10-13T08:17:22.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:22.852+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:22.853181+0000) 2019-10-13T08:17:26.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:26.039+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.040816+0000) 2019-10-13T08:17:26.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:26.039+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.041116+0000) 2019-10-13T08:17:26.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:26.040+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.041361+0000) 2019-10-13T08:17:26.042 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:26.039+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.041264+0000) 2019-10-13T08:17:26.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:26.039+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.041374+0000) 2019-10-13T08:17:26.043 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:26.040+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.041735+0000) 2019-10-13T08:17:26.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:26.040+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.042199+0000) 2019-10-13T08:17:26.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:26.042+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.043439+0000) 2019-10-13T08:17:26.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:26.045+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140293 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:27.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:27.853+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:27.854373+0000) 2019-10-13T08:17:28.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:28.402+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:28.403205+0000) 2019-10-13T08:17:28.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:28.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:28.740805+0000) 2019-10-13T08:17:28.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:28.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:28.740805+0000) 2019-10-13T08:17:31.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:31.048+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.048673+0000) 2019-10-13T08:17:31.050 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:31.048+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.048797+0000) 2019-10-13T08:17:31.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:31.048+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.049143+0000) 2019-10-13T08:17:31.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:31.047+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.048736+0000) 2019-10-13T08:17:31.051 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:31.049+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.049499+0000) 2019-10-13T08:17:31.051 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:31.049+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.049698+0000) 2019-10-13T08:17:31.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:31.049+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.049980+0000) 2019-10-13T08:17:31.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:31.049+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.051678+0000) 2019-10-13T08:17:31.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:31.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140322 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:31.393 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:17:31.393 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:31.397 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:17:31.397 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:31.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:31.993+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.993824+0000) 2019-10-13T08:17:32.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:32.712+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:32.713324+0000) 2019-10-13T08:17:32.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:32.852+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:32.853309+0000) 2019-10-13T08:17:32.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:32.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:32.853386+0000) 2019-10-13T08:17:36.057 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:36.055+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.056047+0000) 2019-10-13T08:17:36.057 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:36.055+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.056201+0000) 2019-10-13T08:17:36.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:36.054+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.056142+0000) 2019-10-13T08:17:36.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:36.056+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.056971+0000) 2019-10-13T08:17:36.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:36.056+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.057127+0000) 2019-10-13T08:17:36.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:36.056+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.057284+0000) 2019-10-13T08:17:36.059 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:36.057+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.057423+0000) 2019-10-13T08:17:36.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:36.056+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.058368+0000) 2019-10-13T08:17:36.065 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:36.061+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:37.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:37.854+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:37.854556+0000) 2019-10-13T08:17:38.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:38.403+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:38.403399+0000) 2019-10-13T08:17:38.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:38.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:38.740990+0000) 2019-10-13T08:17:38.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:38.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:38.740990+0000) 2019-10-13T08:17:41.065 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:41.063+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.064022+0000) 2019-10-13T08:17:41.065 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:41.063+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.064283+0000) 2019-10-13T08:17:41.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:41.064+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.064465+0000) 2019-10-13T08:17:41.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:41.063+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.065200+0000) 2019-10-13T08:17:41.066 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:41.064+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.064832+0000) 2019-10-13T08:17:41.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:41.064+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.065028+0000) 2019-10-13T08:17:41.067 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:41.065+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.065510+0000) 2019-10-13T08:17:41.067 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:41.065+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.067127+0000) 2019-10-13T08:17:41.070 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:41.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140386 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:41.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:41.993+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.993969+0000) 2019-10-13T08:17:42.717 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:42.715+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:42.716429+0000) 2019-10-13T08:17:42.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:42.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:42.853562+0000) 2019-10-13T08:17:42.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:42.852+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:42.853496+0000) 2019-10-13T08:17:46.071 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:46.070+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.071351+0000) 2019-10-13T08:17:46.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:46.071+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.071845+0000) 2019-10-13T08:17:46.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:46.071+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.072067+0000) 2019-10-13T08:17:46.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:46.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.072370+0000) 2019-10-13T08:17:46.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:46.071+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.072437+0000) 2019-10-13T08:17:46.074 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:46.072+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.072767+0000) 2019-10-13T08:17:46.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:46.072+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.072969+0000) 2019-10-13T08:17:46.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:46.073+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.074179+0000) 2019-10-13T08:17:46.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:46.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:47.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:47.853+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:47.854754+0000) 2019-10-13T08:17:48.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:48.402+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:48.403542+0000) 2019-10-13T08:17:48.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:48.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:48.741201+0000) 2019-10-13T08:17:48.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:48.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:48.741202+0000) 2019-10-13T08:17:51.079 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:51.077+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.078970+0000) 2019-10-13T08:17:51.080 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:51.077+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.079209+0000) 2019-10-13T08:17:51.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:51.077+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.079364+0000) 2019-10-13T08:17:51.080 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:51.078+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.079534+0000) 2019-10-13T08:17:51.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:51.079+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.080073+0000) 2019-10-13T08:17:51.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:51.078+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.080081+0000) 2019-10-13T08:17:51.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:51.079+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.080443+0000) 2019-10-13T08:17:51.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:51.081+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.082446+0000) 2019-10-13T08:17:51.085 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:51.083+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:51.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:51.993+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.994135+0000) 2019-10-13T08:17:52.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:52.718+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:52.718887+0000) 2019-10-13T08:17:52.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:52.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:52.853669+0000) 2019-10-13T08:17:52.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:52.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:52.853753+0000) 2019-10-13T08:17:56.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:56.085+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.087038+0000) 2019-10-13T08:17:56.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:56.085+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.087223+0000) 2019-10-13T08:17:56.088 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:56.086+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.087467+0000) 2019-10-13T08:17:56.089 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:56.086+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.087758+0000) 2019-10-13T08:17:56.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:17:56.087+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.087877+0000) 2019-10-13T08:17:56.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:17:56.086+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.088094+0000) 2019-10-13T08:17:56.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:17:56.086+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.088345+0000) 2019-10-13T08:17:56.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:17:56.089+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.090468+0000) 2019-10-13T08:17:56.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:17:56.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:17:57.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:17:57.854+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:57.854951+0000) 2019-10-13T08:17:58.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:17:58.402+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:58.403752+0000) 2019-10-13T08:17:58.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:17:58.740+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:58.741390+0000) 2019-10-13T08:17:58.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:17:58.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:58.741390+0000) 2019-10-13T08:18:01.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:01.094+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.094949+0000) 2019-10-13T08:18:01.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:01.093+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.095313+0000) 2019-10-13T08:18:01.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:01.094+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.095668+0000) 2019-10-13T08:18:01.097 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:01.094+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.095913+0000) 2019-10-13T08:18:01.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:01.094+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.096047+0000) 2019-10-13T08:18:01.097 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:01.094+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.096239+0000) 2019-10-13T08:18:01.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:01.094+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.096340+0000) 2019-10-13T08:18:01.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:01.096+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.097434+0000) 2019-10-13T08:18:01.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:01.099+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140512 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:01.615 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:18:01.615 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:01.619 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:18:01.619 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:01.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:01.993+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.994297+0000) 2019-10-13T08:18:02.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:02.720+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:02.721147+0000) 2019-10-13T08:18:02.854 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:02.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:02.853942+0000) 2019-10-13T08:18:02.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:02.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:02.853831+0000) 2019-10-13T08:18:06.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:06.101+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.102252+0000) 2019-10-13T08:18:06.104 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:06.101+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.102945+0000) 2019-10-13T08:18:06.104 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:06.101+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.103238+0000) 2019-10-13T08:18:06.104 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:06.101+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.103063+0000) 2019-10-13T08:18:06.104 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:06.102+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.103411+0000) 2019-10-13T08:18:06.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:06.102+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.103599+0000) 2019-10-13T08:18:06.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:06.103+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.104696+0000) 2019-10-13T08:18:06.105 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:06.102+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.104012+0000) 2019-10-13T08:18:06.108 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:06.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:07.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:07.854+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:07.855145+0000) 2019-10-13T08:18:08.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:08.403+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:08.403929+0000) 2019-10-13T08:18:08.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:08.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:08.741582+0000) 2019-10-13T08:18:08.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:08.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:08.741582+0000) 2019-10-13T08:18:11.110 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:11.108+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.109934+0000) 2019-10-13T08:18:11.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:11.108+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.110257+0000) 2019-10-13T08:18:11.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:11.109+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.110614+0000) 2019-10-13T08:18:11.112 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:11.109+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.110581+0000) 2019-10-13T08:18:11.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:11.109+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.110443+0000) 2019-10-13T08:18:11.112 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:11.109+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.110907+0000) 2019-10-13T08:18:11.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:11.109+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.111259+0000) 2019-10-13T08:18:11.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:11.112+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.112884+0000) 2019-10-13T08:18:11.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:11.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:11.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:11.993+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.994466+0000) 2019-10-13T08:18:12.725 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:12.722+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:12.723678+0000) 2019-10-13T08:18:12.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:12.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:12.853925+0000) 2019-10-13T08:18:12.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:12.852+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:12.854117+0000) 2019-10-13T08:18:16.119 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:16.117+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.118158+0000) 2019-10-13T08:18:16.120 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:16.117+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.118216+0000) 2019-10-13T08:18:16.120 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:16.118+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.118450+0000) 2019-10-13T08:18:16.120 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:16.118+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.118833+0000) 2019-10-13T08:18:16.120 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:16.118+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.119046+0000) 2019-10-13T08:18:16.121 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:16.118+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.119124+0000) 2019-10-13T08:18:16.121 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:16.117+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.118399+0000) 2019-10-13T08:18:16.122 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:16.120+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.121444+0000) 2019-10-13T08:18:16.124 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:16.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:17.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:17.854+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:17.855359+0000) 2019-10-13T08:18:18.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:18.403+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:18.404164+0000) 2019-10-13T08:18:18.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:18.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:18.741755+0000) 2019-10-13T08:18:18.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:18.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:18.741755+0000) 2019-10-13T08:18:21.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:21.124+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.125464+0000) 2019-10-13T08:18:21.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:21.127+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.127888+0000) 2019-10-13T08:18:21.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:21.124+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.125301+0000) 2019-10-13T08:18:21.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:21.125+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.125484+0000) 2019-10-13T08:18:21.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:21.125+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.126248+0000) 2019-10-13T08:18:21.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:21.126+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.126423+0000) 2019-10-13T08:18:21.131 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:21.126+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.126632+0000) 2019-10-13T08:18:21.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:21.126+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.126687+0000) 2019-10-13T08:18:21.132 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:21.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140638 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:21.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:21.994+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.994632+0000) 2019-10-13T08:18:22.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:22.725+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:22.726774+0000) 2019-10-13T08:18:22.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:22.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:22.854250+0000) 2019-10-13T08:18:22.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:22.852+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:22.854034+0000) 2019-10-13T08:18:26.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:26.131+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.133060+0000) 2019-10-13T08:18:26.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:26.133+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.133514+0000) 2019-10-13T08:18:26.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:26.133+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.133784+0000) 2019-10-13T08:18:26.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:26.133+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.134013+0000) 2019-10-13T08:18:26.135 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:26.133+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.134133+0000) 2019-10-13T08:18:26.135 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:26.133+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.134308+0000) 2019-10-13T08:18:26.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:26.134+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.134873+0000) 2019-10-13T08:18:26.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:26.135+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.137142+0000) 2019-10-13T08:18:26.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:26.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:27.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:27.855+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:27.855579+0000) 2019-10-13T08:18:28.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:28.404+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:28.404323+0000) 2019-10-13T08:18:28.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:28.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:28.741926+0000) 2019-10-13T08:18:28.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:28.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:28.741926+0000) 2019-10-13T08:18:31.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:31.139+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.140990+0000) 2019-10-13T08:18:31.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:31.140+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.141257+0000) 2019-10-13T08:18:31.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:31.140+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.141371+0000) 2019-10-13T08:18:31.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:31.141+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.141866+0000) 2019-10-13T08:18:31.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:31.141+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.141987+0000) 2019-10-13T08:18:31.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:31.141+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.142132+0000) 2019-10-13T08:18:31.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:31.141+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.142333+0000) 2019-10-13T08:18:31.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:31.141+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.142932+0000) 2019-10-13T08:18:31.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:31.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:31.788 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:18:31.789 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:31.793 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:18:31.794 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:31.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:31.994+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.994801+0000) 2019-10-13T08:18:32.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:32.728+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:32.729802+0000) 2019-10-13T08:18:32.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:32.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:32.854190+0000) 2019-10-13T08:18:32.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:32.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:32.854411+0000) 2019-10-13T08:18:36.149 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:36.147+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.148729+0000) 2019-10-13T08:18:36.150 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:36.147+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.148739+0000) 2019-10-13T08:18:36.150 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:36.147+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.149173+0000) 2019-10-13T08:18:36.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:36.148+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.149673+0000) 2019-10-13T08:18:36.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:36.148+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.150076+0000) 2019-10-13T08:18:36.151 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:36.148+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.150223+0000) 2019-10-13T08:18:36.151 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:36.149+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.150603+0000) 2019-10-13T08:18:36.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:36.149+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.151277+0000) 2019-10-13T08:18:36.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:36.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:37.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:37.854+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:37.855760+0000) 2019-10-13T08:18:38.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:38.403+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:38.404491+0000) 2019-10-13T08:18:38.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:38.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:38.742105+0000) 2019-10-13T08:18:38.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:38.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:38.742105+0000) 2019-10-13T08:18:41.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:41.155+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.157047+0000) 2019-10-13T08:18:41.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:41.155+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.157262+0000) 2019-10-13T08:18:41.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:41.155+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.157161+0000) 2019-10-13T08:18:41.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:41.156+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.157455+0000) 2019-10-13T08:18:41.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:41.156+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.157720+0000) 2019-10-13T08:18:41.159 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:41.156+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.157928+0000) 2019-10-13T08:18:41.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:41.157+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.158046+0000) 2019-10-13T08:18:41.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:41.159+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.160352+0000) 2019-10-13T08:18:41.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:41.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:41.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:41.994+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.994991+0000) 2019-10-13T08:18:42.733 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:42.731+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:42.732314+0000) 2019-10-13T08:18:42.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:42.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:42.854315+0000) 2019-10-13T08:18:42.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:42.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:42.854602+0000) 2019-10-13T08:18:46.163 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:46.161+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.162533+0000) 2019-10-13T08:18:46.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:46.161+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.162665+0000) 2019-10-13T08:18:46.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:46.161+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.163184+0000) 2019-10-13T08:18:46.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:46.161+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.163293+0000) 2019-10-13T08:18:46.165 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:46.162+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.163559+0000) 2019-10-13T08:18:46.165 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:46.162+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.163726+0000) 2019-10-13T08:18:46.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:46.162+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.163921+0000) 2019-10-13T08:18:46.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:46.164+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.165349+0000) 2019-10-13T08:18:46.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:46.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:47.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:47.854+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:47.855922+0000) 2019-10-13T08:18:48.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:48.403+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:48.404682+0000) 2019-10-13T08:18:48.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:48.741+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:48.742321+0000) 2019-10-13T08:18:48.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:48.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:48.742321+0000) 2019-10-13T08:18:51.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:51.169+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.170723+0000) 2019-10-13T08:18:51.172 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:51.169+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.170859+0000) 2019-10-13T08:18:51.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:51.169+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.171210+0000) 2019-10-13T08:18:51.172 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:51.169+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.171360+0000) 2019-10-13T08:18:51.172 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:51.170+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.171572+0000) 2019-10-13T08:18:51.173 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:51.170+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.170908+0000) 2019-10-13T08:18:51.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:51.170+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.172316+0000) 2019-10-13T08:18:51.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:51.173+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.173958+0000) 2019-10-13T08:18:51.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:51.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:51.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:51.994+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.995149+0000) 2019-10-13T08:18:52.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:52.733+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:52.734583+0000) 2019-10-13T08:18:52.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:52.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:52.854762+0000) 2019-10-13T08:18:52.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:52.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:52.854446+0000) 2019-10-13T08:18:56.178 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:18:56.176+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.178240+0000) 2019-10-13T08:18:56.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:56.177+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.178462+0000) 2019-10-13T08:18:56.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:18:56.177+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.178445+0000) 2019-10-13T08:18:56.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:56.177+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.178824+0000) 2019-10-13T08:18:56.180 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:18:56.177+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.179093+0000) 2019-10-13T08:18:56.180 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:56.177+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.179234+0000) 2019-10-13T08:18:56.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:56.178+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.179787+0000) 2019-10-13T08:18:56.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:18:56.180+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.181533+0000) 2019-10-13T08:18:56.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:18:56.182+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:18:57.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:18:57.855+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:57.856127+0000) 2019-10-13T08:18:58.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:18:58.404+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:58.404881+0000) 2019-10-13T08:18:58.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:18:58.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:58.742561+0000) 2019-10-13T08:18:58.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:18:58.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:58.742561+0000) 2019-10-13T08:19:01.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:01.184+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.185834+0000) 2019-10-13T08:19:01.187 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:01.185+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.186031+0000) 2019-10-13T08:19:01.187 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:01.184+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.186323+0000) 2019-10-13T08:19:01.187 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:01.185+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.186496+0000) 2019-10-13T08:19:01.187 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:01.185+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.186798+0000) 2019-10-13T08:19:01.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:01.185+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.187156+0000) 2019-10-13T08:19:01.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:01.185+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.187335+0000) 2019-10-13T08:19:01.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:01.187+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.188064+0000) 2019-10-13T08:19:01.191 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:01.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:01.943 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:19:01.943 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:01.947 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:19:01.948 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:01.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:01.994+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.995307+0000) 2019-10-13T08:19:02.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:02.737+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:02.737814+0000) 2019-10-13T08:19:02.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:02.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:02.854641+0000) 2019-10-13T08:19:02.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:02.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:02.854906+0000) 2019-10-13T08:19:06.193 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:06.192+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.193274+0000) 2019-10-13T08:19:06.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:06.193+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.193504+0000) 2019-10-13T08:19:06.195 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:06.192+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.193264+0000) 2019-10-13T08:19:06.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:06.193+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.193991+0000) 2019-10-13T08:19:06.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:06.193+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.194185+0000) 2019-10-13T08:19:06.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:06.194+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.194462+0000) 2019-10-13T08:19:06.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:06.194+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.195011+0000) 2019-10-13T08:19:06.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:06.195+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.195813+0000) 2019-10-13T08:19:06.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:06.198+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140922 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:07.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:07.856+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:07.856361+0000) 2019-10-13T08:19:08.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:08.404+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:08.405076+0000) 2019-10-13T08:19:08.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:08.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:08.742742+0000) 2019-10-13T08:19:08.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:08.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:08.742742+0000) 2019-10-13T08:19:11.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:11.199+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.199955+0000) 2019-10-13T08:19:11.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:11.199+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.200317+0000) 2019-10-13T08:19:11.201 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:11.200+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.200466+0000) 2019-10-13T08:19:11.202 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:11.199+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.200698+0000) 2019-10-13T08:19:11.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:11.200+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.200772+0000) 2019-10-13T08:19:11.202 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:11.200+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.201079+0000) 2019-10-13T08:19:11.202 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:11.200+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.201338+0000) 2019-10-13T08:19:11.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:11.202+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.202974+0000) 2019-10-13T08:19:11.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:11.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:11.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:11.995+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.995459+0000) 2019-10-13T08:19:12.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:12.740+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:12.740141+0000) 2019-10-13T08:19:12.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:12.854+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:12.854792+0000) 2019-10-13T08:19:12.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:12.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:12.855076+0000) 2019-10-13T08:19:16.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:16.206+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.206760+0000) 2019-10-13T08:19:16.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:16.206+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.207063+0000) 2019-10-13T08:19:16.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:16.206+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.207206+0000) 2019-10-13T08:19:16.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:16.205+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.207703+0000) 2019-10-13T08:19:16.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:16.207+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.207530+0000) 2019-10-13T08:19:16.209 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:16.207+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.207813+0000) 2019-10-13T08:19:16.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:16.207+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.208133+0000) 2019-10-13T08:19:16.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:16.208+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.210035+0000) 2019-10-13T08:19:16.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:16.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 140983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:17.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:17.856+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:17.856568+0000) 2019-10-13T08:19:18.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:18.405+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:18.405272+0000) 2019-10-13T08:19:18.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:18.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:18.742890+0000) 2019-10-13T08:19:18.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:18.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:18.742890+0000) 2019-10-13T08:19:21.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:21.212+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.214596+0000) 2019-10-13T08:19:21.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:21.213+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.215116+0000) 2019-10-13T08:19:21.216 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:21.213+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.215299+0000) 2019-10-13T08:19:21.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:21.214+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.215507+0000) 2019-10-13T08:19:21.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:21.214+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.215534+0000) 2019-10-13T08:19:21.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:21.214+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.215739+0000) 2019-10-13T08:19:21.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:21.214+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.215938+0000) 2019-10-13T08:19:21.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:21.214+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.216478+0000) 2019-10-13T08:19:21.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:21.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:21.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:21.994+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.995627+0000) 2019-10-13T08:19:22.745 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:22.741+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:22.742490+0000) 2019-10-13T08:19:22.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:22.853+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:22.854898+0000) 2019-10-13T08:19:22.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:22.853+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:22.855234+0000) 2019-10-13T08:19:26.222 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:26.220+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.221775+0000) 2019-10-13T08:19:26.223 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:26.220+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.222123+0000) 2019-10-13T08:19:26.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:26.220+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.222208+0000) 2019-10-13T08:19:26.223 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:26.221+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.222393+0000) 2019-10-13T08:19:26.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:26.221+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.222790+0000) 2019-10-13T08:19:26.224 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:26.221+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.222924+0000) 2019-10-13T08:19:26.224 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:26.221+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.223340+0000) 2019-10-13T08:19:26.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:26.223+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.224915+0000) 2019-10-13T08:19:26.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:26.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:27.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:27.855+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:27.856774+0000) 2019-10-13T08:19:28.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:28.404+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:28.405411+0000) 2019-10-13T08:19:28.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:28.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:28.743068+0000) 2019-10-13T08:19:28.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:28.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:28.743068+0000) 2019-10-13T08:19:31.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:31.226+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.228241+0000) 2019-10-13T08:19:31.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:31.227+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.228784+0000) 2019-10-13T08:19:31.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:31.227+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.228976+0000) 2019-10-13T08:19:31.230 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:31.227+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.229190+0000) 2019-10-13T08:19:31.230 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:31.228+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.229408+0000) 2019-10-13T08:19:31.231 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:31.228+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.230224+0000) 2019-10-13T08:19:31.231 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:31.229+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.230494+0000) 2019-10-13T08:19:31.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:31.232+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.232878+0000) 2019-10-13T08:19:31.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:31.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:31.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:31.994+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.995774+0000) 2019-10-13T08:19:32.041 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:19:32.041 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:32.045 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:19:32.045 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:32.745 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:32.744+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:32.744721+0000) 2019-10-13T08:19:32.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:32.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:32.855357+0000) 2019-10-13T08:19:32.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:32.854+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:32.855054+0000) 2019-10-13T08:19:36.237 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:36.235+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.236621+0000) 2019-10-13T08:19:36.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:36.235+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.237036+0000) 2019-10-13T08:19:36.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:36.236+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.237498+0000) 2019-10-13T08:19:36.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:36.235+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.237263+0000) 2019-10-13T08:19:36.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:36.236+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.237675+0000) 2019-10-13T08:19:36.239 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:36.236+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.237384+0000) 2019-10-13T08:19:36.239 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:36.236+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.237987+0000) 2019-10-13T08:19:36.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:36.239+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.239821+0000) 2019-10-13T08:19:36.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:36.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:37.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:37.855+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:37.856972+0000) 2019-10-13T08:19:38.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:38.404+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:38.405580+0000) 2019-10-13T08:19:38.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:38.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:38.743252+0000) 2019-10-13T08:19:38.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:38.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:38.743253+0000) 2019-10-13T08:19:41.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:41.243+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.244537+0000) 2019-10-13T08:19:41.247 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:41.243+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.244948+0000) 2019-10-13T08:19:41.247 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:41.244+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.245752+0000) 2019-10-13T08:19:41.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:41.244+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.245779+0000) 2019-10-13T08:19:41.247 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:41.244+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.245870+0000) 2019-10-13T08:19:41.248 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:41.244+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.246153+0000) 2019-10-13T08:19:41.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:41.245+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.246430+0000) 2019-10-13T08:19:41.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:41.247+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.247997+0000) 2019-10-13T08:19:41.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:41.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:41.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:41.995+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.995948+0000) 2019-10-13T08:19:42.747 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:42.746+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:42.746916+0000) 2019-10-13T08:19:42.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:42.854+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:42.855234+0000) 2019-10-13T08:19:42.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:42.854+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:42.855484+0000) 2019-10-13T08:19:46.254 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:46.252+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.253035+0000) 2019-10-13T08:19:46.255 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:46.251+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.253373+0000) 2019-10-13T08:19:46.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:46.252+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.253569+0000) 2019-10-13T08:19:46.255 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:46.252+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.253915+0000) 2019-10-13T08:19:46.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:46.252+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.254092+0000) 2019-10-13T08:19:46.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:46.252+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.254177+0000) 2019-10-13T08:19:46.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:46.252+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.254307+0000) 2019-10-13T08:19:46.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:46.255+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.256553+0000) 2019-10-13T08:19:46.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:46.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:47.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:47.856+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:47.857173+0000) 2019-10-13T08:19:48.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:48.405+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:48.405722+0000) 2019-10-13T08:19:48.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:48.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:48.743478+0000) 2019-10-13T08:19:48.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:48.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:48.743478+0000) 2019-10-13T08:19:51.261 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:51.260+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.261144+0000) 2019-10-13T08:19:51.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:51.260+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.261298+0000) 2019-10-13T08:19:51.263 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:51.260+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.261282+0000) 2019-10-13T08:19:51.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:51.261+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.261661+0000) 2019-10-13T08:19:51.263 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:51.261+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.262046+0000) 2019-10-13T08:19:51.263 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:51.261+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.262269+0000) 2019-10-13T08:19:51.264 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:51.262+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.262599+0000) 2019-10-13T08:19:51.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:51.263+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.264541+0000) 2019-10-13T08:19:51.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:51.266+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:52.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:51.995+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.996099+0000) 2019-10-13T08:19:52.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:52.748+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:52.749260+0000) 2019-10-13T08:19:52.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:52.855+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:52.855420+0000) 2019-10-13T08:19:52.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:52.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:52.855645+0000) 2019-10-13T08:19:56.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:19:56.267+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.268479+0000) 2019-10-13T08:19:56.269 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:56.268+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.268926+0000) 2019-10-13T08:19:56.270 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:19:56.268+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.269172+0000) 2019-10-13T08:19:56.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:56.268+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.269382+0000) 2019-10-13T08:19:56.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:56.269+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.269936+0000) 2019-10-13T08:19:56.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:19:56.269+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.270030+0000) 2019-10-13T08:19:56.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:56.269+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.270189+0000) 2019-10-13T08:19:56.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:19:56.270+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.271520+0000) 2019-10-13T08:19:56.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:19:56.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:19:57.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:19:57.856+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:57.857377+0000) 2019-10-13T08:19:58.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:19:58.405+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:58.405898+0000) 2019-10-13T08:19:58.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:19:58.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:58.743640+0000) 2019-10-13T08:19:58.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:19:58.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:58.743640+0000) 2019-10-13T08:20:01.277 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:01.276+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.276479+0000) 2019-10-13T08:20:01.277 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:01.276+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.276625+0000) 2019-10-13T08:20:01.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:01.275+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.276618+0000) 2019-10-13T08:20:01.278 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:01.276+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.276997+0000) 2019-10-13T08:20:01.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:01.276+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.277125+0000) 2019-10-13T08:20:01.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:01.276+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.277338+0000) 2019-10-13T08:20:01.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:01.277+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.277702+0000) 2019-10-13T08:20:01.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:01.278+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.279487+0000) 2019-10-13T08:20:01.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:01.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141263 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:01.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:01.996+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.996240+0000) 2019-10-13T08:20:02.246 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:20:02.246 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:02.250 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:20:02.251 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:02.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:02.751+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:02.751779+0000) 2019-10-13T08:20:02.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:02.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:02.855838+0000) 2019-10-13T08:20:02.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:02.855+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:02.855614+0000) 2019-10-13T08:20:06.285 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:06.282+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.284027+0000) 2019-10-13T08:20:06.287 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:06.282+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.284300+0000) 2019-10-13T08:20:06.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:06.283+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.284470+0000) 2019-10-13T08:20:06.288 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:06.283+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.284697+0000) 2019-10-13T08:20:06.288 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:06.283+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.285061+0000) 2019-10-13T08:20:06.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:06.283+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.283951+0000) 2019-10-13T08:20:06.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:06.283+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.285339+0000) 2019-10-13T08:20:06.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:06.285+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.285790+0000) 2019-10-13T08:20:06.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:06.288+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:07.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:07.856+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:07.857597+0000) 2019-10-13T08:20:08.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:08.405+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:08.406095+0000) 2019-10-13T08:20:08.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:08.742+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:08.743842+0000) 2019-10-13T08:20:08.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:08.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:08.743842+0000) 2019-10-13T08:20:11.292 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:11.290+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.292116+0000) 2019-10-13T08:20:11.293 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:11.290+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.292682+0000) 2019-10-13T08:20:11.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:11.291+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.292423+0000) 2019-10-13T08:20:11.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:11.291+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.292568+0000) 2019-10-13T08:20:11.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:11.291+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.293075+0000) 2019-10-13T08:20:11.294 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:11.291+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.293241+0000) 2019-10-13T08:20:11.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:11.291+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.293385+0000) 2019-10-13T08:20:11.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:11.292+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.294574+0000) 2019-10-13T08:20:11.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:11.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:11.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:11.995+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.996423+0000) 2019-10-13T08:20:12.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:12.753+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:12.754249+0000) 2019-10-13T08:20:12.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:12.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:12.856059+0000) 2019-10-13T08:20:12.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:12.854+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:12.855796+0000) 2019-10-13T08:20:16.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:16.298+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.299534+0000) 2019-10-13T08:20:16.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:16.298+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.299614+0000) 2019-10-13T08:20:16.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:16.298+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.300370+0000) 2019-10-13T08:20:16.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:16.299+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.300523+0000) 2019-10-13T08:20:16.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:16.299+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.300700+0000) 2019-10-13T08:20:16.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:16.299+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.300793+0000) 2019-10-13T08:20:16.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:16.299+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.301078+0000) 2019-10-13T08:20:16.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:16.300+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.302394+0000) 2019-10-13T08:20:16.306 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:16.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:17.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:17.856+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:17.857760+0000) 2019-10-13T08:20:18.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:18.405+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:18.406277+0000) 2019-10-13T08:20:18.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:18.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:18.744025+0000) 2019-10-13T08:20:18.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:18.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:18.744025+0000) 2019-10-13T08:20:21.308 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:21.306+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.307722+0000) 2019-10-13T08:20:21.309 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:21.306+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.307776+0000) 2019-10-13T08:20:21.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:21.306+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.308211+0000) 2019-10-13T08:20:21.309 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:21.307+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.308423+0000) 2019-10-13T08:20:21.310 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:21.307+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.308621+0000) 2019-10-13T08:20:21.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:21.307+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.308818+0000) 2019-10-13T08:20:21.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:21.307+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.308991+0000) 2019-10-13T08:20:21.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:21.310+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.311968+0000) 2019-10-13T08:20:21.314 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:21.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:21.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:21.995+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.996590+0000) 2019-10-13T08:20:22.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:22.756+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:22.756769+0000) 2019-10-13T08:20:22.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:22.855+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:22.855932+0000) 2019-10-13T08:20:22.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:22.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:22.856226+0000) 2019-10-13T08:20:26.316 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:26.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.315970+0000) 2019-10-13T08:20:26.317 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:26.314+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.316277+0000) 2019-10-13T08:20:26.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:26.315+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.316457+0000) 2019-10-13T08:20:26.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:26.315+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.316706+0000) 2019-10-13T08:20:26.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:26.315+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.316976+0000) 2019-10-13T08:20:26.318 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:26.315+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.317109+0000) 2019-10-13T08:20:26.318 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:26.316+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.317544+0000) 2019-10-13T08:20:26.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:26.318+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.318857+0000) 2019-10-13T08:20:26.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:26.321+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141418 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:27.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:27.857+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:27.857895+0000) 2019-10-13T08:20:28.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:28.405+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:28.406470+0000) 2019-10-13T08:20:28.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:28.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:28.744213+0000) 2019-10-13T08:20:28.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:28.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:28.744213+0000) 2019-10-13T08:20:31.325 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:31.323+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.324382+0000) 2019-10-13T08:20:31.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:31.323+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.324609+0000) 2019-10-13T08:20:31.326 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:31.323+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.324786+0000) 2019-10-13T08:20:31.326 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:31.324+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.325424+0000) 2019-10-13T08:20:31.326 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:31.324+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.325592+0000) 2019-10-13T08:20:31.327 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:31.324+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.325861+0000) 2019-10-13T08:20:31.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:31.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.325956+0000) 2019-10-13T08:20:31.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:31.326+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.327292+0000) 2019-10-13T08:20:31.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:31.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:31.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:31.995+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.996772+0000) 2019-10-13T08:20:32.440 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:20:32.440 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:32.444 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:20:32.444 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:32.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:32.758+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:32.758949+0000) 2019-10-13T08:20:32.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:32.855+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:32.856071+0000) 2019-10-13T08:20:32.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:32.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:32.856423+0000) 2019-10-13T08:20:36.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:36.331+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.332026+0000) 2019-10-13T08:20:36.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:36.331+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.332300+0000) 2019-10-13T08:20:36.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:36.331+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.332296+0000) 2019-10-13T08:20:36.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:36.332+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.332668+0000) 2019-10-13T08:20:36.334 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:36.332+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.333025+0000) 2019-10-13T08:20:36.334 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:36.333+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.333528+0000) 2019-10-13T08:20:36.334 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:36.333+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.333672+0000) 2019-10-13T08:20:36.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:36.334+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.335197+0000) 2019-10-13T08:20:36.339 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:36.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141479 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:37.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:37.857+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:37.858022+0000) 2019-10-13T08:20:38.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:38.406+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:38.406668+0000) 2019-10-13T08:20:38.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:38.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:38.744436+0000) 2019-10-13T08:20:38.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:38.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:38.744436+0000) 2019-10-13T08:20:41.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:41.340+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.340442+0000) 2019-10-13T08:20:41.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:41.340+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.340824+0000) 2019-10-13T08:20:41.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:41.340+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.340864+0000) 2019-10-13T08:20:41.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:41.340+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.340997+0000) 2019-10-13T08:20:41.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:41.341+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.341432+0000) 2019-10-13T08:20:41.343 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:41.340+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.340992+0000) 2019-10-13T08:20:41.343 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:41.341+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.341750+0000) 2019-10-13T08:20:41.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:41.342+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.343267+0000) 2019-10-13T08:20:41.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:41.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141511 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:41.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:41.996+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.996896+0000) 2019-10-13T08:20:42.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:42.761+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:42.761457+0000) 2019-10-13T08:20:42.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:42.855+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:42.856259+0000) 2019-10-13T08:20:42.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:42.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:42.856599+0000) 2019-10-13T08:20:46.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:46.348+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.348442+0000) 2019-10-13T08:20:46.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:46.348+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.348599+0000) 2019-10-13T08:20:46.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:46.348+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.348806+0000) 2019-10-13T08:20:46.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:46.348+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.348969+0000) 2019-10-13T08:20:46.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:46.348+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.349096+0000) 2019-10-13T08:20:46.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:46.348+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.349125+0000) 2019-10-13T08:20:46.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:46.349+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.349804+0000) 2019-10-13T08:20:46.351 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:46.349+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.349545+0000) 2019-10-13T08:20:46.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:46.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141542 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:47.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:47.858+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:47.858234+0000) 2019-10-13T08:20:48.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:48.406+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:48.406852+0000) 2019-10-13T08:20:48.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:48.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:48.744644+0000) 2019-10-13T08:20:48.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:48.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:48.744644+0000) 2019-10-13T08:20:51.356 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:51.354+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.355290+0000) 2019-10-13T08:20:51.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:51.355+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.355481+0000) 2019-10-13T08:20:51.356 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:51.355+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.355846+0000) 2019-10-13T08:20:51.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:51.355+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.356044+0000) 2019-10-13T08:20:51.357 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:51.354+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.355565+0000) 2019-10-13T08:20:51.358 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:51.355+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.356257+0000) 2019-10-13T08:20:51.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:51.355+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.356040+0000) 2019-10-13T08:20:51.358 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:51.357+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.358351+0000) 2019-10-13T08:20:51.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:51.360+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141572 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:51.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:51.997+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.997045+0000) 2019-10-13T08:20:52.764 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:52.763+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:52.763699+0000) 2019-10-13T08:20:52.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:52.856+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:52.856378+0000) 2019-10-13T08:20:52.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:52.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:52.856774+0000) 2019-10-13T08:20:56.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:20:56.361+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.362286+0000) 2019-10-13T08:20:56.363 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:20:56.361+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.362862+0000) 2019-10-13T08:20:56.363 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:56.361+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.363054+0000) 2019-10-13T08:20:56.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:56.361+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.363236+0000) 2019-10-13T08:20:56.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:56.362+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.363574+0000) 2019-10-13T08:20:56.364 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:20:56.362+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.363705+0000) 2019-10-13T08:20:56.365 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:56.362+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.363452+0000) 2019-10-13T08:20:56.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:20:56.363+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.364559+0000) 2019-10-13T08:20:56.368 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:20:56.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:20:57.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:20:57.857+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:57.858441+0000) 2019-10-13T08:20:58.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:20:58.406+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:58.407003+0000) 2019-10-13T08:20:58.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:20:58.743+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:58.744828+0000) 2019-10-13T08:20:58.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:20:58.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:58.744828+0000) 2019-10-13T08:21:01.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:01.368+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.370309+0000) 2019-10-13T08:21:01.371 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:01.368+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.370443+0000) 2019-10-13T08:21:01.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:01.369+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.370871+0000) 2019-10-13T08:21:01.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:01.369+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.371002+0000) 2019-10-13T08:21:01.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:01.369+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.371162+0000) 2019-10-13T08:21:01.372 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:01.370+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.371435+0000) 2019-10-13T08:21:01.373 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:01.370+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.371681+0000) 2019-10-13T08:21:01.373 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:01.370+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.372695+0000) 2019-10-13T08:21:01.376 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:01.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:01.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:01.995+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.997232+0000) 2019-10-13T08:21:02.594 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:21:02.594 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:02.598 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:21:02.598 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:02.766 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:02.764+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:02.765850+0000) 2019-10-13T08:21:02.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:02.855+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:02.856492+0000) 2019-10-13T08:21:02.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:02.855+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:02.856876+0000) 2019-10-13T08:21:06.378 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:06.376+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.377999+0000) 2019-10-13T08:21:06.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:06.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.378474+0000) 2019-10-13T08:21:06.379 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:06.377+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.378571+0000) 2019-10-13T08:21:06.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:06.377+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.378825+0000) 2019-10-13T08:21:06.379 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:06.377+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.379175+0000) 2019-10-13T08:21:06.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:06.377+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.379390+0000) 2019-10-13T08:21:06.380 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:06.378+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.379424+0000) 2019-10-13T08:21:06.381 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:06.378+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.380724+0000) 2019-10-13T08:21:06.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:06.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:07.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:07.857+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:07.858636+0000) 2019-10-13T08:21:08.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:08.406+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:08.407217+0000) 2019-10-13T08:21:08.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:08.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:08.744981+0000) 2019-10-13T08:21:08.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:08.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:08.744981+0000) 2019-10-13T08:21:11.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:11.384+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.386130+0000) 2019-10-13T08:21:11.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:11.385+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.386429+0000) 2019-10-13T08:21:11.388 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:11.384+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.386488+0000) 2019-10-13T08:21:11.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:11.385+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.386958+0000) 2019-10-13T08:21:11.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:11.385+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.387166+0000) 2019-10-13T08:21:11.388 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:11.385+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.387349+0000) 2019-10-13T08:21:11.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:11.385+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.387367+0000) 2019-10-13T08:21:11.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:11.386+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.388671+0000) 2019-10-13T08:21:11.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:11.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:11.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:11.996+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.997394+0000) 2019-10-13T08:21:12.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:12.768+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:12.768942+0000) 2019-10-13T08:21:12.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:12.855+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:12.856654+0000) 2019-10-13T08:21:12.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:12.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:12.857034+0000) 2019-10-13T08:21:16.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:16.392+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.393275+0000) 2019-10-13T08:21:16.394 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:16.392+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.393700+0000) 2019-10-13T08:21:16.394 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:16.392+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.393998+0000) 2019-10-13T08:21:16.395 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:16.392+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.394179+0000) 2019-10-13T08:21:16.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:16.393+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.394428+0000) 2019-10-13T08:21:16.395 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:16.393+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.394553+0000) 2019-10-13T08:21:16.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:16.393+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.394657+0000) 2019-10-13T08:21:16.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:16.395+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.395806+0000) 2019-10-13T08:21:16.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:16.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:17.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:17.858+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:17.858823+0000) 2019-10-13T08:21:18.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:18.406+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:18.407420+0000) 2019-10-13T08:21:18.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:18.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:18.745146+0000) 2019-10-13T08:21:18.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:18.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:18.745146+0000) 2019-10-13T08:21:21.402 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:21.401+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.401675+0000) 2019-10-13T08:21:21.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:21.401+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.402249+0000) 2019-10-13T08:21:21.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:21.402+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.402526+0000) 2019-10-13T08:21:21.403 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:21.402+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.402640+0000) 2019-10-13T08:21:21.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:21.401+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.401986+0000) 2019-10-13T08:21:21.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:21.402+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.403148+0000) 2019-10-13T08:21:21.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:21.402+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.402972+0000) 2019-10-13T08:21:21.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:21.403+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.404393+0000) 2019-10-13T08:21:21.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:21.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141759 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:21.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:21.997+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.997570+0000) 2019-10-13T08:21:22.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:22.770+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.771206+0000) 2019-10-13T08:21:22.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:22.856+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.856800+0000) 2019-10-13T08:21:22.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:22.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:22.857188+0000) 2019-10-13T08:21:26.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:26.409+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.409890+0000) 2019-10-13T08:21:26.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:26.409+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.410135+0000) 2019-10-13T08:21:26.411 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:26.409+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.410246+0000) 2019-10-13T08:21:26.411 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:26.410+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.410443+0000) 2019-10-13T08:21:26.412 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:26.409+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.410398+0000) 2019-10-13T08:21:26.412 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:26.410+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.410655+0000) 2019-10-13T08:21:26.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:26.410+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.410818+0000) 2019-10-13T08:21:26.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:26.412+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.413011+0000) 2019-10-13T08:21:26.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:26.414+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:27.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:27.858+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.859000+0000) 2019-10-13T08:21:28.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:28.407+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:28.407621+0000) 2019-10-13T08:21:28.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:28.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:28.745330+0000) 2019-10-13T08:21:28.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:28.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:28.745331+0000) 2019-10-13T08:21:31.416 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:31.415+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.416132+0000) 2019-10-13T08:21:31.417 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:31.416+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.416664+0000) 2019-10-13T08:21:31.418 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:31.416+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.417107+0000) 2019-10-13T08:21:31.418 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:31.416+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.417228+0000) 2019-10-13T08:21:31.418 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:31.417+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.417448+0000) 2019-10-13T08:21:31.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:31.417+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.417625+0000) 2019-10-13T08:21:31.419 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:31.417+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.418416+0000) 2019-10-13T08:21:31.419 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:31.417+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.417871+0000) 2019-10-13T08:21:31.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:31.421+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:31.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:31.997+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.997713+0000) 2019-10-13T08:21:32.756 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:21:32.756 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:32.762 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:21:32.762 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:32.773 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:32.772+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.773452+0000) 2019-10-13T08:21:32.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:32.856+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.856977+0000) 2019-10-13T08:21:32.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:32.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:32.857359+0000) 2019-10-13T08:21:36.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:36.423+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.424078+0000) 2019-10-13T08:21:36.425 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:36.423+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.424245+0000) 2019-10-13T08:21:36.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:36.424+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.424440+0000) 2019-10-13T08:21:36.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:36.424+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.424640+0000) 2019-10-13T08:21:36.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:36.424+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.424849+0000) 2019-10-13T08:21:36.427 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:36.424+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.425197+0000) 2019-10-13T08:21:36.427 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:36.424+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.425279+0000) 2019-10-13T08:21:36.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:36.426+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.427471+0000) 2019-10-13T08:21:36.430 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:36.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:37.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:37.858+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.859210+0000) 2019-10-13T08:21:38.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:38.407+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:38.407816+0000) 2019-10-13T08:21:38.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:38.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:38.745555+0000) 2019-10-13T08:21:38.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:38.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:38.745555+0000) 2019-10-13T08:21:41.432 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:41.430+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.431760+0000) 2019-10-13T08:21:41.433 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:41.430+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.431968+0000) 2019-10-13T08:21:41.433 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:41.430+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.432133+0000) 2019-10-13T08:21:41.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:41.431+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.432437+0000) 2019-10-13T08:21:41.434 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:41.432+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.433138+0000) 2019-10-13T08:21:41.434 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:41.431+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.432931+0000) 2019-10-13T08:21:41.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:41.431+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.432763+0000) 2019-10-13T08:21:41.435 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:41.434+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.435356+0000) 2019-10-13T08:21:41.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:41.436+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141883 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:41.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:41.996+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.997902+0000) 2019-10-13T08:21:42.776 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:42.775+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.775783+0000) 2019-10-13T08:21:42.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:42.856+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.857130+0000) 2019-10-13T08:21:42.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:42.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:42.857518+0000) 2019-10-13T08:21:46.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:46.438+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.439110+0000) 2019-10-13T08:21:46.440 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:46.437+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.439113+0000) 2019-10-13T08:21:46.441 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:46.438+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.439689+0000) 2019-10-13T08:21:46.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:46.438+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.440054+0000) 2019-10-13T08:21:46.441 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:46.438+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.440192+0000) 2019-10-13T08:21:46.441 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:46.439+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.440411+0000) 2019-10-13T08:21:46.441 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:46.439+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.440666+0000) 2019-10-13T08:21:46.442 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:46.440+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.441355+0000) 2019-10-13T08:21:46.446 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:46.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141914 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:47.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:47.858+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.859416+0000) 2019-10-13T08:21:48.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:48.407+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:48.407988+0000) 2019-10-13T08:21:48.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:48.744+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:48.745724+0000) 2019-10-13T08:21:48.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:48.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:48.745724+0000) 2019-10-13T08:21:51.448 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:51.446+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.446782+0000) 2019-10-13T08:21:51.448 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:51.445+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.446666+0000) 2019-10-13T08:21:51.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:51.445+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.447113+0000) 2019-10-13T08:21:51.448 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:51.446+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.447412+0000) 2019-10-13T08:21:51.448 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:51.446+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.447521+0000) 2019-10-13T08:21:51.449 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:51.446+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.448106+0000) 2019-10-13T08:21:51.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:51.446+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.448221+0000) 2019-10-13T08:21:51.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:51.449+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.450656+0000) 2019-10-13T08:21:51.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:51.451+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:51.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:51.997+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.998057+0000) 2019-10-13T08:21:52.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:52.778+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.778223+0000) 2019-10-13T08:21:52.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:52.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.857239+0000) 2019-10-13T08:21:52.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:52.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:52.857683+0000) 2019-10-13T08:21:56.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:21:56.453+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.455158+0000) 2019-10-13T08:21:56.456 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:56.454+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.455391+0000) 2019-10-13T08:21:56.456 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:21:56.454+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.455614+0000) 2019-10-13T08:21:56.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:56.454+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.455786+0000) 2019-10-13T08:21:56.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:56.454+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.455923+0000) 2019-10-13T08:21:56.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:21:56.453+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.455388+0000) 2019-10-13T08:21:56.458 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:56.455+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.456619+0000) 2019-10-13T08:21:56.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:21:56.456+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.458372+0000) 2019-10-13T08:21:56.461 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:21:56.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 141975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:21:57.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:21:57.858+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.859601+0000) 2019-10-13T08:21:58.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:21:58.406+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:58.408183+0000) 2019-10-13T08:21:58.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:21:58.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:58.745911+0000) 2019-10-13T08:21:58.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:21:58.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:58.745911+0000) 2019-10-13T08:22:01.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:01.460+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.462362+0000) 2019-10-13T08:22:01.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:01.461+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.462661+0000) 2019-10-13T08:22:01.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:01.461+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.463166+0000) 2019-10-13T08:22:01.464 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:01.461+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.463313+0000) 2019-10-13T08:22:01.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:01.462+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.463459+0000) 2019-10-13T08:22:01.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:01.462+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.463642+0000) 2019-10-13T08:22:01.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:01.462+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.463822+0000) 2019-10-13T08:22:01.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:01.464+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.466529+0000) 2019-10-13T08:22:01.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:01.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142007 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:01.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:01.997+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.998205+0000) 2019-10-13T08:22:02.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:02.779+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.780786+0000) 2019-10-13T08:22:02.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:02.856+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.857346+0000) 2019-10-13T08:22:02.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:02.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:02.857784+0000) 2019-10-13T08:22:02.969 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:22:02.969 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:02.974 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:22:02.974 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:06.470 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:06.469+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.469561+0000) 2019-10-13T08:22:06.471 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:06.469+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.469688+0000) 2019-10-13T08:22:06.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:06.469+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.470180+0000) 2019-10-13T08:22:06.471 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:06.469+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.470353+0000) 2019-10-13T08:22:06.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:06.470+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.470484+0000) 2019-10-13T08:22:06.472 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:06.468+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.470508+0000) 2019-10-13T08:22:06.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:06.470+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.471992+0000) 2019-10-13T08:22:06.473 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:06.470+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.470891+0000) 2019-10-13T08:22:06.475 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:06.475+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:07.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:07.859+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.859773+0000) 2019-10-13T08:22:08.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:08.408+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:08.408381+0000) 2019-10-13T08:22:08.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:08.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:08.746094+0000) 2019-10-13T08:22:08.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:08.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:08.746094+0000) 2019-10-13T08:22:11.477 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:11.476+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.477214+0000) 2019-10-13T08:22:11.478 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:11.477+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.477427+0000) 2019-10-13T08:22:11.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:11.477+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.477607+0000) 2019-10-13T08:22:11.479 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:11.476+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.477668+0000) 2019-10-13T08:22:11.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:11.477+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.478079+0000) 2019-10-13T08:22:11.479 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:11.478+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.478462+0000) 2019-10-13T08:22:11.479 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:11.478+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.479037+0000) 2019-10-13T08:22:11.481 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:11.480+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.480943+0000) 2019-10-13T08:22:11.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:11.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:11.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:11.998+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.998368+0000) 2019-10-13T08:22:12.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:12.782+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.783258+0000) 2019-10-13T08:22:12.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:12.856+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.857444+0000) 2019-10-13T08:22:12.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:12.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:12.857961+0000) 2019-10-13T08:22:16.485 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:16.485+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.485427+0000) 2019-10-13T08:22:16.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:16.484+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.485480+0000) 2019-10-13T08:22:16.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:16.485+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.485787+0000) 2019-10-13T08:22:16.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:16.485+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.485923+0000) 2019-10-13T08:22:16.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:16.485+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.486237+0000) 2019-10-13T08:22:16.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:16.486+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.486512+0000) 2019-10-13T08:22:16.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:16.486+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.486770+0000) 2019-10-13T08:22:16.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:16.486+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.486897+0000) 2019-10-13T08:22:16.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:16.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:17.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:17.859+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.859945+0000) 2019-10-13T08:22:18.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:18.408+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:18.408551+0000) 2019-10-13T08:22:18.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:18.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:18.746249+0000) 2019-10-13T08:22:18.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:18.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:18.746249+0000) 2019-10-13T08:22:21.493 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:21.492+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.492484+0000) 2019-10-13T08:22:21.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:21.492+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.492654+0000) 2019-10-13T08:22:21.494 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:21.492+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.492992+0000) 2019-10-13T08:22:21.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:21.492+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.493385+0000) 2019-10-13T08:22:21.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:21.493+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.493412+0000) 2019-10-13T08:22:21.495 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:21.493+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.493660+0000) 2019-10-13T08:22:21.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:21.493+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.493793+0000) 2019-10-13T08:22:21.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:21.494+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.494997+0000) 2019-10-13T08:22:21.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:21.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:21.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:21.998+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.998564+0000) 2019-10-13T08:22:22.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:22.785+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.785603+0000) 2019-10-13T08:22:22.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:22.856+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.857611+0000) 2019-10-13T08:22:22.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:22.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:22.858118+0000) 2019-10-13T08:22:26.500 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:26.499+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.499863+0000) 2019-10-13T08:22:26.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:26.498+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.500176+0000) 2019-10-13T08:22:26.501 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:26.498+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.500298+0000) 2019-10-13T08:22:26.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:26.499+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.500502+0000) 2019-10-13T08:22:26.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:26.499+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.500593+0000) 2019-10-13T08:22:26.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:26.499+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.500743+0000) 2019-10-13T08:22:26.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:26.499+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.501052+0000) 2019-10-13T08:22:26.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:26.501+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.501806+0000) 2019-10-13T08:22:26.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:26.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:27.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:27.859+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.860142+0000) 2019-10-13T08:22:28.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:28.407+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:28.408741+0000) 2019-10-13T08:22:28.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:28.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:28.746437+0000) 2019-10-13T08:22:28.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:28.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:28.746437+0000) 2019-10-13T08:22:31.507 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:31.506+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.506827+0000) 2019-10-13T08:22:31.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:31.505+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.507164+0000) 2019-10-13T08:22:31.508 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:31.505+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.507300+0000) 2019-10-13T08:22:31.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:31.506+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.507704+0000) 2019-10-13T08:22:31.509 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:31.506+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.507860+0000) 2019-10-13T08:22:31.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:31.506+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.508070+0000) 2019-10-13T08:22:31.509 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:31.506+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.508263+0000) 2019-10-13T08:22:31.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:31.510+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.510743+0000) 2019-10-13T08:22:31.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:31.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:32.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:31.997+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.998695+0000) 2019-10-13T08:22:32.788 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:32.787+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.787874+0000) 2019-10-13T08:22:32.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:32.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.857799+0000) 2019-10-13T08:22:32.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:32.856+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:32.858247+0000) 2019-10-13T08:22:33.176 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:22:33.176 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:33.180 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:22:33.181 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:36.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:36.513+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.515090+0000) 2019-10-13T08:22:36.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:36.514+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.515590+0000) 2019-10-13T08:22:36.516 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:36.514+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.515753+0000) 2019-10-13T08:22:36.517 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:36.514+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.515931+0000) 2019-10-13T08:22:36.517 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:36.514+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.516193+0000) 2019-10-13T08:22:36.517 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:36.515+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.516302+0000) 2019-10-13T08:22:36.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:36.515+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.516688+0000) 2019-10-13T08:22:36.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:36.516+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.517463+0000) 2019-10-13T08:22:36.521 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:36.519+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:37.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:37.859+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.860317+0000) 2019-10-13T08:22:38.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:38.408+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:38.408933+0000) 2019-10-13T08:22:38.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:38.745+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:38.746649+0000) 2019-10-13T08:22:38.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:38.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:38.746649+0000) 2019-10-13T08:22:41.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:41.521+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.522917+0000) 2019-10-13T08:22:41.523 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:41.521+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.523105+0000) 2019-10-13T08:22:41.524 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:41.522+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.523651+0000) 2019-10-13T08:22:41.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:41.521+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.523345+0000) 2019-10-13T08:22:41.525 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:41.522+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.523572+0000) 2019-10-13T08:22:41.525 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:41.522+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.523899+0000) 2019-10-13T08:22:41.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:41.522+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.524302+0000) 2019-10-13T08:22:41.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:41.525+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.526114+0000) 2019-10-13T08:22:41.529 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:41.527+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142255 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:42.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:41.997+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.998881+0000) 2019-10-13T08:22:42.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:42.790+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.790130+0000) 2019-10-13T08:22:42.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:42.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.857974+0000) 2019-10-13T08:22:42.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:42.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:42.858447+0000) 2019-10-13T08:22:46.531 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:46.529+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.530728+0000) 2019-10-13T08:22:46.532 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:46.529+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.531387+0000) 2019-10-13T08:22:46.532 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:46.529+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.531283+0000) 2019-10-13T08:22:46.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:46.529+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.531302+0000) 2019-10-13T08:22:46.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:46.530+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.531672+0000) 2019-10-13T08:22:46.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:46.530+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.532013+0000) 2019-10-13T08:22:46.533 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:46.530+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.532228+0000) 2019-10-13T08:22:46.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:46.532+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.533848+0000) 2019-10-13T08:22:46.537 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:46.535+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:47.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:47.859+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.860506+0000) 2019-10-13T08:22:48.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:48.408+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:48.409147+0000) 2019-10-13T08:22:48.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:48.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:48.746835+0000) 2019-10-13T08:22:48.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:48.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:48.746836+0000) 2019-10-13T08:22:51.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:51.537+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.539105+0000) 2019-10-13T08:22:51.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:51.537+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.539257+0000) 2019-10-13T08:22:51.540 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:51.538+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.539460+0000) 2019-10-13T08:22:51.541 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:51.538+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.540045+0000) 2019-10-13T08:22:51.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:51.538+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.540220+0000) 2019-10-13T08:22:51.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:51.539+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.540835+0000) 2019-10-13T08:22:51.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:51.538+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.540501+0000) 2019-10-13T08:22:51.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:51.540+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.541727+0000) 2019-10-13T08:22:51.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:51.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:51.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:51.998+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.999070+0000) 2019-10-13T08:22:52.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:52.791+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.792350+0000) 2019-10-13T08:22:52.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:52.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.858125+0000) 2019-10-13T08:22:52.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:52.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:52.858601+0000) 2019-10-13T08:22:56.547 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:22:56.545+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.547391+0000) 2019-10-13T08:22:56.548 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:22:56.547+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.547830+0000) 2019-10-13T08:22:56.549 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:22:56.547+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.547933+0000) 2019-10-13T08:22:56.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:56.547+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.548127+0000) 2019-10-13T08:22:56.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:56.548+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.548457+0000) 2019-10-13T08:22:56.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:56.548+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.548728+0000) 2019-10-13T08:22:56.550 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:22:56.547+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.549267+0000) 2019-10-13T08:22:56.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:56.548+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.548783+0000) 2019-10-13T08:22:56.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:22:56.552+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:22:57.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:22:57.860+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.860709+0000) 2019-10-13T08:22:58.409 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:22:58.408+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:58.409354+0000) 2019-10-13T08:22:58.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:22:58.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:58.747011+0000) 2019-10-13T08:22:58.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:22:58.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:58.747011+0000) 2019-10-13T08:23:01.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:01.554+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.554960+0000) 2019-10-13T08:23:01.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:01.554+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.555083+0000) 2019-10-13T08:23:01.556 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:01.554+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.555237+0000) 2019-10-13T08:23:01.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:01.554+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.555323+0000) 2019-10-13T08:23:01.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:01.555+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.555763+0000) 2019-10-13T08:23:01.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:01.555+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.555995+0000) 2019-10-13T08:23:01.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:01.556+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.556698+0000) 2019-10-13T08:23:01.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:01.556+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.557589+0000) 2019-10-13T08:23:01.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:01.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142379 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:01.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:01.999+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.999233+0000) 2019-10-13T08:23:02.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:02.794+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:02.794894+0000) 2019-10-13T08:23:02.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:02.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:02.858228+0000) 2019-10-13T08:23:02.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:02.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:02.858710+0000) 2019-10-13T08:23:03.286 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:23:03.287 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:03.291 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:23:03.292 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:06.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:06.562+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.562954+0000) 2019-10-13T08:23:06.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:06.562+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.563304+0000) 2019-10-13T08:23:06.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:06.563+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.563499+0000) 2019-10-13T08:23:06.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:06.562+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.563702+0000) 2019-10-13T08:23:06.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:06.563+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.563905+0000) 2019-10-13T08:23:06.565 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:06.563+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.564108+0000) 2019-10-13T08:23:06.565 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:06.564+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.564449+0000) 2019-10-13T08:23:06.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:06.565+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.566320+0000) 2019-10-13T08:23:06.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:06.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:07.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:07.860+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:07.860914+0000) 2019-10-13T08:23:08.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:08.409+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.409571+0000) 2019-10-13T08:23:08.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:08.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.747256+0000) 2019-10-13T08:23:08.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:08.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:08.747256+0000) 2019-10-13T08:23:11.571 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:11.569+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.570577+0000) 2019-10-13T08:23:11.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:11.569+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.570741+0000) 2019-10-13T08:23:11.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:11.569+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.571049+0000) 2019-10-13T08:23:11.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:11.569+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.571232+0000) 2019-10-13T08:23:11.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:11.569+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.571294+0000) 2019-10-13T08:23:11.573 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:11.570+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.570813+0000) 2019-10-13T08:23:11.573 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:11.571+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.572267+0000) 2019-10-13T08:23:11.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:11.570+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.571489+0000) 2019-10-13T08:23:11.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:11.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142440 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:11.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:11.998+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.999403+0000) 2019-10-13T08:23:12.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:12.796+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:12.797473+0000) 2019-10-13T08:23:12.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:12.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:12.858360+0000) 2019-10-13T08:23:12.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:12.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:12.858869+0000) 2019-10-13T08:23:16.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:16.576+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.578288+0000) 2019-10-13T08:23:16.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:16.576+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.578122+0000) 2019-10-13T08:23:16.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:16.577+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.578761+0000) 2019-10-13T08:23:16.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:16.577+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.578878+0000) 2019-10-13T08:23:16.580 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:16.577+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.579069+0000) 2019-10-13T08:23:16.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:16.577+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.579206+0000) 2019-10-13T08:23:16.581 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:16.577+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.578537+0000) 2019-10-13T08:23:16.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:16.579+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.579984+0000) 2019-10-13T08:23:16.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:16.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142471 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:17.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:17.860+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:17.861128+0000) 2019-10-13T08:23:18.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:18.408+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.409750+0000) 2019-10-13T08:23:18.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:18.746+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.747502+0000) 2019-10-13T08:23:18.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:18.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:18.747502+0000) 2019-10-13T08:23:21.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:21.583+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.585275+0000) 2019-10-13T08:23:21.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:21.584+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.585574+0000) 2019-10-13T08:23:21.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:21.584+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.585357+0000) 2019-10-13T08:23:21.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:21.584+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.585728+0000) 2019-10-13T08:23:21.587 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:21.584+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.585787+0000) 2019-10-13T08:23:21.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:21.584+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.585960+0000) 2019-10-13T08:23:21.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:21.585+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.586523+0000) 2019-10-13T08:23:21.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:21.587+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.588134+0000) 2019-10-13T08:23:21.589 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:21.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:22.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:21.998+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.999566+0000) 2019-10-13T08:23:22.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:22.799+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:22.799786+0000) 2019-10-13T08:23:22.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:22.858+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:22.858481+0000) 2019-10-13T08:23:22.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:22.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:22.858980+0000) 2019-10-13T08:23:26.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:26.590+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.591534+0000) 2019-10-13T08:23:26.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:26.590+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.591748+0000) 2019-10-13T08:23:26.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:26.590+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.591935+0000) 2019-10-13T08:23:26.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:26.590+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.592051+0000) 2019-10-13T08:23:26.593 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:26.590+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.592238+0000) 2019-10-13T08:23:26.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:26.591+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.592382+0000) 2019-10-13T08:23:26.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:26.591+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.592702+0000) 2019-10-13T08:23:26.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:26.593+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.594440+0000) 2019-10-13T08:23:26.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:26.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142534 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:27.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:27.859+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:27.861342+0000) 2019-10-13T08:23:28.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:28.408+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.409911+0000) 2019-10-13T08:23:28.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:28.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.747713+0000) 2019-10-13T08:23:28.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:28.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:28.747712+0000) 2019-10-13T08:23:31.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:31.597+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.599167+0000) 2019-10-13T08:23:31.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:31.597+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.599358+0000) 2019-10-13T08:23:31.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:31.598+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.599426+0000) 2019-10-13T08:23:31.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:31.598+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.599725+0000) 2019-10-13T08:23:31.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:31.598+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.599815+0000) 2019-10-13T08:23:31.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:31.598+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.599449+0000) 2019-10-13T08:23:31.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:31.598+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.600109+0000) 2019-10-13T08:23:31.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:31.601+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.602133+0000) 2019-10-13T08:23:31.604 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:31.602+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:32.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:31.998+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.999672+0000) 2019-10-13T08:23:32.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:32.802+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:32.802361+0000) 2019-10-13T08:23:32.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:32.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:32.858656+0000) 2019-10-13T08:23:32.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:32.857+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:32.859145+0000) 2019-10-13T08:23:33.475 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:23:33.476 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:33.480 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:23:33.480 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:36.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:36.605+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.606024+0000) 2019-10-13T08:23:36.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:36.604+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.606290+0000) 2019-10-13T08:23:36.607 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:36.605+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.606496+0000) 2019-10-13T08:23:36.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:36.605+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.606843+0000) 2019-10-13T08:23:36.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:36.605+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.607047+0000) 2019-10-13T08:23:36.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:36.605+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.607203+0000) 2019-10-13T08:23:36.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:36.606+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.608190+0000) 2019-10-13T08:23:36.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:36.608+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.609506+0000) 2019-10-13T08:23:36.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:36.610+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:37.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:37.861+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:37.861583+0000) 2019-10-13T08:23:38.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:38.409+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.410092+0000) 2019-10-13T08:23:38.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:38.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.747930+0000) 2019-10-13T08:23:38.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:38.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:38.747931+0000) 2019-10-13T08:23:41.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:41.613+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.613772+0000) 2019-10-13T08:23:41.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:41.613+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.613935+0000) 2019-10-13T08:23:41.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:41.613+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.614133+0000) 2019-10-13T08:23:41.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:41.614+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.614393+0000) 2019-10-13T08:23:41.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:41.612+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.614617+0000) 2019-10-13T08:23:41.616 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:41.613+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.615634+0000) 2019-10-13T08:23:41.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:41.614+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.614818+0000) 2019-10-13T08:23:41.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:41.614+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.614962+0000) 2019-10-13T08:23:41.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:41.619+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:42.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:41.999+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.999834+0000) 2019-10-13T08:23:42.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:42.803+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:42.804708+0000) 2019-10-13T08:23:42.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:42.857+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:42.858807+0000) 2019-10-13T08:23:42.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:42.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:42.859261+0000) 2019-10-13T08:23:46.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:46.621+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.621439+0000) 2019-10-13T08:23:46.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:46.621+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.621597+0000) 2019-10-13T08:23:46.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:46.621+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.621894+0000) 2019-10-13T08:23:46.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:46.621+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.621771+0000) 2019-10-13T08:23:46.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:46.620+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.621927+0000) 2019-10-13T08:23:46.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:46.622+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.622664+0000) 2019-10-13T08:23:46.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:46.622+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.622866+0000) 2019-10-13T08:23:46.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:46.622+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.624418+0000) 2019-10-13T08:23:46.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:46.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:47.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:47.861+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:47.861762+0000) 2019-10-13T08:23:48.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:48.409+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.410271+0000) 2019-10-13T08:23:48.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:48.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.748136+0000) 2019-10-13T08:23:48.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:48.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:48.748136+0000) 2019-10-13T08:23:51.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:51.627+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.628878+0000) 2019-10-13T08:23:51.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:51.628+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.629090+0000) 2019-10-13T08:23:51.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:51.628+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.629230+0000) 2019-10-13T08:23:51.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:51.629+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.629408+0000) 2019-10-13T08:23:51.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:51.629+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.629703+0000) 2019-10-13T08:23:51.631 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:51.629+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.629855+0000) 2019-10-13T08:23:51.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:51.629+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.630082+0000) 2019-10-13T08:23:51.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:51.630+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.632399+0000) 2019-10-13T08:23:51.635 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:51.634+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:52.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:51.999+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:52.000013+0000) 2019-10-13T08:23:52.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:52.806+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:52.806972+0000) 2019-10-13T08:23:52.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:52.858+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:52.858935+0000) 2019-10-13T08:23:52.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:52.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:52.859351+0000) 2019-10-13T08:23:56.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:23:56.635+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.636221+0000) 2019-10-13T08:23:56.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:56.636+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.636612+0000) 2019-10-13T08:23:56.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:56.636+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.636763+0000) 2019-10-13T08:23:56.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:23:56.636+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.637285+0000) 2019-10-13T08:23:56.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:56.637+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.637467+0000) 2019-10-13T08:23:56.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:56.637+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.637561+0000) 2019-10-13T08:23:56.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:23:56.637+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.637847+0000) 2019-10-13T08:23:56.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:23:56.638+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.638984+0000) 2019-10-13T08:23:56.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:23:56.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142719 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:23:57.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:23:57.860+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:57.861941+0000) 2019-10-13T08:23:58.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:23:58.409+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.410459+0000) 2019-10-13T08:23:58.748 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:23:58.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.748322+0000) 2019-10-13T08:23:58.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:23:58.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:58.748322+0000) 2019-10-13T08:24:01.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:01.643+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.644531+0000) 2019-10-13T08:24:01.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:01.643+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.644867+0000) 2019-10-13T08:24:01.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:01.644+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.645323+0000) 2019-10-13T08:24:01.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:01.643+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.645162+0000) 2019-10-13T08:24:01.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:01.644+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.645673+0000) 2019-10-13T08:24:01.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:01.644+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.645580+0000) 2019-10-13T08:24:01.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:01.644+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.645986+0000) 2019-10-13T08:24:01.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:01.647+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.648309+0000) 2019-10-13T08:24:01.651 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:01.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142751 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:02.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:01.999+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:02.000181+0000) 2019-10-13T08:24:02.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:02.808+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:02.809361+0000) 2019-10-13T08:24:02.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:02.858+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:02.859049+0000) 2019-10-13T08:24:02.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:02.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:02.859445+0000) 2019-10-13T08:24:03.612 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:24:03.613 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:03.617 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:24:03.617 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:06.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:06.651+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.652583+0000) 2019-10-13T08:24:06.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:06.651+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.652863+0000) 2019-10-13T08:24:06.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:06.652+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.652957+0000) 2019-10-13T08:24:06.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:06.651+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.653223+0000) 2019-10-13T08:24:06.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:06.652+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.653517+0000) 2019-10-13T08:24:06.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:06.652+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.653718+0000) 2019-10-13T08:24:06.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:06.652+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.653972+0000) 2019-10-13T08:24:06.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:06.654+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.655628+0000) 2019-10-13T08:24:06.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:06.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:07.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:07.861+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:07.862145+0000) 2019-10-13T08:24:08.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:08.409+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.410639+0000) 2019-10-13T08:24:08.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:08.747+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.748471+0000) 2019-10-13T08:24:08.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:08.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:08.748471+0000) 2019-10-13T08:24:11.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:11.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.659788+0000) 2019-10-13T08:24:11.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:11.658+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.660021+0000) 2019-10-13T08:24:11.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:11.659+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.660345+0000) 2019-10-13T08:24:11.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:11.658+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.660210+0000) 2019-10-13T08:24:11.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:11.659+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.660542+0000) 2019-10-13T08:24:11.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:11.659+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.660822+0000) 2019-10-13T08:24:11.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:11.659+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.660929+0000) 2019-10-13T08:24:11.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:11.661+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.662687+0000) 2019-10-13T08:24:11.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:11.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:12.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:11.999+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:12.000350+0000) 2019-10-13T08:24:12.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:12.810+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:12.811596+0000) 2019-10-13T08:24:12.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:12.858+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:12.859193+0000) 2019-10-13T08:24:12.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:12.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:12.859586+0000) 2019-10-13T08:24:16.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:16.665+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.667102+0000) 2019-10-13T08:24:16.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:16.665+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.667300+0000) 2019-10-13T08:24:16.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:16.666+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.667686+0000) 2019-10-13T08:24:16.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:16.666+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.667681+0000) 2019-10-13T08:24:16.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:16.666+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.667908+0000) 2019-10-13T08:24:16.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:16.666+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.668321+0000) 2019-10-13T08:24:16.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:16.667+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.668528+0000) 2019-10-13T08:24:16.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:16.668+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.669431+0000) 2019-10-13T08:24:16.673 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:16.671+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:17.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:17.861+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:17.862347+0000) 2019-10-13T08:24:18.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:18.410+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.410827+0000) 2019-10-13T08:24:18.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:18.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.748655+0000) 2019-10-13T08:24:18.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:18.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:18.748656+0000) 2019-10-13T08:24:21.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:21.673+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.674942+0000) 2019-10-13T08:24:21.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:21.674+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.675485+0000) 2019-10-13T08:24:21.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:21.674+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.675589+0000) 2019-10-13T08:24:21.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:21.674+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.675710+0000) 2019-10-13T08:24:21.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:21.674+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.676008+0000) 2019-10-13T08:24:21.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:21.674+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.676063+0000) 2019-10-13T08:24:21.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:21.675+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.676733+0000) 2019-10-13T08:24:21.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:21.677+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.677983+0000) 2019-10-13T08:24:21.682 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:21.680+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:22.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:22.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:22.000516+0000) 2019-10-13T08:24:22.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:22.813+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:22.813643+0000) 2019-10-13T08:24:22.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:22.859+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:22.859312+0000) 2019-10-13T08:24:22.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:22.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:22.859682+0000) 2019-10-13T08:24:26.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:26.683+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.683915+0000) 2019-10-13T08:24:26.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:26.683+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.684091+0000) 2019-10-13T08:24:26.685 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:26.683+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.684248+0000) 2019-10-13T08:24:26.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:26.684+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.684521+0000) 2019-10-13T08:24:26.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:26.683+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.684710+0000) 2019-10-13T08:24:26.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:26.684+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.684806+0000) 2019-10-13T08:24:26.686 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:26.684+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.685103+0000) 2019-10-13T08:24:26.687 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:26.686+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.686928+0000) 2019-10-13T08:24:26.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:26.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142906 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:27.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:27.862+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:27.862552+0000) 2019-10-13T08:24:28.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:28.410+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.411069+0000) 2019-10-13T08:24:28.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:28.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.748867+0000) 2019-10-13T08:24:28.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:28.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:28.748866+0000) 2019-10-13T08:24:31.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:31.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.692138+0000) 2019-10-13T08:24:31.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:31.691+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.692352+0000) 2019-10-13T08:24:31.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:31.692+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.692590+0000) 2019-10-13T08:24:31.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:31.692+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.692812+0000) 2019-10-13T08:24:31.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:31.691+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.692883+0000) 2019-10-13T08:24:31.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:31.692+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.693252+0000) 2019-10-13T08:24:31.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:31.693+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.693467+0000) 2019-10-13T08:24:31.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:31.694+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.695884+0000) 2019-10-13T08:24:31.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:31.697+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142936 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:32.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:32.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:32.000721+0000) 2019-10-13T08:24:32.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:32.814+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:32.815944+0000) 2019-10-13T08:24:32.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:32.858+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:32.859436+0000) 2019-10-13T08:24:32.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:32.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:32.859774+0000) 2019-10-13T08:24:33.793 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:24:33.793 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:33.797 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:24:33.797 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:36.701 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:36.699+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.699405+0000) 2019-10-13T08:24:36.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:36.699+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.699505+0000) 2019-10-13T08:24:36.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:36.699+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.699780+0000) 2019-10-13T08:24:36.701 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:36.699+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.699951+0000) 2019-10-13T08:24:36.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:36.699+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.700183+0000) 2019-10-13T08:24:36.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:36.700+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.700662+0000) 2019-10-13T08:24:36.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:36.698+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.700421+0000) 2019-10-13T08:24:36.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:36.700+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.702362+0000) 2019-10-13T08:24:36.704 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:36.703+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:37.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:37.862+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:37.862741+0000) 2019-10-13T08:24:38.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:38.411+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.411316+0000) 2019-10-13T08:24:38.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:38.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.749053+0000) 2019-10-13T08:24:38.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:38.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:38.749053+0000) 2019-10-13T08:24:41.706 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:41.705+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.705947+0000) 2019-10-13T08:24:41.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:41.704+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.706645+0000) 2019-10-13T08:24:41.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:41.705+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.706337+0000) 2019-10-13T08:24:41.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:41.706+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.706499+0000) 2019-10-13T08:24:41.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:41.706+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.706597+0000) 2019-10-13T08:24:41.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:41.706+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.706863+0000) 2019-10-13T08:24:41.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:41.706+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.707073+0000) 2019-10-13T08:24:41.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:41.707+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.708849+0000) 2019-10-13T08:24:41.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:41.709+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 142999 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:42.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:42.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:42.000888+0000) 2019-10-13T08:24:42.818 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:42.817+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:42.818258+0000) 2019-10-13T08:24:42.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:42.858+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:42.859575+0000) 2019-10-13T08:24:42.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:42.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:42.859894+0000) 2019-10-13T08:24:46.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:46.710+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.711975+0000) 2019-10-13T08:24:46.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:46.710+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.712231+0000) 2019-10-13T08:24:46.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:46.711+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.712427+0000) 2019-10-13T08:24:46.713 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:46.711+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.712631+0000) 2019-10-13T08:24:46.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:46.711+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.712728+0000) 2019-10-13T08:24:46.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:46.712+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.712895+0000) 2019-10-13T08:24:46.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:46.711+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.713115+0000) 2019-10-13T08:24:46.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:46.715+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.715816+0000) 2019-10-13T08:24:46.718 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:46.717+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143030 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:47.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:47.861+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:47.862957+0000) 2019-10-13T08:24:48.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:48.410+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.411505+0000) 2019-10-13T08:24:48.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:48.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.749235+0000) 2019-10-13T08:24:48.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:48.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:48.749236+0000) 2019-10-13T08:24:51.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:51.718+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.719656+0000) 2019-10-13T08:24:51.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:51.718+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.720030+0000) 2019-10-13T08:24:51.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:51.719+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.719765+0000) 2019-10-13T08:24:51.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:51.718+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.720382+0000) 2019-10-13T08:24:51.722 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:51.719+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.720415+0000) 2019-10-13T08:24:51.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:51.719+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.720597+0000) 2019-10-13T08:24:51.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:51.719+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.720724+0000) 2019-10-13T08:24:51.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:51.721+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.721968+0000) 2019-10-13T08:24:51.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:51.723+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:52.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:52.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:52.001048+0000) 2019-10-13T08:24:52.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:52.820+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:52.820652+0000) 2019-10-13T08:24:52.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:52.859+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:52.859766+0000) 2019-10-13T08:24:52.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:52.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:52.859995+0000) 2019-10-13T08:24:56.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:24:56.725+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.726212+0000) 2019-10-13T08:24:56.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:56.725+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.726488+0000) 2019-10-13T08:24:56.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:56.725+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.726729+0000) 2019-10-13T08:24:56.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:24:56.725+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.727076+0000) 2019-10-13T08:24:56.728 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:56.725+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.727372+0000) 2019-10-13T08:24:56.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:56.726+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.727604+0000) 2019-10-13T08:24:56.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:24:56.726+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.727709+0000) 2019-10-13T08:24:56.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:24:56.728+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.729412+0000) 2019-10-13T08:24:56.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:24:56.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:24:57.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:24:57.862+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:57.863161+0000) 2019-10-13T08:24:58.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:24:58.411+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.411709+0000) 2019-10-13T08:24:58.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:24:58.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.749436+0000) 2019-10-13T08:24:58.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:24:58.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:58.749436+0000) 2019-10-13T08:25:01.734 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:01.732+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.733446+0000) 2019-10-13T08:25:01.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:01.732+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.733809+0000) 2019-10-13T08:25:01.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:01.732+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.733990+0000) 2019-10-13T08:25:01.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:01.733+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.734500+0000) 2019-10-13T08:25:01.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:01.733+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.734422+0000) 2019-10-13T08:25:01.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:01.733+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.734786+0000) 2019-10-13T08:25:01.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:01.733+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.735050+0000) 2019-10-13T08:25:01.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:01.735+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.736171+0000) 2019-10-13T08:25:01.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:01.737+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:02.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:02.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:02.001210+0000) 2019-10-13T08:25:02.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:02.822+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:02.822917+0000) 2019-10-13T08:25:02.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:02.858+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:02.860109+0000) 2019-10-13T08:25:02.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:02.859+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:02.859904+0000) 2019-10-13T08:25:03.938 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:25:03.938 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:03.946 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:25:03.947 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:06.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:06.739+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.740778+0000) 2019-10-13T08:25:06.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:06.739+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.741041+0000) 2019-10-13T08:25:06.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:06.739+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.741282+0000) 2019-10-13T08:25:06.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:06.740+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.741457+0000) 2019-10-13T08:25:06.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:06.740+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.741754+0000) 2019-10-13T08:25:06.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:06.740+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.741609+0000) 2019-10-13T08:25:06.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:06.742+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.743307+0000) 2019-10-13T08:25:06.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:06.740+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.742227+0000) 2019-10-13T08:25:06.747 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:06.744+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143154 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:07.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:07.862+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:07.863356+0000) 2019-10-13T08:25:08.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:08.411+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.411884+0000) 2019-10-13T08:25:08.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:08.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.749647+0000) 2019-10-13T08:25:08.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:08.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:08.749647+0000) 2019-10-13T08:25:11.749 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:11.747+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.747873+0000) 2019-10-13T08:25:11.749 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:11.747+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.748098+0000) 2019-10-13T08:25:11.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:11.747+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.748327+0000) 2019-10-13T08:25:11.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:11.748+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.748677+0000) 2019-10-13T08:25:11.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:11.747+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.748470+0000) 2019-10-13T08:25:11.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:11.748+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.749260+0000) 2019-10-13T08:25:11.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:11.749+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.749393+0000) 2019-10-13T08:25:11.751 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:11.750+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.750989+0000) 2019-10-13T08:25:11.754 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:11.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:12.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:12.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:12.001370+0000) 2019-10-13T08:25:12.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:12.824+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:12.825261+0000) 2019-10-13T08:25:12.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:12.859+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:12.860053+0000) 2019-10-13T08:25:12.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:12.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:12.860289+0000) 2019-10-13T08:25:16.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:16.755+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.755850+0000) 2019-10-13T08:25:16.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:16.755+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.756006+0000) 2019-10-13T08:25:16.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:16.755+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.756515+0000) 2019-10-13T08:25:16.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:16.756+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.757019+0000) 2019-10-13T08:25:16.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:16.756+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.757183+0000) 2019-10-13T08:25:16.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:16.756+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.757382+0000) 2019-10-13T08:25:16.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:16.757+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.757536+0000) 2019-10-13T08:25:16.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:16.757+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.758474+0000) 2019-10-13T08:25:16.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:16.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:17.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:17.863+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:17.863598+0000) 2019-10-13T08:25:18.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:18.411+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.412081+0000) 2019-10-13T08:25:18.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:18.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.749855+0000) 2019-10-13T08:25:18.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:18.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:18.749854+0000) 2019-10-13T08:25:21.764 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:21.762+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.763288+0000) 2019-10-13T08:25:21.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:21.763+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.763565+0000) 2019-10-13T08:25:21.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:21.763+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.763797+0000) 2019-10-13T08:25:21.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:21.763+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.764022+0000) 2019-10-13T08:25:21.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:21.763+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.764357+0000) 2019-10-13T08:25:21.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:21.764+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.764777+0000) 2019-10-13T08:25:21.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:21.764+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.765054+0000) 2019-10-13T08:25:21.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:21.765+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.766647+0000) 2019-10-13T08:25:21.769 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:21.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:22.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:22.001+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:22.001551+0000) 2019-10-13T08:25:22.828 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:22.827+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:22.827675+0000) 2019-10-13T08:25:22.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:22.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:22.860176+0000) 2019-10-13T08:25:22.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:22.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:22.860415+0000) 2019-10-13T08:25:26.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:26.769+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.771105+0000) 2019-10-13T08:25:26.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:26.771+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.771663+0000) 2019-10-13T08:25:26.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:26.771+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.771856+0000) 2019-10-13T08:25:26.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:26.771+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.772110+0000) 2019-10-13T08:25:26.773 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:26.771+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.772299+0000) 2019-10-13T08:25:26.773 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:26.772+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.772581+0000) 2019-10-13T08:25:26.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:26.773+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.773491+0000) 2019-10-13T08:25:26.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:26.773+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.774980+0000) 2019-10-13T08:25:26.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:26.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:27.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:27.863+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:27.863762+0000) 2019-10-13T08:25:28.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:28.412+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.412280+0000) 2019-10-13T08:25:28.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:28.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.750076+0000) 2019-10-13T08:25:28.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:28.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:28.750076+0000) 2019-10-13T08:25:31.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:31.777+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.778961+0000) 2019-10-13T08:25:31.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:31.778+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.779468+0000) 2019-10-13T08:25:31.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:31.778+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.779933+0000) 2019-10-13T08:25:31.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:31.778+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.780050+0000) 2019-10-13T08:25:31.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:31.779+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.780529+0000) 2019-10-13T08:25:31.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:31.779+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.780678+0000) 2019-10-13T08:25:31.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:31.779+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.780856+0000) 2019-10-13T08:25:31.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:31.780+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.782345+0000) 2019-10-13T08:25:31.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:31.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:32.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:32.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:32.001711+0000) 2019-10-13T08:25:32.830 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:32.829+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:32.829972+0000) 2019-10-13T08:25:32.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:32.859+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:32.860326+0000) 2019-10-13T08:25:32.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:32.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:32.860579+0000) 2019-10-13T08:25:34.100 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:25:34.100 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:34.105 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:25:34.106 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:36.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:36.785+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.787104+0000) 2019-10-13T08:25:36.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:36.785+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.787335+0000) 2019-10-13T08:25:36.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:36.786+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.787582+0000) 2019-10-13T08:25:36.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:36.785+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.787198+0000) 2019-10-13T08:25:36.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:36.786+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.787792+0000) 2019-10-13T08:25:36.790 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:36.786+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.787969+0000) 2019-10-13T08:25:36.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:36.786+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.788094+0000) 2019-10-13T08:25:36.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:36.787+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.789662+0000) 2019-10-13T08:25:36.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:36.790+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143339 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:37.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:37.863+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:37.863959+0000) 2019-10-13T08:25:38.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:38.411+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.412450+0000) 2019-10-13T08:25:38.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:38.748+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.750285+0000) 2019-10-13T08:25:38.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:38.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:38.750285+0000) 2019-10-13T08:25:41.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:41.793+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.794082+0000) 2019-10-13T08:25:41.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:41.793+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.794401+0000) 2019-10-13T08:25:41.796 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:41.793+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.794526+0000) 2019-10-13T08:25:41.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:41.793+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.794726+0000) 2019-10-13T08:25:41.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:41.793+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.794840+0000) 2019-10-13T08:25:41.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:41.793+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.795094+0000) 2019-10-13T08:25:41.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:41.793+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.795345+0000) 2019-10-13T08:25:41.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:41.794+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.795395+0000) 2019-10-13T08:25:41.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:41.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:42.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:42.000+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:42.001878+0000) 2019-10-13T08:25:42.833 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:42.831+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:42.832352+0000) 2019-10-13T08:25:42.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:42.859+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:42.860469+0000) 2019-10-13T08:25:42.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:42.859+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:42.860725+0000) 2019-10-13T08:25:46.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:46.800+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.800882+0000) 2019-10-13T08:25:46.806 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:46.802+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.803698+0000) 2019-10-13T08:25:46.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:46.799+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.800820+0000) 2019-10-13T08:25:46.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:46.799+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.801130+0000) 2019-10-13T08:25:46.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:46.799+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.801314+0000) 2019-10-13T08:25:46.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:46.800+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.801450+0000) 2019-10-13T08:25:46.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:46.800+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.801620+0000) 2019-10-13T08:25:46.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:46.800+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.801987+0000) 2019-10-13T08:25:46.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:46.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143402 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:47.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:47.863+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:47.864163+0000) 2019-10-13T08:25:48.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:48.411+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.412608+0000) 2019-10-13T08:25:48.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:48.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.750445+0000) 2019-10-13T08:25:48.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:48.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:48.750445+0000) 2019-10-13T08:25:51.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:51.807+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.808107+0000) 2019-10-13T08:25:51.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:51.806+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.808344+0000) 2019-10-13T08:25:51.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:51.807+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.808758+0000) 2019-10-13T08:25:51.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:51.807+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.809039+0000) 2019-10-13T08:25:51.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:51.807+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.809218+0000) 2019-10-13T08:25:51.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:51.807+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.809388+0000) 2019-10-13T08:25:51.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:51.807+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.808894+0000) 2019-10-13T08:25:51.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:51.809+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.810514+0000) 2019-10-13T08:25:51.814 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:51.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143432 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:52.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:52.001+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:52.002018+0000) 2019-10-13T08:25:52.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:52.834+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:52.834634+0000) 2019-10-13T08:25:52.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:52.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:52.860600+0000) 2019-10-13T08:25:52.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:52.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:52.860873+0000) 2019-10-13T08:25:56.817 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:25:56.814+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.816066+0000) 2019-10-13T08:25:56.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:56.814+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.816248+0000) 2019-10-13T08:25:56.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:56.815+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.816618+0000) 2019-10-13T08:25:56.818 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:56.815+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.816676+0000) 2019-10-13T08:25:56.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:25:56.815+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.816826+0000) 2019-10-13T08:25:56.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:56.815+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.817089+0000) 2019-10-13T08:25:56.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:25:56.815+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.816376+0000) 2019-10-13T08:25:56.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:25:56.817+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.818578+0000) 2019-10-13T08:25:56.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:25:56.820+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:25:57.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:25:57.863+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:57.864371+0000) 2019-10-13T08:25:58.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:25:58.412+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.412765+0000) 2019-10-13T08:25:58.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:25:58.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.750634+0000) 2019-10-13T08:25:58.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:25:58.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:58.750634+0000) 2019-10-13T08:26:01.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:01.823+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.823805+0000) 2019-10-13T08:26:01.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:01.823+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.824020+0000) 2019-10-13T08:26:01.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:01.823+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.824179+0000) 2019-10-13T08:26:01.825 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:01.823+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.824317+0000) 2019-10-13T08:26:01.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:01.824+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.824508+0000) 2019-10-13T08:26:01.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:01.824+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.824664+0000) 2019-10-13T08:26:01.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:01.824+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.824735+0000) 2019-10-13T08:26:01.827 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:01.826+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.826950+0000) 2019-10-13T08:26:01.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:01.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143495 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:02.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:02.001+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:02.002149+0000) 2019-10-13T08:26:02.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:02.836+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:02.836861+0000) 2019-10-13T08:26:02.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:02.859+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:02.860698+0000) 2019-10-13T08:26:02.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:02.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:02.861054+0000) 2019-10-13T08:26:04.255 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:26:04.255 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:04.259 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:26:04.259 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:06.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:06.831+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.831770+0000) 2019-10-13T08:26:06.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:06.831+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.832004+0000) 2019-10-13T08:26:06.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:06.831+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.832195+0000) 2019-10-13T08:26:06.833 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:06.832+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.832394+0000) 2019-10-13T08:26:06.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:06.832+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.832607+0000) 2019-10-13T08:26:06.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:06.832+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.832660+0000) 2019-10-13T08:26:06.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:06.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.832928+0000) 2019-10-13T08:26:06.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:06.832+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.833490+0000) 2019-10-13T08:26:06.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:06.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:07.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:07.864+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:07.864567+0000) 2019-10-13T08:26:08.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:08.412+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.412925+0000) 2019-10-13T08:26:08.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:08.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.750795+0000) 2019-10-13T08:26:08.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:08.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:08.750795+0000) 2019-10-13T08:26:11.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:11.838+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.839053+0000) 2019-10-13T08:26:11.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:11.839+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.839506+0000) 2019-10-13T08:26:11.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:11.839+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.839624+0000) 2019-10-13T08:26:11.841 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:11.839+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.839752+0000) 2019-10-13T08:26:11.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:11.839+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.840017+0000) 2019-10-13T08:26:11.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:11.839+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.840301+0000) 2019-10-13T08:26:11.841 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:11.840+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.840449+0000) 2019-10-13T08:26:11.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:11.841+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.842355+0000) 2019-10-13T08:26:11.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:11.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:12.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:12.002+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:12.002309+0000) 2019-10-13T08:26:12.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:12.838+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:12.838983+0000) 2019-10-13T08:26:12.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:12.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:12.860791+0000) 2019-10-13T08:26:12.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:12.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:12.861185+0000) 2019-10-13T08:26:16.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:16.845+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.847099+0000) 2019-10-13T08:26:16.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:16.845+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.846797+0000) 2019-10-13T08:26:16.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:16.845+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.846924+0000) 2019-10-13T08:26:16.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:16.845+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.847291+0000) 2019-10-13T08:26:16.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:16.846+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.847605+0000) 2019-10-13T08:26:16.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:16.846+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.847957+0000) 2019-10-13T08:26:16.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:16.846+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.848026+0000) 2019-10-13T08:26:16.850 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:16.848+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.849886+0000) 2019-10-13T08:26:16.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:16.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:17.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:17.863+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:17.864763+0000) 2019-10-13T08:26:18.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:18.412+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.413115+0000) 2019-10-13T08:26:18.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:18.749+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.750978+0000) 2019-10-13T08:26:18.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:18.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:18.750978+0000) 2019-10-13T08:26:21.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:21.852+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.853937+0000) 2019-10-13T08:26:21.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:21.852+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.853900+0000) 2019-10-13T08:26:21.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:21.852+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.854089+0000) 2019-10-13T08:26:21.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:21.853+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.854662+0000) 2019-10-13T08:26:21.857 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:21.853+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.854423+0000) 2019-10-13T08:26:21.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:21.853+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.854815+0000) 2019-10-13T08:26:21.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:21.853+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.855165+0000) 2019-10-13T08:26:21.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:21.854+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.856050+0000) 2019-10-13T08:26:21.860 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:21.858+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:22.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:22.001+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:22.002479+0000) 2019-10-13T08:26:22.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:22.839+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:22.841210+0000) 2019-10-13T08:26:22.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:22.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:22.860893+0000) 2019-10-13T08:26:22.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:22.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:22.861285+0000) 2019-10-13T08:26:26.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:26.860+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.862179+0000) 2019-10-13T08:26:26.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:26.861+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.862449+0000) 2019-10-13T08:26:26.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:26.861+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.862830+0000) 2019-10-13T08:26:26.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:26.861+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.863088+0000) 2019-10-13T08:26:26.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:26.861+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.863261+0000) 2019-10-13T08:26:26.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:26.862+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.863477+0000) 2019-10-13T08:26:26.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:26.860+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.862663+0000) 2019-10-13T08:26:26.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:26.862+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.864623+0000) 2019-10-13T08:26:26.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:26.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:27.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:27.864+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:27.864945+0000) 2019-10-13T08:26:28.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:28.412+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.413307+0000) 2019-10-13T08:26:28.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:28.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.751216+0000) 2019-10-13T08:26:28.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:28.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:28.751216+0000) 2019-10-13T08:26:31.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:31.868+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.869182+0000) 2019-10-13T08:26:31.871 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:31.868+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.869403+0000) 2019-10-13T08:26:31.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:31.868+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.869730+0000) 2019-10-13T08:26:31.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:31.868+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.869764+0000) 2019-10-13T08:26:31.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:31.868+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.869928+0000) 2019-10-13T08:26:31.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:31.868+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.869987+0000) 2019-10-13T08:26:31.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:31.868+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.870259+0000) 2019-10-13T08:26:31.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:31.870+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.871076+0000) 2019-10-13T08:26:31.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:31.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:32.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:32.001+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:32.002622+0000) 2019-10-13T08:26:32.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:32.842+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:32.843509+0000) 2019-10-13T08:26:32.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:32.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:32.860994+0000) 2019-10-13T08:26:32.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:32.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:32.861418+0000) 2019-10-13T08:26:34.409 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:26:34.410 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:34.414 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:26:34.414 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:36.877 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:36.875+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.876829+0000) 2019-10-13T08:26:36.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:36.877+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.877754+0000) 2019-10-13T08:26:36.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:36.875+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.877235+0000) 2019-10-13T08:26:36.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:36.876+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.877447+0000) 2019-10-13T08:26:36.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:36.876+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.877993+0000) 2019-10-13T08:26:36.880 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:36.876+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.878148+0000) 2019-10-13T08:26:36.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:36.877+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.878495+0000) 2019-10-13T08:26:36.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:36.878+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.879698+0000) 2019-10-13T08:26:36.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:36.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:37.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:37.864+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:37.865155+0000) 2019-10-13T08:26:38.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:38.412+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.413494+0000) 2019-10-13T08:26:38.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:38.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.751439+0000) 2019-10-13T08:26:38.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:38.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:38.751439+0000) 2019-10-13T08:26:41.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:41.883+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.884589+0000) 2019-10-13T08:26:41.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:41.884+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.885276+0000) 2019-10-13T08:26:41.886 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:41.883+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.885134+0000) 2019-10-13T08:26:41.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:41.884+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.885529+0000) 2019-10-13T08:26:41.886 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:41.884+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.885626+0000) 2019-10-13T08:26:41.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:41.884+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.886181+0000) 2019-10-13T08:26:41.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:41.884+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.886083+0000) 2019-10-13T08:26:41.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:41.887+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.888011+0000) 2019-10-13T08:26:41.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:41.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:42.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:42.002+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:42.002749+0000) 2019-10-13T08:26:42.846 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:42.845+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:42.846097+0000) 2019-10-13T08:26:42.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:42.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:42.861125+0000) 2019-10-13T08:26:42.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:42.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:42.861577+0000) 2019-10-13T08:26:46.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:46.892+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.892769+0000) 2019-10-13T08:26:46.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:46.892+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.893025+0000) 2019-10-13T08:26:46.895 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:46.892+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.893163+0000) 2019-10-13T08:26:46.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:46.892+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.893071+0000) 2019-10-13T08:26:46.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:46.893+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.893567+0000) 2019-10-13T08:26:46.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:46.893+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.893703+0000) 2019-10-13T08:26:46.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:46.892+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.893325+0000) 2019-10-13T08:26:46.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:46.895+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.896146+0000) 2019-10-13T08:26:46.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:46.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143774 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:47.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:47.864+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:47.865299+0000) 2019-10-13T08:26:48.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:48.413+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.413657+0000) 2019-10-13T08:26:48.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:48.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.751641+0000) 2019-10-13T08:26:48.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:48.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:48.751641+0000) 2019-10-13T08:26:51.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:51.899+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.900042+0000) 2019-10-13T08:26:51.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:51.899+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.900347+0000) 2019-10-13T08:26:51.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:51.899+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.900645+0000) 2019-10-13T08:26:51.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:51.900+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.900799+0000) 2019-10-13T08:26:51.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:51.900+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.900917+0000) 2019-10-13T08:26:51.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:51.900+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.901109+0000) 2019-10-13T08:26:51.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:51.900+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.901233+0000) 2019-10-13T08:26:51.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:51.901+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.902277+0000) 2019-10-13T08:26:51.907 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:51.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:52.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:52.002+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:52.002915+0000) 2019-10-13T08:26:52.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:52.848+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:52.848353+0000) 2019-10-13T08:26:52.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:52.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:52.861229+0000) 2019-10-13T08:26:52.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:52.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:52.861712+0000) 2019-10-13T08:26:56.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:56.907+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.908037+0000) 2019-10-13T08:26:56.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:56.907+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.908193+0000) 2019-10-13T08:26:56.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:56.908+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.908444+0000) 2019-10-13T08:26:56.909 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:56.908+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.908689+0000) 2019-10-13T08:26:56.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:26:56.907+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.908493+0000) 2019-10-13T08:26:56.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:26:56.908+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.908909+0000) 2019-10-13T08:26:56.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:26:56.909+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.909420+0000) 2019-10-13T08:26:56.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:26:56.910+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.911214+0000) 2019-10-13T08:26:56.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:26:56.912+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:26:57.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:26:57.865+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.865474+0000) 2019-10-13T08:26:58.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:26:58.413+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.413836+0000) 2019-10-13T08:26:58.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:26:58.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.751835+0000) 2019-10-13T08:26:58.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:26:58.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:58.751835+0000) 2019-10-13T08:27:01.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:01.914+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.915029+0000) 2019-10-13T08:27:01.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:01.913+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.914753+0000) 2019-10-13T08:27:01.918 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:01.914+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.915449+0000) 2019-10-13T08:27:01.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:01.914+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.915628+0000) 2019-10-13T08:27:01.919 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:01.914+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.915978+0000) 2019-10-13T08:27:01.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:01.914+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.916080+0000) 2019-10-13T08:27:01.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:01.914+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.916200+0000) 2019-10-13T08:27:01.920 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:01.917+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.917929+0000) 2019-10-13T08:27:01.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:01.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:02.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:02.002+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.003083+0000) 2019-10-13T08:27:02.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:02.850+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.850686+0000) 2019-10-13T08:27:02.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:02.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.861816+0000) 2019-10-13T08:27:02.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:02.861+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.861365+0000) 2019-10-13T08:27:04.573 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:27:04.573 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:04.580 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:27:04.581 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:06.923 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:06.921+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.922744+0000) 2019-10-13T08:27:06.923 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:06.921+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.923055+0000) 2019-10-13T08:27:06.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:06.921+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.923138+0000) 2019-10-13T08:27:06.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:06.922+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.923310+0000) 2019-10-13T08:27:06.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:06.922+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.923636+0000) 2019-10-13T08:27:06.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:06.922+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.923918+0000) 2019-10-13T08:27:06.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:06.922+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.924005+0000) 2019-10-13T08:27:06.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:06.925+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:06.926279+0000) 2019-10-13T08:27:06.929 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:06.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:07.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:07.864+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.865612+0000) 2019-10-13T08:27:08.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:08.412+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.414030+0000) 2019-10-13T08:27:08.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:08.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.752023+0000) 2019-10-13T08:27:08.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:08.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:08.752023+0000) 2019-10-13T08:27:11.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:11.928+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.930357+0000) 2019-10-13T08:27:11.932 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:11.929+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.930470+0000) 2019-10-13T08:27:11.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:11.929+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.930967+0000) 2019-10-13T08:27:11.933 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:11.929+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.931290+0000) 2019-10-13T08:27:11.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:11.929+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.931157+0000) 2019-10-13T08:27:11.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:11.931+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.932899+0000) 2019-10-13T08:27:11.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:11.930+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.931564+0000) 2019-10-13T08:27:11.935 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:11.930+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.931913+0000) 2019-10-13T08:27:11.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:11.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:12.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:12.002+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.003236+0000) 2019-10-13T08:27:12.853 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:12.852+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.852960+0000) 2019-10-13T08:27:12.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:12.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.861458+0000) 2019-10-13T08:27:12.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:12.860+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.861972+0000) 2019-10-13T08:27:16.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:16.936+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.938019+0000) 2019-10-13T08:27:16.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:16.937+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.938620+0000) 2019-10-13T08:27:16.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:16.937+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.938802+0000) 2019-10-13T08:27:16.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:16.937+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.939214+0000) 2019-10-13T08:27:16.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:16.937+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.939359+0000) 2019-10-13T08:27:16.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:16.938+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.939500+0000) 2019-10-13T08:27:16.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:16.938+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.939825+0000) 2019-10-13T08:27:16.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:16.939+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:16.941064+0000) 2019-10-13T08:27:16.946 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:16.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:17.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:17.865+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.865788+0000) 2019-10-13T08:27:18.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:18.413+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.414271+0000) 2019-10-13T08:27:18.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:18.750+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.752226+0000) 2019-10-13T08:27:18.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:18.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:18.752226+0000) 2019-10-13T08:27:21.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:21.944+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.945717+0000) 2019-10-13T08:27:21.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:21.944+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.946024+0000) 2019-10-13T08:27:21.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:21.944+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.946356+0000) 2019-10-13T08:27:21.947 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:21.945+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.946635+0000) 2019-10-13T08:27:21.947 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:21.945+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.947098+0000) 2019-10-13T08:27:21.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:21.945+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.946914+0000) 2019-10-13T08:27:21.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:21.945+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.947332+0000) 2019-10-13T08:27:21.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:21.947+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.948565+0000) 2019-10-13T08:27:21.952 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:21.950+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 143991 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:22.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:22.002+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.003347+0000) 2019-10-13T08:27:22.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:22.854+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.855415+0000) 2019-10-13T08:27:22.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:22.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.861543+0000) 2019-10-13T08:27:22.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:22.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.862097+0000) 2019-10-13T08:27:26.955 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:26.952+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.954040+0000) 2019-10-13T08:27:26.955 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:26.952+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.954145+0000) 2019-10-13T08:27:26.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:26.952+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.954298+0000) 2019-10-13T08:27:26.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:26.954+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.955084+0000) 2019-10-13T08:27:26.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:26.953+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.954897+0000) 2019-10-13T08:27:26.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:26.953+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.955385+0000) 2019-10-13T08:27:26.956 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:26.954+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.955731+0000) 2019-10-13T08:27:26.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:26.956+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:26.957149+0000) 2019-10-13T08:27:26.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:26.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:27.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:27.865+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.865968+0000) 2019-10-13T08:27:28.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:28.414+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.414491+0000) 2019-10-13T08:27:28.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:28.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.752426+0000) 2019-10-13T08:27:28.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:28.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:28.752426+0000) 2019-10-13T08:27:31.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:31.961+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.961936+0000) 2019-10-13T08:27:31.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:31.962+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.962441+0000) 2019-10-13T08:27:31.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:31.962+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.962726+0000) 2019-10-13T08:27:31.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:31.962+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.962604+0000) 2019-10-13T08:27:31.964 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:31.962+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.963088+0000) 2019-10-13T08:27:31.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:31.962+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.963313+0000) 2019-10-13T08:27:31.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:31.962+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.963500+0000) 2019-10-13T08:27:31.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:31.963+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.964434+0000) 2019-10-13T08:27:31.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:31.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:32.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:32.003+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.003445+0000) 2019-10-13T08:27:32.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:32.857+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.857717+0000) 2019-10-13T08:27:32.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:32.860+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.861630+0000) 2019-10-13T08:27:32.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:32.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.862284+0000) 2019-10-13T08:27:34.735 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:27:34.735 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:34.740 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:27:34.740 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:36.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:36.969+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.969996+0000) 2019-10-13T08:27:36.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:36.969+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.970231+0000) 2019-10-13T08:27:36.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:36.970+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.970436+0000) 2019-10-13T08:27:36.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:36.970+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.970508+0000) 2019-10-13T08:27:36.972 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:36.970+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.970682+0000) 2019-10-13T08:27:36.972 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:36.970+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.970932+0000) 2019-10-13T08:27:36.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:36.969+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.970132+0000) 2019-10-13T08:27:36.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:36.971+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:36.972474+0000) 2019-10-13T08:27:36.974 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:36.973+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:37.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:37.865+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.866179+0000) 2019-10-13T08:27:38.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:38.414+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.414658+0000) 2019-10-13T08:27:38.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:38.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.752636+0000) 2019-10-13T08:27:38.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:38.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:38.752636+0000) 2019-10-13T08:27:41.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:41.974+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.975255+0000) 2019-10-13T08:27:41.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:41.975+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.975818+0000) 2019-10-13T08:27:41.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:41.975+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.975854+0000) 2019-10-13T08:27:41.979 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:41.975+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.976106+0000) 2019-10-13T08:27:41.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:41.975+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.976014+0000) 2019-10-13T08:27:41.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:41.975+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.976208+0000) 2019-10-13T08:27:41.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:41.976+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.976650+0000) 2019-10-13T08:27:41.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:41.977+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.977861+0000) 2019-10-13T08:27:41.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:41.979+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144115 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:42.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:42.003+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.003573+0000) 2019-10-13T08:27:42.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:42.859+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.860020+0000) 2019-10-13T08:27:42.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:42.861+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.861758+0000) 2019-10-13T08:27:42.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:42.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.862411+0000) 2019-10-13T08:27:46.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:46.981+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.981682+0000) 2019-10-13T08:27:46.982 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:46.981+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.981987+0000) 2019-10-13T08:27:46.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:46.981+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.982200+0000) 2019-10-13T08:27:46.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:46.982+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.982786+0000) 2019-10-13T08:27:46.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:46.982+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.982966+0000) 2019-10-13T08:27:46.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:46.982+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.983178+0000) 2019-10-13T08:27:46.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:46.982+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.983354+0000) 2019-10-13T08:27:46.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:46.987+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:46.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:46.985+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:46.986121+0000) 2019-10-13T08:27:47.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:47.865+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.866357+0000) 2019-10-13T08:27:48.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:48.413+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.414817+0000) 2019-10-13T08:27:48.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:48.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.752839+0000) 2019-10-13T08:27:48.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:48.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:48.752839+0000) 2019-10-13T08:27:51.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:51.988+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.989787+0000) 2019-10-13T08:27:51.991 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:51.988+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.990022+0000) 2019-10-13T08:27:51.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:51.988+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.990187+0000) 2019-10-13T08:27:51.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:51.989+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.990324+0000) 2019-10-13T08:27:51.992 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:51.989+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.990502+0000) 2019-10-13T08:27:51.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:51.989+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.990942+0000) 2019-10-13T08:27:51.992 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:51.989+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.991117+0000) 2019-10-13T08:27:51.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:51.991+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.991886+0000) 2019-10-13T08:27:51.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:51.994+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144176 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:52.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:52.002+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.003660+0000) 2019-10-13T08:27:52.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:52.861+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.861881+0000) 2019-10-13T08:27:52.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:52.861+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.862417+0000) 2019-10-13T08:27:52.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:52.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.862560+0000) 2019-10-13T08:27:56.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:56.996+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.997554+0000) 2019-10-13T08:27:56.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:27:56.996+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.997807+0000) 2019-10-13T08:27:56.998 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:56.996+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.998001+0000) 2019-10-13T08:27:56.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:27:56.997+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.998470+0000) 2019-10-13T08:27:56.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:56.996+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.998189+0000) 2019-10-13T08:27:57.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:56.997+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.998708+0000) 2019-10-13T08:27:57.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:27:56.997+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:56.999178+0000) 2019-10-13T08:27:57.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:27:56.999+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.000651+0000) 2019-10-13T08:27:57.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:27:57.001+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:27:57.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:27:57.865+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.866568+0000) 2019-10-13T08:27:58.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:27:58.414+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.414968+0000) 2019-10-13T08:27:58.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:27:58.751+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.753006+0000) 2019-10-13T08:27:58.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:27:58.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:58.753006+0000) 2019-10-13T08:28:02.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:02.003+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.005179+0000) 2019-10-13T08:28:02.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:02.004+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.005530+0000) 2019-10-13T08:28:02.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:02.004+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.005697+0000) 2019-10-13T08:28:02.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:02.004+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.005990+0000) 2019-10-13T08:28:02.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:02.004+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.006221+0000) 2019-10-13T08:28:02.007 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:02.005+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.006460+0000) 2019-10-13T08:28:02.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:02.004+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.006313+0000) 2019-10-13T08:28:02.008 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:02.004+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.006714+0000) 2019-10-13T08:28:02.011 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:02.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:02.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:02.861+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.862022+0000) 2019-10-13T08:28:02.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:02.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.862697+0000) 2019-10-13T08:28:02.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:02.863+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.864660+0000) 2019-10-13T08:28:04.890 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:28:04.890 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:04.896 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:28:04.896 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:07.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:07.011+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.012985+0000) 2019-10-13T08:28:07.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:07.011+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.013195+0000) 2019-10-13T08:28:07.015 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:07.011+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.013340+0000) 2019-10-13T08:28:07.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:07.012+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.013451+0000) 2019-10-13T08:28:07.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:07.012+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.013793+0000) 2019-10-13T08:28:07.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:07.012+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.014095+0000) 2019-10-13T08:28:07.016 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:07.012+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.014291+0000) 2019-10-13T08:28:07.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:07.014+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.015985+0000) 2019-10-13T08:28:07.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:07.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:07.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:07.866+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.866716+0000) 2019-10-13T08:28:08.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:08.414+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.415127+0000) 2019-10-13T08:28:08.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:08.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.753180+0000) 2019-10-13T08:28:08.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:08.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.753180+0000) 2019-10-13T08:28:08.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:08.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:08.754340+0000) 2019-10-13T08:28:12.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:12.018+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.020264+0000) 2019-10-13T08:28:12.021 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:12.019+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.020533+0000) 2019-10-13T08:28:12.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:12.019+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.021240+0000) 2019-10-13T08:28:12.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:12.019+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.020890+0000) 2019-10-13T08:28:12.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:12.019+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.021020+0000) 2019-10-13T08:28:12.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:12.019+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.021050+0000) 2019-10-13T08:28:12.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:12.020+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.021444+0000) 2019-10-13T08:28:12.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:12.020+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.022480+0000) 2019-10-13T08:28:12.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:12.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:12.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:12.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.862803+0000) 2019-10-13T08:28:12.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:12.861+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.862186+0000) 2019-10-13T08:28:12.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:12.866+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.867030+0000) 2019-10-13T08:28:17.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:17.026+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.027078+0000) 2019-10-13T08:28:17.028 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:17.026+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.027266+0000) 2019-10-13T08:28:17.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:17.027+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.027548+0000) 2019-10-13T08:28:17.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:17.027+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.028103+0000) 2019-10-13T08:28:17.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:17.027+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.028285+0000) 2019-10-13T08:28:17.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:17.028+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.028462+0000) 2019-10-13T08:28:17.030 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:17.028+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.028633+0000) 2019-10-13T08:28:17.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:17.029+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.029994+0000) 2019-10-13T08:28:17.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:17.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:17.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:17.866+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.866867+0000) 2019-10-13T08:28:18.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:18.414+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.415300+0000) 2019-10-13T08:28:18.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:18.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.753390+0000) 2019-10-13T08:28:18.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:18.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.753390+0000) 2019-10-13T08:28:18.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:18.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:18.754465+0000) 2019-10-13T08:28:22.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:22.034+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.034883+0000) 2019-10-13T08:28:22.036 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:22.034+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.034703+0000) 2019-10-13T08:28:22.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:22.034+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.035007+0000) 2019-10-13T08:28:22.037 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:22.035+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.035390+0000) 2019-10-13T08:28:22.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:22.035+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.035832+0000) 2019-10-13T08:28:22.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:22.036+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.037265+0000) 2019-10-13T08:28:22.039 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:22.035+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.036166+0000) 2019-10-13T08:28:22.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:22.036+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.036639+0000) 2019-10-13T08:28:22.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:22.040+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144362 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:22.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:22.861+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.862344+0000) 2019-10-13T08:28:22.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:22.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.862936+0000) 2019-10-13T08:28:22.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:22.868+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.869326+0000) 2019-10-13T08:28:27.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:27.042+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.042491+0000) 2019-10-13T08:28:27.044 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:27.043+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.043543+0000) 2019-10-13T08:28:27.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:27.043+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.043750+0000) 2019-10-13T08:28:27.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:27.043+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.044083+0000) 2019-10-13T08:28:27.045 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:27.043+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.044276+0000) 2019-10-13T08:28:27.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:27.044+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.044513+0000) 2019-10-13T08:28:27.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:27.043+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.044708+0000) 2019-10-13T08:28:27.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:27.044+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.045484+0000) 2019-10-13T08:28:27.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:27.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144392 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:27.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:27.866+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.867059+0000) 2019-10-13T08:28:28.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:28.415+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.415516+0000) 2019-10-13T08:28:28.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:28.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.753623+0000) 2019-10-13T08:28:28.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:28.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.753624+0000) 2019-10-13T08:28:28.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:28.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:28.754644+0000) 2019-10-13T08:28:32.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:32.050+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.051114+0000) 2019-10-13T08:28:32.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:32.050+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.051275+0000) 2019-10-13T08:28:32.053 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:32.051+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.051425+0000) 2019-10-13T08:28:32.053 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:32.051+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.051773+0000) 2019-10-13T08:28:32.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:32.051+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.052177+0000) 2019-10-13T08:28:32.053 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:32.051+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.052232+0000) 2019-10-13T08:28:32.053 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:32.052+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.052571+0000) 2019-10-13T08:28:32.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:32.053+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.054036+0000) 2019-10-13T08:28:32.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:32.056+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:32.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:32.862+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.862525+0000) 2019-10-13T08:28:32.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:32.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.863103+0000) 2019-10-13T08:28:32.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:32.871+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.872173+0000) 2019-10-13T08:28:35.052 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:28:35.052 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:35.057 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:28:35.057 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:37.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:37.056+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.058162+0000) 2019-10-13T08:28:37.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:37.057+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.058307+0000) 2019-10-13T08:28:37.060 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:37.057+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.058720+0000) 2019-10-13T08:28:37.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:37.057+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.058928+0000) 2019-10-13T08:28:37.060 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:37.057+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.059014+0000) 2019-10-13T08:28:37.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:37.057+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.059117+0000) 2019-10-13T08:28:37.061 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:37.057+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.059224+0000) 2019-10-13T08:28:37.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:37.059+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.060652+0000) 2019-10-13T08:28:37.063 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:37.061+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:37.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:37.866+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.867229+0000) 2019-10-13T08:28:38.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:38.414+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.415704+0000) 2019-10-13T08:28:38.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:38.752+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.753838+0000) 2019-10-13T08:28:38.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:38.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.753838+0000) 2019-10-13T08:28:38.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:38.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:38.754814+0000) 2019-10-13T08:28:42.065 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:42.063+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.064468+0000) 2019-10-13T08:28:42.065 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:42.064+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.065502+0000) 2019-10-13T08:28:42.066 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:42.064+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.065743+0000) 2019-10-13T08:28:42.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:42.064+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.065953+0000) 2019-10-13T08:28:42.067 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:42.064+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.066012+0000) 2019-10-13T08:28:42.067 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:42.064+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.066384+0000) 2019-10-13T08:28:42.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:42.065+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.066211+0000) 2019-10-13T08:28:42.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:42.065+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.066656+0000) 2019-10-13T08:28:42.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:42.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144482 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:42.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:42.862+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.862651+0000) 2019-10-13T08:28:42.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:42.861+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.863188+0000) 2019-10-13T08:28:42.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:42.874+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.874529+0000) 2019-10-13T08:28:47.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:47.071+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.072405+0000) 2019-10-13T08:28:47.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:47.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.072720+0000) 2019-10-13T08:28:47.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:47.071+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.072896+0000) 2019-10-13T08:28:47.074 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:47.071+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.073043+0000) 2019-10-13T08:28:47.074 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:47.071+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.073323+0000) 2019-10-13T08:28:47.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:47.072+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.073654+0000) 2019-10-13T08:28:47.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:47.073+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.074017+0000) 2019-10-13T08:28:47.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:47.075+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.075954+0000) 2019-10-13T08:28:47.078 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:47.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:47.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:47.866+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.867394+0000) 2019-10-13T08:28:48.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:48.415+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.415865+0000) 2019-10-13T08:28:48.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:48.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.754019+0000) 2019-10-13T08:28:48.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:48.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.754048+0000) 2019-10-13T08:28:48.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:48.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:48.754996+0000) 2019-10-13T08:28:52.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:52.079+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.079941+0000) 2019-10-13T08:28:52.081 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:52.078+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.080258+0000) 2019-10-13T08:28:52.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:52.079+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.080592+0000) 2019-10-13T08:28:52.082 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:52.079+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.080429+0000) 2019-10-13T08:28:52.082 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:52.079+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.080650+0000) 2019-10-13T08:28:52.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:52.079+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.081032+0000) 2019-10-13T08:28:52.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:52.079+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.081198+0000) 2019-10-13T08:28:52.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:52.082+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.082985+0000) 2019-10-13T08:28:52.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:52.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:52.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:52.862+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.862814+0000) 2019-10-13T08:28:52.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:52.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.863271+0000) 2019-10-13T08:28:52.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:52.876+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.876706+0000) 2019-10-13T08:28:57.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:28:57.085+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.087696+0000) 2019-10-13T08:28:57.089 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:57.086+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.087917+0000) 2019-10-13T08:28:57.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:28:57.086+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.088264+0000) 2019-10-13T08:28:57.090 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:57.086+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.088344+0000) 2019-10-13T08:28:57.090 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:57.087+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.088541+0000) 2019-10-13T08:28:57.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:57.087+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.088759+0000) 2019-10-13T08:28:57.090 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:57.087+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.088950+0000) 2019-10-13T08:28:57.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:28:57.088+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.090209+0000) 2019-10-13T08:28:57.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:28:57.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144568 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:28:57.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:28:57.867+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.867564+0000) 2019-10-13T08:28:58.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:28:58.415+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.416052+0000) 2019-10-13T08:28:58.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:28:58.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.754227+0000) 2019-10-13T08:28:58.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:28:58.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.754238+0000) 2019-10-13T08:28:58.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:28:58.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:58.755178+0000) 2019-10-13T08:29:02.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:02.094+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.095144+0000) 2019-10-13T08:29:02.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:02.094+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.095982+0000) 2019-10-13T08:29:02.097 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:02.095+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.095541+0000) 2019-10-13T08:29:02.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:02.095+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.095624+0000) 2019-10-13T08:29:02.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:02.095+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.095792+0000) 2019-10-13T08:29:02.097 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:02.095+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.096204+0000) 2019-10-13T08:29:02.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:02.095+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.097660+0000) 2019-10-13T08:29:02.101 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:02.095+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.096356+0000) 2019-10-13T08:29:02.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:02.100+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:02.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:02.862+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.863002+0000) 2019-10-13T08:29:02.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:02.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.863421+0000) 2019-10-13T08:29:02.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:02.878+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.879272+0000) 2019-10-13T08:29:05.206 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:29:05.207 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:05.211 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:29:05.211 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:07.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:07.100+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.102615+0000) 2019-10-13T08:29:07.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:07.102+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.102418+0000) 2019-10-13T08:29:07.104 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:07.102+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.103175+0000) 2019-10-13T08:29:07.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:07.102+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.103365+0000) 2019-10-13T08:29:07.104 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:07.103+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.103518+0000) 2019-10-13T08:29:07.105 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:07.103+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.103829+0000) 2019-10-13T08:29:07.105 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:07.103+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.104157+0000) 2019-10-13T08:29:07.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:07.104+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.105997+0000) 2019-10-13T08:29:07.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:07.108+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:07.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:07.867+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.867735+0000) 2019-10-13T08:29:08.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:08.415+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.416207+0000) 2019-10-13T08:29:08.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:08.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.754439+0000) 2019-10-13T08:29:08.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:08.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.754462+0000) 2019-10-13T08:29:08.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:08.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:08.755367+0000) 2019-10-13T08:29:12.110 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:12.109+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.110165+0000) 2019-10-13T08:29:12.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:12.109+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.110475+0000) 2019-10-13T08:29:12.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:12.110+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.110981+0000) 2019-10-13T08:29:12.112 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:12.110+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.110843+0000) 2019-10-13T08:29:12.112 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:12.110+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.111355+0000) 2019-10-13T08:29:12.112 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:12.110+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.111312+0000) 2019-10-13T08:29:12.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:12.110+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.111126+0000) 2019-10-13T08:29:12.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:12.111+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.112118+0000) 2019-10-13T08:29:12.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:12.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144655 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:12.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:12.862+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.863137+0000) 2019-10-13T08:29:12.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:12.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.863558+0000) 2019-10-13T08:29:12.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:12.881+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.881772+0000) 2019-10-13T08:29:17.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:17.117+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.117742+0000) 2019-10-13T08:29:17.119 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:17.117+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.117944+0000) 2019-10-13T08:29:17.119 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:17.117+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.118202+0000) 2019-10-13T08:29:17.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:17.117+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.118180+0000) 2019-10-13T08:29:17.119 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:17.118+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.118525+0000) 2019-10-13T08:29:17.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:17.118+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.118796+0000) 2019-10-13T08:29:17.120 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:17.118+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.118971+0000) 2019-10-13T08:29:17.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:17.120+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.120802+0000) 2019-10-13T08:29:17.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:17.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:17.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:17.867+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.867931+0000) 2019-10-13T08:29:18.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:18.416+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.416378+0000) 2019-10-13T08:29:18.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:18.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.754618+0000) 2019-10-13T08:29:18.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:18.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.754619+0000) 2019-10-13T08:29:18.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:18.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:18.755581+0000) 2019-10-13T08:29:22.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:22.123+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.125151+0000) 2019-10-13T08:29:22.126 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:22.124+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.125441+0000) 2019-10-13T08:29:22.127 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:22.125+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.126277+0000) 2019-10-13T08:29:22.127 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:22.124+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.125713+0000) 2019-10-13T08:29:22.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:22.125+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.126603+0000) 2019-10-13T08:29:22.127 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:22.125+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.126776+0000) 2019-10-13T08:29:22.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:22.125+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.126950+0000) 2019-10-13T08:29:22.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:22.127+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.128017+0000) 2019-10-13T08:29:22.132 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:22.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:22.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:22.862+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.863271+0000) 2019-10-13T08:29:22.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:22.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.863734+0000) 2019-10-13T08:29:22.884 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:22.883+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.884032+0000) 2019-10-13T08:29:27.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:27.132+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.133281+0000) 2019-10-13T08:29:27.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:27.132+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.133714+0000) 2019-10-13T08:29:27.134 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:27.132+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.134181+0000) 2019-10-13T08:29:27.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:27.132+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.134347+0000) 2019-10-13T08:29:27.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:27.133+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.134620+0000) 2019-10-13T08:29:27.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:27.133+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.134999+0000) 2019-10-13T08:29:27.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:27.133+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.135187+0000) 2019-10-13T08:29:27.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:27.135+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.136192+0000) 2019-10-13T08:29:27.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:27.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:27.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:27.867+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.868104+0000) 2019-10-13T08:29:28.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:28.415+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.416561+0000) 2019-10-13T08:29:28.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:28.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.754856+0000) 2019-10-13T08:29:28.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:28.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.754857+0000) 2019-10-13T08:29:28.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:28.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:28.755786+0000) 2019-10-13T08:29:32.141 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:32.139+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.141092+0000) 2019-10-13T08:29:32.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:32.140+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.141506+0000) 2019-10-13T08:29:32.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:32.140+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.141728+0000) 2019-10-13T08:29:32.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:32.140+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.142072+0000) 2019-10-13T08:29:32.143 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:32.140+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.142218+0000) 2019-10-13T08:29:32.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:32.141+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.142600+0000) 2019-10-13T08:29:32.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:32.141+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.142791+0000) 2019-10-13T08:29:32.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:32.143+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.144483+0000) 2019-10-13T08:29:32.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:32.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144772 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:32.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:32.863+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.863453+0000) 2019-10-13T08:29:32.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:32.862+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.863888+0000) 2019-10-13T08:29:32.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:32.886+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.886470+0000) 2019-10-13T08:29:35.369 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:29:35.370 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:35.374 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:29:35.374 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:37.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:37.149+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.150453+0000) 2019-10-13T08:29:37.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:37.149+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.150725+0000) 2019-10-13T08:29:37.152 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:37.149+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.150915+0000) 2019-10-13T08:29:37.152 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:37.149+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.151099+0000) 2019-10-13T08:29:37.152 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:37.149+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.151219+0000) 2019-10-13T08:29:37.153 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:37.150+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.151568+0000) 2019-10-13T08:29:37.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:37.150+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.151701+0000) 2019-10-13T08:29:37.153 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:37.151+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.152437+0000) 2019-10-13T08:29:37.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:37.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:37.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:37.867+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.868256+0000) 2019-10-13T08:29:38.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:38.416+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.416754+0000) 2019-10-13T08:29:38.755 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:38.753+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.754986+0000) 2019-10-13T08:29:38.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:38.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.755000+0000) 2019-10-13T08:29:38.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:38.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:38.755918+0000) 2019-10-13T08:29:42.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:42.155+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.156990+0000) 2019-10-13T08:29:42.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:42.156+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.157161+0000) 2019-10-13T08:29:42.159 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:42.156+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.157613+0000) 2019-10-13T08:29:42.159 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:42.156+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.157732+0000) 2019-10-13T08:29:42.159 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:42.156+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.157874+0000) 2019-10-13T08:29:42.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:42.156+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.158109+0000) 2019-10-13T08:29:42.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:42.156+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.158265+0000) 2019-10-13T08:29:42.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:42.158+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.159204+0000) 2019-10-13T08:29:42.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:42.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:42.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:42.863+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.863608+0000) 2019-10-13T08:29:42.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:42.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.864021+0000) 2019-10-13T08:29:42.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:42.888+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.888789+0000) 2019-10-13T08:29:47.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:47.164+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.165424+0000) 2019-10-13T08:29:47.167 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:47.164+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.165838+0000) 2019-10-13T08:29:47.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:47.164+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.166144+0000) 2019-10-13T08:29:47.168 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:47.164+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.166363+0000) 2019-10-13T08:29:47.168 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:47.165+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.166864+0000) 2019-10-13T08:29:47.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:47.165+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.167177+0000) 2019-10-13T08:29:47.169 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:47.165+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.167223+0000) 2019-10-13T08:29:47.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:47.167+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.169490+0000) 2019-10-13T08:29:47.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:47.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:47.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:47.868+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.868424+0000) 2019-10-13T08:29:48.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:48.416+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.416919+0000) 2019-10-13T08:29:48.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:48.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.755172+0000) 2019-10-13T08:29:48.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:48.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.755174+0000) 2019-10-13T08:29:48.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:48.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:48.756028+0000) 2019-10-13T08:29:52.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:52.171+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.173696+0000) 2019-10-13T08:29:52.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:52.173+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.173843+0000) 2019-10-13T08:29:52.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:52.173+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.173885+0000) 2019-10-13T08:29:52.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:52.173+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.174033+0000) 2019-10-13T08:29:52.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:52.173+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.174151+0000) 2019-10-13T08:29:52.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:52.174+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.174438+0000) 2019-10-13T08:29:52.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:52.174+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.174595+0000) 2019-10-13T08:29:52.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:52.175+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.176848+0000) 2019-10-13T08:29:52.181 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:52.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:52.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:52.862+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.863765+0000) 2019-10-13T08:29:52.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:52.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.864126+0000) 2019-10-13T08:29:52.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:52.890+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.890999+0000) 2019-10-13T08:29:57.180 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:29:57.179+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.180259+0000) 2019-10-13T08:29:57.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:29:57.178+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.180645+0000) 2019-10-13T08:29:57.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:57.180+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.180826+0000) 2019-10-13T08:29:57.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:57.180+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.180882+0000) 2019-10-13T08:29:57.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:57.180+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.181061+0000) 2019-10-13T08:29:57.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:57.180+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.181329+0000) 2019-10-13T08:29:57.182 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:57.181+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.181448+0000) 2019-10-13T08:29:57.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:29:57.181+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.183463+0000) 2019-10-13T08:29:57.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:29:57.186+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:29:57.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:29:57.868+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.868595+0000) 2019-10-13T08:29:58.417 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:29:58.416+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.417104+0000) 2019-10-13T08:29:58.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:29:58.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.755386+0000) 2019-10-13T08:29:58.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:29:58.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.755388+0000) 2019-10-13T08:29:58.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:29:58.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:58.756173+0000) 2019-10-13T08:30:02.188 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:02.187+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.188241+0000) 2019-10-13T08:30:02.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:02.188+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.188572+0000) 2019-10-13T08:30:02.190 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:02.188+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.188567+0000) 2019-10-13T08:30:02.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:02.188+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.188797+0000) 2019-10-13T08:30:02.190 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:02.188+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.189152+0000) 2019-10-13T08:30:02.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:02.188+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.189113+0000) 2019-10-13T08:30:02.190 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:02.188+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.189274+0000) 2019-10-13T08:30:02.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:02.190+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.191058+0000) 2019-10-13T08:30:02.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:02.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:02.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:02.863+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.863917+0000) 2019-10-13T08:30:02.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:02.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.864246+0000) 2019-10-13T08:30:02.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:02.892+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.893580+0000) 2019-10-13T08:30:05.514 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:30:05.515 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:05.519 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:30:05.520 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:07.199 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:07.196+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.196755+0000) 2019-10-13T08:30:07.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:07.195+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.196627+0000) 2019-10-13T08:30:07.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:07.195+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.196835+0000) 2019-10-13T08:30:07.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:07.195+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.196966+0000) 2019-10-13T08:30:07.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:07.195+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.197135+0000) 2019-10-13T08:30:07.200 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:07.196+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.197591+0000) 2019-10-13T08:30:07.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:07.196+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.197749+0000) 2019-10-13T08:30:07.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:07.197+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.198521+0000) 2019-10-13T08:30:07.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:07.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 144973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:07.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:07.867+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.868767+0000) 2019-10-13T08:30:08.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:08.416+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.417289+0000) 2019-10-13T08:30:08.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:08.754+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.755596+0000) 2019-10-13T08:30:08.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:08.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.755622+0000) 2019-10-13T08:30:08.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:08.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:08.756336+0000) 2019-10-13T08:30:12.205 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:12.202+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.204341+0000) 2019-10-13T08:30:12.205 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:12.203+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.204587+0000) 2019-10-13T08:30:12.205 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:12.203+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.204868+0000) 2019-10-13T08:30:12.206 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:12.204+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.205664+0000) 2019-10-13T08:30:12.206 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:12.203+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.205095+0000) 2019-10-13T08:30:12.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:12.204+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.205457+0000) 2019-10-13T08:30:12.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:12.204+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.205793+0000) 2019-10-13T08:30:12.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:12.206+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.207642+0000) 2019-10-13T08:30:12.210 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:12.208+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145005 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:12.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:12.863+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.864074+0000) 2019-10-13T08:30:12.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:12.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.864434+0000) 2019-10-13T08:30:12.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:12.895+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.896027+0000) 2019-10-13T08:30:17.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:17.211+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.211873+0000) 2019-10-13T08:30:17.213 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:17.210+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.211943+0000) 2019-10-13T08:30:17.213 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:17.210+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.212061+0000) 2019-10-13T08:30:17.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:17.210+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.212212+0000) 2019-10-13T08:30:17.214 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:17.210+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.212323+0000) 2019-10-13T08:30:17.214 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:17.211+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.212725+0000) 2019-10-13T08:30:17.216 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:17.211+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.213121+0000) 2019-10-13T08:30:17.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:17.212+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.213530+0000) 2019-10-13T08:30:17.218 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:17.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:17.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:17.868+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.868935+0000) 2019-10-13T08:30:18.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:18.416+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.417421+0000) 2019-10-13T08:30:18.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:18.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.755783+0000) 2019-10-13T08:30:18.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:18.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.755815+0000) 2019-10-13T08:30:18.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:18.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:18.756464+0000) 2019-10-13T08:30:22.220 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:22.218+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.219727+0000) 2019-10-13T08:30:22.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:22.218+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.219965+0000) 2019-10-13T08:30:22.221 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:22.219+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.220366+0000) 2019-10-13T08:30:22.222 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:22.219+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.220428+0000) 2019-10-13T08:30:22.222 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:22.219+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.220658+0000) 2019-10-13T08:30:22.222 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:22.219+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.220823+0000) 2019-10-13T08:30:22.222 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:22.219+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.220983+0000) 2019-10-13T08:30:22.223 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:22.221+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.221852+0000) 2019-10-13T08:30:22.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:22.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:22.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:22.863+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.864207+0000) 2019-10-13T08:30:22.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:22.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.864585+0000) 2019-10-13T08:30:22.899 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:22.897+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.898399+0000) 2019-10-13T08:30:27.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:27.226+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.227266+0000) 2019-10-13T08:30:27.228 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:27.225+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.227375+0000) 2019-10-13T08:30:27.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:27.226+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.227536+0000) 2019-10-13T08:30:27.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:27.226+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.227652+0000) 2019-10-13T08:30:27.229 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:27.226+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.227758+0000) 2019-10-13T08:30:27.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:27.226+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.228139+0000) 2019-10-13T08:30:27.231 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:27.228+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.229162+0000) 2019-10-13T08:30:27.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:27.227+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.228477+0000) 2019-10-13T08:30:27.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:27.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:27.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:27.868+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.869096+0000) 2019-10-13T08:30:28.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:28.417+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.417566+0000) 2019-10-13T08:30:28.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:28.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.755913+0000) 2019-10-13T08:30:28.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:28.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.755998+0000) 2019-10-13T08:30:28.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:28.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:28.756627+0000) 2019-10-13T08:30:32.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:32.233+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.234794+0000) 2019-10-13T08:30:32.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:32.233+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.234968+0000) 2019-10-13T08:30:32.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:32.233+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.235156+0000) 2019-10-13T08:30:32.236 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:32.234+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.235478+0000) 2019-10-13T08:30:32.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:32.235+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.235816+0000) 2019-10-13T08:30:32.237 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:32.234+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.235995+0000) 2019-10-13T08:30:32.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:32.234+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.236199+0000) 2019-10-13T08:30:32.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:32.237+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.238353+0000) 2019-10-13T08:30:32.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:32.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:32.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:32.864+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.864400+0000) 2019-10-13T08:30:32.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:32.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.864722+0000) 2019-10-13T08:30:32.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:32.900+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.900515+0000) 2019-10-13T08:30:35.677 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:30:35.677 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:35.681 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:30:35.681 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:37.243 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:37.242+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.242614+0000) 2019-10-13T08:30:37.243 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:37.242+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.242793+0000) 2019-10-13T08:30:37.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:37.242+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.242948+0000) 2019-10-13T08:30:37.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:37.242+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.243095+0000) 2019-10-13T08:30:37.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:37.242+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.243537+0000) 2019-10-13T08:30:37.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:37.242+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.243336+0000) 2019-10-13T08:30:37.245 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:37.243+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.243591+0000) 2019-10-13T08:30:37.249 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:37.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145151 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:37.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:37.246+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.246763+0000) 2019-10-13T08:30:37.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:37.868+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.869226+0000) 2019-10-13T08:30:38.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:38.417+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.417749+0000) 2019-10-13T08:30:38.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:38.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.756025+0000) 2019-10-13T08:30:38.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:38.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.756130+0000) 2019-10-13T08:30:38.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:38.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:38.756778+0000) 2019-10-13T08:30:42.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:42.248+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.248682+0000) 2019-10-13T08:30:42.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:42.248+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.248885+0000) 2019-10-13T08:30:42.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:42.248+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.249238+0000) 2019-10-13T08:30:42.250 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:42.248+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.249356+0000) 2019-10-13T08:30:42.251 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:42.248+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.250374+0000) 2019-10-13T08:30:42.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:42.249+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.249520+0000) 2019-10-13T08:30:42.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:42.249+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.249745+0000) 2019-10-13T08:30:42.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:42.250+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.252410+0000) 2019-10-13T08:30:42.255 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:42.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145183 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:42.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:42.863+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.864534+0000) 2019-10-13T08:30:42.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:42.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.864838+0000) 2019-10-13T08:30:42.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:42.901+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.902566+0000) 2019-10-13T08:30:47.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:47.256+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.256775+0000) 2019-10-13T08:30:47.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:47.256+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.256976+0000) 2019-10-13T08:30:47.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:47.256+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.257157+0000) 2019-10-13T08:30:47.260 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:47.256+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.257335+0000) 2019-10-13T08:30:47.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:47.257+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.257628+0000) 2019-10-13T08:30:47.260 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:47.257+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.258009+0000) 2019-10-13T08:30:47.261 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:47.256+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.257790+0000) 2019-10-13T08:30:47.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:47.257+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.259667+0000) 2019-10-13T08:30:47.263 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:47.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:47.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:47.869+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.869373+0000) 2019-10-13T08:30:48.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:48.417+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.417962+0000) 2019-10-13T08:30:48.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:48.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.756179+0000) 2019-10-13T08:30:48.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:48.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.756289+0000) 2019-10-13T08:30:48.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:48.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:48.756963+0000) 2019-10-13T08:30:52.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:52.263+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.264746+0000) 2019-10-13T08:30:52.266 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:52.263+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.265030+0000) 2019-10-13T08:30:52.266 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:52.263+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.265354+0000) 2019-10-13T08:30:52.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:52.263+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.264822+0000) 2019-10-13T08:30:52.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:52.264+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.265469+0000) 2019-10-13T08:30:52.267 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:52.263+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.265299+0000) 2019-10-13T08:30:52.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:52.265+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.266762+0000) 2019-10-13T08:30:52.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:52.266+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.267762+0000) 2019-10-13T08:30:52.270 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:52.268+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:52.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:52.863+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.864707+0000) 2019-10-13T08:30:52.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:52.863+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.864995+0000) 2019-10-13T08:30:52.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:52.903+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.904630+0000) 2019-10-13T08:30:57.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:57.270+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.272035+0000) 2019-10-13T08:30:57.273 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:57.270+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.272102+0000) 2019-10-13T08:30:57.273 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:57.270+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.272362+0000) 2019-10-13T08:30:57.274 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:57.270+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.272218+0000) 2019-10-13T08:30:57.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:57.271+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.272544+0000) 2019-10-13T08:30:57.274 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:30:57.271+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.272855+0000) 2019-10-13T08:30:57.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:30:57.271+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.272707+0000) 2019-10-13T08:30:57.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:30:57.274+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.275110+0000) 2019-10-13T08:30:57.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:30:57.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:30:57.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:30:57.868+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.869562+0000) 2019-10-13T08:30:58.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:30:58.417+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.418163+0000) 2019-10-13T08:30:58.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:30:58.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.756334+0000) 2019-10-13T08:30:58.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:30:58.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.756442+0000) 2019-10-13T08:30:58.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:30:58.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:58.757090+0000) 2019-10-13T08:31:02.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:02.278+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.279958+0000) 2019-10-13T08:31:02.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:02.278+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.280311+0000) 2019-10-13T08:31:02.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:02.279+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.280439+0000) 2019-10-13T08:31:02.282 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:02.279+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.280677+0000) 2019-10-13T08:31:02.282 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:02.279+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.280940+0000) 2019-10-13T08:31:02.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:02.279+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.281141+0000) 2019-10-13T08:31:02.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:02.280+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.281439+0000) 2019-10-13T08:31:02.283 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:02.282+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.283478+0000) 2019-10-13T08:31:02.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:02.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145301 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:02.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:02.864+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.864822+0000) 2019-10-13T08:31:02.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:02.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.865125+0000) 2019-10-13T08:31:02.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:02.906+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.906916+0000) 2019-10-13T08:31:05.857 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:31:05.858 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:05.862 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:31:05.863 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:07.289 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:07.287+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.288628+0000) 2019-10-13T08:31:07.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:07.287+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.288753+0000) 2019-10-13T08:31:07.290 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:07.287+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.288885+0000) 2019-10-13T08:31:07.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:07.288+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.289185+0000) 2019-10-13T08:31:07.291 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:07.288+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.289696+0000) 2019-10-13T08:31:07.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:07.288+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.289855+0000) 2019-10-13T08:31:07.291 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:07.288+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.290190+0000) 2019-10-13T08:31:07.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:07.290+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.291471+0000) 2019-10-13T08:31:07.296 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:07.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145327 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:07.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:07.869+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.869731+0000) 2019-10-13T08:31:08.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:08.417+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.418304+0000) 2019-10-13T08:31:08.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:08.755+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.756517+0000) 2019-10-13T08:31:08.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:08.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.756595+0000) 2019-10-13T08:31:08.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:08.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:08.757255+0000) 2019-10-13T08:31:12.298 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:12.295+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.297276+0000) 2019-10-13T08:31:12.298 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:12.296+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.297393+0000) 2019-10-13T08:31:12.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:12.296+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.297591+0000) 2019-10-13T08:31:12.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:12.296+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.297852+0000) 2019-10-13T08:31:12.299 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:12.297+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.298316+0000) 2019-10-13T08:31:12.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:12.298+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.299204+0000) 2019-10-13T08:31:12.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:12.296+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.298104+0000) 2019-10-13T08:31:12.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:12.297+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.298454+0000) 2019-10-13T08:31:12.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:12.302+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145359 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:12.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:12.864+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.864980+0000) 2019-10-13T08:31:12.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:12.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.865206+0000) 2019-10-13T08:31:12.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:12.908+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.909373+0000) 2019-10-13T08:31:17.305 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:17.303+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.304593+0000) 2019-10-13T08:31:17.305 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:17.303+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.304859+0000) 2019-10-13T08:31:17.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:17.303+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.305188+0000) 2019-10-13T08:31:17.306 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:17.305+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.305804+0000) 2019-10-13T08:31:17.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:17.303+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.305358+0000) 2019-10-13T08:31:17.307 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:17.304+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.305823+0000) 2019-10-13T08:31:17.307 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:17.304+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.305942+0000) 2019-10-13T08:31:17.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:17.307+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.307865+0000) 2019-10-13T08:31:17.311 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:17.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:17.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:17.869+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.869858+0000) 2019-10-13T08:31:18.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:18.418+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.418490+0000) 2019-10-13T08:31:18.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:18.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.756704+0000) 2019-10-13T08:31:18.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:18.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.756716+0000) 2019-10-13T08:31:18.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:18.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:18.757431+0000) 2019-10-13T08:31:22.312 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:22.311+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.312292+0000) 2019-10-13T08:31:22.314 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:22.311+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.312373+0000) 2019-10-13T08:31:22.314 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:22.312+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.312712+0000) 2019-10-13T08:31:22.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:22.312+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.312796+0000) 2019-10-13T08:31:22.314 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:22.312+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.312941+0000) 2019-10-13T08:31:22.314 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:22.312+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.313061+0000) 2019-10-13T08:31:22.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:22.313+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.313487+0000) 2019-10-13T08:31:22.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:22.314+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.315369+0000) 2019-10-13T08:31:22.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:22.317+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:22.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:22.864+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.865135+0000) 2019-10-13T08:31:22.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:22.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.865340+0000) 2019-10-13T08:31:22.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:22.911+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.912156+0000) 2019-10-13T08:31:27.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:27.319+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.320228+0000) 2019-10-13T08:31:27.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:27.319+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.320252+0000) 2019-10-13T08:31:27.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:27.320+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.320526+0000) 2019-10-13T08:31:27.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:27.320+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.320491+0000) 2019-10-13T08:31:27.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:27.320+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.320622+0000) 2019-10-13T08:31:27.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:27.320+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.320701+0000) 2019-10-13T08:31:27.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:27.320+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.321124+0000) 2019-10-13T08:31:27.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:27.321+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.322136+0000) 2019-10-13T08:31:27.325 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:27.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145445 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:27.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:27.869+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.870034+0000) 2019-10-13T08:31:28.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:28.418+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.418649+0000) 2019-10-13T08:31:28.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:28.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.756895+0000) 2019-10-13T08:31:28.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:28.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.756895+0000) 2019-10-13T08:31:28.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:28.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:28.757604+0000) 2019-10-13T08:31:32.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:32.326+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.327122+0000) 2019-10-13T08:31:32.328 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:32.326+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.327271+0000) 2019-10-13T08:31:32.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:32.325+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.327526+0000) 2019-10-13T08:31:32.329 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:32.327+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.327628+0000) 2019-10-13T08:31:32.329 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:32.327+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.327950+0000) 2019-10-13T08:31:32.329 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:32.327+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.328129+0000) 2019-10-13T08:31:32.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:32.327+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.328383+0000) 2019-10-13T08:31:32.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:32.328+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.330582+0000) 2019-10-13T08:31:32.333 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:32.332+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:32.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:32.864+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.865292+0000) 2019-10-13T08:31:32.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:32.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.865476+0000) 2019-10-13T08:31:32.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:32.913+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.914370+0000) 2019-10-13T08:31:35.962 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:31:35.962 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:35.966 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:31:35.966 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:37.335 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:37.334+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.334951+0000) 2019-10-13T08:31:37.336 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:37.334+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.335104+0000) 2019-10-13T08:31:37.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:37.333+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.335364+0000) 2019-10-13T08:31:37.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:37.335+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.335443+0000) 2019-10-13T08:31:37.337 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:37.335+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.335464+0000) 2019-10-13T08:31:37.337 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:37.335+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.335918+0000) 2019-10-13T08:31:37.337 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:37.335+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.335750+0000) 2019-10-13T08:31:37.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:37.336+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.337879+0000) 2019-10-13T08:31:37.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:37.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:37.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:37.870+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.870215+0000) 2019-10-13T08:31:38.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:38.417+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.418826+0000) 2019-10-13T08:31:38.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:38.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.757099+0000) 2019-10-13T08:31:38.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:38.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.757099+0000) 2019-10-13T08:31:38.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:38.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:38.757750+0000) 2019-10-13T08:31:42.342 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:42.339+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.341580+0000) 2019-10-13T08:31:42.343 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:42.340+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.341828+0000) 2019-10-13T08:31:42.343 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:42.340+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.341999+0000) 2019-10-13T08:31:42.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:42.340+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.342105+0000) 2019-10-13T08:31:42.343 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:42.341+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.342419+0000) 2019-10-13T08:31:42.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:42.340+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.342326+0000) 2019-10-13T08:31:42.344 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:42.341+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.342496+0000) 2019-10-13T08:31:42.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:42.342+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.343973+0000) 2019-10-13T08:31:42.348 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:42.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:42.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:42.864+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.865424+0000) 2019-10-13T08:31:42.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:42.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.865604+0000) 2019-10-13T08:31:42.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:42.915+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.916407+0000) 2019-10-13T08:31:47.350 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:47.349+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.349830+0000) 2019-10-13T08:31:47.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:47.348+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.349893+0000) 2019-10-13T08:31:47.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:47.348+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.350261+0000) 2019-10-13T08:31:47.352 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:47.349+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.350554+0000) 2019-10-13T08:31:47.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:47.349+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.350652+0000) 2019-10-13T08:31:47.352 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:47.349+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.350806+0000) 2019-10-13T08:31:47.352 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:47.349+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.351043+0000) 2019-10-13T08:31:47.353 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:47.351+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.352678+0000) 2019-10-13T08:31:47.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:47.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145563 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:47.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:47.869+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.870385+0000) 2019-10-13T08:31:48.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:48.417+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.419023+0000) 2019-10-13T08:31:48.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:48.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.757333+0000) 2019-10-13T08:31:48.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:48.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.757333+0000) 2019-10-13T08:31:48.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:48.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:48.757849+0000) 2019-10-13T08:31:52.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:52.356+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.357825+0000) 2019-10-13T08:31:52.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:52.356+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.357957+0000) 2019-10-13T08:31:52.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:52.356+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.358228+0000) 2019-10-13T08:31:52.360 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:52.357+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.358458+0000) 2019-10-13T08:31:52.360 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:52.357+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.358466+0000) 2019-10-13T08:31:52.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:52.357+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.358864+0000) 2019-10-13T08:31:52.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:52.357+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.359133+0000) 2019-10-13T08:31:52.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:52.361+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.361972+0000) 2019-10-13T08:31:52.364 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:52.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:52.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:52.864+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.865577+0000) 2019-10-13T08:31:52.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:52.864+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.865711+0000) 2019-10-13T08:31:52.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:52.917+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.918521+0000) 2019-10-13T08:31:57.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:57.364+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.365897+0000) 2019-10-13T08:31:57.367 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:31:57.365+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.366178+0000) 2019-10-13T08:31:57.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:57.364+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.366366+0000) 2019-10-13T08:31:57.368 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:57.365+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.366585+0000) 2019-10-13T08:31:57.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:57.365+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.366741+0000) 2019-10-13T08:31:57.368 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:57.365+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.367074+0000) 2019-10-13T08:31:57.368 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:31:57.365+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.367283+0000) 2019-10-13T08:31:57.369 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:31:57.368+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.368758+0000) 2019-10-13T08:31:57.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:31:57.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:31:57.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:31:57.869+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.870583+0000) 2019-10-13T08:31:58.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:31:58.418+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.419178+0000) 2019-10-13T08:31:58.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:31:58.756+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.757547+0000) 2019-10-13T08:31:58.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:31:58.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.757547+0000) 2019-10-13T08:31:58.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:31:58.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:58.757968+0000) 2019-10-13T08:32:02.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:02.373+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.374499+0000) 2019-10-13T08:32:02.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:02.371+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.373375+0000) 2019-10-13T08:32:02.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:02.372+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.373717+0000) 2019-10-13T08:32:02.378 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:02.372+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.373904+0000) 2019-10-13T08:32:02.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:02.372+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.374006+0000) 2019-10-13T08:32:02.378 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:02.372+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.374292+0000) 2019-10-13T08:32:02.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:02.373+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.374783+0000) 2019-10-13T08:32:02.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:02.375+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.375781+0000) 2019-10-13T08:32:02.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:02.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145655 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:02.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:02.865+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.865748+0000) 2019-10-13T08:32:02.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:02.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.865839+0000) 2019-10-13T08:32:02.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:02.920+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.921059+0000) 2019-10-13T08:32:06.124 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:32:06.124 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:06.128 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:32:06.128 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:07.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:07.381+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.381572+0000) 2019-10-13T08:32:07.383 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:07.381+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.381614+0000) 2019-10-13T08:32:07.383 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:07.381+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.381843+0000) 2019-10-13T08:32:07.383 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:07.381+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.382043+0000) 2019-10-13T08:32:07.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:07.381+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.382169+0000) 2019-10-13T08:32:07.384 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:07.381+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.382699+0000) 2019-10-13T08:32:07.384 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:07.382+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.382864+0000) 2019-10-13T08:32:07.385 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:07.384+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.384935+0000) 2019-10-13T08:32:07.388 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:07.387+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145681 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:07.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:07.870+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.870735+0000) 2019-10-13T08:32:08.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:08.418+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.419339+0000) 2019-10-13T08:32:08.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:08.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.757730+0000) 2019-10-13T08:32:08.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:08.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.757730+0000) 2019-10-13T08:32:08.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:08.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:08.758093+0000) 2019-10-13T08:32:12.390 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:12.389+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.389626+0000) 2019-10-13T08:32:12.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:12.389+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.389978+0000) 2019-10-13T08:32:12.392 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:12.389+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.390236+0000) 2019-10-13T08:32:12.392 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:12.389+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.390324+0000) 2019-10-13T08:32:12.392 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:12.390+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.390607+0000) 2019-10-13T08:32:12.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:12.390+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.390860+0000) 2019-10-13T08:32:12.393 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:12.390+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.391121+0000) 2019-10-13T08:32:12.393 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:12.391+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.392636+0000) 2019-10-13T08:32:12.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:12.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:12.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:12.865+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.865881+0000) 2019-10-13T08:32:12.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:12.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.865997+0000) 2019-10-13T08:32:12.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:12.923+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.923515+0000) 2019-10-13T08:32:17.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:17.397+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.398095+0000) 2019-10-13T08:32:17.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:17.397+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.398308+0000) 2019-10-13T08:32:17.400 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:17.397+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.398220+0000) 2019-10-13T08:32:17.400 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:17.398+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.398697+0000) 2019-10-13T08:32:17.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:17.398+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.398895+0000) 2019-10-13T08:32:17.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:17.399+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.399414+0000) 2019-10-13T08:32:17.400 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:17.399+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.399568+0000) 2019-10-13T08:32:17.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:17.400+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.401274+0000) 2019-10-13T08:32:17.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:17.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:17.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:17.870+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.870917+0000) 2019-10-13T08:32:18.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:18.419+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.419506+0000) 2019-10-13T08:32:18.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:18.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.757913+0000) 2019-10-13T08:32:18.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:18.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.757913+0000) 2019-10-13T08:32:18.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:18.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:18.758193+0000) 2019-10-13T08:32:22.406 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:22.404+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.405498+0000) 2019-10-13T08:32:22.407 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:22.405+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.405621+0000) 2019-10-13T08:32:22.408 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:22.405+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.406135+0000) 2019-10-13T08:32:22.409 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:22.405+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.405886+0000) 2019-10-13T08:32:22.409 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:22.406+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.406408+0000) 2019-10-13T08:32:22.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:22.406+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.406514+0000) 2019-10-13T08:32:22.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:22.406+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.406738+0000) 2019-10-13T08:32:22.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:22.408+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.408932+0000) 2019-10-13T08:32:22.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:22.411+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:22.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:22.865+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.866006+0000) 2019-10-13T08:32:22.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:22.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.866123+0000) 2019-10-13T08:32:22.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:22.925+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.925935+0000) 2019-10-13T08:32:27.413 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:27.411+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.413110+0000) 2019-10-13T08:32:27.415 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:27.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.413312+0000) 2019-10-13T08:32:27.415 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:27.412+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.413578+0000) 2019-10-13T08:32:27.416 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:27.412+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.413781+0000) 2019-10-13T08:32:27.416 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:27.412+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.414307+0000) 2019-10-13T08:32:27.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:27.413+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.414465+0000) 2019-10-13T08:32:27.416 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:27.413+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.414792+0000) 2019-10-13T08:32:27.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:27.415+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.417135+0000) 2019-10-13T08:32:27.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:27.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:27.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:27.869+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.871116+0000) 2019-10-13T08:32:28.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:28.418+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.419703+0000) 2019-10-13T08:32:28.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:28.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.758126+0000) 2019-10-13T08:32:28.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:28.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.758126+0000) 2019-10-13T08:32:28.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:28.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:28.758301+0000) 2019-10-13T08:32:32.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:32.419+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.420767+0000) 2019-10-13T08:32:32.421 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:32.419+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.420899+0000) 2019-10-13T08:32:32.422 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:32.419+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.421501+0000) 2019-10-13T08:32:32.423 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:32.420+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.421667+0000) 2019-10-13T08:32:32.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:32.420+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.421833+0000) 2019-10-13T08:32:32.423 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:32.420+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.421967+0000) 2019-10-13T08:32:32.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:32.420+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.422112+0000) 2019-10-13T08:32:32.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:32.422+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.423895+0000) 2019-10-13T08:32:32.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:32.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:32.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:32.865+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.866173+0000) 2019-10-13T08:32:32.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:32.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.866274+0000) 2019-10-13T08:32:32.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:32.927+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.928368+0000) 2019-10-13T08:32:36.340 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:32:36.341 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:36.345 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:32:36.345 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:37.428 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:37.426+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.428036+0000) 2019-10-13T08:32:37.429 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:37.426+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.428046+0000) 2019-10-13T08:32:37.429 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:37.427+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.428491+0000) 2019-10-13T08:32:37.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:37.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.428570+0000) 2019-10-13T08:32:37.430 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:37.427+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.428779+0000) 2019-10-13T08:32:37.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:37.427+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.429072+0000) 2019-10-13T08:32:37.430 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:37.427+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.429251+0000) 2019-10-13T08:32:37.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:37.430+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.431998+0000) 2019-10-13T08:32:37.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:37.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:37.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:37.870+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.871263+0000) 2019-10-13T08:32:38.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:38.418+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.419922+0000) 2019-10-13T08:32:38.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:38.757+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.758344+0000) 2019-10-13T08:32:38.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:38.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.758343+0000) 2019-10-13T08:32:38.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:38.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:38.758471+0000) 2019-10-13T08:32:42.436 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:42.433+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.435234+0000) 2019-10-13T08:32:42.436 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:42.434+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.435394+0000) 2019-10-13T08:32:42.436 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:42.434+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.435592+0000) 2019-10-13T08:32:42.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:42.434+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.435801+0000) 2019-10-13T08:32:42.437 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:42.435+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.436098+0000) 2019-10-13T08:32:42.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:42.434+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.436386+0000) 2019-10-13T08:32:42.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:42.435+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.436580+0000) 2019-10-13T08:32:42.438 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:42.437+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.438336+0000) 2019-10-13T08:32:42.440 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:42.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145891 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:42.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:42.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.866424+0000) 2019-10-13T08:32:42.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:42.865+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.866313+0000) 2019-10-13T08:32:42.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:42.930+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.930708+0000) 2019-10-13T08:32:47.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:47.440+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.441988+0000) 2019-10-13T08:32:47.443 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:47.441+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.442401+0000) 2019-10-13T08:32:47.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:47.441+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.442539+0000) 2019-10-13T08:32:47.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:47.441+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.442776+0000) 2019-10-13T08:32:47.444 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:47.442+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.443289+0000) 2019-10-13T08:32:47.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:47.441+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.443319+0000) 2019-10-13T08:32:47.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:47.442+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.443655+0000) 2019-10-13T08:32:47.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:47.444+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.445262+0000) 2019-10-13T08:32:47.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:47.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145917 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:47.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:47.870+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.871442+0000) 2019-10-13T08:32:48.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:48.418+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.420109+0000) 2019-10-13T08:32:48.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:48.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.758553+0000) 2019-10-13T08:32:48.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:48.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.758554+0000) 2019-10-13T08:32:48.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:48.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:48.758617+0000) 2019-10-13T08:32:52.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:52.449+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.449590+0000) 2019-10-13T08:32:52.450 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:52.449+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.449838+0000) 2019-10-13T08:32:52.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:52.449+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.449838+0000) 2019-10-13T08:32:52.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:52.449+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.450191+0000) 2019-10-13T08:32:52.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:52.449+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.450327+0000) 2019-10-13T08:32:52.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:52.450+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.450523+0000) 2019-10-13T08:32:52.452 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:52.450+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.450692+0000) 2019-10-13T08:32:52.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:52.451+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.452287+0000) 2019-10-13T08:32:52.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:52.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:52.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:52.866+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.866511+0000) 2019-10-13T08:32:52.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:52.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.866571+0000) 2019-10-13T08:32:52.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:52.932+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.933022+0000) 2019-10-13T08:32:57.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:32:57.456+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.457144+0000) 2019-10-13T08:32:57.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:57.456+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.457338+0000) 2019-10-13T08:32:57.458 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:57.457+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.457778+0000) 2019-10-13T08:32:57.459 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:57.457+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.457916+0000) 2019-10-13T08:32:57.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:57.457+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.458103+0000) 2019-10-13T08:32:57.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:32:57.458+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.458495+0000) 2019-10-13T08:32:57.460 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:57.458+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.458667+0000) 2019-10-13T08:32:57.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:32:57.460+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.461303+0000) 2019-10-13T08:32:57.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:32:57.462+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 145977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:32:57.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:32:57.871+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.871614+0000) 2019-10-13T08:32:58.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:32:58.419+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.420279+0000) 2019-10-13T08:32:58.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:32:58.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.758737+0000) 2019-10-13T08:32:58.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:32:58.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.758737+0000) 2019-10-13T08:32:58.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:32:58.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:58.758772+0000) 2019-10-13T08:33:02.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:02.463+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.464620+0000) 2019-10-13T08:33:02.466 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:02.464+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.464746+0000) 2019-10-13T08:33:02.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:02.464+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.464959+0000) 2019-10-13T08:33:02.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:02.464+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.465240+0000) 2019-10-13T08:33:02.467 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:02.465+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.465462+0000) 2019-10-13T08:33:02.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:02.465+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.465625+0000) 2019-10-13T08:33:02.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:02.465+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.465912+0000) 2019-10-13T08:33:02.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:02.466+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.467396+0000) 2019-10-13T08:33:02.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:02.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:02.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:02.866+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.866699+0000) 2019-10-13T08:33:02.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:02.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.866738+0000) 2019-10-13T08:33:02.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:02.934+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.935305+0000) 2019-10-13T08:33:06.483 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:33:06.484 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:06.488 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:33:06.488 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:07.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:07.472+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.473485+0000) 2019-10-13T08:33:07.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:07.473+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.473659+0000) 2019-10-13T08:33:07.476 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:07.473+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.473844+0000) 2019-10-13T08:33:07.476 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:07.473+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.474328+0000) 2019-10-13T08:33:07.476 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:07.474+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.474559+0000) 2019-10-13T08:33:07.477 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:07.474+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.474609+0000) 2019-10-13T08:33:07.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:07.474+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.475072+0000) 2019-10-13T08:33:07.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:07.475+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.476387+0000) 2019-10-13T08:33:07.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:07.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:07.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:07.871+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.871811+0000) 2019-10-13T08:33:08.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:08.420+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.420421+0000) 2019-10-13T08:33:08.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:08.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.758987+0000) 2019-10-13T08:33:08.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:08.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.758987+0000) 2019-10-13T08:33:08.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:08.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:08.758981+0000) 2019-10-13T08:33:12.481 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:12.480+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.480971+0000) 2019-10-13T08:33:12.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:12.479+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.481099+0000) 2019-10-13T08:33:12.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:12.479+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.481310+0000) 2019-10-13T08:33:12.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:12.480+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.481591+0000) 2019-10-13T08:33:12.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:12.480+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.481744+0000) 2019-10-13T08:33:12.483 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:12.480+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.482089+0000) 2019-10-13T08:33:12.483 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:12.480+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.482276+0000) 2019-10-13T08:33:12.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:12.482+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.483627+0000) 2019-10-13T08:33:12.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:12.485+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146067 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:12.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:12.866+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.866856+0000) 2019-10-13T08:33:12.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:12.865+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.866902+0000) 2019-10-13T08:33:12.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:12.937+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.937573+0000) 2019-10-13T08:33:17.490 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:17.487+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.488635+0000) 2019-10-13T08:33:17.490 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:17.487+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.488818+0000) 2019-10-13T08:33:17.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:17.487+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.488956+0000) 2019-10-13T08:33:17.490 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:17.487+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.489050+0000) 2019-10-13T08:33:17.491 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:17.488+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.489424+0000) 2019-10-13T08:33:17.491 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:17.487+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.489322+0000) 2019-10-13T08:33:17.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:17.488+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.489619+0000) 2019-10-13T08:33:17.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:17.490+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.492073+0000) 2019-10-13T08:33:17.494 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:17.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146096 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:17.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:17.871+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.871985+0000) 2019-10-13T08:33:18.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:18.419+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.420542+0000) 2019-10-13T08:33:18.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:18.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.759214+0000) 2019-10-13T08:33:18.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:18.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.759211+0000) 2019-10-13T08:33:18.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:18.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:18.759214+0000) 2019-10-13T08:33:22.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:22.494+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.496003+0000) 2019-10-13T08:33:22.497 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:22.494+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.495794+0000) 2019-10-13T08:33:22.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:22.494+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.496163+0000) 2019-10-13T08:33:22.498 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:22.495+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.496540+0000) 2019-10-13T08:33:22.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:22.495+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.496682+0000) 2019-10-13T08:33:22.498 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:22.495+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.496887+0000) 2019-10-13T08:33:22.498 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:22.495+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.497045+0000) 2019-10-13T08:33:22.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:22.497+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.498935+0000) 2019-10-13T08:33:22.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:22.501+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:22.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:22.866+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.867019+0000) 2019-10-13T08:33:22.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:22.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.867048+0000) 2019-10-13T08:33:22.940 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:22.939+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.939988+0000) 2019-10-13T08:33:27.505 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:27.502+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.504222+0000) 2019-10-13T08:33:27.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:27.503+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.504441+0000) 2019-10-13T08:33:27.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:27.502+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.504601+0000) 2019-10-13T08:33:27.506 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:27.503+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.504798+0000) 2019-10-13T08:33:27.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:27.503+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.504951+0000) 2019-10-13T08:33:27.507 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:27.503+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.505193+0000) 2019-10-13T08:33:27.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:27.503+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.505328+0000) 2019-10-13T08:33:27.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:27.505+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.506868+0000) 2019-10-13T08:33:27.511 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:27.509+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146154 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:27.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:27.871+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.872132+0000) 2019-10-13T08:33:28.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:28.419+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.420660+0000) 2019-10-13T08:33:28.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:28.758+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.759410+0000) 2019-10-13T08:33:28.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:28.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.759413+0000) 2019-10-13T08:33:28.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:28.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:28.759390+0000) 2019-10-13T08:33:32.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:32.510+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.512121+0000) 2019-10-13T08:33:32.513 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:32.510+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.512350+0000) 2019-10-13T08:33:32.513 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:32.511+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.512558+0000) 2019-10-13T08:33:32.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:32.512+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.512965+0000) 2019-10-13T08:33:32.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:32.511+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.513067+0000) 2019-10-13T08:33:32.515 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:32.511+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.513232+0000) 2019-10-13T08:33:32.515 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:32.512+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.513538+0000) 2019-10-13T08:33:32.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:32.514+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.515576+0000) 2019-10-13T08:33:32.519 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:32.516+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:32.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:32.866+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.867156+0000) 2019-10-13T08:33:32.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:32.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.867214+0000) 2019-10-13T08:33:32.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:32.941+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.942424+0000) 2019-10-13T08:33:36.682 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:33:36.682 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:36.686 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:33:36.687 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:37.522 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:37.519+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.520448+0000) 2019-10-13T08:33:37.522 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:37.518+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.520308+0000) 2019-10-13T08:33:37.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:37.519+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.520553+0000) 2019-10-13T08:33:37.522 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:37.519+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.520720+0000) 2019-10-13T08:33:37.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:37.519+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.520818+0000) 2019-10-13T08:33:37.523 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:37.519+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.521060+0000) 2019-10-13T08:33:37.523 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:37.519+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.521269+0000) 2019-10-13T08:33:37.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:37.522+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.523109+0000) 2019-10-13T08:33:37.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:37.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:37.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:37.871+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.872304+0000) 2019-10-13T08:33:38.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:38.420+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.420826+0000) 2019-10-13T08:33:38.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:38.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.759618+0000) 2019-10-13T08:33:38.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:38.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.759618+0000) 2019-10-13T08:33:38.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:38.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:38.759608+0000) 2019-10-13T08:33:42.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:42.526+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.527117+0000) 2019-10-13T08:33:42.528 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:42.527+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.527558+0000) 2019-10-13T08:33:42.530 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:42.527+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.528053+0000) 2019-10-13T08:33:42.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:42.527+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.527803+0000) 2019-10-13T08:33:42.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:42.527+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.528077+0000) 2019-10-13T08:33:42.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:42.528+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.528411+0000) 2019-10-13T08:33:42.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:42.528+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.528586+0000) 2019-10-13T08:33:42.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:42.528+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.529610+0000) 2019-10-13T08:33:42.534 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:42.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:42.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:42.866+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.867310+0000) 2019-10-13T08:33:42.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:42.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.867408+0000) 2019-10-13T08:33:42.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:42.944+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.945243+0000) 2019-10-13T08:33:47.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:47.535+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.536144+0000) 2019-10-13T08:33:47.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:47.535+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.536290+0000) 2019-10-13T08:33:47.538 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:47.536+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.536644+0000) 2019-10-13T08:33:47.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:47.536+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.536895+0000) 2019-10-13T08:33:47.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:47.536+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.536914+0000) 2019-10-13T08:33:47.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:47.536+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.537036+0000) 2019-10-13T08:33:47.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:47.536+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.537338+0000) 2019-10-13T08:33:47.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:47.537+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.538688+0000) 2019-10-13T08:33:47.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:47.539+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146272 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:47.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:47.872+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.872471+0000) 2019-10-13T08:33:48.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:48.420+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.420968+0000) 2019-10-13T08:33:48.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:48.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.759848+0000) 2019-10-13T08:33:48.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:48.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.759848+0000) 2019-10-13T08:33:48.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:48.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:48.759843+0000) 2019-10-13T08:33:52.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:52.541+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.541932+0000) 2019-10-13T08:33:52.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:52.541+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.542225+0000) 2019-10-13T08:33:52.544 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:52.541+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.542124+0000) 2019-10-13T08:33:52.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:52.542+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.542405+0000) 2019-10-13T08:33:52.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:52.542+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.542580+0000) 2019-10-13T08:33:52.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:52.542+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.543217+0000) 2019-10-13T08:33:52.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:52.542+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.542897+0000) 2019-10-13T08:33:52.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:52.544+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.545105+0000) 2019-10-13T08:33:52.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:52.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146303 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:52.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:52.867+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.867474+0000) 2019-10-13T08:33:52.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:52.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.867584+0000) 2019-10-13T08:33:52.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:52.947+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.947796+0000) 2019-10-13T08:33:57.548 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:57.546+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.547747+0000) 2019-10-13T08:33:57.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:57.546+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.547891+0000) 2019-10-13T08:33:57.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:57.546+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.548120+0000) 2019-10-13T08:33:57.549 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:57.546+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.548271+0000) 2019-10-13T08:33:57.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:33:57.547+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.548146+0000) 2019-10-13T08:33:57.550 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:33:57.548+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.549665+0000) 2019-10-13T08:33:57.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:33:57.547+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.548412+0000) 2019-10-13T08:33:57.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:57.547+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.548582+0000) 2019-10-13T08:33:57.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:33:57.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:33:57.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:33:57.871+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.872625+0000) 2019-10-13T08:33:58.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:33:58.420+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.421152+0000) 2019-10-13T08:33:58.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:33:58.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.760053+0000) 2019-10-13T08:33:58.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:33:58.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.760045+0000) 2019-10-13T08:33:58.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:33:58.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:58.760054+0000) 2019-10-13T08:34:02.555 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:02.553+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.554490+0000) 2019-10-13T08:34:02.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:02.553+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.554657+0000) 2019-10-13T08:34:02.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:02.553+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.554702+0000) 2019-10-13T08:34:02.556 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:02.553+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.554917+0000) 2019-10-13T08:34:02.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:02.553+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.555266+0000) 2019-10-13T08:34:02.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:02.553+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.555236+0000) 2019-10-13T08:34:02.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:02.554+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.555666+0000) 2019-10-13T08:34:02.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:02.556+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.557215+0000) 2019-10-13T08:34:02.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:02.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:02.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:02.867+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.867616+0000) 2019-10-13T08:34:02.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:02.866+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.867742+0000) 2019-10-13T08:34:02.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:02.950+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.950126+0000) 2019-10-13T08:34:06.834 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:34:06.834 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:06.838 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:34:06.839 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:07.562 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:07.561+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.562281+0000) 2019-10-13T08:34:07.563 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:07.560+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.562350+0000) 2019-10-13T08:34:07.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:07.561+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.562631+0000) 2019-10-13T08:34:07.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:07.561+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.562802+0000) 2019-10-13T08:34:07.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:07.561+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.562964+0000) 2019-10-13T08:34:07.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:07.561+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.563319+0000) 2019-10-13T08:34:07.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:07.562+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.563522+0000) 2019-10-13T08:34:07.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:07.563+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.564616+0000) 2019-10-13T08:34:07.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:07.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:07.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:07.871+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.872783+0000) 2019-10-13T08:34:08.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:08.420+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.421329+0000) 2019-10-13T08:34:08.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:08.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.760235+0000) 2019-10-13T08:34:08.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:08.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.760236+0000) 2019-10-13T08:34:08.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:08.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:08.760269+0000) 2019-10-13T08:34:12.571 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:12.568+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.570154+0000) 2019-10-13T08:34:12.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:12.568+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.570295+0000) 2019-10-13T08:34:12.572 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:12.569+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.571339+0000) 2019-10-13T08:34:12.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:12.569+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.570750+0000) 2019-10-13T08:34:12.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:12.569+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.570904+0000) 2019-10-13T08:34:12.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:12.570+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.571517+0000) 2019-10-13T08:34:12.573 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:12.570+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.571648+0000) 2019-10-13T08:34:12.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:12.572+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.574085+0000) 2019-10-13T08:34:12.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:12.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146421 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:12.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:12.866+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.867777+0000) 2019-10-13T08:34:12.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:12.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.867875+0000) 2019-10-13T08:34:12.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:12.951+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.952073+0000) 2019-10-13T08:34:17.582 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:17.577+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.579243+0000) 2019-10-13T08:34:17.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:17.579+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.581591+0000) 2019-10-13T08:34:17.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:17.577+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.578637+0000) 2019-10-13T08:34:17.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:17.577+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.578912+0000) 2019-10-13T08:34:17.584 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:17.577+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.578985+0000) 2019-10-13T08:34:17.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:17.578+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.579424+0000) 2019-10-13T08:34:17.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:17.578+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.579783+0000) 2019-10-13T08:34:17.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:17.578+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.580200+0000) 2019-10-13T08:34:17.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:17.583+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:17.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:17.872+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.872951+0000) 2019-10-13T08:34:18.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:18.420+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.421502+0000) 2019-10-13T08:34:18.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:18.759+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.760418+0000) 2019-10-13T08:34:18.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:18.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.760418+0000) 2019-10-13T08:34:18.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:18.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:18.760416+0000) 2019-10-13T08:34:22.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:22.585+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.586865+0000) 2019-10-13T08:34:22.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:22.585+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.587245+0000) 2019-10-13T08:34:22.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:22.585+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.587384+0000) 2019-10-13T08:34:22.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:22.586+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.587709+0000) 2019-10-13T08:34:22.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:22.585+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.587639+0000) 2019-10-13T08:34:22.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:22.586+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.587927+0000) 2019-10-13T08:34:22.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:22.586+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.588110+0000) 2019-10-13T08:34:22.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:22.590+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.591360+0000) 2019-10-13T08:34:22.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:22.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:22.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:22.867+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.867938+0000) 2019-10-13T08:34:22.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:22.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.868038+0000) 2019-10-13T08:34:22.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:22.954+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.955020+0000) 2019-10-13T08:34:27.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:27.595+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.596204+0000) 2019-10-13T08:34:27.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:27.596+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.597055+0000) 2019-10-13T08:34:27.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:27.594+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.594953+0000) 2019-10-13T08:34:27.600 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:27.594+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.595140+0000) 2019-10-13T08:34:27.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:27.595+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.595611+0000) 2019-10-13T08:34:27.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:27.595+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.596345+0000) 2019-10-13T08:34:27.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:27.596+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.596479+0000) 2019-10-13T08:34:27.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:27.596+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.596832+0000) 2019-10-13T08:34:27.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:27.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146508 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:27.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:27.872+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.873084+0000) 2019-10-13T08:34:28.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:28.421+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.421713+0000) 2019-10-13T08:34:28.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:28.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.760610+0000) 2019-10-13T08:34:28.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:28.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.760609+0000) 2019-10-13T08:34:28.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:28.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:28.760608+0000) 2019-10-13T08:34:32.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:32.602+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.602929+0000) 2019-10-13T08:34:32.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:32.602+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.603602+0000) 2019-10-13T08:34:32.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:32.602+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.603376+0000) 2019-10-13T08:34:32.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:32.603+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.603719+0000) 2019-10-13T08:34:32.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:32.603+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.603928+0000) 2019-10-13T08:34:32.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:32.603+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.604159+0000) 2019-10-13T08:34:32.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:32.603+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.604127+0000) 2019-10-13T08:34:32.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:32.604+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.605207+0000) 2019-10-13T08:34:32.609 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:32.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:32.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:32.867+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.868125+0000) 2019-10-13T08:34:32.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:32.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.868200+0000) 2019-10-13T08:34:32.958 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:32.956+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.957353+0000) 2019-10-13T08:34:37.047 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:34:37.047 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:37.054 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:34:37.054 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:37.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:37.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.610697+0000) 2019-10-13T08:34:37.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:37.610+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.610748+0000) 2019-10-13T08:34:37.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:37.610+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.611157+0000) 2019-10-13T08:34:37.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:37.610+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.611236+0000) 2019-10-13T08:34:37.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:37.611+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.611436+0000) 2019-10-13T08:34:37.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:37.611+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.611705+0000) 2019-10-13T08:34:37.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:37.611+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.611619+0000) 2019-10-13T08:34:37.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:37.612+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.612966+0000) 2019-10-13T08:34:37.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:37.616+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146568 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:37.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:37.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.873256+0000) 2019-10-13T08:34:38.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:38.421+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.421882+0000) 2019-10-13T08:34:38.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:38.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.760809+0000) 2019-10-13T08:34:38.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:38.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.760809+0000) 2019-10-13T08:34:38.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:38.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:38.760808+0000) 2019-10-13T08:34:42.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:42.617+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.618136+0000) 2019-10-13T08:34:42.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:42.617+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.618251+0000) 2019-10-13T08:34:42.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:42.618+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.618489+0000) 2019-10-13T08:34:42.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:42.618+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.618732+0000) 2019-10-13T08:34:42.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:42.618+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.619096+0000) 2019-10-13T08:34:42.620 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:42.618+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.619345+0000) 2019-10-13T08:34:42.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:42.619+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.619587+0000) 2019-10-13T08:34:42.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:42.620+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.621329+0000) 2019-10-13T08:34:42.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:42.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:42.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:42.867+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.868275+0000) 2019-10-13T08:34:42.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:42.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.868350+0000) 2019-10-13T08:34:42.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:42.959+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.959831+0000) 2019-10-13T08:34:47.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:47.624+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.626130+0000) 2019-10-13T08:34:47.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:47.624+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.626325+0000) 2019-10-13T08:34:47.627 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:47.625+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.626479+0000) 2019-10-13T08:34:47.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:47.625+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.626795+0000) 2019-10-13T08:34:47.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:47.626+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.627047+0000) 2019-10-13T08:34:47.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:47.625+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.627109+0000) 2019-10-13T08:34:47.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:47.626+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.627428+0000) 2019-10-13T08:34:47.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:47.629+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.630153+0000) 2019-10-13T08:34:47.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:47.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146625 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:47.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:47.872+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.873446+0000) 2019-10-13T08:34:48.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:48.421+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.422057+0000) 2019-10-13T08:34:48.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:48.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.760985+0000) 2019-10-13T08:34:48.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:48.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.760965+0000) 2019-10-13T08:34:48.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:48.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:48.760967+0000) 2019-10-13T08:34:52.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:52.633+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.634105+0000) 2019-10-13T08:34:52.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:52.632+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.634130+0000) 2019-10-13T08:34:52.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:52.632+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.634330+0000) 2019-10-13T08:34:52.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:52.633+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.634717+0000) 2019-10-13T08:34:52.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:52.633+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.634812+0000) 2019-10-13T08:34:52.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:52.633+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.635220+0000) 2019-10-13T08:34:52.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:52.633+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.635281+0000) 2019-10-13T08:34:52.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:52.636+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.636758+0000) 2019-10-13T08:34:52.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:52.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146655 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:52.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:52.867+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.868509+0000) 2019-10-13T08:34:52.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:52.868+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.868429+0000) 2019-10-13T08:34:52.962 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:52.961+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.961971+0000) 2019-10-13T08:34:57.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:57.640+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.641419+0000) 2019-10-13T08:34:57.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:34:57.640+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.641663+0000) 2019-10-13T08:34:57.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:57.640+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.642118+0000) 2019-10-13T08:34:57.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:57.640+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.642029+0000) 2019-10-13T08:34:57.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:57.641+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.642559+0000) 2019-10-13T08:34:57.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:34:57.642+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.642964+0000) 2019-10-13T08:34:57.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:57.641+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.642957+0000) 2019-10-13T08:34:57.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:34:57.644+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.644926+0000) 2019-10-13T08:34:57.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:34:57.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:34:57.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:34:57.872+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.873602+0000) 2019-10-13T08:34:58.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:34:58.421+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.422238+0000) 2019-10-13T08:34:58.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:34:58.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.761153+0000) 2019-10-13T08:34:58.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:34:58.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.761153+0000) 2019-10-13T08:34:58.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:34:58.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:58.761147+0000) 2019-10-13T08:35:02.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:02.648+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.649453+0000) 2019-10-13T08:35:02.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:02.648+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.649638+0000) 2019-10-13T08:35:02.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:02.648+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.649773+0000) 2019-10-13T08:35:02.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:02.649+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.650796+0000) 2019-10-13T08:35:02.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:02.648+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.650223+0000) 2019-10-13T08:35:02.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:02.648+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.650284+0000) 2019-10-13T08:35:02.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:02.649+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.650493+0000) 2019-10-13T08:35:02.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:02.651+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.652987+0000) 2019-10-13T08:35:02.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:02.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:02.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:02.867+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.868591+0000) 2019-10-13T08:35:02.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:02.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.868666+0000) 2019-10-13T08:35:02.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:02.963+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.964369+0000) 2019-10-13T08:35:07.159 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:35:07.160 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:07.164 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:35:07.164 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:07.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:07.655+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.657102+0000) 2019-10-13T08:35:07.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:07.656+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.657422+0000) 2019-10-13T08:35:07.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:07.656+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.657539+0000) 2019-10-13T08:35:07.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:07.656+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.657702+0000) 2019-10-13T08:35:07.660 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:07.656+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.658177+0000) 2019-10-13T08:35:07.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:07.657+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.658425+0000) 2019-10-13T08:35:07.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:07.656+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.658203+0000) 2019-10-13T08:35:07.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:07.659+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.660875+0000) 2019-10-13T08:35:07.664 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:07.662+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:07.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:07.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.873784+0000) 2019-10-13T08:35:08.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:08.421+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.422436+0000) 2019-10-13T08:35:08.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:08.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.761331+0000) 2019-10-13T08:35:08.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:08.760+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.761369+0000) 2019-10-13T08:35:08.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:08.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:08.761369+0000) 2019-10-13T08:35:12.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:12.664+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.665362+0000) 2019-10-13T08:35:12.667 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:12.663+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.665666+0000) 2019-10-13T08:35:12.667 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:12.665+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.665883+0000) 2019-10-13T08:35:12.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:12.665+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.665928+0000) 2019-10-13T08:35:12.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:12.665+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.666194+0000) 2019-10-13T08:35:12.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:12.666+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.666466+0000) 2019-10-13T08:35:12.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:12.666+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.666511+0000) 2019-10-13T08:35:12.668 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:12.666+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.668288+0000) 2019-10-13T08:35:12.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:12.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:12.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:12.867+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.868769+0000) 2019-10-13T08:35:12.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:12.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.868823+0000) 2019-10-13T08:35:12.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:12.965+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.966618+0000) 2019-10-13T08:35:17.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:17.672+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.673133+0000) 2019-10-13T08:35:17.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:17.673+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.673416+0000) 2019-10-13T08:35:17.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:17.672+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.673613+0000) 2019-10-13T08:35:17.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:17.673+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.673695+0000) 2019-10-13T08:35:17.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:17.673+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.673863+0000) 2019-10-13T08:35:17.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:17.673+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.674227+0000) 2019-10-13T08:35:17.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:17.674+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.674599+0000) 2019-10-13T08:35:17.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:17.675+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.676456+0000) 2019-10-13T08:35:17.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:17.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:17.881 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:17.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.873986+0000) 2019-10-13T08:35:18.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:18.422+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.422632+0000) 2019-10-13T08:35:18.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:18.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.761512+0000) 2019-10-13T08:35:18.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:18.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.761545+0000) 2019-10-13T08:35:18.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:18.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:18.761545+0000) 2019-10-13T08:35:22.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:22.680+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.681180+0000) 2019-10-13T08:35:22.682 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:22.681+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.681577+0000) 2019-10-13T08:35:22.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:22.681+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.681812+0000) 2019-10-13T08:35:22.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:22.681+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.682716+0000) 2019-10-13T08:35:22.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:22.681+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.681915+0000) 2019-10-13T08:35:22.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:22.681+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.682192+0000) 2019-10-13T08:35:22.684 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:22.682+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.682665+0000) 2019-10-13T08:35:22.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:22.683+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.683817+0000) 2019-10-13T08:35:22.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:22.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:22.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:22.868+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.868944+0000) 2019-10-13T08:35:22.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:22.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.868988+0000) 2019-10-13T08:35:22.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:22.968+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.969109+0000) 2019-10-13T08:35:27.690 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:27.688+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.688974+0000) 2019-10-13T08:35:27.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:27.688+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.689154+0000) 2019-10-13T08:35:27.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:27.688+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.689306+0000) 2019-10-13T08:35:27.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:27.689+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.689605+0000) 2019-10-13T08:35:27.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:27.688+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.689570+0000) 2019-10-13T08:35:27.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:27.689+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.689775+0000) 2019-10-13T08:35:27.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:27.690+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.690424+0000) 2019-10-13T08:35:27.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:27.691+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.692442+0000) 2019-10-13T08:35:27.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:27.693+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:27.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:27.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.874144+0000) 2019-10-13T08:35:28.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:28.422+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.422802+0000) 2019-10-13T08:35:28.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:28.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.761748+0000) 2019-10-13T08:35:28.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:28.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.761748+0000) 2019-10-13T08:35:28.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:28.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:28.761745+0000) 2019-10-13T08:35:32.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:32.695+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.695980+0000) 2019-10-13T08:35:32.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:32.694+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.696205+0000) 2019-10-13T08:35:32.697 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:32.694+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.696333+0000) 2019-10-13T08:35:32.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:32.695+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.696530+0000) 2019-10-13T08:35:32.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:32.695+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.696694+0000) 2019-10-13T08:35:32.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:32.695+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.696838+0000) 2019-10-13T08:35:32.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:32.695+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.697228+0000) 2019-10-13T08:35:32.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:32.699+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.699997+0000) 2019-10-13T08:35:32.702 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:32.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:32.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:32.868+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.869108+0000) 2019-10-13T08:35:32.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:32.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.869159+0000) 2019-10-13T08:35:32.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:32.970+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.971143+0000) 2019-10-13T08:35:37.239 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:35:37.239 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:37.246 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:35:37.246 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:37.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:37.703+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.703878+0000) 2019-10-13T08:35:37.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:37.702+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.703906+0000) 2019-10-13T08:35:37.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:37.702+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.704232+0000) 2019-10-13T08:35:37.706 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:37.703+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.704533+0000) 2019-10-13T08:35:37.706 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:37.703+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.704729+0000) 2019-10-13T08:35:37.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:37.703+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.704919+0000) 2019-10-13T08:35:37.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:37.703+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.705047+0000) 2019-10-13T08:35:37.707 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:37.705+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.706428+0000) 2019-10-13T08:35:37.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:37.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:37.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:37.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.874293+0000) 2019-10-13T08:35:38.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:38.422+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.422943+0000) 2019-10-13T08:35:38.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:38.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.761932+0000) 2019-10-13T08:35:38.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:38.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.761929+0000) 2019-10-13T08:35:38.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:38.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:38.761933+0000) 2019-10-13T08:35:42.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:42.709+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.711103+0000) 2019-10-13T08:35:42.712 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:42.709+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.711318+0000) 2019-10-13T08:35:42.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:42.709+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.711372+0000) 2019-10-13T08:35:42.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:42.710+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.711496+0000) 2019-10-13T08:35:42.713 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:42.710+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.711570+0000) 2019-10-13T08:35:42.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:42.710+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.711704+0000) 2019-10-13T08:35:42.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:42.711+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.712245+0000) 2019-10-13T08:35:42.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:42.713+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.714473+0000) 2019-10-13T08:35:42.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:42.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:42.869 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:42.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.869260+0000) 2019-10-13T08:35:42.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:42.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.869306+0000) 2019-10-13T08:35:42.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:42.973+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.973435+0000) 2019-10-13T08:35:47.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:47.717+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.718489+0000) 2019-10-13T08:35:47.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:47.717+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.718707+0000) 2019-10-13T08:35:47.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:47.717+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.718814+0000) 2019-10-13T08:35:47.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:47.717+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.718991+0000) 2019-10-13T08:35:47.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:47.719+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.719749+0000) 2019-10-13T08:35:47.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:47.719+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.719889+0000) 2019-10-13T08:35:47.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:47.717+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.719345+0000) 2019-10-13T08:35:47.721 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:47.718+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.719932+0000) 2019-10-13T08:35:47.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:47.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 146973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:47.884 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:47.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.874466+0000) 2019-10-13T08:35:48.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:48.422+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.423105+0000) 2019-10-13T08:35:48.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:48.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.762119+0000) 2019-10-13T08:35:48.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:48.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.762120+0000) 2019-10-13T08:35:48.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:48.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:48.762120+0000) 2019-10-13T08:35:52.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:52.724+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.726361+0000) 2019-10-13T08:35:52.727 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:52.725+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.726652+0000) 2019-10-13T08:35:52.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:52.725+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.726887+0000) 2019-10-13T08:35:52.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:52.726+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.726792+0000) 2019-10-13T08:35:52.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:52.725+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.727277+0000) 2019-10-13T08:35:52.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:52.725+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.727381+0000) 2019-10-13T08:35:52.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:52.726+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.727478+0000) 2019-10-13T08:35:52.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:52.728+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.729244+0000) 2019-10-13T08:35:52.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:52.729+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147003 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:52.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:52.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.869412+0000) 2019-10-13T08:35:52.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:52.868+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.869465+0000) 2019-10-13T08:35:52.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:52.975+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.975758+0000) 2019-10-13T08:35:57.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:57.732+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.732666+0000) 2019-10-13T08:35:57.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:57.732+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.732914+0000) 2019-10-13T08:35:57.734 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:57.732+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.733133+0000) 2019-10-13T08:35:57.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:35:57.732+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.733854+0000) 2019-10-13T08:35:57.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:35:57.733+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.733561+0000) 2019-10-13T08:35:57.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:57.733+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.733841+0000) 2019-10-13T08:35:57.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:57.733+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.733968+0000) 2019-10-13T08:35:57.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:35:57.733+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.735664+0000) 2019-10-13T08:35:57.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:35:57.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147032 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:35:57.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:35:57.874+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.874579+0000) 2019-10-13T08:35:58.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:35:58.422+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.423280+0000) 2019-10-13T08:35:58.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:35:58.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.762330+0000) 2019-10-13T08:35:58.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:35:58.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.762308+0000) 2019-10-13T08:35:58.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:35:58.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:58.762330+0000) 2019-10-13T08:36:02.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:02.740+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.740620+0000) 2019-10-13T08:36:02.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:02.740+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.740781+0000) 2019-10-13T08:36:02.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:02.740+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.741173+0000) 2019-10-13T08:36:02.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:02.740+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.741024+0000) 2019-10-13T08:36:02.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:02.741+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.741642+0000) 2019-10-13T08:36:02.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:02.741+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.741781+0000) 2019-10-13T08:36:02.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:02.740+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.741990+0000) 2019-10-13T08:36:02.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:02.742+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.744403+0000) 2019-10-13T08:36:02.747 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:02.746+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147062 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:02.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:02.868+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.869570+0000) 2019-10-13T08:36:02.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:02.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.869611+0000) 2019-10-13T08:36:02.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:02.977+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.978081+0000) 2019-10-13T08:36:07.355 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:36:07.355 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:07.359 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:36:07.359 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:07.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:07.748+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.748465+0000) 2019-10-13T08:36:07.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:07.747+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.748890+0000) 2019-10-13T08:36:07.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:07.748+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.748591+0000) 2019-10-13T08:36:07.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:07.748+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.749280+0000) 2019-10-13T08:36:07.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:07.749+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.749500+0000) 2019-10-13T08:36:07.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:07.749+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.749650+0000) 2019-10-13T08:36:07.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:07.749+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.749773+0000) 2019-10-13T08:36:07.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:07.749+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.750910+0000) 2019-10-13T08:36:07.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:07.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147089 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:07.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:07.874+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.874729+0000) 2019-10-13T08:36:08.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:08.423+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.423451+0000) 2019-10-13T08:36:08.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:08.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.762497+0000) 2019-10-13T08:36:08.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:08.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.762498+0000) 2019-10-13T08:36:08.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:08.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:08.762491+0000) 2019-10-13T08:36:12.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:12.756+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.756860+0000) 2019-10-13T08:36:12.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:12.756+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.757117+0000) 2019-10-13T08:36:12.758 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:12.756+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.757315+0000) 2019-10-13T08:36:12.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:12.756+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.757125+0000) 2019-10-13T08:36:12.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:12.757+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.757610+0000) 2019-10-13T08:36:12.759 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:12.757+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.758111+0000) 2019-10-13T08:36:12.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:12.757+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.758196+0000) 2019-10-13T08:36:12.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:12.759+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.760088+0000) 2019-10-13T08:36:12.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:12.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147119 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:12.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:12.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.869747+0000) 2019-10-13T08:36:12.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:12.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.869745+0000) 2019-10-13T08:36:12.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:12.979+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.980540+0000) 2019-10-13T08:36:17.765 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:17.763+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.764725+0000) 2019-10-13T08:36:17.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:17.763+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.764999+0000) 2019-10-13T08:36:17.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:17.763+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.765199+0000) 2019-10-13T08:36:17.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:17.765+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.766121+0000) 2019-10-13T08:36:17.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:17.763+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.765363+0000) 2019-10-13T08:36:17.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:17.764+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.765583+0000) 2019-10-13T08:36:17.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:17.765+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.766474+0000) 2019-10-13T08:36:17.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:17.767+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.767949+0000) 2019-10-13T08:36:17.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:17.770+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147148 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:17.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:17.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.874901+0000) 2019-10-13T08:36:18.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:18.422+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.423661+0000) 2019-10-13T08:36:18.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:18.761+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.762704+0000) 2019-10-13T08:36:18.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:18.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.762700+0000) 2019-10-13T08:36:18.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:18.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:18.762705+0000) 2019-10-13T08:36:22.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:22.771+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.772890+0000) 2019-10-13T08:36:22.774 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:22.771+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.773224+0000) 2019-10-13T08:36:22.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:22.771+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.773110+0000) 2019-10-13T08:36:22.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:22.772+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.773408+0000) 2019-10-13T08:36:22.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:22.772+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.773607+0000) 2019-10-13T08:36:22.775 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:22.772+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.773763+0000) 2019-10-13T08:36:22.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:22.773+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.774018+0000) 2019-10-13T08:36:22.776 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:22.775+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.775837+0000) 2019-10-13T08:36:22.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:22.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:22.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:22.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.869886+0000) 2019-10-13T08:36:22.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:22.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.869944+0000) 2019-10-13T08:36:22.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:22.982+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.982796+0000) 2019-10-13T08:36:27.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:27.778+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.780129+0000) 2019-10-13T08:36:27.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:27.778+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.780226+0000) 2019-10-13T08:36:27.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:27.779+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.780457+0000) 2019-10-13T08:36:27.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:27.779+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.780700+0000) 2019-10-13T08:36:27.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:27.779+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.780747+0000) 2019-10-13T08:36:27.783 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:27.779+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.781218+0000) 2019-10-13T08:36:27.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:27.781+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.781851+0000) 2019-10-13T08:36:27.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:27.782+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.783669+0000) 2019-10-13T08:36:27.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:27.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:27.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:27.874+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.875096+0000) 2019-10-13T08:36:28.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:28.423+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.423829+0000) 2019-10-13T08:36:28.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:28.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.762895+0000) 2019-10-13T08:36:28.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:28.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.762899+0000) 2019-10-13T08:36:28.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:28.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:28.762899+0000) 2019-10-13T08:36:32.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:32.785+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.787136+0000) 2019-10-13T08:36:32.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:32.785+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.787309+0000) 2019-10-13T08:36:32.788 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:32.786+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.787429+0000) 2019-10-13T08:36:32.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:32.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.787835+0000) 2019-10-13T08:36:32.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:32.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.787922+0000) 2019-10-13T08:36:32.789 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:32.786+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.788116+0000) 2019-10-13T08:36:32.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:32.786+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.788256+0000) 2019-10-13T08:36:32.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:32.789+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.789901+0000) 2019-10-13T08:36:32.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:32.791+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:32.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:32.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.870044+0000) 2019-10-13T08:36:32.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:32.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.870088+0000) 2019-10-13T08:36:32.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:32.984+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.985109+0000) 2019-10-13T08:36:37.444 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:36:37.444 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:37.448 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:36:37.448 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:37.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:37.792+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.794221+0000) 2019-10-13T08:36:37.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:37.794+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.794800+0000) 2019-10-13T08:36:37.796 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:37.793+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.794786+0000) 2019-10-13T08:36:37.796 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:37.793+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.795079+0000) 2019-10-13T08:36:37.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:37.793+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.795308+0000) 2019-10-13T08:36:37.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:37.793+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.794906+0000) 2019-10-13T08:36:37.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:37.794+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.795497+0000) 2019-10-13T08:36:37.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:37.795+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.796517+0000) 2019-10-13T08:36:37.800 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:37.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147264 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:37.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:37.873+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.875307+0000) 2019-10-13T08:36:38.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:38.423+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.423997+0000) 2019-10-13T08:36:38.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:38.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.763009+0000) 2019-10-13T08:36:38.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:38.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.763009+0000) 2019-10-13T08:36:38.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:38.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:38.763047+0000) 2019-10-13T08:36:42.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:42.800+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.801420+0000) 2019-10-13T08:36:42.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:42.800+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.802193+0000) 2019-10-13T08:36:42.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:42.800+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.802194+0000) 2019-10-13T08:36:42.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:42.801+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.802565+0000) 2019-10-13T08:36:42.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:42.801+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.802831+0000) 2019-10-13T08:36:42.804 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:42.801+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.802917+0000) 2019-10-13T08:36:42.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:42.801+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.803051+0000) 2019-10-13T08:36:42.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:42.803+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.804086+0000) 2019-10-13T08:36:42.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:42.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:42.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:42.870+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.870221+0000) 2019-10-13T08:36:42.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:42.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.870247+0000) 2019-10-13T08:36:42.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:42.987+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.987249+0000) 2019-10-13T08:36:47.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:47.809+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.809670+0000) 2019-10-13T08:36:47.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:47.809+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.809877+0000) 2019-10-13T08:36:47.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:47.808+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.810248+0000) 2019-10-13T08:36:47.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:47.809+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.810281+0000) 2019-10-13T08:36:47.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:47.810+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.810490+0000) 2019-10-13T08:36:47.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:47.810+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.810813+0000) 2019-10-13T08:36:47.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:47.810+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.811147+0000) 2019-10-13T08:36:47.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:47.811+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.813221+0000) 2019-10-13T08:36:47.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:47.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:47.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:47.875+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.875481+0000) 2019-10-13T08:36:48.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:48.423+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.424135+0000) 2019-10-13T08:36:48.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:48.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.763189+0000) 2019-10-13T08:36:48.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:48.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.763189+0000) 2019-10-13T08:36:48.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:48.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:48.763188+0000) 2019-10-13T08:36:52.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:52.817+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.817633+0000) 2019-10-13T08:36:52.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:52.817+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.818001+0000) 2019-10-13T08:36:52.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:52.816+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.818606+0000) 2019-10-13T08:36:52.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:52.818+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.818730+0000) 2019-10-13T08:36:52.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:52.818+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.818966+0000) 2019-10-13T08:36:52.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:52.818+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.819000+0000) 2019-10-13T08:36:52.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:52.818+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.819184+0000) 2019-10-13T08:36:52.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:52.818+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.820132+0000) 2019-10-13T08:36:52.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:52.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:52.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:52.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.870337+0000) 2019-10-13T08:36:52.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:52.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.870382+0000) 2019-10-13T08:36:52.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:52.988+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.989667+0000) 2019-10-13T08:36:57.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:57.825+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.825652+0000) 2019-10-13T08:36:57.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:36:57.825+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.825855+0000) 2019-10-13T08:36:57.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:57.825+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.826055+0000) 2019-10-13T08:36:57.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:36:57.824+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.826105+0000) 2019-10-13T08:36:57.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:57.826+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.826607+0000) 2019-10-13T08:36:57.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:57.826+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.826830+0000) 2019-10-13T08:36:57.829 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:57.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.827048+0000) 2019-10-13T08:36:57.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:36:57.827+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.828984+0000) 2019-10-13T08:36:57.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:36:57.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147380 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:36:57.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:36:57.875+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.875640+0000) 2019-10-13T08:36:58.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:36:58.424+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.424300+0000) 2019-10-13T08:36:58.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:36:58.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.763378+0000) 2019-10-13T08:36:58.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:36:58.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.763377+0000) 2019-10-13T08:36:58.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:36:58.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:58.763361+0000) 2019-10-13T08:37:02.836 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:02.832+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.833613+0000) 2019-10-13T08:37:02.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:02.832+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.833867+0000) 2019-10-13T08:37:02.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:02.833+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.834786+0000) 2019-10-13T08:37:02.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:02.833+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.834829+0000) 2019-10-13T08:37:02.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:02.833+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.835021+0000) 2019-10-13T08:37:02.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:02.833+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.835173+0000) 2019-10-13T08:37:02.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:02.833+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.834390+0000) 2019-10-13T08:37:02.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:02.836+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.837521+0000) 2019-10-13T08:37:02.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:02.838+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:02.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:02.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.870462+0000) 2019-10-13T08:37:02.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:02.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.870545+0000) 2019-10-13T08:37:02.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:02.991+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.991918+0000) 2019-10-13T08:37:07.550 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:37:07.550 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:07.555 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:37:07.555 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:07.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:07.841+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.842562+0000) 2019-10-13T08:37:07.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:07.841+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.842693+0000) 2019-10-13T08:37:07.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:07.841+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.842956+0000) 2019-10-13T08:37:07.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:07.841+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.843090+0000) 2019-10-13T08:37:07.844 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:07.842+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.843766+0000) 2019-10-13T08:37:07.845 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:07.842+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.844157+0000) 2019-10-13T08:37:07.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:07.843+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.844641+0000) 2019-10-13T08:37:07.846 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:07.844+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.845561+0000) 2019-10-13T08:37:07.850 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:07.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147437 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:07.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:07.874+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.875787+0000) 2019-10-13T08:37:08.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:08.423+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.424498+0000) 2019-10-13T08:37:08.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:08.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.763515+0000) 2019-10-13T08:37:08.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:08.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.763571+0000) 2019-10-13T08:37:08.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:08.762+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:08.763593+0000) 2019-10-13T08:37:12.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:12.849+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.851245+0000) 2019-10-13T08:37:12.852 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:12.850+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.851426+0000) 2019-10-13T08:37:12.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:12.850+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.851593+0000) 2019-10-13T08:37:12.853 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:12.850+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.851728+0000) 2019-10-13T08:37:12.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:12.850+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.852048+0000) 2019-10-13T08:37:12.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:12.851+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.852068+0000) 2019-10-13T08:37:12.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:12.853+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.854143+0000) 2019-10-13T08:37:12.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:12.851+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.852485+0000) 2019-10-13T08:37:12.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:12.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147468 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:12.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:12.870+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.870593+0000) 2019-10-13T08:37:12.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:12.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.870697+0000) 2019-10-13T08:37:12.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:12.993+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.994149+0000) 2019-10-13T08:37:17.860 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:17.857+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.859241+0000) 2019-10-13T08:37:17.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:17.858+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.859688+0000) 2019-10-13T08:37:17.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:17.858+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.859789+0000) 2019-10-13T08:37:17.861 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:17.858+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.859918+0000) 2019-10-13T08:37:17.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:17.858+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.860292+0000) 2019-10-13T08:37:17.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:17.858+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.860248+0000) 2019-10-13T08:37:17.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:17.859+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.860634+0000) 2019-10-13T08:37:17.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:17.862+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.862918+0000) 2019-10-13T08:37:17.866 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:17.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:17.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:17.875+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.875897+0000) 2019-10-13T08:37:18.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:18.424+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.424668+0000) 2019-10-13T08:37:18.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:18.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.763729+0000) 2019-10-13T08:37:18.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:18.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.763684+0000) 2019-10-13T08:37:18.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:18.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:18.763735+0000) 2019-10-13T08:37:22.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:22.865+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.867331+0000) 2019-10-13T08:37:22.869 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:22.866+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.867891+0000) 2019-10-13T08:37:22.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:22.866+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.867792+0000) 2019-10-13T08:37:22.869 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:22.866+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.868201+0000) 2019-10-13T08:37:22.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:22.867+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.868433+0000) 2019-10-13T08:37:22.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:22.867+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.868606+0000) 2019-10-13T08:37:22.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:22.867+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.868512+0000) 2019-10-13T08:37:22.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:22.869+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.870720+0000) 2019-10-13T08:37:22.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:22.869+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.870808+0000) 2019-10-13T08:37:22.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:22.870+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.871443+0000) 2019-10-13T08:37:22.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:22.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:22.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:22.995+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.996303+0000) 2019-10-13T08:37:27.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:27.874+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.876012+0000) 2019-10-13T08:37:27.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:27.874+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.876176+0000) 2019-10-13T08:37:27.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:27.876+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.876856+0000) 2019-10-13T08:37:27.879 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:27.875+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.877176+0000) 2019-10-13T08:37:27.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:27.875+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.877280+0000) 2019-10-13T08:37:27.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:27.876+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.877852+0000) 2019-10-13T08:37:27.879 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:27.876+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.878081+0000) 2019-10-13T08:37:27.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:27.877+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.877997+0000) 2019-10-13T08:37:27.882 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:27.880+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:28.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:28.424+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.424814+0000) 2019-10-13T08:37:28.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:28.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.763842+0000) 2019-10-13T08:37:28.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:28.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.763821+0000) 2019-10-13T08:37:28.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:28.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:28.763861+0000) 2019-10-13T08:37:32.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:32.870+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.870851+0000) 2019-10-13T08:37:32.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:32.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.870969+0000) 2019-10-13T08:37:32.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:32.883+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.883645+0000) 2019-10-13T08:37:32.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:32.883+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.884052+0000) 2019-10-13T08:37:32.885 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:32.883+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.884105+0000) 2019-10-13T08:37:32.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:32.884+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.884576+0000) 2019-10-13T08:37:32.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:32.884+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.884635+0000) 2019-10-13T08:37:32.886 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:32.884+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.884391+0000) 2019-10-13T08:37:32.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:32.884+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.885050+0000) 2019-10-13T08:37:32.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:32.886+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.887184+0000) 2019-10-13T08:37:32.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:32.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:32.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:32.998+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.998261+0000) 2019-10-13T08:37:34.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:34.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:34.626328+0000) 2019-10-13T08:37:37.664 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:37:37.664 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:37.671 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:37:37.672 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:37.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:37.891+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.891920+0000) 2019-10-13T08:37:37.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:37.891+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.891878+0000) 2019-10-13T08:37:37.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:37.891+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.892154+0000) 2019-10-13T08:37:37.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:37.891+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.892357+0000) 2019-10-13T08:37:37.894 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:37.892+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.892629+0000) 2019-10-13T08:37:37.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:37.892+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.892645+0000) 2019-10-13T08:37:37.894 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:37.892+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.893052+0000) 2019-10-13T08:37:37.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:37.894+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.895335+0000) 2019-10-13T08:37:37.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:37.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:38.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:38.424+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.425021+0000) 2019-10-13T08:37:38.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:38.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.764002+0000) 2019-10-13T08:37:38.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:38.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.763986+0000) 2019-10-13T08:37:38.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:38.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:38.763988+0000) 2019-10-13T08:37:42.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:42.870+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.871047+0000) 2019-10-13T08:37:42.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:42.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.871184+0000) 2019-10-13T08:37:42.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:42.899+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.899657+0000) 2019-10-13T08:37:42.900 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:42.899+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.899745+0000) 2019-10-13T08:37:42.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:42.898+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.900451+0000) 2019-10-13T08:37:42.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:42.900+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.900404+0000) 2019-10-13T08:37:42.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:42.900+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.900618+0000) 2019-10-13T08:37:42.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:42.900+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.900788+0000) 2019-10-13T08:37:42.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:42.900+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.901033+0000) 2019-10-13T08:37:42.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:42.900+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.902203+0000) 2019-10-13T08:37:42.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:42.904+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147642 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:43.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:42.999+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.000376+0000) 2019-10-13T08:37:44.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:44.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:44.626515+0000) 2019-10-13T08:37:47.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:47.905+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.907369+0000) 2019-10-13T08:37:47.908 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:47.906+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.907642+0000) 2019-10-13T08:37:47.908 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:47.906+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.907863+0000) 2019-10-13T08:37:47.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:47.906+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.907948+0000) 2019-10-13T08:37:47.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:47.906+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.908091+0000) 2019-10-13T08:37:47.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:47.907+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.909312+0000) 2019-10-13T08:37:47.910 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:47.906+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.908321+0000) 2019-10-13T08:37:47.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:47.907+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.908577+0000) 2019-10-13T08:37:47.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:47.912+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:48.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:48.424+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.425197+0000) 2019-10-13T08:37:48.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:48.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.764210+0000) 2019-10-13T08:37:48.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:48.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.764213+0000) 2019-10-13T08:37:48.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:48.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:48.764212+0000) 2019-10-13T08:37:52.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:52.870+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.871227+0000) 2019-10-13T08:37:52.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:52.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.871311+0000) 2019-10-13T08:37:52.915 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:52.913+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.915044+0000) 2019-10-13T08:37:52.916 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:52.913+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.915199+0000) 2019-10-13T08:37:52.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:52.915+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.915821+0000) 2019-10-13T08:37:52.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:52.914+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.915561+0000) 2019-10-13T08:37:52.917 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:52.914+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.915781+0000) 2019-10-13T08:37:52.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:52.914+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.915928+0000) 2019-10-13T08:37:52.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:52.914+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.916356+0000) 2019-10-13T08:37:52.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:52.916+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.917185+0000) 2019-10-13T08:37:52.920 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:52.918+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:53.003 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:53.001+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.002484+0000) 2019-10-13T08:37:54.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:54.625+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:54.626651+0000) 2019-10-13T08:37:57.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:37:57.920+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.921398+0000) 2019-10-13T08:37:57.922 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:37:57.920+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.921457+0000) 2019-10-13T08:37:57.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:57.920+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.921750+0000) 2019-10-13T08:37:57.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:57.920+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.921963+0000) 2019-10-13T08:37:57.923 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:57.920+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.922148+0000) 2019-10-13T08:37:57.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:37:57.921+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.922578+0000) 2019-10-13T08:37:57.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:57.921+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.922760+0000) 2019-10-13T08:37:57.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:37:57.924+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.925352+0000) 2019-10-13T08:37:57.928 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:37:57.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:37:58.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:37:58.424+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.425372+0000) 2019-10-13T08:37:58.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:37:58.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.764388+0000) 2019-10-13T08:37:58.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:37:58.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.764389+0000) 2019-10-13T08:37:58.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:37:58.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:58.764389+0000) 2019-10-13T08:38:02.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:02.870+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.871415+0000) 2019-10-13T08:38:02.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:02.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.871474+0000) 2019-10-13T08:38:02.930 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:02.927+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.929300+0000) 2019-10-13T08:38:02.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:02.928+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.929486+0000) 2019-10-13T08:38:02.930 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:02.928+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.929709+0000) 2019-10-13T08:38:02.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:02.929+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.930412+0000) 2019-10-13T08:38:02.931 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:02.928+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.930022+0000) 2019-10-13T08:38:02.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:02.929+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.930664+0000) 2019-10-13T08:38:02.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:02.929+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.930896+0000) 2019-10-13T08:38:02.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:02.932+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.933334+0000) 2019-10-13T08:38:02.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:02.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:03.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:03.004+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.004601+0000) 2019-10-13T08:38:04.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:04.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:04.626827+0000) 2019-10-13T08:38:07.779 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:38:07.779 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:07.783 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:38:07.783 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:07.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:07.936+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.937459+0000) 2019-10-13T08:38:07.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:07.935+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.937339+0000) 2019-10-13T08:38:07.939 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:07.936+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.937565+0000) 2019-10-13T08:38:07.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:07.936+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.937681+0000) 2019-10-13T08:38:07.939 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:07.936+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.937816+0000) 2019-10-13T08:38:07.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:07.936+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.937945+0000) 2019-10-13T08:38:07.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:07.936+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.938137+0000) 2019-10-13T08:38:07.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:07.940+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.941182+0000) 2019-10-13T08:38:07.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:07.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:08.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:08.424+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.425539+0000) 2019-10-13T08:38:08.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:08.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.764562+0000) 2019-10-13T08:38:08.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:08.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.764561+0000) 2019-10-13T08:38:08.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:08.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:08.764577+0000) 2019-10-13T08:38:12.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:12.871+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.871595+0000) 2019-10-13T08:38:12.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:12.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.871623+0000) 2019-10-13T08:38:12.944 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:12.942+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.943393+0000) 2019-10-13T08:38:12.945 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:12.942+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.943622+0000) 2019-10-13T08:38:12.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:12.942+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.943870+0000) 2019-10-13T08:38:12.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:12.943+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.944192+0000) 2019-10-13T08:38:12.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:12.942+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.944231+0000) 2019-10-13T08:38:12.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:12.943+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.944442+0000) 2019-10-13T08:38:12.947 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:12.943+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.944583+0000) 2019-10-13T08:38:12.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:12.945+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.946234+0000) 2019-10-13T08:38:12.949 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:12.947+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147817 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:13.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:13.006+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.006681+0000) 2019-10-13T08:38:14.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:14.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:14.627035+0000) 2019-10-13T08:38:17.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:17.950+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.951128+0000) 2019-10-13T08:38:17.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:17.950+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.951634+0000) 2019-10-13T08:38:17.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:17.951+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.951665+0000) 2019-10-13T08:38:17.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:17.951+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.951899+0000) 2019-10-13T08:38:17.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:17.951+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.952034+0000) 2019-10-13T08:38:17.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:17.952+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.952391+0000) 2019-10-13T08:38:17.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:17.952+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.952507+0000) 2019-10-13T08:38:17.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:17.953+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.954430+0000) 2019-10-13T08:38:17.958 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:17.957+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:18.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:18.425+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.425710+0000) 2019-10-13T08:38:18.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:18.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.764729+0000) 2019-10-13T08:38:18.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:18.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.764726+0000) 2019-10-13T08:38:18.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:18.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:18.764727+0000) 2019-10-13T08:38:22.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:22.871+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.871798+0000) 2019-10-13T08:38:22.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:22.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.871793+0000) 2019-10-13T08:38:22.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:22.958+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.959348+0000) 2019-10-13T08:38:22.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:22.959+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.959703+0000) 2019-10-13T08:38:22.960 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:22.959+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.959828+0000) 2019-10-13T08:38:22.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:22.959+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.960297+0000) 2019-10-13T08:38:22.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:22.959+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.960205+0000) 2019-10-13T08:38:22.962 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:22.960+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.960497+0000) 2019-10-13T08:38:22.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:22.960+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.960700+0000) 2019-10-13T08:38:22.962 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:22.961+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.961783+0000) 2019-10-13T08:38:22.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:22.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:23.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:23.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.008811+0000) 2019-10-13T08:38:24.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:24.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:24.627191+0000) 2019-10-13T08:38:27.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:27.966+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.967296+0000) 2019-10-13T08:38:27.968 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:27.966+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.966918+0000) 2019-10-13T08:38:27.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:27.966+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.967304+0000) 2019-10-13T08:38:27.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:27.967+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.967411+0000) 2019-10-13T08:38:27.969 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:27.967+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.967653+0000) 2019-10-13T08:38:27.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:27.967+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.967921+0000) 2019-10-13T08:38:27.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:27.967+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.968083+0000) 2019-10-13T08:38:27.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:27.969+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.970090+0000) 2019-10-13T08:38:27.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:27.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:28.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:28.425+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.425907+0000) 2019-10-13T08:38:28.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:28.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.764894+0000) 2019-10-13T08:38:28.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:28.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.764892+0000) 2019-10-13T08:38:28.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:28.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:28.764895+0000) 2019-10-13T08:38:32.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:32.870+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.871965+0000) 2019-10-13T08:38:32.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:32.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.871970+0000) 2019-10-13T08:38:32.975 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:32.973+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.973922+0000) 2019-10-13T08:38:32.975 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:32.973+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.974226+0000) 2019-10-13T08:38:32.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:32.973+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.974326+0000) 2019-10-13T08:38:32.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:32.974+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.974483+0000) 2019-10-13T08:38:32.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:32.973+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.975192+0000) 2019-10-13T08:38:32.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:32.974+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.974809+0000) 2019-10-13T08:38:32.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:32.974+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.975085+0000) 2019-10-13T08:38:32.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:32.975+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.977505+0000) 2019-10-13T08:38:32.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:32.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:33.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:33.009+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.010992+0000) 2019-10-13T08:38:34.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:34.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:34.627377+0000) 2019-10-13T08:38:37.901 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:38:37.901 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:37.905 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:38:37.905 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:37.981 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:37.979+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.980423+0000) 2019-10-13T08:38:37.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:37.979+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.980707+0000) 2019-10-13T08:38:37.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:37.978+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.980719+0000) 2019-10-13T08:38:37.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:37.979+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.980872+0000) 2019-10-13T08:38:37.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:37.979+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.981011+0000) 2019-10-13T08:38:37.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:37.979+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.981275+0000) 2019-10-13T08:38:37.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:37.980+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.981534+0000) 2019-10-13T08:38:37.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:37.980+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.982430+0000) 2019-10-13T08:38:37.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:37.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:38.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:38.425+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.426104+0000) 2019-10-13T08:38:38.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:38.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.765070+0000) 2019-10-13T08:38:38.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:38.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.765067+0000) 2019-10-13T08:38:38.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:38.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:38.765072+0000) 2019-10-13T08:38:42.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:42.871+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.872157+0000) 2019-10-13T08:38:42.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:42.870+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.872144+0000) 2019-10-13T08:38:42.988 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:42.985+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.987023+0000) 2019-10-13T08:38:42.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:42.985+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.987237+0000) 2019-10-13T08:38:42.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:42.985+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.986889+0000) 2019-10-13T08:38:42.988 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:42.986+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.987427+0000) 2019-10-13T08:38:42.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:42.986+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.987950+0000) 2019-10-13T08:38:42.989 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:42.986+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.987902+0000) 2019-10-13T08:38:42.990 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:42.986+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.988166+0000) 2019-10-13T08:38:42.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:42.988+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.989986+0000) 2019-10-13T08:38:42.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:42.991+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 147981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:43.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:43.012+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.013111+0000) 2019-10-13T08:38:44.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:44.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:44.627584+0000) 2019-10-13T08:38:47.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:47.994+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.995479+0000) 2019-10-13T08:38:47.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:47.993+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.995137+0000) 2019-10-13T08:38:47.997 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:47.994+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.995672+0000) 2019-10-13T08:38:47.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:47.994+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.995763+0000) 2019-10-13T08:38:47.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:47.994+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.995867+0000) 2019-10-13T08:38:47.997 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:47.994+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.996303+0000) 2019-10-13T08:38:47.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:47.995+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.996388+0000) 2019-10-13T08:38:47.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:47.997+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.998529+0000) 2019-10-13T08:38:48.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:48.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148008 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:48.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:48.425+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.426274+0000) 2019-10-13T08:38:48.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:48.763+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.765234+0000) 2019-10-13T08:38:48.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:48.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.765234+0000) 2019-10-13T08:38:48.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:48.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:48.765234+0000) 2019-10-13T08:38:52.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:52.871+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.872356+0000) 2019-10-13T08:38:52.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:52.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.872375+0000) 2019-10-13T08:38:53.005 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:53.002+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.003632+0000) 2019-10-13T08:38:53.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:53.002+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.003979+0000) 2019-10-13T08:38:53.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:53.002+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.004063+0000) 2019-10-13T08:38:53.006 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:53.002+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.004195+0000) 2019-10-13T08:38:53.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:53.003+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.004404+0000) 2019-10-13T08:38:53.006 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:53.003+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.004755+0000) 2019-10-13T08:38:53.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:53.003+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.004677+0000) 2019-10-13T08:38:53.008 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:53.005+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.006145+0000) 2019-10-13T08:38:53.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:53.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148038 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:53.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:53.014+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.015267+0000) 2019-10-13T08:38:54.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:54.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:54.627755+0000) 2019-10-13T08:38:58.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:38:58.011+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.012275+0000) 2019-10-13T08:38:58.013 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:58.010+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.012232+0000) 2019-10-13T08:38:58.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:58.010+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.012353+0000) 2019-10-13T08:38:58.014 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:38:58.011+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.012582+0000) 2019-10-13T08:38:58.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:58.011+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.012675+0000) 2019-10-13T08:38:58.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:58.011+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.012871+0000) 2019-10-13T08:38:58.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:38:58.011+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.013109+0000) 2019-10-13T08:38:58.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:38:58.015+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.015837+0000) 2019-10-13T08:38:58.018 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:38:58.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:38:58.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:38:58.425+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.426457+0000) 2019-10-13T08:38:58.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:38:58.764+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.765385+0000) 2019-10-13T08:38:58.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:38:58.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.765385+0000) 2019-10-13T08:38:58.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:38:58.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:58.765388+0000) 2019-10-13T08:39:02.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:02.871+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.872553+0000) 2019-10-13T08:39:02.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:02.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.872572+0000) 2019-10-13T08:39:03.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:03.016+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.017405+0000) 2019-10-13T08:39:03.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:03.019+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.019792+0000) 2019-10-13T08:39:03.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:03.019+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.019895+0000) 2019-10-13T08:39:03.021 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:03.019+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.020167+0000) 2019-10-13T08:39:03.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:03.019+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.020250+0000) 2019-10-13T08:39:03.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:03.020+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.020479+0000) 2019-10-13T08:39:03.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:03.020+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.020804+0000) 2019-10-13T08:39:03.022 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:03.020+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.021024+0000) 2019-10-13T08:39:03.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:03.022+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.023020+0000) 2019-10-13T08:39:03.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:03.024+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148092 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:04.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:04.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:04.627914+0000) 2019-10-13T08:39:07.983 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:39:07.983 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:07.987 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:39:07.988 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:08.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:08.026+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.026850+0000) 2019-10-13T08:39:08.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:08.026+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.026948+0000) 2019-10-13T08:39:08.028 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:08.026+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.027117+0000) 2019-10-13T08:39:08.029 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:08.027+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.027421+0000) 2019-10-13T08:39:08.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:08.027+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.027674+0000) 2019-10-13T08:39:08.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:08.027+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.027863+0000) 2019-10-13T08:39:08.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:08.027+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.028027+0000) 2019-10-13T08:39:08.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:08.032+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.032834+0000) 2019-10-13T08:39:08.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:08.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:08.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:08.426+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.426646+0000) 2019-10-13T08:39:08.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:08.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.765595+0000) 2019-10-13T08:39:08.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:08.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.765599+0000) 2019-10-13T08:39:08.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:08.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:08.765595+0000) 2019-10-13T08:39:12.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:12.872+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.872749+0000) 2019-10-13T08:39:12.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:12.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.872738+0000) 2019-10-13T08:39:13.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:13.019+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.019967+0000) 2019-10-13T08:39:13.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:13.034+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.034459+0000) 2019-10-13T08:39:13.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:13.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.034794+0000) 2019-10-13T08:39:13.036 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:13.034+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.034882+0000) 2019-10-13T08:39:13.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:13.034+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.035088+0000) 2019-10-13T08:39:13.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:13.034+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.035235+0000) 2019-10-13T08:39:13.037 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:13.035+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.035628+0000) 2019-10-13T08:39:13.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:13.035+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.035863+0000) 2019-10-13T08:39:13.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:13.036+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.036947+0000) 2019-10-13T08:39:13.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:13.040+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148148 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:14.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:14.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.628104+0000) 2019-10-13T08:39:18.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:18.042+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.042615+0000) 2019-10-13T08:39:18.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:18.042+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.042886+0000) 2019-10-13T08:39:18.044 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:18.042+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.043093+0000) 2019-10-13T08:39:18.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:18.042+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.043643+0000) 2019-10-13T08:39:18.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:18.045+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.045863+0000) 2019-10-13T08:39:18.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:18.043+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.043813+0000) 2019-10-13T08:39:18.053 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:18.043+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.043943+0000) 2019-10-13T08:39:18.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:18.043+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.044069+0000) 2019-10-13T08:39:18.053 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:18.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148174 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:18.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:18.426+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.426825+0000) 2019-10-13T08:39:18.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:18.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.765785+0000) 2019-10-13T08:39:18.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:18.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.765786+0000) 2019-10-13T08:39:18.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:18.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:18.765771+0000) 2019-10-13T08:39:22.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:22.872+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.872944+0000) 2019-10-13T08:39:22.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:22.871+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.872921+0000) 2019-10-13T08:39:23.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:23.022+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.022414+0000) 2019-10-13T08:39:23.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:23.050+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.051025+0000) 2019-10-13T08:39:23.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:23.049+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.051086+0000) 2019-10-13T08:39:23.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:23.050+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.051394+0000) 2019-10-13T08:39:23.052 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:23.050+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.051417+0000) 2019-10-13T08:39:23.053 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:23.050+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.051567+0000) 2019-10-13T08:39:23.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:23.050+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.051813+0000) 2019-10-13T08:39:23.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:23.052+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.052903+0000) 2019-10-13T08:39:23.054 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:23.050+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.052186+0000) 2019-10-13T08:39:23.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:23.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148202 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:24.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:24.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.628283+0000) 2019-10-13T08:39:28.059 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:28.057+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.058652+0000) 2019-10-13T08:39:28.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:28.057+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.058907+0000) 2019-10-13T08:39:28.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:28.057+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.059038+0000) 2019-10-13T08:39:28.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:28.057+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.059572+0000) 2019-10-13T08:39:28.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:28.058+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.060296+0000) 2019-10-13T08:39:28.061 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:28.058+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.059485+0000) 2019-10-13T08:39:28.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:28.058+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.059846+0000) 2019-10-13T08:39:28.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:28.058+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.060153+0000) 2019-10-13T08:39:28.065 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:28.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148228 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:28.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:28.425+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.426993+0000) 2019-10-13T08:39:28.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:28.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.765962+0000) 2019-10-13T08:39:28.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:28.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.765962+0000) 2019-10-13T08:39:28.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:28.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:28.765958+0000) 2019-10-13T08:39:32.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:32.872+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.873172+0000) 2019-10-13T08:39:32.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:32.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.873102+0000) 2019-10-13T08:39:33.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:33.023+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.024808+0000) 2019-10-13T08:39:33.067 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:33.064+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.066266+0000) 2019-10-13T08:39:33.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:33.065+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.066875+0000) 2019-10-13T08:39:33.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:33.065+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.066799+0000) 2019-10-13T08:39:33.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:33.065+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.067182+0000) 2019-10-13T08:39:33.069 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:33.066+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.067403+0000) 2019-10-13T08:39:33.069 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:33.066+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.067598+0000) 2019-10-13T08:39:33.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:33.066+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.067887+0000) 2019-10-13T08:39:33.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:33.067+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.068760+0000) 2019-10-13T08:39:33.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:33.071+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148260 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:34.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:34.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.628443+0000) 2019-10-13T08:39:38.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:38.073+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.074425+0000) 2019-10-13T08:39:38.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:38.074+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.074819+0000) 2019-10-13T08:39:38.076 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:38.073+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.074881+0000) 2019-10-13T08:39:38.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:38.073+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.075178+0000) 2019-10-13T08:39:38.076 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:38.074+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.075394+0000) 2019-10-13T08:39:38.077 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:38.074+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.075598+0000) 2019-10-13T08:39:38.077 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:38.074+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.076032+0000) 2019-10-13T08:39:38.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:38.078+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.078782+0000) 2019-10-13T08:39:38.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:38.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148287 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:38.088 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:39:38.088 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:38.091 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:39:38.092 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:38.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:38.426+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.427214+0000) 2019-10-13T08:39:38.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:38.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.766122+0000) 2019-10-13T08:39:38.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:38.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.766126+0000) 2019-10-13T08:39:38.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:38.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.766119+0000) 2019-10-13T08:39:42.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:42.872+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:42.873349+0000) 2019-10-13T08:39:42.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:42.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:42.873288+0000) 2019-10-13T08:39:43.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:43.026+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.027253+0000) 2019-10-13T08:39:43.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:43.081+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.083321+0000) 2019-10-13T08:39:43.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:43.082+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.083884+0000) 2019-10-13T08:39:43.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:43.083+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.084567+0000) 2019-10-13T08:39:43.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:43.082+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.084148+0000) 2019-10-13T08:39:43.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:43.083+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.084753+0000) 2019-10-13T08:39:43.086 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:43.083+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.084827+0000) 2019-10-13T08:39:43.086 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:43.083+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.084969+0000) 2019-10-13T08:39:43.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:43.086+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.086934+0000) 2019-10-13T08:39:43.090 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:43.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:44.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:44.628+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.628581+0000) 2019-10-13T08:39:48.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:48.090+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.090866+0000) 2019-10-13T08:39:48.095 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:48.090+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.091128+0000) 2019-10-13T08:39:48.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:48.090+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.091230+0000) 2019-10-13T08:39:48.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:48.091+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.091426+0000) 2019-10-13T08:39:48.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:48.093+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.094051+0000) 2019-10-13T08:39:48.101 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:48.091+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.091490+0000) 2019-10-13T08:39:48.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:48.091+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.091619+0000) 2019-10-13T08:39:48.102 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:48.091+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.091941+0000) 2019-10-13T08:39:48.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:48.095+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148344 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:48.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:48.426+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.427374+0000) 2019-10-13T08:39:48.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:48.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.766294+0000) 2019-10-13T08:39:48.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:48.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.766286+0000) 2019-10-13T08:39:48.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:48.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.766294+0000) 2019-10-13T08:39:52.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:52.873+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:52.873553+0000) 2019-10-13T08:39:52.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:52.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:52.873483+0000) 2019-10-13T08:39:53.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:53.029+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.029698+0000) 2019-10-13T08:39:53.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:53.097+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.097568+0000) 2019-10-13T08:39:53.099 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:53.097+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.097711+0000) 2019-10-13T08:39:53.099 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:53.097+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.098247+0000) 2019-10-13T08:39:53.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:53.097+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.098239+0000) 2019-10-13T08:39:53.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:53.098+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.099697+0000) 2019-10-13T08:39:53.101 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:53.098+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.098510+0000) 2019-10-13T08:39:53.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:53.098+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.098612+0000) 2019-10-13T08:39:53.101 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:53.098+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.098839+0000) 2019-10-13T08:39:53.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:53.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:54.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:54.628+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.628750+0000) 2019-10-13T08:39:58.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:39:58.104+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.105636+0000) 2019-10-13T08:39:58.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:58.105+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.105620+0000) 2019-10-13T08:39:58.107 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:58.105+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.105933+0000) 2019-10-13T08:39:58.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:58.105+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.106236+0000) 2019-10-13T08:39:58.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:39:58.106+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.106411+0000) 2019-10-13T08:39:58.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:58.106+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.106508+0000) 2019-10-13T08:39:58.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:39:58.106+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.106729+0000) 2019-10-13T08:39:58.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:39:58.107+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.108073+0000) 2019-10-13T08:39:58.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:39:58.111+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:39:58.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:39:58.427+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.427574+0000) 2019-10-13T08:39:58.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:39:58.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.766453+0000) 2019-10-13T08:39:58.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:39:58.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.766457+0000) 2019-10-13T08:39:58.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:39:58.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.766458+0000) 2019-10-13T08:40:02.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:02.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:02.873665+0000) 2019-10-13T08:40:02.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:02.873+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:02.873764+0000) 2019-10-13T08:40:03.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:03.031+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.032134+0000) 2019-10-13T08:40:03.114 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:03.113+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.113639+0000) 2019-10-13T08:40:03.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:03.113+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.113837+0000) 2019-10-13T08:40:03.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:03.113+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.114559+0000) 2019-10-13T08:40:03.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:03.114+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.114708+0000) 2019-10-13T08:40:03.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:03.114+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.114577+0000) 2019-10-13T08:40:03.117 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:03.114+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.114897+0000) 2019-10-13T08:40:03.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:03.114+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.115342+0000) 2019-10-13T08:40:03.118 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:03.115+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.116285+0000) 2019-10-13T08:40:03.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:03.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148433 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:04.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:04.628+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.628864+0000) 2019-10-13T08:40:08.124 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:08.120+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.122275+0000) 2019-10-13T08:40:08.124 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:08.121+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.122435+0000) 2019-10-13T08:40:08.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:08.121+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.122605+0000) 2019-10-13T08:40:08.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:08.121+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.123009+0000) 2019-10-13T08:40:08.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:08.121+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.123123+0000) 2019-10-13T08:40:08.125 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:08.121+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.123296+0000) 2019-10-13T08:40:08.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:08.122+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.122965+0000) 2019-10-13T08:40:08.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:08.123+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.124581+0000) 2019-10-13T08:40:08.129 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:08.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:08.218 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:40:08.219 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:08.224 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:40:08.224 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:08.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:08.426+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.427727+0000) 2019-10-13T08:40:08.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:08.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.766632+0000) 2019-10-13T08:40:08.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:08.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.766631+0000) 2019-10-13T08:40:08.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:08.765+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.766633+0000) 2019-10-13T08:40:12.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:12.873+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:12.873933+0000) 2019-10-13T08:40:12.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:12.872+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:12.873892+0000) 2019-10-13T08:40:13.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:13.034+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.034455+0000) 2019-10-13T08:40:13.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:13.129+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.130452+0000) 2019-10-13T08:40:13.132 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:13.128+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.130390+0000) 2019-10-13T08:40:13.132 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:13.129+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.130910+0000) 2019-10-13T08:40:13.133 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:13.129+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.130996+0000) 2019-10-13T08:40:13.133 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:13.129+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.131069+0000) 2019-10-13T08:40:13.133 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:13.129+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.131231+0000) 2019-10-13T08:40:13.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:13.130+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.131604+0000) 2019-10-13T08:40:13.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:13.132+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.133594+0000) 2019-10-13T08:40:13.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:13.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:14.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:14.628+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.629036+0000) 2019-10-13T08:40:18.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:18.137+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.138717+0000) 2019-10-13T08:40:18.140 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:18.137+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.139097+0000) 2019-10-13T08:40:18.141 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:18.138+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.139638+0000) 2019-10-13T08:40:18.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:18.137+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.139627+0000) 2019-10-13T08:40:18.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:18.138+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.139905+0000) 2019-10-13T08:40:18.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:18.138+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.140057+0000) 2019-10-13T08:40:18.146 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:18.138+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.140163+0000) 2019-10-13T08:40:18.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:18.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148519 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:18.150 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:18.139+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.141488+0000) 2019-10-13T08:40:18.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:18.427+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.427857+0000) 2019-10-13T08:40:18.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:18.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.766808+0000) 2019-10-13T08:40:18.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:18.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.766806+0000) 2019-10-13T08:40:18.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:18.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.766807+0000) 2019-10-13T08:40:22.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:22.873+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:22.874073+0000) 2019-10-13T08:40:22.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:22.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:22.874077+0000) 2019-10-13T08:40:23.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:23.036+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.037100+0000) 2019-10-13T08:40:23.147 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:23.144+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.146336+0000) 2019-10-13T08:40:23.147 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:23.145+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.146783+0000) 2019-10-13T08:40:23.148 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:23.145+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.147402+0000) 2019-10-13T08:40:23.148 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:23.145+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.147198+0000) 2019-10-13T08:40:23.148 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:23.146+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.147440+0000) 2019-10-13T08:40:23.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:23.146+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.147791+0000) 2019-10-13T08:40:23.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:23.146+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.147971+0000) 2019-10-13T08:40:23.150 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:23.148+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.150596+0000) 2019-10-13T08:40:23.153 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:23.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:24.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:24.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.629167+0000) 2019-10-13T08:40:28.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:28.152+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.154141+0000) 2019-10-13T08:40:28.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:28.152+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.154307+0000) 2019-10-13T08:40:28.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:28.153+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.154634+0000) 2019-10-13T08:40:28.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:28.153+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.154728+0000) 2019-10-13T08:40:28.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:28.153+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.155048+0000) 2019-10-13T08:40:28.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:28.153+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.155292+0000) 2019-10-13T08:40:28.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:28.153+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.155439+0000) 2019-10-13T08:40:28.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:28.155+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.157034+0000) 2019-10-13T08:40:28.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:28.157+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148576 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:28.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:28.427+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.427971+0000) 2019-10-13T08:40:28.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:28.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.766967+0000) 2019-10-13T08:40:28.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:28.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.766977+0000) 2019-10-13T08:40:28.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:28.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.766968+0000) 2019-10-13T08:40:32.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:32.873+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:32.874257+0000) 2019-10-13T08:40:32.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:32.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:32.874262+0000) 2019-10-13T08:40:33.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:33.038+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.039505+0000) 2019-10-13T08:40:33.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:33.159+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.160633+0000) 2019-10-13T08:40:33.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:33.159+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.161152+0000) 2019-10-13T08:40:33.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:33.160+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.161652+0000) 2019-10-13T08:40:33.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:33.160+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.161813+0000) 2019-10-13T08:40:33.163 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:33.160+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.162039+0000) 2019-10-13T08:40:33.163 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:33.161+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.162149+0000) 2019-10-13T08:40:33.163 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:33.161+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.162473+0000) 2019-10-13T08:40:33.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:33.162+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.163682+0000) 2019-10-13T08:40:33.167 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:33.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:34.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:34.628+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.629346+0000) 2019-10-13T08:40:38.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:38.167+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.168111+0000) 2019-10-13T08:40:38.169 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:38.168+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.168443+0000) 2019-10-13T08:40:38.169 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:38.167+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.168295+0000) 2019-10-13T08:40:38.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:38.168+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.168702+0000) 2019-10-13T08:40:38.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:38.168+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.168906+0000) 2019-10-13T08:40:38.171 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:38.168+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.168827+0000) 2019-10-13T08:40:38.171 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:38.168+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.169245+0000) 2019-10-13T08:40:38.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:38.170+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.171226+0000) 2019-10-13T08:40:38.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:38.172+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:38.390 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:40:38.390 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:38.395 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:40:38.395 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:38.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:38.427+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.428087+0000) 2019-10-13T08:40:38.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:38.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.767125+0000) 2019-10-13T08:40:38.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:38.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.767129+0000) 2019-10-13T08:40:38.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:38.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.767125+0000) 2019-10-13T08:40:42.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:42.873+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:42.874464+0000) 2019-10-13T08:40:42.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:42.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:42.874472+0000) 2019-10-13T08:40:43.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:43.041+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.041870+0000) 2019-10-13T08:40:43.175 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:43.174+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.175088+0000) 2019-10-13T08:40:43.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:43.174+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.175103+0000) 2019-10-13T08:40:43.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:43.174+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.175310+0000) 2019-10-13T08:40:43.177 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:43.175+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.175635+0000) 2019-10-13T08:40:43.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:43.175+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.175655+0000) 2019-10-13T08:40:43.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:43.175+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.175899+0000) 2019-10-13T08:40:43.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:43.175+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.176167+0000) 2019-10-13T08:40:43.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:43.177+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.177809+0000) 2019-10-13T08:40:43.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:43.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:44.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:44.629+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.629557+0000) 2019-10-13T08:40:48.184 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:48.182+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.183293+0000) 2019-10-13T08:40:48.184 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:48.183+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.183480+0000) 2019-10-13T08:40:48.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:48.183+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.183935+0000) 2019-10-13T08:40:48.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:48.183+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.183865+0000) 2019-10-13T08:40:48.186 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:48.183+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.184052+0000) 2019-10-13T08:40:48.186 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:48.183+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.184301+0000) 2019-10-13T08:40:48.187 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:48.184+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.184822+0000) 2019-10-13T08:40:48.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:48.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148692 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:48.190 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:48.186+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.187457+0000) 2019-10-13T08:40:48.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:48.428+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.428263+0000) 2019-10-13T08:40:48.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:48.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.767277+0000) 2019-10-13T08:40:48.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:48.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.767276+0000) 2019-10-13T08:40:48.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:48.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.767276+0000) 2019-10-13T08:40:52.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:52.874+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:52.874636+0000) 2019-10-13T08:40:52.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:52.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:52.874674+0000) 2019-10-13T08:40:53.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:53.044+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.044366+0000) 2019-10-13T08:40:53.192 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:53.189+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.191046+0000) 2019-10-13T08:40:53.192 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:53.189+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.191233+0000) 2019-10-13T08:40:53.192 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:53.190+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.191423+0000) 2019-10-13T08:40:53.192 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:53.190+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.191571+0000) 2019-10-13T08:40:53.193 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:53.191+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.191882+0000) 2019-10-13T08:40:53.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:53.190+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.192155+0000) 2019-10-13T08:40:53.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:53.190+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.191999+0000) 2019-10-13T08:40:53.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:53.193+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.193890+0000) 2019-10-13T08:40:53.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:53.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148724 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:54.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:54.628+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.629743+0000) 2019-10-13T08:40:58.199 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:40:58.198+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.198858+0000) 2019-10-13T08:40:58.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:58.197+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.198894+0000) 2019-10-13T08:40:58.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:58.197+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.199000+0000) 2019-10-13T08:40:58.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:40:58.197+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.199282+0000) 2019-10-13T08:40:58.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:40:58.197+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.199355+0000) 2019-10-13T08:40:58.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:58.198+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.199497+0000) 2019-10-13T08:40:58.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:40:58.200+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.201048+0000) 2019-10-13T08:40:58.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:58.198+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.200036+0000) 2019-10-13T08:40:58.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:40:58.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148751 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:40:58.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:40:58.427+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.428458+0000) 2019-10-13T08:40:58.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:40:58.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.767431+0000) 2019-10-13T08:40:58.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:40:58.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.767433+0000) 2019-10-13T08:40:58.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:40:58.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.767431+0000) 2019-10-13T08:41:02.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:02.874+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:02.874818+0000) 2019-10-13T08:41:02.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:02.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:02.874858+0000) 2019-10-13T08:41:03.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:03.046+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.046797+0000) 2019-10-13T08:41:03.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:03.206+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.206822+0000) 2019-10-13T08:41:03.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:03.205+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.206693+0000) 2019-10-13T08:41:03.208 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:03.205+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.206811+0000) 2019-10-13T08:41:03.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:03.205+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.206935+0000) 2019-10-13T08:41:03.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:03.205+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.207281+0000) 2019-10-13T08:41:03.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:03.206+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.207439+0000) 2019-10-13T08:41:03.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:03.206+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.207704+0000) 2019-10-13T08:41:03.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:03.209+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.210197+0000) 2019-10-13T08:41:03.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:03.209+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148781 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:04.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:04.629+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.629902+0000) 2019-10-13T08:41:08.214 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:08.211+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.212796+0000) 2019-10-13T08:41:08.214 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:08.211+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.212979+0000) 2019-10-13T08:41:08.214 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:08.211+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.213209+0000) 2019-10-13T08:41:08.214 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:08.211+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.213364+0000) 2019-10-13T08:41:08.215 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:08.212+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.214149+0000) 2019-10-13T08:41:08.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:08.212+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.213782+0000) 2019-10-13T08:41:08.215 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:08.212+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.214048+0000) 2019-10-13T08:41:08.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:08.215+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.217218+0000) 2019-10-13T08:41:08.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:08.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148809 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:08.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:08.427+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.428604+0000) 2019-10-13T08:41:08.528 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:41:08.528 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:08.533 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:41:08.533 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:08.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:08.766+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.767586+0000) 2019-10-13T08:41:08.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:08.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.767591+0000) 2019-10-13T08:41:08.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:08.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.767585+0000) 2019-10-13T08:41:12.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:12.874+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:12.874987+0000) 2019-10-13T08:41:12.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:12.873+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:12.875007+0000) 2019-10-13T08:41:13.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:13.048+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.049172+0000) 2019-10-13T08:41:13.222 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:13.219+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.221360+0000) 2019-10-13T08:41:13.223 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:13.220+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.222177+0000) 2019-10-13T08:41:13.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:13.220+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.221898+0000) 2019-10-13T08:41:13.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:13.220+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.222060+0000) 2019-10-13T08:41:13.224 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:13.220+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.222162+0000) 2019-10-13T08:41:13.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:13.220+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.222309+0000) 2019-10-13T08:41:13.225 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:13.221+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.222695+0000) 2019-10-13T08:41:13.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:13.222+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.223853+0000) 2019-10-13T08:41:13.228 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:13.226+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:14.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:14.629+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.630062+0000) 2019-10-13T08:41:18.230 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:18.227+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.229439+0000) 2019-10-13T08:41:18.231 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:18.228+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.229471+0000) 2019-10-13T08:41:18.231 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:18.228+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.229610+0000) 2019-10-13T08:41:18.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:18.228+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.229878+0000) 2019-10-13T08:41:18.231 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:18.228+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.230081+0000) 2019-10-13T08:41:18.231 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:18.228+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.230118+0000) 2019-10-13T08:41:18.232 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:18.228+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.230239+0000) 2019-10-13T08:41:18.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:18.231+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.233550+0000) 2019-10-13T08:41:18.236 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:18.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:18.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:18.428+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.428739+0000) 2019-10-13T08:41:18.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:18.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.767727+0000) 2019-10-13T08:41:18.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:18.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.767728+0000) 2019-10-13T08:41:18.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:18.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.767729+0000) 2019-10-13T08:41:22.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:22.874+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:22.875158+0000) 2019-10-13T08:41:22.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:22.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:22.875182+0000) 2019-10-13T08:41:23.052 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:23.050+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.051496+0000) 2019-10-13T08:41:23.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:23.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.237213+0000) 2019-10-13T08:41:23.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:23.236+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.237210+0000) 2019-10-13T08:41:23.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:23.236+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.237337+0000) 2019-10-13T08:41:23.239 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:23.237+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.237531+0000) 2019-10-13T08:41:23.239 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:23.237+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.237812+0000) 2019-10-13T08:41:23.239 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:23.237+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.238318+0000) 2019-10-13T08:41:23.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:23.238+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.238591+0000) 2019-10-13T08:41:23.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:23.239+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.240280+0000) 2019-10-13T08:41:23.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:23.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148899 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:24.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:24.629+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.630265+0000) 2019-10-13T08:41:28.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:28.244+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.245159+0000) 2019-10-13T08:41:28.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:28.245+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.245530+0000) 2019-10-13T08:41:28.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:28.244+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.245613+0000) 2019-10-13T08:41:28.247 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:28.245+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.245835+0000) 2019-10-13T08:41:28.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:28.246+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.246494+0000) 2019-10-13T08:41:28.247 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:28.246+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.246572+0000) 2019-10-13T08:41:28.248 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:28.246+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.246731+0000) 2019-10-13T08:41:28.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:28.247+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.247990+0000) 2019-10-13T08:41:28.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:28.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:28.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:28.428+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.428925+0000) 2019-10-13T08:41:28.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:28.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.767915+0000) 2019-10-13T08:41:28.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:28.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.767913+0000) 2019-10-13T08:41:28.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:28.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.767915+0000) 2019-10-13T08:41:32.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:32.874+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:32.875339+0000) 2019-10-13T08:41:32.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:32.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:32.875321+0000) 2019-10-13T08:41:33.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:33.053+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.053839+0000) 2019-10-13T08:41:33.253 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:33.252+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.252801+0000) 2019-10-13T08:41:33.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:33.252+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.253009+0000) 2019-10-13T08:41:33.254 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:33.252+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.253180+0000) 2019-10-13T08:41:33.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:33.253+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.253826+0000) 2019-10-13T08:41:33.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:33.253+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.253484+0000) 2019-10-13T08:41:33.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:33.253+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.253894+0000) 2019-10-13T08:41:33.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:33.253+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.254247+0000) 2019-10-13T08:41:33.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:33.255+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.256225+0000) 2019-10-13T08:41:33.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:33.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:34.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:34.630+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.630371+0000) 2019-10-13T08:41:38.261 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:38.259+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.260681+0000) 2019-10-13T08:41:38.261 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:38.259+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.260955+0000) 2019-10-13T08:41:38.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:38.261+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.261874+0000) 2019-10-13T08:41:38.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:38.259+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.261030+0000) 2019-10-13T08:41:38.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:38.260+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.261818+0000) 2019-10-13T08:41:38.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:38.260+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.262077+0000) 2019-10-13T08:41:38.263 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:38.261+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.262415+0000) 2019-10-13T08:41:38.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:38.263+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.263835+0000) 2019-10-13T08:41:38.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:38.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 148987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:38.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:38.428+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.429120+0000) 2019-10-13T08:41:38.610 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:41:38.610 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:38.614 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:41:38.614 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:38.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:38.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.768100+0000) 2019-10-13T08:41:38.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:38.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.768095+0000) 2019-10-13T08:41:38.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:38.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.768104+0000) 2019-10-13T08:41:42.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:42.875+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:42.875523+0000) 2019-10-13T08:41:42.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:42.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:42.875508+0000) 2019-10-13T08:41:43.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:43.055+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.056301+0000) 2019-10-13T08:41:43.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:43.268+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.268773+0000) 2019-10-13T08:41:43.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:43.267+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.268955+0000) 2019-10-13T08:41:43.270 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:43.267+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.269288+0000) 2019-10-13T08:41:43.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:43.268+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.269530+0000) 2019-10-13T08:41:43.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:43.268+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.269809+0000) 2019-10-13T08:41:43.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:43.268+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.269972+0000) 2019-10-13T08:41:43.271 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:43.268+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.270355+0000) 2019-10-13T08:41:43.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:43.270+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.270821+0000) 2019-10-13T08:41:43.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:43.273+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:44.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:44.629+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.630474+0000) 2019-10-13T08:41:48.277 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:48.276+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.277142+0000) 2019-10-13T08:41:48.278 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:48.275+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.277038+0000) 2019-10-13T08:41:48.278 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:48.276+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.277449+0000) 2019-10-13T08:41:48.278 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:48.276+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.277715+0000) 2019-10-13T08:41:48.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:48.276+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.278031+0000) 2019-10-13T08:41:48.279 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:48.276+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.278174+0000) 2019-10-13T08:41:48.280 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:48.277+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.278448+0000) 2019-10-13T08:41:48.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:48.279+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.280204+0000) 2019-10-13T08:41:48.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:48.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149045 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:48.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:48.428+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.429267+0000) 2019-10-13T08:41:48.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:48.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.768252+0000) 2019-10-13T08:41:48.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:48.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.768254+0000) 2019-10-13T08:41:48.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:48.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.768255+0000) 2019-10-13T08:41:52.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:52.875+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:52.875749+0000) 2019-10-13T08:41:52.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:52.874+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:52.875680+0000) 2019-10-13T08:41:53.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:53.057+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.058610+0000) 2019-10-13T08:41:53.286 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:53.283+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.284919+0000) 2019-10-13T08:41:53.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:53.283+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.285238+0000) 2019-10-13T08:41:53.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:53.283+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.285356+0000) 2019-10-13T08:41:53.288 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:53.285+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.285830+0000) 2019-10-13T08:41:53.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:53.284+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.285982+0000) 2019-10-13T08:41:53.288 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:53.284+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.286144+0000) 2019-10-13T08:41:53.289 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:53.285+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.286828+0000) 2019-10-13T08:41:53.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:53.287+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.287898+0000) 2019-10-13T08:41:53.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:53.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:54.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:54.629+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.630612+0000) 2019-10-13T08:41:58.293 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:41:58.291+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.292494+0000) 2019-10-13T08:41:58.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:58.291+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.292697+0000) 2019-10-13T08:41:58.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:58.291+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.293001+0000) 2019-10-13T08:41:58.294 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:58.291+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.293090+0000) 2019-10-13T08:41:58.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:58.292+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.293474+0000) 2019-10-13T08:41:58.295 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:41:58.292+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.293626+0000) 2019-10-13T08:41:58.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:41:58.293+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.294223+0000) 2019-10-13T08:41:58.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:41:58.294+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.295566+0000) 2019-10-13T08:41:58.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:41:58.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:41:58.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:41:58.428+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.429443+0000) 2019-10-13T08:41:58.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:41:58.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.768417+0000) 2019-10-13T08:41:58.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:41:58.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.768413+0000) 2019-10-13T08:41:58.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:41:58.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.768418+0000) 2019-10-13T08:42:02.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:02.874+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:02.875958+0000) 2019-10-13T08:42:02.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:02.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:02.875880+0000) 2019-10-13T08:42:03.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:03.060+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.061617+0000) 2019-10-13T08:42:03.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:03.298+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.300294+0000) 2019-10-13T08:42:03.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:03.298+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.300326+0000) 2019-10-13T08:42:03.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:03.298+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.300221+0000) 2019-10-13T08:42:03.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:03.299+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.300528+0000) 2019-10-13T08:42:03.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:03.299+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.300678+0000) 2019-10-13T08:42:03.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:03.299+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.300887+0000) 2019-10-13T08:42:03.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:03.300+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.301405+0000) 2019-10-13T08:42:03.304 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:03.302+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.304250+0000) 2019-10-13T08:42:03.307 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:03.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:04.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:04.630+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.630781+0000) 2019-10-13T08:42:08.309 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:08.307+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.308234+0000) 2019-10-13T08:42:08.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:08.307+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.308778+0000) 2019-10-13T08:42:08.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:08.308+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.308955+0000) 2019-10-13T08:42:08.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:08.308+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.309027+0000) 2019-10-13T08:42:08.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:08.309+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.309424+0000) 2019-10-13T08:42:08.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:08.308+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.309144+0000) 2019-10-13T08:42:08.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:08.309+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.309833+0000) 2019-10-13T08:42:08.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:08.309+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.311165+0000) 2019-10-13T08:42:08.315 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:08.314+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:08.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:08.429+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.429581+0000) 2019-10-13T08:42:08.756 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:42:08.757 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:08.766 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:42:08.766 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:08.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:08.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.768541+0000) 2019-10-13T08:42:08.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:08.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.768551+0000) 2019-10-13T08:42:08.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:08.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.768550+0000) 2019-10-13T08:42:12.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:12.874+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:12.876111+0000) 2019-10-13T08:42:12.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:12.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:12.876010+0000) 2019-10-13T08:42:13.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:13.063+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.063906+0000) 2019-10-13T08:42:13.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:13.315+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.316307+0000) 2019-10-13T08:42:13.318 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:13.316+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.316666+0000) 2019-10-13T08:42:13.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:13.316+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.316743+0000) 2019-10-13T08:42:13.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:13.316+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.317075+0000) 2019-10-13T08:42:13.318 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:13.316+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.317186+0000) 2019-10-13T08:42:13.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:13.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.317148+0000) 2019-10-13T08:42:13.319 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:13.317+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.317528+0000) 2019-10-13T08:42:13.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:13.317+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.319707+0000) 2019-10-13T08:42:13.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:13.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:14.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:14.630+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.630967+0000) 2019-10-13T08:42:18.325 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:18.323+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.324275+0000) 2019-10-13T08:42:18.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:18.324+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.324865+0000) 2019-10-13T08:42:18.326 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:18.324+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.324894+0000) 2019-10-13T08:42:18.326 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:18.324+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.324957+0000) 2019-10-13T08:42:18.326 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:18.324+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.325374+0000) 2019-10-13T08:42:18.326 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:18.325+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.325556+0000) 2019-10-13T08:42:18.327 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:18.324+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.325651+0000) 2019-10-13T08:42:18.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:18.326+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.326878+0000) 2019-10-13T08:42:18.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:18.330+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:18.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:18.429+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.429741+0000) 2019-10-13T08:42:18.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:18.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.768632+0000) 2019-10-13T08:42:18.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:18.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.768679+0000) 2019-10-13T08:42:18.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:18.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.768668+0000) 2019-10-13T08:42:22.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:22.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:22.876199+0000) 2019-10-13T08:42:22.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:22.875+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:22.876263+0000) 2019-10-13T08:42:23.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:23.065+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.066375+0000) 2019-10-13T08:42:23.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:23.331+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.332325+0000) 2019-10-13T08:42:23.333 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:23.332+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.332429+0000) 2019-10-13T08:42:23.333 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:23.332+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.332553+0000) 2019-10-13T08:42:23.334 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:23.332+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.332918+0000) 2019-10-13T08:42:23.334 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:23.332+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.333060+0000) 2019-10-13T08:42:23.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:23.333+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.333799+0000) 2019-10-13T08:42:23.335 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:23.333+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.334345+0000) 2019-10-13T08:42:23.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:23.334+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.335552+0000) 2019-10-13T08:42:23.339 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:23.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:24.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:24.629+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.631084+0000) 2019-10-13T08:42:28.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:28.339+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.340600+0000) 2019-10-13T08:42:28.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:28.339+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.340790+0000) 2019-10-13T08:42:28.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:28.339+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.340948+0000) 2019-10-13T08:42:28.343 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:28.340+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.341334+0000) 2019-10-13T08:42:28.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:28.340+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.341515+0000) 2019-10-13T08:42:28.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:28.340+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.341588+0000) 2019-10-13T08:42:28.344 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:28.340+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.342091+0000) 2019-10-13T08:42:28.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:28.343+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.344715+0000) 2019-10-13T08:42:28.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:28.345+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:28.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:28.428+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.429913+0000) 2019-10-13T08:42:28.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:28.767+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.768820+0000) 2019-10-13T08:42:28.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:28.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.768814+0000) 2019-10-13T08:42:28.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:28.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.768853+0000) 2019-10-13T08:42:32.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:32.876+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:32.876467+0000) 2019-10-13T08:42:32.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:32.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:32.876352+0000) 2019-10-13T08:42:33.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:33.068+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.068810+0000) 2019-10-13T08:42:33.348 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:33.347+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.348428+0000) 2019-10-13T08:42:33.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:33.347+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.348434+0000) 2019-10-13T08:42:33.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:33.347+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.348585+0000) 2019-10-13T08:42:33.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:33.347+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.348960+0000) 2019-10-13T08:42:33.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:33.347+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.349189+0000) 2019-10-13T08:42:33.350 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:33.347+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.349379+0000) 2019-10-13T08:42:33.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:33.348+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.349703+0000) 2019-10-13T08:42:33.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:33.350+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.350870+0000) 2019-10-13T08:42:33.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:33.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:34.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:34.630+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.631192+0000) 2019-10-13T08:42:38.357 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:38.354+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.356114+0000) 2019-10-13T08:42:38.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:38.354+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.356289+0000) 2019-10-13T08:42:38.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:38.355+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.356583+0000) 2019-10-13T08:42:38.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:38.355+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.356620+0000) 2019-10-13T08:42:38.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:38.356+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.357017+0000) 2019-10-13T08:42:38.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:38.355+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.356897+0000) 2019-10-13T08:42:38.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:38.355+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.357136+0000) 2019-10-13T08:42:38.360 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:38.359+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.359900+0000) 2019-10-13T08:42:38.362 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:38.360+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149342 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:38.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:38.429+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.430053+0000) 2019-10-13T08:42:38.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:38.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.769028+0000) 2019-10-13T08:42:38.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:38.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.769030+0000) 2019-10-13T08:42:38.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:38.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.769029+0000) 2019-10-13T08:42:38.916 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:42:38.917 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:38.920 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:42:38.920 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:42.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:42.875+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:42.876664+0000) 2019-10-13T08:42:42.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:42.875+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:42.876526+0000) 2019-10-13T08:42:43.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:43.070+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.071161+0000) 2019-10-13T08:42:43.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:43.362+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.363570+0000) 2019-10-13T08:42:43.365 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:43.362+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.363618+0000) 2019-10-13T08:42:43.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:43.362+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.363990+0000) 2019-10-13T08:42:43.365 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:43.362+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.364149+0000) 2019-10-13T08:42:43.366 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:43.362+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.364268+0000) 2019-10-13T08:42:43.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:43.363+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.364468+0000) 2019-10-13T08:42:43.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:43.363+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.364535+0000) 2019-10-13T08:42:43.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:43.365+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.366306+0000) 2019-10-13T08:42:43.370 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:43.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:44.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:44.630+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.631378+0000) 2019-10-13T08:42:48.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:48.370+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.371460+0000) 2019-10-13T08:42:48.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:48.370+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.371657+0000) 2019-10-13T08:42:48.373 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:48.370+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.371657+0000) 2019-10-13T08:42:48.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:48.370+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.371928+0000) 2019-10-13T08:42:48.373 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:48.370+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.372072+0000) 2019-10-13T08:42:48.374 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:48.371+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.372049+0000) 2019-10-13T08:42:48.374 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:48.371+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.372523+0000) 2019-10-13T08:42:48.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:48.374+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.374836+0000) 2019-10-13T08:42:48.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:48.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:48.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:48.429+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.430193+0000) 2019-10-13T08:42:48.770 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:48.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.769199+0000) 2019-10-13T08:42:48.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:48.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.769198+0000) 2019-10-13T08:42:48.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:48.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.769185+0000) 2019-10-13T08:42:52.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:52.875+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:52.876851+0000) 2019-10-13T08:42:52.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:52.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:52.876698+0000) 2019-10-13T08:42:53.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:53.072+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.073562+0000) 2019-10-13T08:42:53.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:53.378+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.378539+0000) 2019-10-13T08:42:53.379 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:53.378+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.378598+0000) 2019-10-13T08:42:53.380 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:53.378+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.378694+0000) 2019-10-13T08:42:53.380 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:53.378+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.378872+0000) 2019-10-13T08:42:53.380 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:53.378+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.380309+0000) 2019-10-13T08:42:53.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:53.379+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.379737+0000) 2019-10-13T08:42:53.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:53.379+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.380265+0000) 2019-10-13T08:42:53.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:53.380+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.382540+0000) 2019-10-13T08:42:53.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:53.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:54.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:54.631+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.631551+0000) 2019-10-13T08:42:58.387 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:42:58.384+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.386714+0000) 2019-10-13T08:42:58.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:42:58.386+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.386850+0000) 2019-10-13T08:42:58.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:58.386+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.386960+0000) 2019-10-13T08:42:58.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:58.386+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.387151+0000) 2019-10-13T08:42:58.388 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:58.386+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.387368+0000) 2019-10-13T08:42:58.389 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:58.387+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.387522+0000) 2019-10-13T08:42:58.389 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:42:58.387+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.387692+0000) 2019-10-13T08:42:58.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:42:58.387+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.389004+0000) 2019-10-13T08:42:58.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:42:58.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:42:58.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:42:58.429+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.430375+0000) 2019-10-13T08:42:58.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:42:58.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.769362+0000) 2019-10-13T08:42:58.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:42:58.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.769324+0000) 2019-10-13T08:42:58.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:42:58.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.769400+0000) 2019-10-13T08:43:02.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:02.876+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:02.877050+0000) 2019-10-13T08:43:02.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:02.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:02.876885+0000) 2019-10-13T08:43:03.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:03.075+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.075851+0000) 2019-10-13T08:43:03.395 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:03.393+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.395092+0000) 2019-10-13T08:43:03.396 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:03.394+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.394991+0000) 2019-10-13T08:43:03.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:03.394+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.395237+0000) 2019-10-13T08:43:03.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:03.394+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.395344+0000) 2019-10-13T08:43:03.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:03.395+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.395599+0000) 2019-10-13T08:43:03.397 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:03.395+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.395946+0000) 2019-10-13T08:43:03.397 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:03.395+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.396039+0000) 2019-10-13T08:43:03.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:03.397+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.398785+0000) 2019-10-13T08:43:03.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:03.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:04.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:04.631+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.631686+0000) 2019-10-13T08:43:08.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:08.402+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.402912+0000) 2019-10-13T08:43:08.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:08.403+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.403751+0000) 2019-10-13T08:43:08.405 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:08.402+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.403345+0000) 2019-10-13T08:43:08.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:08.403+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.403623+0000) 2019-10-13T08:43:08.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:08.403+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.403798+0000) 2019-10-13T08:43:08.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:08.403+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.404068+0000) 2019-10-13T08:43:08.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:08.404+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.404534+0000) 2019-10-13T08:43:08.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:08.404+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.405205+0000) 2019-10-13T08:43:08.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:08.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:08.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:08.430+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.430576+0000) 2019-10-13T08:43:08.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:08.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.769492+0000) 2019-10-13T08:43:08.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:08.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.769491+0000) 2019-10-13T08:43:08.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:08.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.769519+0000) 2019-10-13T08:43:09.079 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:43:09.079 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:09.084 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:43:09.084 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:12.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:12.876+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:12.877219+0000) 2019-10-13T08:43:12.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:12.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:12.877107+0000) 2019-10-13T08:43:13.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:13.077+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.078178+0000) 2019-10-13T08:43:13.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:13.409+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.411278+0000) 2019-10-13T08:43:13.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:13.411+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.412030+0000) 2019-10-13T08:43:13.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:13.410+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.411927+0000) 2019-10-13T08:43:13.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:13.410+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.412232+0000) 2019-10-13T08:43:13.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:13.411+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.412491+0000) 2019-10-13T08:43:13.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:13.411+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.412613+0000) 2019-10-13T08:43:13.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:13.410+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.412062+0000) 2019-10-13T08:43:13.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:13.413+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.414674+0000) 2019-10-13T08:43:13.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:13.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:14.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:14.630+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.631858+0000) 2019-10-13T08:43:18.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:18.419+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.419781+0000) 2019-10-13T08:43:18.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:18.418+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.419706+0000) 2019-10-13T08:43:18.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:18.418+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.420151+0000) 2019-10-13T08:43:18.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:18.418+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.420275+0000) 2019-10-13T08:43:18.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:18.419+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.420680+0000) 2019-10-13T08:43:18.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:18.419+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.420736+0000) 2019-10-13T08:43:18.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:18.419+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.420974+0000) 2019-10-13T08:43:18.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:18.421+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.422288+0000) 2019-10-13T08:43:18.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:18.425+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:18.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:18.429+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.430687+0000) 2019-10-13T08:43:18.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:18.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.769689+0000) 2019-10-13T08:43:18.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:18.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.769688+0000) 2019-10-13T08:43:18.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:18.768+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.769691+0000) 2019-10-13T08:43:22.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:22.876+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:22.877392+0000) 2019-10-13T08:43:22.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:22.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:22.877291+0000) 2019-10-13T08:43:23.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:23.080+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.080582+0000) 2019-10-13T08:43:23.430 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:23.428+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.429535+0000) 2019-10-13T08:43:23.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:23.427+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.429140+0000) 2019-10-13T08:43:23.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:23.427+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.428442+0000) 2019-10-13T08:43:23.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:23.427+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.428597+0000) 2019-10-13T08:43:23.432 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:23.427+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.428983+0000) 2019-10-13T08:43:23.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:23.428+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.429407+0000) 2019-10-13T08:43:23.432 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:23.428+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.429807+0000) 2019-10-13T08:43:23.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:23.429+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.430625+0000) 2019-10-13T08:43:23.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:23.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:24.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:24.631+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.632019+0000) 2019-10-13T08:43:28.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:28.430+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.430843+0000) 2019-10-13T08:43:28.437 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:28.435+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.436511+0000) 2019-10-13T08:43:28.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:28.435+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.436673+0000) 2019-10-13T08:43:28.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:28.435+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.436868+0000) 2019-10-13T08:43:28.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:28.436+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.437236+0000) 2019-10-13T08:43:28.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:28.435+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.437315+0000) 2019-10-13T08:43:28.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:28.436+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.437679+0000) 2019-10-13T08:43:28.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:28.436+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.437870+0000) 2019-10-13T08:43:28.440 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:28.439+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.439833+0000) 2019-10-13T08:43:28.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:28.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:28.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:28.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.769838+0000) 2019-10-13T08:43:28.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:28.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.769839+0000) 2019-10-13T08:43:28.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:28.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.769839+0000) 2019-10-13T08:43:32.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:32.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:32.877543+0000) 2019-10-13T08:43:32.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:32.876+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:32.877487+0000) 2019-10-13T08:43:33.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:33.082+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.083095+0000) 2019-10-13T08:43:33.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:33.442+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.444148+0000) 2019-10-13T08:43:33.445 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:33.443+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.444559+0000) 2019-10-13T08:43:33.446 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:33.444+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.445363+0000) 2019-10-13T08:43:33.446 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:33.443+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.444900+0000) 2019-10-13T08:43:33.446 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:33.443+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.444892+0000) 2019-10-13T08:43:33.446 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:33.443+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.445294+0000) 2019-10-13T08:43:33.447 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:33.444+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.445665+0000) 2019-10-13T08:43:33.448 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:33.447+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.448639+0000) 2019-10-13T08:43:33.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:33.449+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:34.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:34.631+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.632188+0000) 2019-10-13T08:43:38.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:38.430+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.431031+0000) 2019-10-13T08:43:38.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:38.452+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.452590+0000) 2019-10-13T08:43:38.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:38.452+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.452768+0000) 2019-10-13T08:43:38.454 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:38.452+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.452912+0000) 2019-10-13T08:43:38.454 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:38.453+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.454233+0000) 2019-10-13T08:43:38.455 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:38.452+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.453204+0000) 2019-10-13T08:43:38.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:38.453+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.453602+0000) 2019-10-13T08:43:38.455 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:38.453+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.453655+0000) 2019-10-13T08:43:38.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:38.455+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.456621+0000) 2019-10-13T08:43:38.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:38.458+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:38.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:38.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.769952+0000) 2019-10-13T08:43:38.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:38.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.770033+0000) 2019-10-13T08:43:38.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:38.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.769998+0000) 2019-10-13T08:43:39.259 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:43:39.259 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:39.263 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:43:39.263 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:42.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:42.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:42.877727+0000) 2019-10-13T08:43:42.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:42.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:42.877667+0000) 2019-10-13T08:43:43.086 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:43.085+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.085498+0000) 2019-10-13T08:43:43.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:43.459+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.460199+0000) 2019-10-13T08:43:43.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:43.460+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.460506+0000) 2019-10-13T08:43:43.461 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:43.460+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.461061+0000) 2019-10-13T08:43:43.462 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:43.460+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.461239+0000) 2019-10-13T08:43:43.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:43.461+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.462113+0000) 2019-10-13T08:43:43.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:43.461+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.461460+0000) 2019-10-13T08:43:43.463 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:43.461+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.462234+0000) 2019-10-13T08:43:43.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:43.461+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.462667+0000) 2019-10-13T08:43:43.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:43.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:44.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:44.631+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.632348+0000) 2019-10-13T08:43:48.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:48.431+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.431228+0000) 2019-10-13T08:43:48.469 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:48.468+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.468592+0000) 2019-10-13T08:43:48.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:48.468+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.468941+0000) 2019-10-13T08:43:48.470 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:48.468+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.469238+0000) 2019-10-13T08:43:48.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:48.468+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.469389+0000) 2019-10-13T08:43:48.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:48.468+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.470031+0000) 2019-10-13T08:43:48.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:48.469+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.469806+0000) 2019-10-13T08:43:48.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:48.469+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.469924+0000) 2019-10-13T08:43:48.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:48.469+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.471642+0000) 2019-10-13T08:43:48.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:48.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149754 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:48.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:48.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.770186+0000) 2019-10-13T08:43:48.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:48.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.770157+0000) 2019-10-13T08:43:48.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:48.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.770158+0000) 2019-10-13T08:43:52.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:52.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:52.877922+0000) 2019-10-13T08:43:52.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:52.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:52.877849+0000) 2019-10-13T08:43:53.088 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:53.087+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.088013+0000) 2019-10-13T08:43:53.477 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:53.476+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.476616+0000) 2019-10-13T08:43:53.477 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:53.476+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.476848+0000) 2019-10-13T08:43:53.478 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:53.476+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.477022+0000) 2019-10-13T08:43:53.478 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:53.475+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.477487+0000) 2019-10-13T08:43:53.479 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:53.477+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.477815+0000) 2019-10-13T08:43:53.479 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:53.477+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.477984+0000) 2019-10-13T08:43:53.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:53.477+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.478154+0000) 2019-10-13T08:43:53.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:53.478+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.479839+0000) 2019-10-13T08:43:53.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:53.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:54.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:54.632+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.632481+0000) 2019-10-13T08:43:58.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:58.430+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.431390+0000) 2019-10-13T08:43:58.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:58.483+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.484776+0000) 2019-10-13T08:43:58.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:58.483+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.485046+0000) 2019-10-13T08:43:58.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:43:58.485+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.486078+0000) 2019-10-13T08:43:58.487 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:43:58.483+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.485375+0000) 2019-10-13T08:43:58.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:43:58.484+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.485494+0000) 2019-10-13T08:43:58.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:43:58.484+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.485936+0000) 2019-10-13T08:43:58.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:58.484+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.486184+0000) 2019-10-13T08:43:58.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:43:58.486+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.488105+0000) 2019-10-13T08:43:58.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:43:58.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:43:58.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:43:58.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.770361+0000) 2019-10-13T08:43:58.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:43:58.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.770361+0000) 2019-10-13T08:43:58.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:43:58.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.770365+0000) 2019-10-13T08:44:02.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:02.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:02.878093+0000) 2019-10-13T08:44:02.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:02.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:02.878004+0000) 2019-10-13T08:44:03.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:03.089+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.090409+0000) 2019-10-13T08:44:03.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:03.491+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.492704+0000) 2019-10-13T08:44:03.494 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:03.491+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.492899+0000) 2019-10-13T08:44:03.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:03.491+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.493074+0000) 2019-10-13T08:44:03.495 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:03.492+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.493640+0000) 2019-10-13T08:44:03.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:03.492+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.493587+0000) 2019-10-13T08:44:03.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:03.492+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.493765+0000) 2019-10-13T08:44:03.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:03.492+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.494120+0000) 2019-10-13T08:44:03.496 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:03.495+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.495971+0000) 2019-10-13T08:44:03.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:03.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:04.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:04.631+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.632617+0000) 2019-10-13T08:44:08.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:08.430+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.431559+0000) 2019-10-13T08:44:08.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:08.499+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.500938+0000) 2019-10-13T08:44:08.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:08.499+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.501117+0000) 2019-10-13T08:44:08.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:08.500+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.501528+0000) 2019-10-13T08:44:08.504 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:08.500+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.501709+0000) 2019-10-13T08:44:08.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:08.500+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.501762+0000) 2019-10-13T08:44:08.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:08.500+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.502070+0000) 2019-10-13T08:44:08.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:08.500+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.502191+0000) 2019-10-13T08:44:08.505 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:08.504+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.504885+0000) 2019-10-13T08:44:08.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:08.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:08.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:08.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.770565+0000) 2019-10-13T08:44:08.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:08.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.770562+0000) 2019-10-13T08:44:08.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:08.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.770564+0000) 2019-10-13T08:44:09.429 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:44:09.429 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:09.435 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:44:09.435 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:12.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:12.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:12.878312+0000) 2019-10-13T08:44:12.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:12.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:12.878157+0000) 2019-10-13T08:44:13.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:13.092+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.092885+0000) 2019-10-13T08:44:13.509 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:13.507+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.508886+0000) 2019-10-13T08:44:13.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:13.507+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.509010+0000) 2019-10-13T08:44:13.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:13.507+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.509193+0000) 2019-10-13T08:44:13.511 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:13.509+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.509904+0000) 2019-10-13T08:44:13.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:13.508+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.509615+0000) 2019-10-13T08:44:13.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:13.508+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.509820+0000) 2019-10-13T08:44:13.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:13.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.510012+0000) 2019-10-13T08:44:13.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:13.510+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.511430+0000) 2019-10-13T08:44:13.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:13.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:14.633 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:14.632+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.632790+0000) 2019-10-13T08:44:18.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:18.431+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.431722+0000) 2019-10-13T08:44:18.516 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:18.515+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.515982+0000) 2019-10-13T08:44:18.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:18.514+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.515967+0000) 2019-10-13T08:44:18.517 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:18.514+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.516177+0000) 2019-10-13T08:44:18.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:18.514+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.516386+0000) 2019-10-13T08:44:18.518 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:18.515+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.516541+0000) 2019-10-13T08:44:18.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:18.515+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.516684+0000) 2019-10-13T08:44:18.518 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:18.515+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.516925+0000) 2019-10-13T08:44:18.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:18.518+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.519443+0000) 2019-10-13T08:44:18.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:18.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:18.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:18.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.770745+0000) 2019-10-13T08:44:18.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:18.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.770741+0000) 2019-10-13T08:44:18.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:18.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.770745+0000) 2019-10-13T08:44:22.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:22.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:22.878504+0000) 2019-10-13T08:44:22.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:22.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:22.878344+0000) 2019-10-13T08:44:23.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:23.095+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.095421+0000) 2019-10-13T08:44:23.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:23.522+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.523692+0000) 2019-10-13T08:44:23.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:23.522+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.523987+0000) 2019-10-13T08:44:23.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:23.524+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.524847+0000) 2019-10-13T08:44:23.526 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:23.522+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.524216+0000) 2019-10-13T08:44:23.526 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:23.522+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.524335+0000) 2019-10-13T08:44:23.526 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:23.523+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.524832+0000) 2019-10-13T08:44:23.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:23.523+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.524754+0000) 2019-10-13T08:44:23.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:23.525+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.525821+0000) 2019-10-13T08:44:23.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:23.529+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:24.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:24.632+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.632973+0000) 2019-10-13T08:44:28.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:28.431+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.431937+0000) 2019-10-13T08:44:28.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:28.530+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.531256+0000) 2019-10-13T08:44:28.533 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:28.531+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.531641+0000) 2019-10-13T08:44:28.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:28.531+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.531794+0000) 2019-10-13T08:44:28.533 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:28.531+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.531916+0000) 2019-10-13T08:44:28.534 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:28.531+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.532336+0000) 2019-10-13T08:44:28.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:28.531+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.532355+0000) 2019-10-13T08:44:28.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:28.532+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.532544+0000) 2019-10-13T08:44:28.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:28.534+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.535012+0000) 2019-10-13T08:44:28.538 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:28.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 149990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:28.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:28.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.770930+0000) 2019-10-13T08:44:28.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:28.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.770927+0000) 2019-10-13T08:44:28.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:28.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.770929+0000) 2019-10-13T08:44:32.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:32.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:32.878673+0000) 2019-10-13T08:44:32.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:32.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:32.878488+0000) 2019-10-13T08:44:33.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:33.097+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.097884+0000) 2019-10-13T08:44:33.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:33.538+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.539322+0000) 2019-10-13T08:44:33.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:33.539+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.539531+0000) 2019-10-13T08:44:33.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:33.539+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.539684+0000) 2019-10-13T08:44:33.541 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:33.539+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.539845+0000) 2019-10-13T08:44:33.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:33.539+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.540030+0000) 2019-10-13T08:44:33.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:33.539+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.539884+0000) 2019-10-13T08:44:33.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:33.540+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.541546+0000) 2019-10-13T08:44:33.542 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:33.539+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.540217+0000) 2019-10-13T08:44:33.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:33.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:34.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:34.632+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.633190+0000) 2019-10-13T08:44:38.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:38.432+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.432107+0000) 2019-10-13T08:44:38.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:38.545+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.545761+0000) 2019-10-13T08:44:38.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:38.545+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.545861+0000) 2019-10-13T08:44:38.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:38.545+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.546004+0000) 2019-10-13T08:44:38.547 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:38.545+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.546179+0000) 2019-10-13T08:44:38.548 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:38.544+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.546592+0000) 2019-10-13T08:44:38.548 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:38.546+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.546589+0000) 2019-10-13T08:44:38.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:38.546+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.546688+0000) 2019-10-13T08:44:38.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:38.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:38.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:38.547+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.549645+0000) 2019-10-13T08:44:38.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:38.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.771103+0000) 2019-10-13T08:44:38.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:38.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.771103+0000) 2019-10-13T08:44:38.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:38.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.771111+0000) 2019-10-13T08:44:39.519 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:44:39.519 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:39.523 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:44:39.523 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:42.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:42.877+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:42.878907+0000) 2019-10-13T08:44:42.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:42.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:42.878683+0000) 2019-10-13T08:44:43.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:43.099+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.100342+0000) 2019-10-13T08:44:43.553 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:43.551+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.552922+0000) 2019-10-13T08:44:43.554 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:43.551+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.553346+0000) 2019-10-13T08:44:43.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:43.551+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.553299+0000) 2019-10-13T08:44:43.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:43.552+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.553512+0000) 2019-10-13T08:44:43.555 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:43.552+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.553725+0000) 2019-10-13T08:44:43.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:43.552+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.553402+0000) 2019-10-13T08:44:43.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:43.552+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.554051+0000) 2019-10-13T08:44:43.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:43.553+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.555013+0000) 2019-10-13T08:44:43.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:43.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150079 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:44.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:44.632+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.633413+0000) 2019-10-13T08:44:48.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:48.431+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.432274+0000) 2019-10-13T08:44:48.564 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:48.559+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.561198+0000) 2019-10-13T08:44:48.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:48.559+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.560613+0000) 2019-10-13T08:44:48.565 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:48.559+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.561049+0000) 2019-10-13T08:44:48.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:48.559+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.561166+0000) 2019-10-13T08:44:48.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:48.559+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.561291+0000) 2019-10-13T08:44:48.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:48.560+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.561395+0000) 2019-10-13T08:44:48.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:48.560+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.561869+0000) 2019-10-13T08:44:48.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:48.562+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.564530+0000) 2019-10-13T08:44:48.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:48.565+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:48.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:48.769+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.771261+0000) 2019-10-13T08:44:48.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:48.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.771261+0000) 2019-10-13T08:44:48.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:48.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.771261+0000) 2019-10-13T08:44:52.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:52.878+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:52.879042+0000) 2019-10-13T08:44:52.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:52.877+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:52.878871+0000) 2019-10-13T08:44:53.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:53.101+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.102665+0000) 2019-10-13T08:44:53.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:53.567+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.569016+0000) 2019-10-13T08:44:53.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:53.567+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.569267+0000) 2019-10-13T08:44:53.570 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:53.568+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.569514+0000) 2019-10-13T08:44:53.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:53.568+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.569734+0000) 2019-10-13T08:44:53.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:53.568+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.569698+0000) 2019-10-13T08:44:53.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:53.568+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.569502+0000) 2019-10-13T08:44:53.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:53.568+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.570291+0000) 2019-10-13T08:44:53.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:53.571+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.571807+0000) 2019-10-13T08:44:53.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:53.572+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:54.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:54.632+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.633598+0000) 2019-10-13T08:44:58.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:58.431+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.432471+0000) 2019-10-13T08:44:58.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:44:58.573+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.574971+0000) 2019-10-13T08:44:58.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:44:58.574+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.575632+0000) 2019-10-13T08:44:58.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:58.574+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.575719+0000) 2019-10-13T08:44:58.577 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:44:58.575+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.576306+0000) 2019-10-13T08:44:58.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:44:58.577+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.577922+0000) 2019-10-13T08:44:58.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:58.574+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.575877+0000) 2019-10-13T08:44:58.579 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:58.574+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.576070+0000) 2019-10-13T08:44:58.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:44:58.575+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.576402+0000) 2019-10-13T08:44:58.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:44:58.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:44:58.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:44:58.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.771410+0000) 2019-10-13T08:44:58.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:44:58.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.771402+0000) 2019-10-13T08:44:58.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:44:58.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.771402+0000) 2019-10-13T08:45:02.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:02.878+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:02.879223+0000) 2019-10-13T08:45:02.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:02.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:02.879052+0000) 2019-10-13T08:45:03.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:03.104+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.104991+0000) 2019-10-13T08:45:03.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:03.580+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.582038+0000) 2019-10-13T08:45:03.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:03.580+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.582231+0000) 2019-10-13T08:45:03.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:03.580+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.582331+0000) 2019-10-13T08:45:03.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:03.581+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.582504+0000) 2019-10-13T08:45:03.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:03.582+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.582977+0000) 2019-10-13T08:45:03.584 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:03.581+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.583001+0000) 2019-10-13T08:45:03.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:03.581+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.583109+0000) 2019-10-13T08:45:03.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:03.585+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.586467+0000) 2019-10-13T08:45:03.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:03.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150197 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:04.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:04.633+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.633807+0000) 2019-10-13T08:45:08.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:08.432+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.432707+0000) 2019-10-13T08:45:08.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:08.587+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.589307+0000) 2019-10-13T08:45:08.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:08.588+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.589521+0000) 2019-10-13T08:45:08.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:08.589+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.589937+0000) 2019-10-13T08:45:08.591 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:08.588+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.590383+0000) 2019-10-13T08:45:08.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:08.588+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.590267+0000) 2019-10-13T08:45:08.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:08.589+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.590719+0000) 2019-10-13T08:45:08.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:08.589+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.590948+0000) 2019-10-13T08:45:08.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:08.591+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.592445+0000) 2019-10-13T08:45:08.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:08.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:08.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:08.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.771560+0000) 2019-10-13T08:45:08.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:08.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.771563+0000) 2019-10-13T08:45:08.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:08.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.771560+0000) 2019-10-13T08:45:09.599 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:45:09.599 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:09.604 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:45:09.605 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:12.879 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:12.879+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:12.879373+0000) 2019-10-13T08:45:12.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:12.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:12.879244+0000) 2019-10-13T08:45:13.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:13.107+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.107420+0000) 2019-10-13T08:45:13.598 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:13.597+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.598148+0000) 2019-10-13T08:45:13.599 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:13.597+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.598071+0000) 2019-10-13T08:45:13.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:13.598+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.598617+0000) 2019-10-13T08:45:13.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:13.598+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.598738+0000) 2019-10-13T08:45:13.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:13.598+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.599024+0000) 2019-10-13T08:45:13.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:13.598+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.599054+0000) 2019-10-13T08:45:13.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:13.598+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.599261+0000) 2019-10-13T08:45:13.601 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:13.600+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.600838+0000) 2019-10-13T08:45:13.605 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:13.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:14.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:14.633+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.634013+0000) 2019-10-13T08:45:18.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:18.432+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.432924+0000) 2019-10-13T08:45:18.607 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:18.605+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.606553+0000) 2019-10-13T08:45:18.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:18.606+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.606431+0000) 2019-10-13T08:45:18.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:18.606+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.606920+0000) 2019-10-13T08:45:18.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:18.606+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.606900+0000) 2019-10-13T08:45:18.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:18.606+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.607293+0000) 2019-10-13T08:45:18.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:18.607+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.607496+0000) 2019-10-13T08:45:18.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:18.607+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.607659+0000) 2019-10-13T08:45:18.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:18.608+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.609448+0000) 2019-10-13T08:45:18.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:18.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:18.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:18.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.771689+0000) 2019-10-13T08:45:18.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:18.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.771726+0000) 2019-10-13T08:45:18.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:18.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.771730+0000) 2019-10-13T08:45:22.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:22.879+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:22.879551+0000) 2019-10-13T08:45:22.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:22.879+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:22.879439+0000) 2019-10-13T08:45:23.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:23.109+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.109966+0000) 2019-10-13T08:45:23.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:23.614+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.614904+0000) 2019-10-13T08:45:23.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:23.614+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.615107+0000) 2019-10-13T08:45:23.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:23.614+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.615236+0000) 2019-10-13T08:45:23.617 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:23.615+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.616116+0000) 2019-10-13T08:45:23.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:23.615+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.615529+0000) 2019-10-13T08:45:23.617 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:23.615+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.615665+0000) 2019-10-13T08:45:23.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:23.615+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.616104+0000) 2019-10-13T08:45:23.619 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:23.618+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.618772+0000) 2019-10-13T08:45:23.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:23.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:24.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:24.634+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.634172+0000) 2019-10-13T08:45:28.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:28.432+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.433133+0000) 2019-10-13T08:45:28.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:28.621+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.623158+0000) 2019-10-13T08:45:28.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:28.621+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.623267+0000) 2019-10-13T08:45:28.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:28.622+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.623597+0000) 2019-10-13T08:45:28.625 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:28.622+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.623814+0000) 2019-10-13T08:45:28.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:28.623+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.624274+0000) 2019-10-13T08:45:28.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:28.622+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.624115+0000) 2019-10-13T08:45:28.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:28.622+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.624357+0000) 2019-10-13T08:45:28.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:28.626+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.626883+0000) 2019-10-13T08:45:28.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:28.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:28.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:28.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.771835+0000) 2019-10-13T08:45:28.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:28.770+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.771896+0000) 2019-10-13T08:45:28.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:28.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.771927+0000) 2019-10-13T08:45:32.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:32.878+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:32.879713+0000) 2019-10-13T08:45:32.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:32.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:32.879599+0000) 2019-10-13T08:45:33.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:33.111+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.112328+0000) 2019-10-13T08:45:33.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:33.630+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.631658+0000) 2019-10-13T08:45:33.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:33.630+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.631973+0000) 2019-10-13T08:45:33.633 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:33.630+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.632143+0000) 2019-10-13T08:45:33.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:33.630+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.632257+0000) 2019-10-13T08:45:33.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:33.631+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.632534+0000) 2019-10-13T08:45:33.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:33.631+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.632689+0000) 2019-10-13T08:45:33.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:33.631+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.632819+0000) 2019-10-13T08:45:33.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:33.632+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.634297+0000) 2019-10-13T08:45:33.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:33.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:34.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:34.633+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.634323+0000) 2019-10-13T08:45:38.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:38.432+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.433327+0000) 2019-10-13T08:45:38.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:38.638+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.639749+0000) 2019-10-13T08:45:38.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:38.638+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.640233+0000) 2019-10-13T08:45:38.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:38.638+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.640271+0000) 2019-10-13T08:45:38.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:38.638+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.640385+0000) 2019-10-13T08:45:38.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:38.639+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.640520+0000) 2019-10-13T08:45:38.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:38.639+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.640793+0000) 2019-10-13T08:45:38.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:38.639+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.641033+0000) 2019-10-13T08:45:38.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:38.640+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.642543+0000) 2019-10-13T08:45:38.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:38.644+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:38.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:38.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.771981+0000) 2019-10-13T08:45:38.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:38.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.772038+0000) 2019-10-13T08:45:38.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:38.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.772059+0000) 2019-10-13T08:45:39.737 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:45:39.738 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:39.752 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:45:39.752 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:42.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:42.879+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:42.879830+0000) 2019-10-13T08:45:42.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:42.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:42.879749+0000) 2019-10-13T08:45:43.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:43.114+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.114819+0000) 2019-10-13T08:45:43.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:43.646+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.648010+0000) 2019-10-13T08:45:43.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:43.646+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.648124+0000) 2019-10-13T08:45:43.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:43.646+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.648319+0000) 2019-10-13T08:45:43.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:43.647+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.648706+0000) 2019-10-13T08:45:43.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:43.648+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.648809+0000) 2019-10-13T08:45:43.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:43.647+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.648974+0000) 2019-10-13T08:45:43.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:43.647+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.649249+0000) 2019-10-13T08:45:43.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:43.650+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.651228+0000) 2019-10-13T08:45:43.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:43.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:44.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:44.633+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.634499+0000) 2019-10-13T08:45:48.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:48.432+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.433514+0000) 2019-10-13T08:45:48.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:48.654+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.655687+0000) 2019-10-13T08:45:48.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:48.654+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.655861+0000) 2019-10-13T08:45:48.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:48.654+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.655957+0000) 2019-10-13T08:45:48.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:48.655+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.656509+0000) 2019-10-13T08:45:48.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:48.654+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.656384+0000) 2019-10-13T08:45:48.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:48.655+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.656525+0000) 2019-10-13T08:45:48.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:48.655+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.656821+0000) 2019-10-13T08:45:48.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:48.658+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.659051+0000) 2019-10-13T08:45:48.661 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:48.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:48.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:48.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.772138+0000) 2019-10-13T08:45:48.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:48.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.772179+0000) 2019-10-13T08:45:48.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:48.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.772185+0000) 2019-10-13T08:45:52.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:52.879+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:52.880007+0000) 2019-10-13T08:45:52.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:52.878+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:52.879927+0000) 2019-10-13T08:45:53.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:53.116+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.116982+0000) 2019-10-13T08:45:53.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:53.661+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.663008+0000) 2019-10-13T08:45:53.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:53.661+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.663262+0000) 2019-10-13T08:45:53.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:53.662+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.663619+0000) 2019-10-13T08:45:53.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:53.663+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.664353+0000) 2019-10-13T08:45:53.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:53.662+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.663884+0000) 2019-10-13T08:45:53.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:53.662+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.664233+0000) 2019-10-13T08:45:53.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:53.662+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.664323+0000) 2019-10-13T08:45:53.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:53.665+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.666380+0000) 2019-10-13T08:45:53.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:53.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:54.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:54.634+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.634665+0000) 2019-10-13T08:45:58.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:58.433+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.433706+0000) 2019-10-13T08:45:58.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:58.671+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.671555+0000) 2019-10-13T08:45:58.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:58.671+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.671723+0000) 2019-10-13T08:45:58.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:45:58.671+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.671975+0000) 2019-10-13T08:45:58.674 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:45:58.671+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.672223+0000) 2019-10-13T08:45:58.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:58.672+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.672633+0000) 2019-10-13T08:45:58.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:45:58.672+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.672743+0000) 2019-10-13T08:45:58.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:45:58.672+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.673284+0000) 2019-10-13T08:45:58.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:45:58.674+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.675689+0000) 2019-10-13T08:45:58.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:45:58.677+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:45:58.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:45:58.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.772304+0000) 2019-10-13T08:45:58.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:45:58.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.772326+0000) 2019-10-13T08:45:58.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:45:58.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.772332+0000) 2019-10-13T08:46:02.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:02.879+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:02.880187+0000) 2019-10-13T08:46:02.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:02.879+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:02.880113+0000) 2019-10-13T08:46:03.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:03.118+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.119294+0000) 2019-10-13T08:46:03.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:03.679+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.679986+0000) 2019-10-13T08:46:03.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:03.679+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.680220+0000) 2019-10-13T08:46:03.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:03.680+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.680928+0000) 2019-10-13T08:46:03.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:03.680+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.680947+0000) 2019-10-13T08:46:03.682 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:03.680+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.681249+0000) 2019-10-13T08:46:03.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:03.680+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.681242+0000) 2019-10-13T08:46:03.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:03.680+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.681067+0000) 2019-10-13T08:46:03.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:03.682+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.683090+0000) 2019-10-13T08:46:03.686 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:03.685+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150555 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:04.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:04.634+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.634830+0000) 2019-10-13T08:46:08.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:08.433+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.433903+0000) 2019-10-13T08:46:08.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:08.687+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.687566+0000) 2019-10-13T08:46:08.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:08.687+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.687626+0000) 2019-10-13T08:46:08.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:08.687+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.688076+0000) 2019-10-13T08:46:08.689 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:08.687+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.688136+0000) 2019-10-13T08:46:08.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:08.687+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.688333+0000) 2019-10-13T08:46:08.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:08.688+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.688560+0000) 2019-10-13T08:46:08.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:08.688+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.688926+0000) 2019-10-13T08:46:08.691 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:08.689+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.690658+0000) 2019-10-13T08:46:08.693 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:08.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:08.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:08.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.772471+0000) 2019-10-13T08:46:08.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:08.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.772500+0000) 2019-10-13T08:46:08.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:08.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.772549+0000) 2019-10-13T08:46:09.902 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:46:09.902 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:09.907 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:46:09.907 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:12.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:12.880+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:12.880369+0000) 2019-10-13T08:46:12.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:12.880+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:12.880304+0000) 2019-10-13T08:46:13.122 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:13.120+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.121655+0000) 2019-10-13T08:46:13.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:13.694+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.694788+0000) 2019-10-13T08:46:13.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:13.693+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.694133+0000) 2019-10-13T08:46:13.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:13.694+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.694574+0000) 2019-10-13T08:46:13.698 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:13.694+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.694692+0000) 2019-10-13T08:46:13.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:13.694+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.695253+0000) 2019-10-13T08:46:13.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:13.695+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.695501+0000) 2019-10-13T08:46:13.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:13.695+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.695605+0000) 2019-10-13T08:46:13.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:13.696+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.697356+0000) 2019-10-13T08:46:13.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:13.698+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:14.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:14.634+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.634995+0000) 2019-10-13T08:46:18.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:18.433+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.434097+0000) 2019-10-13T08:46:18.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:18.699+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.701062+0000) 2019-10-13T08:46:18.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:18.700+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.701471+0000) 2019-10-13T08:46:18.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:18.700+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.701491+0000) 2019-10-13T08:46:18.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:18.700+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.701664+0000) 2019-10-13T08:46:18.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:18.700+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.702354+0000) 2019-10-13T08:46:18.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:18.701+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.702480+0000) 2019-10-13T08:46:18.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:18.701+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.702655+0000) 2019-10-13T08:46:18.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:18.703+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.704567+0000) 2019-10-13T08:46:18.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:18.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:18.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:18.771+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.772641+0000) 2019-10-13T08:46:18.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:18.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.772696+0000) 2019-10-13T08:46:18.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:18.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.772716+0000) 2019-10-13T08:46:22.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:22.878+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:22.880562+0000) 2019-10-13T08:46:22.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:22.879+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:22.880477+0000) 2019-10-13T08:46:23.124 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:23.123+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.124083+0000) 2019-10-13T08:46:23.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:23.707+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.709414+0000) 2019-10-13T08:46:23.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:23.707+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.709241+0000) 2019-10-13T08:46:23.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:23.708+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.709410+0000) 2019-10-13T08:46:23.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:23.708+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.709568+0000) 2019-10-13T08:46:23.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:23.708+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.709667+0000) 2019-10-13T08:46:23.711 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:23.708+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.710018+0000) 2019-10-13T08:46:23.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:23.709+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.710429+0000) 2019-10-13T08:46:23.713 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:23.711+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.713374+0000) 2019-10-13T08:46:23.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:23.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:24.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:24.634+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.635176+0000) 2019-10-13T08:46:28.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:28.433+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.434249+0000) 2019-10-13T08:46:28.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:28.715+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.716878+0000) 2019-10-13T08:46:28.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:28.715+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.717269+0000) 2019-10-13T08:46:28.719 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:28.717+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.718742+0000) 2019-10-13T08:46:28.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:28.717+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.719346+0000) 2019-10-13T08:46:28.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:28.717+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.719093+0000) 2019-10-13T08:46:28.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:28.717+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.719223+0000) 2019-10-13T08:46:28.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:28.718+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.719651+0000) 2019-10-13T08:46:28.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:28.721+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.722786+0000) 2019-10-13T08:46:28.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:28.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:28.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:28.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.772825+0000) 2019-10-13T08:46:28.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:28.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.772859+0000) 2019-10-13T08:46:28.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:28.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.772841+0000) 2019-10-13T08:46:32.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:32.879+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:32.880769+0000) 2019-10-13T08:46:32.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:32.880+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:32.880677+0000) 2019-10-13T08:46:33.127 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:33.124+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.126473+0000) 2019-10-13T08:46:33.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:33.724+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.726044+0000) 2019-10-13T08:46:33.727 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:33.724+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.726310+0000) 2019-10-13T08:46:33.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:33.725+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.726629+0000) 2019-10-13T08:46:33.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:33.724+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.726557+0000) 2019-10-13T08:46:33.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:33.725+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.727215+0000) 2019-10-13T08:46:33.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:33.726+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.727528+0000) 2019-10-13T08:46:33.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:33.726+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.727733+0000) 2019-10-13T08:46:33.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:33.727+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.729135+0000) 2019-10-13T08:46:33.733 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:33.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:34.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:34.634+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.635336+0000) 2019-10-13T08:46:38.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:38.433+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.434433+0000) 2019-10-13T08:46:38.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:38.733+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.734697+0000) 2019-10-13T08:46:38.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:38.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.734953+0000) 2019-10-13T08:46:38.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:38.733+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.735367+0000) 2019-10-13T08:46:38.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:38.734+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.735502+0000) 2019-10-13T08:46:38.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:38.734+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.735674+0000) 2019-10-13T08:46:38.737 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:38.734+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.735862+0000) 2019-10-13T08:46:38.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:38.734+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.735590+0000) 2019-10-13T08:46:38.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:38.736+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.736988+0000) 2019-10-13T08:46:38.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:38.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:38.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:38.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.772989+0000) 2019-10-13T08:46:38.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:38.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.773042+0000) 2019-10-13T08:46:38.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:38.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.773034+0000) 2019-10-13T08:46:40.015 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:46:40.016 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:40.020 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:46:40.020 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:42.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:42.880+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:42.880841+0000) 2019-10-13T08:46:42.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:42.880+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:42.880956+0000) 2019-10-13T08:46:43.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:43.128+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.128818+0000) 2019-10-13T08:46:43.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:43.742+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.742516+0000) 2019-10-13T08:46:43.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:43.742+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.742873+0000) 2019-10-13T08:46:43.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:43.742+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.743040+0000) 2019-10-13T08:46:43.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:43.742+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.743230+0000) 2019-10-13T08:46:43.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:43.743+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.743409+0000) 2019-10-13T08:46:43.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:43.743+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.743560+0000) 2019-10-13T08:46:43.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:43.743+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.743830+0000) 2019-10-13T08:46:43.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:43.744+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.745124+0000) 2019-10-13T08:46:43.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:43.748+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:44.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:44.635+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.635508+0000) 2019-10-13T08:46:48.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:48.434+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.434607+0000) 2019-10-13T08:46:48.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:48.750+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.751359+0000) 2019-10-13T08:46:48.752 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:48.750+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.751214+0000) 2019-10-13T08:46:48.752 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:48.751+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.751484+0000) 2019-10-13T08:46:48.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:48.751+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.751575+0000) 2019-10-13T08:46:48.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:48.751+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.751859+0000) 2019-10-13T08:46:48.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:48.751+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.752217+0000) 2019-10-13T08:46:48.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:48.752+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.752503+0000) 2019-10-13T08:46:48.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:48.753+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.754123+0000) 2019-10-13T08:46:48.757 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:48.756+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:48.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:48.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.773130+0000) 2019-10-13T08:46:48.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:48.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.773233+0000) 2019-10-13T08:46:48.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:48.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.773249+0000) 2019-10-13T08:46:52.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:52.880+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:52.881123+0000) 2019-10-13T08:46:52.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:52.880+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:52.881022+0000) 2019-10-13T08:46:53.132 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:53.131+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.131464+0000) 2019-10-13T08:46:53.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:53.758+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.758858+0000) 2019-10-13T08:46:53.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:53.758+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.758433+0000) 2019-10-13T08:46:53.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:53.758+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.758723+0000) 2019-10-13T08:46:53.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:53.758+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.759049+0000) 2019-10-13T08:46:53.761 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:53.758+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.759301+0000) 2019-10-13T08:46:53.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:53.760+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.761253+0000) 2019-10-13T08:46:53.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:53.759+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.759866+0000) 2019-10-13T08:46:53.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:53.759+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.760021+0000) 2019-10-13T08:46:53.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:53.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:46:54.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:54.635+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.635677+0000) 2019-10-13T08:46:58.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:58.434+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.434769+0000) 2019-10-13T08:46:58.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:58.766+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.766773+0000) 2019-10-13T08:46:58.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:58.766+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.766923+0000) 2019-10-13T08:46:58.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:58.766+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.767053+0000) 2019-10-13T08:46:58.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:46:58.767+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.767899+0000) 2019-10-13T08:46:58.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:46:58.767+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.767934+0000) 2019-10-13T08:46:58.769 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:46:58.767+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.768066+0000) 2019-10-13T08:46:58.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:46:58.767+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.768352+0000) 2019-10-13T08:46:58.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:46:58.768+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.769708+0000) 2019-10-13T08:46:58.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:46:58.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.773277+0000) 2019-10-13T08:46:58.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:46:58.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.773339+0000) 2019-10-13T08:46:58.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:46:58.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.773356+0000) 2019-10-13T08:46:58.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:46:58.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:02.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:02.881+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:02.881292+0000) 2019-10-13T08:47:02.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:02.880+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:02.881196+0000) 2019-10-13T08:47:03.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:03.133+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.133799+0000) 2019-10-13T08:47:03.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:03.773+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.774980+0000) 2019-10-13T08:47:03.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:03.773+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.775197+0000) 2019-10-13T08:47:03.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:03.774+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.775410+0000) 2019-10-13T08:47:03.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:03.775+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.775903+0000) 2019-10-13T08:47:03.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:03.774+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.775699+0000) 2019-10-13T08:47:03.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:03.774+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.775899+0000) 2019-10-13T08:47:03.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:03.774+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.776069+0000) 2019-10-13T08:47:03.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:03.777+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.778672+0000) 2019-10-13T08:47:03.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:03.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:04.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:04.634+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.635883+0000) 2019-10-13T08:47:08.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:08.433+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.434967+0000) 2019-10-13T08:47:08.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:08.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.773438+0000) 2019-10-13T08:47:08.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:08.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.773506+0000) 2019-10-13T08:47:08.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:08.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.773516+0000) 2019-10-13T08:47:08.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:08.782+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.783615+0000) 2019-10-13T08:47:08.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:08.782+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.783656+0000) 2019-10-13T08:47:08.785 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:08.782+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.784060+0000) 2019-10-13T08:47:08.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:08.782+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.784239+0000) 2019-10-13T08:47:08.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:08.782+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.784343+0000) 2019-10-13T08:47:08.785 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:08.783+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.784707+0000) 2019-10-13T08:47:08.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:08.785+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.785889+0000) 2019-10-13T08:47:08.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:08.783+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.784880+0000) 2019-10-13T08:47:08.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:08.788+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:10.116 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:47:10.116 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:10.121 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:47:10.121 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:12.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:12.881+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:12.881477+0000) 2019-10-13T08:47:12.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:12.880+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:12.881366+0000) 2019-10-13T08:47:13.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:13.136+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.136253+0000) 2019-10-13T08:47:13.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:13.791+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.792403+0000) 2019-10-13T08:47:13.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:13.790+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.792382+0000) 2019-10-13T08:47:13.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:13.791+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.792567+0000) 2019-10-13T08:47:13.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:13.791+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.792840+0000) 2019-10-13T08:47:13.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:13.791+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.792969+0000) 2019-10-13T08:47:13.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:13.791+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.793200+0000) 2019-10-13T08:47:13.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:13.791+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.793385+0000) 2019-10-13T08:47:13.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:13.794+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.795505+0000) 2019-10-13T08:47:13.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:13.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 150977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:14.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:14.635+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.636046+0000) 2019-10-13T08:47:18.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:18.434+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.435153+0000) 2019-10-13T08:47:18.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:18.772+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.773636+0000) 2019-10-13T08:47:18.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:18.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.773685+0000) 2019-10-13T08:47:18.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:18.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.773668+0000) 2019-10-13T08:47:18.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:18.799+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.801308+0000) 2019-10-13T08:47:18.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:18.799+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.801173+0000) 2019-10-13T08:47:18.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:18.799+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.801303+0000) 2019-10-13T08:47:18.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:18.800+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.801500+0000) 2019-10-13T08:47:18.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:18.800+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.801714+0000) 2019-10-13T08:47:18.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:18.800+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.802218+0000) 2019-10-13T08:47:18.805 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:18.801+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.802418+0000) 2019-10-13T08:47:18.806 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:18.802+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.804356+0000) 2019-10-13T08:47:18.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:18.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:22.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:22.880+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:22.881660+0000) 2019-10-13T08:47:22.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:22.880+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:22.881540+0000) 2019-10-13T08:47:23.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:23.137+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.138647+0000) 2019-10-13T08:47:23.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:23.807+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.809353+0000) 2019-10-13T08:47:23.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:23.807+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.809156+0000) 2019-10-13T08:47:23.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:23.808+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.809619+0000) 2019-10-13T08:47:23.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:23.808+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.809832+0000) 2019-10-13T08:47:23.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:23.808+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.810046+0000) 2019-10-13T08:47:23.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:23.808+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.810150+0000) 2019-10-13T08:47:23.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:23.810+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.811764+0000) 2019-10-13T08:47:23.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:23.809+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.810488+0000) 2019-10-13T08:47:23.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:23.813+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:24.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:24.635+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.636193+0000) 2019-10-13T08:47:28.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:28.434+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.435341+0000) 2019-10-13T08:47:28.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:28.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.773849+0000) 2019-10-13T08:47:28.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:28.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.773880+0000) 2019-10-13T08:47:28.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:28.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.773882+0000) 2019-10-13T08:47:28.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:28.816+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.817291+0000) 2019-10-13T08:47:28.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:28.816+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.817370+0000) 2019-10-13T08:47:28.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:28.817+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.817561+0000) 2019-10-13T08:47:28.818 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:28.817+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.818075+0000) 2019-10-13T08:47:28.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:28.817+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.818283+0000) 2019-10-13T08:47:28.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:28.818+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.818567+0000) 2019-10-13T08:47:28.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:28.818+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.818820+0000) 2019-10-13T08:47:28.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:28.819+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.820322+0000) 2019-10-13T08:47:28.824 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:28.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:32.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:32.881+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:32.881811+0000) 2019-10-13T08:47:32.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:32.881+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:32.881730+0000) 2019-10-13T08:47:33.141 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:33.140+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.141046+0000) 2019-10-13T08:47:33.825 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:33.824+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.825193+0000) 2019-10-13T08:47:33.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:33.824+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.825120+0000) 2019-10-13T08:47:33.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:33.825+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.825689+0000) 2019-10-13T08:47:33.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:33.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.825800+0000) 2019-10-13T08:47:33.827 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:33.825+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.825971+0000) 2019-10-13T08:47:33.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:33.825+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.826138+0000) 2019-10-13T08:47:33.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:33.825+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.826262+0000) 2019-10-13T08:47:33.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:33.828+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.828819+0000) 2019-10-13T08:47:33.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:33.831+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:34.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:34.636+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.636342+0000) 2019-10-13T08:47:38.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:38.435+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.435511+0000) 2019-10-13T08:47:38.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:38.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.774048+0000) 2019-10-13T08:47:38.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:38.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.774092+0000) 2019-10-13T08:47:38.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:38.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.774111+0000) 2019-10-13T08:47:38.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:38.833+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.833586+0000) 2019-10-13T08:47:38.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:38.833+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.833791+0000) 2019-10-13T08:47:38.835 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:38.833+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.833915+0000) 2019-10-13T08:47:38.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:38.834+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.835026+0000) 2019-10-13T08:47:38.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:38.834+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.834489+0000) 2019-10-13T08:47:38.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:38.834+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.835063+0000) 2019-10-13T08:47:38.836 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:38.833+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.834211+0000) 2019-10-13T08:47:38.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:38.835+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.835906+0000) 2019-10-13T08:47:38.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:38.839+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:40.260 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:47:40.260 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:40.267 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:47:40.267 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:42.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:42.881+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:42.881873+0000) 2019-10-13T08:47:42.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:42.881+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:42.881999+0000) 2019-10-13T08:47:43.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:43.142+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.143602+0000) 2019-10-13T08:47:43.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:43.841+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.841867+0000) 2019-10-13T08:47:43.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:43.841+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.841727+0000) 2019-10-13T08:47:43.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:43.842+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.843273+0000) 2019-10-13T08:47:43.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:43.842+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.842594+0000) 2019-10-13T08:47:43.844 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:43.842+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.842485+0000) 2019-10-13T08:47:43.844 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:43.842+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.842754+0000) 2019-10-13T08:47:43.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:43.842+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.843086+0000) 2019-10-13T08:47:43.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:43.843+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.844686+0000) 2019-10-13T08:47:43.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:43.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:44.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:44.636+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.636487+0000) 2019-10-13T08:47:48.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:48.434+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.435703+0000) 2019-10-13T08:47:48.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:48.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.774260+0000) 2019-10-13T08:47:48.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:48.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.774267+0000) 2019-10-13T08:47:48.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:48.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.774268+0000) 2019-10-13T08:47:48.850 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:48.848+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.849454+0000) 2019-10-13T08:47:48.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:48.849+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.850562+0000) 2019-10-13T08:47:48.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:48.848+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.850061+0000) 2019-10-13T08:47:48.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:48.849+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.850774+0000) 2019-10-13T08:47:48.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:48.849+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.850836+0000) 2019-10-13T08:47:48.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:48.849+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.850503+0000) 2019-10-13T08:47:48.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:48.849+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.851307+0000) 2019-10-13T08:47:48.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:48.851+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.852684+0000) 2019-10-13T08:47:48.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:48.854+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:52.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:52.881+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:52.882146+0000) 2019-10-13T08:47:52.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:52.881+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:52.882046+0000) 2019-10-13T08:47:53.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:53.145+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.146055+0000) 2019-10-13T08:47:53.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:53.856+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.857521+0000) 2019-10-13T08:47:53.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:53.857+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.858237+0000) 2019-10-13T08:47:53.859 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:53.856+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.857992+0000) 2019-10-13T08:47:53.859 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:53.856+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.858095+0000) 2019-10-13T08:47:53.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:53.856+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.858195+0000) 2019-10-13T08:47:53.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:53.857+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.858432+0000) 2019-10-13T08:47:53.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:53.857+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.858580+0000) 2019-10-13T08:47:53.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:53.859+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.859757+0000) 2019-10-13T08:47:53.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:53.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151216 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:47:54.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:54.635+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.636591+0000) 2019-10-13T08:47:58.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:58.435+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.435866+0000) 2019-10-13T08:47:58.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:58.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.774414+0000) 2019-10-13T08:47:58.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:58.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.774413+0000) 2019-10-13T08:47:58.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:58.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.774413+0000) 2019-10-13T08:47:58.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:47:58.862+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.864023+0000) 2019-10-13T08:47:58.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:47:58.862+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.864159+0000) 2019-10-13T08:47:58.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:47:58.862+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.864286+0000) 2019-10-13T08:47:58.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:47:58.864+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.864990+0000) 2019-10-13T08:47:58.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:47:58.863+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.864844+0000) 2019-10-13T08:47:58.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:47:58.863+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.865044+0000) 2019-10-13T08:47:58.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:47:58.863+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.865289+0000) 2019-10-13T08:47:58.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:47:58.867+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.867983+0000) 2019-10-13T08:47:58.870 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:47:58.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:02.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:02.882+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:02.882331+0000) 2019-10-13T08:48:02.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:02.881+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:02.882242+0000) 2019-10-13T08:48:03.148 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:03.147+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.148073+0000) 2019-10-13T08:48:03.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:03.870+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.871424+0000) 2019-10-13T08:48:03.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:03.870+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.871625+0000) 2019-10-13T08:48:03.872 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:03.870+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.871772+0000) 2019-10-13T08:48:03.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:03.871+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.872301+0000) 2019-10-13T08:48:03.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:03.870+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.872076+0000) 2019-10-13T08:48:03.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:03.871+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.872578+0000) 2019-10-13T08:48:03.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:03.871+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.872792+0000) 2019-10-13T08:48:03.874 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:03.874+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.874817+0000) 2019-10-13T08:48:03.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:03.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151274 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:04.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:04.635+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.636748+0000) 2019-10-13T08:48:08.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:08.435+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.436040+0000) 2019-10-13T08:48:08.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:08.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.774530+0000) 2019-10-13T08:48:08.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:08.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.774507+0000) 2019-10-13T08:48:08.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:08.773+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.774546+0000) 2019-10-13T08:48:08.881 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:08.878+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.880027+0000) 2019-10-13T08:48:08.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:08.878+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.880155+0000) 2019-10-13T08:48:08.881 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:08.879+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.880487+0000) 2019-10-13T08:48:08.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:08.879+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.881427+0000) 2019-10-13T08:48:08.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:08.879+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.880753+0000) 2019-10-13T08:48:08.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:08.879+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.880870+0000) 2019-10-13T08:48:08.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:08.880+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.881424+0000) 2019-10-13T08:48:08.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:08.880+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.882720+0000) 2019-10-13T08:48:08.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:08.882+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151303 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:10.385 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:48:10.385 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:10.391 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:48:10.391 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:12.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:12.881+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:12.882505+0000) 2019-10-13T08:48:12.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:12.881+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:12.882407+0000) 2019-10-13T08:48:13.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:13.149+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.150514+0000) 2019-10-13T08:48:13.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:13.884+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.886097+0000) 2019-10-13T08:48:13.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:13.884+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.886115+0000) 2019-10-13T08:48:13.887 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:13.884+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.886344+0000) 2019-10-13T08:48:13.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:13.885+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.886725+0000) 2019-10-13T08:48:13.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:13.885+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.886866+0000) 2019-10-13T08:48:13.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:13.885+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.887060+0000) 2019-10-13T08:48:13.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:13.885+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.886570+0000) 2019-10-13T08:48:13.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:13.887+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.889454+0000) 2019-10-13T08:48:13.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:13.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:14.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:14.636+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.636939+0000) 2019-10-13T08:48:18.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:18.435+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.436231+0000) 2019-10-13T08:48:18.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:18.774+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.774698+0000) 2019-10-13T08:48:18.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:18.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.774691+0000) 2019-10-13T08:48:18.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:18.774+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.774692+0000) 2019-10-13T08:48:18.895 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:18.893+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.893999+0000) 2019-10-13T08:48:18.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:18.893+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.894241+0000) 2019-10-13T08:48:18.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:18.893+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.894374+0000) 2019-10-13T08:48:18.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:18.894+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.894522+0000) 2019-10-13T08:48:18.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:18.893+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.895059+0000) 2019-10-13T08:48:18.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:18.894+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.895015+0000) 2019-10-13T08:48:18.897 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:18.894+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.895067+0000) 2019-10-13T08:48:18.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:18.895+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.897334+0000) 2019-10-13T08:48:18.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:18.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151362 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:22.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:22.881+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:22.882696+0000) 2019-10-13T08:48:22.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:22.882+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:22.882545+0000) 2019-10-13T08:48:23.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:23.152+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.153517+0000) 2019-10-13T08:48:23.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:23.901+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.902150+0000) 2019-10-13T08:48:23.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:23.902+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.902420+0000) 2019-10-13T08:48:23.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:23.902+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.902620+0000) 2019-10-13T08:48:23.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:23.902+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.903372+0000) 2019-10-13T08:48:23.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:23.902+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.903156+0000) 2019-10-13T08:48:23.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:23.902+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.903308+0000) 2019-10-13T08:48:23.906 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:23.903+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.903713+0000) 2019-10-13T08:48:23.906 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:23.904+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.904980+0000) 2019-10-13T08:48:23.909 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:23.908+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151392 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:24.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:24.636+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.637109+0000) 2019-10-13T08:48:28.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:28.436+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.436432+0000) 2019-10-13T08:48:28.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:28.774+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.774867+0000) 2019-10-13T08:48:28.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:28.774+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.774868+0000) 2019-10-13T08:48:28.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:28.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.774871+0000) 2019-10-13T08:48:28.911 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:28.910+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.910574+0000) 2019-10-13T08:48:28.911 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:28.910+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.910797+0000) 2019-10-13T08:48:28.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:28.910+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.911028+0000) 2019-10-13T08:48:28.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:28.911+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.911889+0000) 2019-10-13T08:48:28.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:28.911+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.911796+0000) 2019-10-13T08:48:28.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:28.911+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.912068+0000) 2019-10-13T08:48:28.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:28.911+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.912256+0000) 2019-10-13T08:48:28.914 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:28.913+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.914528+0000) 2019-10-13T08:48:28.918 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:28.917+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:32.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:32.882+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:32.882913+0000) 2019-10-13T08:48:32.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:32.881+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:32.882685+0000) 2019-10-13T08:48:33.156 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:33.155+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.155912+0000) 2019-10-13T08:48:33.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:33.917+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.919249+0000) 2019-10-13T08:48:33.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:33.917+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.919335+0000) 2019-10-13T08:48:33.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:33.919+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.920053+0000) 2019-10-13T08:48:33.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:33.918+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.919861+0000) 2019-10-13T08:48:33.921 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:33.918+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.919995+0000) 2019-10-13T08:48:33.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:33.918+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.920097+0000) 2019-10-13T08:48:33.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:33.918+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.920371+0000) 2019-10-13T08:48:33.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:33.922+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.923000+0000) 2019-10-13T08:48:33.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:33.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:34.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:34.635+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.637279+0000) 2019-10-13T08:48:38.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:38.435+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.436595+0000) 2019-10-13T08:48:38.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:38.774+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.775059+0000) 2019-10-13T08:48:38.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:38.774+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.775091+0000) 2019-10-13T08:48:38.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:38.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.775084+0000) 2019-10-13T08:48:38.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:38.924+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.926200+0000) 2019-10-13T08:48:38.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:38.925+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.926435+0000) 2019-10-13T08:48:38.927 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:38.925+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.926589+0000) 2019-10-13T08:48:38.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:38.926+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.927093+0000) 2019-10-13T08:48:38.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:38.925+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.927041+0000) 2019-10-13T08:48:38.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:38.925+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.927104+0000) 2019-10-13T08:48:38.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:38.925+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.927325+0000) 2019-10-13T08:48:38.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:38.928+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.929580+0000) 2019-10-13T08:48:38.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:38.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151480 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:40.480 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:48:40.481 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:40.485 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:48:40.485 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:42.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:42.882+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:42.883100+0000) 2019-10-13T08:48:42.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:42.882+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:42.882857+0000) 2019-10-13T08:48:43.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:43.157+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.158270+0000) 2019-10-13T08:48:43.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:43.933+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.934369+0000) 2019-10-13T08:48:43.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:43.932+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.934128+0000) 2019-10-13T08:48:43.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:43.933+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.934603+0000) 2019-10-13T08:48:43.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:43.933+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.934533+0000) 2019-10-13T08:48:43.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:43.933+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.934729+0000) 2019-10-13T08:48:43.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:43.933+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.934823+0000) 2019-10-13T08:48:43.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:43.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.935274+0000) 2019-10-13T08:48:43.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:43.936+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.936810+0000) 2019-10-13T08:48:43.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:43.938+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:44.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:44.636+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.637474+0000) 2019-10-13T08:48:48.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:48.436+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.436769+0000) 2019-10-13T08:48:48.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:48.774+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.775271+0000) 2019-10-13T08:48:48.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:48.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.775279+0000) 2019-10-13T08:48:48.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:48.774+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.775290+0000) 2019-10-13T08:48:48.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:48.940+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.941807+0000) 2019-10-13T08:48:48.943 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:48.940+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.942097+0000) 2019-10-13T08:48:48.943 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:48.940+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.942209+0000) 2019-10-13T08:48:48.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:48.940+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.942370+0000) 2019-10-13T08:48:48.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:48.942+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.943395+0000) 2019-10-13T08:48:48.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:48.941+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.942544+0000) 2019-10-13T08:48:48.944 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:48.941+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.942955+0000) 2019-10-13T08:48:48.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:48.945+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.946368+0000) 2019-10-13T08:48:48.948 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:48.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:52.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:52.883+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:52.883285+0000) 2019-10-13T08:48:52.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:52.882+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:52.883063+0000) 2019-10-13T08:48:53.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:53.160+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.160573+0000) 2019-10-13T08:48:53.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:53.948+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.949422+0000) 2019-10-13T08:48:53.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:53.948+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.949751+0000) 2019-10-13T08:48:53.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:53.948+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.949757+0000) 2019-10-13T08:48:53.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:53.949+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.949946+0000) 2019-10-13T08:48:53.952 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:53.948+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.950019+0000) 2019-10-13T08:48:53.952 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:53.948+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.950172+0000) 2019-10-13T08:48:53.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:53.948+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.950325+0000) 2019-10-13T08:48:53.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:53.951+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.952417+0000) 2019-10-13T08:48:53.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:53.954+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151570 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:48:54.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:54.637+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.637653+0000) 2019-10-13T08:48:58.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:58.436+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.436927+0000) 2019-10-13T08:48:58.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:58.774+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.775442+0000) 2019-10-13T08:48:58.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:58.774+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.775446+0000) 2019-10-13T08:48:58.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:58.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.775443+0000) 2019-10-13T08:48:58.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:48:58.956+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.957914+0000) 2019-10-13T08:48:58.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:48:58.956+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.958369+0000) 2019-10-13T08:48:58.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:48:58.957+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.958766+0000) 2019-10-13T08:48:58.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:48:58.957+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.958634+0000) 2019-10-13T08:48:58.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:48:58.957+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.958839+0000) 2019-10-13T08:48:58.960 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:48:58.957+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.959016+0000) 2019-10-13T08:48:58.961 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:48:58.957+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.959102+0000) 2019-10-13T08:48:58.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:48:58.959+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.961198+0000) 2019-10-13T08:48:58.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:48:58.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151598 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:02.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:02.882+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:02.883485+0000) 2019-10-13T08:49:02.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:02.882+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:02.883258+0000) 2019-10-13T08:49:03.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:03.161+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.162914+0000) 2019-10-13T08:49:03.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:03.966+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.966458+0000) 2019-10-13T08:49:03.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:03.966+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.966587+0000) 2019-10-13T08:49:03.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:03.965+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.967115+0000) 2019-10-13T08:49:03.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:03.966+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.967084+0000) 2019-10-13T08:49:03.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:03.966+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.967240+0000) 2019-10-13T08:49:03.969 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:03.967+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.967576+0000) 2019-10-13T08:49:03.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:03.967+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.967809+0000) 2019-10-13T08:49:03.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:03.967+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.969427+0000) 2019-10-13T08:49:03.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:03.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151628 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:04.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:04.637+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.637821+0000) 2019-10-13T08:49:08.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:08.436+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.437099+0000) 2019-10-13T08:49:08.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:08.775+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.775638+0000) 2019-10-13T08:49:08.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:08.775+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.775634+0000) 2019-10-13T08:49:08.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:08.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.775660+0000) 2019-10-13T08:49:08.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:08.974+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.974802+0000) 2019-10-13T08:49:08.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:08.974+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.974995+0000) 2019-10-13T08:49:08.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:08.974+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.975229+0000) 2019-10-13T08:49:08.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:08.974+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.975293+0000) 2019-10-13T08:49:08.978 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:08.975+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.975591+0000) 2019-10-13T08:49:08.978 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:08.975+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.975914+0000) 2019-10-13T08:49:08.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:08.974+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.975858+0000) 2019-10-13T08:49:08.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:08.976+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.978267+0000) 2019-10-13T08:49:08.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:08.980+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151661 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:10.585 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:49:10.585 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:10.590 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:49:10.590 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:12.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:12.882+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:12.883635+0000) 2019-10-13T08:49:12.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:12.883+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:12.883449+0000) 2019-10-13T08:49:13.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:13.164+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.165393+0000) 2019-10-13T08:49:13.984 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:13.982+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.982800+0000) 2019-10-13T08:49:13.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:13.982+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.983281+0000) 2019-10-13T08:49:13.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:13.983+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.983482+0000) 2019-10-13T08:49:13.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:13.983+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.983669+0000) 2019-10-13T08:49:13.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:13.983+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.983794+0000) 2019-10-13T08:49:13.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:13.983+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.984509+0000) 2019-10-13T08:49:13.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:13.984+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.985550+0000) 2019-10-13T08:49:13.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:13.983+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.984273+0000) 2019-10-13T08:49:13.990 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:13.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:14.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:14.637+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.637991+0000) 2019-10-13T08:49:18.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:18.436+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.437278+0000) 2019-10-13T08:49:18.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:18.774+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.775855+0000) 2019-10-13T08:49:18.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:18.774+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.775830+0000) 2019-10-13T08:49:18.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:18.774+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.775862+0000) 2019-10-13T08:49:18.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:18.990+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.991431+0000) 2019-10-13T08:49:18.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:18.990+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.991476+0000) 2019-10-13T08:49:18.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:18.990+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.991505+0000) 2019-10-13T08:49:18.993 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:18.990+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.991764+0000) 2019-10-13T08:49:18.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:18.990+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.992137+0000) 2019-10-13T08:49:18.994 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:18.990+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.992293+0000) 2019-10-13T08:49:18.994 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:18.991+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.992490+0000) 2019-10-13T08:49:18.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:18.993+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.994699+0000) 2019-10-13T08:49:18.998 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:18.996+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151722 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:22.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:22.883+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:22.883782+0000) 2019-10-13T08:49:22.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:22.882+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:22.883609+0000) 2019-10-13T08:49:23.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:23.166+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.167469+0000) 2019-10-13T08:49:24.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:23.998+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.999592+0000) 2019-10-13T08:49:24.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:23.999+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.000409+0000) 2019-10-13T08:49:24.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:23.998+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.000061+0000) 2019-10-13T08:49:24.002 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:23.998+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.000366+0000) 2019-10-13T08:49:24.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:23.999+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.000497+0000) 2019-10-13T08:49:24.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:23.999+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.000790+0000) 2019-10-13T08:49:24.002 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:23.999+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.000950+0000) 2019-10-13T08:49:24.003 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:24.002+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.002814+0000) 2019-10-13T08:49:24.006 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:24.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:24.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:24.637+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.638124+0000) 2019-10-13T08:49:28.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:28.436+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.437444+0000) 2019-10-13T08:49:28.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:28.775+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.775991+0000) 2019-10-13T08:49:28.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:28.775+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.776023+0000) 2019-10-13T08:49:28.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:28.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.776054+0000) 2019-10-13T08:49:29.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:29.006+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.007540+0000) 2019-10-13T08:49:29.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:29.005+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.007243+0000) 2019-10-13T08:49:29.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:29.006+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.007428+0000) 2019-10-13T08:49:29.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:29.006+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.007663+0000) 2019-10-13T08:49:29.009 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:29.006+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.007782+0000) 2019-10-13T08:49:29.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:29.006+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.007899+0000) 2019-10-13T08:49:29.010 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:29.006+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.008071+0000) 2019-10-13T08:49:29.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:29.010+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:29.010830+0000) 2019-10-13T08:49:29.012 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:29.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:32.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:32.883+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:32.883983+0000) 2019-10-13T08:49:32.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:32.882+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:32.883810+0000) 2019-10-13T08:49:33.170 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:33.169+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.169778+0000) 2019-10-13T08:49:34.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:34.012+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.014071+0000) 2019-10-13T08:49:34.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:34.012+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.014240+0000) 2019-10-13T08:49:34.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:34.012+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.014240+0000) 2019-10-13T08:49:34.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:34.014+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.015070+0000) 2019-10-13T08:49:34.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:34.013+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.014971+0000) 2019-10-13T08:49:34.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:34.013+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.015062+0000) 2019-10-13T08:49:34.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:34.013+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.015220+0000) 2019-10-13T08:49:34.017 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:34.016+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.017336+0000) 2019-10-13T08:49:34.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:34.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:34.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:34.637+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.638253+0000) 2019-10-13T08:49:38.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:38.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.437615+0000) 2019-10-13T08:49:38.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:38.775+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.776141+0000) 2019-10-13T08:49:38.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:38.775+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.776177+0000) 2019-10-13T08:49:38.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:38.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.776209+0000) 2019-10-13T08:49:39.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:39.020+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.021568+0000) 2019-10-13T08:49:39.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:39.020+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.021875+0000) 2019-10-13T08:49:39.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:39.020+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.021940+0000) 2019-10-13T08:49:39.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:39.020+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.022360+0000) 2019-10-13T08:49:39.023 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:39.021+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.022512+0000) 2019-10-13T08:49:39.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:39.021+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.022494+0000) 2019-10-13T08:49:39.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:39.021+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.022963+0000) 2019-10-13T08:49:39.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:39.024+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:39.024767+0000) 2019-10-13T08:49:39.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:39.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:40.665 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:49:40.665 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:40.670 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:49:40.670 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:42.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:42.884+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:42.884202+0000) 2019-10-13T08:49:42.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:42.883+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:42.884030+0000) 2019-10-13T08:49:43.172 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:43.171+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.172238+0000) 2019-10-13T08:49:44.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:44.029+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.030300+0000) 2019-10-13T08:49:44.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:44.028+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.030199+0000) 2019-10-13T08:49:44.032 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:44.029+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.030414+0000) 2019-10-13T08:49:44.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:44.029+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.030552+0000) 2019-10-13T08:49:44.032 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:44.029+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.030742+0000) 2019-10-13T08:49:44.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:44.029+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.030868+0000) 2019-10-13T08:49:44.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:44.029+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.031072+0000) 2019-10-13T08:49:44.034 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:44.033+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.033936+0000) 2019-10-13T08:49:44.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:44.035+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151876 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:44.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:44.637+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.638410+0000) 2019-10-13T08:49:48.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:48.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.437711+0000) 2019-10-13T08:49:48.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:48.775+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.776299+0000) 2019-10-13T08:49:48.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:48.775+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.776337+0000) 2019-10-13T08:49:48.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:48.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.776364+0000) 2019-10-13T08:49:49.039 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:49.038+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.038631+0000) 2019-10-13T08:49:49.040 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:49.038+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.038818+0000) 2019-10-13T08:49:49.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:49.038+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.039093+0000) 2019-10-13T08:49:49.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:49.038+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.039636+0000) 2019-10-13T08:49:49.042 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:49.039+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.039411+0000) 2019-10-13T08:49:49.042 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:49.039+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.039534+0000) 2019-10-13T08:49:49.042 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:49.039+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.039873+0000) 2019-10-13T08:49:49.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:49.041+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:49.042242+0000) 2019-10-13T08:49:49.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:49.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:52.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:52.882+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:52.884387+0000) 2019-10-13T08:49:52.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:52.883+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:52.884188+0000) 2019-10-13T08:49:53.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:53.173+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.174674+0000) 2019-10-13T08:49:54.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:54.046+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.046623+0000) 2019-10-13T08:49:54.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:54.046+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.046935+0000) 2019-10-13T08:49:54.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:54.046+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.047818+0000) 2019-10-13T08:49:54.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:54.046+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.047226+0000) 2019-10-13T08:49:54.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:54.047+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.047557+0000) 2019-10-13T08:49:54.049 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:54.047+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.047758+0000) 2019-10-13T08:49:54.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:54.047+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.048015+0000) 2019-10-13T08:49:54.050 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:54.048+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.050328+0000) 2019-10-13T08:49:54.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:54.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:49:54.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:54.638+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.638577+0000) 2019-10-13T08:49:58.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:58.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.437869+0000) 2019-10-13T08:49:58.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:58.776+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.776479+0000) 2019-10-13T08:49:58.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:58.776+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.776504+0000) 2019-10-13T08:49:58.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:58.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.776542+0000) 2019-10-13T08:49:59.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:49:59.054+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.055855+0000) 2019-10-13T08:49:59.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:49:59.055+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.055602+0000) 2019-10-13T08:49:59.057 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:49:59.055+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.055956+0000) 2019-10-13T08:49:59.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:49:59.055+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.056173+0000) 2019-10-13T08:49:59.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:49:59.055+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.056335+0000) 2019-10-13T08:49:59.058 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:49:59.056+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.056771+0000) 2019-10-13T08:49:59.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:49:59.056+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.056956+0000) 2019-10-13T08:49:59.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:49:59.056+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:59.058416+0000) 2019-10-13T08:49:59.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:49:59.061+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 151970 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:02.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:02.883+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:02.884557+0000) 2019-10-13T08:50:02.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:02.884+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:02.884346+0000) 2019-10-13T08:50:03.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:03.176+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.176944+0000) 2019-10-13T08:50:04.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:04.062+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.063846+0000) 2019-10-13T08:50:04.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:04.062+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.064142+0000) 2019-10-13T08:50:04.065 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:04.062+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.064300+0000) 2019-10-13T08:50:04.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:04.063+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.065140+0000) 2019-10-13T08:50:04.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:04.063+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.064554+0000) 2019-10-13T08:50:04.066 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:04.063+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.064815+0000) 2019-10-13T08:50:04.066 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:04.063+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.065304+0000) 2019-10-13T08:50:04.067 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:04.065+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.067009+0000) 2019-10-13T08:50:04.070 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:04.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:04.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:04.637+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.638747+0000) 2019-10-13T08:50:08.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:08.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.438076+0000) 2019-10-13T08:50:08.777 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:08.775+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.776678+0000) 2019-10-13T08:50:08.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:08.775+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.776693+0000) 2019-10-13T08:50:08.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:08.775+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.776694+0000) 2019-10-13T08:50:09.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:09.069+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.071355+0000) 2019-10-13T08:50:09.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:09.070+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.071890+0000) 2019-10-13T08:50:09.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:09.070+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.072087+0000) 2019-10-13T08:50:09.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:09.070+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.072214+0000) 2019-10-13T08:50:09.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:09.070+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.072366+0000) 2019-10-13T08:50:09.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:09.072+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.073075+0000) 2019-10-13T08:50:09.074 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:09.071+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.072971+0000) 2019-10-13T08:50:09.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:09.073+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:09.074361+0000) 2019-10-13T08:50:09.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:09.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:10.762 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:50:10.762 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:10.767 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:50:10.767 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:12.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:12.884+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:12.884785+0000) 2019-10-13T08:50:12.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:12.883+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:12.884561+0000) 2019-10-13T08:50:13.179 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:13.178+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.179260+0000) 2019-10-13T08:50:14.080 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:14.077+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.079019+0000) 2019-10-13T08:50:14.081 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:14.077+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.079111+0000) 2019-10-13T08:50:14.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:14.078+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.079459+0000) 2019-10-13T08:50:14.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:14.078+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.079887+0000) 2019-10-13T08:50:14.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:14.078+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.079956+0000) 2019-10-13T08:50:14.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:14.079+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.080597+0000) 2019-10-13T08:50:14.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:14.081+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.081868+0000) 2019-10-13T08:50:14.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:14.079+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.080486+0000) 2019-10-13T08:50:14.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:14.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152063 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:14.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:14.638+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.638907+0000) 2019-10-13T08:50:18.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:18.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.438246+0000) 2019-10-13T08:50:18.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:18.776+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.776901+0000) 2019-10-13T08:50:18.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:18.776+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.776900+0000) 2019-10-13T08:50:18.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:18.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.776882+0000) 2019-10-13T08:50:19.088 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:19.085+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.087353+0000) 2019-10-13T08:50:19.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:19.086+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.087574+0000) 2019-10-13T08:50:19.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:19.086+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.087668+0000) 2019-10-13T08:50:19.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:19.087+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.088253+0000) 2019-10-13T08:50:19.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:19.086+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.088006+0000) 2019-10-13T08:50:19.090 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:19.086+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.088231+0000) 2019-10-13T08:50:19.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:19.087+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.088582+0000) 2019-10-13T08:50:19.090 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:19.088+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:19.089690+0000) 2019-10-13T08:50:19.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:19.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152094 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:22.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:22.884+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:22.884947+0000) 2019-10-13T08:50:22.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:22.884+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:22.884744+0000) 2019-10-13T08:50:23.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:23.181+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.181721+0000) 2019-10-13T08:50:24.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:24.093+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.095227+0000) 2019-10-13T08:50:24.096 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:24.093+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.095375+0000) 2019-10-13T08:50:24.096 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:24.094+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.095489+0000) 2019-10-13T08:50:24.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:24.094+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.095795+0000) 2019-10-13T08:50:24.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:24.094+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.095962+0000) 2019-10-13T08:50:24.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:24.095+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.096632+0000) 2019-10-13T08:50:24.098 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:24.095+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.096440+0000) 2019-10-13T08:50:24.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:24.097+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.098192+0000) 2019-10-13T08:50:24.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:24.099+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152124 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:24.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:24.638+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.639064+0000) 2019-10-13T08:50:28.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:28.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.438403+0000) 2019-10-13T08:50:28.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:28.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.777048+0000) 2019-10-13T08:50:28.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:28.776+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.777127+0000) 2019-10-13T08:50:28.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:28.776+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.777112+0000) 2019-10-13T08:50:29.103 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:29.101+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.102591+0000) 2019-10-13T08:50:29.104 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:29.101+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.102903+0000) 2019-10-13T08:50:29.104 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:29.101+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.103092+0000) 2019-10-13T08:50:29.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:29.103+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.104217+0000) 2019-10-13T08:50:29.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:29.101+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.103316+0000) 2019-10-13T08:50:29.106 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:29.102+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.103557+0000) 2019-10-13T08:50:29.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:29.102+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.103614+0000) 2019-10-13T08:50:29.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:29.104+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:29.105339+0000) 2019-10-13T08:50:29.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:29.108+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152157 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:32.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:32.884+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:32.885093+0000) 2019-10-13T08:50:32.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:32.884+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:32.884931+0000) 2019-10-13T08:50:33.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:33.183+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.184049+0000) 2019-10-13T08:50:34.112 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:34.111+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.111624+0000) 2019-10-13T08:50:34.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:34.111+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.111928+0000) 2019-10-13T08:50:34.113 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:34.111+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.111927+0000) 2019-10-13T08:50:34.113 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:34.111+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.112285+0000) 2019-10-13T08:50:34.113 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:34.112+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.112495+0000) 2019-10-13T08:50:34.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:34.112+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.113293+0000) 2019-10-13T08:50:34.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:34.112+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.113057+0000) 2019-10-13T08:50:34.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:34.114+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.115315+0000) 2019-10-13T08:50:34.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:34.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:34.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:34.638+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.639230+0000) 2019-10-13T08:50:38.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:38.438+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.438561+0000) 2019-10-13T08:50:38.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:38.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.777228+0000) 2019-10-13T08:50:38.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:38.776+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.777290+0000) 2019-10-13T08:50:38.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:38.776+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.777297+0000) 2019-10-13T08:50:39.120 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:39.119+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.119498+0000) 2019-10-13T08:50:39.121 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:39.119+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.119989+0000) 2019-10-13T08:50:39.121 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:39.119+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.120258+0000) 2019-10-13T08:50:39.121 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:39.119+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.120156+0000) 2019-10-13T08:50:39.121 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:39.120+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.120429+0000) 2019-10-13T08:50:39.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:39.120+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.120589+0000) 2019-10-13T08:50:39.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:39.120+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.120966+0000) 2019-10-13T08:50:39.122 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:39.121+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:39.122354+0000) 2019-10-13T08:50:39.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:39.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152218 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:40.928 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:50:40.928 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:40.933 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:50:40.933 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:42.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:42.884+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:42.885272+0000) 2019-10-13T08:50:42.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:42.884+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:42.885104+0000) 2019-10-13T08:50:43.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:43.184+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.186629+0000) 2019-10-13T08:50:44.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:44.126+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.127317+0000) 2019-10-13T08:50:44.128 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:44.127+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.127515+0000) 2019-10-13T08:50:44.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:44.126+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.128108+0000) 2019-10-13T08:50:44.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:44.127+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.127839+0000) 2019-10-13T08:50:44.129 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:44.127+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.128030+0000) 2019-10-13T08:50:44.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:44.128+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.128492+0000) 2019-10-13T08:50:44.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:44.128+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.128526+0000) 2019-10-13T08:50:44.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:44.129+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.131306+0000) 2019-10-13T08:50:44.135 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:44.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152248 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:44.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:44.639+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.639405+0000) 2019-10-13T08:50:48.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:48.438+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.438710+0000) 2019-10-13T08:50:48.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:48.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.777392+0000) 2019-10-13T08:50:48.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:48.777+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.777479+0000) 2019-10-13T08:50:48.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:48.777+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.777479+0000) 2019-10-13T08:50:49.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:49.135+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.135819+0000) 2019-10-13T08:50:49.137 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:49.135+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.136098+0000) 2019-10-13T08:50:49.137 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:49.135+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.136120+0000) 2019-10-13T08:50:49.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:49.136+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.137083+0000) 2019-10-13T08:50:49.138 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:49.135+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.137680+0000) 2019-10-13T08:50:49.138 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:49.137+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.137436+0000) 2019-10-13T08:50:49.138 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:49.137+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.137655+0000) 2019-10-13T08:50:49.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:49.137+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:49.139088+0000) 2019-10-13T08:50:49.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:49.141+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:52.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:52.883+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:52.885477+0000) 2019-10-13T08:50:52.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:52.883+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:52.885279+0000) 2019-10-13T08:50:53.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:53.188+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.189011+0000) 2019-10-13T08:50:54.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:54.142+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.144054+0000) 2019-10-13T08:50:54.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:54.142+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.144180+0000) 2019-10-13T08:50:54.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:54.143+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.144699+0000) 2019-10-13T08:50:54.146 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:54.143+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.145108+0000) 2019-10-13T08:50:54.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:54.143+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.144911+0000) 2019-10-13T08:50:54.146 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:54.143+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.145592+0000) 2019-10-13T08:50:54.146 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:54.144+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.145511+0000) 2019-10-13T08:50:54.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:54.145+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.146798+0000) 2019-10-13T08:50:54.150 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:54.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152311 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:50:54.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:54.638+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.639592+0000) 2019-10-13T08:50:58.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:58.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.438850+0000) 2019-10-13T08:50:58.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:58.776+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.777538+0000) 2019-10-13T08:50:58.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:58.776+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.777606+0000) 2019-10-13T08:50:58.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:58.776+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.777631+0000) 2019-10-13T08:50:59.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:50:59.150+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.152188+0000) 2019-10-13T08:50:59.153 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:50:59.150+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.152326+0000) 2019-10-13T08:50:59.154 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:50:59.152+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.153218+0000) 2019-10-13T08:50:59.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:50:59.151+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.152898+0000) 2019-10-13T08:50:59.154 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:50:59.151+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.153052+0000) 2019-10-13T08:50:59.154 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:50:59.151+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.153177+0000) 2019-10-13T08:50:59.155 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:50:59.152+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.153402+0000) 2019-10-13T08:50:59.155 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:50:59.154+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:59.155513+0000) 2019-10-13T08:50:59.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:50:59.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152342 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:02.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:02.884+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:02.885644+0000) 2019-10-13T08:51:02.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:02.884+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:02.885462+0000) 2019-10-13T08:51:03.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:03.190+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.191551+0000) 2019-10-13T08:51:04.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:04.156+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.157775+0000) 2019-10-13T08:51:04.159 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:04.156+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.157643+0000) 2019-10-13T08:51:04.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:04.156+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.158163+0000) 2019-10-13T08:51:04.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:04.158+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.159131+0000) 2019-10-13T08:51:04.160 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:04.157+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.158831+0000) 2019-10-13T08:51:04.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:04.157+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.159045+0000) 2019-10-13T08:51:04.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:04.157+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.159107+0000) 2019-10-13T08:51:04.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:04.160+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.160905+0000) 2019-10-13T08:51:04.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:04.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:04.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:04.639+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.639758+0000) 2019-10-13T08:51:08.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:08.437+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.439030+0000) 2019-10-13T08:51:08.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:08.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.777712+0000) 2019-10-13T08:51:08.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:08.777+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.777788+0000) 2019-10-13T08:51:08.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:08.777+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.777770+0000) 2019-10-13T08:51:09.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:09.164+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.166157+0000) 2019-10-13T08:51:09.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:09.165+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.166393+0000) 2019-10-13T08:51:09.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:09.165+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.166605+0000) 2019-10-13T08:51:09.168 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:09.165+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.166787+0000) 2019-10-13T08:51:09.168 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:09.165+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.167098+0000) 2019-10-13T08:51:09.169 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:09.165+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.167244+0000) 2019-10-13T08:51:09.169 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:09.166+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.167379+0000) 2019-10-13T08:51:09.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:09.168+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:09.169312+0000) 2019-10-13T08:51:09.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:09.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152405 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:11.100 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:51:11.100 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:11.104 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:51:11.104 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:12.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:12.885+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:12.885788+0000) 2019-10-13T08:51:12.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:12.885+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:12.885614+0000) 2019-10-13T08:51:13.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:13.193+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.193910+0000) 2019-10-13T08:51:14.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:14.173+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.174091+0000) 2019-10-13T08:51:14.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:14.172+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.173995+0000) 2019-10-13T08:51:14.175 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:14.172+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.174154+0000) 2019-10-13T08:51:14.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:14.173+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.174625+0000) 2019-10-13T08:51:14.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:14.173+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.175137+0000) 2019-10-13T08:51:14.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:14.173+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.174949+0000) 2019-10-13T08:51:14.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:14.173+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.174792+0000) 2019-10-13T08:51:14.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:14.175+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.176572+0000) 2019-10-13T08:51:14.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:14.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:14.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:14.639+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.639910+0000) 2019-10-13T08:51:18.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:18.438+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.439241+0000) 2019-10-13T08:51:18.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:18.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.777896+0000) 2019-10-13T08:51:18.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:18.777+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.778008+0000) 2019-10-13T08:51:18.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:18.777+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.777966+0000) 2019-10-13T08:51:19.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:19.181+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.182308+0000) 2019-10-13T08:51:19.183 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:19.182+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.182428+0000) 2019-10-13T08:51:19.183 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:19.182+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.182726+0000) 2019-10-13T08:51:19.184 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:19.182+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.183376+0000) 2019-10-13T08:51:19.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:19.182+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.183200+0000) 2019-10-13T08:51:19.185 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:19.183+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.183506+0000) 2019-10-13T08:51:19.185 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:19.183+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.183733+0000) 2019-10-13T08:51:19.185 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:19.184+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:19.185215+0000) 2019-10-13T08:51:19.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:19.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152466 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:22.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:22.885+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:22.886000+0000) 2019-10-13T08:51:22.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:22.885+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:22.885804+0000) 2019-10-13T08:51:23.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:23.195+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.196211+0000) 2019-10-13T08:51:24.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:24.189+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.190132+0000) 2019-10-13T08:51:24.191 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:24.190+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.190439+0000) 2019-10-13T08:51:24.192 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:24.190+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.191476+0000) 2019-10-13T08:51:24.192 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:24.190+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.191127+0000) 2019-10-13T08:51:24.192 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:24.190+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.191381+0000) 2019-10-13T08:51:24.193 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:24.191+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.191487+0000) 2019-10-13T08:51:24.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:24.191+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.191658+0000) 2019-10-13T08:51:24.195 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:24.194+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.195017+0000) 2019-10-13T08:51:24.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:24.197+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:24.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:24.639+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.640077+0000) 2019-10-13T08:51:28.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:28.439+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.439477+0000) 2019-10-13T08:51:28.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:28.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.778105+0000) 2019-10-13T08:51:28.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:28.777+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.778164+0000) 2019-10-13T08:51:28.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:28.777+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.778222+0000) 2019-10-13T08:51:29.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:29.199+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.199912+0000) 2019-10-13T08:51:29.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:29.199+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.199672+0000) 2019-10-13T08:51:29.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:29.199+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.199935+0000) 2019-10-13T08:51:29.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:29.199+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.200081+0000) 2019-10-13T08:51:29.202 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:29.199+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.200205+0000) 2019-10-13T08:51:29.202 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:29.200+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.200780+0000) 2019-10-13T08:51:29.202 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:29.200+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.200927+0000) 2019-10-13T08:51:29.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:29.201+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:29.202351+0000) 2019-10-13T08:51:29.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:29.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152529 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:32.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:32.886+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:32.886183+0000) 2019-10-13T08:51:32.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:32.885+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:32.886002+0000) 2019-10-13T08:51:33.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:33.198+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.198559+0000) 2019-10-13T08:51:34.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:34.207+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.207681+0000) 2019-10-13T08:51:34.209 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:34.207+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.208396+0000) 2019-10-13T08:51:34.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:34.207+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.208120+0000) 2019-10-13T08:51:34.210 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:34.207+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.208243+0000) 2019-10-13T08:51:34.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:34.207+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.208386+0000) 2019-10-13T08:51:34.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:34.208+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.208593+0000) 2019-10-13T08:51:34.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:34.208+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.208997+0000) 2019-10-13T08:51:34.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:34.209+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.210001+0000) 2019-10-13T08:51:34.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:34.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152559 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:34.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:34.640+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.640242+0000) 2019-10-13T08:51:38.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:38.438+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.439657+0000) 2019-10-13T08:51:38.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:38.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.778260+0000) 2019-10-13T08:51:38.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:38.777+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.778331+0000) 2019-10-13T08:51:38.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:38.777+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.778323+0000) 2019-10-13T08:51:39.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:39.213+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.214902+0000) 2019-10-13T08:51:39.216 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:39.213+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.215316+0000) 2019-10-13T08:51:39.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:39.213+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.215203+0000) 2019-10-13T08:51:39.216 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:39.214+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.215533+0000) 2019-10-13T08:51:39.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:39.214+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.215651+0000) 2019-10-13T08:51:39.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:39.213+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.215471+0000) 2019-10-13T08:51:39.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:39.214+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.215820+0000) 2019-10-13T08:51:39.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:39.215+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:39.217345+0000) 2019-10-13T08:51:39.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:39.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152590 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:41.220 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:51:41.220 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:41.224 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:51:41.225 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:42.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:42.885+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:42.886388+0000) 2019-10-13T08:51:42.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:42.885+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:42.886209+0000) 2019-10-13T08:51:43.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:43.200+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.201068+0000) 2019-10-13T08:51:44.222 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:44.220+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.221771+0000) 2019-10-13T08:51:44.222 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:44.220+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.222002+0000) 2019-10-13T08:51:44.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:44.221+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.222461+0000) 2019-10-13T08:51:44.223 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:44.221+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.222719+0000) 2019-10-13T08:51:44.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:44.221+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.222857+0000) 2019-10-13T08:51:44.224 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:44.221+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.223167+0000) 2019-10-13T08:51:44.224 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:44.221+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.223533+0000) 2019-10-13T08:51:44.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:44.224+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.226128+0000) 2019-10-13T08:51:44.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:44.226+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:44.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:44.639+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.640439+0000) 2019-10-13T08:51:48.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:48.439+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.439828+0000) 2019-10-13T08:51:48.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:48.777+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.778442+0000) 2019-10-13T08:51:48.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:48.777+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.778420+0000) 2019-10-13T08:51:48.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:48.777+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.778476+0000) 2019-10-13T08:51:49.231 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:49.228+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.230024+0000) 2019-10-13T08:51:49.231 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:49.228+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.230112+0000) 2019-10-13T08:51:49.231 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:49.229+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.230535+0000) 2019-10-13T08:51:49.231 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:49.229+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.230729+0000) 2019-10-13T08:51:49.232 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:49.230+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.231360+0000) 2019-10-13T08:51:49.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:49.229+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.230957+0000) 2019-10-13T08:51:49.233 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:49.229+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.231123+0000) 2019-10-13T08:51:49.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:49.232+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:49.233528+0000) 2019-10-13T08:51:49.237 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:49.235+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:52.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:52.885+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:52.886586+0000) 2019-10-13T08:51:52.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:52.885+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:52.886350+0000) 2019-10-13T08:51:53.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:53.202+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.203599+0000) 2019-10-13T08:51:54.239 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:54.238+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.238835+0000) 2019-10-13T08:51:54.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:54.237+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.238542+0000) 2019-10-13T08:51:54.240 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:54.237+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.238795+0000) 2019-10-13T08:51:54.240 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:54.237+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.239108+0000) 2019-10-13T08:51:54.240 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:54.237+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.239388+0000) 2019-10-13T08:51:54.241 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:54.238+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.239666+0000) 2019-10-13T08:51:54.241 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:54.238+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.239805+0000) 2019-10-13T08:51:54.242 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:54.241+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.242104+0000) 2019-10-13T08:51:54.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:54.244+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:51:54.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:54.640+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.640607+0000) 2019-10-13T08:51:58.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:58.439+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:58.440015+0000) 2019-10-13T08:51:58.779 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:58.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:58.778608+0000) 2019-10-13T08:51:58.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:58.778+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:58.778656+0000) 2019-10-13T08:51:58.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:58.778+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:58.778670+0000) 2019-10-13T08:51:59.248 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:51:59.247+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.247914+0000) 2019-10-13T08:51:59.249 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:51:59.246+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.247620+0000) 2019-10-13T08:51:59.249 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:51:59.246+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.247910+0000) 2019-10-13T08:51:59.249 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:51:59.246+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.248012+0000) 2019-10-13T08:51:59.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:51:59.246+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.248252+0000) 2019-10-13T08:51:59.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:51:59.246+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.248307+0000) 2019-10-13T08:51:59.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:51:59.247+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.248711+0000) 2019-10-13T08:51:59.251 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:51:59.250+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.251005+0000) 2019-10-13T08:51:59.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:51:59.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:02.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:02.886+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:02.886718+0000) 2019-10-13T08:52:02.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:02.886+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:02.886503+0000) 2019-10-13T08:52:03.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:03.205+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:03.205937+0000) 2019-10-13T08:52:04.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:04.255+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.255452+0000) 2019-10-13T08:52:04.257 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:04.255+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.256124+0000) 2019-10-13T08:52:04.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:04.255+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.256156+0000) 2019-10-13T08:52:04.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:04.256+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.256392+0000) 2019-10-13T08:52:04.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:04.256+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.256642+0000) 2019-10-13T08:52:04.258 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:04.256+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.257052+0000) 2019-10-13T08:52:04.258 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:04.257+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.257460+0000) 2019-10-13T08:52:04.265 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:04.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:04.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:04.258+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.259445+0000) 2019-10-13T08:52:04.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:04.640+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.640756+0000) 2019-10-13T08:52:08.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:08.439+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:08.440246+0000) 2019-10-13T08:52:08.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:08.778+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:08.778869+0000) 2019-10-13T08:52:08.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:08.778+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:08.778881+0000) 2019-10-13T08:52:08.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:08.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:08.778864+0000) 2019-10-13T08:52:09.265 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:09.264+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.264409+0000) 2019-10-13T08:52:09.265 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:09.263+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.264275+0000) 2019-10-13T08:52:09.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:09.264+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.264771+0000) 2019-10-13T08:52:09.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:09.265+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.265741+0000) 2019-10-13T08:52:09.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:09.264+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.265042+0000) 2019-10-13T08:52:09.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:09.265+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.265577+0000) 2019-10-13T08:52:09.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:09.265+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.265771+0000) 2019-10-13T08:52:09.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:09.266+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.267169+0000) 2019-10-13T08:52:09.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:09.270+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:11.399 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:52:11.399 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:11.403 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:52:11.403 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:12.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:12.886+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:12.886711+0000) 2019-10-13T08:52:12.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:12.886+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:12.886898+0000) 2019-10-13T08:52:13.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:13.207+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:13.208347+0000) 2019-10-13T08:52:14.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:14.272+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.273158+0000) 2019-10-13T08:52:14.274 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:14.272+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.272916+0000) 2019-10-13T08:52:14.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:14.272+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.273177+0000) 2019-10-13T08:52:14.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:14.273+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.273608+0000) 2019-10-13T08:52:14.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:14.273+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.273718+0000) 2019-10-13T08:52:14.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:14.273+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.274021+0000) 2019-10-13T08:52:14.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:14.273+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.274190+0000) 2019-10-13T08:52:14.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:14.275+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.276642+0000) 2019-10-13T08:52:14.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:14.278+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152807 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:14.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:14.640+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.640967+0000) 2019-10-13T08:52:18.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:18.440+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:18.440436+0000) 2019-10-13T08:52:18.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:18.778+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:18.779041+0000) 2019-10-13T08:52:18.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:18.778+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:18.779057+0000) 2019-10-13T08:52:18.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:18.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:18.779053+0000) 2019-10-13T08:52:19.281 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:19.280+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.281072+0000) 2019-10-13T08:52:19.282 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:19.280+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.281373+0000) 2019-10-13T08:52:19.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:19.281+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.281637+0000) 2019-10-13T08:52:19.283 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:19.281+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.281822+0000) 2019-10-13T08:52:19.283 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:19.281+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.282014+0000) 2019-10-13T08:52:19.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:19.281+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.282134+0000) 2019-10-13T08:52:19.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:19.282+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.282560+0000) 2019-10-13T08:52:19.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:19.283+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.284047+0000) 2019-10-13T08:52:19.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:19.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152838 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:22.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:22.886+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:22.887090+0000) 2019-10-13T08:52:22.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:22.885+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:22.886901+0000) 2019-10-13T08:52:23.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:23.210+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:23.210727+0000) 2019-10-13T08:52:24.290 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:24.288+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.289940+0000) 2019-10-13T08:52:24.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:24.289+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.290609+0000) 2019-10-13T08:52:24.292 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:24.289+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.290651+0000) 2019-10-13T08:52:24.292 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:24.289+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.290803+0000) 2019-10-13T08:52:24.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:24.289+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.290951+0000) 2019-10-13T08:52:24.292 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:24.289+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.291135+0000) 2019-10-13T08:52:24.292 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:24.289+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.291259+0000) 2019-10-13T08:52:24.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:24.292+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.292951+0000) 2019-10-13T08:52:24.297 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:24.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152868 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:24.641 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:24.640+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.641136+0000) 2019-10-13T08:52:28.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:28.439+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:28.440624+0000) 2019-10-13T08:52:28.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:28.778+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:28.779277+0000) 2019-10-13T08:52:28.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:28.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:28.779275+0000) 2019-10-13T08:52:28.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:28.778+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:28.779275+0000) 2019-10-13T08:52:29.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:29.297+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.299167+0000) 2019-10-13T08:52:29.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:29.298+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.299407+0000) 2019-10-13T08:52:29.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:29.298+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.299653+0000) 2019-10-13T08:52:29.301 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:29.298+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.299799+0000) 2019-10-13T08:52:29.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:29.298+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.299935+0000) 2019-10-13T08:52:29.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:29.298+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.300526+0000) 2019-10-13T08:52:29.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:29.298+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.300215+0000) 2019-10-13T08:52:29.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:29.300+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.302275+0000) 2019-10-13T08:52:29.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:29.302+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152901 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:32.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:32.886+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:32.887258+0000) 2019-10-13T08:52:32.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:32.886+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:32.887018+0000) 2019-10-13T08:52:33.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:33.211+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:33.213005+0000) 2019-10-13T08:52:34.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:34.303+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.305293+0000) 2019-10-13T08:52:34.306 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:34.304+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.305496+0000) 2019-10-13T08:52:34.306 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:34.304+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.305753+0000) 2019-10-13T08:52:34.307 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:34.304+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.306415+0000) 2019-10-13T08:52:34.307 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:34.304+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.306171+0000) 2019-10-13T08:52:34.307 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:34.305+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.306409+0000) 2019-10-13T08:52:34.308 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:34.305+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.306660+0000) 2019-10-13T08:52:34.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:34.306+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.308583+0000) 2019-10-13T08:52:34.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:34.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:34.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:34.639+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.641308+0000) 2019-10-13T08:52:38.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:38.439+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:38.440817+0000) 2019-10-13T08:52:38.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:38.778+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:38.779473+0000) 2019-10-13T08:52:38.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:38.778+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:38.779477+0000) 2019-10-13T08:52:38.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:38.778+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:38.779504+0000) 2019-10-13T08:52:39.315 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:39.313+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.314777+0000) 2019-10-13T08:52:39.316 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:39.312+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.314189+0000) 2019-10-13T08:52:39.316 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:39.313+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.314531+0000) 2019-10-13T08:52:39.316 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:39.313+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.314708+0000) 2019-10-13T08:52:39.316 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:39.313+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.314826+0000) 2019-10-13T08:52:39.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:39.313+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.315105+0000) 2019-10-13T08:52:39.317 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:39.313+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.315257+0000) 2019-10-13T08:52:39.317 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:39.314+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.316408+0000) 2019-10-13T08:52:39.321 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:39.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:41.562 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:52:41.562 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:41.567 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:52:41.567 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:42.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:42.886+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:42.887438+0000) 2019-10-13T08:52:42.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:42.886+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:42.887182+0000) 2019-10-13T08:52:43.215 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:43.214+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:43.215395+0000) 2019-10-13T08:52:44.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:44.320+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.321486+0000) 2019-10-13T08:52:44.323 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:44.320+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.321903+0000) 2019-10-13T08:52:44.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:44.320+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.322231+0000) 2019-10-13T08:52:44.323 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:44.321+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.322577+0000) 2019-10-13T08:52:44.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:44.321+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.322728+0000) 2019-10-13T08:52:44.324 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:44.322+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.322910+0000) 2019-10-13T08:52:44.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:44.321+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.322945+0000) 2019-10-13T08:52:44.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:44.323+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.324375+0000) 2019-10-13T08:52:44.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:44.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 152992 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:44.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:44.640+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.641472+0000) 2019-10-13T08:52:48.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:48.440+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:48.440974+0000) 2019-10-13T08:52:48.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:48.779+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:48.779674+0000) 2019-10-13T08:52:48.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:48.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:48.779641+0000) 2019-10-13T08:52:48.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:48.779+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:48.779643+0000) 2019-10-13T08:52:49.328 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:49.327+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.328679+0000) 2019-10-13T08:52:49.329 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:49.327+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.328364+0000) 2019-10-13T08:52:49.330 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:49.328+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.328867+0000) 2019-10-13T08:52:49.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:49.328+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.328976+0000) 2019-10-13T08:52:49.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:49.328+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.329183+0000) 2019-10-13T08:52:49.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:49.329+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.329394+0000) 2019-10-13T08:52:49.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:49.330+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.331097+0000) 2019-10-13T08:52:49.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:49.329+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.329724+0000) 2019-10-13T08:52:49.335 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:49.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:52.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:52.886+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:52.887626+0000) 2019-10-13T08:52:52.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:52.887+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:52.887410+0000) 2019-10-13T08:52:53.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:53.217+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:53.217941+0000) 2019-10-13T08:52:54.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:54.336+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.337297+0000) 2019-10-13T08:52:54.338 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:54.337+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.337424+0000) 2019-10-13T08:52:54.338 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:54.337+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.337582+0000) 2019-10-13T08:52:54.339 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:54.337+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.337797+0000) 2019-10-13T08:52:54.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:54.338+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.338988+0000) 2019-10-13T08:52:54.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:54.337+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.338006+0000) 2019-10-13T08:52:54.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:54.337+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.338333+0000) 2019-10-13T08:52:54.341 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:54.341+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.341841+0000) 2019-10-13T08:52:54.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:54.343+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:52:54.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:54.641+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.641635+0000) 2019-10-13T08:52:58.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:58.440+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:58.441167+0000) 2019-10-13T08:52:58.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:58.779+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:58.779796+0000) 2019-10-13T08:52:58.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:58.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:58.779827+0000) 2019-10-13T08:52:58.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:58.779+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:58.779882+0000) 2019-10-13T08:52:59.347 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:52:59.345+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.346444+0000) 2019-10-13T08:52:59.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:52:59.345+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.345757+0000) 2019-10-13T08:52:59.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:52:59.345+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.345904+0000) 2019-10-13T08:52:59.347 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:52:59.345+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.346228+0000) 2019-10-13T08:52:59.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:52:59.346+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.346772+0000) 2019-10-13T08:52:59.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:52:59.346+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.347103+0000) 2019-10-13T08:52:59.348 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:52:59.346+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.347303+0000) 2019-10-13T08:52:59.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:52:59.348+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.348834+0000) 2019-10-13T08:52:59.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:52:59.351+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:02.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:02.887+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:02.887819+0000) 2019-10-13T08:53:02.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:02.887+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:02.887586+0000) 2019-10-13T08:53:03.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:03.219+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:03.220379+0000) 2019-10-13T08:53:04.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:04.353+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.353879+0000) 2019-10-13T08:53:04.355 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:04.353+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.354713+0000) 2019-10-13T08:53:04.356 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:04.354+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.354416+0000) 2019-10-13T08:53:04.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:04.354+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.354565+0000) 2019-10-13T08:53:04.356 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:04.354+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.354802+0000) 2019-10-13T08:53:04.356 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:04.354+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.354976+0000) 2019-10-13T08:53:04.357 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:04.354+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.355228+0000) 2019-10-13T08:53:04.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:04.356+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.357302+0000) 2019-10-13T08:53:04.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:04.359+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:04.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:04.641+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.641797+0000) 2019-10-13T08:53:08.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:08.439+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:08.441369+0000) 2019-10-13T08:53:08.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:08.779+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:08.780012+0000) 2019-10-13T08:53:08.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:08.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:08.780021+0000) 2019-10-13T08:53:08.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:08.779+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:08.780110+0000) 2019-10-13T08:53:09.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:09.360+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.361521+0000) 2019-10-13T08:53:09.363 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:09.361+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.362546+0000) 2019-10-13T08:53:09.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:09.360+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.361953+0000) 2019-10-13T08:53:09.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:09.361+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.362571+0000) 2019-10-13T08:53:09.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:09.361+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.362786+0000) 2019-10-13T08:53:09.364 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:09.361+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.362926+0000) 2019-10-13T08:53:09.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:09.361+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.363068+0000) 2019-10-13T08:53:09.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:09.363+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.363974+0000) 2019-10-13T08:53:09.369 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:09.366+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:11.874 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:53:11.874 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:11.879 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:53:11.879 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:12.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:12.887+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:12.888001+0000) 2019-10-13T08:53:12.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:12.886+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:12.887764+0000) 2019-10-13T08:53:13.223 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:13.222+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:13.222874+0000) 2019-10-13T08:53:14.371 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:14.368+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.370100+0000) 2019-10-13T08:53:14.371 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:14.369+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.370517+0000) 2019-10-13T08:53:14.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:14.369+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.370818+0000) 2019-10-13T08:53:14.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:14.369+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.370889+0000) 2019-10-13T08:53:14.372 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:14.371+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.371895+0000) 2019-10-13T08:53:14.372 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:14.370+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.371706+0000) 2019-10-13T08:53:14.373 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:14.370+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.371886+0000) 2019-10-13T08:53:14.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:14.374+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.374794+0000) 2019-10-13T08:53:14.376 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:14.374+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153180 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:14.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:14.640+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.641983+0000) 2019-10-13T08:53:18.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:18.440+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:18.441531+0000) 2019-10-13T08:53:18.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:18.779+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:18.780233+0000) 2019-10-13T08:53:18.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:18.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:18.780234+0000) 2019-10-13T08:53:18.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:18.779+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:18.780310+0000) 2019-10-13T08:53:19.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:19.376+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.377624+0000) 2019-10-13T08:53:19.378 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:19.376+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.377918+0000) 2019-10-13T08:53:19.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:19.377+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.378237+0000) 2019-10-13T08:53:19.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:19.376+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.378134+0000) 2019-10-13T08:53:19.379 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:19.377+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.378424+0000) 2019-10-13T08:53:19.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:19.377+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.378654+0000) 2019-10-13T08:53:19.380 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:19.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.378914+0000) 2019-10-13T08:53:19.381 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:19.380+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.381352+0000) 2019-10-13T08:53:19.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:19.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153210 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:22.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:22.887+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:22.888187+0000) 2019-10-13T08:53:22.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:22.887+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:22.887957+0000) 2019-10-13T08:53:23.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:23.224+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:23.225188+0000) 2019-10-13T08:53:24.385 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:24.383+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.385150+0000) 2019-10-13T08:53:24.386 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:24.383+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.384881+0000) 2019-10-13T08:53:24.386 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:24.383+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.385151+0000) 2019-10-13T08:53:24.386 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:24.383+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.385384+0000) 2019-10-13T08:53:24.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:24.384+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.385678+0000) 2019-10-13T08:53:24.387 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:24.384+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.385828+0000) 2019-10-13T08:53:24.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:24.385+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.387103+0000) 2019-10-13T08:53:24.388 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:24.384+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.386115+0000) 2019-10-13T08:53:24.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:24.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:24.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:24.641+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.642152+0000) 2019-10-13T08:53:28.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:28.441+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:28.441695+0000) 2019-10-13T08:53:28.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:28.779+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:28.780423+0000) 2019-10-13T08:53:28.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:28.779+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:28.780427+0000) 2019-10-13T08:53:28.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:28.779+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:28.780505+0000) 2019-10-13T08:53:29.394 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:29.391+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.393116+0000) 2019-10-13T08:53:29.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:29.392+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.393516+0000) 2019-10-13T08:53:29.395 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:29.392+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.393613+0000) 2019-10-13T08:53:29.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:29.392+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.394315+0000) 2019-10-13T08:53:29.396 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:29.392+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.394272+0000) 2019-10-13T08:53:29.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:29.393+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.394395+0000) 2019-10-13T08:53:29.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:29.393+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.394870+0000) 2019-10-13T08:53:29.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:29.394+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.396716+0000) 2019-10-13T08:53:29.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:29.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:32.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:32.887+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:32.888366+0000) 2019-10-13T08:53:32.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:32.886+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:32.888146+0000) 2019-10-13T08:53:33.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:33.226+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:33.227546+0000) 2019-10-13T08:53:34.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:34.400+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.401632+0000) 2019-10-13T08:53:34.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:34.400+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.401980+0000) 2019-10-13T08:53:34.403 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:34.400+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.402118+0000) 2019-10-13T08:53:34.403 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:34.402+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.403180+0000) 2019-10-13T08:53:34.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:34.400+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.402293+0000) 2019-10-13T08:53:34.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:34.401+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.402535+0000) 2019-10-13T08:53:34.404 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:34.401+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.402543+0000) 2019-10-13T08:53:34.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:34.404+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.405469+0000) 2019-10-13T08:53:34.408 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:34.406+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:34.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:34.641+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.642301+0000) 2019-10-13T08:53:38.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:38.441+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:38.441879+0000) 2019-10-13T08:53:38.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:38.780+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:38.780629+0000) 2019-10-13T08:53:38.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:38.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:38.780621+0000) 2019-10-13T08:53:38.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:38.780+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:38.780695+0000) 2019-10-13T08:53:39.410 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:39.408+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.409308+0000) 2019-10-13T08:53:39.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:39.409+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.410338+0000) 2019-10-13T08:53:39.411 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:39.410+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.410945+0000) 2019-10-13T08:53:39.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:39.410+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.410606+0000) 2019-10-13T08:53:39.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:39.410+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.410916+0000) 2019-10-13T08:53:39.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:39.410+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.411057+0000) 2019-10-13T08:53:39.412 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:39.410+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.411262+0000) 2019-10-13T08:53:39.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:39.412+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.413390+0000) 2019-10-13T08:53:39.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:39.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:42.037 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:53:42.038 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:42.042 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:53:42.042 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:42.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:42.887+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:42.888524+0000) 2019-10-13T08:53:42.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:42.888+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:42.888324+0000) 2019-10-13T08:53:43.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:43.229+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:43.229926+0000) 2019-10-13T08:53:44.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:44.417+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.417879+0000) 2019-10-13T08:53:44.419 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:44.417+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.418184+0000) 2019-10-13T08:53:44.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:44.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.418545+0000) 2019-10-13T08:53:44.420 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:44.418+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.418700+0000) 2019-10-13T08:53:44.420 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:44.418+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.419054+0000) 2019-10-13T08:53:44.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:44.418+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.419182+0000) 2019-10-13T08:53:44.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:44.418+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.419311+0000) 2019-10-13T08:53:44.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:44.420+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.421674+0000) 2019-10-13T08:53:44.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:44.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:44.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:44.642+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.642424+0000) 2019-10-13T08:53:48.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:48.441+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:48.442076+0000) 2019-10-13T08:53:48.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:48.780+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:48.780825+0000) 2019-10-13T08:53:48.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:48.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:48.780822+0000) 2019-10-13T08:53:48.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:48.780+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:48.780910+0000) 2019-10-13T08:53:49.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:49.425+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.426463+0000) 2019-10-13T08:53:49.427 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:49.425+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.426279+0000) 2019-10-13T08:53:49.428 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:49.425+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.426151+0000) 2019-10-13T08:53:49.428 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:49.426+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.426401+0000) 2019-10-13T08:53:49.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:49.426+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.426581+0000) 2019-10-13T08:53:49.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:49.426+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.426652+0000) 2019-10-13T08:53:49.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:49.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.426847+0000) 2019-10-13T08:53:49.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:49.427+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.428284+0000) 2019-10-13T08:53:49.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:49.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153397 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:52.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:52.888+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:52.888722+0000) 2019-10-13T08:53:52.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:52.887+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:52.888503+0000) 2019-10-13T08:53:53.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:53.231+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:53.232229+0000) 2019-10-13T08:53:54.434 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:54.432+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.434055+0000) 2019-10-13T08:53:54.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:54.434+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.435261+0000) 2019-10-13T08:53:54.436 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:54.433+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.434527+0000) 2019-10-13T08:53:54.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:54.433+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.435104+0000) 2019-10-13T08:53:54.436 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:54.433+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.435171+0000) 2019-10-13T08:53:54.436 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:54.434+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.435510+0000) 2019-10-13T08:53:54.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:54.434+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.435560+0000) 2019-10-13T08:53:54.438 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:54.437+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.438249+0000) 2019-10-13T08:53:54.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:54.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:53:54.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:54.641+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.642537+0000) 2019-10-13T08:53:58.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:58.441+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:58.442274+0000) 2019-10-13T08:53:58.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:58.780+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:58.781032+0000) 2019-10-13T08:53:58.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:58.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:58.781022+0000) 2019-10-13T08:53:58.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:58.780+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:58.781102+0000) 2019-10-13T08:53:59.444 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:53:59.441+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.442645+0000) 2019-10-13T08:53:59.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:53:59.441+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.442736+0000) 2019-10-13T08:53:59.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:53:59.441+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.442916+0000) 2019-10-13T08:53:59.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:53:59.441+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.443232+0000) 2019-10-13T08:53:59.445 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:53:59.442+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.443559+0000) 2019-10-13T08:53:59.446 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:53:59.442+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.443650+0000) 2019-10-13T08:53:59.446 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:53:59.442+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.443487+0000) 2019-10-13T08:53:59.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:53:59.445+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.445985+0000) 2019-10-13T08:53:59.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:53:59.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:02.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:02.888+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:02.888914+0000) 2019-10-13T08:54:02.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:02.887+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:02.888684+0000) 2019-10-13T08:54:03.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:03.234+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:03.234508+0000) 2019-10-13T08:54:04.450 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:04.447+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.449289+0000) 2019-10-13T08:54:04.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:04.448+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.449552+0000) 2019-10-13T08:54:04.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:04.448+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.449836+0000) 2019-10-13T08:54:04.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:04.449+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.450651+0000) 2019-10-13T08:54:04.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:04.448+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.450014+0000) 2019-10-13T08:54:04.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:04.449+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.450416+0000) 2019-10-13T08:54:04.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:04.449+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.450658+0000) 2019-10-13T08:54:04.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:04.452+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.452854+0000) 2019-10-13T08:54:04.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:04.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153490 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:04.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:04.641+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.642664+0000) 2019-10-13T08:54:08.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:08.441+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:08.442451+0000) 2019-10-13T08:54:08.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:08.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:08.781225+0000) 2019-10-13T08:54:08.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:08.780+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:08.781232+0000) 2019-10-13T08:54:08.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:08.780+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:08.781307+0000) 2019-10-13T08:54:09.458 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:09.456+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.457572+0000) 2019-10-13T08:54:09.458 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:09.456+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.457985+0000) 2019-10-13T08:54:09.459 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:09.456+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.458153+0000) 2019-10-13T08:54:09.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:09.458+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.459228+0000) 2019-10-13T08:54:09.460 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:09.457+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.458818+0000) 2019-10-13T08:54:09.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:09.457+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.459157+0000) 2019-10-13T08:54:09.460 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:09.457+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.459377+0000) 2019-10-13T08:54:09.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:09.459+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.460464+0000) 2019-10-13T08:54:09.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:09.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:12.200 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:54:12.201 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:12.205 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:54:12.205 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:12.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:12.888+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:12.889103+0000) 2019-10-13T08:54:12.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:12.888+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:12.888847+0000) 2019-10-13T08:54:13.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:13.236+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:13.237029+0000) 2019-10-13T08:54:14.467 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:14.465+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.467309+0000) 2019-10-13T08:54:14.468 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:14.465+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.466944+0000) 2019-10-13T08:54:14.468 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:14.465+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.467244+0000) 2019-10-13T08:54:14.469 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:14.465+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.467387+0000) 2019-10-13T08:54:14.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:14.466+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.467499+0000) 2019-10-13T08:54:14.469 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:14.466+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.467741+0000) 2019-10-13T08:54:14.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:14.466+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.467861+0000) 2019-10-13T08:54:14.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:14.467+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.468995+0000) 2019-10-13T08:54:14.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:14.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:14.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:14.642+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.642832+0000) 2019-10-13T08:54:18.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:18.442+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:18.442645+0000) 2019-10-13T08:54:18.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:18.780+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:18.781452+0000) 2019-10-13T08:54:18.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:18.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:18.781438+0000) 2019-10-13T08:54:18.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:18.780+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:18.781507+0000) 2019-10-13T08:54:19.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:19.473+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.474778+0000) 2019-10-13T08:54:19.476 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:19.473+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.475102+0000) 2019-10-13T08:54:19.476 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:19.473+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.475207+0000) 2019-10-13T08:54:19.477 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:19.474+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.475886+0000) 2019-10-13T08:54:19.477 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:19.474+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.475568+0000) 2019-10-13T08:54:19.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:19.474+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.475798+0000) 2019-10-13T08:54:19.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:19.474+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.476176+0000) 2019-10-13T08:54:19.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:19.476+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.478576+0000) 2019-10-13T08:54:19.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:19.479+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153582 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:22.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:22.888+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:22.889273+0000) 2019-10-13T08:54:22.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:22.888+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:22.888987+0000) 2019-10-13T08:54:23.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:23.238+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:23.239296+0000) 2019-10-13T08:54:24.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:24.481+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.483627+0000) 2019-10-13T08:54:24.485 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:24.482+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.483252+0000) 2019-10-13T08:54:24.485 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:24.482+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.483381+0000) 2019-10-13T08:54:24.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:24.483+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.483750+0000) 2019-10-13T08:54:24.485 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:24.483+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.483867+0000) 2019-10-13T08:54:24.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:24.483+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.484192+0000) 2019-10-13T08:54:24.486 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:24.484+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.484507+0000) 2019-10-13T08:54:24.486 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:24.484+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.486509+0000) 2019-10-13T08:54:24.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:24.489+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153614 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:24.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:24.642+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.643009+0000) 2019-10-13T08:54:28.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:28.442+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:28.442846+0000) 2019-10-13T08:54:28.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:28.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:28.781606+0000) 2019-10-13T08:54:28.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:28.781+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:28.781643+0000) 2019-10-13T08:54:28.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:28.781+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:28.781673+0000) 2019-10-13T08:54:29.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:29.491+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.492113+0000) 2019-10-13T08:54:29.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:29.491+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.491776+0000) 2019-10-13T08:54:29.493 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:29.491+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.492141+0000) 2019-10-13T08:54:29.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:29.492+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.492677+0000) 2019-10-13T08:54:29.494 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:29.492+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.492834+0000) 2019-10-13T08:54:29.494 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:29.492+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.493057+0000) 2019-10-13T08:54:29.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:29.492+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.493290+0000) 2019-10-13T08:54:29.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:29.493+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.494604+0000) 2019-10-13T08:54:29.505 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:29.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:32.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:32.888+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:32.889444+0000) 2019-10-13T08:54:32.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:32.888+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:32.889185+0000) 2019-10-13T08:54:33.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:33.240+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:33.241581+0000) 2019-10-13T08:54:34.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:34.499+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.500265+0000) 2019-10-13T08:54:34.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:34.500+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.500524+0000) 2019-10-13T08:54:34.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:34.500+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.501648+0000) 2019-10-13T08:54:34.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:34.500+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.500824+0000) 2019-10-13T08:54:34.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:34.500+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.501265+0000) 2019-10-13T08:54:34.503 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:34.501+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.501416+0000) 2019-10-13T08:54:34.503 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:34.501+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.501700+0000) 2019-10-13T08:54:34.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:34.503+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.504038+0000) 2019-10-13T08:54:34.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:34.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153677 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:34.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:34.643+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.643181+0000) 2019-10-13T08:54:38.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:38.442+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:38.443041+0000) 2019-10-13T08:54:38.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:38.780+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:38.781835+0000) 2019-10-13T08:54:38.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:38.780+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:38.781806+0000) 2019-10-13T08:54:38.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:38.780+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:38.781812+0000) 2019-10-13T08:54:39.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:39.506+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.508192+0000) 2019-10-13T08:54:39.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:39.507+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.508462+0000) 2019-10-13T08:54:39.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:39.508+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.508843+0000) 2019-10-13T08:54:39.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:39.507+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.509033+0000) 2019-10-13T08:54:39.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:39.507+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.509227+0000) 2019-10-13T08:54:39.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:39.508+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.509425+0000) 2019-10-13T08:54:39.511 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:39.508+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.509856+0000) 2019-10-13T08:54:39.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:39.510+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.511381+0000) 2019-10-13T08:54:39.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:39.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:42.306 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:54:42.306 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:42.310 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:54:42.310 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:42.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:42.889+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:42.889634+0000) 2019-10-13T08:54:42.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:42.888+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:42.889364+0000) 2019-10-13T08:54:43.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:43.243+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:43.243864+0000) 2019-10-13T08:54:44.517 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:44.515+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.516632+0000) 2019-10-13T08:54:44.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:44.515+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.516962+0000) 2019-10-13T08:54:44.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:44.516+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.517481+0000) 2019-10-13T08:54:44.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:44.515+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.517344+0000) 2019-10-13T08:54:44.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:44.516+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.517595+0000) 2019-10-13T08:54:44.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:44.516+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.518198+0000) 2019-10-13T08:54:44.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:44.517+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.518523+0000) 2019-10-13T08:54:44.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:44.519+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.520499+0000) 2019-10-13T08:54:44.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:44.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:44.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:44.642+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.643324+0000) 2019-10-13T08:54:48.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:48.442+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:48.443238+0000) 2019-10-13T08:54:48.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:48.781+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:48.782048+0000) 2019-10-13T08:54:48.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:48.781+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:48.782049+0000) 2019-10-13T08:54:48.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:48.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:48.782049+0000) 2019-10-13T08:54:49.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:49.523+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.525103+0000) 2019-10-13T08:54:49.526 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:49.523+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.525371+0000) 2019-10-13T08:54:49.526 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:49.524+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.525658+0000) 2019-10-13T08:54:49.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:49.524+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.526240+0000) 2019-10-13T08:54:49.527 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:49.524+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.525990+0000) 2019-10-13T08:54:49.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:49.525+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.526437+0000) 2019-10-13T08:54:49.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:49.526+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.527352+0000) 2019-10-13T08:54:49.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:49.527+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.528513+0000) 2019-10-13T08:54:49.532 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:49.529+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:52.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:52.889+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:52.889827+0000) 2019-10-13T08:54:52.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:52.888+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:52.889561+0000) 2019-10-13T08:54:53.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:53.245+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:53.246332+0000) 2019-10-13T08:54:54.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:54.531+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.533063+0000) 2019-10-13T08:54:54.534 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:54.531+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.533373+0000) 2019-10-13T08:54:54.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:54.532+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.533682+0000) 2019-10-13T08:54:54.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:54.534+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.534757+0000) 2019-10-13T08:54:54.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:54.532+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.533899+0000) 2019-10-13T08:54:54.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:54.532+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.534300+0000) 2019-10-13T08:54:54.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:54.533+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.534709+0000) 2019-10-13T08:54:54.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:54.535+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.536469+0000) 2019-10-13T08:54:54.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:54.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:54:54.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:54.642+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.643455+0000) 2019-10-13T08:54:58.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:58.442+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:58.443407+0000) 2019-10-13T08:54:58.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:58.781+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:58.782248+0000) 2019-10-13T08:54:58.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:58.781+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:58.782249+0000) 2019-10-13T08:54:58.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:58.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:58.782253+0000) 2019-10-13T08:54:59.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:54:59.539+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.541208+0000) 2019-10-13T08:54:59.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:54:59.539+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.541346+0000) 2019-10-13T08:54:59.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:54:59.540+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.541484+0000) 2019-10-13T08:54:59.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:54:59.540+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.541608+0000) 2019-10-13T08:54:59.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:54:59.540+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.541811+0000) 2019-10-13T08:54:59.543 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:54:59.542+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.542826+0000) 2019-10-13T08:54:59.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:54:59.540+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.541948+0000) 2019-10-13T08:54:59.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:54:59.544+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.545116+0000) 2019-10-13T08:54:59.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:54:59.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:02.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:02.889+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:02.890015+0000) 2019-10-13T08:55:02.890 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:02.889+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:02.889736+0000) 2019-10-13T08:55:03.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:03.248+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:03.248843+0000) 2019-10-13T08:55:04.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:04.546+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.548301+0000) 2019-10-13T08:55:04.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:04.547+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.548457+0000) 2019-10-13T08:55:04.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:04.547+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.548616+0000) 2019-10-13T08:55:04.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:04.547+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.548854+0000) 2019-10-13T08:55:04.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:04.547+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.548900+0000) 2019-10-13T08:55:04.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:04.547+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.549161+0000) 2019-10-13T08:55:04.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:04.548+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.550217+0000) 2019-10-13T08:55:04.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:04.550+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.552131+0000) 2019-10-13T08:55:04.555 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:04.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:04.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:04.643+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.643627+0000) 2019-10-13T08:55:08.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:08.443+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:08.443595+0000) 2019-10-13T08:55:08.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:08.782+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:08.782426+0000) 2019-10-13T08:55:08.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:08.782+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:08.782425+0000) 2019-10-13T08:55:08.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:08.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:08.782425+0000) 2019-10-13T08:55:09.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:09.556+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.557137+0000) 2019-10-13T08:55:09.558 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:09.556+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.557350+0000) 2019-10-13T08:55:09.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:09.557+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.557620+0000) 2019-10-13T08:55:09.559 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:09.556+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.557797+0000) 2019-10-13T08:55:09.559 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:09.557+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.557998+0000) 2019-10-13T08:55:09.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:09.557+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.558196+0000) 2019-10-13T08:55:09.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:09.557+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.558373+0000) 2019-10-13T08:55:09.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:09.558+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.560232+0000) 2019-10-13T08:55:09.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:09.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153893 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:12.467 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:55:12.467 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:12.471 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:55:12.471 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:12.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:12.888+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:12.890199+0000) 2019-10-13T08:55:12.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:12.889+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:12.889914+0000) 2019-10-13T08:55:13.251 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:13.250+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:13.251252+0000) 2019-10-13T08:55:14.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:14.563+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.565197+0000) 2019-10-13T08:55:14.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:14.564+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.564763+0000) 2019-10-13T08:55:14.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:14.564+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.564878+0000) 2019-10-13T08:55:14.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:14.564+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.565134+0000) 2019-10-13T08:55:14.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:14.564+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.565175+0000) 2019-10-13T08:55:14.567 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:14.565+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.565567+0000) 2019-10-13T08:55:14.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:14.565+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.565643+0000) 2019-10-13T08:55:14.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:14.566+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.567911+0000) 2019-10-13T08:55:14.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:14.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153925 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:14.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:14.643+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.643767+0000) 2019-10-13T08:55:18.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:18.443+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:18.443785+0000) 2019-10-13T08:55:18.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:18.782+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:18.782588+0000) 2019-10-13T08:55:18.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:18.782+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:18.782589+0000) 2019-10-13T08:55:18.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:18.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:18.782587+0000) 2019-10-13T08:55:19.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:19.570+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.571266+0000) 2019-10-13T08:55:19.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:19.570+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.570848+0000) 2019-10-13T08:55:19.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:19.570+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.571058+0000) 2019-10-13T08:55:19.572 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:19.570+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.571222+0000) 2019-10-13T08:55:19.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:19.570+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.571367+0000) 2019-10-13T08:55:19.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:19.571+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.571600+0000) 2019-10-13T08:55:19.573 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:19.571+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.571817+0000) 2019-10-13T08:55:19.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:19.573+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.574125+0000) 2019-10-13T08:55:19.577 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:19.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:22.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:22.889+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:22.890410+0000) 2019-10-13T08:55:22.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:22.890+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:22.890102+0000) 2019-10-13T08:55:23.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:23.253+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:23.253659+0000) 2019-10-13T08:55:24.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:24.578+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.578762+0000) 2019-10-13T08:55:24.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:24.577+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.578509+0000) 2019-10-13T08:55:24.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:24.577+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.579152+0000) 2019-10-13T08:55:24.580 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:24.577+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.578379+0000) 2019-10-13T08:55:24.580 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:24.577+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.579376+0000) 2019-10-13T08:55:24.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:24.578+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.579586+0000) 2019-10-13T08:55:24.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:24.578+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.579960+0000) 2019-10-13T08:55:24.582 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:24.581+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.582154+0000) 2019-10-13T08:55:24.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:24.583+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 153986 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:24.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:24.642+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.643937+0000) 2019-10-13T08:55:28.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:28.443+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:28.444015+0000) 2019-10-13T08:55:28.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:28.781+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:28.782746+0000) 2019-10-13T08:55:28.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:28.781+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:28.782746+0000) 2019-10-13T08:55:28.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:28.781+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:28.782758+0000) 2019-10-13T08:55:29.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:29.585+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.586551+0000) 2019-10-13T08:55:29.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:29.584+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.586391+0000) 2019-10-13T08:55:29.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:29.585+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.586471+0000) 2019-10-13T08:55:29.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:29.585+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.586604+0000) 2019-10-13T08:55:29.588 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:29.585+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.586944+0000) 2019-10-13T08:55:29.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:29.585+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.587234+0000) 2019-10-13T08:55:29.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:29.585+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.587337+0000) 2019-10-13T08:55:29.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:29.588+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.588891+0000) 2019-10-13T08:55:29.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:29.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:32.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:32.890+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:32.890615+0000) 2019-10-13T08:55:32.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:32.889+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:32.890272+0000) 2019-10-13T08:55:33.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:33.255+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:33.255979+0000) 2019-10-13T08:55:34.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:34.592+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.594009+0000) 2019-10-13T08:55:34.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:34.592+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.594083+0000) 2019-10-13T08:55:34.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:34.592+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.594327+0000) 2019-10-13T08:55:34.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:34.594+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.595237+0000) 2019-10-13T08:55:34.596 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:34.593+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.594841+0000) 2019-10-13T08:55:34.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:34.593+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.595071+0000) 2019-10-13T08:55:34.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:34.593+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.595221+0000) 2019-10-13T08:55:34.597 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:34.596+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.597498+0000) 2019-10-13T08:55:34.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:34.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:34.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:34.643+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.644128+0000) 2019-10-13T08:55:38.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:38.443+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:38.444155+0000) 2019-10-13T08:55:38.784 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:38.782+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:38.782924+0000) 2019-10-13T08:55:38.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:38.782+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:38.782950+0000) 2019-10-13T08:55:38.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:38.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:38.782925+0000) 2019-10-13T08:55:39.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:39.599+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.600504+0000) 2019-10-13T08:55:39.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:39.601+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.601736+0000) 2019-10-13T08:55:39.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:39.599+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.600874+0000) 2019-10-13T08:55:39.603 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:39.600+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.601477+0000) 2019-10-13T08:55:39.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:39.600+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.601695+0000) 2019-10-13T08:55:39.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:39.599+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.601114+0000) 2019-10-13T08:55:39.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:39.600+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.602090+0000) 2019-10-13T08:55:39.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:39.603+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.604509+0000) 2019-10-13T08:55:39.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:39.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:42.738 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:55:42.738 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:42.742 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:55:42.742 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:42.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:42.890+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:42.890799+0000) 2019-10-13T08:55:42.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:42.889+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:42.890449+0000) 2019-10-13T08:55:43.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:43.257+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:43.258277+0000) 2019-10-13T08:55:44.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:44.607+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.609265+0000) 2019-10-13T08:55:44.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:44.608+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.609454+0000) 2019-10-13T08:55:44.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:44.608+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.609754+0000) 2019-10-13T08:55:44.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:44.608+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.610062+0000) 2019-10-13T08:55:44.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:44.608+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.610210+0000) 2019-10-13T08:55:44.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:44.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.610486+0000) 2019-10-13T08:55:44.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:44.611+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.612246+0000) 2019-10-13T08:55:44.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:44.609+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.610747+0000) 2019-10-13T08:55:44.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:44.614+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:44.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:44.642+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.644254+0000) 2019-10-13T08:55:48.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:48.443+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:48.444323+0000) 2019-10-13T08:55:48.784 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:48.782+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:48.783144+0000) 2019-10-13T08:55:48.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:48.782+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:48.783144+0000) 2019-10-13T08:55:48.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:48.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:48.783144+0000) 2019-10-13T08:55:49.617 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:49.615+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.617239+0000) 2019-10-13T08:55:49.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:49.617+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.618015+0000) 2019-10-13T08:55:49.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:49.616+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.617729+0000) 2019-10-13T08:55:49.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:49.616+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.618015+0000) 2019-10-13T08:55:49.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:49.616+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.618172+0000) 2019-10-13T08:55:49.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:49.617+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.618616+0000) 2019-10-13T08:55:49.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:49.617+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.618425+0000) 2019-10-13T08:55:49.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:49.621+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.621852+0000) 2019-10-13T08:55:49.625 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:49.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:52.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:52.890+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:52.890983+0000) 2019-10-13T08:55:52.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:52.890+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:52.890600+0000) 2019-10-13T08:55:53.260 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:53.260+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:53.260680+0000) 2019-10-13T08:55:54.627 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:54.625+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.626347+0000) 2019-10-13T08:55:54.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:54.625+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.626138+0000) 2019-10-13T08:55:54.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:54.626+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.626489+0000) 2019-10-13T08:55:54.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:54.626+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.626619+0000) 2019-10-13T08:55:54.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:54.626+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.626988+0000) 2019-10-13T08:55:54.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:54.626+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.627161+0000) 2019-10-13T08:55:54.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:54.626+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.627292+0000) 2019-10-13T08:55:54.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:54.628+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.629435+0000) 2019-10-13T08:55:54.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:54.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154173 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:55:54.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:54.644+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.644432+0000) 2019-10-13T08:55:58.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:58.444+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:58.444513+0000) 2019-10-13T08:55:58.784 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:58.782+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:58.783323+0000) 2019-10-13T08:55:58.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:58.782+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:58.783321+0000) 2019-10-13T08:55:58.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:58.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:58.783323+0000) 2019-10-13T08:55:59.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:55:59.634+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.634533+0000) 2019-10-13T08:55:59.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:55:59.634+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.634769+0000) 2019-10-13T08:55:59.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:55:59.634+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.634946+0000) 2019-10-13T08:55:59.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:55:59.634+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.636135+0000) 2019-10-13T08:55:59.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:55:59.634+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.635069+0000) 2019-10-13T08:55:59.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:55:59.634+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.635332+0000) 2019-10-13T08:55:59.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:55:59.635+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.635528+0000) 2019-10-13T08:55:59.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:55:59.635+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.636910+0000) 2019-10-13T08:55:59.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:55:59.640+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154202 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:02.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:02.890+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:02.891130+0000) 2019-10-13T08:56:02.891 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:02.890+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:02.890730+0000) 2019-10-13T08:56:03.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:03.261+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:03.262943+0000) 2019-10-13T08:56:04.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:04.642+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.643112+0000) 2019-10-13T08:56:04.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:04.642+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.643336+0000) 2019-10-13T08:56:04.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:04.643+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.643521+0000) 2019-10-13T08:56:04.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:04.643+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.643695+0000) 2019-10-13T08:56:04.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:04.642+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.644658+0000) 2019-10-13T08:56:04.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:04.643+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.644208+0000) 2019-10-13T08:56:04.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:04.644+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.644545+0000) 2019-10-13T08:56:04.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:04.644+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.644460+0000) 2019-10-13T08:56:04.647 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:04.645+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.647020+0000) 2019-10-13T08:56:04.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:04.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:08.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:08.444+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:08.444688+0000) 2019-10-13T08:56:08.784 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:08.783+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:08.783491+0000) 2019-10-13T08:56:08.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:08.783+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:08.783475+0000) 2019-10-13T08:56:08.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:08.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:08.783491+0000) 2019-10-13T08:56:09.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:09.651+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.651442+0000) 2019-10-13T08:56:09.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:09.651+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.651563+0000) 2019-10-13T08:56:09.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:09.650+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.652318+0000) 2019-10-13T08:56:09.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:09.651+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.652241+0000) 2019-10-13T08:56:09.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:09.652+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.652464+0000) 2019-10-13T08:56:09.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:09.652+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.652653+0000) 2019-10-13T08:56:09.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:09.652+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.652833+0000) 2019-10-13T08:56:09.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:09.652+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.654537+0000) 2019-10-13T08:56:09.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:09.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:12.818 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:56:12.818 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:12.824 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:56:12.824 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:12.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:12.890+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:12.891297+0000) 2019-10-13T08:56:12.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:12.889+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:12.890917+0000) 2019-10-13T08:56:13.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:13.264+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:13.264994+0000) 2019-10-13T08:56:14.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:14.643+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.644695+0000) 2019-10-13T08:56:14.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:14.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.659818+0000) 2019-10-13T08:56:14.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:14.658+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.660064+0000) 2019-10-13T08:56:14.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:14.658+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.660227+0000) 2019-10-13T08:56:14.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:14.659+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.660474+0000) 2019-10-13T08:56:14.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:14.659+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.660665+0000) 2019-10-13T08:56:14.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:14.660+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.661429+0000) 2019-10-13T08:56:14.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:14.659+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.661058+0000) 2019-10-13T08:56:14.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:14.663+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.663953+0000) 2019-10-13T08:56:14.667 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:14.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154297 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:18.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:18.444+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:18.444869+0000) 2019-10-13T08:56:18.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:18.782+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:18.783711+0000) 2019-10-13T08:56:18.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:18.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:18.783710+0000) 2019-10-13T08:56:18.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:18.782+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:18.783707+0000) 2019-10-13T08:56:19.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:19.666+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.668233+0000) 2019-10-13T08:56:19.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:19.668+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.668979+0000) 2019-10-13T08:56:19.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:19.667+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.668852+0000) 2019-10-13T08:56:19.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:19.667+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.669159+0000) 2019-10-13T08:56:19.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:19.667+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.669362+0000) 2019-10-13T08:56:19.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:19.667+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.669368+0000) 2019-10-13T08:56:19.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:19.668+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.669661+0000) 2019-10-13T08:56:19.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:19.670+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.671575+0000) 2019-10-13T08:56:19.675 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:19.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:22.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:22.890+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:22.891489+0000) 2019-10-13T08:56:22.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:22.890+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:22.891106+0000) 2019-10-13T08:56:23.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:23.266+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:23.267583+0000) 2019-10-13T08:56:24.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:24.644+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.644922+0000) 2019-10-13T08:56:24.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:24.675+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.676575+0000) 2019-10-13T08:56:24.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:24.676+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.677721+0000) 2019-10-13T08:56:24.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:24.675+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.677017+0000) 2019-10-13T08:56:24.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:24.676+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.677408+0000) 2019-10-13T08:56:24.678 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:24.676+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.677662+0000) 2019-10-13T08:56:24.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:24.676+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.677783+0000) 2019-10-13T08:56:24.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:24.676+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.678293+0000) 2019-10-13T08:56:24.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:24.678+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.679536+0000) 2019-10-13T08:56:24.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:24.681+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154358 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:28.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:28.443+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:28.445076+0000) 2019-10-13T08:56:28.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:28.782+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:28.783907+0000) 2019-10-13T08:56:28.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:28.782+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:28.783902+0000) 2019-10-13T08:56:28.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:28.782+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:28.783915+0000) 2019-10-13T08:56:29.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:29.683+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.684559+0000) 2019-10-13T08:56:29.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:29.684+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.685268+0000) 2019-10-13T08:56:29.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:29.683+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.685112+0000) 2019-10-13T08:56:29.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:29.683+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.685278+0000) 2019-10-13T08:56:29.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:29.684+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.685419+0000) 2019-10-13T08:56:29.687 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:29.684+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.685757+0000) 2019-10-13T08:56:29.687 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:29.686+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.687478+0000) 2019-10-13T08:56:29.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:29.684+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.686164+0000) 2019-10-13T08:56:29.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:29.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154389 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:32.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:32.890+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:32.891258+0000) 2019-10-13T08:56:32.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:32.891+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:32.891638+0000) 2019-10-13T08:56:33.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:33.269+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:33.270060+0000) 2019-10-13T08:56:34.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:34.644+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.645110+0000) 2019-10-13T08:56:34.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:34.691+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.693181+0000) 2019-10-13T08:56:34.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:34.691+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.693352+0000) 2019-10-13T08:56:34.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:34.692+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.693660+0000) 2019-10-13T08:56:34.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:34.692+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.693730+0000) 2019-10-13T08:56:34.696 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:34.693+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.694435+0000) 2019-10-13T08:56:34.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:34.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.694090+0000) 2019-10-13T08:56:34.696 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:34.692+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.694159+0000) 2019-10-13T08:56:34.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:34.696+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.697151+0000) 2019-10-13T08:56:34.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:34.698+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154421 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:38.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:38.444+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:38.445294+0000) 2019-10-13T08:56:38.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:38.783+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:38.784108+0000) 2019-10-13T08:56:38.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:38.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:38.784105+0000) 2019-10-13T08:56:38.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:38.783+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:38.784105+0000) 2019-10-13T08:56:39.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:39.700+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.701091+0000) 2019-10-13T08:56:39.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:39.701+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.701898+0000) 2019-10-13T08:56:39.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:39.701+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.701659+0000) 2019-10-13T08:56:39.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:39.701+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.701805+0000) 2019-10-13T08:56:39.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:39.701+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.701942+0000) 2019-10-13T08:56:39.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:39.701+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.702100+0000) 2019-10-13T08:56:39.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:39.701+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.702302+0000) 2019-10-13T08:56:39.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:39.703+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.704161+0000) 2019-10-13T08:56:39.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:39.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:42.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:42.891+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:42.891820+0000) 2019-10-13T08:56:42.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:42.891+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:42.891431+0000) 2019-10-13T08:56:42.947 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:56:42.947 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:42.952 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:56:42.952 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:43.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:43.272+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:43.272402+0000) 2019-10-13T08:56:44.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:44.645+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.645296+0000) 2019-10-13T08:56:44.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:44.709+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.709592+0000) 2019-10-13T08:56:44.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:44.709+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.710300+0000) 2019-10-13T08:56:44.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:44.709+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.710164+0000) 2019-10-13T08:56:44.712 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:44.710+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.710391+0000) 2019-10-13T08:56:44.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:44.710+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.710537+0000) 2019-10-13T08:56:44.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:44.710+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.710774+0000) 2019-10-13T08:56:44.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:44.710+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.711158+0000) 2019-10-13T08:56:44.713 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:44.711+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.712577+0000) 2019-10-13T08:56:44.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:44.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154482 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:48.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:48.445+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:48.445472+0000) 2019-10-13T08:56:48.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:48.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:48.784302+0000) 2019-10-13T08:56:48.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:48.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:48.784302+0000) 2019-10-13T08:56:48.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:48.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:48.784303+0000) 2019-10-13T08:56:49.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:49.717+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.718232+0000) 2019-10-13T08:56:49.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:49.717+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.718870+0000) 2019-10-13T08:56:49.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:49.718+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.718727+0000) 2019-10-13T08:56:49.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:49.718+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.718910+0000) 2019-10-13T08:56:49.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:49.718+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.719221+0000) 2019-10-13T08:56:49.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:49.718+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.719385+0000) 2019-10-13T08:56:49.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:49.719+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.719695+0000) 2019-10-13T08:56:49.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:49.720+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.722016+0000) 2019-10-13T08:56:49.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:49.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:52.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:52.891+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:52.891999+0000) 2019-10-13T08:56:52.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:52.891+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:52.891615+0000) 2019-10-13T08:56:53.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:53.273+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:53.274875+0000) 2019-10-13T08:56:54.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:54.645+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.645452+0000) 2019-10-13T08:56:54.726 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:54.725+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.725795+0000) 2019-10-13T08:56:54.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:54.725+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.726038+0000) 2019-10-13T08:56:54.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:54.725+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.726882+0000) 2019-10-13T08:56:54.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:54.725+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.726373+0000) 2019-10-13T08:56:54.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:54.726+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.726543+0000) 2019-10-13T08:56:54.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:54.726+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.726737+0000) 2019-10-13T08:56:54.728 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:54.726+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.727085+0000) 2019-10-13T08:56:54.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:54.727+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.728787+0000) 2019-10-13T08:56:54.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:54.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:56:58.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:58.444+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:58.445612+0000) 2019-10-13T08:56:58.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:58.783+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:58.784482+0000) 2019-10-13T08:56:58.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:58.783+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:58.784478+0000) 2019-10-13T08:56:58.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:58.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:58.784486+0000) 2019-10-13T08:56:59.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:56:59.732+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.733525+0000) 2019-10-13T08:56:59.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:56:59.732+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.733857+0000) 2019-10-13T08:56:59.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:56:59.732+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.733986+0000) 2019-10-13T08:56:59.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:56:59.732+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.734410+0000) 2019-10-13T08:56:59.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:56:59.732+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.734304+0000) 2019-10-13T08:56:59.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:56:59.732+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.734342+0000) 2019-10-13T08:56:59.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:56:59.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.734538+0000) 2019-10-13T08:56:59.737 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:56:59.735+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.737463+0000) 2019-10-13T08:56:59.740 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:56:59.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:02.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:02.891+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:02.892182+0000) 2019-10-13T08:57:02.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:02.890+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:02.891855+0000) 2019-10-13T08:57:03.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:03.276+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:03.277296+0000) 2019-10-13T08:57:04.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:04.644+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.645605+0000) 2019-10-13T08:57:04.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:04.739+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.741316+0000) 2019-10-13T08:57:04.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:04.740+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.741459+0000) 2019-10-13T08:57:04.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:04.740+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.741810+0000) 2019-10-13T08:57:04.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:04.741+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.742692+0000) 2019-10-13T08:57:04.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:04.740+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.741908+0000) 2019-10-13T08:57:04.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:04.740+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.742148+0000) 2019-10-13T08:57:04.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:04.740+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.742332+0000) 2019-10-13T08:57:04.750 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:04.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:04.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:04.744+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.745129+0000) 2019-10-13T08:57:08.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:08.444+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:08.445762+0000) 2019-10-13T08:57:08.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:08.783+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:08.784697+0000) 2019-10-13T08:57:08.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:08.783+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:08.784683+0000) 2019-10-13T08:57:08.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:08.783+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:08.784691+0000) 2019-10-13T08:57:09.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:09.748+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.749024+0000) 2019-10-13T08:57:09.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:09.747+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.748622+0000) 2019-10-13T08:57:09.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:09.747+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.748778+0000) 2019-10-13T08:57:09.750 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:09.747+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.749178+0000) 2019-10-13T08:57:09.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:09.748+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.749429+0000) 2019-10-13T08:57:09.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:09.748+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.749423+0000) 2019-10-13T08:57:09.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:09.748+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.749684+0000) 2019-10-13T08:57:09.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:09.751+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.752238+0000) 2019-10-13T08:57:09.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:09.753+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:12.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:12.891+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:12.892386+0000) 2019-10-13T08:57:12.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:12.891+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:12.892048+0000) 2019-10-13T08:57:13.126 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:57:13.126 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:13.131 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:57:13.131 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:13.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:13.279+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:13.279723+0000) 2019-10-13T08:57:14.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:14.645+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.645782+0000) 2019-10-13T08:57:14.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:14.756+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.756795+0000) 2019-10-13T08:57:14.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:14.755+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.756416+0000) 2019-10-13T08:57:14.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:14.755+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.756592+0000) 2019-10-13T08:57:14.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:14.755+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.756700+0000) 2019-10-13T08:57:14.758 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:14.755+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.756983+0000) 2019-10-13T08:57:14.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:14.755+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.757259+0000) 2019-10-13T08:57:14.759 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:14.756+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.757403+0000) 2019-10-13T08:57:14.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:14.759+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.760114+0000) 2019-10-13T08:57:14.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:14.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:18.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:18.445+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:18.445919+0000) 2019-10-13T08:57:18.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:18.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:18.784914+0000) 2019-10-13T08:57:18.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:18.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:18.784917+0000) 2019-10-13T08:57:18.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:18.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:18.784908+0000) 2019-10-13T08:57:19.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:19.762+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.764377+0000) 2019-10-13T08:57:19.765 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:19.763+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.764695+0000) 2019-10-13T08:57:19.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:19.763+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.764804+0000) 2019-10-13T08:57:19.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:19.764+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.765090+0000) 2019-10-13T08:57:19.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:19.763+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.765178+0000) 2019-10-13T08:57:19.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:19.763+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.765317+0000) 2019-10-13T08:57:19.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:19.764+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.765924+0000) 2019-10-13T08:57:19.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:19.766+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.767640+0000) 2019-10-13T08:57:19.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:19.769+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:22.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:22.892+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:22.892565+0000) 2019-10-13T08:57:22.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:22.891+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:22.892243+0000) 2019-10-13T08:57:23.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:23.281+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:23.282034+0000) 2019-10-13T08:57:24.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:24.645+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.645956+0000) 2019-10-13T08:57:24.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:24.772+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.772628+0000) 2019-10-13T08:57:24.773 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:24.772+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.772797+0000) 2019-10-13T08:57:24.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:24.773+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.773724+0000) 2019-10-13T08:57:24.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:24.772+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.773283+0000) 2019-10-13T08:57:24.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:24.773+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.773580+0000) 2019-10-13T08:57:24.775 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:24.773+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.773817+0000) 2019-10-13T08:57:24.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:24.773+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.773747+0000) 2019-10-13T08:57:24.776 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:24.774+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.775650+0000) 2019-10-13T08:57:24.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:24.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154730 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:28.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:28.445+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:28.446092+0000) 2019-10-13T08:57:28.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:28.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:28.785116+0000) 2019-10-13T08:57:28.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:28.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:28.785131+0000) 2019-10-13T08:57:28.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:28.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:28.785133+0000) 2019-10-13T08:57:29.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:29.779+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.780289+0000) 2019-10-13T08:57:29.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:29.779+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.779823+0000) 2019-10-13T08:57:29.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:29.779+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.780081+0000) 2019-10-13T08:57:29.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:29.779+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.780140+0000) 2019-10-13T08:57:29.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:29.779+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.780278+0000) 2019-10-13T08:57:29.783 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:29.780+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.780416+0000) 2019-10-13T08:57:29.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:29.780+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.780555+0000) 2019-10-13T08:57:29.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:29.781+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.782358+0000) 2019-10-13T08:57:29.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:29.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:32.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:32.892+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:32.892763+0000) 2019-10-13T08:57:32.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:32.892+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:32.892410+0000) 2019-10-13T08:57:33.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:33.283+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:33.284500+0000) 2019-10-13T08:57:34.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:34.645+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.646130+0000) 2019-10-13T08:57:34.787 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:34.785+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.786202+0000) 2019-10-13T08:57:34.787 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:34.786+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.786660+0000) 2019-10-13T08:57:34.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:34.786+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.786774+0000) 2019-10-13T08:57:34.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:34.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.786951+0000) 2019-10-13T08:57:34.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:34.787+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.787839+0000) 2019-10-13T08:57:34.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:34.787+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.787459+0000) 2019-10-13T08:57:34.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:34.787+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.787744+0000) 2019-10-13T08:57:34.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:34.789+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.789871+0000) 2019-10-13T08:57:34.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:34.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:38.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:38.446+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:38.446259+0000) 2019-10-13T08:57:38.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:38.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:38.785280+0000) 2019-10-13T08:57:38.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:38.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:38.785291+0000) 2019-10-13T08:57:38.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:38.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:38.785289+0000) 2019-10-13T08:57:39.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:39.794+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.794491+0000) 2019-10-13T08:57:39.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:39.794+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.794795+0000) 2019-10-13T08:57:39.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:39.794+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.794971+0000) 2019-10-13T08:57:39.796 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:39.794+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.795053+0000) 2019-10-13T08:57:39.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:39.794+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.795228+0000) 2019-10-13T08:57:39.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:39.793+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.795210+0000) 2019-10-13T08:57:39.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:39.795+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.795822+0000) 2019-10-13T08:57:39.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:39.796+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.798295+0000) 2019-10-13T08:57:39.800 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:39.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154822 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:42.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:42.891+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:42.892945+0000) 2019-10-13T08:57:42.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:42.891+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:42.892568+0000) 2019-10-13T08:57:43.281 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:57:43.282 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:43.286 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:57:43.286 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:43.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:43.285+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:43.286908+0000) 2019-10-13T08:57:44.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:44.645+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.646306+0000) 2019-10-13T08:57:44.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:44.800+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.801931+0000) 2019-10-13T08:57:44.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:44.800+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.802153+0000) 2019-10-13T08:57:44.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:44.800+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.802372+0000) 2019-10-13T08:57:44.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:44.801+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.802553+0000) 2019-10-13T08:57:44.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:44.801+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.802801+0000) 2019-10-13T08:57:44.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:44.801+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.802941+0000) 2019-10-13T08:57:44.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:44.801+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.803071+0000) 2019-10-13T08:57:44.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:44.802+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.804265+0000) 2019-10-13T08:57:44.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:44.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:48.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:48.445+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:48.446415+0000) 2019-10-13T08:57:48.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:48.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:48.785439+0000) 2019-10-13T08:57:48.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:48.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:48.785433+0000) 2019-10-13T08:57:48.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:48.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:48.785464+0000) 2019-10-13T08:57:49.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:49.808+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.810135+0000) 2019-10-13T08:57:49.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:49.808+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.809713+0000) 2019-10-13T08:57:49.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:49.808+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.809951+0000) 2019-10-13T08:57:49.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:49.808+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.810167+0000) 2019-10-13T08:57:49.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:49.809+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.810409+0000) 2019-10-13T08:57:49.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:49.809+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.810498+0000) 2019-10-13T08:57:49.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:49.809+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.810982+0000) 2019-10-13T08:57:49.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:49.811+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.813358+0000) 2019-10-13T08:57:49.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:49.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:52.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:52.892+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:52.893086+0000) 2019-10-13T08:57:52.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:52.891+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:52.892738+0000) 2019-10-13T08:57:53.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:53.289+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:53.289839+0000) 2019-10-13T08:57:54.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:54.645+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.646513+0000) 2019-10-13T08:57:54.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:54.816+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.818140+0000) 2019-10-13T08:57:54.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:54.817+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.818494+0000) 2019-10-13T08:57:54.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:54.817+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.818641+0000) 2019-10-13T08:57:54.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:54.817+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.818810+0000) 2019-10-13T08:57:54.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:54.818+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.819561+0000) 2019-10-13T08:57:54.821 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:54.818+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.819762+0000) 2019-10-13T08:57:54.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:54.818+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.820130+0000) 2019-10-13T08:57:54.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:54.820+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.820934+0000) 2019-10-13T08:57:54.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:54.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154917 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:57:58.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:58.445+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:58.446553+0000) 2019-10-13T08:57:58.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:58.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:58.785582+0000) 2019-10-13T08:57:58.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:58.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:58.785612+0000) 2019-10-13T08:57:58.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:58.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:58.785591+0000) 2019-10-13T08:57:59.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:57:59.825+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.826518+0000) 2019-10-13T08:57:59.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:57:59.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.826930+0000) 2019-10-13T08:57:59.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:57:59.825+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.827163+0000) 2019-10-13T08:57:59.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:57:59.825+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.827262+0000) 2019-10-13T08:57:59.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:57:59.826+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.827542+0000) 2019-10-13T08:57:59.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:57:59.827+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.828442+0000) 2019-10-13T08:57:59.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:57:59.828+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.829530+0000) 2019-10-13T08:57:59.830 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:57:59.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.828182+0000) 2019-10-13T08:57:59.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:57:59.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:02.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:02.892+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:02.893250+0000) 2019-10-13T08:58:02.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:02.892+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:02.892907+0000) 2019-10-13T08:58:03.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:03.291+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:03.292184+0000) 2019-10-13T08:58:04.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:04.646+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.646665+0000) 2019-10-13T08:58:04.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:04.833+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.835251+0000) 2019-10-13T08:58:04.836 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:04.834+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.835620+0000) 2019-10-13T08:58:04.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:04.834+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.835822+0000) 2019-10-13T08:58:04.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:04.834+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.836018+0000) 2019-10-13T08:58:04.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:04.834+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.836234+0000) 2019-10-13T08:58:04.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:04.836+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.837307+0000) 2019-10-13T08:58:04.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:04.835+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.836567+0000) 2019-10-13T08:58:04.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:04.838+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.838795+0000) 2019-10-13T08:58:04.842 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:04.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 154978 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:08.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:08.446+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:08.446709+0000) 2019-10-13T08:58:08.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:08.785+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:08.785700+0000) 2019-10-13T08:58:08.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:08.785+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:08.785687+0000) 2019-10-13T08:58:08.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:08.785+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:08.785740+0000) 2019-10-13T08:58:09.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:09.843+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.843448+0000) 2019-10-13T08:58:09.844 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:09.843+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.844049+0000) 2019-10-13T08:58:09.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:09.843+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.844716+0000) 2019-10-13T08:58:09.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:09.843+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.844260+0000) 2019-10-13T08:58:09.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:09.844+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.844523+0000) 2019-10-13T08:58:09.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:09.844+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.844917+0000) 2019-10-13T08:58:09.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:09.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.844972+0000) 2019-10-13T08:58:09.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:09.846+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.846982+0000) 2019-10-13T08:58:09.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:09.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:12.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:12.892+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:12.893089+0000) 2019-10-13T08:58:12.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:12.893+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:12.893417+0000) 2019-10-13T08:58:13.294 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:13.294+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:13.294533+0000) 2019-10-13T08:58:13.410 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:58:13.410 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:13.415 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:58:13.415 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:14.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:14.646+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.646808+0000) 2019-10-13T08:58:14.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:14.850+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.850427+0000) 2019-10-13T08:58:14.852 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:14.850+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.850836+0000) 2019-10-13T08:58:14.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:14.850+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.850669+0000) 2019-10-13T08:58:14.852 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:14.850+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.850920+0000) 2019-10-13T08:58:14.852 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:14.850+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.851008+0000) 2019-10-13T08:58:14.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:14.850+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.851229+0000) 2019-10-13T08:58:14.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:14.851+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.851831+0000) 2019-10-13T08:58:14.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:14.853+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.854231+0000) 2019-10-13T08:58:14.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:14.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155042 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:18.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:18.446+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:18.446890+0000) 2019-10-13T08:58:18.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:18.785+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:18.785865+0000) 2019-10-13T08:58:18.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:18.785+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:18.785875+0000) 2019-10-13T08:58:18.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:18.785+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:18.785891+0000) 2019-10-13T08:58:19.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:19.856+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.857280+0000) 2019-10-13T08:58:19.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:19.857+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.857473+0000) 2019-10-13T08:58:19.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:19.857+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.858162+0000) 2019-10-13T08:58:19.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:19.857+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.857822+0000) 2019-10-13T08:58:19.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:19.857+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.858335+0000) 2019-10-13T08:58:19.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:19.858+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.858545+0000) 2019-10-13T08:58:19.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:19.858+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.858730+0000) 2019-10-13T08:58:19.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:19.859+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.860710+0000) 2019-10-13T08:58:19.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:19.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155071 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:22.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:22.893+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:22.893601+0000) 2019-10-13T08:58:22.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:22.893+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:22.893268+0000) 2019-10-13T08:58:23.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:23.296+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:23.296799+0000) 2019-10-13T08:58:24.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:24.646+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.646976+0000) 2019-10-13T08:58:24.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:24.865+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.865675+0000) 2019-10-13T08:58:24.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:24.865+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.865921+0000) 2019-10-13T08:58:24.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:24.865+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.866674+0000) 2019-10-13T08:58:24.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:24.865+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.866284+0000) 2019-10-13T08:58:24.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:24.866+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.866556+0000) 2019-10-13T08:58:24.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:24.866+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.866860+0000) 2019-10-13T08:58:24.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:24.866+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.867203+0000) 2019-10-13T08:58:24.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:24.868+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.869204+0000) 2019-10-13T08:58:24.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:24.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:28.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:28.446+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:28.447074+0000) 2019-10-13T08:58:28.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:28.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:28.786095+0000) 2019-10-13T08:58:28.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:28.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:28.786097+0000) 2019-10-13T08:58:28.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:28.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:28.786098+0000) 2019-10-13T08:58:29.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:29.872+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.873825+0000) 2019-10-13T08:58:29.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:29.873+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.874665+0000) 2019-10-13T08:58:29.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:29.872+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.874195+0000) 2019-10-13T08:58:29.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:29.872+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.874375+0000) 2019-10-13T08:58:29.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:29.873+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.874634+0000) 2019-10-13T08:58:29.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:29.873+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.874736+0000) 2019-10-13T08:58:29.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:29.873+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.874504+0000) 2019-10-13T08:58:29.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:29.875+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.876687+0000) 2019-10-13T08:58:29.879 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:29.877+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:32.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:32.892+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:32.893762+0000) 2019-10-13T08:58:32.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:32.892+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:32.893439+0000) 2019-10-13T08:58:33.299 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:33.298+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:33.299053+0000) 2019-10-13T08:58:34.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:34.646+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.647134+0000) 2019-10-13T08:58:34.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:34.879+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.881225+0000) 2019-10-13T08:58:34.882 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:34.880+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.881456+0000) 2019-10-13T08:58:34.883 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:34.880+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.881677+0000) 2019-10-13T08:58:34.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:34.880+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.881809+0000) 2019-10-13T08:58:34.883 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:34.880+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.881957+0000) 2019-10-13T08:58:34.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:34.881+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.882994+0000) 2019-10-13T08:58:34.884 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:34.882+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.883826+0000) 2019-10-13T08:58:34.884 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:34.880+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.882186+0000) 2019-10-13T08:58:34.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:34.886+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:38.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:38.446+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:38.447212+0000) 2019-10-13T08:58:38.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:38.784+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:38.786318+0000) 2019-10-13T08:58:38.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:38.784+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:38.786320+0000) 2019-10-13T08:58:38.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:38.784+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:38.786323+0000) 2019-10-13T08:58:39.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:39.888+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.890147+0000) 2019-10-13T08:58:39.891 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:39.888+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.890170+0000) 2019-10-13T08:58:39.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:39.888+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.890349+0000) 2019-10-13T08:58:39.891 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:39.889+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.890466+0000) 2019-10-13T08:58:39.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:39.889+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.890829+0000) 2019-10-13T08:58:39.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:39.889+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.890996+0000) 2019-10-13T08:58:39.892 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:39.889+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.891138+0000) 2019-10-13T08:58:39.892 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:39.890+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.892674+0000) 2019-10-13T08:58:39.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:39.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155198 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:42.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:42.893+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:42.893916+0000) 2019-10-13T08:58:42.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:42.892+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:42.893618+0000) 2019-10-13T08:58:43.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:43.300+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:43.301441+0000) 2019-10-13T08:58:43.564 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:58:43.565 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:43.569 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:58:43.569 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:44.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:44.646+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.647328+0000) 2019-10-13T08:58:44.899 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:44.896+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.897972+0000) 2019-10-13T08:58:44.899 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:44.896+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.898292+0000) 2019-10-13T08:58:44.899 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:44.897+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.898446+0000) 2019-10-13T08:58:44.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:44.896+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.898694+0000) 2019-10-13T08:58:44.900 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:44.898+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.900397+0000) 2019-10-13T08:58:44.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:44.897+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.898688+0000) 2019-10-13T08:58:44.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:44.897+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.898750+0000) 2019-10-13T08:58:44.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:44.897+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.899112+0000) 2019-10-13T08:58:44.904 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:44.902+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:48.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:48.446+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:48.447391+0000) 2019-10-13T08:58:48.787 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:48.785+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:48.786543+0000) 2019-10-13T08:58:48.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:48.785+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:48.786526+0000) 2019-10-13T08:58:48.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:48.785+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:48.786547+0000) 2019-10-13T08:58:49.907 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:49.904+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.905574+0000) 2019-10-13T08:58:49.907 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:49.903+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.905232+0000) 2019-10-13T08:58:49.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:49.904+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.905806+0000) 2019-10-13T08:58:49.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:49.904+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.906127+0000) 2019-10-13T08:58:49.908 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:49.905+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.906607+0000) 2019-10-13T08:58:49.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:49.905+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.906629+0000) 2019-10-13T08:58:49.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:49.906+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.907050+0000) 2019-10-13T08:58:49.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:49.908+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.909230+0000) 2019-10-13T08:58:49.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:49.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:52.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:52.893+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:52.894090+0000) 2019-10-13T08:58:52.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:52.893+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:52.893798+0000) 2019-10-13T08:58:53.304 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:53.303+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:53.303837+0000) 2019-10-13T08:58:54.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:54.647+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.647507+0000) 2019-10-13T08:58:54.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:54.911+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.912955+0000) 2019-10-13T08:58:54.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:54.911+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.913290+0000) 2019-10-13T08:58:54.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:54.912+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.913541+0000) 2019-10-13T08:58:54.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:54.914+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.914884+0000) 2019-10-13T08:58:54.915 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:54.912+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.913957+0000) 2019-10-13T08:58:54.915 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:54.913+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.914449+0000) 2019-10-13T08:58:54.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:54.913+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.914769+0000) 2019-10-13T08:58:54.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:54.915+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.916623+0000) 2019-10-13T08:58:54.920 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:54.918+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:58:58.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:58.447+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:58.447631+0000) 2019-10-13T08:58:58.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:58.786+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:58.786713+0000) 2019-10-13T08:58:58.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:58.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:58.786736+0000) 2019-10-13T08:58:58.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:58.786+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:58.786745+0000) 2019-10-13T08:58:59.922 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:58:59.921+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.921475+0000) 2019-10-13T08:58:59.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:58:59.921+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.921554+0000) 2019-10-13T08:58:59.923 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:58:59.921+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.921755+0000) 2019-10-13T08:58:59.923 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:58:59.921+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.922042+0000) 2019-10-13T08:58:59.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:58:59.921+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.922136+0000) 2019-10-13T08:58:59.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:58:59.922+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.922850+0000) 2019-10-13T08:58:59.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:58:59.922+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.922465+0000) 2019-10-13T08:58:59.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:58:59.924+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.925423+0000) 2019-10-13T08:58:59.927 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:58:59.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:02.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:02.893+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:02.894297+0000) 2019-10-13T08:59:02.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:02.893+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:02.893999+0000) 2019-10-13T08:59:03.306 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:03.305+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:03.306204+0000) 2019-10-13T08:59:04.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:04.647+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.647724+0000) 2019-10-13T08:59:04.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:04.927+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.927892+0000) 2019-10-13T08:59:04.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:04.927+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.928073+0000) 2019-10-13T08:59:04.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:04.928+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.928952+0000) 2019-10-13T08:59:04.930 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:04.928+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.928589+0000) 2019-10-13T08:59:04.930 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:04.928+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.928656+0000) 2019-10-13T08:59:04.930 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:04.928+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.928881+0000) 2019-10-13T08:59:04.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:04.928+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.929016+0000) 2019-10-13T08:59:04.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:04.930+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.931483+0000) 2019-10-13T08:59:04.933 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:04.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155358 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:08.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:08.447+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:08.447861+0000) 2019-10-13T08:59:08.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:08.786+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:08.786937+0000) 2019-10-13T08:59:08.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:08.786+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:08.786928+0000) 2019-10-13T08:59:08.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:08.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:08.786929+0000) 2019-10-13T08:59:09.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:09.934+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.934605+0000) 2019-10-13T08:59:09.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:09.934+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.935110+0000) 2019-10-13T08:59:09.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:09.934+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.935349+0000) 2019-10-13T08:59:09.937 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:09.935+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.935728+0000) 2019-10-13T08:59:09.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:09.935+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.935912+0000) 2019-10-13T08:59:09.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:09.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.936089+0000) 2019-10-13T08:59:09.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:09.935+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.936365+0000) 2019-10-13T08:59:09.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:09.937+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.938003+0000) 2019-10-13T08:59:09.941 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:09.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155388 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:12.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:12.894+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:12.894499+0000) 2019-10-13T08:59:12.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:12.894+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:12.894180+0000) 2019-10-13T08:59:13.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:13.308+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:13.308458+0000) 2019-10-13T08:59:13.718 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:59:13.719 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:13.723 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:59:13.723 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:14.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:14.646+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.647868+0000) 2019-10-13T08:59:14.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:14.943+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.944349+0000) 2019-10-13T08:59:14.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:14.945+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.945809+0000) 2019-10-13T08:59:14.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:14.941+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.942865+0000) 2019-10-13T08:59:14.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:14.941+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.943265+0000) 2019-10-13T08:59:14.951 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:14.942+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.943412+0000) 2019-10-13T08:59:14.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:14.942+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.943961+0000) 2019-10-13T08:59:14.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:14.942+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.944164+0000) 2019-10-13T08:59:14.951 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:14.942+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.944384+0000) 2019-10-13T08:59:14.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:14.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155420 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:18.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:18.447+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:18.448049+0000) 2019-10-13T08:59:18.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:18.786+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:18.787104+0000) 2019-10-13T08:59:18.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:18.786+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:18.787105+0000) 2019-10-13T08:59:18.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:18.786+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:18.787103+0000) 2019-10-13T08:59:19.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:19.950+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.951803+0000) 2019-10-13T08:59:19.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:19.950+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.951881+0000) 2019-10-13T08:59:19.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:19.950+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.952183+0000) 2019-10-13T08:59:19.954 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:19.950+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.952224+0000) 2019-10-13T08:59:19.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:19.951+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.952567+0000) 2019-10-13T08:59:19.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:19.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.952752+0000) 2019-10-13T08:59:19.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:19.952+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.952991+0000) 2019-10-13T08:59:19.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:19.953+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.954630+0000) 2019-10-13T08:59:19.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:19.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155450 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:22.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:22.893+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:22.894687+0000) 2019-10-13T08:59:22.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:22.892+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:22.894374+0000) 2019-10-13T08:59:23.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:23.309+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:23.310760+0000) 2019-10-13T08:59:24.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:24.646+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.648031+0000) 2019-10-13T08:59:24.960 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:24.958+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.959745+0000) 2019-10-13T08:59:24.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:24.958+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.960196+0000) 2019-10-13T08:59:24.961 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:24.958+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.960373+0000) 2019-10-13T08:59:24.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:24.959+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.960693+0000) 2019-10-13T08:59:24.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:24.959+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.961683+0000) 2019-10-13T08:59:24.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:24.959+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.960893+0000) 2019-10-13T08:59:24.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:24.959+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.961254+0000) 2019-10-13T08:59:24.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:24.961+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.963383+0000) 2019-10-13T08:59:24.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:24.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:28.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:28.447+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:28.448240+0000) 2019-10-13T08:59:28.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:28.785+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:28.787285+0000) 2019-10-13T08:59:28.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:28.785+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:28.787285+0000) 2019-10-13T08:59:28.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:28.785+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:28.787283+0000) 2019-10-13T08:59:29.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:29.967+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.969194+0000) 2019-10-13T08:59:29.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:29.968+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.969882+0000) 2019-10-13T08:59:29.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:29.968+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.969504+0000) 2019-10-13T08:59:29.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:29.968+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.969595+0000) 2019-10-13T08:59:29.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:29.968+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.969714+0000) 2019-10-13T08:59:29.971 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:29.968+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.969898+0000) 2019-10-13T08:59:29.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:29.969+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.970460+0000) 2019-10-13T08:59:29.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:29.969+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.971626+0000) 2019-10-13T08:59:29.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:29.974+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155514 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:32.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:32.893+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:32.894863+0000) 2019-10-13T08:59:32.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:32.893+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:32.894494+0000) 2019-10-13T08:59:33.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:33.312+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:33.313098+0000) 2019-10-13T08:59:34.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:34.647+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.648205+0000) 2019-10-13T08:59:34.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:34.975+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.976921+0000) 2019-10-13T08:59:34.978 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:34.975+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.977007+0000) 2019-10-13T08:59:34.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:34.975+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.977373+0000) 2019-10-13T08:59:34.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:34.976+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.977808+0000) 2019-10-13T08:59:34.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:34.976+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.977487+0000) 2019-10-13T08:59:34.979 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:34.976+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.977745+0000) 2019-10-13T08:59:34.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:34.976+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.978257+0000) 2019-10-13T08:59:34.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:34.978+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.980604+0000) 2019-10-13T08:59:34.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:34.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:38.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:38.447+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:38.448378+0000) 2019-10-13T08:59:38.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:38.787+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:38.787493+0000) 2019-10-13T08:59:38.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:38.787+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:38.787488+0000) 2019-10-13T08:59:38.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:38.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:38.787488+0000) 2019-10-13T08:59:39.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:39.984+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.984845+0000) 2019-10-13T08:59:39.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:39.983+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.984515+0000) 2019-10-13T08:59:39.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:39.983+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.984791+0000) 2019-10-13T08:59:39.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:39.983+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.984959+0000) 2019-10-13T08:59:39.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:39.986+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.987240+0000) 2019-10-13T08:59:40.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:39.983+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.985268+0000) 2019-10-13T08:59:40.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:39.984+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.985621+0000) 2019-10-13T08:59:40.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:39.984+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.985734+0000) 2019-10-13T08:59:40.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:39.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155576 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:42.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:42.894+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:42.895019+0000) 2019-10-13T08:59:42.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:42.894+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:42.894677+0000) 2019-10-13T08:59:43.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:43.314+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:43.315424+0000) 2019-10-13T08:59:43.873 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T08:59:43.873 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:43.878 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T08:59:43.879 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:44.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:44.647+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.648336+0000) 2019-10-13T08:59:44.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:44.993+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.994006+0000) 2019-10-13T08:59:44.995 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:44.993+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.993629+0000) 2019-10-13T08:59:44.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:44.993+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.993760+0000) 2019-10-13T08:59:44.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:44.993+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.993906+0000) 2019-10-13T08:59:44.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:44.993+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.994120+0000) 2019-10-13T08:59:44.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:44.994+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.994530+0000) 2019-10-13T08:59:44.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:44.994+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.994492+0000) 2019-10-13T08:59:44.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:44.996+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.997478+0000) 2019-10-13T08:59:45.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:44.999+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155610 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:48.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:48.448+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:48.448531+0000) 2019-10-13T08:59:48.788 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:48.787+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:48.787709+0000) 2019-10-13T08:59:48.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:48.787+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:48.787708+0000) 2019-10-13T08:59:48.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:48.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:48.787710+0000) 2019-10-13T08:59:50.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:50.001+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.001526+0000) 2019-10-13T08:59:50.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:50.001+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.002220+0000) 2019-10-13T08:59:50.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:50.001+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.001879+0000) 2019-10-13T08:59:50.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:50.001+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.002181+0000) 2019-10-13T08:59:50.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:50.002+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.002463+0000) 2019-10-13T08:59:50.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:50.002+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.002693+0000) 2019-10-13T08:59:50.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:50.002+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.002864+0000) 2019-10-13T08:59:50.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:50.004+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:50.005658+0000) 2019-10-13T08:59:50.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:50.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:52.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:52.894+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:52.895184+0000) 2019-10-13T08:59:52.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:52.894+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:52.894845+0000) 2019-10-13T08:59:53.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:53.317+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:53.317764+0000) 2019-10-13T08:59:54.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:54.648+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.648532+0000) 2019-10-13T08:59:55.011 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:55.010+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.010848+0000) 2019-10-13T08:59:55.012 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:55.010+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.011233+0000) 2019-10-13T08:59:55.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T08:59:55.010+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.011125+0000) 2019-10-13T08:59:55.012 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:55.011+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.011509+0000) 2019-10-13T08:59:55.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T08:59:55.011+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.011828+0000) 2019-10-13T08:59:55.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T08:59:55.011+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.012225+0000) 2019-10-13T08:59:55.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T08:59:55.012+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.013381+0000) 2019-10-13T08:59:55.014 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:55.011+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:55.012096+0000) 2019-10-13T08:59:55.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T08:59:55.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T08:59:58.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T08:59:58.448+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:58.448681+0000) 2019-10-13T08:59:58.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T08:59:58.787+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:58.787896+0000) 2019-10-13T08:59:58.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T08:59:58.787+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:58.787898+0000) 2019-10-13T08:59:58.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T08:59:58.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:58.787894+0000) 2019-10-13T09:00:00.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:00.017+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.018814+0000) 2019-10-13T09:00:00.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:00.017+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.019261+0000) 2019-10-13T09:00:00.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:00.018+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.019505+0000) 2019-10-13T09:00:00.021 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:00.018+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.019836+0000) 2019-10-13T09:00:00.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:00.018+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.019964+0000) 2019-10-13T09:00:00.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:00.019+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.020544+0000) 2019-10-13T09:00:00.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:00.020+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.021037+0000) 2019-10-13T09:00:00.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:00.021+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:00.022168+0000) 2019-10-13T09:00:00.026 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:00.024+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155702 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:02.895 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:02.895+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:02.895367+0000) 2019-10-13T09:00:02.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:02.894+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:02.895053+0000) 2019-10-13T09:00:03.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:03.319+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:03.320150+0000) 2019-10-13T09:00:04.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:04.647+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.648685+0000) 2019-10-13T09:00:05.028 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:05.025+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.026895+0000) 2019-10-13T09:00:05.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:05.025+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.027128+0000) 2019-10-13T09:00:05.029 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:05.026+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.027466+0000) 2019-10-13T09:00:05.029 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:05.026+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.027583+0000) 2019-10-13T09:00:05.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:05.026+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.027807+0000) 2019-10-13T09:00:05.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:05.026+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.027838+0000) 2019-10-13T09:00:05.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:05.027+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.028607+0000) 2019-10-13T09:00:05.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:05.030+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:05.030839+0000) 2019-10-13T09:00:05.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:05.030+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155736 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:08.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:08.448+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:08.448840+0000) 2019-10-13T09:00:08.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:08.787+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:08.788066+0000) 2019-10-13T09:00:08.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:08.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:08.788060+0000) 2019-10-13T09:00:08.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:08.787+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:08.788061+0000) 2019-10-13T09:00:10.034 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:10.032+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.033892+0000) 2019-10-13T09:00:10.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:10.032+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.034074+0000) 2019-10-13T09:00:10.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:10.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.034800+0000) 2019-10-13T09:00:10.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:10.033+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.034555+0000) 2019-10-13T09:00:10.036 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:10.033+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.034618+0000) 2019-10-13T09:00:10.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:10.033+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.035054+0000) 2019-10-13T09:00:10.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:10.033+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.035230+0000) 2019-10-13T09:00:10.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:10.037+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:10.038038+0000) 2019-10-13T09:00:10.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:10.039+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:12.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:12.895+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:12.895581+0000) 2019-10-13T09:00:12.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:12.894+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:12.895231+0000) 2019-10-13T09:00:13.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:13.322+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:13.322516+0000) 2019-10-13T09:00:14.004 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:00:14.004 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:14.009 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:00:14.009 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:14.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:14.648+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.648824+0000) 2019-10-13T09:00:15.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:15.041+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.042885+0000) 2019-10-13T09:00:15.044 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:15.041+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.042497+0000) 2019-10-13T09:00:15.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:15.041+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.042741+0000) 2019-10-13T09:00:15.044 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:15.041+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.042959+0000) 2019-10-13T09:00:15.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:15.041+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.043138+0000) 2019-10-13T09:00:15.045 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:15.041+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.043353+0000) 2019-10-13T09:00:15.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:15.042+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.043529+0000) 2019-10-13T09:00:15.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:15.043+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:15.045446+0000) 2019-10-13T09:00:15.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:15.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:18.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:18.448+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:18.449004+0000) 2019-10-13T09:00:18.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:18.787+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:18.788272+0000) 2019-10-13T09:00:18.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:18.787+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:18.788270+0000) 2019-10-13T09:00:18.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:18.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:18.788253+0000) 2019-10-13T09:00:20.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:20.048+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.050214+0000) 2019-10-13T09:00:20.051 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:20.048+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.049974+0000) 2019-10-13T09:00:20.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:20.049+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.050832+0000) 2019-10-13T09:00:20.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:20.049+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.051017+0000) 2019-10-13T09:00:20.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:20.049+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.051139+0000) 2019-10-13T09:00:20.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:20.049+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.051352+0000) 2019-10-13T09:00:20.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:20.048+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.050149+0000) 2019-10-13T09:00:20.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:20.051+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:20.053433+0000) 2019-10-13T09:00:20.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:20.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:22.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:22.894+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:22.895752+0000) 2019-10-13T09:00:22.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:22.894+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:22.895412+0000) 2019-10-13T09:00:23.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:23.323+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:23.324765+0000) 2019-10-13T09:00:24.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:24.648+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.649004+0000) 2019-10-13T09:00:25.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:25.057+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.058811+0000) 2019-10-13T09:00:25.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:25.057+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.059226+0000) 2019-10-13T09:00:25.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:25.057+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.059661+0000) 2019-10-13T09:00:25.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:25.058+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.059763+0000) 2019-10-13T09:00:25.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:25.058+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.059915+0000) 2019-10-13T09:00:25.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:25.058+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.060201+0000) 2019-10-13T09:00:25.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:25.059+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.060529+0000) 2019-10-13T09:00:25.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:25.060+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:25.062651+0000) 2019-10-13T09:00:25.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:25.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:28.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:28.448+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:28.449173+0000) 2019-10-13T09:00:28.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:28.788+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:28.788508+0000) 2019-10-13T09:00:28.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:28.788+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:28.788507+0000) 2019-10-13T09:00:28.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:28.788+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:28.788507+0000) 2019-10-13T09:00:30.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:30.067+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.068046+0000) 2019-10-13T09:00:30.069 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:30.066+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.067169+0000) 2019-10-13T09:00:30.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:30.067+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.067830+0000) 2019-10-13T09:00:30.069 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:30.067+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.067944+0000) 2019-10-13T09:00:30.069 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:30.067+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.068189+0000) 2019-10-13T09:00:30.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:30.069+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.070099+0000) 2019-10-13T09:00:30.070 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:30.068+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.068620+0000) 2019-10-13T09:00:30.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:30.068+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:30.068778+0000) 2019-10-13T09:00:30.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:30.073+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:32.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:32.895+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:32.895941+0000) 2019-10-13T09:00:32.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:32.895+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:32.895564+0000) 2019-10-13T09:00:33.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:33.326+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:33.327299+0000) 2019-10-13T09:00:34.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:34.648+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.649181+0000) 2019-10-13T09:00:35.077 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:35.075+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.076342+0000) 2019-10-13T09:00:35.077 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:35.076+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.076504+0000) 2019-10-13T09:00:35.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:35.076+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.077550+0000) 2019-10-13T09:00:35.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:35.076+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.076786+0000) 2019-10-13T09:00:35.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:35.076+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.076617+0000) 2019-10-13T09:00:35.079 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:35.076+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.077304+0000) 2019-10-13T09:00:35.079 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:35.076+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.077189+0000) 2019-10-13T09:00:35.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:35.079+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:35.079803+0000) 2019-10-13T09:00:35.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:35.081+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:38.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:38.449+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:38.449390+0000) 2019-10-13T09:00:38.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:38.788+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:38.788681+0000) 2019-10-13T09:00:38.790 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:38.788+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:38.788683+0000) 2019-10-13T09:00:38.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:38.788+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:38.788677+0000) 2019-10-13T09:00:40.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:40.083+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.084204+0000) 2019-10-13T09:00:40.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:40.084+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.084419+0000) 2019-10-13T09:00:40.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:40.084+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.085440+0000) 2019-10-13T09:00:40.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:40.084+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.084697+0000) 2019-10-13T09:00:40.086 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:40.084+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.085053+0000) 2019-10-13T09:00:40.087 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:40.084+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.085158+0000) 2019-10-13T09:00:40.087 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:40.085+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.085455+0000) 2019-10-13T09:00:40.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:40.088+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:40.089122+0000) 2019-10-13T09:00:40.091 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:40.090+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155954 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:42.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:42.895+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:42.896129+0000) 2019-10-13T09:00:42.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:42.895+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:42.895746+0000) 2019-10-13T09:00:43.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:43.329+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:43.329598+0000) 2019-10-13T09:00:44.166 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:00:44.167 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:44.171 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:00:44.171 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:44.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:44.649+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.649348+0000) 2019-10-13T09:00:45.092 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:45.091+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.091680+0000) 2019-10-13T09:00:45.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:45.091+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.091947+0000) 2019-10-13T09:00:45.093 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:45.091+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.092081+0000) 2019-10-13T09:00:45.093 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:45.091+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.092341+0000) 2019-10-13T09:00:45.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:45.091+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.092523+0000) 2019-10-13T09:00:45.094 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:45.092+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.092443+0000) 2019-10-13T09:00:45.094 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:45.092+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.092697+0000) 2019-10-13T09:00:45.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:45.094+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:45.094883+0000) 2019-10-13T09:00:45.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:45.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 155988 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:48.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:48.448+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:48.449643+0000) 2019-10-13T09:00:48.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:48.787+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:48.788880+0000) 2019-10-13T09:00:48.790 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:48.787+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:48.788879+0000) 2019-10-13T09:00:48.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:48.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:48.788900+0000) 2019-10-13T09:00:50.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:50.098+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.099683+0000) 2019-10-13T09:00:50.101 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:50.097+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.099329+0000) 2019-10-13T09:00:50.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:50.098+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.099530+0000) 2019-10-13T09:00:50.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:50.098+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.099852+0000) 2019-10-13T09:00:50.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:50.098+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.100060+0000) 2019-10-13T09:00:50.103 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:50.098+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.100228+0000) 2019-10-13T09:00:50.103 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:50.099+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.100643+0000) 2019-10-13T09:00:50.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:50.101+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:50.102711+0000) 2019-10-13T09:00:50.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:50.105+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:52.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:52.895+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:52.896325+0000) 2019-10-13T09:00:52.897 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:52.894+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:52.895925+0000) 2019-10-13T09:00:53.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:53.331+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:53.331904+0000) 2019-10-13T09:00:54.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:54.648+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.649530+0000) 2019-10-13T09:00:55.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:00:55.107+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.108705+0000) 2019-10-13T09:00:55.109 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:55.106+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.108383+0000) 2019-10-13T09:00:55.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:00:55.107+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.108806+0000) 2019-10-13T09:00:55.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:00:55.107+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.108879+0000) 2019-10-13T09:00:55.110 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:55.107+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.109253+0000) 2019-10-13T09:00:55.110 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:55.108+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.109446+0000) 2019-10-13T09:00:55.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:55.108+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.109639+0000) 2019-10-13T09:00:55.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:00:55.111+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:55.111790+0000) 2019-10-13T09:00:55.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:00:55.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:00:58.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:00:58.449+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:58.449810+0000) 2019-10-13T09:00:58.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:00:58.787+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:58.789101+0000) 2019-10-13T09:00:58.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:00:58.787+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:58.789099+0000) 2019-10-13T09:00:58.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:00:58.787+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:58.789106+0000) 2019-10-13T09:01:00.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:00.117+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.117893+0000) 2019-10-13T09:01:00.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:00.116+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.117553+0000) 2019-10-13T09:01:00.119 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:00.116+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.118031+0000) 2019-10-13T09:01:00.119 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:00.116+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.118259+0000) 2019-10-13T09:01:00.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:00.117+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.118511+0000) 2019-10-13T09:01:00.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:00.117+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.118908+0000) 2019-10-13T09:01:00.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:00.117+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.119275+0000) 2019-10-13T09:01:00.122 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:00.120+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:00.120861+0000) 2019-10-13T09:01:00.125 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:00.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:02.897 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:02.895+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:02.896128+0000) 2019-10-13T09:01:02.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:02.896+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:02.896486+0000) 2019-10-13T09:01:03.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:03.334+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:03.334177+0000) 2019-10-13T09:01:04.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:04.649+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.649638+0000) 2019-10-13T09:01:05.127 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:05.124+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.126250+0000) 2019-10-13T09:01:05.127 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:05.125+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.126426+0000) 2019-10-13T09:01:05.127 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:05.125+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.126754+0000) 2019-10-13T09:01:05.128 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:05.125+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.126920+0000) 2019-10-13T09:01:05.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:05.125+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.127214+0000) 2019-10-13T09:01:05.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:05.126+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.127467+0000) 2019-10-13T09:01:05.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:05.127+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.128303+0000) 2019-10-13T09:01:05.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:05.128+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:05.128901+0000) 2019-10-13T09:01:05.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:05.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156114 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:08.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:08.449+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:08.449945+0000) 2019-10-13T09:01:08.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:08.788+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:08.789339+0000) 2019-10-13T09:01:08.790 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:08.788+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:08.789339+0000) 2019-10-13T09:01:08.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:08.788+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:08.789339+0000) 2019-10-13T09:01:10.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:10.133+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.134810+0000) 2019-10-13T09:01:10.136 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:10.133+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.134833+0000) 2019-10-13T09:01:10.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:10.133+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.135034+0000) 2019-10-13T09:01:10.137 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:10.133+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.135344+0000) 2019-10-13T09:01:10.137 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:10.133+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.135231+0000) 2019-10-13T09:01:10.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:10.134+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.135506+0000) 2019-10-13T09:01:10.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:10.134+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.136276+0000) 2019-10-13T09:01:10.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:10.135+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:10.137178+0000) 2019-10-13T09:01:10.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:10.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156144 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:12.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:12.895+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:12.896661+0000) 2019-10-13T09:01:12.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:12.895+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:12.896296+0000) 2019-10-13T09:01:13.337 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:13.336+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:13.337339+0000) 2019-10-13T09:01:14.337 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:01:14.337 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:14.351 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:01:14.351 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:14.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:14.649+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.649773+0000) 2019-10-13T09:01:15.143 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:15.142+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.142663+0000) 2019-10-13T09:01:15.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:15.142+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.143068+0000) 2019-10-13T09:01:15.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:15.141+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.143419+0000) 2019-10-13T09:01:15.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:15.143+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.143584+0000) 2019-10-13T09:01:15.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:15.143+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.143853+0000) 2019-10-13T09:01:15.145 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:15.143+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.143887+0000) 2019-10-13T09:01:15.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:15.143+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.144009+0000) 2019-10-13T09:01:15.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:15.144+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:15.146089+0000) 2019-10-13T09:01:15.150 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:15.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156176 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:18.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:18.449+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:18.450120+0000) 2019-10-13T09:01:18.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:18.789+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:18.789568+0000) 2019-10-13T09:01:18.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:18.789+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:18.789550+0000) 2019-10-13T09:01:18.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:18.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:18.789565+0000) 2019-10-13T09:01:20.153 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:20.151+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.151772+0000) 2019-10-13T09:01:20.153 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:20.151+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.151920+0000) 2019-10-13T09:01:20.153 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:20.151+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.152001+0000) 2019-10-13T09:01:20.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:20.151+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.152268+0000) 2019-10-13T09:01:20.154 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:20.152+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.152577+0000) 2019-10-13T09:01:20.154 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:20.151+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.152007+0000) 2019-10-13T09:01:20.154 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:20.152+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.152777+0000) 2019-10-13T09:01:20.155 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:20.154+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:20.155070+0000) 2019-10-13T09:01:20.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:20.157+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:22.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:22.896+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:22.896839+0000) 2019-10-13T09:01:22.897 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:22.896+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:22.896488+0000) 2019-10-13T09:01:23.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:23.339+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:23.339756+0000) 2019-10-13T09:01:24.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:24.649+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.649942+0000) 2019-10-13T09:01:25.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:25.159+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.159953+0000) 2019-10-13T09:01:25.161 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:25.159+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.160125+0000) 2019-10-13T09:01:25.162 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:25.160+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.160956+0000) 2019-10-13T09:01:25.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:25.160+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.160608+0000) 2019-10-13T09:01:25.162 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:25.160+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.160958+0000) 2019-10-13T09:01:25.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:25.160+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.161010+0000) 2019-10-13T09:01:25.163 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:25.160+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.161356+0000) 2019-10-13T09:01:25.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:25.163+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:25.164308+0000) 2019-10-13T09:01:25.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:25.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156240 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:28.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:28.450+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:28.450312+0000) 2019-10-13T09:01:28.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:28.789+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:28.789752+0000) 2019-10-13T09:01:28.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:28.789+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:28.789757+0000) 2019-10-13T09:01:28.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:28.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:28.789749+0000) 2019-10-13T09:01:30.168 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:30.167+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.168369+0000) 2019-10-13T09:01:30.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:30.167+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.167931+0000) 2019-10-13T09:01:30.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:30.167+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.168165+0000) 2019-10-13T09:01:30.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:30.167+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.168291+0000) 2019-10-13T09:01:30.170 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:30.168+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.168693+0000) 2019-10-13T09:01:30.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:30.168+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.169022+0000) 2019-10-13T09:01:30.170 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:30.168+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.169241+0000) 2019-10-13T09:01:30.172 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:30.171+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:30.172218+0000) 2019-10-13T09:01:30.175 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:30.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:32.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:32.896+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:32.897021+0000) 2019-10-13T09:01:32.897 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:32.895+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:32.896669+0000) 2019-10-13T09:01:33.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:33.341+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:33.342050+0000) 2019-10-13T09:01:34.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:34.648+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.650166+0000) 2019-10-13T09:01:35.177 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:35.174+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.176159+0000) 2019-10-13T09:01:35.177 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:35.176+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.176909+0000) 2019-10-13T09:01:35.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:35.175+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.176744+0000) 2019-10-13T09:01:35.178 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:35.175+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.176928+0000) 2019-10-13T09:01:35.178 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:35.176+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.177411+0000) 2019-10-13T09:01:35.178 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:35.176+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.177583+0000) 2019-10-13T09:01:35.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:35.176+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.177915+0000) 2019-10-13T09:01:35.179 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:35.179+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:35.179814+0000) 2019-10-13T09:01:35.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:35.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:38.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:38.449+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:38.450542+0000) 2019-10-13T09:01:38.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:38.789+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:38.789920+0000) 2019-10-13T09:01:38.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:38.789+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:38.789918+0000) 2019-10-13T09:01:38.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:38.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:38.789912+0000) 2019-10-13T09:01:40.185 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:40.182+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.183965+0000) 2019-10-13T09:01:40.185 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:40.182+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.184296+0000) 2019-10-13T09:01:40.186 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:40.184+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.184795+0000) 2019-10-13T09:01:40.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:40.183+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.184544+0000) 2019-10-13T09:01:40.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:40.183+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.184749+0000) 2019-10-13T09:01:40.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:40.183+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.185064+0000) 2019-10-13T09:01:40.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:40.183+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.185326+0000) 2019-10-13T09:01:40.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:40.186+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:40.187464+0000) 2019-10-13T09:01:40.191 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:40.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:42.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:42.896+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:42.897207+0000) 2019-10-13T09:01:42.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:42.896+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:42.896834+0000) 2019-10-13T09:01:43.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:43.344+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:43.344446+0000) 2019-10-13T09:01:44.525 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:01:44.525 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:44.555 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:01:44.555 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:44.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:44.648+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.650383+0000) 2019-10-13T09:01:45.193 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:45.191+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.192638+0000) 2019-10-13T09:01:45.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:45.193+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.193795+0000) 2019-10-13T09:01:45.194 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:45.191+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.193024+0000) 2019-10-13T09:01:45.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:45.191+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.193366+0000) 2019-10-13T09:01:45.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:45.192+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.193588+0000) 2019-10-13T09:01:45.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:45.192+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.193817+0000) 2019-10-13T09:01:45.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:45.192+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.193957+0000) 2019-10-13T09:01:45.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:45.195+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:45.196271+0000) 2019-10-13T09:01:45.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:45.197+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:48.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:48.449+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:48.450729+0000) 2019-10-13T09:01:48.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:48.789+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:48.790123+0000) 2019-10-13T09:01:48.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:48.789+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:48.790129+0000) 2019-10-13T09:01:48.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:48.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:48.790125+0000) 2019-10-13T09:01:50.201 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:50.200+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.201158+0000) 2019-10-13T09:01:50.202 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:50.199+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.200729+0000) 2019-10-13T09:01:50.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:50.199+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.201254+0000) 2019-10-13T09:01:50.202 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:50.199+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.201294+0000) 2019-10-13T09:01:50.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:50.200+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.201559+0000) 2019-10-13T09:01:50.203 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:50.200+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.201788+0000) 2019-10-13T09:01:50.203 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:50.200+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.201990+0000) 2019-10-13T09:01:50.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:50.202+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:50.203472+0000) 2019-10-13T09:01:50.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:50.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:52.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:52.897+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:52.897385+0000) 2019-10-13T09:01:52.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:52.896+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:52.896969+0000) 2019-10-13T09:01:53.347 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:53.346+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:53.346879+0000) 2019-10-13T09:01:54.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:54.649+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.650590+0000) 2019-10-13T09:01:55.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:55.207+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.208904+0000) 2019-10-13T09:01:55.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:01:55.207+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.209117+0000) 2019-10-13T09:01:55.211 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:01:55.209+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.210373+0000) 2019-10-13T09:01:55.211 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:55.208+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.209612+0000) 2019-10-13T09:01:55.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:55.208+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.209943+0000) 2019-10-13T09:01:55.211 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:55.208+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.210084+0000) 2019-10-13T09:01:55.212 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:01:55.209+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.210702+0000) 2019-10-13T09:01:55.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:01:55.212+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:55.213073+0000) 2019-10-13T09:01:55.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:01:55.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:01:58.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:01:58.450+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:58.450902+0000) 2019-10-13T09:01:58.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:01:58.789+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:58.790337+0000) 2019-10-13T09:01:58.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:01:58.789+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:58.790339+0000) 2019-10-13T09:01:58.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:01:58.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:58.790338+0000) 2019-10-13T09:02:00.218 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:00.217+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.217699+0000) 2019-10-13T09:02:00.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:00.217+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.217959+0000) 2019-10-13T09:02:00.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:00.217+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.218098+0000) 2019-10-13T09:02:00.219 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:00.217+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.218251+0000) 2019-10-13T09:02:00.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:00.217+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.219009+0000) 2019-10-13T09:02:00.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:00.218+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.218778+0000) 2019-10-13T09:02:00.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:00.218+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.219078+0000) 2019-10-13T09:02:00.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:00.219+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:00.221720+0000) 2019-10-13T09:02:00.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:00.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:02.897 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:02.896+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:02.897556+0000) 2019-10-13T09:02:02.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:02.896+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:02.897165+0000) 2019-10-13T09:02:03.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:03.348+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:03.349383+0000) 2019-10-13T09:02:04.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:04.650+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.650709+0000) 2019-10-13T09:02:05.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:05.225+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.226269+0000) 2019-10-13T09:02:05.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:05.226+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.226495+0000) 2019-10-13T09:02:05.228 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:05.226+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.227875+0000) 2019-10-13T09:02:05.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:05.226+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.227018+0000) 2019-10-13T09:02:05.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:05.226+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.227131+0000) 2019-10-13T09:02:05.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:05.227+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.227407+0000) 2019-10-13T09:02:05.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:05.227+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.227561+0000) 2019-10-13T09:02:05.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:05.228+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:05.230590+0000) 2019-10-13T09:02:05.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:05.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:08.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:08.450+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:08.451063+0000) 2019-10-13T09:02:08.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:08.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:08.790556+0000) 2019-10-13T09:02:08.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:08.790+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:08.790572+0000) 2019-10-13T09:02:08.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:08.790+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:08.790575+0000) 2019-10-13T09:02:10.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:10.234+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.235033+0000) 2019-10-13T09:02:10.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:10.235+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.235552+0000) 2019-10-13T09:02:10.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:10.235+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.235756+0000) 2019-10-13T09:02:10.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:10.235+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.235941+0000) 2019-10-13T09:02:10.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:10.235+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.236252+0000) 2019-10-13T09:02:10.239 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:10.236+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.236967+0000) 2019-10-13T09:02:10.239 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:10.235+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.237442+0000) 2019-10-13T09:02:10.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:10.236+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:10.238098+0000) 2019-10-13T09:02:10.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:10.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:12.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:12.897+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:12.897724+0000) 2019-10-13T09:02:12.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:12.897+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:12.897356+0000) 2019-10-13T09:02:13.352 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:13.350+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:13.351700+0000) 2019-10-13T09:02:14.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:14.650+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.650880+0000) 2019-10-13T09:02:14.716 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:02:14.716 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:14.764 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:02:14.765 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:15.260 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:15.243+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.244681+0000) 2019-10-13T09:02:15.260 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:15.243+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.244185+0000) 2019-10-13T09:02:15.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:15.243+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.244340+0000) 2019-10-13T09:02:15.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:15.247+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.248595+0000) 2019-10-13T09:02:15.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:15.244+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.244478+0000) 2019-10-13T09:02:15.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:15.244+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.244643+0000) 2019-10-13T09:02:15.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:15.244+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.244976+0000) 2019-10-13T09:02:15.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:15.244+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:15.245365+0000) 2019-10-13T09:02:15.877 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:15.248+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:18.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:18.449+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:18.451179+0000) 2019-10-13T09:02:18.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:18.789+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:18.790776+0000) 2019-10-13T09:02:18.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:18.789+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:18.790777+0000) 2019-10-13T09:02:18.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:18.789+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:18.790758+0000) 2019-10-13T09:02:20.251 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:20.249+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.250778+0000) 2019-10-13T09:02:20.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:20.249+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.251219+0000) 2019-10-13T09:02:20.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:20.249+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.251346+0000) 2019-10-13T09:02:20.253 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:20.251+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.252197+0000) 2019-10-13T09:02:20.253 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:20.250+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.251883+0000) 2019-10-13T09:02:20.253 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:20.250+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.252059+0000) 2019-10-13T09:02:20.253 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:20.251+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.252406+0000) 2019-10-13T09:02:20.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:20.253+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:20.254549+0000) 2019-10-13T09:02:20.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:20.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:22.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:22.897+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:22.897912+0000) 2019-10-13T09:02:22.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:22.896+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:22.897554+0000) 2019-10-13T09:02:23.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:23.353+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:23.354053+0000) 2019-10-13T09:02:24.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:24.649+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.651042+0000) 2019-10-13T09:02:25.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:25.257+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.258557+0000) 2019-10-13T09:02:25.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:25.257+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.258953+0000) 2019-10-13T09:02:25.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:25.257+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.259222+0000) 2019-10-13T09:02:25.260 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:25.258+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.259404+0000) 2019-10-13T09:02:25.261 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:25.259+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.260202+0000) 2019-10-13T09:02:25.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:25.258+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.259917+0000) 2019-10-13T09:02:25.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:25.258+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.260100+0000) 2019-10-13T09:02:25.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:25.261+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:25.262104+0000) 2019-10-13T09:02:25.266 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:25.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156618 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:28.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:28.450+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:28.451355+0000) 2019-10-13T09:02:28.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:28.790+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:28.790969+0000) 2019-10-13T09:02:28.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:28.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:28.790962+0000) 2019-10-13T09:02:28.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:28.790+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:28.790970+0000) 2019-10-13T09:02:30.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:30.265+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.267326+0000) 2019-10-13T09:02:30.268 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:30.266+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.267510+0000) 2019-10-13T09:02:30.269 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:30.266+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.267747+0000) 2019-10-13T09:02:30.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:30.268+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.268809+0000) 2019-10-13T09:02:30.270 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:30.266+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.268068+0000) 2019-10-13T09:02:30.270 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:30.266+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.268331+0000) 2019-10-13T09:02:30.270 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:30.267+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.268533+0000) 2019-10-13T09:02:30.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:30.270+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:30.270782+0000) 2019-10-13T09:02:30.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:30.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156648 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:32.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:32.897+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:32.898075+0000) 2019-10-13T09:02:32.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:32.897+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:32.897709+0000) 2019-10-13T09:02:33.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:33.356+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:33.356369+0000) 2019-10-13T09:02:34.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:34.649+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.651205+0000) 2019-10-13T09:02:35.276 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:35.274+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.275781+0000) 2019-10-13T09:02:35.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:35.276+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.276835+0000) 2019-10-13T09:02:35.278 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:35.275+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.276475+0000) 2019-10-13T09:02:35.278 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:35.275+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.276678+0000) 2019-10-13T09:02:35.278 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:35.275+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.276823+0000) 2019-10-13T09:02:35.279 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:35.275+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.277065+0000) 2019-10-13T09:02:35.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:35.275+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.277142+0000) 2019-10-13T09:02:35.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:35.279+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:35.280059+0000) 2019-10-13T09:02:35.283 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:35.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:38.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:38.450+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:38.451538+0000) 2019-10-13T09:02:38.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:38.790+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:38.791154+0000) 2019-10-13T09:02:38.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:38.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:38.791139+0000) 2019-10-13T09:02:38.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:38.790+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:38.791175+0000) 2019-10-13T09:02:40.285 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:40.283+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.284460+0000) 2019-10-13T09:02:40.285 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:40.283+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.284667+0000) 2019-10-13T09:02:40.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:40.283+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.285173+0000) 2019-10-13T09:02:40.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:40.284+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.285484+0000) 2019-10-13T09:02:40.287 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:40.284+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.285715+0000) 2019-10-13T09:02:40.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:40.284+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.285767+0000) 2019-10-13T09:02:40.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:40.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156710 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:40.340 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:40.284+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.285377+0000) 2019-10-13T09:02:40.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:40.339+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:40.339862+0000) 2019-10-13T09:02:42.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:42.897+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:42.898284+0000) 2019-10-13T09:02:42.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:42.897+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:42.897876+0000) 2019-10-13T09:02:43.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:43.358+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:43.358661+0000) 2019-10-13T09:02:44.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:44.650+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.651338+0000) 2019-10-13T09:02:44.872 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:02:44.872 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:44.877 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:02:44.877 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:45.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:45.292+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.292933+0000) 2019-10-13T09:02:45.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:45.292+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.293068+0000) 2019-10-13T09:02:45.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:45.292+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.293382+0000) 2019-10-13T09:02:45.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:45.293+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.293536+0000) 2019-10-13T09:02:45.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:45.293+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.294512+0000) 2019-10-13T09:02:45.296 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:45.293+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.294089+0000) 2019-10-13T09:02:45.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:45.294+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.294647+0000) 2019-10-13T09:02:45.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:45.296+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:45.297467+0000) 2019-10-13T09:02:45.300 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:45.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:48.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:48.451+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:48.451747+0000) 2019-10-13T09:02:48.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:48.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:48.791241+0000) 2019-10-13T09:02:48.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:48.790+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:48.791332+0000) 2019-10-13T09:02:48.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:48.790+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:48.791301+0000) 2019-10-13T09:02:50.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:50.300+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.301344+0000) 2019-10-13T09:02:50.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:50.301+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.302051+0000) 2019-10-13T09:02:50.303 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:50.301+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.302360+0000) 2019-10-13T09:02:50.303 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:50.302+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.302393+0000) 2019-10-13T09:02:50.303 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:50.302+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.302621+0000) 2019-10-13T09:02:50.304 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:50.302+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.304166+0000) 2019-10-13T09:02:50.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:50.302+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.303237+0000) 2019-10-13T09:02:50.305 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:50.302+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:50.304707+0000) 2019-10-13T09:02:50.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:50.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156774 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:52.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:52.897+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:52.898086+0000) 2019-10-13T09:02:52.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:52.896+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:52.898486+0000) 2019-10-13T09:02:53.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:53.359+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:53.360961+0000) 2019-10-13T09:02:54.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:54.651+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.651501+0000) 2019-10-13T09:02:55.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:55.310+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.310790+0000) 2019-10-13T09:02:55.312 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:55.310+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.311077+0000) 2019-10-13T09:02:55.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:02:55.310+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.311284+0000) 2019-10-13T09:02:55.313 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:02:55.309+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.311632+0000) 2019-10-13T09:02:55.313 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:55.311+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.311665+0000) 2019-10-13T09:02:55.313 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:55.311+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.311844+0000) 2019-10-13T09:02:55.313 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:02:55.311+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.312123+0000) 2019-10-13T09:02:55.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:02:55.312+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:55.314171+0000) 2019-10-13T09:02:55.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:02:55.317+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:02:58.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:02:58.451+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:58.451905+0000) 2019-10-13T09:02:58.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:02:58.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:58.791431+0000) 2019-10-13T09:02:58.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:02:58.791+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:58.791470+0000) 2019-10-13T09:02:58.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:02:58.791+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:58.791523+0000) 2019-10-13T09:03:00.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:00.319+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.319515+0000) 2019-10-13T09:03:00.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:00.318+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.320415+0000) 2019-10-13T09:03:00.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:00.319+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.319933+0000) 2019-10-13T09:03:00.321 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:00.320+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.320464+0000) 2019-10-13T09:03:00.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:00.320+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.320542+0000) 2019-10-13T09:03:00.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:00.320+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.320816+0000) 2019-10-13T09:03:00.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:00.320+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.320870+0000) 2019-10-13T09:03:00.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:00.321+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:00.323273+0000) 2019-10-13T09:03:00.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:00.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:02.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:02.897+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:02.898671+0000) 2019-10-13T09:03:02.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:02.897+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:02.898254+0000) 2019-10-13T09:03:03.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:03.362+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:03.363269+0000) 2019-10-13T09:03:04.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:04.650+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.651709+0000) 2019-10-13T09:03:05.329 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:05.327+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.328627+0000) 2019-10-13T09:03:05.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:05.327+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.328896+0000) 2019-10-13T09:03:05.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:05.327+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.329183+0000) 2019-10-13T09:03:05.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:05.328+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.329401+0000) 2019-10-13T09:03:05.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:05.328+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.329544+0000) 2019-10-13T09:03:05.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:05.328+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.329555+0000) 2019-10-13T09:03:05.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:05.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156870 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:05.355 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:05.327+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.328614+0000) 2019-10-13T09:03:05.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:05.354+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:05.355397+0000) 2019-10-13T09:03:08.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:08.451+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:08.452119+0000) 2019-10-13T09:03:08.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:08.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:08.791628+0000) 2019-10-13T09:03:08.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:08.790+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:08.791717+0000) 2019-10-13T09:03:08.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:08.790+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:08.791738+0000) 2019-10-13T09:03:10.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:10.335+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.336811+0000) 2019-10-13T09:03:10.338 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:10.335+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.337033+0000) 2019-10-13T09:03:10.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:10.335+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.337287+0000) 2019-10-13T09:03:10.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:10.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.337839+0000) 2019-10-13T09:03:10.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:10.336+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.337531+0000) 2019-10-13T09:03:10.339 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:10.336+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.337945+0000) 2019-10-13T09:03:10.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:10.336+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.338344+0000) 2019-10-13T09:03:10.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:10.339+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:10.340434+0000) 2019-10-13T09:03:10.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:10.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:12.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:12.898+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:12.898858+0000) 2019-10-13T09:03:12.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:12.897+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:12.898457+0000) 2019-10-13T09:03:13.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:13.365+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:13.365575+0000) 2019-10-13T09:03:14.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:14.651+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.651905+0000) 2019-10-13T09:03:15.038 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:03:15.038 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:15.042 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:03:15.042 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:15.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:15.344+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.345676+0000) 2019-10-13T09:03:15.347 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:15.344+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.345803+0000) 2019-10-13T09:03:15.347 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:15.344+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.345986+0000) 2019-10-13T09:03:15.347 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:15.344+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.346160+0000) 2019-10-13T09:03:15.348 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:15.346+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.347203+0000) 2019-10-13T09:03:15.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:15.345+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.346765+0000) 2019-10-13T09:03:15.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:15.345+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.346851+0000) 2019-10-13T09:03:15.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:15.348+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:15.349713+0000) 2019-10-13T09:03:15.352 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:15.350+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:18.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:18.451+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:18.452238+0000) 2019-10-13T09:03:18.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:18.790+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:18.791812+0000) 2019-10-13T09:03:18.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:18.790+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:18.791904+0000) 2019-10-13T09:03:18.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:18.790+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:18.791918+0000) 2019-10-13T09:03:20.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:20.353+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.354321+0000) 2019-10-13T09:03:20.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:20.352+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.353838+0000) 2019-10-13T09:03:20.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:20.352+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.353980+0000) 2019-10-13T09:03:20.356 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:20.352+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.354073+0000) 2019-10-13T09:03:20.356 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:20.353+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.354766+0000) 2019-10-13T09:03:20.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:20.353+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.354609+0000) 2019-10-13T09:03:20.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:20.353+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.354946+0000) 2019-10-13T09:03:20.358 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:20.357+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:20.357891+0000) 2019-10-13T09:03:20.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:20.358+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:22.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:22.898+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:22.899068+0000) 2019-10-13T09:03:22.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:22.898+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:22.898659+0000) 2019-10-13T09:03:23.368 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:23.367+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:23.367862+0000) 2019-10-13T09:03:24.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:24.650+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.652057+0000) 2019-10-13T09:03:25.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:25.361+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.362397+0000) 2019-10-13T09:03:25.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:25.360+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.362145+0000) 2019-10-13T09:03:25.363 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:25.361+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.362487+0000) 2019-10-13T09:03:25.363 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:25.361+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.362684+0000) 2019-10-13T09:03:25.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:25.361+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.362953+0000) 2019-10-13T09:03:25.364 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:25.361+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.363126+0000) 2019-10-13T09:03:25.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:25.362+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.363501+0000) 2019-10-13T09:03:25.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:25.365+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:25.366058+0000) 2019-10-13T09:03:25.369 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:25.366+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 156996 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:28.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:28.451+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:28.452482+0000) 2019-10-13T09:03:28.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:28.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:28.791945+0000) 2019-10-13T09:03:28.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:28.791+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:28.792085+0000) 2019-10-13T09:03:28.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:28.791+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:28.792088+0000) 2019-10-13T09:03:30.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:30.369+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.369924+0000) 2019-10-13T09:03:30.372 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:30.370+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.371069+0000) 2019-10-13T09:03:30.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:30.370+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.370417+0000) 2019-10-13T09:03:30.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:30.370+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.370773+0000) 2019-10-13T09:03:30.373 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:30.370+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.370974+0000) 2019-10-13T09:03:30.373 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:30.370+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.371119+0000) 2019-10-13T09:03:30.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:30.371+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.371455+0000) 2019-10-13T09:03:30.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:30.372+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:30.373313+0000) 2019-10-13T09:03:30.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:30.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157026 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:32.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:32.899+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:32.899226+0000) 2019-10-13T09:03:32.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:32.898+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:32.898837+0000) 2019-10-13T09:03:33.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:33.369+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:33.370166+0000) 2019-10-13T09:03:34.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:34.651+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.652238+0000) 2019-10-13T09:03:35.379 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:35.378+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.378632+0000) 2019-10-13T09:03:35.380 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:35.378+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.379447+0000) 2019-10-13T09:03:35.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:35.378+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.379010+0000) 2019-10-13T09:03:35.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:35.379+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.379393+0000) 2019-10-13T09:03:35.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:35.379+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.379626+0000) 2019-10-13T09:03:35.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:35.379+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.379883+0000) 2019-10-13T09:03:35.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:35.379+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.380146+0000) 2019-10-13T09:03:35.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:35.382+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.382737+0000) 2019-10-13T09:03:35.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:35.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157058 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:38.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:38.452+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:38.452700+0000) 2019-10-13T09:03:38.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:38.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:38.792136+0000) 2019-10-13T09:03:38.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:38.791+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:38.792246+0000) 2019-10-13T09:03:38.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:38.791+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:38.792248+0000) 2019-10-13T09:03:40.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:40.386+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.387147+0000) 2019-10-13T09:03:40.389 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:40.386+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.387349+0000) 2019-10-13T09:03:40.389 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:40.387+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.387578+0000) 2019-10-13T09:03:40.389 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:40.387+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.387839+0000) 2019-10-13T09:03:40.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:40.387+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.387970+0000) 2019-10-13T09:03:40.390 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:40.388+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.389287+0000) 2019-10-13T09:03:40.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:40.387+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.388271+0000) 2019-10-13T09:03:40.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:40.390+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.391182+0000) 2019-10-13T09:03:40.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:40.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:42.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:42.897+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:42.899421+0000) 2019-10-13T09:03:42.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:42.898+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:42.899012+0000) 2019-10-13T09:03:43.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:43.371+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:43.372477+0000) 2019-10-13T09:03:44.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:44.652+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:44.652440+0000) 2019-10-13T09:03:45.126 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:03:45.126 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:45.130 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:03:45.130 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:45.397 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:45.395+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.395392+0000) 2019-10-13T09:03:45.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:45.395+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.395690+0000) 2019-10-13T09:03:45.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:45.395+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.395813+0000) 2019-10-13T09:03:45.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:45.395+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.396039+0000) 2019-10-13T09:03:45.398 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:45.395+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.396146+0000) 2019-10-13T09:03:45.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:45.395+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.397046+0000) 2019-10-13T09:03:45.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:45.396+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.397955+0000) 2019-10-13T09:03:45.399 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:45.396+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.396612+0000) 2019-10-13T09:03:45.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:45.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157122 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:48.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:48.451+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:48.452862+0000) 2019-10-13T09:03:48.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:48.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:48.792329+0000) 2019-10-13T09:03:48.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:48.791+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:48.792413+0000) 2019-10-13T09:03:48.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:48.791+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:48.792424+0000) 2019-10-13T09:03:50.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:50.401+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.403605+0000) 2019-10-13T09:03:50.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:50.401+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.402884+0000) 2019-10-13T09:03:50.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:50.401+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.403213+0000) 2019-10-13T09:03:50.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:50.402+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.403803+0000) 2019-10-13T09:03:50.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:50.402+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.403829+0000) 2019-10-13T09:03:50.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:50.402+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.403925+0000) 2019-10-13T09:03:50.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:50.402+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.404107+0000) 2019-10-13T09:03:50.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:50.404+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.406102+0000) 2019-10-13T09:03:50.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:50.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:52.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:52.898+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:52.899605+0000) 2019-10-13T09:03:52.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:52.898+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:52.899209+0000) 2019-10-13T09:03:53.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:53.374+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:53.374817+0000) 2019-10-13T09:03:54.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:54.651+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:54.652642+0000) 2019-10-13T09:03:55.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:55.410+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.412203+0000) 2019-10-13T09:03:55.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:55.411+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.412472+0000) 2019-10-13T09:03:55.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:03:55.411+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.412759+0000) 2019-10-13T09:03:55.414 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:03:55.412+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.413546+0000) 2019-10-13T09:03:55.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:55.411+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.413374+0000) 2019-10-13T09:03:55.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:03:55.412+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.413574+0000) 2019-10-13T09:03:55.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:55.412+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.413818+0000) 2019-10-13T09:03:55.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:03:55.415+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.415965+0000) 2019-10-13T09:03:55.420 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:03:55.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:03:58.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:03:58.451+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:58.453092+0000) 2019-10-13T09:03:58.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:03:58.791+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:58.792538+0000) 2019-10-13T09:03:58.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:03:58.791+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:58.792507+0000) 2019-10-13T09:03:58.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:03:58.791+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:58.792589+0000) 2019-10-13T09:04:00.422 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:00.419+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.420669+0000) 2019-10-13T09:04:00.422 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:00.419+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.420893+0000) 2019-10-13T09:04:00.422 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:00.419+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.420950+0000) 2019-10-13T09:04:00.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:00.421+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.421749+0000) 2019-10-13T09:04:00.423 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:00.420+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.421478+0000) 2019-10-13T09:04:00.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:00.420+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.421639+0000) 2019-10-13T09:04:00.423 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:00.420+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.421917+0000) 2019-10-13T09:04:00.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:00.423+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.423986+0000) 2019-10-13T09:04:00.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:00.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:02.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:02.899+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:02.899783+0000) 2019-10-13T09:04:02.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:02.898+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:02.899361+0000) 2019-10-13T09:04:03.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:03.376+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:03.377236+0000) 2019-10-13T09:04:04.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:04.652+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:04.652860+0000) 2019-10-13T09:04:05.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:05.426+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.427758+0000) 2019-10-13T09:04:05.429 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:05.427+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.428667+0000) 2019-10-13T09:04:05.430 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:05.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.428257+0000) 2019-10-13T09:04:05.430 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:05.427+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.428460+0000) 2019-10-13T09:04:05.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:05.426+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.428345+0000) 2019-10-13T09:04:05.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:05.427+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.428616+0000) 2019-10-13T09:04:05.431 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:05.427+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.429326+0000) 2019-10-13T09:04:05.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:05.431+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.431738+0000) 2019-10-13T09:04:05.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:05.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157248 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:08.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:08.452+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:08.453285+0000) 2019-10-13T09:04:08.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:08.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:08.792703+0000) 2019-10-13T09:04:08.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:08.792+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:08.792750+0000) 2019-10-13T09:04:08.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:08.792+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:08.792818+0000) 2019-10-13T09:04:10.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:10.434+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.435887+0000) 2019-10-13T09:04:10.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:10.434+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.436099+0000) 2019-10-13T09:04:10.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:10.434+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.436319+0000) 2019-10-13T09:04:10.438 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:10.435+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.436539+0000) 2019-10-13T09:04:10.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:10.435+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.436771+0000) 2019-10-13T09:04:10.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:10.435+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.436545+0000) 2019-10-13T09:04:10.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:10.436+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.436846+0000) 2019-10-13T09:04:10.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:10.438+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.439630+0000) 2019-10-13T09:04:10.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:10.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:12.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:12.899+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:12.899931+0000) 2019-10-13T09:04:12.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:12.898+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:12.899549+0000) 2019-10-13T09:04:13.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:13.378+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:13.379602+0000) 2019-10-13T09:04:14.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:14.652+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:14.653034+0000) 2019-10-13T09:04:15.207 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:04:15.207 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:15.211 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:04:15.211 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:15.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:15.440+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.442329+0000) 2019-10-13T09:04:15.443 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:15.441+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.442563+0000) 2019-10-13T09:04:15.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:15.441+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.442716+0000) 2019-10-13T09:04:15.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:15.441+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.442970+0000) 2019-10-13T09:04:15.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:15.443+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.444275+0000) 2019-10-13T09:04:15.445 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:15.441+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.443245+0000) 2019-10-13T09:04:15.445 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:15.442+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.443412+0000) 2019-10-13T09:04:15.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:15.445+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.446468+0000) 2019-10-13T09:04:15.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:15.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:18.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:18.453+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:18.453440+0000) 2019-10-13T09:04:18.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:18.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:18.792870+0000) 2019-10-13T09:04:18.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:18.792+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:18.792944+0000) 2019-10-13T09:04:18.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:18.792+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:18.792968+0000) 2019-10-13T09:04:20.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:20.449+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.450067+0000) 2019-10-13T09:04:20.451 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:20.449+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.450381+0000) 2019-10-13T09:04:20.452 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:20.451+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.451875+0000) 2019-10-13T09:04:20.452 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:20.450+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.450713+0000) 2019-10-13T09:04:20.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:20.450+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.450857+0000) 2019-10-13T09:04:20.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:20.450+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.450969+0000) 2019-10-13T09:04:20.453 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:20.450+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.451166+0000) 2019-10-13T09:04:20.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:20.452+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.453324+0000) 2019-10-13T09:04:20.457 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:20.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:22.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:22.899+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:22.900082+0000) 2019-10-13T09:04:22.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:22.899+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:22.899751+0000) 2019-10-13T09:04:23.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:23.381+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:23.381818+0000) 2019-10-13T09:04:24.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:24.652+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:24.653236+0000) 2019-10-13T09:04:25.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:25.457+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.458236+0000) 2019-10-13T09:04:25.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:25.458+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.458431+0000) 2019-10-13T09:04:25.460 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:25.458+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.458707+0000) 2019-10-13T09:04:25.460 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:25.458+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.459001+0000) 2019-10-13T09:04:25.461 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:25.459+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.460418+0000) 2019-10-13T09:04:25.461 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:25.459+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.459902+0000) 2019-10-13T09:04:25.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:25.459+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.460114+0000) 2019-10-13T09:04:25.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:25.461+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.462011+0000) 2019-10-13T09:04:25.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:25.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:28.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:28.453+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:28.453641+0000) 2019-10-13T09:04:28.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:28.792+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:28.793105+0000) 2019-10-13T09:04:28.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:28.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:28.793071+0000) 2019-10-13T09:04:28.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:28.792+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:28.793139+0000) 2019-10-13T09:04:30.467 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:30.466+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.466979+0000) 2019-10-13T09:04:30.468 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:30.466+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.466566+0000) 2019-10-13T09:04:30.468 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:30.466+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.466931+0000) 2019-10-13T09:04:30.468 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:30.466+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.467105+0000) 2019-10-13T09:04:30.469 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:30.467+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.467679+0000) 2019-10-13T09:04:30.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:30.467+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.467854+0000) 2019-10-13T09:04:30.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:30.467+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.468063+0000) 2019-10-13T09:04:30.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:30.469+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.469939+0000) 2019-10-13T09:04:30.474 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:30.472+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:32.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:32.898+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:32.900259+0000) 2019-10-13T09:04:32.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:32.899+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:32.899918+0000) 2019-10-13T09:04:33.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:33.383+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:33.384317+0000) 2019-10-13T09:04:34.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:34.652+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:34.653409+0000) 2019-10-13T09:04:35.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:35.473+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.475079+0000) 2019-10-13T09:04:35.476 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:35.473+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.475356+0000) 2019-10-13T09:04:35.476 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:35.474+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.476104+0000) 2019-10-13T09:04:35.477 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:35.475+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.477038+0000) 2019-10-13T09:04:35.477 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:35.475+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.476550+0000) 2019-10-13T09:04:35.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:35.475+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.476913+0000) 2019-10-13T09:04:35.478 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:35.475+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.477267+0000) 2019-10-13T09:04:35.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:35.477+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.479503+0000) 2019-10-13T09:04:35.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:35.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:38.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:38.452+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:38.453838+0000) 2019-10-13T09:04:38.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:38.792+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:38.793310+0000) 2019-10-13T09:04:38.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:38.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:38.793270+0000) 2019-10-13T09:04:38.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:38.792+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:38.793348+0000) 2019-10-13T09:04:40.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:40.482+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.484151+0000) 2019-10-13T09:04:40.485 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:40.482+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.484388+0000) 2019-10-13T09:04:40.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:40.483+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.484517+0000) 2019-10-13T09:04:40.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:40.483+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.485431+0000) 2019-10-13T09:04:40.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:40.483+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.485317+0000) 2019-10-13T09:04:40.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:40.484+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.485529+0000) 2019-10-13T09:04:40.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:40.484+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.485567+0000) 2019-10-13T09:04:40.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:40.486+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.488637+0000) 2019-10-13T09:04:40.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:40.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157466 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:42.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:42.899+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:42.900429+0000) 2019-10-13T09:04:42.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:42.899+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:42.900099+0000) 2019-10-13T09:04:43.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:43.385+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:43.386650+0000) 2019-10-13T09:04:44.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:44.652+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:44.653569+0000) 2019-10-13T09:04:45.318 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:04:45.318 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:45.322 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:04:45.322 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:45.493 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:45.490+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.492028+0000) 2019-10-13T09:04:45.493 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:45.490+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.492171+0000) 2019-10-13T09:04:45.494 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:45.491+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.493251+0000) 2019-10-13T09:04:45.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:45.491+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.492548+0000) 2019-10-13T09:04:45.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:45.491+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.492736+0000) 2019-10-13T09:04:45.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:45.491+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.492905+0000) 2019-10-13T09:04:45.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:45.491+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.493039+0000) 2019-10-13T09:04:45.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:45.492+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.494517+0000) 2019-10-13T09:04:45.499 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:45.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157500 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:48.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:48.453+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:48.454026+0000) 2019-10-13T09:04:48.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:48.792+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:48.793492+0000) 2019-10-13T09:04:48.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:48.792+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:48.793448+0000) 2019-10-13T09:04:48.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:48.792+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:48.793532+0000) 2019-10-13T09:04:50.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:50.498+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.499746+0000) 2019-10-13T09:04:50.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:50.498+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.499859+0000) 2019-10-13T09:04:50.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:50.498+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.500248+0000) 2019-10-13T09:04:50.501 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:50.499+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.500408+0000) 2019-10-13T09:04:50.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:50.500+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.501431+0000) 2019-10-13T09:04:50.502 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:50.499+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.500842+0000) 2019-10-13T09:04:50.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:50.499+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.501061+0000) 2019-10-13T09:04:50.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:50.502+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.503477+0000) 2019-10-13T09:04:50.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:50.504+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:52.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:52.899+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:52.900609+0000) 2019-10-13T09:04:52.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:52.899+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:52.900251+0000) 2019-10-13T09:04:53.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:53.388+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:53.388934+0000) 2019-10-13T09:04:54.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:54.653+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:54.653794+0000) 2019-10-13T09:04:55.508 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:55.506+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.507460+0000) 2019-10-13T09:04:55.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:04:55.506+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.507925+0000) 2019-10-13T09:04:55.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:04:55.506+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.508053+0000) 2019-10-13T09:04:55.509 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:55.506+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.508194+0000) 2019-10-13T09:04:55.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:55.506+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.508376+0000) 2019-10-13T09:04:55.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:04:55.508+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.509368+0000) 2019-10-13T09:04:55.510 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:55.507+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.509178+0000) 2019-10-13T09:04:55.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:04:55.510+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.511343+0000) 2019-10-13T09:04:55.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:04:55.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:04:58.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:04:58.453+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:58.454215+0000) 2019-10-13T09:04:58.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:04:58.793+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:58.793658+0000) 2019-10-13T09:04:58.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:04:58.793+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:58.793691+0000) 2019-10-13T09:04:58.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:04:58.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:58.793663+0000) 2019-10-13T09:05:00.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:00.514+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.515952+0000) 2019-10-13T09:05:00.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:00.514+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.516277+0000) 2019-10-13T09:05:00.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:00.514+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.516369+0000) 2019-10-13T09:05:00.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:00.516+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.517441+0000) 2019-10-13T09:05:00.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:00.515+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.517003+0000) 2019-10-13T09:05:00.518 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:00.516+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.517560+0000) 2019-10-13T09:05:00.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:00.516+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.517707+0000) 2019-10-13T09:05:00.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:00.519+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.520402+0000) 2019-10-13T09:05:00.523 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:00.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157592 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:02.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:02.900+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:02.900815+0000) 2019-10-13T09:05:02.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:02.900+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:02.900470+0000) 2019-10-13T09:05:03.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:03.390+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:03.391212+0000) 2019-10-13T09:05:04.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:04.653+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:04.653984+0000) 2019-10-13T09:05:05.533 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:05.525+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.526117+0000) 2019-10-13T09:05:05.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:05.526+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.527695+0000) 2019-10-13T09:05:05.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:05.524+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.525142+0000) 2019-10-13T09:05:05.534 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:05.525+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.525779+0000) 2019-10-13T09:05:05.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:05.525+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.525986+0000) 2019-10-13T09:05:05.535 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:05.525+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.526355+0000) 2019-10-13T09:05:05.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:05.525+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.526325+0000) 2019-10-13T09:05:05.536 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:05.525+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.526074+0000) 2019-10-13T09:05:05.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:05.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157626 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:08.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:08.454+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:08.454427+0000) 2019-10-13T09:05:08.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:08.793+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:08.793865+0000) 2019-10-13T09:05:08.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:08.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:08.793874+0000) 2019-10-13T09:05:08.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:08.793+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:08.793897+0000) 2019-10-13T09:05:10.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:10.532+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.533136+0000) 2019-10-13T09:05:10.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:10.533+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.534468+0000) 2019-10-13T09:05:10.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:10.533+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.533630+0000) 2019-10-13T09:05:10.536 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:10.533+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.533582+0000) 2019-10-13T09:05:10.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:10.533+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.534088+0000) 2019-10-13T09:05:10.536 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:10.533+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.534145+0000) 2019-10-13T09:05:10.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:10.533+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.534350+0000) 2019-10-13T09:05:10.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:10.536+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.537409+0000) 2019-10-13T09:05:10.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:10.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157657 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:12.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:12.900+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:12.900660+0000) 2019-10-13T09:05:12.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:12.900+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:12.901033+0000) 2019-10-13T09:05:13.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:13.393+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:13.393545+0000) 2019-10-13T09:05:14.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:14.654+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:14.654181+0000) 2019-10-13T09:05:15.409 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:05:15.409 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:15.413 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:05:15.414 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:15.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:15.540+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.541144+0000) 2019-10-13T09:05:15.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:15.540+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.541221+0000) 2019-10-13T09:05:15.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:15.541+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.541488+0000) 2019-10-13T09:05:15.544 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:15.541+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.542471+0000) 2019-10-13T09:05:15.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:15.541+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.541642+0000) 2019-10-13T09:05:15.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:15.541+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.541902+0000) 2019-10-13T09:05:15.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:15.541+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.541987+0000) 2019-10-13T09:05:15.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:15.543+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.544707+0000) 2019-10-13T09:05:15.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:15.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:18.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:18.454+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:18.454618+0000) 2019-10-13T09:05:18.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:18.794+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:18.794079+0000) 2019-10-13T09:05:18.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:18.794+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:18.794099+0000) 2019-10-13T09:05:18.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:18.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:18.794096+0000) 2019-10-13T09:05:20.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:20.546+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.548122+0000) 2019-10-13T09:05:20.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:20.547+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.548608+0000) 2019-10-13T09:05:20.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:20.548+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.549438+0000) 2019-10-13T09:05:20.551 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:20.547+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.549030+0000) 2019-10-13T09:05:20.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:20.547+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.549267+0000) 2019-10-13T09:05:20.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:20.548+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.549590+0000) 2019-10-13T09:05:20.552 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:20.548+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.549842+0000) 2019-10-13T09:05:20.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:20.551+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.552185+0000) 2019-10-13T09:05:20.556 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:20.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157719 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:22.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:22.901+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:22.901201+0000) 2019-10-13T09:05:22.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:22.899+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:22.900817+0000) 2019-10-13T09:05:23.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:23.395+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:23.395881+0000) 2019-10-13T09:05:24.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:24.653+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:24.654375+0000) 2019-10-13T09:05:25.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:25.555+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.557551+0000) 2019-10-13T09:05:25.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:25.556+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.557441+0000) 2019-10-13T09:05:25.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:25.556+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.557606+0000) 2019-10-13T09:05:25.559 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:25.556+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.557992+0000) 2019-10-13T09:05:25.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:25.556+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.557989+0000) 2019-10-13T09:05:25.559 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:25.556+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.558237+0000) 2019-10-13T09:05:25.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:25.557+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.558407+0000) 2019-10-13T09:05:25.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:25.558+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.560324+0000) 2019-10-13T09:05:25.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:25.561+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:28.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:28.453+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:28.454804+0000) 2019-10-13T09:05:28.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:28.793+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:28.794293+0000) 2019-10-13T09:05:28.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:28.793+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:28.794300+0000) 2019-10-13T09:05:28.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:28.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:28.794303+0000) 2019-10-13T09:05:30.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:30.564+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.566526+0000) 2019-10-13T09:05:30.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:30.566+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.567880+0000) 2019-10-13T09:05:30.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:30.563+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.564933+0000) 2019-10-13T09:05:30.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:30.564+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.565568+0000) 2019-10-13T09:05:30.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:30.564+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.565470+0000) 2019-10-13T09:05:30.570 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:30.564+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.565678+0000) 2019-10-13T09:05:30.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:30.564+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.566207+0000) 2019-10-13T09:05:30.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:30.564+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.566356+0000) 2019-10-13T09:05:30.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:30.570+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157783 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:32.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:32.900+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:32.901380+0000) 2019-10-13T09:05:32.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:32.900+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:32.900971+0000) 2019-10-13T09:05:33.398 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:33.397+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:33.398258+0000) 2019-10-13T09:05:34.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:34.653+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:34.654571+0000) 2019-10-13T09:05:35.575 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:35.572+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.573796+0000) 2019-10-13T09:05:35.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:35.572+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.574002+0000) 2019-10-13T09:05:35.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:35.572+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.574153+0000) 2019-10-13T09:05:35.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:35.572+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.574221+0000) 2019-10-13T09:05:35.576 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:35.573+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.575063+0000) 2019-10-13T09:05:35.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:35.573+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.574638+0000) 2019-10-13T09:05:35.576 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:35.573+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.575101+0000) 2019-10-13T09:05:35.577 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:35.575+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.577705+0000) 2019-10-13T09:05:35.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:35.578+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:38.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:38.454+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:38.454979+0000) 2019-10-13T09:05:38.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:38.793+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:38.794490+0000) 2019-10-13T09:05:38.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:38.793+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:38.794517+0000) 2019-10-13T09:05:38.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:38.793+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:38.794520+0000) 2019-10-13T09:05:40.582 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:40.581+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.582312+0000) 2019-10-13T09:05:40.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:40.580+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.581833+0000) 2019-10-13T09:05:40.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:40.580+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.582172+0000) 2019-10-13T09:05:40.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:40.580+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.582096+0000) 2019-10-13T09:05:40.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:40.581+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.582488+0000) 2019-10-13T09:05:40.584 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:40.581+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.582547+0000) 2019-10-13T09:05:40.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:40.581+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.582768+0000) 2019-10-13T09:05:40.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:40.583+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.584697+0000) 2019-10-13T09:05:40.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:40.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157845 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:42.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:42.900+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:42.901581+0000) 2019-10-13T09:05:42.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:42.900+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:42.901127+0000) 2019-10-13T09:05:43.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:43.400+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:43.400752+0000) 2019-10-13T09:05:44.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:44.654+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:44.654666+0000) 2019-10-13T09:05:45.498 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:05:45.498 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:45.502 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:05:45.503 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:45.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:45.588+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.588893+0000) 2019-10-13T09:05:45.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:45.586+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.588117+0000) 2019-10-13T09:05:45.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:45.587+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.588591+0000) 2019-10-13T09:05:45.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:45.587+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.588642+0000) 2019-10-13T09:05:45.591 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:45.587+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.588869+0000) 2019-10-13T09:05:45.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:45.587+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.588801+0000) 2019-10-13T09:05:45.591 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:45.587+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.589026+0000) 2019-10-13T09:05:45.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:45.591+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.591839+0000) 2019-10-13T09:05:45.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:45.591+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:48.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:48.454+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:48.455204+0000) 2019-10-13T09:05:48.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:48.794+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:48.794708+0000) 2019-10-13T09:05:48.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:48.794+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:48.794712+0000) 2019-10-13T09:05:48.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:48.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:48.794713+0000) 2019-10-13T09:05:50.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:50.594+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.595081+0000) 2019-10-13T09:05:50.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:50.594+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.594875+0000) 2019-10-13T09:05:50.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:50.594+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.594853+0000) 2019-10-13T09:05:50.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:50.594+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.595274+0000) 2019-10-13T09:05:50.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:50.595+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.595511+0000) 2019-10-13T09:05:50.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:50.594+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.595377+0000) 2019-10-13T09:05:50.597 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:50.595+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.595717+0000) 2019-10-13T09:05:50.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:50.597+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.597939+0000) 2019-10-13T09:05:50.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:50.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:52.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:52.901+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:52.901761+0000) 2019-10-13T09:05:52.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:52.900+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:52.901349+0000) 2019-10-13T09:05:53.403 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:53.402+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:53.403089+0000) 2019-10-13T09:05:54.655 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:54.654+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:54.654832+0000) 2019-10-13T09:05:55.604 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:05:55.602+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.602989+0000) 2019-10-13T09:05:55.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:55.602+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.603134+0000) 2019-10-13T09:05:55.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:55.602+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.603214+0000) 2019-10-13T09:05:55.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:05:55.603+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.603507+0000) 2019-10-13T09:05:55.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:55.603+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.603820+0000) 2019-10-13T09:05:55.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:55.603+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.603917+0000) 2019-10-13T09:05:55.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:05:55.603+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.604234+0000) 2019-10-13T09:05:55.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:05:55.604+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.605710+0000) 2019-10-13T09:05:55.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:05:55.609+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:05:58.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:05:58.455+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:58.455427+0000) 2019-10-13T09:05:58.796 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:05:58.794+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:58.794868+0000) 2019-10-13T09:05:58.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:05:58.794+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:58.794857+0000) 2019-10-13T09:05:58.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:05:58.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:58.794885+0000) 2019-10-13T09:06:00.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:00.611+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.611609+0000) 2019-10-13T09:06:00.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:00.611+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.611844+0000) 2019-10-13T09:06:00.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:00.611+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.611877+0000) 2019-10-13T09:06:00.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:00.611+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.612697+0000) 2019-10-13T09:06:00.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:00.611+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.612166+0000) 2019-10-13T09:06:00.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:00.612+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.612577+0000) 2019-10-13T09:06:00.615 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:00.612+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.612839+0000) 2019-10-13T09:06:00.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:00.613+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.614175+0000) 2019-10-13T09:06:00.619 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:00.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 157972 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:02.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:02.901+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:02.901961+0000) 2019-10-13T09:06:02.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:02.901+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:02.901516+0000) 2019-10-13T09:06:03.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:03.405+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:03.405484+0000) 2019-10-13T09:06:04.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:04.654+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:04.654989+0000) 2019-10-13T09:06:05.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:05.619+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.620081+0000) 2019-10-13T09:06:05.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:05.619+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.620325+0000) 2019-10-13T09:06:05.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:05.620+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.620487+0000) 2019-10-13T09:06:05.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:05.620+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.621513+0000) 2019-10-13T09:06:05.622 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:05.620+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.620694+0000) 2019-10-13T09:06:05.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:05.620+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.620981+0000) 2019-10-13T09:06:05.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:05.620+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.621270+0000) 2019-10-13T09:06:05.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:05.623+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.624209+0000) 2019-10-13T09:06:05.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:05.626+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:08.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:08.454+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:08.455595+0000) 2019-10-13T09:06:08.796 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:08.794+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:08.795054+0000) 2019-10-13T09:06:08.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:08.794+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:08.795038+0000) 2019-10-13T09:06:08.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:08.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:08.795059+0000) 2019-10-13T09:06:10.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:10.627+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.629283+0000) 2019-10-13T09:06:10.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:10.628+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.629537+0000) 2019-10-13T09:06:10.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:10.628+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.629759+0000) 2019-10-13T09:06:10.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:10.628+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.629673+0000) 2019-10-13T09:06:10.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:10.630+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.630885+0000) 2019-10-13T09:06:10.632 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:10.628+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.629973+0000) 2019-10-13T09:06:10.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:10.629+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.630493+0000) 2019-10-13T09:06:10.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:10.632+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.633565+0000) 2019-10-13T09:06:10.636 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:10.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:12.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:12.902+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:12.902112+0000) 2019-10-13T09:06:12.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:12.900+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:12.901683+0000) 2019-10-13T09:06:13.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:13.407+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:13.407799+0000) 2019-10-13T09:06:14.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:14.654+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:14.655152+0000) 2019-10-13T09:06:15.592 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:06:15.592 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:15.597 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:06:15.597 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:15.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:15.637+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.638075+0000) 2019-10-13T09:06:15.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:15.635+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.637117+0000) 2019-10-13T09:06:15.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:15.635+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.637275+0000) 2019-10-13T09:06:15.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:15.636+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.637572+0000) 2019-10-13T09:06:15.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:15.636+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.638180+0000) 2019-10-13T09:06:15.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:15.637+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.638542+0000) 2019-10-13T09:06:15.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:15.637+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.638902+0000) 2019-10-13T09:06:15.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:15.641+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.642353+0000) 2019-10-13T09:06:15.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:15.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:18.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:18.454+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:18.455745+0000) 2019-10-13T09:06:18.809 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:18.794+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:18.795256+0000) 2019-10-13T09:06:18.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:18.794+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:18.795255+0000) 2019-10-13T09:06:18.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:18.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:18.795273+0000) 2019-10-13T09:06:20.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:20.643+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.645343+0000) 2019-10-13T09:06:20.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:20.644+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.646206+0000) 2019-10-13T09:06:20.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:20.644+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.646015+0000) 2019-10-13T09:06:20.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:20.644+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.646155+0000) 2019-10-13T09:06:20.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:20.644+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.646220+0000) 2019-10-13T09:06:20.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:20.645+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.646573+0000) 2019-10-13T09:06:20.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:20.645+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.646772+0000) 2019-10-13T09:06:20.649 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:20.647+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.649199+0000) 2019-10-13T09:06:20.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:20.650+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158098 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:22.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:22.901+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:22.902303+0000) 2019-10-13T09:06:22.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:22.901+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:22.901850+0000) 2019-10-13T09:06:23.410 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:23.409+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:23.410049+0000) 2019-10-13T09:06:24.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:24.654+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:24.655243+0000) 2019-10-13T09:06:25.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:25.652+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.653557+0000) 2019-10-13T09:06:25.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:25.652+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.654065+0000) 2019-10-13T09:06:25.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:25.652+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.654171+0000) 2019-10-13T09:06:25.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:25.652+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.654275+0000) 2019-10-13T09:06:25.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:25.653+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.655303+0000) 2019-10-13T09:06:25.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:25.653+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.654947+0000) 2019-10-13T09:06:25.657 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:25.653+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.655157+0000) 2019-10-13T09:06:25.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:25.654+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.656652+0000) 2019-10-13T09:06:25.661 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:25.658+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158130 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:28.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:28.455+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:28.455914+0000) 2019-10-13T09:06:28.796 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:28.794+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:28.795452+0000) 2019-10-13T09:06:28.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:28.794+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:28.795452+0000) 2019-10-13T09:06:28.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:28.794+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:28.795450+0000) 2019-10-13T09:06:30.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:30.660+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.662139+0000) 2019-10-13T09:06:30.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:30.660+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.662275+0000) 2019-10-13T09:06:30.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:30.661+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.662872+0000) 2019-10-13T09:06:30.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:30.661+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.663100+0000) 2019-10-13T09:06:30.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:30.661+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.663240+0000) 2019-10-13T09:06:30.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:30.662+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.663437+0000) 2019-10-13T09:06:30.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:30.663+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.664245+0000) 2019-10-13T09:06:30.665 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:30.665+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.665885+0000) 2019-10-13T09:06:30.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:30.667+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:32.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:32.901+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:32.902472+0000) 2019-10-13T09:06:32.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:32.901+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:32.902018+0000) 2019-10-13T09:06:33.412 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:33.411+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:33.412320+0000) 2019-10-13T09:06:34.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:34.655+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:34.655417+0000) 2019-10-13T09:06:35.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:35.669+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.670310+0000) 2019-10-13T09:06:35.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:35.670+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.670451+0000) 2019-10-13T09:06:35.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:35.670+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.670592+0000) 2019-10-13T09:06:35.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:35.670+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.670812+0000) 2019-10-13T09:06:35.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:35.670+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.671600+0000) 2019-10-13T09:06:35.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:35.670+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.671192+0000) 2019-10-13T09:06:35.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:35.670+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.671368+0000) 2019-10-13T09:06:35.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:35.673+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.673822+0000) 2019-10-13T09:06:35.677 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:35.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:38.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:38.455+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:38.456091+0000) 2019-10-13T09:06:38.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:38.795+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:38.795575+0000) 2019-10-13T09:06:38.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:38.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:38.795640+0000) 2019-10-13T09:06:38.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:38.795+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:38.795655+0000) 2019-10-13T09:06:40.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:40.677+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.678102+0000) 2019-10-13T09:06:40.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:40.678+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.678513+0000) 2019-10-13T09:06:40.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:40.678+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.678876+0000) 2019-10-13T09:06:40.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:40.678+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.678837+0000) 2019-10-13T09:06:40.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:40.678+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.678987+0000) 2019-10-13T09:06:40.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:40.678+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.679150+0000) 2019-10-13T09:06:40.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:40.679+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.679489+0000) 2019-10-13T09:06:40.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:40.681+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.682148+0000) 2019-10-13T09:06:40.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:40.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:42.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:42.902+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:42.902650+0000) 2019-10-13T09:06:42.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:42.901+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:42.902208+0000) 2019-10-13T09:06:43.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:43.414+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:43.414946+0000) 2019-10-13T09:06:44.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:44.655+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:44.655573+0000) 2019-10-13T09:06:45.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:45.686+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.686429+0000) 2019-10-13T09:06:45.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:45.686+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.687297+0000) 2019-10-13T09:06:45.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:45.686+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.686930+0000) 2019-10-13T09:06:45.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:45.686+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.686852+0000) 2019-10-13T09:06:45.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:45.686+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.687260+0000) 2019-10-13T09:06:45.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:45.687+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.687646+0000) 2019-10-13T09:06:45.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:45.687+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.687798+0000) 2019-10-13T09:06:45.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:45.689+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.690725+0000) 2019-10-13T09:06:45.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:45.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158256 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:45.751 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:06:45.752 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:45.756 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:06:45.757 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:48.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:48.456+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:48.456277+0000) 2019-10-13T09:06:48.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:48.795+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:48.795777+0000) 2019-10-13T09:06:48.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:48.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:48.795803+0000) 2019-10-13T09:06:48.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:48.795+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:48.795829+0000) 2019-10-13T09:06:50.695 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:50.694+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.694712+0000) 2019-10-13T09:06:50.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:50.694+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.694527+0000) 2019-10-13T09:06:50.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:50.694+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.695513+0000) 2019-10-13T09:06:50.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:50.694+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.695327+0000) 2019-10-13T09:06:50.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:50.695+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.695593+0000) 2019-10-13T09:06:50.697 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:50.695+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.695720+0000) 2019-10-13T09:06:50.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:50.695+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.695862+0000) 2019-10-13T09:06:50.698 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:50.697+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.698204+0000) 2019-10-13T09:06:50.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:50.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:52.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:52.902+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:52.902831+0000) 2019-10-13T09:06:52.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:52.901+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:52.902386+0000) 2019-10-13T09:06:53.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:53.417+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:53.417385+0000) 2019-10-13T09:06:54.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:54.654+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:54.655752+0000) 2019-10-13T09:06:55.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:06:55.701+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.702726+0000) 2019-10-13T09:06:55.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:06:55.701+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.702977+0000) 2019-10-13T09:06:55.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:55.701+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.703187+0000) 2019-10-13T09:06:55.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:55.701+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.703310+0000) 2019-10-13T09:06:55.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:06:55.703+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.704566+0000) 2019-10-13T09:06:55.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:06:55.704+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.705089+0000) 2019-10-13T09:06:55.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:55.702+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.703444+0000) 2019-10-13T09:06:55.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:55.702+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.703629+0000) 2019-10-13T09:06:55.709 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:06:55.707+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158318 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:06:58.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:06:58.455+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:58.456446+0000) 2019-10-13T09:06:58.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:06:58.795+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:58.795990+0000) 2019-10-13T09:06:58.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:06:58.795+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:58.795953+0000) 2019-10-13T09:06:58.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:06:58.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:58.795964+0000) 2019-10-13T09:07:00.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:00.709+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.710503+0000) 2019-10-13T09:07:00.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:00.709+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.710714+0000) 2019-10-13T09:07:00.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:00.711+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.711765+0000) 2019-10-13T09:07:00.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:00.709+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.711200+0000) 2019-10-13T09:07:00.713 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:00.709+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.711308+0000) 2019-10-13T09:07:00.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:00.710+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.711470+0000) 2019-10-13T09:07:00.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:00.710+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.711681+0000) 2019-10-13T09:07:00.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:00.712+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.713421+0000) 2019-10-13T09:07:00.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:00.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158350 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:02.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:02.902+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:02.903008+0000) 2019-10-13T09:07:02.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:02.901+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:02.902585+0000) 2019-10-13T09:07:03.419 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:03.419+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:03.419677+0000) 2019-10-13T09:07:04.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:04.655+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:04.655943+0000) 2019-10-13T09:07:05.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:05.718+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.719544+0000) 2019-10-13T09:07:05.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:05.717+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.719179+0000) 2019-10-13T09:07:05.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:05.718+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.719451+0000) 2019-10-13T09:07:05.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:05.718+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.719473+0000) 2019-10-13T09:07:05.721 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:05.718+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.719803+0000) 2019-10-13T09:07:05.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:05.718+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.720215+0000) 2019-10-13T09:07:05.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:05.718+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.720156+0000) 2019-10-13T09:07:05.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:05.721+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.722275+0000) 2019-10-13T09:07:05.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:05.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:08.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:08.455+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:08.456638+0000) 2019-10-13T09:07:08.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:08.795+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:08.796187+0000) 2019-10-13T09:07:08.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:08.795+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:08.796166+0000) 2019-10-13T09:07:08.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:08.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:08.796184+0000) 2019-10-13T09:07:10.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:10.725+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.727701+0000) 2019-10-13T09:07:10.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:10.726+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.727610+0000) 2019-10-13T09:07:10.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:10.726+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.727835+0000) 2019-10-13T09:07:10.730 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:10.726+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.727936+0000) 2019-10-13T09:07:10.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:10.726+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.728088+0000) 2019-10-13T09:07:10.730 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:10.726+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.728255+0000) 2019-10-13T09:07:10.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:10.727+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.728426+0000) 2019-10-13T09:07:10.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:10.728+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.730122+0000) 2019-10-13T09:07:10.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:10.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:12.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:12.902+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:12.903195+0000) 2019-10-13T09:07:12.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:12.902+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:12.902777+0000) 2019-10-13T09:07:13.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:13.421+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:13.421996+0000) 2019-10-13T09:07:14.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:14.655+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:14.656114+0000) 2019-10-13T09:07:15.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:15.734+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.735661+0000) 2019-10-13T09:07:15.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:15.734+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.735822+0000) 2019-10-13T09:07:15.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:15.734+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.736017+0000) 2019-10-13T09:07:15.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:15.735+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.736958+0000) 2019-10-13T09:07:15.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:15.735+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.736419+0000) 2019-10-13T09:07:15.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:15.735+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.736709+0000) 2019-10-13T09:07:15.738 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:15.735+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.737114+0000) 2019-10-13T09:07:15.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:15.738+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.739814+0000) 2019-10-13T09:07:15.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:15.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158444 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:15.881 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:07:15.881 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:15.885 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:07:15.886 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:18.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:18.456+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:18.456846+0000) 2019-10-13T09:07:18.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:18.795+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:18.796382+0000) 2019-10-13T09:07:18.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:18.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:18.796388+0000) 2019-10-13T09:07:18.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:18.795+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:18.796405+0000) 2019-10-13T09:07:20.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:20.743+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.744208+0000) 2019-10-13T09:07:20.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:20.744+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.744597+0000) 2019-10-13T09:07:20.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:20.744+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.744824+0000) 2019-10-13T09:07:20.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:20.744+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.745209+0000) 2019-10-13T09:07:20.746 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:20.745+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.745435+0000) 2019-10-13T09:07:20.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:20.745+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.745792+0000) 2019-10-13T09:07:20.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:20.744+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.746248+0000) 2019-10-13T09:07:20.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:20.747+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.748887+0000) 2019-10-13T09:07:20.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:20.750+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158476 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:22.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:22.902+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:22.903381+0000) 2019-10-13T09:07:22.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:22.902+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:22.902963+0000) 2019-10-13T09:07:23.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:23.423+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:23.424271+0000) 2019-10-13T09:07:24.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:24.655+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:24.656307+0000) 2019-10-13T09:07:25.753 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:25.752+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.753112+0000) 2019-10-13T09:07:25.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:25.751+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.752336+0000) 2019-10-13T09:07:25.757 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:25.752+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.752680+0000) 2019-10-13T09:07:25.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:25.752+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.752820+0000) 2019-10-13T09:07:25.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:25.752+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.753142+0000) 2019-10-13T09:07:25.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:25.752+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.753304+0000) 2019-10-13T09:07:25.758 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:25.753+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.753550+0000) 2019-10-13T09:07:25.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:25.755+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.756696+0000) 2019-10-13T09:07:25.760 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:25.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158508 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:28.458 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:28.456+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:28.457057+0000) 2019-10-13T09:07:28.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:28.796+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:28.796540+0000) 2019-10-13T09:07:28.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:28.796+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:28.796544+0000) 2019-10-13T09:07:28.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:28.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:28.796548+0000) 2019-10-13T09:07:30.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:30.760+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.760916+0000) 2019-10-13T09:07:30.762 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:30.760+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.761192+0000) 2019-10-13T09:07:30.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:30.761+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.762406+0000) 2019-10-13T09:07:30.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:30.761+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.761575+0000) 2019-10-13T09:07:30.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:30.761+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.762038+0000) 2019-10-13T09:07:30.763 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:30.761+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.762144+0000) 2019-10-13T09:07:30.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:30.761+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.762276+0000) 2019-10-13T09:07:30.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:30.764+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.764918+0000) 2019-10-13T09:07:30.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:30.766+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:32.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:32.903+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:32.903550+0000) 2019-10-13T09:07:32.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:32.902+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:32.903139+0000) 2019-10-13T09:07:33.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:33.426+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:33.426695+0000) 2019-10-13T09:07:34.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:34.656+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:34.656505+0000) 2019-10-13T09:07:35.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:35.768+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.769550+0000) 2019-10-13T09:07:35.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:35.768+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.769137+0000) 2019-10-13T09:07:35.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:35.769+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.769443+0000) 2019-10-13T09:07:35.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:35.769+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.769539+0000) 2019-10-13T09:07:35.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:35.769+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.769665+0000) 2019-10-13T09:07:35.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:35.769+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.770070+0000) 2019-10-13T09:07:35.772 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:35.770+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.770630+0000) 2019-10-13T09:07:35.773 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:35.772+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.773335+0000) 2019-10-13T09:07:35.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:35.775+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158570 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:38.458 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:38.456+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:38.457273+0000) 2019-10-13T09:07:38.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:38.795+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:38.796689+0000) 2019-10-13T09:07:38.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:38.795+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:38.796677+0000) 2019-10-13T09:07:38.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:38.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:38.796697+0000) 2019-10-13T09:07:40.778 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:40.776+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.777569+0000) 2019-10-13T09:07:40.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:40.778+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.778870+0000) 2019-10-13T09:07:40.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:40.776+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.778069+0000) 2019-10-13T09:07:40.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:40.776+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.778357+0000) 2019-10-13T09:07:40.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:40.777+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.778710+0000) 2019-10-13T09:07:40.780 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:40.777+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.778817+0000) 2019-10-13T09:07:40.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:40.777+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.778471+0000) 2019-10-13T09:07:40.781 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:40.780+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.781397+0000) 2019-10-13T09:07:40.784 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:40.782+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158602 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:42.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:42.903+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:42.903740+0000) 2019-10-13T09:07:42.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:42.902+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:42.903310+0000) 2019-10-13T09:07:43.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:43.428+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:43.429112+0000) 2019-10-13T09:07:44.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:44.655+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:44.656694+0000) 2019-10-13T09:07:45.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:45.784+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.785604+0000) 2019-10-13T09:07:45.787 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:45.784+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.785825+0000) 2019-10-13T09:07:45.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:45.785+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.786701+0000) 2019-10-13T09:07:45.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:45.784+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.786241+0000) 2019-10-13T09:07:45.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:45.785+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.786850+0000) 2019-10-13T09:07:45.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:45.785+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.787008+0000) 2019-10-13T09:07:45.789 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:45.785+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.786677+0000) 2019-10-13T09:07:45.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:45.788+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.789106+0000) 2019-10-13T09:07:45.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:45.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:45.995 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:07:45.995 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:45.999 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:07:45.999 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:48.458 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:48.456+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:48.457494+0000) 2019-10-13T09:07:48.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:48.796+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:48.796847+0000) 2019-10-13T09:07:48.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:48.796+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:48.796849+0000) 2019-10-13T09:07:48.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:48.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:48.796848+0000) 2019-10-13T09:07:50.797 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:50.794+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.795617+0000) 2019-10-13T09:07:50.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:50.794+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.795878+0000) 2019-10-13T09:07:50.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:50.794+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.796053+0000) 2019-10-13T09:07:50.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:50.794+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.796177+0000) 2019-10-13T09:07:50.797 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:50.794+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.796348+0000) 2019-10-13T09:07:50.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:50.796+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.797170+0000) 2019-10-13T09:07:50.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:50.795+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.796805+0000) 2019-10-13T09:07:50.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:50.798+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.799489+0000) 2019-10-13T09:07:50.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:50.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:52.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:52.903+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:52.903958+0000) 2019-10-13T09:07:52.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:52.902+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:52.903488+0000) 2019-10-13T09:07:53.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:53.431+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:53.431416+0000) 2019-10-13T09:07:54.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:54.656+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:54.656882+0000) 2019-10-13T09:07:55.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:55.802+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.803866+0000) 2019-10-13T09:07:55.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:55.802+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.804242+0000) 2019-10-13T09:07:55.805 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:55.803+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.804460+0000) 2019-10-13T09:07:55.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:07:55.804+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.805410+0000) 2019-10-13T09:07:55.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:55.803+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.804847+0000) 2019-10-13T09:07:55.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:07:55.803+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.804921+0000) 2019-10-13T09:07:55.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:07:55.803+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.805169+0000) 2019-10-13T09:07:55.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:07:55.806+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.807632+0000) 2019-10-13T09:07:55.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:07:55.809+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:07:58.458 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:07:58.457+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:58.457687+0000) 2019-10-13T09:07:58.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:07:58.795+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:58.797033+0000) 2019-10-13T09:07:58.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:07:58.795+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:58.797035+0000) 2019-10-13T09:07:58.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:07:58.795+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:58.797035+0000) 2019-10-13T09:08:00.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:00.810+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.811930+0000) 2019-10-13T09:08:00.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:00.810+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.812024+0000) 2019-10-13T09:08:00.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:00.810+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.812187+0000) 2019-10-13T09:08:00.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:00.810+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.812325+0000) 2019-10-13T09:08:00.814 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:00.811+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.813664+0000) 2019-10-13T09:08:00.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:00.811+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.812682+0000) 2019-10-13T09:08:00.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:00.811+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.812712+0000) 2019-10-13T09:08:00.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:00.813+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.815671+0000) 2019-10-13T09:08:00.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:00.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:02.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:02.903+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:02.904161+0000) 2019-10-13T09:08:02.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:02.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:02.903678+0000) 2019-10-13T09:08:03.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:03.432+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:03.433677+0000) 2019-10-13T09:08:04.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:04.656+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:04.657070+0000) 2019-10-13T09:08:05.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:05.818+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.819124+0000) 2019-10-13T09:08:05.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:05.818+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.819338+0000) 2019-10-13T09:08:05.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:05.819+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.819613+0000) 2019-10-13T09:08:05.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:05.818+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.820482+0000) 2019-10-13T09:08:05.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:05.819+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.819903+0000) 2019-10-13T09:08:05.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:05.819+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.820038+0000) 2019-10-13T09:08:05.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:05.819+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.820302+0000) 2019-10-13T09:08:05.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:05.819+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.821694+0000) 2019-10-13T09:08:05.826 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:05.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158760 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:08.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:08.457+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:08.457904+0000) 2019-10-13T09:08:08.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:08.796+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:08.797229+0000) 2019-10-13T09:08:08.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:08.796+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:08.797222+0000) 2019-10-13T09:08:08.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:08.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:08.797224+0000) 2019-10-13T09:08:10.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:10.826+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.828367+0000) 2019-10-13T09:08:10.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:10.827+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.827796+0000) 2019-10-13T09:08:10.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:10.827+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.828002+0000) 2019-10-13T09:08:10.830 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:10.827+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.828325+0000) 2019-10-13T09:08:10.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:10.828+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.828537+0000) 2019-10-13T09:08:10.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:10.828+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.828582+0000) 2019-10-13T09:08:10.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:10.828+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.829020+0000) 2019-10-13T09:08:10.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:10.829+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.831676+0000) 2019-10-13T09:08:10.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:10.834+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:12.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:12.903+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:12.904378+0000) 2019-10-13T09:08:12.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:12.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:12.903850+0000) 2019-10-13T09:08:13.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:13.435+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:13.436119+0000) 2019-10-13T09:08:14.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:14.657+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:14.657247+0000) 2019-10-13T09:08:15.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:15.835+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.836273+0000) 2019-10-13T09:08:15.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:15.836+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.836635+0000) 2019-10-13T09:08:15.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:15.836+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.836522+0000) 2019-10-13T09:08:15.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:15.837+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.838279+0000) 2019-10-13T09:08:15.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:15.836+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.837112+0000) 2019-10-13T09:08:15.839 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:15.836+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.837278+0000) 2019-10-13T09:08:15.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:15.837+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.837705+0000) 2019-10-13T09:08:15.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:15.839+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.839791+0000) 2019-10-13T09:08:15.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:15.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158822 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:16.166 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:08:16.166 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:16.171 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:08:16.171 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:18.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:18.457+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:18.458108+0000) 2019-10-13T09:08:18.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:18.797+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:18.797390+0000) 2019-10-13T09:08:18.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:18.797+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:18.797388+0000) 2019-10-13T09:08:18.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:18.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:18.797389+0000) 2019-10-13T09:08:20.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:20.844+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.845311+0000) 2019-10-13T09:08:20.847 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:20.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.844805+0000) 2019-10-13T09:08:20.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:20.844+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.845025+0000) 2019-10-13T09:08:20.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:20.844+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.845060+0000) 2019-10-13T09:08:20.848 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:20.844+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.845243+0000) 2019-10-13T09:08:20.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:20.844+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.845294+0000) 2019-10-13T09:08:20.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:20.845+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.845455+0000) 2019-10-13T09:08:20.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:20.848+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.849205+0000) 2019-10-13T09:08:20.851 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:20.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:22.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:22.903+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:22.904582+0000) 2019-10-13T09:08:22.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:22.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:22.904071+0000) 2019-10-13T09:08:23.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:23.438+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:23.438673+0000) 2019-10-13T09:08:24.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:24.656+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:24.657462+0000) 2019-10-13T09:08:25.853 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:25.850+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.851995+0000) 2019-10-13T09:08:25.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:25.851+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.852425+0000) 2019-10-13T09:08:25.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:25.853+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.853919+0000) 2019-10-13T09:08:25.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:25.852+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.853563+0000) 2019-10-13T09:08:25.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:25.852+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.853727+0000) 2019-10-13T09:08:25.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:25.852+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.853986+0000) 2019-10-13T09:08:25.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:25.852+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.854299+0000) 2019-10-13T09:08:25.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:25.855+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.856160+0000) 2019-10-13T09:08:25.860 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:25.858+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:28.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:28.457+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:28.458278+0000) 2019-10-13T09:08:28.798 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:28.796+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:28.797564+0000) 2019-10-13T09:08:28.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:28.796+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:28.797560+0000) 2019-10-13T09:08:28.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:28.796+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:28.797561+0000) 2019-10-13T09:08:30.862 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:30.859+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.861212+0000) 2019-10-13T09:08:30.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:30.861+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.862227+0000) 2019-10-13T09:08:30.863 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:30.860+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.861784+0000) 2019-10-13T09:08:30.863 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:30.860+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.862208+0000) 2019-10-13T09:08:30.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:30.860+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.862156+0000) 2019-10-13T09:08:30.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:30.860+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.862081+0000) 2019-10-13T09:08:30.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:30.861+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.862575+0000) 2019-10-13T09:08:30.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:30.864+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.864945+0000) 2019-10-13T09:08:30.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:30.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:32.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:32.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:32.904254+0000) 2019-10-13T09:08:32.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:32.904+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:32.904766+0000) 2019-10-13T09:08:33.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:33.440+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:33.440963+0000) 2019-10-13T09:08:34.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:34.656+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:34.657683+0000) 2019-10-13T09:08:35.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:35.868+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.869637+0000) 2019-10-13T09:08:35.871 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:35.868+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.870253+0000) 2019-10-13T09:08:35.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:35.870+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.871207+0000) 2019-10-13T09:08:35.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:35.869+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.870832+0000) 2019-10-13T09:08:35.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:35.869+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.871032+0000) 2019-10-13T09:08:35.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:35.869+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.871177+0000) 2019-10-13T09:08:35.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:35.869+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.871382+0000) 2019-10-13T09:08:35.874 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:35.873+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.874605+0000) 2019-10-13T09:08:35.876 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:35.874+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158948 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:38.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:38.457+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:38.458413+0000) 2019-10-13T09:08:38.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:38.797+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:38.797757+0000) 2019-10-13T09:08:38.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:38.797+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:38.797765+0000) 2019-10-13T09:08:38.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:38.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:38.797765+0000) 2019-10-13T09:08:40.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:40.876+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.877953+0000) 2019-10-13T09:08:40.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:40.876+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.878328+0000) 2019-10-13T09:08:40.880 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:40.877+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.878538+0000) 2019-10-13T09:08:40.880 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:40.877+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.878676+0000) 2019-10-13T09:08:40.881 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:40.877+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.878750+0000) 2019-10-13T09:08:40.881 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:40.877+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.878894+0000) 2019-10-13T09:08:40.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:40.879+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.879877+0000) 2019-10-13T09:08:40.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:40.880+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.881003+0000) 2019-10-13T09:08:40.882 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:40.880+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 158980 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:42.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:42.904+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:42.904956+0000) 2019-10-13T09:08:42.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:42.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:42.904429+0000) 2019-10-13T09:08:43.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:43.443+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:43.443272+0000) 2019-10-13T09:08:44.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:44.657+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:44.657879+0000) 2019-10-13T09:08:45.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:45.882+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.883901+0000) 2019-10-13T09:08:45.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:45.882+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.884041+0000) 2019-10-13T09:08:45.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:45.884+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.885444+0000) 2019-10-13T09:08:45.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:45.883+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.884609+0000) 2019-10-13T09:08:45.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:45.883+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.885067+0000) 2019-10-13T09:08:45.886 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:45.883+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.885252+0000) 2019-10-13T09:08:45.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:45.886+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.887085+0000) 2019-10-13T09:08:45.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:45.884+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.885483+0000) 2019-10-13T09:08:45.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:45.889+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159012 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:46.303 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:08:46.303 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:46.308 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:08:46.308 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:48.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:48.458+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:48.458591+0000) 2019-10-13T09:08:48.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:48.797+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:48.797902+0000) 2019-10-13T09:08:48.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:48.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:48.797946+0000) 2019-10-13T09:08:48.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:48.797+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:48.797978+0000) 2019-10-13T09:08:50.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:50.890+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.892331+0000) 2019-10-13T09:08:50.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:50.891+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.892476+0000) 2019-10-13T09:08:50.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:50.891+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.893549+0000) 2019-10-13T09:08:50.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:50.891+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.893025+0000) 2019-10-13T09:08:50.895 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:50.891+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.893292+0000) 2019-10-13T09:08:50.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:50.892+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.893466+0000) 2019-10-13T09:08:50.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:50.892+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.893823+0000) 2019-10-13T09:08:50.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:50.894+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.896437+0000) 2019-10-13T09:08:50.900 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:50.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:52.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:52.903+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:52.905141+0000) 2019-10-13T09:08:52.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:52.904+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:52.904598+0000) 2019-10-13T09:08:53.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:53.445+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:53.446321+0000) 2019-10-13T09:08:54.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:54.657+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:54.658010+0000) 2019-10-13T09:08:55.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:08:55.900+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.901822+0000) 2019-10-13T09:08:55.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:55.900+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.901326+0000) 2019-10-13T09:08:55.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:08:55.901+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.901433+0000) 2019-10-13T09:08:55.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:55.901+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.901601+0000) 2019-10-13T09:08:55.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:55.901+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.901919+0000) 2019-10-13T09:08:55.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:55.901+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.902094+0000) 2019-10-13T09:08:55.904 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:08:55.901+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.902274+0000) 2019-10-13T09:08:55.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:08:55.902+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.904610+0000) 2019-10-13T09:08:55.908 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:08:55.907+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159074 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:08:58.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:08:58.458+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:58.458766+0000) 2019-10-13T09:08:58.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:08:58.797+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:58.798078+0000) 2019-10-13T09:08:58.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:08:58.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:58.798097+0000) 2019-10-13T09:08:58.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:08:58.797+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:58.798168+0000) 2019-10-13T09:09:00.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:00.908+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.909041+0000) 2019-10-13T09:09:00.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:00.908+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.909253+0000) 2019-10-13T09:09:00.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:00.909+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.909632+0000) 2019-10-13T09:09:00.912 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:00.909+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.910107+0000) 2019-10-13T09:09:00.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:00.909+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.910243+0000) 2019-10-13T09:09:00.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:00.910+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.910479+0000) 2019-10-13T09:09:00.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:00.908+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.910083+0000) 2019-10-13T09:09:00.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:00.911+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.913443+0000) 2019-10-13T09:09:00.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:00.915+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:02.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:02.904+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:02.905349+0000) 2019-10-13T09:09:02.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:02.904+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:02.904752+0000) 2019-10-13T09:09:03.449 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:03.448+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:03.448874+0000) 2019-10-13T09:09:04.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:04.658+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:04.658174+0000) 2019-10-13T09:09:05.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:05.916+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.917385+0000) 2019-10-13T09:09:05.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:05.917+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.918220+0000) 2019-10-13T09:09:05.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:05.917+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.917878+0000) 2019-10-13T09:09:05.920 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:05.917+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.918014+0000) 2019-10-13T09:09:05.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:05.917+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.918279+0000) 2019-10-13T09:09:05.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:05.918+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.918518+0000) 2019-10-13T09:09:05.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:05.918+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.919005+0000) 2019-10-13T09:09:05.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:05.920+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.921726+0000) 2019-10-13T09:09:05.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:05.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159138 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:08.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:08.458+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:08.458943+0000) 2019-10-13T09:09:08.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:08.798+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:08.798319+0000) 2019-10-13T09:09:08.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:08.798+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:08.798334+0000) 2019-10-13T09:09:08.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:08.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:08.798320+0000) 2019-10-13T09:09:10.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:10.925+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.926477+0000) 2019-10-13T09:09:10.928 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:10.925+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.926515+0000) 2019-10-13T09:09:10.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:10.925+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.926774+0000) 2019-10-13T09:09:10.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:10.925+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.926851+0000) 2019-10-13T09:09:10.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:10.926+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.927585+0000) 2019-10-13T09:09:10.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:10.926+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.927598+0000) 2019-10-13T09:09:10.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:10.926+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.927819+0000) 2019-10-13T09:09:10.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:10.929+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.930208+0000) 2019-10-13T09:09:10.934 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:10.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159170 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:12.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:12.904+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:12.905533+0000) 2019-10-13T09:09:12.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:12.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:12.904869+0000) 2019-10-13T09:09:13.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:13.450+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:13.451219+0000) 2019-10-13T09:09:14.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:14.657+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:14.658370+0000) 2019-10-13T09:09:15.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:15.934+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.935418+0000) 2019-10-13T09:09:15.937 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:15.933+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.935200+0000) 2019-10-13T09:09:15.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:15.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.935294+0000) 2019-10-13T09:09:15.937 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:15.934+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.935442+0000) 2019-10-13T09:09:15.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:15.934+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.935796+0000) 2019-10-13T09:09:15.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:15.934+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.935527+0000) 2019-10-13T09:09:15.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:15.934+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.936100+0000) 2019-10-13T09:09:15.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:15.937+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.937757+0000) 2019-10-13T09:09:15.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:15.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:16.386 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:09:16.386 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:16.412 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:09:16.412 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:18.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:18.458+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:18.459158+0000) 2019-10-13T09:09:18.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:18.797+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:18.798505+0000) 2019-10-13T09:09:18.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:18.797+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:18.798507+0000) 2019-10-13T09:09:18.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:18.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:18.798511+0000) 2019-10-13T09:09:20.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:20.942+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.943616+0000) 2019-10-13T09:09:20.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:20.941+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.943177+0000) 2019-10-13T09:09:20.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:20.941+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.943310+0000) 2019-10-13T09:09:20.945 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:20.942+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.943597+0000) 2019-10-13T09:09:20.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:20.942+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.943725+0000) 2019-10-13T09:09:20.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:20.942+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.943941+0000) 2019-10-13T09:09:20.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:20.943+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.944674+0000) 2019-10-13T09:09:20.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:20.946+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.947173+0000) 2019-10-13T09:09:20.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:20.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:22.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:22.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:22.905054+0000) 2019-10-13T09:09:22.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:22.905+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:22.905713+0000) 2019-10-13T09:09:23.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:23.453+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:23.453654+0000) 2019-10-13T09:09:24.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:24.657+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:24.658574+0000) 2019-10-13T09:09:25.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:25.949+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.951322+0000) 2019-10-13T09:09:25.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:25.950+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.951657+0000) 2019-10-13T09:09:25.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:25.950+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.951775+0000) 2019-10-13T09:09:25.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:25.951+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.952521+0000) 2019-10-13T09:09:25.955 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:25.950+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.952321+0000) 2019-10-13T09:09:25.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:25.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.952492+0000) 2019-10-13T09:09:25.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:25.951+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.952649+0000) 2019-10-13T09:09:25.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:25.954+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.955154+0000) 2019-10-13T09:09:25.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:25.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159264 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:28.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:28.458+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:28.459347+0000) 2019-10-13T09:09:28.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:28.798+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:28.798700+0000) 2019-10-13T09:09:28.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:28.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:28.798701+0000) 2019-10-13T09:09:28.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:28.798+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:28.798708+0000) 2019-10-13T09:09:30.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:30.960+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.960772+0000) 2019-10-13T09:09:30.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:30.958+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.960290+0000) 2019-10-13T09:09:30.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:30.958+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.960374+0000) 2019-10-13T09:09:30.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:30.959+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.960523+0000) 2019-10-13T09:09:30.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:30.959+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.960884+0000) 2019-10-13T09:09:30.963 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:30.959+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.960907+0000) 2019-10-13T09:09:30.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:30.960+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.961534+0000) 2019-10-13T09:09:30.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:30.963+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.963992+0000) 2019-10-13T09:09:30.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:30.965+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159296 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:32.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:32.905+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:32.905917+0000) 2019-10-13T09:09:32.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:32.903+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:32.905215+0000) 2019-10-13T09:09:33.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:33.455+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:33.455991+0000) 2019-10-13T09:09:34.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:34.657+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:34.658738+0000) 2019-10-13T09:09:35.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:35.968+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.969304+0000) 2019-10-13T09:09:35.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:35.967+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.969011+0000) 2019-10-13T09:09:35.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:35.968+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.969490+0000) 2019-10-13T09:09:35.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:35.967+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.969300+0000) 2019-10-13T09:09:35.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:35.968+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.969709+0000) 2019-10-13T09:09:35.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:35.970+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.971275+0000) 2019-10-13T09:09:35.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:35.968+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.970006+0000) 2019-10-13T09:09:35.972 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:35.968+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.969968+0000) 2019-10-13T09:09:35.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:35.974+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:38.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:38.459+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:38.459548+0000) 2019-10-13T09:09:38.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:38.798+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:38.798866+0000) 2019-10-13T09:09:38.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:38.798+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:38.798865+0000) 2019-10-13T09:09:38.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:38.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:38.798863+0000) 2019-10-13T09:09:40.978 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:40.976+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.976979+0000) 2019-10-13T09:09:40.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:40.976+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.977135+0000) 2019-10-13T09:09:40.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:40.977+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.977967+0000) 2019-10-13T09:09:40.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:40.977+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.977713+0000) 2019-10-13T09:09:40.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:40.977+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.977993+0000) 2019-10-13T09:09:40.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:40.977+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.978168+0000) 2019-10-13T09:09:40.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:40.977+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.978298+0000) 2019-10-13T09:09:40.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:40.979+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.980306+0000) 2019-10-13T09:09:40.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:40.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159360 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:42.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:42.904+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:42.906076+0000) 2019-10-13T09:09:42.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:42.904+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:42.905384+0000) 2019-10-13T09:09:43.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:43.457+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:43.458321+0000) 2019-10-13T09:09:44.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:44.658+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:44.658896+0000) 2019-10-13T09:09:45.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:45.984+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.985236+0000) 2019-10-13T09:09:45.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:45.985+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.985528+0000) 2019-10-13T09:09:45.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:45.984+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.986594+0000) 2019-10-13T09:09:45.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:45.985+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.985622+0000) 2019-10-13T09:09:45.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:45.985+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.986126+0000) 2019-10-13T09:09:45.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:45.985+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.986383+0000) 2019-10-13T09:09:45.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:45.986+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.986621+0000) 2019-10-13T09:09:45.989 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:45.988+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.989802+0000) 2019-10-13T09:09:45.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:45.991+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159390 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:46.501 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:09:46.501 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:46.505 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:09:46.506 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:48.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:48.459+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:48.459739+0000) 2019-10-13T09:09:48.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:48.798+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:48.799076+0000) 2019-10-13T09:09:48.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:48.798+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:48.799078+0000) 2019-10-13T09:09:48.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:48.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:48.799073+0000) 2019-10-13T09:09:50.995 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:50.993+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.993824+0000) 2019-10-13T09:09:50.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:50.993+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.994072+0000) 2019-10-13T09:09:50.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:50.993+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.994371+0000) 2019-10-13T09:09:50.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:50.994+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.994667+0000) 2019-10-13T09:09:50.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:50.994+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.994988+0000) 2019-10-13T09:09:50.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:50.993+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.995187+0000) 2019-10-13T09:09:50.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:50.995+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.996877+0000) 2019-10-13T09:09:50.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:50.994+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.995309+0000) 2019-10-13T09:09:51.001 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:51.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:52.906 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:52.905+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:52.906245+0000) 2019-10-13T09:09:52.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:52.905+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:52.905568+0000) 2019-10-13T09:09:53.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:53.460+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:53.461629+0000) 2019-10-13T09:09:54.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:54.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:54.659082+0000) 2019-10-13T09:09:56.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:09:56.000+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.002333+0000) 2019-10-13T09:09:56.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:56.001+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.002466+0000) 2019-10-13T09:09:56.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:56.001+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.002647+0000) 2019-10-13T09:09:56.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:09:56.002+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.003594+0000) 2019-10-13T09:09:56.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:56.001+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.003160+0000) 2019-10-13T09:09:56.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:56.001+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.003325+0000) 2019-10-13T09:09:56.005 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:09:56.002+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.004022+0000) 2019-10-13T09:09:56.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:09:56.003+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.005086+0000) 2019-10-13T09:09:56.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:09:56.007+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:09:58.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:09:58.458+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:58.459957+0000) 2019-10-13T09:09:58.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:09:58.797+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:58.799300+0000) 2019-10-13T09:09:58.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:09:58.797+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:58.799298+0000) 2019-10-13T09:09:58.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:09:58.797+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:58.799303+0000) 2019-10-13T09:10:01.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:01.009+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.010439+0000) 2019-10-13T09:10:01.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:01.009+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.010618+0000) 2019-10-13T09:10:01.013 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:01.009+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.010756+0000) 2019-10-13T09:10:01.013 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:01.009+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.011051+0000) 2019-10-13T09:10:01.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:01.009+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.011192+0000) 2019-10-13T09:10:01.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:01.011+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.012111+0000) 2019-10-13T09:10:01.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:01.012+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.012918+0000) 2019-10-13T09:10:01.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:01.010+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.011608+0000) 2019-10-13T09:10:01.016 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:01.014+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:02.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:02.905+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:02.906450+0000) 2019-10-13T09:10:02.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:02.904+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:02.905741+0000) 2019-10-13T09:10:03.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:03.463+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:03.463976+0000) 2019-10-13T09:10:04.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:04.658+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:04.659277+0000) 2019-10-13T09:10:06.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:06.016+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.017701+0000) 2019-10-13T09:10:06.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:06.018+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.018930+0000) 2019-10-13T09:10:06.021 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:06.016+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.018133+0000) 2019-10-13T09:10:06.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:06.016+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.018058+0000) 2019-10-13T09:10:06.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:06.017+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.018498+0000) 2019-10-13T09:10:06.021 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:06.017+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.018920+0000) 2019-10-13T09:10:06.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:06.017+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.019068+0000) 2019-10-13T09:10:06.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:06.021+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.022306+0000) 2019-10-13T09:10:06.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:06.022+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:08.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:08.459+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:08.460144+0000) 2019-10-13T09:10:08.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:08.798+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:08.799507+0000) 2019-10-13T09:10:08.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:08.798+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:08.799505+0000) 2019-10-13T09:10:08.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:08.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:08.799538+0000) 2019-10-13T09:10:11.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:11.025+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.026668+0000) 2019-10-13T09:10:11.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:11.024+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.026282+0000) 2019-10-13T09:10:11.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:11.025+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.026451+0000) 2019-10-13T09:10:11.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:11.025+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.026990+0000) 2019-10-13T09:10:11.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:11.025+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.027330+0000) 2019-10-13T09:10:11.029 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:11.025+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.027387+0000) 2019-10-13T09:10:11.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:11.026+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.027591+0000) 2019-10-13T09:10:11.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:11.028+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.029658+0000) 2019-10-13T09:10:11.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:11.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:12.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:12.906+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:12.906634+0000) 2019-10-13T09:10:12.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:12.905+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:12.905939+0000) 2019-10-13T09:10:13.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:13.465+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:13.466333+0000) 2019-10-13T09:10:14.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:14.658+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:14.659480+0000) 2019-10-13T09:10:16.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:16.034+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.035593+0000) 2019-10-13T09:10:16.036 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:16.033+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.035134+0000) 2019-10-13T09:10:16.036 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:16.034+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.035460+0000) 2019-10-13T09:10:16.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:16.034+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.035893+0000) 2019-10-13T09:10:16.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:16.034+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.036200+0000) 2019-10-13T09:10:16.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:16.035+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.036451+0000) 2019-10-13T09:10:16.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:16.035+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.036631+0000) 2019-10-13T09:10:16.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:16.037+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.038669+0000) 2019-10-13T09:10:16.044 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:16.040+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:16.597 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:10:16.597 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:16.602 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:10:16.603 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:18.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:18.459+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:18.460338+0000) 2019-10-13T09:10:18.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:18.798+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:18.799658+0000) 2019-10-13T09:10:18.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:18.798+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:18.799703+0000) 2019-10-13T09:10:18.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:18.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:18.799737+0000) 2019-10-13T09:10:21.045 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:21.042+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.043862+0000) 2019-10-13T09:10:21.045 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:21.042+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.044327+0000) 2019-10-13T09:10:21.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:21.044+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.045220+0000) 2019-10-13T09:10:21.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:21.043+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.044970+0000) 2019-10-13T09:10:21.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:21.043+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.045016+0000) 2019-10-13T09:10:21.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:21.043+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.045195+0000) 2019-10-13T09:10:21.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:21.044+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.045426+0000) 2019-10-13T09:10:21.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:21.046+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.047572+0000) 2019-10-13T09:10:21.051 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:21.049+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:22.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:22.906+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:22.906813+0000) 2019-10-13T09:10:22.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:22.905+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:22.906125+0000) 2019-10-13T09:10:23.469 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:23.468+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:23.468751+0000) 2019-10-13T09:10:24.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:24.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:24.659678+0000) 2019-10-13T09:10:26.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:26.052+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.053353+0000) 2019-10-13T09:10:26.054 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:26.052+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.052884+0000) 2019-10-13T09:10:26.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:26.052+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.053066+0000) 2019-10-13T09:10:26.055 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:26.052+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.053186+0000) 2019-10-13T09:10:26.055 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:26.053+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.053526+0000) 2019-10-13T09:10:26.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:26.053+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.053661+0000) 2019-10-13T09:10:26.056 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:26.053+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.054081+0000) 2019-10-13T09:10:26.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:26.055+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.056133+0000) 2019-10-13T09:10:26.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:26.059+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159642 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:28.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:28.460+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:28.460561+0000) 2019-10-13T09:10:28.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:28.799+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:28.799832+0000) 2019-10-13T09:10:28.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:28.799+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:28.799895+0000) 2019-10-13T09:10:28.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:28.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:28.799921+0000) 2019-10-13T09:10:31.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:31.060+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.060912+0000) 2019-10-13T09:10:31.062 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:31.060+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.061044+0000) 2019-10-13T09:10:31.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:31.061+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.061954+0000) 2019-10-13T09:10:31.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:31.061+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.061460+0000) 2019-10-13T09:10:31.064 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:31.061+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.061638+0000) 2019-10-13T09:10:31.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:31.061+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.061864+0000) 2019-10-13T09:10:31.064 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:31.061+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.062131+0000) 2019-10-13T09:10:31.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:31.064+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.064753+0000) 2019-10-13T09:10:31.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:31.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:32.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:32.906+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:32.906318+0000) 2019-10-13T09:10:32.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:32.906+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:32.907025+0000) 2019-10-13T09:10:33.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:33.471+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:33.471117+0000) 2019-10-13T09:10:34.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:34.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:34.659867+0000) 2019-10-13T09:10:36.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:36.068+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.069210+0000) 2019-10-13T09:10:36.070 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:36.069+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.069547+0000) 2019-10-13T09:10:36.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:36.069+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.069524+0000) 2019-10-13T09:10:36.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:36.069+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.069841+0000) 2019-10-13T09:10:36.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:36.069+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.071450+0000) 2019-10-13T09:10:36.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:36.070+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.071031+0000) 2019-10-13T09:10:36.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:36.070+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.071153+0000) 2019-10-13T09:10:36.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:36.071+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.073297+0000) 2019-10-13T09:10:36.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:36.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:38.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:38.460+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:38.460764+0000) 2019-10-13T09:10:38.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:38.799+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:38.800004+0000) 2019-10-13T09:10:38.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:38.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:38.800080+0000) 2019-10-13T09:10:38.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:38.799+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:38.800068+0000) 2019-10-13T09:10:41.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:41.077+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.078513+0000) 2019-10-13T09:10:41.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:41.077+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.078913+0000) 2019-10-13T09:10:41.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:41.078+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.080058+0000) 2019-10-13T09:10:41.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:41.077+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.079069+0000) 2019-10-13T09:10:41.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:41.078+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.079672+0000) 2019-10-13T09:10:41.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:41.078+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.079888+0000) 2019-10-13T09:10:41.082 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:41.078+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.079972+0000) 2019-10-13T09:10:41.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:41.080+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.082375+0000) 2019-10-13T09:10:41.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:41.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:42.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:42.906+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:42.907185+0000) 2019-10-13T09:10:42.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:42.905+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:42.906470+0000) 2019-10-13T09:10:43.474 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:43.472+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:43.473568+0000) 2019-10-13T09:10:44.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:44.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:44.660056+0000) 2019-10-13T09:10:46.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:46.086+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.088058+0000) 2019-10-13T09:10:46.089 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:46.086+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.088154+0000) 2019-10-13T09:10:46.089 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:46.087+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.088499+0000) 2019-10-13T09:10:46.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:46.087+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.088776+0000) 2019-10-13T09:10:46.090 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:46.087+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.089032+0000) 2019-10-13T09:10:46.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:46.087+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.089330+0000) 2019-10-13T09:10:46.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:46.088+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.089630+0000) 2019-10-13T09:10:46.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:46.089+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.091210+0000) 2019-10-13T09:10:46.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:46.093+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159768 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:46.694 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:10:46.694 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:46.699 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:10:46.699 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:48.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:48.460+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:48.461024+0000) 2019-10-13T09:10:48.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:48.798+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:48.800207+0000) 2019-10-13T09:10:48.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:48.798+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:48.800299+0000) 2019-10-13T09:10:48.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:48.798+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:48.800296+0000) 2019-10-13T09:10:51.097 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:51.094+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.096198+0000) 2019-10-13T09:10:51.098 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:51.095+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.096532+0000) 2019-10-13T09:10:51.098 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:51.095+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.096820+0000) 2019-10-13T09:10:51.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:51.095+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.096976+0000) 2019-10-13T09:10:51.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:51.096+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.097092+0000) 2019-10-13T09:10:51.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:51.096+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.097449+0000) 2019-10-13T09:10:51.099 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:51.096+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.097665+0000) 2019-10-13T09:10:51.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:51.099+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.100018+0000) 2019-10-13T09:10:51.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:51.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:52.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:52.905+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:52.906627+0000) 2019-10-13T09:10:52.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:52.906+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:52.907347+0000) 2019-10-13T09:10:53.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:53.475+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:53.475904+0000) 2019-10-13T09:10:54.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:54.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:54.660189+0000) 2019-10-13T09:10:56.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:56.103+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.104525+0000) 2019-10-13T09:10:56.106 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:10:56.103+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.105116+0000) 2019-10-13T09:10:56.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:56.103+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.105266+0000) 2019-10-13T09:10:56.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:10:56.105+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.106218+0000) 2019-10-13T09:10:56.107 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:10:56.104+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.105776+0000) 2019-10-13T09:10:56.107 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:56.104+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.105974+0000) 2019-10-13T09:10:56.108 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:56.104+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.106111+0000) 2019-10-13T09:10:56.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:10:56.107+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.108556+0000) 2019-10-13T09:10:56.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:10:56.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:10:58.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:10:58.460+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:58.461239+0000) 2019-10-13T09:10:58.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:10:58.799+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:58.800392+0000) 2019-10-13T09:10:58.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:10:58.799+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:58.800470+0000) 2019-10-13T09:10:58.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:10:58.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:58.800504+0000) 2019-10-13T09:11:01.114 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:01.111+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.112964+0000) 2019-10-13T09:11:01.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:01.111+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.113240+0000) 2019-10-13T09:11:01.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:01.111+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.113366+0000) 2019-10-13T09:11:01.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:01.113+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.114715+0000) 2019-10-13T09:11:01.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:01.112+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.113650+0000) 2019-10-13T09:11:01.115 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:01.112+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.113724+0000) 2019-10-13T09:11:01.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:01.112+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.114246+0000) 2019-10-13T09:11:01.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:01.115+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.115999+0000) 2019-10-13T09:11:01.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:01.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:02.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:02.906+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:02.907507+0000) 2019-10-13T09:11:02.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:02.906+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:02.906846+0000) 2019-10-13T09:11:03.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:03.477+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:03.478226+0000) 2019-10-13T09:11:04.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:04.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:04.660324+0000) 2019-10-13T09:11:06.122 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:06.119+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.120929+0000) 2019-10-13T09:11:06.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:06.119+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.121142+0000) 2019-10-13T09:11:06.122 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:06.120+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.121493+0000) 2019-10-13T09:11:06.123 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:06.121+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.122530+0000) 2019-10-13T09:11:06.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:06.120+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.121908+0000) 2019-10-13T09:11:06.123 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:06.120+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.122107+0000) 2019-10-13T09:11:06.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:06.120+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.122376+0000) 2019-10-13T09:11:06.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:06.124+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.125439+0000) 2019-10-13T09:11:06.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:06.126+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:08.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:08.461+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:08.461456+0000) 2019-10-13T09:11:08.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:08.800+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:08.800580+0000) 2019-10-13T09:11:08.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:08.800+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:08.800660+0000) 2019-10-13T09:11:08.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:08.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:08.800673+0000) 2019-10-13T09:11:11.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:11.128+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.129253+0000) 2019-10-13T09:11:11.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:11.129+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.129489+0000) 2019-10-13T09:11:11.131 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:11.129+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.129650+0000) 2019-10-13T09:11:11.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:11.130+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.131079+0000) 2019-10-13T09:11:11.132 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:11.129+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.129845+0000) 2019-10-13T09:11:11.132 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:11.129+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.130058+0000) 2019-10-13T09:11:11.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:11.129+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.130247+0000) 2019-10-13T09:11:11.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:11.132+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.133453+0000) 2019-10-13T09:11:11.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:11.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:12.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:12.907+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:12.907658+0000) 2019-10-13T09:11:12.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:12.906+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:12.907040+0000) 2019-10-13T09:11:13.481 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:13.480+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:13.480488+0000) 2019-10-13T09:11:14.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:14.660+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:14.660516+0000) 2019-10-13T09:11:16.139 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:16.137+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.138010+0000) 2019-10-13T09:11:16.139 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:16.137+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.138271+0000) 2019-10-13T09:11:16.139 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:16.138+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.139186+0000) 2019-10-13T09:11:16.140 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:16.138+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.139275+0000) 2019-10-13T09:11:16.140 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:16.139+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.139430+0000) 2019-10-13T09:11:16.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:16.139+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.139577+0000) 2019-10-13T09:11:16.141 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:16.139+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.139871+0000) 2019-10-13T09:11:16.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:16.141+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.142544+0000) 2019-10-13T09:11:16.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:16.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159957 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:16.776 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:11:16.776 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:16.780 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:11:16.780 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:18.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:18.461+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:18.461628+0000) 2019-10-13T09:11:18.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:18.800+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:18.800759+0000) 2019-10-13T09:11:18.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:18.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:18.800790+0000) 2019-10-13T09:11:18.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:18.800+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:18.800841+0000) 2019-10-13T09:11:21.149 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:21.147+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.147984+0000) 2019-10-13T09:11:21.149 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:21.146+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.147211+0000) 2019-10-13T09:11:21.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:21.147+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.147429+0000) 2019-10-13T09:11:21.149 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:21.147+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.147513+0000) 2019-10-13T09:11:21.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:21.147+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.147696+0000) 2019-10-13T09:11:21.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:21.147+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.147832+0000) 2019-10-13T09:11:21.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:21.147+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.147965+0000) 2019-10-13T09:11:21.150 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:21.149+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.149838+0000) 2019-10-13T09:11:21.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:21.152+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 159990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:22.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:22.907+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:22.907836+0000) 2019-10-13T09:11:22.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:22.907+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:22.907253+0000) 2019-10-13T09:11:23.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:23.482+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:23.482938+0000) 2019-10-13T09:11:24.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:24.660+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:24.660706+0000) 2019-10-13T09:11:26.156 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:26.154+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.155789+0000) 2019-10-13T09:11:26.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:26.155+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.155520+0000) 2019-10-13T09:11:26.157 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:26.155+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.155681+0000) 2019-10-13T09:11:26.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:26.155+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.155823+0000) 2019-10-13T09:11:26.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:26.155+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.155946+0000) 2019-10-13T09:11:26.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:26.155+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.156131+0000) 2019-10-13T09:11:26.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:26.155+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.156378+0000) 2019-10-13T09:11:26.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:26.156+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.157993+0000) 2019-10-13T09:11:26.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:26.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:28.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:28.460+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:28.461785+0000) 2019-10-13T09:11:28.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:28.799+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:28.800980+0000) 2019-10-13T09:11:28.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:28.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:28.800986+0000) 2019-10-13T09:11:28.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:28.799+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:28.800997+0000) 2019-10-13T09:11:31.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:31.162+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.164191+0000) 2019-10-13T09:11:31.165 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:31.162+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.163782+0000) 2019-10-13T09:11:31.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:31.162+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.163998+0000) 2019-10-13T09:11:31.166 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:31.162+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.164182+0000) 2019-10-13T09:11:31.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:31.163+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.164575+0000) 2019-10-13T09:11:31.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:31.163+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.164455+0000) 2019-10-13T09:11:31.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:31.163+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.164721+0000) 2019-10-13T09:11:31.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:31.164+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.166288+0000) 2019-10-13T09:11:31.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:31.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:32.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:32.907+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:32.907988+0000) 2019-10-13T09:11:32.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:32.906+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:32.907436+0000) 2019-10-13T09:11:33.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:33.484+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:33.485277+0000) 2019-10-13T09:11:34.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:34.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:34.660887+0000) 2019-10-13T09:11:36.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:36.170+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.171621+0000) 2019-10-13T09:11:36.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:36.170+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.171741+0000) 2019-10-13T09:11:36.173 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:36.171+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.173151+0000) 2019-10-13T09:11:36.174 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:36.170+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.172147+0000) 2019-10-13T09:11:36.174 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:36.171+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.172852+0000) 2019-10-13T09:11:36.174 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:36.171+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.173143+0000) 2019-10-13T09:11:36.174 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:36.172+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.173449+0000) 2019-10-13T09:11:36.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:36.174+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.176460+0000) 2019-10-13T09:11:36.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:36.177+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:38.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:38.461+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:38.461968+0000) 2019-10-13T09:11:38.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:38.800+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:38.801154+0000) 2019-10-13T09:11:38.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:38.800+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:38.801154+0000) 2019-10-13T09:11:38.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:38.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:38.801156+0000) 2019-10-13T09:11:41.181 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:41.178+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.180293+0000) 2019-10-13T09:11:41.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:41.179+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.180666+0000) 2019-10-13T09:11:41.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:41.179+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.180989+0000) 2019-10-13T09:11:41.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:41.179+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.181137+0000) 2019-10-13T09:11:41.182 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:41.179+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.181375+0000) 2019-10-13T09:11:41.183 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:41.180+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.181769+0000) 2019-10-13T09:11:41.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:41.181+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.182283+0000) 2019-10-13T09:11:41.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:41.182+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.183671+0000) 2019-10-13T09:11:41.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:41.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:42.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:42.907+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:42.908187+0000) 2019-10-13T09:11:42.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:42.906+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:42.907623+0000) 2019-10-13T09:11:43.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:43.486+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:43.487544+0000) 2019-10-13T09:11:44.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:44.659+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:44.661096+0000) 2019-10-13T09:11:46.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:46.186+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.188313+0000) 2019-10-13T09:11:46.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:46.188+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.189321+0000) 2019-10-13T09:11:46.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:46.187+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.189107+0000) 2019-10-13T09:11:46.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:46.187+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.189337+0000) 2019-10-13T09:11:46.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:46.188+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.189648+0000) 2019-10-13T09:11:46.191 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:46.188+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.189834+0000) 2019-10-13T09:11:46.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:46.188+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.190049+0000) 2019-10-13T09:11:46.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:46.190+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.191711+0000) 2019-10-13T09:11:46.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:46.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:46.872 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:11:46.872 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:46.876 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:11:46.876 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:48.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:48.461+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:48.462147+0000) 2019-10-13T09:11:48.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:48.799+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:48.801363+0000) 2019-10-13T09:11:48.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:48.799+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:48.801341+0000) 2019-10-13T09:11:48.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:48.799+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:48.801360+0000) 2019-10-13T09:11:51.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:51.195+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.197313+0000) 2019-10-13T09:11:51.199 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:51.197+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.198568+0000) 2019-10-13T09:11:51.199 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:51.196+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.197841+0000) 2019-10-13T09:11:51.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:51.196+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.198104+0000) 2019-10-13T09:11:51.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:51.196+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.198232+0000) 2019-10-13T09:11:51.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:51.196+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.198323+0000) 2019-10-13T09:11:51.200 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:51.197+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.198680+0000) 2019-10-13T09:11:51.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:51.200+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.201319+0000) 2019-10-13T09:11:51.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:51.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:52.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:52.907+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:52.908331+0000) 2019-10-13T09:11:52.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:52.907+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:52.907800+0000) 2019-10-13T09:11:53.490 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:53.489+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:53.489794+0000) 2019-10-13T09:11:54.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:54.660+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:54.661283+0000) 2019-10-13T09:11:56.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:56.205+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.205870+0000) 2019-10-13T09:11:56.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:56.205+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.205911+0000) 2019-10-13T09:11:56.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:11:56.205+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.206245+0000) 2019-10-13T09:11:56.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:11:56.207+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.207775+0000) 2019-10-13T09:11:56.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:11:56.208+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.209653+0000) 2019-10-13T09:11:56.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:56.206+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.206457+0000) 2019-10-13T09:11:56.213 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:11:56.206+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.207169+0000) 2019-10-13T09:11:56.214 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:56.206+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.207260+0000) 2019-10-13T09:11:56.214 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:11:56.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:11:58.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:11:58.461+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:58.462323+0000) 2019-10-13T09:11:58.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:11:58.801+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:58.801544+0000) 2019-10-13T09:11:58.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:11:58.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:58.801579+0000) 2019-10-13T09:11:58.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:11:58.801+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:58.801590+0000) 2019-10-13T09:12:01.215 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:01.214+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.214515+0000) 2019-10-13T09:12:01.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:01.214+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.214638+0000) 2019-10-13T09:12:01.216 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:01.214+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.214797+0000) 2019-10-13T09:12:01.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:01.215+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.216507+0000) 2019-10-13T09:12:01.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:01.214+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.215258+0000) 2019-10-13T09:12:01.217 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:01.215+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.215445+0000) 2019-10-13T09:12:01.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:01.215+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.215839+0000) 2019-10-13T09:12:01.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:01.218+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.218936+0000) 2019-10-13T09:12:01.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:01.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:02.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:02.907+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:02.908502+0000) 2019-10-13T09:12:02.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:02.907+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:02.908001+0000) 2019-10-13T09:12:03.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:03.491+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:03.492183+0000) 2019-10-13T09:12:04.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:04.661+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:04.661470+0000) 2019-10-13T09:12:06.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:06.222+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.223264+0000) 2019-10-13T09:12:06.224 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:06.223+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.223499+0000) 2019-10-13T09:12:06.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:06.223+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.223764+0000) 2019-10-13T09:12:06.225 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:06.223+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.223638+0000) 2019-10-13T09:12:06.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:06.223+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.224059+0000) 2019-10-13T09:12:06.225 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:06.224+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.224566+0000) 2019-10-13T09:12:06.226 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:06.224+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.225604+0000) 2019-10-13T09:12:06.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:06.226+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.227364+0000) 2019-10-13T09:12:06.230 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:06.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:08.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:08.462+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:08.462500+0000) 2019-10-13T09:12:08.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:08.801+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:08.801696+0000) 2019-10-13T09:12:08.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:08.801+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:08.801764+0000) 2019-10-13T09:12:08.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:08.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:08.801766+0000) 2019-10-13T09:12:11.232 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:11.231+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.231577+0000) 2019-10-13T09:12:11.233 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:11.231+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.232516+0000) 2019-10-13T09:12:11.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:11.231+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.232206+0000) 2019-10-13T09:12:11.234 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:11.232+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.232422+0000) 2019-10-13T09:12:11.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:11.232+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.232842+0000) 2019-10-13T09:12:11.234 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:11.232+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.233067+0000) 2019-10-13T09:12:11.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:11.232+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.233257+0000) 2019-10-13T09:12:11.235 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:11.235+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.235871+0000) 2019-10-13T09:12:11.240 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:11.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:12.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:12.908+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:12.908614+0000) 2019-10-13T09:12:12.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:12.906+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:12.908129+0000) 2019-10-13T09:12:13.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:13.493+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:13.494575+0000) 2019-10-13T09:12:14.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:14.660+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:14.661646+0000) 2019-10-13T09:12:16.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:16.239+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.240579+0000) 2019-10-13T09:12:16.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:16.239+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.240684+0000) 2019-10-13T09:12:16.242 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:16.239+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.241187+0000) 2019-10-13T09:12:16.243 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:16.241+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.242211+0000) 2019-10-13T09:12:16.243 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:16.240+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.241767+0000) 2019-10-13T09:12:16.243 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:16.240+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.242071+0000) 2019-10-13T09:12:16.243 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:16.240+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.242178+0000) 2019-10-13T09:12:16.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:16.243+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.244454+0000) 2019-10-13T09:12:16.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:16.246+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:16.962 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:12:16.962 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:16.967 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:12:16.967 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:18.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:18.461+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:18.462697+0000) 2019-10-13T09:12:18.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:18.800+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:18.801836+0000) 2019-10-13T09:12:18.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:18.800+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:18.801918+0000) 2019-10-13T09:12:18.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:18.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:18.801917+0000) 2019-10-13T09:12:21.251 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:21.248+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.250308+0000) 2019-10-13T09:12:21.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:21.248+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.250213+0000) 2019-10-13T09:12:21.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:21.249+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.250697+0000) 2019-10-13T09:12:21.252 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:21.249+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.250844+0000) 2019-10-13T09:12:21.253 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:21.250+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.252051+0000) 2019-10-13T09:12:21.253 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:21.249+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.251284+0000) 2019-10-13T09:12:21.253 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:21.250+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.251467+0000) 2019-10-13T09:12:21.255 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:21.253+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.254932+0000) 2019-10-13T09:12:21.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:21.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:22.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:22.907+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:22.908769+0000) 2019-10-13T09:12:22.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:22.906+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:22.908287+0000) 2019-10-13T09:12:23.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:23.496+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:23.497022+0000) 2019-10-13T09:12:24.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:24.660+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:24.661810+0000) 2019-10-13T09:12:26.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:26.256+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.257906+0000) 2019-10-13T09:12:26.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:26.256+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.258114+0000) 2019-10-13T09:12:26.260 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:26.257+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.259349+0000) 2019-10-13T09:12:26.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:26.257+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.258745+0000) 2019-10-13T09:12:26.260 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:26.257+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.258858+0000) 2019-10-13T09:12:26.260 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:26.257+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.259115+0000) 2019-10-13T09:12:26.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:26.257+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.259112+0000) 2019-10-13T09:12:26.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:26.259+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.261377+0000) 2019-10-13T09:12:26.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:26.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:28.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:28.462+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:28.462860+0000) 2019-10-13T09:12:28.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:28.800+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:28.801976+0000) 2019-10-13T09:12:28.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:28.800+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:28.802055+0000) 2019-10-13T09:12:28.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:28.800+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:28.802049+0000) 2019-10-13T09:12:31.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:31.264+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.266198+0000) 2019-10-13T09:12:31.268 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:31.265+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.267238+0000) 2019-10-13T09:12:31.268 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:31.265+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.266922+0000) 2019-10-13T09:12:31.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:31.265+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.267210+0000) 2019-10-13T09:12:31.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:31.265+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.267381+0000) 2019-10-13T09:12:31.269 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:31.266+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.267696+0000) 2019-10-13T09:12:31.269 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:31.266+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.267933+0000) 2019-10-13T09:12:31.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:31.267+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.269565+0000) 2019-10-13T09:12:31.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:31.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:32.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:32.908+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:32.908940+0000) 2019-10-13T09:12:32.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:32.907+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:32.908416+0000) 2019-10-13T09:12:33.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:33.498+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:33.499307+0000) 2019-10-13T09:12:34.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:34.660+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:34.661981+0000) 2019-10-13T09:12:36.275 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:36.274+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.275590+0000) 2019-10-13T09:12:36.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:36.273+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.275127+0000) 2019-10-13T09:12:36.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:36.273+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.275355+0000) 2019-10-13T09:12:36.277 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:36.274+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.275822+0000) 2019-10-13T09:12:36.277 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:36.274+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.276101+0000) 2019-10-13T09:12:36.277 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:36.274+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.276359+0000) 2019-10-13T09:12:36.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:36.275+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.276543+0000) 2019-10-13T09:12:36.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:36.278+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.279240+0000) 2019-10-13T09:12:36.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:36.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:38.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:38.462+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:38.463055+0000) 2019-10-13T09:12:38.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:38.801+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:38.802148+0000) 2019-10-13T09:12:38.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:38.801+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:38.802212+0000) 2019-10-13T09:12:38.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:38.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:38.802235+0000) 2019-10-13T09:12:41.284 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:41.283+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.284401+0000) 2019-10-13T09:12:41.285 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:41.283+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.283911+0000) 2019-10-13T09:12:41.285 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:41.283+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.284037+0000) 2019-10-13T09:12:41.286 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:41.283+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.284351+0000) 2019-10-13T09:12:41.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:41.284+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.284588+0000) 2019-10-13T09:12:41.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:41.284+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.285015+0000) 2019-10-13T09:12:41.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:41.284+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.285166+0000) 2019-10-13T09:12:41.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:41.286+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.286997+0000) 2019-10-13T09:12:41.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:41.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:42.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:42.908+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:42.909113+0000) 2019-10-13T09:12:42.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:42.908+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:42.908583+0000) 2019-10-13T09:12:43.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:43.501+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:43.501711+0000) 2019-10-13T09:12:44.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:44.661+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:44.662155+0000) 2019-10-13T09:12:46.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:46.291+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.292034+0000) 2019-10-13T09:12:46.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:46.291+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.292339+0000) 2019-10-13T09:12:46.294 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:46.291+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.292374+0000) 2019-10-13T09:12:46.294 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:46.292+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.292742+0000) 2019-10-13T09:12:46.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:46.292+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.293549+0000) 2019-10-13T09:12:46.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:46.292+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.293154+0000) 2019-10-13T09:12:46.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:46.292+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.293309+0000) 2019-10-13T09:12:46.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:46.294+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.295663+0000) 2019-10-13T09:12:46.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:46.298+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:47.096 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:12:47.096 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:47.100 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:12:47.100 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:48.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:48.462+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:48.463256+0000) 2019-10-13T09:12:48.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:48.802+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:48.802341+0000) 2019-10-13T09:12:48.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:48.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:48.802412+0000) 2019-10-13T09:12:48.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:48.802+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:48.802417+0000) 2019-10-13T09:12:51.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:51.300+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.300619+0000) 2019-10-13T09:12:51.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:51.300+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.300792+0000) 2019-10-13T09:12:51.303 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:51.300+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.301702+0000) 2019-10-13T09:12:51.303 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:51.300+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.301144+0000) 2019-10-13T09:12:51.303 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:51.300+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.301351+0000) 2019-10-13T09:12:51.303 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:51.301+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.301694+0000) 2019-10-13T09:12:51.303 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:51.301+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.301827+0000) 2019-10-13T09:12:51.304 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:51.302+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.303304+0000) 2019-10-13T09:12:51.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:51.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:52.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:52.908+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:52.909289+0000) 2019-10-13T09:12:52.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:52.908+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:52.908773+0000) 2019-10-13T09:12:53.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:53.503+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:53.504220+0000) 2019-10-13T09:12:54.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:54.661+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:54.662319+0000) 2019-10-13T09:12:56.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:56.308+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.308455+0000) 2019-10-13T09:12:56.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:56.308+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.308652+0000) 2019-10-13T09:12:56.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:12:56.309+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.310031+0000) 2019-10-13T09:12:56.311 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:12:56.308+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.309037+0000) 2019-10-13T09:12:56.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:12:56.308+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.309136+0000) 2019-10-13T09:12:56.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:56.309+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.309544+0000) 2019-10-13T09:12:56.311 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:56.309+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.309696+0000) 2019-10-13T09:12:56.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:12:56.311+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.312099+0000) 2019-10-13T09:12:56.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:12:56.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:12:58.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:12:58.463+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:58.463389+0000) 2019-10-13T09:12:58.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:12:58.801+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:58.802516+0000) 2019-10-13T09:12:58.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:12:58.801+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:58.802578+0000) 2019-10-13T09:12:58.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:12:58.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:58.802574+0000) 2019-10-13T09:13:01.316 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:01.313+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.315230+0000) 2019-10-13T09:13:01.316 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:01.314+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.315409+0000) 2019-10-13T09:13:01.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:01.314+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.315590+0000) 2019-10-13T09:13:01.317 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:01.316+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.316763+0000) 2019-10-13T09:13:01.318 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:01.314+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.316291+0000) 2019-10-13T09:13:01.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:01.315+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.316452+0000) 2019-10-13T09:13:01.318 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:01.315+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.316462+0000) 2019-10-13T09:13:01.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:01.318+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.319635+0000) 2019-10-13T09:13:01.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:01.320+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:02.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:02.908+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:02.909453+0000) 2019-10-13T09:13:02.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:02.907+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:02.908946+0000) 2019-10-13T09:13:03.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:03.506+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:03.506521+0000) 2019-10-13T09:13:04.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:04.661+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:04.662536+0000) 2019-10-13T09:13:06.325 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:06.322+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.323669+0000) 2019-10-13T09:13:06.325 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:06.322+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.323795+0000) 2019-10-13T09:13:06.326 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:06.324+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.324794+0000) 2019-10-13T09:13:06.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:06.322+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.324384+0000) 2019-10-13T09:13:06.327 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:06.323+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.324490+0000) 2019-10-13T09:13:06.327 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:06.323+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.324557+0000) 2019-10-13T09:13:06.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:06.323+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.324772+0000) 2019-10-13T09:13:06.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:06.326+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.327122+0000) 2019-10-13T09:13:06.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:06.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:08.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:08.462+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:08.463596+0000) 2019-10-13T09:13:08.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:08.801+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:08.802738+0000) 2019-10-13T09:13:08.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:08.801+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:08.802726+0000) 2019-10-13T09:13:08.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:08.801+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:08.802738+0000) 2019-10-13T09:13:11.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:11.329+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.330554+0000) 2019-10-13T09:13:11.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:11.329+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.330811+0000) 2019-10-13T09:13:11.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:11.330+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.331755+0000) 2019-10-13T09:13:11.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:11.329+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.330909+0000) 2019-10-13T09:13:11.334 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:11.330+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.331429+0000) 2019-10-13T09:13:11.334 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:11.330+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.331541+0000) 2019-10-13T09:13:11.335 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:11.330+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.331832+0000) 2019-10-13T09:13:11.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:11.333+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.335320+0000) 2019-10-13T09:13:11.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:11.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:12.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:12.908+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:12.909672+0000) 2019-10-13T09:13:12.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:12.908+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:12.909103+0000) 2019-10-13T09:13:13.509 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:13.507+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:13.508932+0000) 2019-10-13T09:13:14.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:14.662+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:14.662739+0000) 2019-10-13T09:13:16.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:16.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.339241+0000) 2019-10-13T09:13:16.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:16.337+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.338750+0000) 2019-10-13T09:13:16.340 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:16.337+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.338989+0000) 2019-10-13T09:13:16.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:16.337+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.339369+0000) 2019-10-13T09:13:16.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:16.338+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.339491+0000) 2019-10-13T09:13:16.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:16.338+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.339628+0000) 2019-10-13T09:13:16.341 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:16.339+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.341633+0000) 2019-10-13T09:13:16.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:16.338+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.339798+0000) 2019-10-13T09:13:16.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:16.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:17.199 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:13:17.200 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:17.214 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:13:17.215 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:18.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:18.463+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:18.463796+0000) 2019-10-13T09:13:18.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:18.802+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:18.802903+0000) 2019-10-13T09:13:18.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:18.802+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:18.802896+0000) 2019-10-13T09:13:18.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:18.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:18.802904+0000) 2019-10-13T09:13:21.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:21.345+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.347298+0000) 2019-10-13T09:13:21.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:21.346+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.347774+0000) 2019-10-13T09:13:21.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:21.346+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.347967+0000) 2019-10-13T09:13:21.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:21.346+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.348144+0000) 2019-10-13T09:13:21.350 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:21.346+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.348377+0000) 2019-10-13T09:13:21.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:21.347+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.348544+0000) 2019-10-13T09:13:21.350 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:21.347+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.348512+0000) 2019-10-13T09:13:21.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:21.348+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.350356+0000) 2019-10-13T09:13:21.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:21.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:22.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:22.909+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:22.909851+0000) 2019-10-13T09:13:22.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:22.908+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:22.909289+0000) 2019-10-13T09:13:23.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:23.510+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:23.511265+0000) 2019-10-13T09:13:24.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:24.662+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:24.662933+0000) 2019-10-13T09:13:26.356 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:26.355+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.355530+0000) 2019-10-13T09:13:26.357 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:26.355+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.356410+0000) 2019-10-13T09:13:26.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:26.355+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.356042+0000) 2019-10-13T09:13:26.357 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:26.355+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.356286+0000) 2019-10-13T09:13:26.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:26.355+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.356138+0000) 2019-10-13T09:13:26.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:26.356+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.356620+0000) 2019-10-13T09:13:26.358 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:26.358+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.358735+0000) 2019-10-13T09:13:26.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:26.356+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.356745+0000) 2019-10-13T09:13:26.363 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:26.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:28.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:28.463+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:28.464002+0000) 2019-10-13T09:13:28.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:28.802+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:28.803132+0000) 2019-10-13T09:13:28.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:28.802+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:28.803134+0000) 2019-10-13T09:13:28.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:28.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:28.803134+0000) 2019-10-13T09:13:31.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:31.363+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.364553+0000) 2019-10-13T09:13:31.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:31.363+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.364078+0000) 2019-10-13T09:13:31.366 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:31.364+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.364477+0000) 2019-10-13T09:13:31.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:31.364+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.364584+0000) 2019-10-13T09:13:31.366 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:31.364+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.364692+0000) 2019-10-13T09:13:31.367 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:31.364+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.364829+0000) 2019-10-13T09:13:31.367 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:31.364+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.364976+0000) 2019-10-13T09:13:31.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:31.366+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.366987+0000) 2019-10-13T09:13:31.370 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:31.369+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160808 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:32.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:32.909+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:32.910005+0000) 2019-10-13T09:13:32.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:32.909+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:32.909474+0000) 2019-10-13T09:13:33.514 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:33.512+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:33.513575+0000) 2019-10-13T09:13:34.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:34.662+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:34.663133+0000) 2019-10-13T09:13:36.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:36.371+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.371596+0000) 2019-10-13T09:13:36.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:36.372+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.372818+0000) 2019-10-13T09:13:36.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:36.371+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.372331+0000) 2019-10-13T09:13:36.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:36.372+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.372590+0000) 2019-10-13T09:13:36.378 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:36.372+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.372552+0000) 2019-10-13T09:13:36.378 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:36.372+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.373069+0000) 2019-10-13T09:13:36.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:36.372+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.373046+0000) 2019-10-13T09:13:36.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:36.373+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.374593+0000) 2019-10-13T09:13:36.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:36.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:38.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:38.464+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:38.464181+0000) 2019-10-13T09:13:38.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:38.803+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:38.803301+0000) 2019-10-13T09:13:38.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:38.803+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:38.803304+0000) 2019-10-13T09:13:38.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:38.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:38.803302+0000) 2019-10-13T09:13:41.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:41.379+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.379997+0000) 2019-10-13T09:13:41.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:41.379+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.380110+0000) 2019-10-13T09:13:41.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:41.379+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.380257+0000) 2019-10-13T09:13:41.382 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:41.380+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.380415+0000) 2019-10-13T09:13:41.382 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:41.380+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.380488+0000) 2019-10-13T09:13:41.382 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:41.380+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.380681+0000) 2019-10-13T09:13:41.383 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:41.381+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.381757+0000) 2019-10-13T09:13:41.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:41.383+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.383898+0000) 2019-10-13T09:13:41.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:41.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:42.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:42.909+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:42.910177+0000) 2019-10-13T09:13:42.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:42.909+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:42.909651+0000) 2019-10-13T09:13:43.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:43.515+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:43.516055+0000) 2019-10-13T09:13:44.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:44.662+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:44.663333+0000) 2019-10-13T09:13:46.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:46.385+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.386773+0000) 2019-10-13T09:13:46.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:46.385+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.387077+0000) 2019-10-13T09:13:46.389 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:46.385+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.386978+0000) 2019-10-13T09:13:46.389 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:46.385+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.387220+0000) 2019-10-13T09:13:46.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:46.386+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.387807+0000) 2019-10-13T09:13:46.389 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:46.386+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.387960+0000) 2019-10-13T09:13:46.390 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:46.387+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.388319+0000) 2019-10-13T09:13:46.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:46.390+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.391079+0000) 2019-10-13T09:13:46.395 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:46.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:47.329 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:13:47.329 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:47.334 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:13:47.334 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:48.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:48.463+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:48.464380+0000) 2019-10-13T09:13:48.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:48.802+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:48.803446+0000) 2019-10-13T09:13:48.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:48.802+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:48.803451+0000) 2019-10-13T09:13:48.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:48.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:48.803451+0000) 2019-10-13T09:13:51.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:51.393+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.394913+0000) 2019-10-13T09:13:51.396 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:51.393+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.395114+0000) 2019-10-13T09:13:51.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:51.393+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.395264+0000) 2019-10-13T09:13:51.397 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:51.394+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.395421+0000) 2019-10-13T09:13:51.397 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:51.396+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.396930+0000) 2019-10-13T09:13:51.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:51.394+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.396143+0000) 2019-10-13T09:13:51.398 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:51.395+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.396805+0000) 2019-10-13T09:13:51.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:51.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:51.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:51.397+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.398682+0000) 2019-10-13T09:13:52.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:52.910+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:52.910385+0000) 2019-10-13T09:13:52.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:52.908+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:52.909837+0000) 2019-10-13T09:13:53.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:53.518+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:53.518461+0000) 2019-10-13T09:13:54.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:54.662+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:54.663549+0000) 2019-10-13T09:13:56.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:13:56.402+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.404233+0000) 2019-10-13T09:13:56.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:56.403+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.404432+0000) 2019-10-13T09:13:56.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:56.403+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.404935+0000) 2019-10-13T09:13:56.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:56.403+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.405042+0000) 2019-10-13T09:13:56.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:56.403+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.405170+0000) 2019-10-13T09:13:56.407 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:13:56.405+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.406413+0000) 2019-10-13T09:13:56.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:13:56.404+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.405900+0000) 2019-10-13T09:13:56.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:13:56.407+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.407873+0000) 2019-10-13T09:13:56.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:13:56.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:13:58.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:13:58.463+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:58.464584+0000) 2019-10-13T09:13:58.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:13:58.802+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:58.803612+0000) 2019-10-13T09:13:58.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:13:58.802+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:58.803614+0000) 2019-10-13T09:13:58.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:13:58.802+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:58.803612+0000) 2019-10-13T09:14:01.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:01.410+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.412350+0000) 2019-10-13T09:14:01.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:01.411+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.412485+0000) 2019-10-13T09:14:01.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:01.411+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.412812+0000) 2019-10-13T09:14:01.414 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:01.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.413609+0000) 2019-10-13T09:14:01.415 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:01.411+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.413028+0000) 2019-10-13T09:14:01.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:01.411+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.413122+0000) 2019-10-13T09:14:01.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:01.412+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.413529+0000) 2019-10-13T09:14:01.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:01.414+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.416058+0000) 2019-10-13T09:14:01.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:01.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 160998 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:02.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:02.909+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:02.910567+0000) 2019-10-13T09:14:02.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:02.909+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:02.910045+0000) 2019-10-13T09:14:03.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:03.520+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:03.521475+0000) 2019-10-13T09:14:04.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:04.663+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:04.663700+0000) 2019-10-13T09:14:06.419 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:06.417+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.419675+0000) 2019-10-13T09:14:06.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:06.417+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.419247+0000) 2019-10-13T09:14:06.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:06.417+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.419372+0000) 2019-10-13T09:14:06.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:06.418+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.419631+0000) 2019-10-13T09:14:06.421 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:06.418+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.419710+0000) 2019-10-13T09:14:06.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:06.418+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.419841+0000) 2019-10-13T09:14:06.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:06.418+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.420280+0000) 2019-10-13T09:14:06.422 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:06.419+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.421652+0000) 2019-10-13T09:14:06.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:06.422+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161029 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:08.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:08.464+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:08.464787+0000) 2019-10-13T09:14:08.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:08.803+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:08.803801+0000) 2019-10-13T09:14:08.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:08.803+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:08.803809+0000) 2019-10-13T09:14:08.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:08.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:08.803809+0000) 2019-10-13T09:14:11.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:11.424+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.426364+0000) 2019-10-13T09:14:11.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:11.424+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.426041+0000) 2019-10-13T09:14:11.428 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:11.424+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.426251+0000) 2019-10-13T09:14:11.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:11.424+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.426300+0000) 2019-10-13T09:14:11.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:11.425+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.426434+0000) 2019-10-13T09:14:11.429 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:11.425+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.426583+0000) 2019-10-13T09:14:11.429 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:11.425+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.427122+0000) 2019-10-13T09:14:11.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:11.427+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.428858+0000) 2019-10-13T09:14:11.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:11.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:12.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:12.909+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:12.910746+0000) 2019-10-13T09:14:12.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:12.909+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:12.910205+0000) 2019-10-13T09:14:13.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:13.523+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:13.523922+0000) 2019-10-13T09:14:14.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:14.663+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:14.663906+0000) 2019-10-13T09:14:16.435 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:16.433+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.433967+0000) 2019-10-13T09:14:16.435 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:16.433+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.434170+0000) 2019-10-13T09:14:16.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:16.433+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.434263+0000) 2019-10-13T09:14:16.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:16.434+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.434451+0000) 2019-10-13T09:14:16.436 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:16.434+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.435286+0000) 2019-10-13T09:14:16.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:16.434+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.434685+0000) 2019-10-13T09:14:16.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:16.434+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.435325+0000) 2019-10-13T09:14:16.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:16.437+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.438136+0000) 2019-10-13T09:14:16.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:16.440+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:17.445 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:14:17.445 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:17.449 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:14:17.449 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:18.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:18.464+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:18.464965+0000) 2019-10-13T09:14:18.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:18.803+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:18.803973+0000) 2019-10-13T09:14:18.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:18.803+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:18.803978+0000) 2019-10-13T09:14:18.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:18.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:18.803974+0000) 2019-10-13T09:14:21.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:21.441+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.442080+0000) 2019-10-13T09:14:21.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:21.442+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.443503+0000) 2019-10-13T09:14:21.444 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:21.442+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.442474+0000) 2019-10-13T09:14:21.444 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:21.442+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.442597+0000) 2019-10-13T09:14:21.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:21.443+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.443444+0000) 2019-10-13T09:14:21.445 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:21.443+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.443581+0000) 2019-10-13T09:14:21.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:21.443+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.443833+0000) 2019-10-13T09:14:21.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:21.444+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.445145+0000) 2019-10-13T09:14:21.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:21.448+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161124 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:22.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:22.910+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:22.910920+0000) 2019-10-13T09:14:22.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:22.910+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:22.910437+0000) 2019-10-13T09:14:23.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:23.525+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:23.526348+0000) 2019-10-13T09:14:24.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:24.663+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:24.664092+0000) 2019-10-13T09:14:26.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:26.450+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.450870+0000) 2019-10-13T09:14:26.452 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:26.451+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.451785+0000) 2019-10-13T09:14:26.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:26.450+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.451230+0000) 2019-10-13T09:14:26.453 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:26.451+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.451678+0000) 2019-10-13T09:14:26.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:26.451+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.451939+0000) 2019-10-13T09:14:26.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:26.452+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.453426+0000) 2019-10-13T09:14:26.454 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:26.451+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.452233+0000) 2019-10-13T09:14:26.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:26.451+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.452297+0000) 2019-10-13T09:14:26.458 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:26.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:28.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:28.465+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:28.465179+0000) 2019-10-13T09:14:28.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:28.804+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:28.804182+0000) 2019-10-13T09:14:28.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:28.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:28.804181+0000) 2019-10-13T09:14:28.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:28.804+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:28.804219+0000) 2019-10-13T09:14:31.460 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:31.457+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.458781+0000) 2019-10-13T09:14:31.460 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:31.457+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.459285+0000) 2019-10-13T09:14:31.461 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:31.459+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.460381+0000) 2019-10-13T09:14:31.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:31.458+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.459781+0000) 2019-10-13T09:14:31.462 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:31.458+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.459979+0000) 2019-10-13T09:14:31.462 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:31.458+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.460128+0000) 2019-10-13T09:14:31.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:31.458+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.460276+0000) 2019-10-13T09:14:31.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:31.461+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.461878+0000) 2019-10-13T09:14:31.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:31.464+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:32.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:32.910+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:32.911126+0000) 2019-10-13T09:14:32.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:32.909+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:32.910607+0000) 2019-10-13T09:14:33.529 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:33.527+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:33.528662+0000) 2019-10-13T09:14:34.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:34.662+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:34.664291+0000) 2019-10-13T09:14:36.468 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:36.467+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.467763+0000) 2019-10-13T09:14:36.468 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:36.466+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.467607+0000) 2019-10-13T09:14:36.469 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:36.466+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.467880+0000) 2019-10-13T09:14:36.469 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:36.466+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.467911+0000) 2019-10-13T09:14:36.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:36.466+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.467997+0000) 2019-10-13T09:14:36.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:36.468+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.469690+0000) 2019-10-13T09:14:36.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:36.467+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.468569+0000) 2019-10-13T09:14:36.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:36.467+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.468658+0000) 2019-10-13T09:14:36.474 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:36.472+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:38.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:38.464+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:38.465439+0000) 2019-10-13T09:14:38.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:38.803+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:38.804415+0000) 2019-10-13T09:14:38.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:38.803+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:38.804418+0000) 2019-10-13T09:14:38.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:38.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:38.804420+0000) 2019-10-13T09:14:41.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:41.474+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.475555+0000) 2019-10-13T09:14:41.478 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:41.475+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.476531+0000) 2019-10-13T09:14:41.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:41.474+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.476244+0000) 2019-10-13T09:14:41.479 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:41.474+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.476388+0000) 2019-10-13T09:14:41.479 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:41.475+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.476469+0000) 2019-10-13T09:14:41.480 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:41.475+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.476734+0000) 2019-10-13T09:14:41.480 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:41.475+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.476950+0000) 2019-10-13T09:14:41.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:41.478+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.478926+0000) 2019-10-13T09:14:41.483 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:41.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161250 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:42.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:42.911+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:42.911281+0000) 2019-10-13T09:14:42.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:42.909+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:42.910812+0000) 2019-10-13T09:14:43.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:43.531+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:43.531691+0000) 2019-10-13T09:14:44.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:44.663+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:44.664472+0000) 2019-10-13T09:14:46.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:46.482+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.484388+0000) 2019-10-13T09:14:46.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:46.483+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.484750+0000) 2019-10-13T09:14:46.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:46.483+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.484792+0000) 2019-10-13T09:14:46.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:46.483+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.484922+0000) 2019-10-13T09:14:46.488 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:46.485+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.486065+0000) 2019-10-13T09:14:46.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:46.487+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.488050+0000) 2019-10-13T09:14:46.489 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:46.484+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.485830+0000) 2019-10-13T09:14:46.489 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:46.484+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.486033+0000) 2019-10-13T09:14:46.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:46.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:47.551 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:14:47.552 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:47.556 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:14:47.556 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:48.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:48.464+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:48.465602+0000) 2019-10-13T09:14:48.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:48.803+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:48.804601+0000) 2019-10-13T09:14:48.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:48.803+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:48.804605+0000) 2019-10-13T09:14:48.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:48.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:48.804603+0000) 2019-10-13T09:14:51.495 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:51.493+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.494201+0000) 2019-10-13T09:14:51.495 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:51.491+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.493086+0000) 2019-10-13T09:14:51.495 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:51.491+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.493310+0000) 2019-10-13T09:14:51.496 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:51.492+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.493533+0000) 2019-10-13T09:14:51.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:51.492+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.493682+0000) 2019-10-13T09:14:51.496 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:51.492+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.493832+0000) 2019-10-13T09:14:51.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:51.492+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.494049+0000) 2019-10-13T09:14:51.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:51.496+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.496875+0000) 2019-10-13T09:14:51.499 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:51.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161312 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:52.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:52.910+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:52.911457+0000) 2019-10-13T09:14:52.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:52.910+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:52.911004+0000) 2019-10-13T09:14:53.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:53.533+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:53.534181+0000) 2019-10-13T09:14:54.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:54.664+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:54.664634+0000) 2019-10-13T09:14:56.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:56.498+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.500003+0000) 2019-10-13T09:14:56.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:14:56.499+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.501045+0000) 2019-10-13T09:14:56.502 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:56.499+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.500608+0000) 2019-10-13T09:14:56.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:14:56.499+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.500825+0000) 2019-10-13T09:14:56.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:56.499+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.501193+0000) 2019-10-13T09:14:56.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:56.499+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.501180+0000) 2019-10-13T09:14:56.503 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:14:56.500+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.501551+0000) 2019-10-13T09:14:56.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:14:56.501+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.503655+0000) 2019-10-13T09:14:56.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:14:56.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:14:58.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:14:58.465+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:58.465786+0000) 2019-10-13T09:14:58.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:14:58.804+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:58.804801+0000) 2019-10-13T09:14:58.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:14:58.804+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:58.804804+0000) 2019-10-13T09:14:58.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:14:58.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:58.804803+0000) 2019-10-13T09:15:01.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:01.507+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.509344+0000) 2019-10-13T09:15:01.510 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:01.508+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.508875+0000) 2019-10-13T09:15:01.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:01.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.509171+0000) 2019-10-13T09:15:01.510 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:01.509+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.509478+0000) 2019-10-13T09:15:01.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:01.509+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.509514+0000) 2019-10-13T09:15:01.511 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:01.509+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.510260+0000) 2019-10-13T09:15:01.511 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:01.510+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.510403+0000) 2019-10-13T09:15:01.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:01.510+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.512485+0000) 2019-10-13T09:15:01.516 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:01.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:02.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:02.910+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:02.911661+0000) 2019-10-13T09:15:02.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:02.910+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:02.911219+0000) 2019-10-13T09:15:03.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:03.535+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:03.536661+0000) 2019-10-13T09:15:04.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:04.664+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:04.664827+0000) 2019-10-13T09:15:06.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:06.517+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.518025+0000) 2019-10-13T09:15:06.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:06.517+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.517562+0000) 2019-10-13T09:15:06.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:06.517+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.517916+0000) 2019-10-13T09:15:06.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:06.517+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.518020+0000) 2019-10-13T09:15:06.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:06.517+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.518358+0000) 2019-10-13T09:15:06.520 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:06.517+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.517817+0000) 2019-10-13T09:15:06.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:06.518+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.518645+0000) 2019-10-13T09:15:06.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:06.520+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.521287+0000) 2019-10-13T09:15:06.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:06.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:08.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:08.465+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:08.465952+0000) 2019-10-13T09:15:08.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:08.804+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:08.804973+0000) 2019-10-13T09:15:08.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:08.804+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:08.804978+0000) 2019-10-13T09:15:08.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:08.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:08.804972+0000) 2019-10-13T09:15:11.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:11.525+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.525952+0000) 2019-10-13T09:15:11.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:11.526+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.526949+0000) 2019-10-13T09:15:11.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:11.528+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.529265+0000) 2019-10-13T09:15:11.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:11.526+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.526507+0000) 2019-10-13T09:15:11.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:11.526+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.526627+0000) 2019-10-13T09:15:11.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:11.526+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.526849+0000) 2019-10-13T09:15:11.625 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:11.526+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.527118+0000) 2019-10-13T09:15:11.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:11.526+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.527336+0000) 2019-10-13T09:15:11.626 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:11.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:12.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:12.911+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:12.911420+0000) 2019-10-13T09:15:12.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:12.911+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:12.911845+0000) 2019-10-13T09:15:13.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:13.538+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:13.539199+0000) 2019-10-13T09:15:14.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:14.664+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:14.664992+0000) 2019-10-13T09:15:16.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:16.532+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.534081+0000) 2019-10-13T09:15:16.536 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:16.533+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.534574+0000) 2019-10-13T09:15:16.537 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:16.533+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.534771+0000) 2019-10-13T09:15:16.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:16.533+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.534979+0000) 2019-10-13T09:15:16.537 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:16.534+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.535479+0000) 2019-10-13T09:15:16.538 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:16.535+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.536602+0000) 2019-10-13T09:15:16.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:16.534+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.536289+0000) 2019-10-13T09:15:16.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:16.538+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.538860+0000) 2019-10-13T09:15:16.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:16.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161469 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:17.692 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:15:17.693 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:17.697 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:15:17.697 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:18.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:18.465+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:18.466114+0000) 2019-10-13T09:15:18.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:18.803+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:18.805202+0000) 2019-10-13T09:15:18.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:18.803+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:18.805197+0000) 2019-10-13T09:15:18.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:18.803+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:18.805199+0000) 2019-10-13T09:15:21.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:21.541+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.543084+0000) 2019-10-13T09:15:21.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:21.543+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.543976+0000) 2019-10-13T09:15:21.545 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:21.542+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.543576+0000) 2019-10-13T09:15:21.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:21.542+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.543716+0000) 2019-10-13T09:15:21.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:21.542+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.543887+0000) 2019-10-13T09:15:21.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:21.542+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.544246+0000) 2019-10-13T09:15:21.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:21.542+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.544387+0000) 2019-10-13T09:15:21.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:21.546+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.547108+0000) 2019-10-13T09:15:21.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:21.547+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:22.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:22.911+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:22.912053+0000) 2019-10-13T09:15:22.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:22.910+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:22.911635+0000) 2019-10-13T09:15:23.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:23.540+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:23.541392+0000) 2019-10-13T09:15:24.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:24.664+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:24.665181+0000) 2019-10-13T09:15:26.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:26.549+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.551244+0000) 2019-10-13T09:15:26.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:26.550+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.551503+0000) 2019-10-13T09:15:26.553 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:26.550+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.551759+0000) 2019-10-13T09:15:26.553 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:26.550+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.551980+0000) 2019-10-13T09:15:26.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:26.550+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.552159+0000) 2019-10-13T09:15:26.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:26.551+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.552647+0000) 2019-10-13T09:15:26.554 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:26.552+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.553153+0000) 2019-10-13T09:15:26.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:26.553+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.554075+0000) 2019-10-13T09:15:26.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:26.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161533 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:28.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:28.465+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:28.466311+0000) 2019-10-13T09:15:28.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:28.804+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:28.805387+0000) 2019-10-13T09:15:28.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:28.804+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:28.805417+0000) 2019-10-13T09:15:28.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:28.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:28.805417+0000) 2019-10-13T09:15:31.564 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:31.561+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.562052+0000) 2019-10-13T09:15:31.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:31.558+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.560239+0000) 2019-10-13T09:15:31.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:31.559+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.560700+0000) 2019-10-13T09:15:31.564 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:31.559+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.560906+0000) 2019-10-13T09:15:31.565 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:31.559+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.561048+0000) 2019-10-13T09:15:31.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:31.559+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.561143+0000) 2019-10-13T09:15:31.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:31.560+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.561635+0000) 2019-10-13T09:15:31.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:31.563+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.564093+0000) 2019-10-13T09:15:31.570 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:31.565+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:32.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:32.911+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:32.912209+0000) 2019-10-13T09:15:32.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:32.911+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:32.911789+0000) 2019-10-13T09:15:33.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:33.543+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:33.543786+0000) 2019-10-13T09:15:34.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:34.664+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:34.665376+0000) 2019-10-13T09:15:36.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:36.566+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.568250+0000) 2019-10-13T09:15:36.570 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:36.567+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.568535+0000) 2019-10-13T09:15:36.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:36.567+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.568720+0000) 2019-10-13T09:15:36.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:36.567+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.569214+0000) 2019-10-13T09:15:36.571 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:36.568+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.569670+0000) 2019-10-13T09:15:36.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:36.568+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.569457+0000) 2019-10-13T09:15:36.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:36.569+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.570594+0000) 2019-10-13T09:15:36.573 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:36.572+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.572995+0000) 2019-10-13T09:15:36.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:36.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:38.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:38.465+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:38.466481+0000) 2019-10-13T09:15:38.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:38.805+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:38.805584+0000) 2019-10-13T09:15:38.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:38.805+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:38.805605+0000) 2019-10-13T09:15:38.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:38.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:38.805606+0000) 2019-10-13T09:15:41.581 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:41.578+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.578967+0000) 2019-10-13T09:15:41.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:41.575+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.577235+0000) 2019-10-13T09:15:41.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:41.575+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.577376+0000) 2019-10-13T09:15:41.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:41.576+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.577655+0000) 2019-10-13T09:15:41.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:41.576+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.578010+0000) 2019-10-13T09:15:41.583 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:41.576+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.578340+0000) 2019-10-13T09:15:41.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:41.577+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.578819+0000) 2019-10-13T09:15:41.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:41.580+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.581467+0000) 2019-10-13T09:15:41.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:41.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161628 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:42.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:42.912+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:42.912365+0000) 2019-10-13T09:15:42.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:42.911+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:42.911912+0000) 2019-10-13T09:15:43.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:43.546+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:43.546304+0000) 2019-10-13T09:15:44.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:44.665+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:44.665591+0000) 2019-10-13T09:15:46.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:46.584+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.584977+0000) 2019-10-13T09:15:46.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:46.585+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.585947+0000) 2019-10-13T09:15:46.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:46.585+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.587032+0000) 2019-10-13T09:15:46.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:46.586+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.586407+0000) 2019-10-13T09:15:46.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:46.586+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.586850+0000) 2019-10-13T09:15:46.588 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:46.586+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.586970+0000) 2019-10-13T09:15:46.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:46.586+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.587106+0000) 2019-10-13T09:15:46.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:46.587+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.589142+0000) 2019-10-13T09:15:46.591 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:46.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:47.864 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:15:47.864 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:47.869 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:15:47.870 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:48.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:48.466+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:48.466641+0000) 2019-10-13T09:15:48.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:48.805+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:48.805750+0000) 2019-10-13T09:15:48.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:48.805+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:48.805764+0000) 2019-10-13T09:15:48.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:48.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:48.805764+0000) 2019-10-13T09:15:51.593 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:51.591+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.592311+0000) 2019-10-13T09:15:51.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:51.592+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.592462+0000) 2019-10-13T09:15:51.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:51.592+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.592652+0000) 2019-10-13T09:15:51.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:51.591+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.593663+0000) 2019-10-13T09:15:51.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:51.592+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.593026+0000) 2019-10-13T09:15:51.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:51.592+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.593356+0000) 2019-10-13T09:15:51.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:51.593+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.593595+0000) 2019-10-13T09:15:51.597 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:51.594+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.596412+0000) 2019-10-13T09:15:51.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:51.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:52.912 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:52.911+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:52.912531+0000) 2019-10-13T09:15:52.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:52.911+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:52.912107+0000) 2019-10-13T09:15:53.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:53.547+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:53.548718+0000) 2019-10-13T09:15:54.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:54.665+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:54.665814+0000) 2019-10-13T09:15:56.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:15:56.598+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.600221+0000) 2019-10-13T09:15:56.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:56.599+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.599666+0000) 2019-10-13T09:15:56.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:56.599+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.599752+0000) 2019-10-13T09:15:56.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:56.599+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.600052+0000) 2019-10-13T09:15:56.602 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:15:56.599+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.600108+0000) 2019-10-13T09:15:56.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:56.600+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.600608+0000) 2019-10-13T09:15:56.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:15:56.600+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.600830+0000) 2019-10-13T09:15:56.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:15:56.601+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.603106+0000) 2019-10-13T09:15:56.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:15:56.607+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:15:58.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:15:58.466+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:58.466839+0000) 2019-10-13T09:15:58.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:15:58.805+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:58.805947+0000) 2019-10-13T09:15:58.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:15:58.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:58.805973+0000) 2019-10-13T09:15:58.807 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:15:58.805+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:58.805981+0000) 2019-10-13T09:16:01.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:01.608+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.608480+0000) 2019-10-13T09:16:01.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:01.608+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.608690+0000) 2019-10-13T09:16:01.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:01.608+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.608876+0000) 2019-10-13T09:16:01.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:01.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.609951+0000) 2019-10-13T09:16:01.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:01.608+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.609330+0000) 2019-10-13T09:16:01.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:01.609+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.609525+0000) 2019-10-13T09:16:01.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:01.609+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.610025+0000) 2019-10-13T09:16:01.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:01.611+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.612601+0000) 2019-10-13T09:16:01.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:01.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161754 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:02.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:02.912+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:02.912699+0000) 2019-10-13T09:16:02.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:02.911+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:02.912242+0000) 2019-10-13T09:16:03.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:03.550+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:03.551219+0000) 2019-10-13T09:16:04.667 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:04.664+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:04.666027+0000) 2019-10-13T09:16:06.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:06.614+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.615710+0000) 2019-10-13T09:16:06.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:06.614+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.615981+0000) 2019-10-13T09:16:06.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:06.615+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.616553+0000) 2019-10-13T09:16:06.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:06.615+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.616422+0000) 2019-10-13T09:16:06.618 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:06.615+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.616666+0000) 2019-10-13T09:16:06.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:06.615+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.616796+0000) 2019-10-13T09:16:06.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:06.615+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.617052+0000) 2019-10-13T09:16:06.619 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:06.619+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.619882+0000) 2019-10-13T09:16:06.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:06.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161785 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:08.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:08.466+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:08.467062+0000) 2019-10-13T09:16:08.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:08.805+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:08.806168+0000) 2019-10-13T09:16:08.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:08.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:08.806174+0000) 2019-10-13T09:16:08.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:08.805+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:08.806185+0000) 2019-10-13T09:16:11.625 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:11.624+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.625702+0000) 2019-10-13T09:16:11.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:11.623+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.625200+0000) 2019-10-13T09:16:11.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:11.624+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.625390+0000) 2019-10-13T09:16:11.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:11.624+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.625564+0000) 2019-10-13T09:16:11.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:11.624+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.625763+0000) 2019-10-13T09:16:11.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:11.624+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.625936+0000) 2019-10-13T09:16:11.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:11.624+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.626180+0000) 2019-10-13T09:16:11.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:11.628+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.629037+0000) 2019-10-13T09:16:11.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:11.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:12.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:12.912+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:12.912873+0000) 2019-10-13T09:16:12.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:12.911+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:12.912414+0000) 2019-10-13T09:16:13.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:13.553+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:13.553522+0000) 2019-10-13T09:16:14.667 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:14.665+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:14.666227+0000) 2019-10-13T09:16:16.634 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:16.631+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.633020+0000) 2019-10-13T09:16:16.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:16.632+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.633417+0000) 2019-10-13T09:16:16.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:16.631+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.633306+0000) 2019-10-13T09:16:16.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:16.633+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.634303+0000) 2019-10-13T09:16:16.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:16.632+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.634219+0000) 2019-10-13T09:16:16.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:16.632+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.634243+0000) 2019-10-13T09:16:16.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:16.632+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.634372+0000) 2019-10-13T09:16:16.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:16.635+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.636637+0000) 2019-10-13T09:16:16.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:16.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:17.968 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:16:17.968 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:17.973 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:16:17.973 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:18.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:18.466+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:18.467258+0000) 2019-10-13T09:16:18.808 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:18.804+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:18.806355+0000) 2019-10-13T09:16:18.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:18.804+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:18.806362+0000) 2019-10-13T09:16:18.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:18.804+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:18.806357+0000) 2019-10-13T09:16:21.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:21.642+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.642957+0000) 2019-10-13T09:16:21.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:21.641+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.642444+0000) 2019-10-13T09:16:21.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:21.641+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.642621+0000) 2019-10-13T09:16:21.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:21.641+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.643230+0000) 2019-10-13T09:16:21.644 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:21.641+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.643323+0000) 2019-10-13T09:16:21.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:21.642+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.643592+0000) 2019-10-13T09:16:21.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:21.642+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.643793+0000) 2019-10-13T09:16:21.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:21.645+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.646002+0000) 2019-10-13T09:16:21.649 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:21.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161880 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:22.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:22.912+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:22.913041+0000) 2019-10-13T09:16:22.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:22.911+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:22.912606+0000) 2019-10-13T09:16:23.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:23.555+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:23.556000+0000) 2019-10-13T09:16:24.667 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:24.665+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:24.666446+0000) 2019-10-13T09:16:26.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:26.649+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.650578+0000) 2019-10-13T09:16:26.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:26.649+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.650944+0000) 2019-10-13T09:16:26.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:26.649+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.651085+0000) 2019-10-13T09:16:26.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:26.650+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.651564+0000) 2019-10-13T09:16:26.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:26.651+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.652341+0000) 2019-10-13T09:16:26.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:26.650+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.652017+0000) 2019-10-13T09:16:26.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:26.650+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.652181+0000) 2019-10-13T09:16:26.654 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:26.653+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.654556+0000) 2019-10-13T09:16:26.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:26.655+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:28.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:28.466+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:28.467486+0000) 2019-10-13T09:16:28.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:28.806+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:28.806594+0000) 2019-10-13T09:16:28.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:28.806+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:28.806582+0000) 2019-10-13T09:16:28.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:28.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:28.806601+0000) 2019-10-13T09:16:31.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:31.659+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.659734+0000) 2019-10-13T09:16:31.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:31.658+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.658864+0000) 2019-10-13T09:16:31.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:31.658+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.659113+0000) 2019-10-13T09:16:31.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:31.658+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.659374+0000) 2019-10-13T09:16:31.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:31.659+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.659854+0000) 2019-10-13T09:16:31.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:31.659+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.659961+0000) 2019-10-13T09:16:31.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:31.659+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.660136+0000) 2019-10-13T09:16:31.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:31.661+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.662421+0000) 2019-10-13T09:16:31.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:31.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161942 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:32.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:32.913+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:32.913216+0000) 2019-10-13T09:16:32.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:32.912+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:32.912804+0000) 2019-10-13T09:16:33.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:33.558+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:33.558401+0000) 2019-10-13T09:16:34.667 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:34.666+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:34.666647+0000) 2019-10-13T09:16:36.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:36.666+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.667033+0000) 2019-10-13T09:16:36.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:36.666+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.667120+0000) 2019-10-13T09:16:36.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:36.667+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.667433+0000) 2019-10-13T09:16:36.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:36.667+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.667626+0000) 2019-10-13T09:16:36.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:36.667+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.667831+0000) 2019-10-13T09:16:36.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:36.667+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.667893+0000) 2019-10-13T09:16:36.670 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:36.666+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.668044+0000) 2019-10-13T09:16:36.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:36.668+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.669745+0000) 2019-10-13T09:16:36.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:36.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 161973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:38.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:38.467+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:38.467683+0000) 2019-10-13T09:16:38.808 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:38.806+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:38.806827+0000) 2019-10-13T09:16:38.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:38.806+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:38.806828+0000) 2019-10-13T09:16:38.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:38.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:38.806827+0000) 2019-10-13T09:16:41.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:41.674+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.676421+0000) 2019-10-13T09:16:41.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:41.675+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.676153+0000) 2019-10-13T09:16:41.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:41.675+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.676069+0000) 2019-10-13T09:16:41.678 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:41.675+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.676347+0000) 2019-10-13T09:16:41.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:41.676+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.676496+0000) 2019-10-13T09:16:41.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:41.676+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.676668+0000) 2019-10-13T09:16:41.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:41.676+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.676874+0000) 2019-10-13T09:16:41.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:41.676+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.678705+0000) 2019-10-13T09:16:41.683 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:41.681+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:42.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:42.912+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:42.913429+0000) 2019-10-13T09:16:42.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:42.912+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:42.912976+0000) 2019-10-13T09:16:43.561 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:43.559+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:43.560720+0000) 2019-10-13T09:16:44.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:44.666+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:44.666841+0000) 2019-10-13T09:16:46.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:46.683+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.684052+0000) 2019-10-13T09:16:46.685 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:46.683+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.684019+0000) 2019-10-13T09:16:46.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:46.683+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.684345+0000) 2019-10-13T09:16:46.686 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:46.683+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.684981+0000) 2019-10-13T09:16:46.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:46.684+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.684666+0000) 2019-10-13T09:16:46.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:46.684+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.684703+0000) 2019-10-13T09:16:46.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:46.684+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.684921+0000) 2019-10-13T09:16:46.688 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:46.686+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.687978+0000) 2019-10-13T09:16:46.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:46.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:48.093 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:16:48.093 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:48.097 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:16:48.098 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:48.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:48.466+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:48.467924+0000) 2019-10-13T09:16:48.808 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:48.805+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:48.807030+0000) 2019-10-13T09:16:48.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:48.805+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:48.806987+0000) 2019-10-13T09:16:48.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:48.805+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:48.807022+0000) 2019-10-13T09:16:51.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:51.690+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.691897+0000) 2019-10-13T09:16:51.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:51.690+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.692200+0000) 2019-10-13T09:16:51.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:51.690+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.692318+0000) 2019-10-13T09:16:51.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:51.692+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.693393+0000) 2019-10-13T09:16:51.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:51.691+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.692938+0000) 2019-10-13T09:16:51.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:51.691+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.692897+0000) 2019-10-13T09:16:51.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:51.691+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.693274+0000) 2019-10-13T09:16:51.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:51.694+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.695319+0000) 2019-10-13T09:16:51.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:51.697+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:52.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:52.912+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:52.913617+0000) 2019-10-13T09:16:52.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:52.912+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:52.913162+0000) 2019-10-13T09:16:53.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:53.562+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:53.563379+0000) 2019-10-13T09:16:54.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:54.666+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:54.667045+0000) 2019-10-13T09:16:56.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:16:56.700+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.701477+0000) 2019-10-13T09:16:56.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:56.699+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.700635+0000) 2019-10-13T09:16:56.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:56.699+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.700854+0000) 2019-10-13T09:16:56.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:16:56.699+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.700962+0000) 2019-10-13T09:16:56.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:56.699+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.701103+0000) 2019-10-13T09:16:56.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:56.700+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.701459+0000) 2019-10-13T09:16:56.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:16:56.699+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.701359+0000) 2019-10-13T09:16:56.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:16:56.703+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.704124+0000) 2019-10-13T09:16:56.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:16:56.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162100 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:16:58.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:16:58.467+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:58.468157+0000) 2019-10-13T09:16:58.808 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:16:58.806+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:58.807170+0000) 2019-10-13T09:16:58.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:16:58.806+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:58.807164+0000) 2019-10-13T09:16:58.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:16:58.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:58.807169+0000) 2019-10-13T09:17:01.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:01.708+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.709165+0000) 2019-10-13T09:17:01.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:01.707+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.708576+0000) 2019-10-13T09:17:01.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:01.707+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.708734+0000) 2019-10-13T09:17:01.711 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:01.707+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.708879+0000) 2019-10-13T09:17:01.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:01.707+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.709266+0000) 2019-10-13T09:17:01.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:01.708+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.709508+0000) 2019-10-13T09:17:01.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:01.708+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.709605+0000) 2019-10-13T09:17:01.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:01.711+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.711957+0000) 2019-10-13T09:17:01.715 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:01.713+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162132 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:02.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:02.913+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:02.913805+0000) 2019-10-13T09:17:02.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:02.912+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:02.913359+0000) 2019-10-13T09:17:03.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:03.565+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:03.565910+0000) 2019-10-13T09:17:04.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:04.666+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:04.667259+0000) 2019-10-13T09:17:06.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:06.716+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.716937+0000) 2019-10-13T09:17:06.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:06.715+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.716510+0000) 2019-10-13T09:17:06.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:06.715+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.716693+0000) 2019-10-13T09:17:06.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:06.715+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.717011+0000) 2019-10-13T09:17:06.719 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:06.715+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.717184+0000) 2019-10-13T09:17:06.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:06.715+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.717360+0000) 2019-10-13T09:17:06.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:06.716+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.717543+0000) 2019-10-13T09:17:06.720 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:06.719+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.720314+0000) 2019-10-13T09:17:06.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:06.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:08.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:08.467+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:08.468308+0000) 2019-10-13T09:17:08.808 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:08.806+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:08.807345+0000) 2019-10-13T09:17:08.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:08.806+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:08.807343+0000) 2019-10-13T09:17:08.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:08.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:08.807341+0000) 2019-10-13T09:17:11.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:11.723+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.725163+0000) 2019-10-13T09:17:11.726 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:11.723+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.725313+0000) 2019-10-13T09:17:11.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:11.725+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.726309+0000) 2019-10-13T09:17:11.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:11.724+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.725731+0000) 2019-10-13T09:17:11.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:11.724+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.726102+0000) 2019-10-13T09:17:11.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:11.725+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.726477+0000) 2019-10-13T09:17:11.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:11.725+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.726642+0000) 2019-10-13T09:17:11.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:11.728+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.728959+0000) 2019-10-13T09:17:11.733 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:11.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162194 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:12.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:12.913+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:12.913986+0000) 2019-10-13T09:17:12.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:12.912+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:12.913516+0000) 2019-10-13T09:17:13.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:13.568+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:13.568908+0000) 2019-10-13T09:17:14.668 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:14.666+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:14.667442+0000) 2019-10-13T09:17:16.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:16.734+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.734520+0000) 2019-10-13T09:17:16.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:16.734+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.734894+0000) 2019-10-13T09:17:16.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:16.734+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.735142+0000) 2019-10-13T09:17:16.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:16.734+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.735724+0000) 2019-10-13T09:17:16.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:16.734+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.735319+0000) 2019-10-13T09:17:16.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:16.735+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.735544+0000) 2019-10-13T09:17:16.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:16.735+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.735700+0000) 2019-10-13T09:17:16.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:16.736+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.737679+0000) 2019-10-13T09:17:16.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:16.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:18.227 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:17:18.228 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:18.232 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:17:18.232 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:18.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:18.468+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:18.468558+0000) 2019-10-13T09:17:18.808 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:18.807+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:18.807563+0000) 2019-10-13T09:17:18.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:18.807+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:18.807569+0000) 2019-10-13T09:17:18.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:18.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:18.807569+0000) 2019-10-13T09:17:21.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:21.742+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.742824+0000) 2019-10-13T09:17:21.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:21.742+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.743076+0000) 2019-10-13T09:17:21.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:21.742+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.742909+0000) 2019-10-13T09:17:21.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:21.743+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.743445+0000) 2019-10-13T09:17:21.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:21.743+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.744023+0000) 2019-10-13T09:17:21.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:21.743+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.743751+0000) 2019-10-13T09:17:21.746 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:21.744+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.744856+0000) 2019-10-13T09:17:21.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:21.745+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.746619+0000) 2019-10-13T09:17:21.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:21.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:22.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:22.913+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:22.914153+0000) 2019-10-13T09:17:22.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:22.913+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:22.913614+0000) 2019-10-13T09:17:23.571 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:23.571+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:23.571294+0000) 2019-10-13T09:17:24.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:24.667+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:24.667693+0000) 2019-10-13T09:17:26.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:26.749+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.750324+0000) 2019-10-13T09:17:26.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:26.749+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.749970+0000) 2019-10-13T09:17:26.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:26.749+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.750188+0000) 2019-10-13T09:17:26.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:26.750+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.750600+0000) 2019-10-13T09:17:26.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:26.750+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.750574+0000) 2019-10-13T09:17:26.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:26.750+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.750868+0000) 2019-10-13T09:17:26.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:26.750+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.750826+0000) 2019-10-13T09:17:26.761 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:26.755+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:26.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:26.753+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.754240+0000) 2019-10-13T09:17:28.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:28.468+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:28.468765+0000) 2019-10-13T09:17:28.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:28.807+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:28.807723+0000) 2019-10-13T09:17:28.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:28.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:28.807723+0000) 2019-10-13T09:17:28.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:28.807+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:28.807738+0000) 2019-10-13T09:17:31.758 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:31.757+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.757658+0000) 2019-10-13T09:17:31.759 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:31.757+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.757917+0000) 2019-10-13T09:17:31.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:31.757+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.758031+0000) 2019-10-13T09:17:31.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:31.757+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.759656+0000) 2019-10-13T09:17:31.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:31.757+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.758237+0000) 2019-10-13T09:17:31.760 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:31.758+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.758968+0000) 2019-10-13T09:17:31.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:31.758+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.759165+0000) 2019-10-13T09:17:31.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:31.759+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.761539+0000) 2019-10-13T09:17:31.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:31.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162320 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:32.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:32.913+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:32.913789+0000) 2019-10-13T09:17:32.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:32.913+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:32.914296+0000) 2019-10-13T09:17:33.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:33.572+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:33.573874+0000) 2019-10-13T09:17:34.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:34.666+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:34.667906+0000) 2019-10-13T09:17:36.767 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:36.764+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.765699+0000) 2019-10-13T09:17:36.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:36.764+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.765990+0000) 2019-10-13T09:17:36.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:36.764+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.766050+0000) 2019-10-13T09:17:36.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:36.765+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.766495+0000) 2019-10-13T09:17:36.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:36.765+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.766644+0000) 2019-10-13T09:17:36.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:36.765+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.767633+0000) 2019-10-13T09:17:36.769 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:36.765+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.766940+0000) 2019-10-13T09:17:36.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:36.768+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.769974+0000) 2019-10-13T09:17:36.773 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:36.770+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162353 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:38.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:38.468+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:38.468957+0000) 2019-10-13T09:17:38.809 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:38.806+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:38.807898+0000) 2019-10-13T09:17:38.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:38.806+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:38.807897+0000) 2019-10-13T09:17:38.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:38.806+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:38.807897+0000) 2019-10-13T09:17:41.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:41.772+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.774138+0000) 2019-10-13T09:17:41.775 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:41.772+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.773972+0000) 2019-10-13T09:17:41.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:41.772+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.774338+0000) 2019-10-13T09:17:41.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:41.773+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.774770+0000) 2019-10-13T09:17:41.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:41.774+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.775545+0000) 2019-10-13T09:17:41.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:41.773+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.775092+0000) 2019-10-13T09:17:41.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:41.773+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.775278+0000) 2019-10-13T09:17:41.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:41.776+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.777274+0000) 2019-10-13T09:17:41.781 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:41.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:42.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:42.913+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:42.914499+0000) 2019-10-13T09:17:42.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:42.913+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:42.913970+0000) 2019-10-13T09:17:43.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:43.575+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:43.576351+0000) 2019-10-13T09:17:44.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:44.667+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:44.668085+0000) 2019-10-13T09:17:46.783 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:46.780+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.782314+0000) 2019-10-13T09:17:46.784 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:46.781+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.782747+0000) 2019-10-13T09:17:46.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:46.781+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.782883+0000) 2019-10-13T09:17:46.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:46.781+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.782789+0000) 2019-10-13T09:17:46.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:46.783+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.784433+0000) 2019-10-13T09:17:46.785 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:46.781+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.783076+0000) 2019-10-13T09:17:46.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:46.782+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.783469+0000) 2019-10-13T09:17:46.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:46.785+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.786095+0000) 2019-10-13T09:17:46.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:46.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162417 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:48.314 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:17:48.314 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:48.319 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:17:48.319 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:48.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:48.467+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:48.469152+0000) 2019-10-13T09:17:48.809 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:48.807+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:48.808094+0000) 2019-10-13T09:17:48.809 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:48.807+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:48.808106+0000) 2019-10-13T09:17:48.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:48.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:48.808106+0000) 2019-10-13T09:17:51.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:51.789+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.790523+0000) 2019-10-13T09:17:51.792 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:51.789+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.790656+0000) 2019-10-13T09:17:51.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:51.789+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.791299+0000) 2019-10-13T09:17:51.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:51.791+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.791911+0000) 2019-10-13T09:17:51.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:51.790+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.791618+0000) 2019-10-13T09:17:51.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:51.790+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.792028+0000) 2019-10-13T09:17:51.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:51.790+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.792226+0000) 2019-10-13T09:17:51.796 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:51.794+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.795039+0000) 2019-10-13T09:17:51.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:51.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:52.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:52.913+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:52.914665+0000) 2019-10-13T09:17:52.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:52.913+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:52.914145+0000) 2019-10-13T09:17:53.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:53.578+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:53.578647+0000) 2019-10-13T09:17:54.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:54.667+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:54.668252+0000) 2019-10-13T09:17:56.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:56.798+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.800252+0000) 2019-10-13T09:17:56.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:17:56.799+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.800518+0000) 2019-10-13T09:17:56.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:56.799+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.800841+0000) 2019-10-13T09:17:56.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:56.799+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.800923+0000) 2019-10-13T09:17:56.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:17:56.800+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.801591+0000) 2019-10-13T09:17:56.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:17:56.799+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.801214+0000) 2019-10-13T09:17:56.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:56.799+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.801266+0000) 2019-10-13T09:17:56.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:17:56.803+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.804449+0000) 2019-10-13T09:17:56.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:17:56.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162479 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:17:58.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:17:58.468+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:58.469346+0000) 2019-10-13T09:17:58.809 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:17:58.807+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:58.808316+0000) 2019-10-13T09:17:58.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:17:58.807+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:58.808318+0000) 2019-10-13T09:17:58.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:17:58.807+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:58.808318+0000) 2019-10-13T09:18:01.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:01.808+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.809547+0000) 2019-10-13T09:18:01.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:01.808+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.809028+0000) 2019-10-13T09:18:01.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:01.808+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.809299+0000) 2019-10-13T09:18:01.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:01.809+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.809454+0000) 2019-10-13T09:18:01.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:01.809+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.809900+0000) 2019-10-13T09:18:01.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:01.809+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.810028+0000) 2019-10-13T09:18:01.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:01.809+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.810164+0000) 2019-10-13T09:18:01.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:01.810+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.811445+0000) 2019-10-13T09:18:01.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:01.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:02.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:02.914+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:02.914844+0000) 2019-10-13T09:18:02.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:02.913+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:02.914335+0000) 2019-10-13T09:18:03.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:03.580+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:03.580973+0000) 2019-10-13T09:18:04.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:04.668+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:04.668417+0000) 2019-10-13T09:18:06.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:06.817+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.817425+0000) 2019-10-13T09:18:06.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:06.817+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.817612+0000) 2019-10-13T09:18:06.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:06.817+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.817767+0000) 2019-10-13T09:18:06.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:06.818+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.819454+0000) 2019-10-13T09:18:06.821 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:06.817+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.818141+0000) 2019-10-13T09:18:06.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:06.817+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.818378+0000) 2019-10-13T09:18:06.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:06.818+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.818585+0000) 2019-10-13T09:18:06.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:06.821+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.822021+0000) 2019-10-13T09:18:06.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:06.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:08.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:08.469+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:08.469526+0000) 2019-10-13T09:18:08.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:08.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:08.808536+0000) 2019-10-13T09:18:08.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:08.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:08.808549+0000) 2019-10-13T09:18:08.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:08.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:08.808549+0000) 2019-10-13T09:18:11.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:11.826+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.826908+0000) 2019-10-13T09:18:11.828 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:11.825+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.825994+0000) 2019-10-13T09:18:11.828 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:11.826+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.826416+0000) 2019-10-13T09:18:11.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:11.826+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.826552+0000) 2019-10-13T09:18:11.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:11.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.826881+0000) 2019-10-13T09:18:11.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:11.826+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.827055+0000) 2019-10-13T09:18:11.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:11.827+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.827632+0000) 2019-10-13T09:18:11.830 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:11.828+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.829519+0000) 2019-10-13T09:18:11.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:11.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162572 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:12.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:12.914+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:12.915026+0000) 2019-10-13T09:18:12.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:12.914+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:12.914517+0000) 2019-10-13T09:18:13.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:13.583+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:13.583336+0000) 2019-10-13T09:18:14.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:14.668+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:14.668601+0000) 2019-10-13T09:18:16.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:16.833+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.834198+0000) 2019-10-13T09:18:16.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:16.833+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.834321+0000) 2019-10-13T09:18:16.836 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:16.834+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.834428+0000) 2019-10-13T09:18:16.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:16.834+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.834610+0000) 2019-10-13T09:18:16.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:16.834+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.834858+0000) 2019-10-13T09:18:16.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:16.834+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.835673+0000) 2019-10-13T09:18:16.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:16.834+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.835316+0000) 2019-10-13T09:18:16.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:16.837+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.838371+0000) 2019-10-13T09:18:16.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:16.839+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:18.428 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:18:18.429 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:18.433 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:18:18.434 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:18.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:18.469+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:18.469654+0000) 2019-10-13T09:18:18.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:18.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:18.808746+0000) 2019-10-13T09:18:18.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:18.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:18.808754+0000) 2019-10-13T09:18:18.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:18.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:18.808755+0000) 2019-10-13T09:18:21.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:21.840+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.842056+0000) 2019-10-13T09:18:21.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:21.841+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.842484+0000) 2019-10-13T09:18:21.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:21.841+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.842496+0000) 2019-10-13T09:18:21.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:21.841+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.842724+0000) 2019-10-13T09:18:21.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:21.841+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.843233+0000) 2019-10-13T09:18:21.847 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:21.842+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.843488+0000) 2019-10-13T09:18:21.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:21.841+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.843513+0000) 2019-10-13T09:18:21.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:21.844+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.845890+0000) 2019-10-13T09:18:21.850 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:21.847+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:22.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:22.914+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:22.915231+0000) 2019-10-13T09:18:22.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:22.913+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:22.914687+0000) 2019-10-13T09:18:23.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:23.584+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:23.585801+0000) 2019-10-13T09:18:24.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:24.667+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:24.668797+0000) 2019-10-13T09:18:26.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:26.849+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.851044+0000) 2019-10-13T09:18:26.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:26.850+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.851547+0000) 2019-10-13T09:18:26.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:26.850+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.851539+0000) 2019-10-13T09:18:26.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:26.851+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.852812+0000) 2019-10-13T09:18:26.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:26.852+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.854013+0000) 2019-10-13T09:18:26.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:26.850+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.851897+0000) 2019-10-13T09:18:26.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:26.850+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.851767+0000) 2019-10-13T09:18:26.869 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:26.850+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.852274+0000) 2019-10-13T09:18:26.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:26.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:28.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:28.469+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:28.469862+0000) 2019-10-13T09:18:28.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:28.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:28.808957+0000) 2019-10-13T09:18:28.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:28.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:28.808986+0000) 2019-10-13T09:18:28.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:28.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:28.808985+0000) 2019-10-13T09:18:31.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:31.857+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.858878+0000) 2019-10-13T09:18:31.860 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:31.857+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.859123+0000) 2019-10-13T09:18:31.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:31.857+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.859361+0000) 2019-10-13T09:18:31.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:31.858+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.859595+0000) 2019-10-13T09:18:31.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:31.858+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.859496+0000) 2019-10-13T09:18:31.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:31.858+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.860051+0000) 2019-10-13T09:18:31.862 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:31.858+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.859822+0000) 2019-10-13T09:18:31.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:31.861+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.862346+0000) 2019-10-13T09:18:31.865 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:31.862+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:32.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:32.914+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:32.915417+0000) 2019-10-13T09:18:32.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:32.914+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:32.914872+0000) 2019-10-13T09:18:33.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:33.587+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:33.588344+0000) 2019-10-13T09:18:34.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:34.668+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:34.668991+0000) 2019-10-13T09:18:36.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:36.864+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.866245+0000) 2019-10-13T09:18:36.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:36.864+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.866298+0000) 2019-10-13T09:18:36.869 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:36.865+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.866520+0000) 2019-10-13T09:18:36.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:36.865+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.866675+0000) 2019-10-13T09:18:36.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:36.865+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.866753+0000) 2019-10-13T09:18:36.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:36.866+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.867444+0000) 2019-10-13T09:18:36.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:36.866+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.867685+0000) 2019-10-13T09:18:36.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:36.870+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.870799+0000) 2019-10-13T09:18:36.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:36.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:38.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:38.469+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:38.470040+0000) 2019-10-13T09:18:38.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:38.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:38.809147+0000) 2019-10-13T09:18:38.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:38.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:38.809172+0000) 2019-10-13T09:18:38.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:38.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:38.809172+0000) 2019-10-13T09:18:41.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:41.873+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.874451+0000) 2019-10-13T09:18:41.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:41.873+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.874665+0000) 2019-10-13T09:18:41.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:41.874+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.875622+0000) 2019-10-13T09:18:41.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:41.873+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.875213+0000) 2019-10-13T09:18:41.877 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:41.873+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.875308+0000) 2019-10-13T09:18:41.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:41.874+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.875487+0000) 2019-10-13T09:18:41.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:41.874+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.875767+0000) 2019-10-13T09:18:41.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:41.877+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.877830+0000) 2019-10-13T09:18:41.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:41.879+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:42.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:42.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:42.915576+0000) 2019-10-13T09:18:42.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:42.914+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:42.915048+0000) 2019-10-13T09:18:43.591 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:43.590+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:43.590841+0000) 2019-10-13T09:18:44.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:44.668+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:44.669186+0000) 2019-10-13T09:18:46.883 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:46.881+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.882455+0000) 2019-10-13T09:18:46.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:46.881+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.882687+0000) 2019-10-13T09:18:46.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:46.881+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.882848+0000) 2019-10-13T09:18:46.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:46.883+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.883762+0000) 2019-10-13T09:18:46.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:46.882+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.883621+0000) 2019-10-13T09:18:46.885 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:46.882+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.883875+0000) 2019-10-13T09:18:46.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:46.882+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.883859+0000) 2019-10-13T09:18:46.886 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:46.885+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.886020+0000) 2019-10-13T09:18:46.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:46.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:48.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:48.469+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:48.470249+0000) 2019-10-13T09:18:48.554 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:18:48.554 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:48.559 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:18:48.559 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:48.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:48.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:48.809301+0000) 2019-10-13T09:18:48.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:48.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:48.809336+0000) 2019-10-13T09:18:48.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:48.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:48.809335+0000) 2019-10-13T09:18:51.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:51.890+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.891308+0000) 2019-10-13T09:18:51.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:51.891+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.891613+0000) 2019-10-13T09:18:51.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:51.891+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.891880+0000) 2019-10-13T09:18:51.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:51.891+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.891932+0000) 2019-10-13T09:18:51.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:51.891+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.892206+0000) 2019-10-13T09:18:51.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:51.891+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.892283+0000) 2019-10-13T09:18:51.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:51.891+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.892331+0000) 2019-10-13T09:18:51.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:51.894+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.894965+0000) 2019-10-13T09:18:51.905 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:51.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:52.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:52.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:52.915718+0000) 2019-10-13T09:18:52.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:52.914+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:52.915197+0000) 2019-10-13T09:18:53.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:53.592+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:53.593339+0000) 2019-10-13T09:18:54.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:54.669+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:54.669422+0000) 2019-10-13T09:18:56.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:18:56.898+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.899135+0000) 2019-10-13T09:18:56.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:56.898+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.899304+0000) 2019-10-13T09:18:56.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:56.899+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.899444+0000) 2019-10-13T09:18:56.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:18:56.899+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.899633+0000) 2019-10-13T09:18:56.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:18:56.900+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.900804+0000) 2019-10-13T09:18:56.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:56.899+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.899788+0000) 2019-10-13T09:18:56.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:56.899+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.900358+0000) 2019-10-13T09:18:56.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:18:56.902+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.902758+0000) 2019-10-13T09:18:56.905 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:18:56.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:18:58.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:18:58.470+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:58.470454+0000) 2019-10-13T09:18:58.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:18:58.809+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:58.809473+0000) 2019-10-13T09:18:58.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:18:58.809+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:58.809492+0000) 2019-10-13T09:18:58.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:18:58.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:58.809492+0000) 2019-10-13T09:19:01.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:01.905+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.905651+0000) 2019-10-13T09:19:01.907 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:01.905+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.905812+0000) 2019-10-13T09:19:01.907 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:01.905+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.906070+0000) 2019-10-13T09:19:01.908 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:01.906+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.906739+0000) 2019-10-13T09:19:01.908 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:01.906+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.906625+0000) 2019-10-13T09:19:01.908 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:01.906+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.907087+0000) 2019-10-13T09:19:01.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:01.906+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.907292+0000) 2019-10-13T09:19:01.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:01.908+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.909164+0000) 2019-10-13T09:19:01.913 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:01.912+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:02.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:02.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:02.915835+0000) 2019-10-13T09:19:02.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:02.915+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:02.915378+0000) 2019-10-13T09:19:03.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:03.595+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:03.595946+0000) 2019-10-13T09:19:04.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:04.669+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:04.669610+0000) 2019-10-13T09:19:06.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:06.912+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.914329+0000) 2019-10-13T09:19:06.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:06.913+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.914416+0000) 2019-10-13T09:19:06.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:06.915+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.915893+0000) 2019-10-13T09:19:06.917 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:06.913+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.915154+0000) 2019-10-13T09:19:06.917 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:06.913+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.915299+0000) 2019-10-13T09:19:06.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:06.914+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.915473+0000) 2019-10-13T09:19:06.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:06.914+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.915612+0000) 2019-10-13T09:19:06.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:06.917+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.917988+0000) 2019-10-13T09:19:06.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:06.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:08.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:08.469+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:08.470638+0000) 2019-10-13T09:19:08.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:08.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:08.809602+0000) 2019-10-13T09:19:08.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:08.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:08.809677+0000) 2019-10-13T09:19:08.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:08.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:08.809699+0000) 2019-10-13T09:19:11.923 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:11.921+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.922639+0000) 2019-10-13T09:19:11.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:11.921+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.922935+0000) 2019-10-13T09:19:11.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:11.921+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.923547+0000) 2019-10-13T09:19:11.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:11.921+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.923311+0000) 2019-10-13T09:19:11.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:11.921+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.923364+0000) 2019-10-13T09:19:11.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:11.922+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.923468+0000) 2019-10-13T09:19:11.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:11.922+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.923659+0000) 2019-10-13T09:19:11.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:11.924+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.925914+0000) 2019-10-13T09:19:11.929 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:11.926+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:12.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:12.914+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:12.915550+0000) 2019-10-13T09:19:12.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:12.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:12.915999+0000) 2019-10-13T09:19:13.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:13.597+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:13.598313+0000) 2019-10-13T09:19:14.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:14.668+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:14.669779+0000) 2019-10-13T09:19:16.932 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:16.928+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.929870+0000) 2019-10-13T09:19:16.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:16.928+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.930101+0000) 2019-10-13T09:19:16.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:16.928+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.930279+0000) 2019-10-13T09:19:16.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:16.929+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.930489+0000) 2019-10-13T09:19:16.933 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:16.929+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.930966+0000) 2019-10-13T09:19:16.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:16.929+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.931107+0000) 2019-10-13T09:19:16.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:16.929+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.931412+0000) 2019-10-13T09:19:16.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:16.931+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.933366+0000) 2019-10-13T09:19:16.938 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:16.935+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 162983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:18.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:18.470+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:18.470833+0000) 2019-10-13T09:19:18.683 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:19:18.683 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:18.687 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:19:18.687 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:18.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:18.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:18.809775+0000) 2019-10-13T09:19:18.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:18.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:18.809834+0000) 2019-10-13T09:19:18.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:18.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:18.809845+0000) 2019-10-13T09:19:21.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:21.937+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.938688+0000) 2019-10-13T09:19:21.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:21.937+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.938826+0000) 2019-10-13T09:19:21.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:21.938+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.940135+0000) 2019-10-13T09:19:21.941 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:21.938+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.939569+0000) 2019-10-13T09:19:21.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:21.938+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.939477+0000) 2019-10-13T09:19:21.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:21.938+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.939734+0000) 2019-10-13T09:19:21.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:21.938+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.939656+0000) 2019-10-13T09:19:21.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:21.941+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.942804+0000) 2019-10-13T09:19:21.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:21.943+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:22.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:22.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:22.916171+0000) 2019-10-13T09:19:22.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:22.914+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:22.915713+0000) 2019-10-13T09:19:23.601 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:23.599+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:23.600624+0000) 2019-10-13T09:19:24.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:24.669+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:24.669962+0000) 2019-10-13T09:19:26.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:26.944+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.946239+0000) 2019-10-13T09:19:26.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:26.945+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.946586+0000) 2019-10-13T09:19:26.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:26.946+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.947541+0000) 2019-10-13T09:19:26.949 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:26.945+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.947263+0000) 2019-10-13T09:19:26.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:26.945+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.947378+0000) 2019-10-13T09:19:26.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:26.946+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.947500+0000) 2019-10-13T09:19:26.949 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:26.946+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.947814+0000) 2019-10-13T09:19:26.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:26.949+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.949796+0000) 2019-10-13T09:19:26.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:26.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:28.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:28.470+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:28.471010+0000) 2019-10-13T09:19:28.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:28.808+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:28.809982+0000) 2019-10-13T09:19:28.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:28.808+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:28.810000+0000) 2019-10-13T09:19:28.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:28.808+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:28.810031+0000) 2019-10-13T09:19:31.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:31.955+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.956026+0000) 2019-10-13T09:19:31.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:31.954+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.955568+0000) 2019-10-13T09:19:31.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:31.954+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.955903+0000) 2019-10-13T09:19:31.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:31.954+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.955962+0000) 2019-10-13T09:19:31.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:31.954+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.956141+0000) 2019-10-13T09:19:31.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:31.954+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.956359+0000) 2019-10-13T09:19:31.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:31.955+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.956476+0000) 2019-10-13T09:19:31.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:31.957+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.958581+0000) 2019-10-13T09:19:31.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:31.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:32.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:32.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:32.916329+0000) 2019-10-13T09:19:32.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:32.915+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:32.915891+0000) 2019-10-13T09:19:33.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:33.602+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:33.602976+0000) 2019-10-13T09:19:34.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:34.669+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:34.670160+0000) 2019-10-13T09:19:36.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:36.962+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.963145+0000) 2019-10-13T09:19:36.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:36.962+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.963284+0000) 2019-10-13T09:19:36.965 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:36.963+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.963420+0000) 2019-10-13T09:19:36.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:36.963+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.963579+0000) 2019-10-13T09:19:36.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:36.963+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.963660+0000) 2019-10-13T09:19:36.965 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:36.963+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.963863+0000) 2019-10-13T09:19:36.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:36.964+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.964894+0000) 2019-10-13T09:19:36.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:36.966+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.967222+0000) 2019-10-13T09:19:36.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:36.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:38.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:38.470+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:38.471206+0000) 2019-10-13T09:19:38.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:38.809+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:38.810164+0000) 2019-10-13T09:19:38.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:38.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:38.810165+0000) 2019-10-13T09:19:38.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:38.809+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:38.810237+0000) 2019-10-13T09:19:41.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:41.970+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.970414+0000) 2019-10-13T09:19:41.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:41.970+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.970611+0000) 2019-10-13T09:19:41.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:41.970+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.970906+0000) 2019-10-13T09:19:41.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:41.971+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.972448+0000) 2019-10-13T09:19:41.973 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:41.970+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.971236+0000) 2019-10-13T09:19:41.973 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:41.971+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.971508+0000) 2019-10-13T09:19:41.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:41.971+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.971952+0000) 2019-10-13T09:19:41.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:41.973+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.974265+0000) 2019-10-13T09:19:41.978 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:41.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:42.916 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:42.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:42.916521+0000) 2019-10-13T09:19:42.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:42.915+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:42.916010+0000) 2019-10-13T09:19:43.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:43.604+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:43.605304+0000) 2019-10-13T09:19:44.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:44.670+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:44.670381+0000) 2019-10-13T09:19:46.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:46.978+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.979380+0000) 2019-10-13T09:19:46.981 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:46.979+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.979670+0000) 2019-10-13T09:19:46.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:46.979+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.980600+0000) 2019-10-13T09:19:46.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:46.979+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.980248+0000) 2019-10-13T09:19:46.982 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:46.980+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.980524+0000) 2019-10-13T09:19:46.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:46.980+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.980914+0000) 2019-10-13T09:19:46.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:46.980+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.981232+0000) 2019-10-13T09:19:46.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:46.982+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.983524+0000) 2019-10-13T09:19:46.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:46.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163173 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:48.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:48.471+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:48.471412+0000) 2019-10-13T09:19:48.791 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:19:48.791 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:48.796 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:19:48.796 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:48.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:48.809+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:48.810359+0000) 2019-10-13T09:19:48.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:48.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:48.810346+0000) 2019-10-13T09:19:48.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:48.809+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:48.810369+0000) 2019-10-13T09:19:51.990 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:51.986+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.988290+0000) 2019-10-13T09:19:51.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:51.987+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.988442+0000) 2019-10-13T09:19:51.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:51.987+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.988786+0000) 2019-10-13T09:19:51.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:51.987+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.988994+0000) 2019-10-13T09:19:51.991 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:51.987+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.989173+0000) 2019-10-13T09:19:51.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:51.988+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.989424+0000) 2019-10-13T09:19:51.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:51.989+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.990538+0000) 2019-10-13T09:19:51.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:51.990+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.991599+0000) 2019-10-13T09:19:51.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:51.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163202 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:52.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:52.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:52.916694+0000) 2019-10-13T09:19:52.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:52.914+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:52.916115+0000) 2019-10-13T09:19:53.608 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:53.607+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:53.607698+0000) 2019-10-13T09:19:54.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:54.669+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:54.670586+0000) 2019-10-13T09:19:56.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:56.995+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.996811+0000) 2019-10-13T09:19:56.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:19:56.997+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.997822+0000) 2019-10-13T09:19:56.999 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:56.995+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.997323+0000) 2019-10-13T09:19:56.999 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:19:56.996+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.997519+0000) 2019-10-13T09:19:56.999 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:56.996+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.997559+0000) 2019-10-13T09:19:56.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:19:56.996+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.997812+0000) 2019-10-13T09:19:57.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:56.996+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.997898+0000) 2019-10-13T09:19:57.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:19:57.000+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:57.000920+0000) 2019-10-13T09:19:57.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:19:57.001+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:19:58.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:19:58.470+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:58.471585+0000) 2019-10-13T09:19:58.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:19:58.809+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:58.810550+0000) 2019-10-13T09:19:58.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:19:58.809+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:58.810517+0000) 2019-10-13T09:19:58.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:19:58.809+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:58.810533+0000) 2019-10-13T09:20:02.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:02.003+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.004450+0000) 2019-10-13T09:20:02.007 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:02.003+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.004586+0000) 2019-10-13T09:20:02.007 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:02.003+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.004780+0000) 2019-10-13T09:20:02.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:02.003+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.005258+0000) 2019-10-13T09:20:02.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:02.004+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.005433+0000) 2019-10-13T09:20:02.008 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:02.004+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.005474+0000) 2019-10-13T09:20:02.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:02.004+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.005906+0000) 2019-10-13T09:20:02.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:02.006+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.008434+0000) 2019-10-13T09:20:02.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:02.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163266 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:02.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:02.915+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.916826+0000) 2019-10-13T09:20:02.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:02.915+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:02.916261+0000) 2019-10-13T09:20:03.610 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:03.609+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:03.610129+0000) 2019-10-13T09:20:04.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:04.670+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:04.670760+0000) 2019-10-13T09:20:07.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:07.010+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.011960+0000) 2019-10-13T09:20:07.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:07.010+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.011615+0000) 2019-10-13T09:20:07.013 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:07.010+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.011723+0000) 2019-10-13T09:20:07.014 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:07.010+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.012027+0000) 2019-10-13T09:20:07.014 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:07.010+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.012217+0000) 2019-10-13T09:20:07.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:07.010+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.012286+0000) 2019-10-13T09:20:07.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:07.010+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.012385+0000) 2019-10-13T09:20:07.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:07.012+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.014459+0000) 2019-10-13T09:20:07.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:07.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:08.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:08.471+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:08.471775+0000) 2019-10-13T09:20:08.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:08.810+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:08.810717+0000) 2019-10-13T09:20:08.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:08.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:08.810679+0000) 2019-10-13T09:20:08.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:08.810+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:08.810749+0000) 2019-10-13T09:20:12.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:12.020+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.021980+0000) 2019-10-13T09:20:12.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:12.020+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.021485+0000) 2019-10-13T09:20:12.023 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:12.020+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.021648+0000) 2019-10-13T09:20:12.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:12.020+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.021969+0000) 2019-10-13T09:20:12.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:12.020+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.022025+0000) 2019-10-13T09:20:12.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:12.020+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.022359+0000) 2019-10-13T09:20:12.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:12.021+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.022705+0000) 2019-10-13T09:20:12.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:12.022+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.024310+0000) 2019-10-13T09:20:12.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:12.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163328 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:12.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:12.916+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.916955+0000) 2019-10-13T09:20:12.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:12.915+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:12.916391+0000) 2019-10-13T09:20:13.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:13.611+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:13.612498+0000) 2019-10-13T09:20:14.672 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:14.670+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:14.670942+0000) 2019-10-13T09:20:17.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:17.027+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.029193+0000) 2019-10-13T09:20:17.030 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:17.027+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.029328+0000) 2019-10-13T09:20:17.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:17.029+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.030393+0000) 2019-10-13T09:20:17.031 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:17.028+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.029925+0000) 2019-10-13T09:20:17.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:17.028+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.030035+0000) 2019-10-13T09:20:17.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:17.028+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.030360+0000) 2019-10-13T09:20:17.032 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:17.029+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.030635+0000) 2019-10-13T09:20:17.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:17.032+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.033081+0000) 2019-10-13T09:20:17.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:17.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:18.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:18.471+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:18.471967+0000) 2019-10-13T09:20:18.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:18.810+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:18.810884+0000) 2019-10-13T09:20:18.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:18.810+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:18.810896+0000) 2019-10-13T09:20:18.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:18.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:18.810883+0000) 2019-10-13T09:20:18.931 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:20:18.931 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:18.935 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:20:18.935 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:22.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:22.036+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.037714+0000) 2019-10-13T09:20:22.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:22.036+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.036889+0000) 2019-10-13T09:20:22.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:22.036+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.037116+0000) 2019-10-13T09:20:22.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:22.036+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.037237+0000) 2019-10-13T09:20:22.039 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:22.037+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.037406+0000) 2019-10-13T09:20:22.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:22.037+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.037942+0000) 2019-10-13T09:20:22.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:22.037+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.038089+0000) 2019-10-13T09:20:22.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:22.039+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.040695+0000) 2019-10-13T09:20:22.044 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:22.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163392 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:22.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:22.916+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.917080+0000) 2019-10-13T09:20:22.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:22.916+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:22.916545+0000) 2019-10-13T09:20:23.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:23.614+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:23.614849+0000) 2019-10-13T09:20:24.672 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:24.670+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:24.671083+0000) 2019-10-13T09:20:27.047 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:27.045+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.045548+0000) 2019-10-13T09:20:27.047 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:27.045+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.045938+0000) 2019-10-13T09:20:27.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:27.045+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.046212+0000) 2019-10-13T09:20:27.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:27.046+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.046819+0000) 2019-10-13T09:20:27.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:27.046+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.046526+0000) 2019-10-13T09:20:27.048 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:27.046+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.046866+0000) 2019-10-13T09:20:27.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:27.046+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.047057+0000) 2019-10-13T09:20:27.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:27.048+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.049543+0000) 2019-10-13T09:20:27.053 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:27.051+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:28.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:28.471+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:28.472153+0000) 2019-10-13T09:20:28.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:28.810+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:28.811042+0000) 2019-10-13T09:20:28.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:28.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:28.811040+0000) 2019-10-13T09:20:28.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:28.810+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:28.811058+0000) 2019-10-13T09:20:32.056 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:32.053+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.054055+0000) 2019-10-13T09:20:32.056 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:32.053+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.053808+0000) 2019-10-13T09:20:32.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:32.053+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.054323+0000) 2019-10-13T09:20:32.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:32.054+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.054477+0000) 2019-10-13T09:20:32.057 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:32.054+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.054588+0000) 2019-10-13T09:20:32.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:32.054+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.054963+0000) 2019-10-13T09:20:32.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:32.054+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.055640+0000) 2019-10-13T09:20:32.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:32.056+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.057588+0000) 2019-10-13T09:20:32.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:32.060+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:32.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:32.916+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.917240+0000) 2019-10-13T09:20:32.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:32.916+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:32.916696+0000) 2019-10-13T09:20:33.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:33.616+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:33.617358+0000) 2019-10-13T09:20:34.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:34.671+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:34.671269+0000) 2019-10-13T09:20:37.064 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:37.061+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.062667+0000) 2019-10-13T09:20:37.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:37.061+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.062800+0000) 2019-10-13T09:20:37.065 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:37.061+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.063101+0000) 2019-10-13T09:20:37.065 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:37.061+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.063230+0000) 2019-10-13T09:20:37.065 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:37.062+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.063507+0000) 2019-10-13T09:20:37.065 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:37.062+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.063806+0000) 2019-10-13T09:20:37.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:37.063+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.064525+0000) 2019-10-13T09:20:37.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:37.065+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.066245+0000) 2019-10-13T09:20:37.070 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:37.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:38.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:38.471+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:38.472312+0000) 2019-10-13T09:20:38.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:38.810+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:38.811251+0000) 2019-10-13T09:20:38.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:38.810+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:38.811246+0000) 2019-10-13T09:20:38.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:38.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:38.811247+0000) 2019-10-13T09:20:42.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:42.071+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.072234+0000) 2019-10-13T09:20:42.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:42.070+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.071646+0000) 2019-10-13T09:20:42.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:42.070+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.071973+0000) 2019-10-13T09:20:42.075 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:42.070+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.071870+0000) 2019-10-13T09:20:42.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:42.070+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.072088+0000) 2019-10-13T09:20:42.076 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:42.070+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.072257+0000) 2019-10-13T09:20:42.076 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:42.071+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.072483+0000) 2019-10-13T09:20:42.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:42.074+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.075516+0000) 2019-10-13T09:20:42.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:42.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:42.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:42.917+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.917404+0000) 2019-10-13T09:20:42.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:42.915+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:42.916842+0000) 2019-10-13T09:20:43.619 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:43.619+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:43.619540+0000) 2019-10-13T09:20:44.672 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:44.670+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:44.671441+0000) 2019-10-13T09:20:47.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:47.078+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.079602+0000) 2019-10-13T09:20:47.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:47.078+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.079811+0000) 2019-10-13T09:20:47.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:47.079+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.080701+0000) 2019-10-13T09:20:47.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:47.078+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.080053+0000) 2019-10-13T09:20:47.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:47.079+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.080524+0000) 2019-10-13T09:20:47.082 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:47.079+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.080712+0000) 2019-10-13T09:20:47.082 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:47.079+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.080761+0000) 2019-10-13T09:20:47.084 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:47.083+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.083909+0000) 2019-10-13T09:20:47.087 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:47.085+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:48.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:48.471+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:48.472558+0000) 2019-10-13T09:20:48.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:48.810+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:48.811416+0000) 2019-10-13T09:20:48.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:48.810+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:48.811420+0000) 2019-10-13T09:20:48.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:48.810+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:48.811442+0000) 2019-10-13T09:20:49.062 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:20:49.062 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:49.075 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:20:49.075 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:52.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:52.086+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.088179+0000) 2019-10-13T09:20:52.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:52.087+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.088587+0000) 2019-10-13T09:20:52.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:52.087+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.088813+0000) 2019-10-13T09:20:52.090 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:52.087+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.088873+0000) 2019-10-13T09:20:52.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:52.088+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.089258+0000) 2019-10-13T09:20:52.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:52.088+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.089417+0000) 2019-10-13T09:20:52.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:52.088+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.089578+0000) 2019-10-13T09:20:52.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:52.091+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.091902+0000) 2019-10-13T09:20:52.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:52.093+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:52.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:52.916+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.917544+0000) 2019-10-13T09:20:52.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:52.916+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:52.916945+0000) 2019-10-13T09:20:53.622 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:53.621+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:53.622032+0000) 2019-10-13T09:20:54.672 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:54.670+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:54.671623+0000) 2019-10-13T09:20:57.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:20:57.094+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.095783+0000) 2019-10-13T09:20:57.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:20:57.095+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.097330+0000) 2019-10-13T09:20:57.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:57.094+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.096325+0000) 2019-10-13T09:20:57.098 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:57.095+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.096657+0000) 2019-10-13T09:20:57.098 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:20:57.095+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.097173+0000) 2019-10-13T09:20:57.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:57.095+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.097316+0000) 2019-10-13T09:20:57.099 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:57.096+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.097643+0000) 2019-10-13T09:20:57.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:20:57.098+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.100357+0000) 2019-10-13T09:20:57.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:20:57.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:20:58.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:20:58.472+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:58.472752+0000) 2019-10-13T09:20:58.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:20:58.811+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:58.811579+0000) 2019-10-13T09:20:58.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:20:58.811+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:58.811599+0000) 2019-10-13T09:20:58.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:20:58.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:58.811576+0000) 2019-10-13T09:21:02.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:02.103+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.105366+0000) 2019-10-13T09:21:02.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:02.103+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.104497+0000) 2019-10-13T09:21:02.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:02.103+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.104838+0000) 2019-10-13T09:21:02.107 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:02.103+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.105273+0000) 2019-10-13T09:21:02.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:02.104+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.105402+0000) 2019-10-13T09:21:02.107 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:02.104+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.105536+0000) 2019-10-13T09:21:02.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:02.104+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.106060+0000) 2019-10-13T09:21:02.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:02.106+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.108691+0000) 2019-10-13T09:21:02.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:02.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:02.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:02.916+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.917662+0000) 2019-10-13T09:21:02.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:02.916+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:02.917082+0000) 2019-10-13T09:21:03.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:03.623+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:03.624622+0000) 2019-10-13T09:21:04.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:04.671+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:04.671815+0000) 2019-10-13T09:21:07.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:07.112+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.113326+0000) 2019-10-13T09:21:07.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:07.113+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.114439+0000) 2019-10-13T09:21:07.115 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:07.113+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.114021+0000) 2019-10-13T09:21:07.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:07.113+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.114307+0000) 2019-10-13T09:21:07.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:07.114+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.114513+0000) 2019-10-13T09:21:07.116 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:07.114+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.114629+0000) 2019-10-13T09:21:07.116 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:07.114+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.114725+0000) 2019-10-13T09:21:07.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:07.116+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.116976+0000) 2019-10-13T09:21:07.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:07.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163677 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:08.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:08.472+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:08.472941+0000) 2019-10-13T09:21:08.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:08.811+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:08.811803+0000) 2019-10-13T09:21:08.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:08.811+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:08.811805+0000) 2019-10-13T09:21:08.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:08.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:08.811803+0000) 2019-10-13T09:21:12.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:12.121+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.122533+0000) 2019-10-13T09:21:12.124 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:12.121+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.122045+0000) 2019-10-13T09:21:12.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:12.122+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.122393+0000) 2019-10-13T09:21:12.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:12.122+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.122741+0000) 2019-10-13T09:21:12.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:12.122+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.123056+0000) 2019-10-13T09:21:12.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:12.122+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.123124+0000) 2019-10-13T09:21:12.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:12.124+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.125137+0000) 2019-10-13T09:21:12.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:12.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.123443+0000) 2019-10-13T09:21:12.129 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:12.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:12.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:12.917+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.917796+0000) 2019-10-13T09:21:12.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:12.916+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:12.917244+0000) 2019-10-13T09:21:13.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:13.626+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:13.626846+0000) 2019-10-13T09:21:14.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:14.671+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:14.671999+0000) 2019-10-13T09:21:17.131 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:17.129+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.130342+0000) 2019-10-13T09:21:17.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:17.130+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.131296+0000) 2019-10-13T09:21:17.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:17.130+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.130772+0000) 2019-10-13T09:21:17.133 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:17.130+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.130917+0000) 2019-10-13T09:21:17.133 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:17.130+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.131193+0000) 2019-10-13T09:21:17.134 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:17.131+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.131425+0000) 2019-10-13T09:21:17.134 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:17.131+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.131505+0000) 2019-10-13T09:21:17.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:17.133+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.134671+0000) 2019-10-13T09:21:17.138 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:17.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163739 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:18.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:18.472+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:18.473118+0000) 2019-10-13T09:21:18.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:18.811+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:18.811990+0000) 2019-10-13T09:21:18.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:18.811+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:18.811999+0000) 2019-10-13T09:21:18.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:18.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:18.811999+0000) 2019-10-13T09:21:19.201 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:21:19.201 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:19.206 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:21:19.206 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:22.141 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:22.138+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.139069+0000) 2019-10-13T09:21:22.141 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:22.139+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.139390+0000) 2019-10-13T09:21:22.141 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:22.139+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.139729+0000) 2019-10-13T09:21:22.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:22.139+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.139807+0000) 2019-10-13T09:21:22.142 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:22.139+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.140116+0000) 2019-10-13T09:21:22.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:22.139+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.140089+0000) 2019-10-13T09:21:22.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:22.141+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.141842+0000) 2019-10-13T09:21:22.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:22.139+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.140345+0000) 2019-10-13T09:21:22.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:22.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163768 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:22.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:22.917+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.918003+0000) 2019-10-13T09:21:22.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:22.916+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:22.917378+0000) 2019-10-13T09:21:23.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:23.628+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:23.629092+0000) 2019-10-13T09:21:24.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:24.671+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:24.672183+0000) 2019-10-13T09:21:27.149 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:27.146+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.147826+0000) 2019-10-13T09:21:27.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:27.146+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.147989+0000) 2019-10-13T09:21:27.149 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:27.146+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.148228+0000) 2019-10-13T09:21:27.150 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:27.148+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.149367+0000) 2019-10-13T09:21:27.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:27.147+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.148700+0000) 2019-10-13T09:21:27.150 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:27.147+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.148913+0000) 2019-10-13T09:21:27.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:27.147+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.149198+0000) 2019-10-13T09:21:27.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:27.150+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.150830+0000) 2019-10-13T09:21:27.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:27.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:28.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:28.472+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:28.473311+0000) 2019-10-13T09:21:28.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:28.811+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:28.812178+0000) 2019-10-13T09:21:28.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:28.811+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:28.812176+0000) 2019-10-13T09:21:28.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:28.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:28.812177+0000) 2019-10-13T09:21:32.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:32.154+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.156087+0000) 2019-10-13T09:21:32.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:32.154+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.156266+0000) 2019-10-13T09:21:32.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:32.155+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.156472+0000) 2019-10-13T09:21:32.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:32.155+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.156691+0000) 2019-10-13T09:21:32.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:32.156+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.157690+0000) 2019-10-13T09:21:32.159 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:32.155+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.156983+0000) 2019-10-13T09:21:32.159 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:32.155+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.157291+0000) 2019-10-13T09:21:32.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:32.158+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.159041+0000) 2019-10-13T09:21:32.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:32.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:32.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:32.917+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.918163+0000) 2019-10-13T09:21:32.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:32.916+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:32.917560+0000) 2019-10-13T09:21:33.631 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:33.631+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:33.631339+0000) 2019-10-13T09:21:34.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:34.671+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:34.672372+0000) 2019-10-13T09:21:37.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:37.162+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.163402+0000) 2019-10-13T09:21:37.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:37.163+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.164329+0000) 2019-10-13T09:21:37.165 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:37.162+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.163821+0000) 2019-10-13T09:21:37.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:37.162+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.164023+0000) 2019-10-13T09:21:37.166 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:37.162+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.164171+0000) 2019-10-13T09:21:37.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:37.163+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.164414+0000) 2019-10-13T09:21:37.166 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:37.163+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.164764+0000) 2019-10-13T09:21:37.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:37.167+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.168657+0000) 2019-10-13T09:21:37.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:37.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:38.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:38.472+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:38.473506+0000) 2019-10-13T09:21:38.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:38.811+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:38.812377+0000) 2019-10-13T09:21:38.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:38.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:38.812379+0000) 2019-10-13T09:21:38.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:38.811+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:38.812381+0000) 2019-10-13T09:21:42.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:42.169+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.170982+0000) 2019-10-13T09:21:42.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:42.170+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.171438+0000) 2019-10-13T09:21:42.173 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:42.170+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.171562+0000) 2019-10-13T09:21:42.173 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:42.170+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.171688+0000) 2019-10-13T09:21:42.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:42.170+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.171932+0000) 2019-10-13T09:21:42.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:42.170+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.172270+0000) 2019-10-13T09:21:42.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:42.172+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.172932+0000) 2019-10-13T09:21:42.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:42.173+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.174675+0000) 2019-10-13T09:21:42.178 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:42.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:42.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:42.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.918296+0000) 2019-10-13T09:21:42.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:42.917+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:42.917700+0000) 2019-10-13T09:21:43.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:43.633+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:43.633512+0000) 2019-10-13T09:21:44.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:44.672+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:44.672569+0000) 2019-10-13T09:21:47.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:47.178+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.179829+0000) 2019-10-13T09:21:47.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:47.177+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.179156+0000) 2019-10-13T09:21:47.181 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:47.178+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.179461+0000) 2019-10-13T09:21:47.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:47.178+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.179925+0000) 2019-10-13T09:21:47.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:47.178+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.180197+0000) 2019-10-13T09:21:47.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:47.178+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.180224+0000) 2019-10-13T09:21:47.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:47.179+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.180695+0000) 2019-10-13T09:21:47.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:47.180+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.182489+0000) 2019-10-13T09:21:47.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:47.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:48.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:48.473+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:48.473670+0000) 2019-10-13T09:21:48.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:48.812+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:48.812608+0000) 2019-10-13T09:21:48.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:48.812+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:48.812608+0000) 2019-10-13T09:21:48.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:48.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:48.812593+0000) 2019-10-13T09:21:49.283 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:21:49.283 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:49.287 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:21:49.288 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:52.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:52.187+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.187889+0000) 2019-10-13T09:21:52.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:52.187+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.188064+0000) 2019-10-13T09:21:52.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:52.187+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.189257+0000) 2019-10-13T09:21:52.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:52.188+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.188542+0000) 2019-10-13T09:21:52.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:52.188+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.188647+0000) 2019-10-13T09:21:52.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:52.188+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.189007+0000) 2019-10-13T09:21:52.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:52.188+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.189173+0000) 2019-10-13T09:21:52.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:52.189+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.191121+0000) 2019-10-13T09:21:52.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:52.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:52.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:52.917+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.918419+0000) 2019-10-13T09:21:52.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:52.917+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:52.917830+0000) 2019-10-13T09:21:53.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:53.634+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:53.635862+0000) 2019-10-13T09:21:54.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:54.672+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:54.672734+0000) 2019-10-13T09:21:57.198 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:57.196+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.196984+0000) 2019-10-13T09:21:57.198 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:57.196+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.197139+0000) 2019-10-13T09:21:57.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:57.196+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.197248+0000) 2019-10-13T09:21:57.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:57.197+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.197593+0000) 2019-10-13T09:21:57.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:21:57.197+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.197840+0000) 2019-10-13T09:21:57.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:21:57.197+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.197503+0000) 2019-10-13T09:21:57.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:21:57.197+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.199179+0000) 2019-10-13T09:21:57.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:21:57.199+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.201149+0000) 2019-10-13T09:21:57.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:21:57.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 163983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:21:58.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:21:58.473+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:58.473908+0000) 2019-10-13T09:21:58.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:21:58.812+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:58.812770+0000) 2019-10-13T09:21:58.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:21:58.812+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:58.812764+0000) 2019-10-13T09:21:58.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:21:58.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:58.812788+0000) 2019-10-13T09:22:02.206 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:02.204+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.205031+0000) 2019-10-13T09:22:02.206 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:02.204+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.205327+0000) 2019-10-13T09:22:02.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:02.206+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.206770+0000) 2019-10-13T09:22:02.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:02.205+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.206159+0000) 2019-10-13T09:22:02.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:02.205+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.206367+0000) 2019-10-13T09:22:02.208 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:02.206+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.206411+0000) 2019-10-13T09:22:02.208 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:02.206+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.206550+0000) 2019-10-13T09:22:02.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:02.207+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.208381+0000) 2019-10-13T09:22:02.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:02.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164012 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:02.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:02.917+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.917935+0000) 2019-10-13T09:22:02.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:02.917+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:02.918554+0000) 2019-10-13T09:22:03.641 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:03.637+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:03.638210+0000) 2019-10-13T09:22:04.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:04.672+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:04.672907+0000) 2019-10-13T09:22:07.215 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:07.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.214337+0000) 2019-10-13T09:22:07.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:07.214+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.214541+0000) 2019-10-13T09:22:07.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:07.215+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.215978+0000) 2019-10-13T09:22:07.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:07.214+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.214755+0000) 2019-10-13T09:22:07.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:07.214+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.214857+0000) 2019-10-13T09:22:07.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:07.214+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.215142+0000) 2019-10-13T09:22:07.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:07.215+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.215424+0000) 2019-10-13T09:22:07.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:07.217+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.218499+0000) 2019-10-13T09:22:07.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:07.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164043 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:08.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:08.473+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:08.474089+0000) 2019-10-13T09:22:08.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:08.811+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:08.812943+0000) 2019-10-13T09:22:08.815 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:08.811+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:08.812927+0000) 2019-10-13T09:22:08.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:08.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:08.812948+0000) 2019-10-13T09:22:12.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:12.221+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.223078+0000) 2019-10-13T09:22:12.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:12.221+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.223307+0000) 2019-10-13T09:22:12.225 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:12.224+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.224742+0000) 2019-10-13T09:22:12.225 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:12.222+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.223527+0000) 2019-10-13T09:22:12.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:12.222+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.224097+0000) 2019-10-13T09:22:12.226 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:12.223+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.224477+0000) 2019-10-13T09:22:12.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:12.223+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.224624+0000) 2019-10-13T09:22:12.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:12.226+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.227035+0000) 2019-10-13T09:22:12.231 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:12.228+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164072 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:12.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:12.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.918700+0000) 2019-10-13T09:22:12.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:12.917+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:12.918091+0000) 2019-10-13T09:22:13.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:13.639+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:13.640438+0000) 2019-10-13T09:22:14.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:14.672+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:14.673109+0000) 2019-10-13T09:22:17.233 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:17.230+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.231819+0000) 2019-10-13T09:22:17.233 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:17.230+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.232014+0000) 2019-10-13T09:22:17.233 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:17.230+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.232115+0000) 2019-10-13T09:22:17.234 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:17.232+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.233206+0000) 2019-10-13T09:22:17.234 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:17.231+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.232549+0000) 2019-10-13T09:22:17.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:17.231+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.232782+0000) 2019-10-13T09:22:17.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:17.231+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.233093+0000) 2019-10-13T09:22:17.235 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:17.233+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.234580+0000) 2019-10-13T09:22:17.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:17.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164105 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:18.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:18.473+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:18.474285+0000) 2019-10-13T09:22:18.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:18.811+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:18.813162+0000) 2019-10-13T09:22:18.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:18.811+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:18.813155+0000) 2019-10-13T09:22:18.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:18.811+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:18.813148+0000) 2019-10-13T09:22:19.462 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:22:19.463 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:19.467 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:22:19.467 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:22.241 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:22.238+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.239829+0000) 2019-10-13T09:22:22.241 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:22.238+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.239917+0000) 2019-10-13T09:22:22.242 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:22.240+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.241584+0000) 2019-10-13T09:22:22.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:22.239+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.240516+0000) 2019-10-13T09:22:22.244 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:22.239+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.240997+0000) 2019-10-13T09:22:22.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:22.239+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.241275+0000) 2019-10-13T09:22:22.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:22.239+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.241372+0000) 2019-10-13T09:22:22.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:22.242+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.243288+0000) 2019-10-13T09:22:22.248 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:22.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164134 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:22.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:22.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.918870+0000) 2019-10-13T09:22:22.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:22.917+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:22.918258+0000) 2019-10-13T09:22:23.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:23.642+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:23.642767+0000) 2019-10-13T09:22:24.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:24.672+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:24.673311+0000) 2019-10-13T09:22:27.249 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:27.248+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.249302+0000) 2019-10-13T09:22:27.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:27.247+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.248685+0000) 2019-10-13T09:22:27.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:27.247+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.249003+0000) 2019-10-13T09:22:27.251 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:27.247+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.249154+0000) 2019-10-13T09:22:27.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:27.247+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.249332+0000) 2019-10-13T09:22:27.251 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:27.248+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.249561+0000) 2019-10-13T09:22:27.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:27.248+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.249686+0000) 2019-10-13T09:22:27.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:27.252+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.252846+0000) 2019-10-13T09:22:27.255 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:27.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164165 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:28.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:28.473+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:28.474458+0000) 2019-10-13T09:22:28.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:28.812+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:28.813326+0000) 2019-10-13T09:22:28.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:28.812+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:28.813340+0000) 2019-10-13T09:22:28.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:28.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:28.813324+0000) 2019-10-13T09:22:32.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:32.253+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.255377+0000) 2019-10-13T09:22:32.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:32.255+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.256428+0000) 2019-10-13T09:22:32.258 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:32.254+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.255936+0000) 2019-10-13T09:22:32.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:32.254+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.255996+0000) 2019-10-13T09:22:32.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:32.254+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.256079+0000) 2019-10-13T09:22:32.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:32.254+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.256298+0000) 2019-10-13T09:22:32.259 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:32.255+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.256450+0000) 2019-10-13T09:22:32.260 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:32.257+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.258719+0000) 2019-10-13T09:22:32.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:32.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164194 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:32.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:32.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.918971+0000) 2019-10-13T09:22:32.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:32.917+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:32.918428+0000) 2019-10-13T09:22:33.645 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:33.645+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:33.645148+0000) 2019-10-13T09:22:34.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:34.673+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:34.673499+0000) 2019-10-13T09:22:37.263 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:37.261+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.262217+0000) 2019-10-13T09:22:37.264 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:37.261+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.263488+0000) 2019-10-13T09:22:37.264 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:37.262+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.262780+0000) 2019-10-13T09:22:37.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:37.262+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.263091+0000) 2019-10-13T09:22:37.265 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:37.262+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.263201+0000) 2019-10-13T09:22:37.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:37.263+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.263570+0000) 2019-10-13T09:22:37.265 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:37.263+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.263877+0000) 2019-10-13T09:22:37.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:37.266+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.268019+0000) 2019-10-13T09:22:37.270 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:37.268+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:38.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:38.474+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:38.474652+0000) 2019-10-13T09:22:38.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:38.812+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:38.813415+0000) 2019-10-13T09:22:38.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:38.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:38.813476+0000) 2019-10-13T09:22:38.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:38.813+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:38.813491+0000) 2019-10-13T09:22:42.273 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:42.271+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.271424+0000) 2019-10-13T09:22:42.273 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:42.271+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.271507+0000) 2019-10-13T09:22:42.273 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:42.271+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.271900+0000) 2019-10-13T09:22:42.273 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:42.271+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.272187+0000) 2019-10-13T09:22:42.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:42.271+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.272261+0000) 2019-10-13T09:22:42.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:42.271+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.273369+0000) 2019-10-13T09:22:42.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:42.272+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.272694+0000) 2019-10-13T09:22:42.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:42.273+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.274891+0000) 2019-10-13T09:22:42.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:42.277+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164256 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:42.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:42.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.919104+0000) 2019-10-13T09:22:42.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:42.918+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:42.918564+0000) 2019-10-13T09:22:43.647 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:43.646+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:43.647395+0000) 2019-10-13T09:22:44.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:44.673+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:44.673709+0000) 2019-10-13T09:22:47.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:47.279+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.279743+0000) 2019-10-13T09:22:47.281 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:47.279+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.280032+0000) 2019-10-13T09:22:47.282 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:47.279+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.280249+0000) 2019-10-13T09:22:47.282 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:47.280+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.280504+0000) 2019-10-13T09:22:47.283 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:47.278+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.280685+0000) 2019-10-13T09:22:47.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:47.280+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.280886+0000) 2019-10-13T09:22:47.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:47.280+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.281242+0000) 2019-10-13T09:22:47.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:47.281+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.283599+0000) 2019-10-13T09:22:47.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:47.286+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164287 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:48.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:48.474+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:48.474777+0000) 2019-10-13T09:22:48.815 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:48.813+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:48.813585+0000) 2019-10-13T09:22:48.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:48.813+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:48.813619+0000) 2019-10-13T09:22:48.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:48.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:48.813666+0000) 2019-10-13T09:22:49.617 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:22:49.617 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:49.621 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:22:49.621 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:52.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:52.288+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.289461+0000) 2019-10-13T09:22:52.290 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:52.288+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.289009+0000) 2019-10-13T09:22:52.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:52.289+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.289456+0000) 2019-10-13T09:22:52.291 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:52.289+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.289635+0000) 2019-10-13T09:22:52.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:52.289+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.289936+0000) 2019-10-13T09:22:52.291 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:52.289+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.290088+0000) 2019-10-13T09:22:52.292 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:52.289+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.289976+0000) 2019-10-13T09:22:52.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:52.291+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.292082+0000) 2019-10-13T09:22:52.296 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:52.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:52.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:52.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.919215+0000) 2019-10-13T09:22:52.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:52.918+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:52.918705+0000) 2019-10-13T09:22:53.650 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:53.649+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:53.649825+0000) 2019-10-13T09:22:54.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:54.672+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:54.673874+0000) 2019-10-13T09:22:57.298 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:22:57.297+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.297993+0000) 2019-10-13T09:22:57.299 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:57.295+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.297377+0000) 2019-10-13T09:22:57.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:57.296+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.297551+0000) 2019-10-13T09:22:57.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:57.296+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.297835+0000) 2019-10-13T09:22:57.299 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:22:57.296+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.297730+0000) 2019-10-13T09:22:57.300 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:22:57.296+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.298141+0000) 2019-10-13T09:22:57.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:22:57.298+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.299613+0000) 2019-10-13T09:22:57.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:57.297+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.298606+0000) 2019-10-13T09:22:57.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:22:57.302+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164349 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:22:58.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:22:58.474+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:58.474970+0000) 2019-10-13T09:22:58.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:22:58.812+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:58.813797+0000) 2019-10-13T09:22:58.815 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:22:58.812+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:58.813782+0000) 2019-10-13T09:22:58.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:22:58.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:58.813856+0000) 2019-10-13T09:23:02.307 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:02.305+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.306671+0000) 2019-10-13T09:23:02.308 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:02.304+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.306354+0000) 2019-10-13T09:23:02.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:02.305+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.306458+0000) 2019-10-13T09:23:02.308 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:02.305+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.306695+0000) 2019-10-13T09:23:02.308 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:02.305+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.306876+0000) 2019-10-13T09:23:02.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:02.305+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.306985+0000) 2019-10-13T09:23:02.309 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:02.305+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.307150+0000) 2019-10-13T09:23:02.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:02.307+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.308686+0000) 2019-10-13T09:23:02.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:02.310+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164379 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:02.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:02.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.919361+0000) 2019-10-13T09:23:02.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:02.917+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:02.918849+0000) 2019-10-13T09:23:03.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:03.651+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:03.652276+0000) 2019-10-13T09:23:04.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:04.672+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:04.674077+0000) 2019-10-13T09:23:07.316 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:07.313+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.314402+0000) 2019-10-13T09:23:07.316 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:07.313+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.314439+0000) 2019-10-13T09:23:07.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:07.313+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.314619+0000) 2019-10-13T09:23:07.316 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:07.313+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.314779+0000) 2019-10-13T09:23:07.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:07.313+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.315226+0000) 2019-10-13T09:23:07.317 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:07.315+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.315740+0000) 2019-10-13T09:23:07.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:07.314+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.315553+0000) 2019-10-13T09:23:07.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:07.317+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.318191+0000) 2019-10-13T09:23:07.322 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:07.320+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164409 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:08.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:08.474+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:08.475145+0000) 2019-10-13T09:23:08.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:08.812+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:08.813981+0000) 2019-10-13T09:23:08.815 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:08.812+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:08.813981+0000) 2019-10-13T09:23:08.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:08.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:08.813994+0000) 2019-10-13T09:23:12.324 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:12.322+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.323711+0000) 2019-10-13T09:23:12.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:12.321+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.323098+0000) 2019-10-13T09:23:12.325 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:12.322+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.323450+0000) 2019-10-13T09:23:12.325 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:12.322+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.323654+0000) 2019-10-13T09:23:12.325 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:12.322+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.323789+0000) 2019-10-13T09:23:12.326 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:12.322+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.324136+0000) 2019-10-13T09:23:12.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:12.322+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.324201+0000) 2019-10-13T09:23:12.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:12.326+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.327206+0000) 2019-10-13T09:23:12.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:12.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164439 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:12.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:12.919+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.919552+0000) 2019-10-13T09:23:12.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:12.918+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:12.918985+0000) 2019-10-13T09:23:13.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:13.653+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:13.654723+0000) 2019-10-13T09:23:14.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:14.673+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:14.674289+0000) 2019-10-13T09:23:17.333 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:17.331+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.332502+0000) 2019-10-13T09:23:17.334 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:17.330+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.332373+0000) 2019-10-13T09:23:17.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:17.331+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.332650+0000) 2019-10-13T09:23:17.335 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:17.333+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.333895+0000) 2019-10-13T09:23:17.335 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:17.331+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.333217+0000) 2019-10-13T09:23:17.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:17.332+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.333456+0000) 2019-10-13T09:23:17.336 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:17.332+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.333767+0000) 2019-10-13T09:23:17.336 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:17.335+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.335851+0000) 2019-10-13T09:23:17.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:17.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164471 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:18.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:18.474+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:18.475283+0000) 2019-10-13T09:23:18.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:18.812+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:18.814175+0000) 2019-10-13T09:23:18.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:18.812+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:18.814180+0000) 2019-10-13T09:23:18.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:18.812+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:18.814177+0000) 2019-10-13T09:23:19.763 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:23:19.763 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:19.768 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:23:19.768 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:22.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:22.340+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.341271+0000) 2019-10-13T09:23:22.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:22.341+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.341475+0000) 2019-10-13T09:23:22.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:22.341+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.341722+0000) 2019-10-13T09:23:22.344 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:22.341+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.342626+0000) 2019-10-13T09:23:22.344 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:22.341+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.342298+0000) 2019-10-13T09:23:22.344 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:22.341+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.342112+0000) 2019-10-13T09:23:22.345 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:22.342+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.342618+0000) 2019-10-13T09:23:22.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:22.344+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.344961+0000) 2019-10-13T09:23:22.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:22.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164501 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:22.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:22.919+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.919712+0000) 2019-10-13T09:23:22.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:22.918+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:22.919124+0000) 2019-10-13T09:23:23.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:23.656+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:23.657073+0000) 2019-10-13T09:23:24.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:24.674+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:24.674481+0000) 2019-10-13T09:23:27.351 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:27.349+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.349987+0000) 2019-10-13T09:23:27.352 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:27.350+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.351553+0000) 2019-10-13T09:23:27.352 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:27.349+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.350317+0000) 2019-10-13T09:23:27.352 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:27.350+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.350517+0000) 2019-10-13T09:23:27.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:27.350+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.350880+0000) 2019-10-13T09:23:27.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:27.350+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.351264+0000) 2019-10-13T09:23:27.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:27.350+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.351347+0000) 2019-10-13T09:23:27.353 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:27.352+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.353152+0000) 2019-10-13T09:23:27.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:27.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:28.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:28.475+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:28.475482+0000) 2019-10-13T09:23:28.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:28.814+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:28.814389+0000) 2019-10-13T09:23:28.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:28.814+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:28.814394+0000) 2019-10-13T09:23:28.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:28.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:28.814388+0000) 2019-10-13T09:23:32.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:32.358+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.358743+0000) 2019-10-13T09:23:32.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:32.358+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.359288+0000) 2019-10-13T09:23:32.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:32.358+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.359825+0000) 2019-10-13T09:23:32.361 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:32.359+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.359522+0000) 2019-10-13T09:23:32.361 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:32.359+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.359648+0000) 2019-10-13T09:23:32.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:32.359+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.359901+0000) 2019-10-13T09:23:32.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:32.359+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.359939+0000) 2019-10-13T09:23:32.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:32.360+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.361882+0000) 2019-10-13T09:23:32.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:32.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164561 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:32.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:32.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.919835+0000) 2019-10-13T09:23:32.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:32.919+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:32.919277+0000) 2019-10-13T09:23:33.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:33.658+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:33.659544+0000) 2019-10-13T09:23:34.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:34.674+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:34.674648+0000) 2019-10-13T09:23:37.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:37.366+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.367963+0000) 2019-10-13T09:23:37.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:37.366+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.367327+0000) 2019-10-13T09:23:37.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:37.367+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.367558+0000) 2019-10-13T09:23:37.369 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:37.367+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.367972+0000) 2019-10-13T09:23:37.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:37.367+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.368200+0000) 2019-10-13T09:23:37.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:37.368+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.368637+0000) 2019-10-13T09:23:37.370 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:37.368+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.368993+0000) 2019-10-13T09:23:37.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:37.369+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.371517+0000) 2019-10-13T09:23:37.375 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:37.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164593 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:38.476 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:38.475+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:38.475665+0000) 2019-10-13T09:23:38.816 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:38.814+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:38.814548+0000) 2019-10-13T09:23:38.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:38.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:38.814581+0000) 2019-10-13T09:23:38.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:38.814+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:38.814588+0000) 2019-10-13T09:23:42.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:42.375+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.376709+0000) 2019-10-13T09:23:42.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:42.374+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.376070+0000) 2019-10-13T09:23:42.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:42.374+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.376266+0000) 2019-10-13T09:23:42.379 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:42.374+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.376291+0000) 2019-10-13T09:23:42.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:42.375+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.376607+0000) 2019-10-13T09:23:42.379 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:42.375+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.376852+0000) 2019-10-13T09:23:42.379 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:42.375+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.377049+0000) 2019-10-13T09:23:42.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:42.377+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.378523+0000) 2019-10-13T09:23:42.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:42.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:42.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:42.919+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.920023+0000) 2019-10-13T09:23:42.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:42.918+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:42.919422+0000) 2019-10-13T09:23:43.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:43.661+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:43.661932+0000) 2019-10-13T09:23:44.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:44.673+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:44.674832+0000) 2019-10-13T09:23:47.385 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:47.384+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.385338+0000) 2019-10-13T09:23:47.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:47.383+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.384674+0000) 2019-10-13T09:23:47.386 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:47.383+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.384865+0000) 2019-10-13T09:23:47.386 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:47.383+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.385016+0000) 2019-10-13T09:23:47.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:47.383+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.385374+0000) 2019-10-13T09:23:47.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:47.384+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.385917+0000) 2019-10-13T09:23:47.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:47.384+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.386062+0000) 2019-10-13T09:23:47.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:47.387+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.388719+0000) 2019-10-13T09:23:47.392 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:47.389+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:48.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:48.474+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:48.475882+0000) 2019-10-13T09:23:48.816 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:48.813+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:48.814686+0000) 2019-10-13T09:23:48.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:48.813+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:48.814730+0000) 2019-10-13T09:23:48.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:48.813+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:48.814745+0000) 2019-10-13T09:23:49.943 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:23:49.943 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:49.947 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:23:49.948 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:52.394 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:52.391+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.393351+0000) 2019-10-13T09:23:52.395 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:52.392+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.393585+0000) 2019-10-13T09:23:52.395 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:52.393+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.394440+0000) 2019-10-13T09:23:52.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:52.392+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.394315+0000) 2019-10-13T09:23:52.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:52.393+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.394508+0000) 2019-10-13T09:23:52.396 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:52.393+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.394553+0000) 2019-10-13T09:23:52.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:52.396+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.396826+0000) 2019-10-13T09:23:52.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:52.393+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.394769+0000) 2019-10-13T09:23:52.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:52.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:52.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:52.919+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.920182+0000) 2019-10-13T09:23:52.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:52.918+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:52.919562+0000) 2019-10-13T09:23:53.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:53.663+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:53.664290+0000) 2019-10-13T09:23:54.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:54.674+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:54.675001+0000) 2019-10-13T09:23:57.403 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:23:57.402+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.403103+0000) 2019-10-13T09:23:57.404 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:23:57.401+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.402473+0000) 2019-10-13T09:23:57.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:57.401+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.402811+0000) 2019-10-13T09:23:57.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:57.401+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.403014+0000) 2019-10-13T09:23:57.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:57.401+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.403146+0000) 2019-10-13T09:23:57.405 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:57.402+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.403393+0000) 2019-10-13T09:23:57.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:23:57.402+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.403769+0000) 2019-10-13T09:23:57.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:23:57.406+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.407017+0000) 2019-10-13T09:23:57.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:23:57.408+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:23:58.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:23:58.475+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:58.476073+0000) 2019-10-13T09:23:58.816 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:23:58.814+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:58.814831+0000) 2019-10-13T09:23:58.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:23:58.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:58.814830+0000) 2019-10-13T09:23:58.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:23:58.814+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:58.814854+0000) 2019-10-13T09:24:02.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:02.410+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.411822+0000) 2019-10-13T09:24:02.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:02.410+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.411999+0000) 2019-10-13T09:24:02.414 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:02.412+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.413476+0000) 2019-10-13T09:24:02.414 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:02.411+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.412436+0000) 2019-10-13T09:24:02.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:02.411+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.412780+0000) 2019-10-13T09:24:02.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:02.411+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.413084+0000) 2019-10-13T09:24:02.415 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:02.411+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.413202+0000) 2019-10-13T09:24:02.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:02.414+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.415681+0000) 2019-10-13T09:24:02.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:02.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:02.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:02.919+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.920293+0000) 2019-10-13T09:24:02.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:02.919+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:02.919711+0000) 2019-10-13T09:24:03.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:03.666+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:03.666825+0000) 2019-10-13T09:24:04.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:04.674+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:04.675189+0000) 2019-10-13T09:24:07.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:07.420+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.420790+0000) 2019-10-13T09:24:07.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:07.420+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.421143+0000) 2019-10-13T09:24:07.422 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:07.420+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.421050+0000) 2019-10-13T09:24:07.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:07.421+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.422155+0000) 2019-10-13T09:24:07.423 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:07.421+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.421610+0000) 2019-10-13T09:24:07.424 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:07.421+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.421682+0000) 2019-10-13T09:24:07.424 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:07.421+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.422133+0000) 2019-10-13T09:24:07.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:07.425+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.425747+0000) 2019-10-13T09:24:07.428 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:07.426+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164775 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:08.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:08.475+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:08.476294+0000) 2019-10-13T09:24:08.816 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:08.814+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:08.815030+0000) 2019-10-13T09:24:08.816 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:08.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:08.815029+0000) 2019-10-13T09:24:08.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:08.814+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:08.815030+0000) 2019-10-13T09:24:12.430 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:12.428+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.428874+0000) 2019-10-13T09:24:12.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:12.428+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.428851+0000) 2019-10-13T09:24:12.430 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:12.428+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.429146+0000) 2019-10-13T09:24:12.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:12.429+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.429528+0000) 2019-10-13T09:24:12.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:12.429+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.430052+0000) 2019-10-13T09:24:12.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:12.429+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.429779+0000) 2019-10-13T09:24:12.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:12.429+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.429710+0000) 2019-10-13T09:24:12.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:12.431+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.432376+0000) 2019-10-13T09:24:12.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:12.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164805 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:12.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:12.920+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.920429+0000) 2019-10-13T09:24:12.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:12.919+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:12.919830+0000) 2019-10-13T09:24:13.669 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:13.668+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:13.669106+0000) 2019-10-13T09:24:14.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:14.675+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:14.675439+0000) 2019-10-13T09:24:17.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:17.435+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.436043+0000) 2019-10-13T09:24:17.437 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:17.435+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.436201+0000) 2019-10-13T09:24:17.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:17.435+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.436346+0000) 2019-10-13T09:24:17.438 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:17.436+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.436610+0000) 2019-10-13T09:24:17.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:17.437+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.437965+0000) 2019-10-13T09:24:17.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:17.436+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.437024+0000) 2019-10-13T09:24:17.439 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:17.436+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.437242+0000) 2019-10-13T09:24:17.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:17.438+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.439550+0000) 2019-10-13T09:24:17.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:17.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:18.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:18.476+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:18.476486+0000) 2019-10-13T09:24:18.816 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:18.814+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:18.815202+0000) 2019-10-13T09:24:18.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:18.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:18.815188+0000) 2019-10-13T09:24:18.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:18.814+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:18.815204+0000) 2019-10-13T09:24:20.088 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:24:20.088 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:20.092 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:24:20.092 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:22.445 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:22.443+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.443856+0000) 2019-10-13T09:24:22.446 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:22.443+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.444123+0000) 2019-10-13T09:24:22.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:22.443+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.444362+0000) 2019-10-13T09:24:22.446 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:22.443+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.444310+0000) 2019-10-13T09:24:22.447 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:22.443+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.445390+0000) 2019-10-13T09:24:22.447 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:22.444+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.444941+0000) 2019-10-13T09:24:22.447 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:22.444+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.445048+0000) 2019-10-13T09:24:22.448 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:22.446+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.448293+0000) 2019-10-13T09:24:22.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:22.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:22.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:22.918+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.920607+0000) 2019-10-13T09:24:22.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:22.919+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:22.919995+0000) 2019-10-13T09:24:23.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:23.670+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:23.671498+0000) 2019-10-13T09:24:24.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:24.675+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:24.675627+0000) 2019-10-13T09:24:27.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:27.451+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.453034+0000) 2019-10-13T09:24:27.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:27.451+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.452394+0000) 2019-10-13T09:24:27.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:27.451+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.452663+0000) 2019-10-13T09:24:27.455 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:27.451+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.452740+0000) 2019-10-13T09:24:27.455 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:27.451+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.452915+0000) 2019-10-13T09:24:27.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:27.451+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.453133+0000) 2019-10-13T09:24:27.456 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:27.451+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.453238+0000) 2019-10-13T09:24:27.457 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:27.454+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.456571+0000) 2019-10-13T09:24:27.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:27.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:28.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:28.475+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:28.476658+0000) 2019-10-13T09:24:28.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:28.814+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:28.815382+0000) 2019-10-13T09:24:28.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:28.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:28.815368+0000) 2019-10-13T09:24:28.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:28.814+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:28.815425+0000) 2019-10-13T09:24:32.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:32.458+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.459789+0000) 2019-10-13T09:24:32.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:32.458+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.460155+0000) 2019-10-13T09:24:32.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:32.460+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.461251+0000) 2019-10-13T09:24:32.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:32.459+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.460449+0000) 2019-10-13T09:24:32.462 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:32.459+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.460704+0000) 2019-10-13T09:24:32.463 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:32.459+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.461028+0000) 2019-10-13T09:24:32.463 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:32.459+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.460809+0000) 2019-10-13T09:24:32.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:32.461+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.463660+0000) 2019-10-13T09:24:32.467 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:32.465+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164927 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:32.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:32.920+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.920772+0000) 2019-10-13T09:24:32.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:32.919+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:32.920094+0000) 2019-10-13T09:24:33.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:33.673+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:33.673918+0000) 2019-10-13T09:24:34.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:34.675+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:34.675827+0000) 2019-10-13T09:24:37.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:37.467+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.468820+0000) 2019-10-13T09:24:37.470 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:37.467+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.469031+0000) 2019-10-13T09:24:37.470 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:37.467+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.469171+0000) 2019-10-13T09:24:37.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:37.467+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.469317+0000) 2019-10-13T09:24:37.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:37.470+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.470938+0000) 2019-10-13T09:24:37.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:37.468+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.470023+0000) 2019-10-13T09:24:37.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:37.468+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.470331+0000) 2019-10-13T09:24:37.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:37.471+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.472245+0000) 2019-10-13T09:24:37.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:37.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:38.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:38.475+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:38.476830+0000) 2019-10-13T09:24:38.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:38.814+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:38.815577+0000) 2019-10-13T09:24:38.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:38.814+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:38.815563+0000) 2019-10-13T09:24:38.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:38.814+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:38.815595+0000) 2019-10-13T09:24:42.479 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:42.475+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.477375+0000) 2019-10-13T09:24:42.479 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:42.476+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.477489+0000) 2019-10-13T09:24:42.479 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:42.476+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.477794+0000) 2019-10-13T09:24:42.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:42.476+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.478090+0000) 2019-10-13T09:24:42.480 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:42.477+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.478435+0000) 2019-10-13T09:24:42.480 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:42.476+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.478280+0000) 2019-10-13T09:24:42.481 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:42.477+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.478438+0000) 2019-10-13T09:24:42.481 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:42.480+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.481473+0000) 2019-10-13T09:24:42.485 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:42.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 164989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:42.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:42.920+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.920940+0000) 2019-10-13T09:24:42.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:42.919+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:42.920240+0000) 2019-10-13T09:24:43.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:43.675+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:43.676352+0000) 2019-10-13T09:24:44.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:44.674+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:44.676031+0000) 2019-10-13T09:24:47.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:47.484+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.485946+0000) 2019-10-13T09:24:47.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:47.484+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.486097+0000) 2019-10-13T09:24:47.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:47.484+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.486209+0000) 2019-10-13T09:24:47.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:47.485+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.486502+0000) 2019-10-13T09:24:47.489 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:47.486+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.487566+0000) 2019-10-13T09:24:47.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:47.485+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.487050+0000) 2019-10-13T09:24:47.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:47.486+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.487621+0000) 2019-10-13T09:24:47.490 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:47.489+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.490494+0000) 2019-10-13T09:24:47.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:47.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:48.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:48.476+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:48.477022+0000) 2019-10-13T09:24:48.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:48.815+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:48.815732+0000) 2019-10-13T09:24:48.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:48.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:48.815725+0000) 2019-10-13T09:24:48.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:48.815+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:48.815759+0000) 2019-10-13T09:24:50.260 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:24:50.260 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:50.264 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:24:50.264 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:52.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:52.493+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.494159+0000) 2019-10-13T09:24:52.495 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:52.494+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.494546+0000) 2019-10-13T09:24:52.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:52.494+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.495138+0000) 2019-10-13T09:24:52.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:52.494+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.494875+0000) 2019-10-13T09:24:52.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:52.494+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.495014+0000) 2019-10-13T09:24:52.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:52.494+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.495334+0000) 2019-10-13T09:24:52.497 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:52.495+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.495456+0000) 2019-10-13T09:24:52.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:52.497+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.498658+0000) 2019-10-13T09:24:52.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:52.499+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:52.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:52.920+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.921037+0000) 2019-10-13T09:24:52.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:52.920+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:52.920418+0000) 2019-10-13T09:24:53.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:53.678+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:53.678748+0000) 2019-10-13T09:24:54.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:54.675+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:54.676223+0000) 2019-10-13T09:24:57.503 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:24:57.502+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.502871+0000) 2019-10-13T09:24:57.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:24:57.501+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.502386+0000) 2019-10-13T09:24:57.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:24:57.502+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.502406+0000) 2019-10-13T09:24:57.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:57.502+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.502745+0000) 2019-10-13T09:24:57.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:57.502+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.502869+0000) 2019-10-13T09:24:57.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:57.502+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.503135+0000) 2019-10-13T09:24:57.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:57.502+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.503265+0000) 2019-10-13T09:24:57.505 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:24:57.503+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.504684+0000) 2019-10-13T09:24:57.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:24:57.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:24:58.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:24:58.476+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:58.477212+0000) 2019-10-13T09:24:58.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:24:58.815+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:58.815960+0000) 2019-10-13T09:24:58.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:24:58.815+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:58.815954+0000) 2019-10-13T09:24:58.817 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:24:58.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:58.815950+0000) 2019-10-13T09:25:02.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:02.510+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.510419+0000) 2019-10-13T09:25:02.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:02.510+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.510532+0000) 2019-10-13T09:25:02.512 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:02.510+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.510753+0000) 2019-10-13T09:25:02.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:02.510+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.510993+0000) 2019-10-13T09:25:02.513 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:02.511+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.511941+0000) 2019-10-13T09:25:02.513 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:02.510+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.511378+0000) 2019-10-13T09:25:02.514 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:02.511+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.511630+0000) 2019-10-13T09:25:02.514 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:02.513+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.514700+0000) 2019-10-13T09:25:02.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:02.517+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:02.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:02.921+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.921192+0000) 2019-10-13T09:25:02.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:02.920+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:02.920585+0000) 2019-10-13T09:25:03.681 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:03.680+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:03.681058+0000) 2019-10-13T09:25:04.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:04.676+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:04.676435+0000) 2019-10-13T09:25:07.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:07.518+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.519355+0000) 2019-10-13T09:25:07.521 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:07.519+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.519586+0000) 2019-10-13T09:25:07.521 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:07.520+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.521164+0000) 2019-10-13T09:25:07.521 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:07.520+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.520466+0000) 2019-10-13T09:25:07.522 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:07.520+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.521048+0000) 2019-10-13T09:25:07.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:07.520+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.521117+0000) 2019-10-13T09:25:07.522 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:07.520+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.521244+0000) 2019-10-13T09:25:07.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:07.523+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.523899+0000) 2019-10-13T09:25:07.527 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:07.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:08.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:08.477+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:08.477423+0000) 2019-10-13T09:25:08.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:08.816+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:08.816169+0000) 2019-10-13T09:25:08.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:08.816+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:08.816168+0000) 2019-10-13T09:25:08.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:08.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:08.816154+0000) 2019-10-13T09:25:12.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:12.526+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.528297+0000) 2019-10-13T09:25:12.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:12.527+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.528728+0000) 2019-10-13T09:25:12.530 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:12.527+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.529698+0000) 2019-10-13T09:25:12.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:12.527+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.529094+0000) 2019-10-13T09:25:12.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:12.527+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.529302+0000) 2019-10-13T09:25:12.531 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:12.528+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.529601+0000) 2019-10-13T09:25:12.531 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:12.528+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.529512+0000) 2019-10-13T09:25:12.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:12.529+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.531462+0000) 2019-10-13T09:25:12.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:12.533+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:12.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:12.920+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.921341+0000) 2019-10-13T09:25:12.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:12.919+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:12.920720+0000) 2019-10-13T09:25:13.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:13.683+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:13.684181+0000) 2019-10-13T09:25:14.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:14.675+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:14.676623+0000) 2019-10-13T09:25:17.537 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:17.535+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.536919+0000) 2019-10-13T09:25:17.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:17.535+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.536463+0000) 2019-10-13T09:25:17.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:17.535+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.536586+0000) 2019-10-13T09:25:17.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:17.535+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.536975+0000) 2019-10-13T09:25:17.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:17.535+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.537147+0000) 2019-10-13T09:25:17.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:17.536+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.537391+0000) 2019-10-13T09:25:17.539 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:17.536+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.537622+0000) 2019-10-13T09:25:17.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:17.537+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.539723+0000) 2019-10-13T09:25:17.544 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:17.541+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:18.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:18.476+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:18.477569+0000) 2019-10-13T09:25:18.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:18.815+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:18.816369+0000) 2019-10-13T09:25:18.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:18.815+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:18.816376+0000) 2019-10-13T09:25:18.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:18.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:18.816344+0000) 2019-10-13T09:25:20.431 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:25:20.431 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:20.435 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:25:20.435 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:22.546 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:22.543+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.545664+0000) 2019-10-13T09:25:22.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:22.543+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.544718+0000) 2019-10-13T09:25:22.547 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:22.543+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.545046+0000) 2019-10-13T09:25:22.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:22.543+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.545198+0000) 2019-10-13T09:25:22.547 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:22.544+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.545544+0000) 2019-10-13T09:25:22.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:22.544+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.545789+0000) 2019-10-13T09:25:22.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:22.544+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.545673+0000) 2019-10-13T09:25:22.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:22.546+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.548211+0000) 2019-10-13T09:25:22.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:22.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165233 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:22.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:22.920+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.921535+0000) 2019-10-13T09:25:22.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:22.920+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:22.920912+0000) 2019-10-13T09:25:23.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:23.685+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:23.686470+0000) 2019-10-13T09:25:24.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:24.676+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:24.676768+0000) 2019-10-13T09:25:27.555 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:27.552+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.554349+0000) 2019-10-13T09:25:27.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:27.553+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.554444+0000) 2019-10-13T09:25:27.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:27.553+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.554773+0000) 2019-10-13T09:25:27.557 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:27.554+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.555546+0000) 2019-10-13T09:25:27.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:27.553+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.555069+0000) 2019-10-13T09:25:27.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:27.553+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.555166+0000) 2019-10-13T09:25:27.557 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:27.554+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.555480+0000) 2019-10-13T09:25:27.558 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:27.557+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.558337+0000) 2019-10-13T09:25:27.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:27.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165263 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:28.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:28.477+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:28.477742+0000) 2019-10-13T09:25:28.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:28.815+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:28.816573+0000) 2019-10-13T09:25:28.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:28.815+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:28.816572+0000) 2019-10-13T09:25:28.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:28.815+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:28.816570+0000) 2019-10-13T09:25:32.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:32.561+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.562848+0000) 2019-10-13T09:25:32.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:32.561+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.563022+0000) 2019-10-13T09:25:32.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:32.561+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.563217+0000) 2019-10-13T09:25:32.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:32.563+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.564322+0000) 2019-10-13T09:25:32.565 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:32.562+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.563789+0000) 2019-10-13T09:25:32.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:32.562+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.563963+0000) 2019-10-13T09:25:32.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:32.562+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.564249+0000) 2019-10-13T09:25:32.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:32.566+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.566926+0000) 2019-10-13T09:25:32.571 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:32.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165293 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:32.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:32.921+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.921721+0000) 2019-10-13T09:25:32.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:32.920+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:32.921093+0000) 2019-10-13T09:25:33.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:33.688+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:33.688970+0000) 2019-10-13T09:25:34.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:34.676+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:34.676952+0000) 2019-10-13T09:25:37.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:37.570+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.572020+0000) 2019-10-13T09:25:37.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:37.570+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.571986+0000) 2019-10-13T09:25:37.574 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:37.570+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.572337+0000) 2019-10-13T09:25:37.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:37.571+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.572576+0000) 2019-10-13T09:25:37.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:37.571+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.572739+0000) 2019-10-13T09:25:37.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:37.571+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.572833+0000) 2019-10-13T09:25:37.575 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:37.573+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.574002+0000) 2019-10-13T09:25:37.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:37.574+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.575091+0000) 2019-10-13T09:25:37.579 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:37.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:38.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:38.477+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:38.477855+0000) 2019-10-13T09:25:38.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:38.816+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:38.816763+0000) 2019-10-13T09:25:38.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:38.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:38.816770+0000) 2019-10-13T09:25:38.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:38.816+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:38.816782+0000) 2019-10-13T09:25:42.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:42.579+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.579836+0000) 2019-10-13T09:25:42.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:42.579+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.580105+0000) 2019-10-13T09:25:42.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:42.579+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.580377+0000) 2019-10-13T09:25:42.582 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:42.580+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.581264+0000) 2019-10-13T09:25:42.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:42.580+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.580992+0000) 2019-10-13T09:25:42.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:42.580+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.581141+0000) 2019-10-13T09:25:42.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:42.581+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.581423+0000) 2019-10-13T09:25:42.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:42.582+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.583522+0000) 2019-10-13T09:25:42.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:42.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:42.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:42.921+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.921856+0000) 2019-10-13T09:25:42.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:42.920+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:42.921251+0000) 2019-10-13T09:25:43.691 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:43.690+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:43.691467+0000) 2019-10-13T09:25:44.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:44.676+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:44.677180+0000) 2019-10-13T09:25:47.590 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:47.588+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.589102+0000) 2019-10-13T09:25:47.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:47.589+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.590107+0000) 2019-10-13T09:25:47.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:47.589+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.589603+0000) 2019-10-13T09:25:47.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:47.589+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.589881+0000) 2019-10-13T09:25:47.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:47.589+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.589740+0000) 2019-10-13T09:25:47.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:47.589+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.589990+0000) 2019-10-13T09:25:47.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:47.589+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.590282+0000) 2019-10-13T09:25:47.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:47.592+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.593526+0000) 2019-10-13T09:25:47.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:47.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:48.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:48.477+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:48.478016+0000) 2019-10-13T09:25:48.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:48.816+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:48.816942+0000) 2019-10-13T09:25:48.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:48.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:48.816953+0000) 2019-10-13T09:25:48.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:48.816+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:48.816965+0000) 2019-10-13T09:25:50.601 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:25:50.602 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:50.606 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:25:50.606 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:52.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:52.597+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.597455+0000) 2019-10-13T09:25:52.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:52.597+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.598665+0000) 2019-10-13T09:25:52.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:52.597+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.597962+0000) 2019-10-13T09:25:52.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:52.597+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.598206+0000) 2019-10-13T09:25:52.600 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:52.598+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.598515+0000) 2019-10-13T09:25:52.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:52.598+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.598700+0000) 2019-10-13T09:25:52.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:52.598+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.598976+0000) 2019-10-13T09:25:52.601 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:52.600+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.600734+0000) 2019-10-13T09:25:52.605 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:52.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:52.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:52.921+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.922041+0000) 2019-10-13T09:25:52.922 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:52.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.921422+0000) 2019-10-13T09:25:53.694 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:53.693+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:53.693926+0000) 2019-10-13T09:25:54.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:54.677+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:54.677377+0000) 2019-10-13T09:25:57.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:25:57.606+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.607558+0000) 2019-10-13T09:25:57.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:57.605+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.606568+0000) 2019-10-13T09:25:57.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:57.605+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.606910+0000) 2019-10-13T09:25:57.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:25:57.605+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.607057+0000) 2019-10-13T09:25:57.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:57.605+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.607379+0000) 2019-10-13T09:25:57.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:57.606+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.607427+0000) 2019-10-13T09:25:57.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:25:57.606+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.608110+0000) 2019-10-13T09:25:57.610 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:25:57.609+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.610214+0000) 2019-10-13T09:25:57.614 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:25:57.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165447 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:25:58.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:25:58.477+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:58.478181+0000) 2019-10-13T09:25:58.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:25:58.816+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:58.817141+0000) 2019-10-13T09:25:58.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:25:58.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:58.817147+0000) 2019-10-13T09:25:58.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:25:58.816+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:58.817185+0000) 2019-10-13T09:26:02.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:02.613+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.615150+0000) 2019-10-13T09:26:02.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:02.614+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.615579+0000) 2019-10-13T09:26:02.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:02.614+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.615755+0000) 2019-10-13T09:26:02.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:02.615+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.617326+0000) 2019-10-13T09:26:02.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:02.615+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.616741+0000) 2019-10-13T09:26:02.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:02.615+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.616822+0000) 2019-10-13T09:26:02.618 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:02.615+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.616973+0000) 2019-10-13T09:26:02.619 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:02.616+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.618306+0000) 2019-10-13T09:26:02.626 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:02.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:02.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:02.921+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.922194+0000) 2019-10-13T09:26:02.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:02.920+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.921588+0000) 2019-10-13T09:26:03.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:03.695+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:03.696827+0000) 2019-10-13T09:26:04.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:04.676+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:04.677586+0000) 2019-10-13T09:26:07.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:07.623+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.624582+0000) 2019-10-13T09:26:07.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:07.623+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.624748+0000) 2019-10-13T09:26:07.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:07.623+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.624997+0000) 2019-10-13T09:26:07.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:07.623+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.625248+0000) 2019-10-13T09:26:07.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:07.623+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.625354+0000) 2019-10-13T09:26:07.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:07.624+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.626352+0000) 2019-10-13T09:26:07.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:07.624+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.625626+0000) 2019-10-13T09:26:07.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:07.626+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.628007+0000) 2019-10-13T09:26:07.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:07.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:08.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:08.476+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:08.478364+0000) 2019-10-13T09:26:08.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:08.816+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:08.817344+0000) 2019-10-13T09:26:08.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:08.816+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:08.817341+0000) 2019-10-13T09:26:08.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:08.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:08.817342+0000) 2019-10-13T09:26:12.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:12.632+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.634181+0000) 2019-10-13T09:26:12.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:12.632+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.634332+0000) 2019-10-13T09:26:12.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:12.634+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.636075+0000) 2019-10-13T09:26:12.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:12.633+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.634551+0000) 2019-10-13T09:26:12.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:12.633+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.634697+0000) 2019-10-13T09:26:12.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:12.633+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.634933+0000) 2019-10-13T09:26:12.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:12.633+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.635115+0000) 2019-10-13T09:26:12.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:12.636+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.638043+0000) 2019-10-13T09:26:12.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:12.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:12.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:12.921+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.922351+0000) 2019-10-13T09:26:12.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:12.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.921725+0000) 2019-10-13T09:26:13.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:13.699+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:13.699826+0000) 2019-10-13T09:26:14.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:14.677+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:14.677784+0000) 2019-10-13T09:26:17.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:17.640+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.642065+0000) 2019-10-13T09:26:17.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:17.640+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.642306+0000) 2019-10-13T09:26:17.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:17.640+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.642380+0000) 2019-10-13T09:26:17.644 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:17.641+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.642581+0000) 2019-10-13T09:26:17.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:17.643+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.643793+0000) 2019-10-13T09:26:17.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:17.641+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.642672+0000) 2019-10-13T09:26:17.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:17.641+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.642871+0000) 2019-10-13T09:26:17.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:17.645+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.646536+0000) 2019-10-13T09:26:17.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:17.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165569 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:18.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:18.478+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:18.478569+0000) 2019-10-13T09:26:18.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:18.816+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:18.817508+0000) 2019-10-13T09:26:18.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:18.816+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:18.817507+0000) 2019-10-13T09:26:18.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:18.816+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:18.817510+0000) 2019-10-13T09:26:20.747 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:26:20.748 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:20.752 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:26:20.753 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:22.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:22.651+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.652512+0000) 2019-10-13T09:26:22.654 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:22.649+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.651195+0000) 2019-10-13T09:26:22.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:22.650+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.651431+0000) 2019-10-13T09:26:22.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:22.650+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.651778+0000) 2019-10-13T09:26:22.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:22.650+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.652258+0000) 2019-10-13T09:26:22.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:22.650+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.652308+0000) 2019-10-13T09:26:22.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:22.651+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.652483+0000) 2019-10-13T09:26:22.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:22.654+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.654856+0000) 2019-10-13T09:26:22.660 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:22.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165599 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:22.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:22.921+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.922533+0000) 2019-10-13T09:26:22.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:22.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.921876+0000) 2019-10-13T09:26:23.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:23.701+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:23.702296+0000) 2019-10-13T09:26:24.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:24.677+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:24.678024+0000) 2019-10-13T09:26:27.661 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:27.660+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.661706+0000) 2019-10-13T09:26:27.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:27.660+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.661154+0000) 2019-10-13T09:26:27.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:27.660+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.661319+0000) 2019-10-13T09:26:27.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:27.661+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.661565+0000) 2019-10-13T09:26:27.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:27.661+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.661677+0000) 2019-10-13T09:26:27.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:27.661+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.661913+0000) 2019-10-13T09:26:27.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:27.661+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.662131+0000) 2019-10-13T09:26:27.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:27.663+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.664337+0000) 2019-10-13T09:26:27.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:27.667+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165629 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:28.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:28.478+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:28.478737+0000) 2019-10-13T09:26:28.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:28.817+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:28.817727+0000) 2019-10-13T09:26:28.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:28.817+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:28.817719+0000) 2019-10-13T09:26:28.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:28.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:28.817722+0000) 2019-10-13T09:26:32.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:32.669+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.669656+0000) 2019-10-13T09:26:32.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:32.669+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.670109+0000) 2019-10-13T09:26:32.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:32.669+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.670348+0000) 2019-10-13T09:26:32.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:32.670+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.671319+0000) 2019-10-13T09:26:32.672 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:32.670+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.670773+0000) 2019-10-13T09:26:32.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:32.670+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.670829+0000) 2019-10-13T09:26:32.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:32.670+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.670946+0000) 2019-10-13T09:26:32.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:32.672+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.672762+0000) 2019-10-13T09:26:32.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:32.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165659 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:32.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:32.922+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.922725+0000) 2019-10-13T09:26:32.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:32.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.922055+0000) 2019-10-13T09:26:33.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:33.704+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:33.704767+0000) 2019-10-13T09:26:34.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:34.678+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:34.678224+0000) 2019-10-13T09:26:37.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:37.678+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.678416+0000) 2019-10-13T09:26:37.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:37.678+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.679457+0000) 2019-10-13T09:26:37.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:37.678+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.678957+0000) 2019-10-13T09:26:37.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:37.678+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.679027+0000) 2019-10-13T09:26:37.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:37.680+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.680448+0000) 2019-10-13T09:26:37.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:37.680+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.680636+0000) 2019-10-13T09:26:37.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:37.680+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.680803+0000) 2019-10-13T09:26:37.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:37.683+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.683728+0000) 2019-10-13T09:26:37.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:37.686+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165691 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:38.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:38.478+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:38.478919+0000) 2019-10-13T09:26:38.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:38.817+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:38.817912+0000) 2019-10-13T09:26:38.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:38.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:38.817913+0000) 2019-10-13T09:26:38.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:38.817+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:38.817934+0000) 2019-10-13T09:26:42.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:42.687+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.688672+0000) 2019-10-13T09:26:42.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:42.687+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.688826+0000) 2019-10-13T09:26:42.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:42.687+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.689193+0000) 2019-10-13T09:26:42.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:42.689+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.689901+0000) 2019-10-13T09:26:42.691 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:42.687+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.689327+0000) 2019-10-13T09:26:42.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:42.688+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.689509+0000) 2019-10-13T09:26:42.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:42.688+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.689768+0000) 2019-10-13T09:26:42.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:42.691+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.691758+0000) 2019-10-13T09:26:42.696 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:42.694+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:42.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:42.922+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.922923+0000) 2019-10-13T09:26:42.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:42.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.922247+0000) 2019-10-13T09:26:43.707 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:43.706+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:43.707161+0000) 2019-10-13T09:26:44.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:44.677+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:44.678440+0000) 2019-10-13T09:26:47.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:47.696+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.697579+0000) 2019-10-13T09:26:47.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:47.696+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.697675+0000) 2019-10-13T09:26:47.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:47.696+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.698005+0000) 2019-10-13T09:26:47.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:47.696+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.698169+0000) 2019-10-13T09:26:47.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:47.698+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.699525+0000) 2019-10-13T09:26:47.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:47.701+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.702107+0000) 2019-10-13T09:26:47.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:47.696+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.698377+0000) 2019-10-13T09:26:47.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:47.697+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.698831+0000) 2019-10-13T09:26:47.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:47.703+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165751 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:48.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:48.478+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:48.479058+0000) 2019-10-13T09:26:48.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:48.817+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:48.818127+0000) 2019-10-13T09:26:48.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:48.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:48.818121+0000) 2019-10-13T09:26:48.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:48.817+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:48.818124+0000) 2019-10-13T09:26:50.902 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:26:50.902 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:50.906 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:26:50.907 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:52.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:52.706+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.707316+0000) 2019-10-13T09:26:52.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:52.705+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.706719+0000) 2019-10-13T09:26:52.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:52.705+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.706869+0000) 2019-10-13T09:26:52.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:52.705+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.707059+0000) 2019-10-13T09:26:52.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:52.705+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.707277+0000) 2019-10-13T09:26:52.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:52.706+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.707528+0000) 2019-10-13T09:26:52.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:52.706+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.707700+0000) 2019-10-13T09:26:52.714 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:52.712+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165780 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:52.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:52.725+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.725796+0000) 2019-10-13T09:26:52.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:52.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.923056+0000) 2019-10-13T09:26:52.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:52.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.922404+0000) 2019-10-13T09:26:53.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:53.709+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:53.709465+0000) 2019-10-13T09:26:54.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:54.677+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:54.678628+0000) 2019-10-13T09:26:57.717 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:57.714+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.715691+0000) 2019-10-13T09:26:57.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:26:57.714+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.715593+0000) 2019-10-13T09:26:57.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:57.714+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.716061+0000) 2019-10-13T09:26:57.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:57.714+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.716153+0000) 2019-10-13T09:26:57.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:57.714+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.716313+0000) 2019-10-13T09:26:57.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:26:57.714+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.716710+0000) 2019-10-13T09:26:57.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:26:57.718+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.719929+0000) 2019-10-13T09:26:57.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:26:57.715+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.716595+0000) 2019-10-13T09:26:57.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:26:57.721+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:26:58.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:26:58.478+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:58.479228+0000) 2019-10-13T09:26:58.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:26:58.817+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:58.818284+0000) 2019-10-13T09:26:58.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:26:58.817+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:58.818286+0000) 2019-10-13T09:26:58.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:26:58.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:58.818278+0000) 2019-10-13T09:27:02.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:02.722+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.724010+0000) 2019-10-13T09:27:02.725 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:02.722+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.724108+0000) 2019-10-13T09:27:02.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:02.722+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.724372+0000) 2019-10-13T09:27:02.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:02.723+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.725088+0000) 2019-10-13T09:27:02.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:02.723+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.724545+0000) 2019-10-13T09:27:02.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:02.723+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.724862+0000) 2019-10-13T09:27:02.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:02.723+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.725194+0000) 2019-10-13T09:27:02.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:02.725+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.727155+0000) 2019-10-13T09:27:02.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:02.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:02.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:02.922+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.923193+0000) 2019-10-13T09:27:02.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:02.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.922505+0000) 2019-10-13T09:27:03.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:03.711+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:03.711923+0000) 2019-10-13T09:27:04.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:04.678+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:04.678822+0000) 2019-10-13T09:27:07.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:07.729+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.731105+0000) 2019-10-13T09:27:07.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:07.729+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.731224+0000) 2019-10-13T09:27:07.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:07.730+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.731671+0000) 2019-10-13T09:27:07.733 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:07.730+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.731782+0000) 2019-10-13T09:27:07.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:07.732+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.733124+0000) 2019-10-13T09:27:07.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:07.730+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.732021+0000) 2019-10-13T09:27:07.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:07.731+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.732488+0000) 2019-10-13T09:27:07.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:07.734+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.735397+0000) 2019-10-13T09:27:07.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:07.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:08.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:08.478+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:08.479410+0000) 2019-10-13T09:27:08.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:08.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:08.818456+0000) 2019-10-13T09:27:08.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:08.818+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:08.818475+0000) 2019-10-13T09:27:08.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:08.818+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:08.818494+0000) 2019-10-13T09:27:12.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:12.739+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.739756+0000) 2019-10-13T09:27:12.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:12.739+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.739947+0000) 2019-10-13T09:27:12.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:12.739+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.740116+0000) 2019-10-13T09:27:12.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:12.740+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.741300+0000) 2019-10-13T09:27:12.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:12.740+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.740555+0000) 2019-10-13T09:27:12.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:12.740+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.740683+0000) 2019-10-13T09:27:12.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:12.740+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.740970+0000) 2019-10-13T09:27:12.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:12.742+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.743359+0000) 2019-10-13T09:27:12.747 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:12.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:12.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:12.922+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.923371+0000) 2019-10-13T09:27:12.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:12.922+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.922710+0000) 2019-10-13T09:27:13.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:13.713+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:13.714223+0000) 2019-10-13T09:27:14.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:14.678+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:14.679030+0000) 2019-10-13T09:27:17.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:17.747+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.747805+0000) 2019-10-13T09:27:17.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:17.747+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.747903+0000) 2019-10-13T09:27:17.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:17.747+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.748034+0000) 2019-10-13T09:27:17.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:17.747+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.748186+0000) 2019-10-13T09:27:17.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:17.748+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.748414+0000) 2019-10-13T09:27:17.750 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:17.748+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.749471+0000) 2019-10-13T09:27:17.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:17.748+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.748707+0000) 2019-10-13T09:27:17.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:17.751+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.751764+0000) 2019-10-13T09:27:17.754 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:17.752+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:18.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:18.479+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:18.479584+0000) 2019-10-13T09:27:18.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:18.818+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:18.818648+0000) 2019-10-13T09:27:18.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:18.818+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:18.818638+0000) 2019-10-13T09:27:18.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:18.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:18.818635+0000) 2019-10-13T09:27:21.011 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:27:21.011 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:21.015 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:27:21.016 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:22.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:22.754+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.754903+0000) 2019-10-13T09:27:22.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:22.754+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.755280+0000) 2019-10-13T09:27:22.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:22.755+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.756262+0000) 2019-10-13T09:27:22.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:22.755+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.755661+0000) 2019-10-13T09:27:22.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:22.755+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.755711+0000) 2019-10-13T09:27:22.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:22.755+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.756075+0000) 2019-10-13T09:27:22.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:22.755+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.756291+0000) 2019-10-13T09:27:22.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:22.758+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.758870+0000) 2019-10-13T09:27:22.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:22.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:22.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:22.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.923530+0000) 2019-10-13T09:27:22.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:22.922+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.922873+0000) 2019-10-13T09:27:23.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:23.716+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:23.716483+0000) 2019-10-13T09:27:24.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:24.679+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:24.679222+0000) 2019-10-13T09:27:27.765 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:27.762+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.763451+0000) 2019-10-13T09:27:27.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:27.762+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.763720+0000) 2019-10-13T09:27:27.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:27.762+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.763927+0000) 2019-10-13T09:27:27.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:27.763+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.764451+0000) 2019-10-13T09:27:27.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:27.762+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.764194+0000) 2019-10-13T09:27:27.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:27.763+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.764488+0000) 2019-10-13T09:27:27.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:27.763+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.764767+0000) 2019-10-13T09:27:27.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:27.766+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.767635+0000) 2019-10-13T09:27:27.771 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:27.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 165991 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:28.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:28.478+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:28.479720+0000) 2019-10-13T09:27:28.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:28.817+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:28.818741+0000) 2019-10-13T09:27:28.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:28.817+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:28.818782+0000) 2019-10-13T09:27:28.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:28.817+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:28.818800+0000) 2019-10-13T09:27:32.774 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:32.770+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.771981+0000) 2019-10-13T09:27:32.774 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:32.770+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.772136+0000) 2019-10-13T09:27:32.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:32.770+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.771986+0000) 2019-10-13T09:27:32.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:32.771+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.772539+0000) 2019-10-13T09:27:32.774 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:32.771+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.772681+0000) 2019-10-13T09:27:32.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:32.771+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.772819+0000) 2019-10-13T09:27:32.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:32.772+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.773105+0000) 2019-10-13T09:27:32.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:32.774+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.775354+0000) 2019-10-13T09:27:32.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:32.775+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166020 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:32.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:32.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.923687+0000) 2019-10-13T09:27:32.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:32.921+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.923004+0000) 2019-10-13T09:27:33.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:33.718+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:33.718886+0000) 2019-10-13T09:27:34.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:34.678+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:34.679438+0000) 2019-10-13T09:27:37.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:37.777+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.778702+0000) 2019-10-13T09:27:37.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:37.777+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.778939+0000) 2019-10-13T09:27:37.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:37.779+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.779965+0000) 2019-10-13T09:27:37.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:37.778+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.779510+0000) 2019-10-13T09:27:37.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:37.778+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.779573+0000) 2019-10-13T09:27:37.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:37.778+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.779742+0000) 2019-10-13T09:27:37.782 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:37.778+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.780076+0000) 2019-10-13T09:27:37.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:37.782+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.782899+0000) 2019-10-13T09:27:37.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:37.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:38.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:38.479+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:38.479862+0000) 2019-10-13T09:27:38.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:38.818+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:38.818904+0000) 2019-10-13T09:27:38.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:38.818+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:38.818972+0000) 2019-10-13T09:27:38.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:38.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:38.818948+0000) 2019-10-13T09:27:42.789 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:42.786+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.787550+0000) 2019-10-13T09:27:42.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:42.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.787676+0000) 2019-10-13T09:27:42.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:42.786+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.787962+0000) 2019-10-13T09:27:42.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:42.788+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.788967+0000) 2019-10-13T09:27:42.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:42.786+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.788326+0000) 2019-10-13T09:27:42.790 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:42.787+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.788432+0000) 2019-10-13T09:27:42.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:42.787+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.788798+0000) 2019-10-13T09:27:42.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:42.790+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.791650+0000) 2019-10-13T09:27:42.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:42.792+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:42.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:42.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.923830+0000) 2019-10-13T09:27:42.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:42.922+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.923157+0000) 2019-10-13T09:27:43.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:43.720+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:43.721144+0000) 2019-10-13T09:27:44.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:44.678+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:44.679648+0000) 2019-10-13T09:27:47.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:47.793+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.795083+0000) 2019-10-13T09:27:47.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:47.794+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.796424+0000) 2019-10-13T09:27:47.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:47.794+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.795708+0000) 2019-10-13T09:27:47.797 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:47.794+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.795843+0000) 2019-10-13T09:27:47.797 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:47.794+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.796020+0000) 2019-10-13T09:27:47.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:47.794+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.796184+0000) 2019-10-13T09:27:47.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:47.795+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.796820+0000) 2019-10-13T09:27:47.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:47.798+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.799767+0000) 2019-10-13T09:27:47.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:47.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:48.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:48.479+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:48.480031+0000) 2019-10-13T09:27:48.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:48.818+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:48.819155+0000) 2019-10-13T09:27:48.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:48.818+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:48.819134+0000) 2019-10-13T09:27:48.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:48.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:48.819175+0000) 2019-10-13T09:27:51.183 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:27:51.183 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:51.188 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:27:51.188 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:52.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:52.802+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.803644+0000) 2019-10-13T09:27:52.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:52.802+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.803859+0000) 2019-10-13T09:27:52.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:52.802+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.804010+0000) 2019-10-13T09:27:52.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:52.802+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.804274+0000) 2019-10-13T09:27:52.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:52.803+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.804435+0000) 2019-10-13T09:27:52.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:52.803+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.804534+0000) 2019-10-13T09:27:52.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:52.802+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.804615+0000) 2019-10-13T09:27:52.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:52.805+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.807118+0000) 2019-10-13T09:27:52.810 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:52.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:52.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:52.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.923965+0000) 2019-10-13T09:27:52.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:52.922+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.923305+0000) 2019-10-13T09:27:53.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:53.721+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:53.723439+0000) 2019-10-13T09:27:54.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:54.679+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:54.679865+0000) 2019-10-13T09:27:57.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:27:57.809+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.811576+0000) 2019-10-13T09:27:57.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:27:57.810+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.811179+0000) 2019-10-13T09:27:57.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:57.810+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.811248+0000) 2019-10-13T09:27:57.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:57.811+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.811510+0000) 2019-10-13T09:27:57.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:57.811+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.811901+0000) 2019-10-13T09:27:57.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:57.811+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.812253+0000) 2019-10-13T09:27:57.814 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:27:57.812+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.812394+0000) 2019-10-13T09:27:57.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:27:57.813+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.815536+0000) 2019-10-13T09:27:57.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:27:57.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:27:58.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:27:58.479+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:58.480243+0000) 2019-10-13T09:27:58.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:27:58.818+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:58.819326+0000) 2019-10-13T09:27:58.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:27:58.818+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:58.819279+0000) 2019-10-13T09:27:58.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:27:58.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:58.819383+0000) 2019-10-13T09:28:02.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:02.820+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.820865+0000) 2019-10-13T09:28:02.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:02.820+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.821178+0000) 2019-10-13T09:28:02.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:02.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.822676+0000) 2019-10-13T09:28:02.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:02.821+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.821445+0000) 2019-10-13T09:28:02.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:02.821+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.821497+0000) 2019-10-13T09:28:02.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:02.821+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.821654+0000) 2019-10-13T09:28:02.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:02.821+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.822060+0000) 2019-10-13T09:28:02.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:02.824+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.825282+0000) 2019-10-13T09:28:02.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:02.826+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:02.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:02.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.924101+0000) 2019-10-13T09:28:02.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:02.923+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.923450+0000) 2019-10-13T09:28:03.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:03.725+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:03.725928+0000) 2019-10-13T09:28:04.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:04.679+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:04.680062+0000) 2019-10-13T09:28:07.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:07.828+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.828578+0000) 2019-10-13T09:28:07.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:07.828+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.829062+0000) 2019-10-13T09:28:07.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:07.828+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.829135+0000) 2019-10-13T09:28:07.837 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:07.828+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.829300+0000) 2019-10-13T09:28:07.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:07.829+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.829474+0000) 2019-10-13T09:28:07.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:07.829+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.829583+0000) 2019-10-13T09:28:07.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:07.828+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.829288+0000) 2019-10-13T09:28:07.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:07.831+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.832684+0000) 2019-10-13T09:28:07.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:07.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:08.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:08.480+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:08.480420+0000) 2019-10-13T09:28:08.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:08.819+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:08.819484+0000) 2019-10-13T09:28:08.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:08.819+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:08.819579+0000) 2019-10-13T09:28:08.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:08.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:08.819574+0000) 2019-10-13T09:28:12.838 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:12.835+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.837319+0000) 2019-10-13T09:28:12.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:12.838+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.838785+0000) 2019-10-13T09:28:12.839 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:12.836+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.838065+0000) 2019-10-13T09:28:12.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:12.836+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.838191+0000) 2019-10-13T09:28:12.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:12.836+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.838342+0000) 2019-10-13T09:28:12.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:12.837+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.838648+0000) 2019-10-13T09:28:12.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:12.837+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.838920+0000) 2019-10-13T09:28:12.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:12.840+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.840814+0000) 2019-10-13T09:28:12.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:12.843+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166260 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:12.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:12.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.924270+0000) 2019-10-13T09:28:12.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:12.922+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.923595+0000) 2019-10-13T09:28:13.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:13.727+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:13.728294+0000) 2019-10-13T09:28:14.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:14.679+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:14.680258+0000) 2019-10-13T09:28:17.848 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:17.845+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.846595+0000) 2019-10-13T09:28:17.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:17.845+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.846897+0000) 2019-10-13T09:28:17.849 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:17.845+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.847143+0000) 2019-10-13T09:28:17.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:17.848+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.848813+0000) 2019-10-13T09:28:17.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:17.846+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.847570+0000) 2019-10-13T09:28:17.851 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:17.846+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.847706+0000) 2019-10-13T09:28:17.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:17.847+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.848483+0000) 2019-10-13T09:28:17.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:17.849+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.849790+0000) 2019-10-13T09:28:17.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:17.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166292 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:18.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:18.479+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:18.480587+0000) 2019-10-13T09:28:18.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:18.818+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:18.819642+0000) 2019-10-13T09:28:18.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:18.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:18.819680+0000) 2019-10-13T09:28:18.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:18.818+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:18.819735+0000) 2019-10-13T09:28:21.398 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:28:21.398 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:21.402 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:28:21.402 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:22.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:22.855+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.856614+0000) 2019-10-13T09:28:22.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:22.854+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.855899+0000) 2019-10-13T09:28:22.857 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:22.854+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.856016+0000) 2019-10-13T09:28:22.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:22.854+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.856177+0000) 2019-10-13T09:28:22.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:22.855+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.856429+0000) 2019-10-13T09:28:22.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:22.855+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.856502+0000) 2019-10-13T09:28:22.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:22.855+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.856667+0000) 2019-10-13T09:28:22.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:22.857+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.858599+0000) 2019-10-13T09:28:22.863 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:22.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166322 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:22.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:22.924+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.924413+0000) 2019-10-13T09:28:22.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:22.922+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.923746+0000) 2019-10-13T09:28:23.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:23.731+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:23.731381+0000) 2019-10-13T09:28:24.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:24.679+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:24.680448+0000) 2019-10-13T09:28:27.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:27.862+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.863926+0000) 2019-10-13T09:28:27.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:27.862+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.864155+0000) 2019-10-13T09:28:27.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:27.864+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.865447+0000) 2019-10-13T09:28:27.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:27.862+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.864351+0000) 2019-10-13T09:28:27.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:27.863+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.864627+0000) 2019-10-13T09:28:27.866 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:27.863+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.864749+0000) 2019-10-13T09:28:27.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:27.863+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.865097+0000) 2019-10-13T09:28:27.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:27.867+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.867822+0000) 2019-10-13T09:28:27.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:27.869+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:28.481 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:28.480+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:28.480755+0000) 2019-10-13T09:28:28.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:28.819+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:28.819802+0000) 2019-10-13T09:28:28.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:28.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:28.819804+0000) 2019-10-13T09:28:28.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:28.819+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:28.819873+0000) 2019-10-13T09:28:32.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:32.871+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.872610+0000) 2019-10-13T09:28:32.874 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:32.873+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.874013+0000) 2019-10-13T09:28:32.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:32.871+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.873023+0000) 2019-10-13T09:28:32.875 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:32.871+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.873343+0000) 2019-10-13T09:28:32.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:32.872+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.873495+0000) 2019-10-13T09:28:32.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:32.872+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.873716+0000) 2019-10-13T09:28:32.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:32.872+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.874207+0000) 2019-10-13T09:28:32.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:32.876+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.877577+0000) 2019-10-13T09:28:32.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:32.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166381 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:32.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:32.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.924559+0000) 2019-10-13T09:28:32.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:32.923+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.923887+0000) 2019-10-13T09:28:33.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:33.733+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:33.733795+0000) 2019-10-13T09:28:34.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:34.680+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:34.680639+0000) 2019-10-13T09:28:37.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:37.880+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.881795+0000) 2019-10-13T09:28:37.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:37.881+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.882755+0000) 2019-10-13T09:28:37.884 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:37.880+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.882119+0000) 2019-10-13T09:28:37.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:37.880+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.882295+0000) 2019-10-13T09:28:37.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:37.881+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.882619+0000) 2019-10-13T09:28:37.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:37.881+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.882776+0000) 2019-10-13T09:28:37.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:37.881+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.883027+0000) 2019-10-13T09:28:37.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:37.883+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.885413+0000) 2019-10-13T09:28:37.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:37.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166412 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:38.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:38.480+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:38.480934+0000) 2019-10-13T09:28:38.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:38.819+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:38.819936+0000) 2019-10-13T09:28:38.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:38.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:38.819933+0000) 2019-10-13T09:28:38.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:38.819+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:38.819990+0000) 2019-10-13T09:28:42.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:42.889+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.891345+0000) 2019-10-13T09:28:42.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:42.890+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.890939+0000) 2019-10-13T09:28:42.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:42.890+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.891212+0000) 2019-10-13T09:28:42.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:42.891+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.891505+0000) 2019-10-13T09:28:42.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:42.891+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.891637+0000) 2019-10-13T09:28:42.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:42.891+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.891754+0000) 2019-10-13T09:28:42.894 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:42.891+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.891886+0000) 2019-10-13T09:28:42.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:42.891+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.893662+0000) 2019-10-13T09:28:42.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:42.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166443 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:42.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:42.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.924672+0000) 2019-10-13T09:28:42.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:42.923+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.923991+0000) 2019-10-13T09:28:43.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:43.735+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:43.736317+0000) 2019-10-13T09:28:44.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:44.680+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:44.680824+0000) 2019-10-13T09:28:47.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:47.899+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.899608+0000) 2019-10-13T09:28:47.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:47.899+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.900022+0000) 2019-10-13T09:28:47.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:47.899+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.901276+0000) 2019-10-13T09:28:47.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:47.900+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.900628+0000) 2019-10-13T09:28:47.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:47.900+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.900921+0000) 2019-10-13T09:28:47.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:47.900+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.901183+0000) 2019-10-13T09:28:47.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:47.901+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.901408+0000) 2019-10-13T09:28:47.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:47.901+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.903133+0000) 2019-10-13T09:28:47.908 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:47.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166471 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:48.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:48.480+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:48.481073+0000) 2019-10-13T09:28:48.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:48.819+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:48.820077+0000) 2019-10-13T09:28:48.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:48.819+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:48.820150+0000) 2019-10-13T09:28:48.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:48.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:48.820075+0000) 2019-10-13T09:28:51.629 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:28:51.629 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:51.635 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:28:51.635 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:52.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:52.908+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.908920+0000) 2019-10-13T09:28:52.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:52.908+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.909192+0000) 2019-10-13T09:28:52.911 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:52.908+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.909289+0000) 2019-10-13T09:28:52.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:52.909+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.909845+0000) 2019-10-13T09:28:52.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:52.909+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.909696+0000) 2019-10-13T09:28:52.912 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:52.909+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.909780+0000) 2019-10-13T09:28:52.912 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:52.909+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.910016+0000) 2019-10-13T09:28:52.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:52.912+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.913149+0000) 2019-10-13T09:28:52.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:52.915+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:52.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:52.924+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.924753+0000) 2019-10-13T09:28:52.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:52.923+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.924081+0000) 2019-10-13T09:28:53.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:53.738+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:53.739493+0000) 2019-10-13T09:28:54.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:54.680+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:54.680998+0000) 2019-10-13T09:28:57.919 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:57.917+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.917411+0000) 2019-10-13T09:28:57.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:57.917+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.917711+0000) 2019-10-13T09:28:57.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:28:57.917+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.917895+0000) 2019-10-13T09:28:57.920 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:57.917+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.918117+0000) 2019-10-13T09:28:57.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:28:57.918+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.918464+0000) 2019-10-13T09:28:57.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:28:57.918+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.919208+0000) 2019-10-13T09:28:57.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:28:57.919+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.920597+0000) 2019-10-13T09:28:57.921 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:57.918+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.918697+0000) 2019-10-13T09:28:57.929 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:28:57.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:28:58.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:28:58.480+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:58.481207+0000) 2019-10-13T09:28:58.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:28:58.818+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:58.820313+0000) 2019-10-13T09:28:58.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:28:58.818+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:58.820297+0000) 2019-10-13T09:28:58.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:28:58.818+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:58.820298+0000) 2019-10-13T09:29:02.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:02.924+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.924882+0000) 2019-10-13T09:29:02.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:02.923+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.924249+0000) 2019-10-13T09:29:02.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:02.926+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.927372+0000) 2019-10-13T09:29:02.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:02.925+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.926957+0000) 2019-10-13T09:29:02.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:02.925+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.927385+0000) 2019-10-13T09:29:02.928 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:02.926+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.927538+0000) 2019-10-13T09:29:02.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:02.926+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.927889+0000) 2019-10-13T09:29:02.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:02.926+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.928062+0000) 2019-10-13T09:29:02.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:02.926+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.927741+0000) 2019-10-13T09:29:02.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:02.929+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.930278+0000) 2019-10-13T09:29:02.935 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:02.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:03.742 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:03.741+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:03.742122+0000) 2019-10-13T09:29:04.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:04.680+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:04.681167+0000) 2019-10-13T09:29:07.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:07.934+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.935733+0000) 2019-10-13T09:29:07.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:07.936+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.936893+0000) 2019-10-13T09:29:07.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:07.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.936440+0000) 2019-10-13T09:29:07.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:07.935+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.936562+0000) 2019-10-13T09:29:07.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:07.934+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.936303+0000) 2019-10-13T09:29:07.938 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:07.935+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.936671+0000) 2019-10-13T09:29:07.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:07.935+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.936923+0000) 2019-10-13T09:29:07.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:07.939+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.939871+0000) 2019-10-13T09:29:07.943 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:07.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:08.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:08.480+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:08.481367+0000) 2019-10-13T09:29:08.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:08.819+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:08.820430+0000) 2019-10-13T09:29:08.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:08.819+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:08.820462+0000) 2019-10-13T09:29:08.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:08.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:08.820462+0000) 2019-10-13T09:29:12.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:12.924+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.925022+0000) 2019-10-13T09:29:12.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:12.922+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.924386+0000) 2019-10-13T09:29:12.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:12.943+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.944474+0000) 2019-10-13T09:29:12.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:12.943+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.944647+0000) 2019-10-13T09:29:12.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:12.945+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.945829+0000) 2019-10-13T09:29:12.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:12.943+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.945053+0000) 2019-10-13T09:29:12.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:12.943+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.945363+0000) 2019-10-13T09:29:12.947 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:12.944+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.945483+0000) 2019-10-13T09:29:12.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:12.944+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.945651+0000) 2019-10-13T09:29:12.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:12.946+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.947380+0000) 2019-10-13T09:29:12.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:12.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:13.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:13.744+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:13.744442+0000) 2019-10-13T09:29:14.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:14.680+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:14.681367+0000) 2019-10-13T09:29:17.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:17.951+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.952457+0000) 2019-10-13T09:29:17.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:17.951+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.952755+0000) 2019-10-13T09:29:17.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:17.951+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.952956+0000) 2019-10-13T09:29:17.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:17.953+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.954244+0000) 2019-10-13T09:29:17.955 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:17.952+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.953472+0000) 2019-10-13T09:29:17.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:17.952+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.953636+0000) 2019-10-13T09:29:17.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:17.952+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.953781+0000) 2019-10-13T09:29:17.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:17.955+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.956172+0000) 2019-10-13T09:29:17.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:17.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166649 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:18.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:18.480+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:18.481552+0000) 2019-10-13T09:29:18.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:18.819+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:18.820585+0000) 2019-10-13T09:29:18.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:18.819+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:18.820605+0000) 2019-10-13T09:29:18.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:18.819+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:18.820605+0000) 2019-10-13T09:29:21.842 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:29:21.842 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:21.847 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:29:21.847 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:22.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:22.924+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.925167+0000) 2019-10-13T09:29:22.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:22.923+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.924538+0000) 2019-10-13T09:29:22.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:22.959+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.961176+0000) 2019-10-13T09:29:22.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:22.960+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.961581+0000) 2019-10-13T09:29:22.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:22.960+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.961817+0000) 2019-10-13T09:29:22.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:22.960+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.961973+0000) 2019-10-13T09:29:22.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:22.962+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.963110+0000) 2019-10-13T09:29:22.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:22.961+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.962431+0000) 2019-10-13T09:29:22.964 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:22.961+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.962664+0000) 2019-10-13T09:29:22.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:22.964+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.965525+0000) 2019-10-13T09:29:22.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:22.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:23.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:23.746+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:23.746551+0000) 2019-10-13T09:29:24.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:24.681+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:24.681596+0000) 2019-10-13T09:29:27.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:27.969+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.970156+0000) 2019-10-13T09:29:27.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:27.970+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.970556+0000) 2019-10-13T09:29:27.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:27.970+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.970632+0000) 2019-10-13T09:29:27.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:27.970+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.971344+0000) 2019-10-13T09:29:27.973 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:27.970+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.971002+0000) 2019-10-13T09:29:27.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:27.970+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.971072+0000) 2019-10-13T09:29:27.973 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:27.970+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.971354+0000) 2019-10-13T09:29:27.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:27.973+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.974371+0000) 2019-10-13T09:29:27.978 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:27.976+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:28.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:28.481+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:28.481726+0000) 2019-10-13T09:29:28.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:28.820+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:28.820752+0000) 2019-10-13T09:29:28.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:28.820+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:28.820775+0000) 2019-10-13T09:29:28.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:28.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:28.820773+0000) 2019-10-13T09:29:32.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:32.923+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.925342+0000) 2019-10-13T09:29:32.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:32.924+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.924690+0000) 2019-10-13T09:29:32.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:32.978+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.978745+0000) 2019-10-13T09:29:32.980 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:32.978+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.978953+0000) 2019-10-13T09:29:32.980 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:32.978+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.979264+0000) 2019-10-13T09:29:32.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:32.978+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.980372+0000) 2019-10-13T09:29:32.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:32.979+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.979563+0000) 2019-10-13T09:29:32.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:32.979+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.979712+0000) 2019-10-13T09:29:32.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:32.979+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.979919+0000) 2019-10-13T09:29:32.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:32.981+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.982997+0000) 2019-10-13T09:29:32.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:32.985+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:33.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:33.747+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:33.748632+0000) 2019-10-13T09:29:34.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:34.681+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:34.681793+0000) 2019-10-13T09:29:37.988 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:37.987+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.987711+0000) 2019-10-13T09:29:37.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:37.987+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.989023+0000) 2019-10-13T09:29:37.989 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:37.987+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.988340+0000) 2019-10-13T09:29:37.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:37.988+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.988524+0000) 2019-10-13T09:29:37.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:37.988+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.988964+0000) 2019-10-13T09:29:37.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:37.988+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.988863+0000) 2019-10-13T09:29:37.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:37.988+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.989205+0000) 2019-10-13T09:29:37.991 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:37.989+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.991522+0000) 2019-10-13T09:29:37.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:37.994+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166767 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:38.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:38.481+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:38.481903+0000) 2019-10-13T09:29:38.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:38.820+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:38.820892+0000) 2019-10-13T09:29:38.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:38.820+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:38.820921+0000) 2019-10-13T09:29:38.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:38.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:38.820921+0000) 2019-10-13T09:29:42.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:42.924+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.925498+0000) 2019-10-13T09:29:42.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:42.924+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.924882+0000) 2019-10-13T09:29:42.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:42.996+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.996771+0000) 2019-10-13T09:29:42.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:42.996+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.997005+0000) 2019-10-13T09:29:42.998 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:42.996+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.997119+0000) 2019-10-13T09:29:42.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:42.997+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.998114+0000) 2019-10-13T09:29:42.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:42.997+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.997441+0000) 2019-10-13T09:29:42.999 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:42.997+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.997596+0000) 2019-10-13T09:29:43.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:42.997+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.998115+0000) 2019-10-13T09:29:43.000 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:43.000+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:43.000778+0000) 2019-10-13T09:29:43.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:43.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:43.751 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:43.750+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:43.751161+0000) 2019-10-13T09:29:44.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:44.680+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:44.682032+0000) 2019-10-13T09:29:48.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:48.004+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.005643+0000) 2019-10-13T09:29:48.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:48.004+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.005848+0000) 2019-10-13T09:29:48.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:48.004+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.005919+0000) 2019-10-13T09:29:48.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:48.006+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.007436+0000) 2019-10-13T09:29:48.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:48.004+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.006135+0000) 2019-10-13T09:29:48.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:48.004+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.006144+0000) 2019-10-13T09:29:48.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:48.004+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.006317+0000) 2019-10-13T09:29:48.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:48.007+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.008082+0000) 2019-10-13T09:29:48.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:48.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:48.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:48.481+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.482098+0000) 2019-10-13T09:29:48.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:48.820+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.821042+0000) 2019-10-13T09:29:48.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:48.820+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.821069+0000) 2019-10-13T09:29:48.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:48.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.821066+0000) 2019-10-13T09:29:51.969 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:29:51.970 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:51.974 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:29:51.974 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:52.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:52.925+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.925642+0000) 2019-10-13T09:29:52.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:52.923+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.925037+0000) 2019-10-13T09:29:53.015 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:53.012+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.014346+0000) 2019-10-13T09:29:53.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:53.013+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.014647+0000) 2019-10-13T09:29:53.016 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:53.013+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.014921+0000) 2019-10-13T09:29:53.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:53.014+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.015382+0000) 2019-10-13T09:29:53.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:53.013+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.015217+0000) 2019-10-13T09:29:53.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:53.013+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.015374+0000) 2019-10-13T09:29:53.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:53.014+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.015961+0000) 2019-10-13T09:29:53.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:53.018+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.019482+0000) 2019-10-13T09:29:53.022 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:53.020+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166858 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:53.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:53.752+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.753484+0000) 2019-10-13T09:29:54.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:54.681+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:54.682246+0000) 2019-10-13T09:29:58.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:58.022+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.023626+0000) 2019-10-13T09:29:58.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:29:58.022+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.023907+0000) 2019-10-13T09:29:58.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:29:58.023+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.024626+0000) 2019-10-13T09:29:58.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:29:58.022+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.024235+0000) 2019-10-13T09:29:58.026 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:58.022+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.024276+0000) 2019-10-13T09:29:58.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:58.023+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.024680+0000) 2019-10-13T09:29:58.027 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:58.023+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.024763+0000) 2019-10-13T09:29:58.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:29:58.026+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.026972+0000) 2019-10-13T09:29:58.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:29:58.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:29:58.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:29:58.481+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.482284+0000) 2019-10-13T09:29:58.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:29:58.820+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.821197+0000) 2019-10-13T09:29:58.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:29:58.820+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.821226+0000) 2019-10-13T09:29:58.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:29:58.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.821226+0000) 2019-10-13T09:30:02.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:02.925+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.925797+0000) 2019-10-13T09:30:02.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:02.924+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.925144+0000) 2019-10-13T09:30:03.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:03.031+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.032431+0000) 2019-10-13T09:30:03.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:03.031+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.032591+0000) 2019-10-13T09:30:03.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:03.032+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.033560+0000) 2019-10-13T09:30:03.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:03.031+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.032997+0000) 2019-10-13T09:30:03.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:03.032+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.033408+0000) 2019-10-13T09:30:03.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:03.032+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.033535+0000) 2019-10-13T09:30:03.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:03.032+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.033692+0000) 2019-10-13T09:30:03.036 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:03.035+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.036621+0000) 2019-10-13T09:30:03.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:03.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:03.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:03.755+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.755911+0000) 2019-10-13T09:30:04.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:04.681+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:04.682409+0000) 2019-10-13T09:30:08.042 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:08.039+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.040588+0000) 2019-10-13T09:30:08.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:08.039+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.040855+0000) 2019-10-13T09:30:08.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:08.041+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.041956+0000) 2019-10-13T09:30:08.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:08.039+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.041379+0000) 2019-10-13T09:30:08.043 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:08.040+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.042076+0000) 2019-10-13T09:30:08.043 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:08.040+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.042179+0000) 2019-10-13T09:30:08.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:08.041+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.042593+0000) 2019-10-13T09:30:08.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:08.043+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.044725+0000) 2019-10-13T09:30:08.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:08.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:08.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:08.481+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.482460+0000) 2019-10-13T09:30:08.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:08.819+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.821359+0000) 2019-10-13T09:30:08.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:08.820+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.821407+0000) 2019-10-13T09:30:08.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:08.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.821407+0000) 2019-10-13T09:30:12.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:12.925+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.925968+0000) 2019-10-13T09:30:12.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:12.924+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.925311+0000) 2019-10-13T09:30:13.050 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:13.049+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.049586+0000) 2019-10-13T09:30:13.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:13.049+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.049977+0000) 2019-10-13T09:30:13.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:13.050+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.050747+0000) 2019-10-13T09:30:13.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:13.049+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.050197+0000) 2019-10-13T09:30:13.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:13.049+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.050245+0000) 2019-10-13T09:30:13.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:13.050+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.050702+0000) 2019-10-13T09:30:13.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:13.050+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.050910+0000) 2019-10-13T09:30:13.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:13.051+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.052297+0000) 2019-10-13T09:30:13.056 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:13.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 166973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:13.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:13.758+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.758201+0000) 2019-10-13T09:30:14.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:14.682+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:14.682576+0000) 2019-10-13T09:30:18.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:18.057+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.057656+0000) 2019-10-13T09:30:18.059 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:18.057+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.057890+0000) 2019-10-13T09:30:18.059 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:18.057+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.058025+0000) 2019-10-13T09:30:18.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:18.058+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.059089+0000) 2019-10-13T09:30:18.060 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:18.058+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.058436+0000) 2019-10-13T09:30:18.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:18.058+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.058503+0000) 2019-10-13T09:30:18.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:18.058+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.058649+0000) 2019-10-13T09:30:18.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:18.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.061419+0000) 2019-10-13T09:30:18.064 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:18.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:18.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:18.482+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.482616+0000) 2019-10-13T09:30:18.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:18.821+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.821497+0000) 2019-10-13T09:30:18.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:18.821+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.821563+0000) 2019-10-13T09:30:18.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:18.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.821564+0000) 2019-10-13T09:30:22.141 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:30:22.141 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:22.146 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:30:22.146 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:22.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:22.925+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.926122+0000) 2019-10-13T09:30:22.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:22.925+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.925519+0000) 2019-10-13T09:30:23.066 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:23.064+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.065114+0000) 2019-10-13T09:30:23.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:23.064+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.065240+0000) 2019-10-13T09:30:23.067 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:23.065+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.065477+0000) 2019-10-13T09:30:23.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:23.065+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.067123+0000) 2019-10-13T09:30:23.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:23.065+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.065737+0000) 2019-10-13T09:30:23.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:23.065+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.066028+0000) 2019-10-13T09:30:23.068 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:23.065+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.066221+0000) 2019-10-13T09:30:23.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:23.066+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.068667+0000) 2019-10-13T09:30:23.072 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:23.070+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:23.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:23.759+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.760551+0000) 2019-10-13T09:30:24.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:24.682+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:24.682778+0000) 2019-10-13T09:30:28.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:28.072+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.073091+0000) 2019-10-13T09:30:28.074 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:28.072+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.073342+0000) 2019-10-13T09:30:28.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:28.073+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.073588+0000) 2019-10-13T09:30:28.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:28.073+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.075375+0000) 2019-10-13T09:30:28.076 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:28.073+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.073834+0000) 2019-10-13T09:30:28.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:28.073+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.074057+0000) 2019-10-13T09:30:28.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:28.073+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.074255+0000) 2019-10-13T09:30:28.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:28.075+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.077649+0000) 2019-10-13T09:30:28.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:28.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:28.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:28.482+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.482794+0000) 2019-10-13T09:30:28.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:28.821+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.821666+0000) 2019-10-13T09:30:28.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:28.821+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.821734+0000) 2019-10-13T09:30:28.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:28.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.821757+0000) 2019-10-13T09:30:32.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:32.925+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.926309+0000) 2019-10-13T09:30:32.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:32.924+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.925649+0000) 2019-10-13T09:30:33.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:33.080+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.082166+0000) 2019-10-13T09:30:33.083 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:33.080+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.082302+0000) 2019-10-13T09:30:33.084 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:33.081+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.082815+0000) 2019-10-13T09:30:33.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:33.081+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.082941+0000) 2019-10-13T09:30:33.085 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:33.081+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.083121+0000) 2019-10-13T09:30:33.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:33.082+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.083510+0000) 2019-10-13T09:30:33.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:33.082+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.084250+0000) 2019-10-13T09:30:33.086 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:33.083+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.085601+0000) 2019-10-13T09:30:33.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:33.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:33.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:33.762+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.762828+0000) 2019-10-13T09:30:34.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:34.682+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:34.682999+0000) 2019-10-13T09:30:38.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:38.088+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.089725+0000) 2019-10-13T09:30:38.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:38.090+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.090916+0000) 2019-10-13T09:30:38.092 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:38.089+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.090434+0000) 2019-10-13T09:30:38.092 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:38.089+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.090431+0000) 2019-10-13T09:30:38.092 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:38.089+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.090573+0000) 2019-10-13T09:30:38.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:38.089+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.090734+0000) 2019-10-13T09:30:38.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:38.090+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.091464+0000) 2019-10-13T09:30:38.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:38.093+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.093841+0000) 2019-10-13T09:30:38.097 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:38.095+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:38.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:38.481+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.482968+0000) 2019-10-13T09:30:38.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:38.820+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.821822+0000) 2019-10-13T09:30:38.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:38.820+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.821911+0000) 2019-10-13T09:30:38.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:38.820+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.821911+0000) 2019-10-13T09:30:42.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:42.925+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.926509+0000) 2019-10-13T09:30:42.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:42.925+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.925840+0000) 2019-10-13T09:30:43.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:43.096+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.098173+0000) 2019-10-13T09:30:43.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:43.098+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.099247+0000) 2019-10-13T09:30:43.100 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:43.097+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.098817+0000) 2019-10-13T09:30:43.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:43.097+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.099040+0000) 2019-10-13T09:30:43.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:43.097+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.099325+0000) 2019-10-13T09:30:43.101 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:43.098+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.099523+0000) 2019-10-13T09:30:43.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:43.100+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.101499+0000) 2019-10-13T09:30:43.102 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:43.098+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.099750+0000) 2019-10-13T09:30:43.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:43.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:43.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:43.764+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.765061+0000) 2019-10-13T09:30:44.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:44.682+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:44.683202+0000) 2019-10-13T09:30:48.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:48.105+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.106850+0000) 2019-10-13T09:30:48.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:48.106+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.107446+0000) 2019-10-13T09:30:48.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:48.106+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.107513+0000) 2019-10-13T09:30:48.109 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:48.106+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.107895+0000) 2019-10-13T09:30:48.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:48.107+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.108632+0000) 2019-10-13T09:30:48.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:48.106+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.108172+0000) 2019-10-13T09:30:48.110 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:48.106+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.108335+0000) 2019-10-13T09:30:48.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:48.110+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.111124+0000) 2019-10-13T09:30:48.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:48.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:48.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:48.482+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.483110+0000) 2019-10-13T09:30:48.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:48.821+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.821963+0000) 2019-10-13T09:30:48.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:48.821+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.822052+0000) 2019-10-13T09:30:48.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:48.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.822042+0000) 2019-10-13T09:30:52.369 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:30:52.369 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:52.373 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:30:52.373 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:52.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:52.926+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.926695+0000) 2019-10-13T09:30:52.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:52.925+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.925951+0000) 2019-10-13T09:30:53.116 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:53.114+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.115474+0000) 2019-10-13T09:30:53.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:53.114+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.115597+0000) 2019-10-13T09:30:53.117 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:53.114+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.115791+0000) 2019-10-13T09:30:53.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:53.116+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.117425+0000) 2019-10-13T09:30:53.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:53.114+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.116115+0000) 2019-10-13T09:30:53.118 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:53.114+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.116287+0000) 2019-10-13T09:30:53.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:53.115+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.116739+0000) 2019-10-13T09:30:53.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:53.118+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.119657+0000) 2019-10-13T09:30:53.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:53.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167197 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:53.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:53.767+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.767525+0000) 2019-10-13T09:30:54.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:54.682+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:54.683430+0000) 2019-10-13T09:30:58.123 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:30:58.122+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.122610+0000) 2019-10-13T09:30:58.124 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:58.122+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.123357+0000) 2019-10-13T09:30:58.124 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:58.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.123506+0000) 2019-10-13T09:30:58.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:58.123+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.123684+0000) 2019-10-13T09:30:58.125 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:30:58.123+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.124713+0000) 2019-10-13T09:30:58.126 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:30:58.123+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.123951+0000) 2019-10-13T09:30:58.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:58.123+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.124124+0000) 2019-10-13T09:30:58.127 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:30:58.126+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.127698+0000) 2019-10-13T09:30:58.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:30:58.129+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167224 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:30:58.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:30:58.482+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.483295+0000) 2019-10-13T09:30:58.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:30:58.821+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.822155+0000) 2019-10-13T09:30:58.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:30:58.821+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.822203+0000) 2019-10-13T09:30:58.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:30:58.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.822202+0000) 2019-10-13T09:31:02.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:02.926+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.926895+0000) 2019-10-13T09:31:02.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:02.925+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.926120+0000) 2019-10-13T09:31:03.134 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:03.130+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.131302+0000) 2019-10-13T09:31:03.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:03.131+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.131717+0000) 2019-10-13T09:31:03.134 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:03.131+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.131839+0000) 2019-10-13T09:31:03.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:03.131+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.132379+0000) 2019-10-13T09:31:03.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:03.131+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.131950+0000) 2019-10-13T09:31:03.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:03.131+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.132084+0000) 2019-10-13T09:31:03.136 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:03.131+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.132291+0000) 2019-10-13T09:31:03.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:03.133+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.134675+0000) 2019-10-13T09:31:03.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:03.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:03.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:03.769+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.769884+0000) 2019-10-13T09:31:04.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:04.683+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:04.683669+0000) 2019-10-13T09:31:08.140 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:08.138+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.139672+0000) 2019-10-13T09:31:08.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:08.138+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.138625+0000) 2019-10-13T09:31:08.140 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:08.138+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.138990+0000) 2019-10-13T09:31:08.141 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:08.138+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.139219+0000) 2019-10-13T09:31:08.141 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:08.138+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.139369+0000) 2019-10-13T09:31:08.141 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:08.139+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.139757+0000) 2019-10-13T09:31:08.141 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:08.139+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.140007+0000) 2019-10-13T09:31:08.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:08.141+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.142632+0000) 2019-10-13T09:31:08.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:08.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:08.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:08.483+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.483464+0000) 2019-10-13T09:31:08.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:08.822+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.822325+0000) 2019-10-13T09:31:08.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:08.822+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.822386+0000) 2019-10-13T09:31:08.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:08.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.822374+0000) 2019-10-13T09:31:12.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:12.926+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.927056+0000) 2019-10-13T09:31:12.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:12.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.926309+0000) 2019-10-13T09:31:13.148 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:13.146+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.147237+0000) 2019-10-13T09:31:13.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:13.147+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.147502+0000) 2019-10-13T09:31:13.150 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:13.147+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.148857+0000) 2019-10-13T09:31:13.150 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:13.147+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.148258+0000) 2019-10-13T09:31:13.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:13.148+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.148432+0000) 2019-10-13T09:31:13.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:13.148+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.148640+0000) 2019-10-13T09:31:13.151 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:13.148+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.148814+0000) 2019-10-13T09:31:13.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:13.149+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.151181+0000) 2019-10-13T09:31:13.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:13.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:13.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:13.770+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.772348+0000) 2019-10-13T09:31:14.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:14.683+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:14.683858+0000) 2019-10-13T09:31:18.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:18.154+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.156147+0000) 2019-10-13T09:31:18.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:18.154+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.156326+0000) 2019-10-13T09:31:18.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:18.155+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.157363+0000) 2019-10-13T09:31:18.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:18.155+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.156827+0000) 2019-10-13T09:31:18.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:18.155+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.157024+0000) 2019-10-13T09:31:18.159 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:18.155+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.157124+0000) 2019-10-13T09:31:18.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:18.156+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.157646+0000) 2019-10-13T09:31:18.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:18.158+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.160471+0000) 2019-10-13T09:31:18.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:18.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:18.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:18.482+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.483626+0000) 2019-10-13T09:31:18.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:18.821+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.822485+0000) 2019-10-13T09:31:18.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:18.821+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.822558+0000) 2019-10-13T09:31:18.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:18.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.822553+0000) 2019-10-13T09:31:22.457 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:31:22.458 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:22.462 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:31:22.462 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:22.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:22.926+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.927172+0000) 2019-10-13T09:31:22.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:22.925+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.926502+0000) 2019-10-13T09:31:23.166 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:23.163+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.165048+0000) 2019-10-13T09:31:23.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:23.163+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.165366+0000) 2019-10-13T09:31:23.167 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:23.164+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.166512+0000) 2019-10-13T09:31:23.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:23.164+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.165700+0000) 2019-10-13T09:31:23.168 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:23.164+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.165814+0000) 2019-10-13T09:31:23.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:23.164+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.166308+0000) 2019-10-13T09:31:23.168 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:23.165+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.166477+0000) 2019-10-13T09:31:23.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:23.166+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.168226+0000) 2019-10-13T09:31:23.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:23.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167370 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:23.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:23.773+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.774760+0000) 2019-10-13T09:31:24.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:24.682+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:24.684028+0000) 2019-10-13T09:31:28.175 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:28.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.174778+0000) 2019-10-13T09:31:28.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:28.172+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.174152+0000) 2019-10-13T09:31:28.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:28.173+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.174439+0000) 2019-10-13T09:31:28.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:28.172+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.174318+0000) 2019-10-13T09:31:28.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:28.173+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.174761+0000) 2019-10-13T09:31:28.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:28.173+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.174502+0000) 2019-10-13T09:31:28.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:28.173+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.175124+0000) 2019-10-13T09:31:28.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:28.177+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.178627+0000) 2019-10-13T09:31:28.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:28.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167397 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:28.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:28.483+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.483766+0000) 2019-10-13T09:31:28.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:28.821+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.822622+0000) 2019-10-13T09:31:28.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:28.821+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.822700+0000) 2019-10-13T09:31:28.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:28.821+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.822730+0000) 2019-10-13T09:31:32.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:32.926+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.927404+0000) 2019-10-13T09:31:32.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:32.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.926692+0000) 2019-10-13T09:31:33.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:33.181+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.182649+0000) 2019-10-13T09:31:33.184 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:33.181+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.183021+0000) 2019-10-13T09:31:33.184 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:33.181+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.183272+0000) 2019-10-13T09:31:33.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:33.183+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.184088+0000) 2019-10-13T09:31:33.186 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:33.182+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.183434+0000) 2019-10-13T09:31:33.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:33.182+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.183559+0000) 2019-10-13T09:31:33.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:33.182+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.184216+0000) 2019-10-13T09:31:33.186 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:33.185+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.186677+0000) 2019-10-13T09:31:33.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:33.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:33.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:33.776+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.777166+0000) 2019-10-13T09:31:34.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:34.683+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:34.684234+0000) 2019-10-13T09:31:38.193 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:38.190+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.191591+0000) 2019-10-13T09:31:38.193 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:38.190+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.191831+0000) 2019-10-13T09:31:38.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:38.190+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.191997+0000) 2019-10-13T09:31:38.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:38.192+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.193041+0000) 2019-10-13T09:31:38.194 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:38.190+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.192265+0000) 2019-10-13T09:31:38.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:38.191+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.192577+0000) 2019-10-13T09:31:38.194 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:38.191+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.192871+0000) 2019-10-13T09:31:38.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:38.195+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.196320+0000) 2019-10-13T09:31:38.199 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:38.197+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:38.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:38.483+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.483908+0000) 2019-10-13T09:31:38.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:38.822+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.822758+0000) 2019-10-13T09:31:38.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:38.822+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.822849+0000) 2019-10-13T09:31:38.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:38.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.822848+0000) 2019-10-13T09:31:42.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:42.926+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.927589+0000) 2019-10-13T09:31:42.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:42.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.926881+0000) 2019-10-13T09:31:43.201 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:43.199+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.200335+0000) 2019-10-13T09:31:43.202 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:43.201+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.201760+0000) 2019-10-13T09:31:43.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:43.200+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.200895+0000) 2019-10-13T09:31:43.202 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:43.200+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.201261+0000) 2019-10-13T09:31:43.203 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:43.201+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.201509+0000) 2019-10-13T09:31:43.203 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:43.201+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.201672+0000) 2019-10-13T09:31:43.203 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:43.201+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.201857+0000) 2019-10-13T09:31:43.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:43.203+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.204106+0000) 2019-10-13T09:31:43.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:43.206+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167486 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:43.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:43.779+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.780231+0000) 2019-10-13T09:31:44.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:44.684+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:44.684415+0000) 2019-10-13T09:31:48.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:48.208+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.208943+0000) 2019-10-13T09:31:48.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:48.208+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.209122+0000) 2019-10-13T09:31:48.211 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:48.208+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.209325+0000) 2019-10-13T09:31:48.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:48.209+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.209609+0000) 2019-10-13T09:31:48.212 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:48.210+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.210841+0000) 2019-10-13T09:31:48.212 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:48.209+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.210014+0000) 2019-10-13T09:31:48.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:48.210+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.210413+0000) 2019-10-13T09:31:48.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:48.212+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.213147+0000) 2019-10-13T09:31:48.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:48.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:48.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:48.483+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.484102+0000) 2019-10-13T09:31:48.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:48.822+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.822872+0000) 2019-10-13T09:31:48.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:48.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.822942+0000) 2019-10-13T09:31:48.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:48.822+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.822970+0000) 2019-10-13T09:31:52.571 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:31:52.571 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:52.575 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:31:52.575 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:52.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:52.927+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.927799+0000) 2019-10-13T09:31:52.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:52.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.927036+0000) 2019-10-13T09:31:53.218 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:53.216+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.217093+0000) 2019-10-13T09:31:53.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:53.218+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.218815+0000) 2019-10-13T09:31:53.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:53.217+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.217570+0000) 2019-10-13T09:31:53.219 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:53.216+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.217390+0000) 2019-10-13T09:31:53.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:53.216+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.217310+0000) 2019-10-13T09:31:53.220 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:53.217+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.217700+0000) 2019-10-13T09:31:53.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:53.217+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.218381+0000) 2019-10-13T09:31:53.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:53.220+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.221119+0000) 2019-10-13T09:31:53.223 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:53.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:53.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:53.781+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.782191+0000) 2019-10-13T09:31:54.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:54.684+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:54.684558+0000) 2019-10-13T09:31:58.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:31:58.224+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.224546+0000) 2019-10-13T09:31:58.226 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:58.224+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.224740+0000) 2019-10-13T09:31:58.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:58.224+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.224805+0000) 2019-10-13T09:31:58.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:31:58.224+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.224954+0000) 2019-10-13T09:31:58.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:31:58.225+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.226718+0000) 2019-10-13T09:31:58.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:58.225+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.225490+0000) 2019-10-13T09:31:58.228 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:58.225+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.225624+0000) 2019-10-13T09:31:58.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:31:58.227+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.228380+0000) 2019-10-13T09:31:58.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:31:58.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167572 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:31:58.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:31:58.484+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.484240+0000) 2019-10-13T09:31:58.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:31:58.822+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.823028+0000) 2019-10-13T09:31:58.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:31:58.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.823046+0000) 2019-10-13T09:31:58.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:31:58.822+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.823096+0000) 2019-10-13T09:32:02.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:02.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.927211+0000) 2019-10-13T09:32:02.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:02.927+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.927997+0000) 2019-10-13T09:32:03.234 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:03.231+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.233154+0000) 2019-10-13T09:32:03.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:03.233+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.234639+0000) 2019-10-13T09:32:03.235 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:03.232+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.233485+0000) 2019-10-13T09:32:03.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:03.232+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.234050+0000) 2019-10-13T09:32:03.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:03.232+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.234248+0000) 2019-10-13T09:32:03.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:03.233+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.234478+0000) 2019-10-13T09:32:03.236 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:03.233+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.234788+0000) 2019-10-13T09:32:03.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:03.235+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.236362+0000) 2019-10-13T09:32:03.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:03.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167602 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:03.785 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:03.783+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.784555+0000) 2019-10-13T09:32:04.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:04.683+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:04.684732+0000) 2019-10-13T09:32:08.243 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:08.240+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.241881+0000) 2019-10-13T09:32:08.244 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:08.241+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.243579+0000) 2019-10-13T09:32:08.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:08.240+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.242290+0000) 2019-10-13T09:32:08.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:08.241+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.242436+0000) 2019-10-13T09:32:08.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:08.240+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.242353+0000) 2019-10-13T09:32:08.245 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:08.241+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.242770+0000) 2019-10-13T09:32:08.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:08.241+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.242889+0000) 2019-10-13T09:32:08.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:08.244+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.246063+0000) 2019-10-13T09:32:08.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:08.247+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167629 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:08.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:08.483+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.484389+0000) 2019-10-13T09:32:08.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:08.822+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.823236+0000) 2019-10-13T09:32:08.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:08.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.823236+0000) 2019-10-13T09:32:08.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:08.822+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.823310+0000) 2019-10-13T09:32:12.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:12.927+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.928140+0000) 2019-10-13T09:32:12.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:12.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.927412+0000) 2019-10-13T09:32:13.252 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:13.249+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.250788+0000) 2019-10-13T09:32:13.253 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:13.250+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.252189+0000) 2019-10-13T09:32:13.253 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:13.249+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.251229+0000) 2019-10-13T09:32:13.253 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:13.249+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.251185+0000) 2019-10-13T09:32:13.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:13.250+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.251593+0000) 2019-10-13T09:32:13.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:13.250+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.251765+0000) 2019-10-13T09:32:13.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:13.250+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.251984+0000) 2019-10-13T09:32:13.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:13.252+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.254518+0000) 2019-10-13T09:32:13.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:13.256+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167661 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:13.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:13.785+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.787037+0000) 2019-10-13T09:32:14.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:14.684+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:14.684932+0000) 2019-10-13T09:32:18.260 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:18.258+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.259499+0000) 2019-10-13T09:32:18.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:18.260+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.260781+0000) 2019-10-13T09:32:18.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:18.258+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.260016+0000) 2019-10-13T09:32:18.262 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:18.258+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.260260+0000) 2019-10-13T09:32:18.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:18.259+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.260476+0000) 2019-10-13T09:32:18.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:18.259+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.260568+0000) 2019-10-13T09:32:18.263 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:18.259+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.260742+0000) 2019-10-13T09:32:18.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:18.261+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.262724+0000) 2019-10-13T09:32:18.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:18.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:18.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:18.483+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.484570+0000) 2019-10-13T09:32:18.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:18.822+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.823438+0000) 2019-10-13T09:32:18.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:18.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.823438+0000) 2019-10-13T09:32:18.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:18.822+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.823445+0000) 2019-10-13T09:32:22.709 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:32:22.709 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:22.713 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:32:22.713 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:22.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:22.927+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.928276+0000) 2019-10-13T09:32:22.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:22.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.927577+0000) 2019-10-13T09:32:23.269 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:23.266+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.268022+0000) 2019-10-13T09:32:23.269 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:23.266+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.268214+0000) 2019-10-13T09:32:23.270 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:23.266+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.268336+0000) 2019-10-13T09:32:23.270 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:23.268+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.269503+0000) 2019-10-13T09:32:23.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:23.267+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.268680+0000) 2019-10-13T09:32:23.271 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:23.267+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.269087+0000) 2019-10-13T09:32:23.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:23.267+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.269121+0000) 2019-10-13T09:32:23.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:23.271+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.272014+0000) 2019-10-13T09:32:23.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:23.273+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:23.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:23.788+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.789456+0000) 2019-10-13T09:32:24.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:24.683+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:24.685149+0000) 2019-10-13T09:32:28.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:28.277+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.277736+0000) 2019-10-13T09:32:28.278 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:28.275+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.276641+0000) 2019-10-13T09:32:28.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:28.275+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.277245+0000) 2019-10-13T09:32:28.279 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:28.275+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.277364+0000) 2019-10-13T09:32:28.279 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:28.276+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.277494+0000) 2019-10-13T09:32:28.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:28.276+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.277678+0000) 2019-10-13T09:32:28.280 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:28.276+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.277893+0000) 2019-10-13T09:32:28.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:28.279+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.280113+0000) 2019-10-13T09:32:28.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:28.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:28.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:28.484+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.484717+0000) 2019-10-13T09:32:28.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:28.823+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.823572+0000) 2019-10-13T09:32:28.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:28.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.823572+0000) 2019-10-13T09:32:28.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:28.823+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.823607+0000) 2019-10-13T09:32:32.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:32.928+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.928506+0000) 2019-10-13T09:32:32.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:32.927+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.927754+0000) 2019-10-13T09:32:33.287 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:33.284+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.285016+0000) 2019-10-13T09:32:33.287 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:33.285+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.285417+0000) 2019-10-13T09:32:33.287 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:33.285+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.285654+0000) 2019-10-13T09:32:33.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:33.285+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.285784+0000) 2019-10-13T09:32:33.287 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:33.285+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.286030+0000) 2019-10-13T09:32:33.288 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:33.286+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.287179+0000) 2019-10-13T09:32:33.288 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:33.286+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.286727+0000) 2019-10-13T09:32:33.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:33.288+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.289507+0000) 2019-10-13T09:32:33.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:33.292+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:33.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:33.791+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.791817+0000) 2019-10-13T09:32:34.686 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:34.685+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:34.685350+0000) 2019-10-13T09:32:38.298 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:38.295+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.296031+0000) 2019-10-13T09:32:38.298 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:38.293+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.294057+0000) 2019-10-13T09:32:38.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:38.293+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.294187+0000) 2019-10-13T09:32:38.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:38.293+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.294367+0000) 2019-10-13T09:32:38.299 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:38.294+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.294538+0000) 2019-10-13T09:32:38.299 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:38.294+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.294716+0000) 2019-10-13T09:32:38.299 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:38.295+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.295423+0000) 2019-10-13T09:32:38.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:38.296+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.297522+0000) 2019-10-13T09:32:38.301 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:38.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:38.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:38.484+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.484874+0000) 2019-10-13T09:32:38.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:38.823+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.823753+0000) 2019-10-13T09:32:38.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:38.823+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.823768+0000) 2019-10-13T09:32:38.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:38.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.823753+0000) 2019-10-13T09:32:42.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:42.928+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.928681+0000) 2019-10-13T09:32:42.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:42.927+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.927938+0000) 2019-10-13T09:32:43.303 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:43.301+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.301700+0000) 2019-10-13T09:32:43.303 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:43.301+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.301967+0000) 2019-10-13T09:32:43.304 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:43.301+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.302204+0000) 2019-10-13T09:32:43.304 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:43.301+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.302092+0000) 2019-10-13T09:32:43.305 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:43.303+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.304020+0000) 2019-10-13T09:32:43.305 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:43.302+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.302720+0000) 2019-10-13T09:32:43.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:43.302+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.303273+0000) 2019-10-13T09:32:43.306 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:43.305+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.306105+0000) 2019-10-13T09:32:43.310 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:43.309+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167834 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:43.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:43.793+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.793919+0000) 2019-10-13T09:32:44.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:44.685+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:44.685580+0000) 2019-10-13T09:32:48.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:48.309+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.310983+0000) 2019-10-13T09:32:48.312 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:48.309+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.311277+0000) 2019-10-13T09:32:48.313 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:48.309+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.311387+0000) 2019-10-13T09:32:48.313 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:48.310+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.311529+0000) 2019-10-13T09:32:48.313 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:48.312+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.313068+0000) 2019-10-13T09:32:48.314 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:48.310+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.311698+0000) 2019-10-13T09:32:48.314 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:48.310+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.312014+0000) 2019-10-13T09:32:48.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:48.314+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.314928+0000) 2019-10-13T09:32:48.317 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:48.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:48.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:48.484+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.485021+0000) 2019-10-13T09:32:48.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:48.822+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.823941+0000) 2019-10-13T09:32:48.825 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:48.822+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.823941+0000) 2019-10-13T09:32:48.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:48.822+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.823941+0000) 2019-10-13T09:32:52.888 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:32:52.888 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:52.892 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:32:52.893 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:52.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:52.928+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.928836+0000) 2019-10-13T09:32:52.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:52.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.928066+0000) 2019-10-13T09:32:53.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:53.317+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.318564+0000) 2019-10-13T09:32:53.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:53.317+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.318988+0000) 2019-10-13T09:32:53.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:53.317+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.319191+0000) 2019-10-13T09:32:53.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:53.318+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.319450+0000) 2019-10-13T09:32:53.321 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:53.318+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.320022+0000) 2019-10-13T09:32:53.322 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:53.320+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.320816+0000) 2019-10-13T09:32:53.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:53.321+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.321823+0000) 2019-10-13T09:32:53.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:53.318+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.319725+0000) 2019-10-13T09:32:53.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:53.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167893 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:53.796 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:53.795+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.795877+0000) 2019-10-13T09:32:54.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:54.684+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:54.685773+0000) 2019-10-13T09:32:58.328 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:58.325+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.326875+0000) 2019-10-13T09:32:58.328 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:32:58.325+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.327125+0000) 2019-10-13T09:32:58.328 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:58.325+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.327221+0000) 2019-10-13T09:32:58.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:32:58.326+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.328379+0000) 2019-10-13T09:32:58.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:32:58.326+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.327668+0000) 2019-10-13T09:32:58.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:58.326+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.327738+0000) 2019-10-13T09:32:58.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:58.326+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.328251+0000) 2019-10-13T09:32:58.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:32:58.329+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.330964+0000) 2019-10-13T09:32:58.333 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:32:58.331+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167920 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:32:58.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:32:58.484+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.485165+0000) 2019-10-13T09:32:58.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:32:58.823+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.824126+0000) 2019-10-13T09:32:58.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:32:58.823+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.824118+0000) 2019-10-13T09:32:58.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:32:58.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.824126+0000) 2019-10-13T09:33:02.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:02.928+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.929013+0000) 2019-10-13T09:33:02.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:02.926+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.928282+0000) 2019-10-13T09:33:03.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:03.333+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.334441+0000) 2019-10-13T09:33:03.336 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:03.333+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.334766+0000) 2019-10-13T09:33:03.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:03.333+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.335511+0000) 2019-10-13T09:33:03.337 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:03.333+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.335159+0000) 2019-10-13T09:33:03.337 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:03.333+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.335270+0000) 2019-10-13T09:33:03.337 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:03.334+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.335422+0000) 2019-10-13T09:33:03.337 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:03.334+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.335826+0000) 2019-10-13T09:33:03.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:03.336+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.338029+0000) 2019-10-13T09:33:03.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:03.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:03.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:03.797+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.798288+0000) 2019-10-13T09:33:04.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:04.685+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:04.685990+0000) 2019-10-13T09:33:08.344 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:08.342+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.343909+0000) 2019-10-13T09:33:08.345 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:08.341+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.343153+0000) 2019-10-13T09:33:08.345 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:08.341+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.343374+0000) 2019-10-13T09:33:08.346 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:08.342+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.343491+0000) 2019-10-13T09:33:08.346 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:08.342+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.343828+0000) 2019-10-13T09:33:08.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:08.342+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.344196+0000) 2019-10-13T09:33:08.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:08.343+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.344574+0000) 2019-10-13T09:33:08.347 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:08.345+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.347083+0000) 2019-10-13T09:33:08.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:08.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 167977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:08.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:08.484+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.485325+0000) 2019-10-13T09:33:08.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:08.823+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.824297+0000) 2019-10-13T09:33:08.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:08.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.824311+0000) 2019-10-13T09:33:08.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:08.823+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.824312+0000) 2019-10-13T09:33:12.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:12.928+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.929191+0000) 2019-10-13T09:33:12.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:12.927+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.928441+0000) 2019-10-13T09:33:13.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:13.350+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.351655+0000) 2019-10-13T09:33:13.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:13.350+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.351753+0000) 2019-10-13T09:33:13.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:13.350+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.351891+0000) 2019-10-13T09:33:13.353 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:13.350+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.352043+0000) 2019-10-13T09:33:13.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:13.352+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.353158+0000) 2019-10-13T09:33:13.354 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:13.351+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.352513+0000) 2019-10-13T09:33:13.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:13.351+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.352590+0000) 2019-10-13T09:33:13.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:13.355+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.356232+0000) 2019-10-13T09:33:13.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:13.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:13.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:13.799+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.800627+0000) 2019-10-13T09:33:14.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:14.685+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:14.686178+0000) 2019-10-13T09:33:18.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:18.358+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.359251+0000) 2019-10-13T09:33:18.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:18.359+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.359679+0000) 2019-10-13T09:33:18.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:18.359+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.359851+0000) 2019-10-13T09:33:18.361 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:18.360+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.360583+0000) 2019-10-13T09:33:18.362 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:18.360+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.360761+0000) 2019-10-13T09:33:18.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:18.361+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.362150+0000) 2019-10-13T09:33:18.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:18.360+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.361353+0000) 2019-10-13T09:33:18.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:18.362+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.363393+0000) 2019-10-13T09:33:18.368 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:18.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:18.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:18.485+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.485523+0000) 2019-10-13T09:33:18.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:18.824+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.824476+0000) 2019-10-13T09:33:18.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:18.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.824502+0000) 2019-10-13T09:33:18.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:18.824+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.824502+0000) 2019-10-13T09:33:22.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:22.928+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.929346+0000) 2019-10-13T09:33:22.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:22.928+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.928570+0000) 2019-10-13T09:33:23.031 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:33:23.031 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:23.035 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:33:23.035 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:23.371 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:23.369+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.369618+0000) 2019-10-13T09:33:23.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:23.369+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.369904+0000) 2019-10-13T09:33:23.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:23.369+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.369988+0000) 2019-10-13T09:33:23.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:23.369+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.370142+0000) 2019-10-13T09:33:23.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:23.369+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.370306+0000) 2019-10-13T09:33:23.372 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:23.370+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.370470+0000) 2019-10-13T09:33:23.373 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:23.371+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.372082+0000) 2019-10-13T09:33:23.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:23.373+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.374394+0000) 2019-10-13T09:33:23.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:23.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:23.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:23.802+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.802950+0000) 2019-10-13T09:33:24.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:24.686+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:24.686342+0000) 2019-10-13T09:33:28.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:28.378+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.379524+0000) 2019-10-13T09:33:28.380 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:28.378+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.378762+0000) 2019-10-13T09:33:28.381 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:28.378+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.378910+0000) 2019-10-13T09:33:28.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:28.378+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.379132+0000) 2019-10-13T09:33:28.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:28.378+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.379386+0000) 2019-10-13T09:33:28.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:28.379+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.379831+0000) 2019-10-13T09:33:28.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:28.379+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.379950+0000) 2019-10-13T09:33:28.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:28.380+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.381370+0000) 2019-10-13T09:33:28.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:28.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:28.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:28.485+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.485718+0000) 2019-10-13T09:33:28.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:28.824+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.824642+0000) 2019-10-13T09:33:28.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:28.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.824664+0000) 2019-10-13T09:33:28.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:28.824+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.824664+0000) 2019-10-13T09:33:32.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:32.927+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.928707+0000) 2019-10-13T09:33:32.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:32.929+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.929543+0000) 2019-10-13T09:33:33.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:33.385+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.386824+0000) 2019-10-13T09:33:33.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:33.385+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.387055+0000) 2019-10-13T09:33:33.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:33.385+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.387355+0000) 2019-10-13T09:33:33.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:33.386+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.387593+0000) 2019-10-13T09:33:33.390 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:33.388+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.389153+0000) 2019-10-13T09:33:33.390 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:33.386+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.387977+0000) 2019-10-13T09:33:33.391 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:33.386+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.388378+0000) 2019-10-13T09:33:33.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:33.389+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.390652+0000) 2019-10-13T09:33:33.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:33.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:33.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:33.804+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.805102+0000) 2019-10-13T09:33:34.687 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:34.685+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:34.686562+0000) 2019-10-13T09:33:38.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:38.395+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.396480+0000) 2019-10-13T09:33:38.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:38.394+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.395544+0000) 2019-10-13T09:33:38.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:38.394+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.395803+0000) 2019-10-13T09:33:38.398 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:38.394+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.395894+0000) 2019-10-13T09:33:38.398 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:38.395+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.396403+0000) 2019-10-13T09:33:38.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:38.395+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.396555+0000) 2019-10-13T09:33:38.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:38.395+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.396810+0000) 2019-10-13T09:33:38.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:38.398+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.399382+0000) 2019-10-13T09:33:38.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:38.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:38.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:38.484+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.485885+0000) 2019-10-13T09:33:38.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:38.823+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.824817+0000) 2019-10-13T09:33:38.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:38.823+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.824807+0000) 2019-10-13T09:33:38.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:38.823+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.824817+0000) 2019-10-13T09:33:42.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:42.929+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.929725+0000) 2019-10-13T09:33:42.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:42.928+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.928861+0000) 2019-10-13T09:33:43.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:43.402+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.403867+0000) 2019-10-13T09:33:43.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:43.402+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.404337+0000) 2019-10-13T09:33:43.407 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:43.405+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.405876+0000) 2019-10-13T09:33:43.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:43.403+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.404682+0000) 2019-10-13T09:33:43.407 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:43.403+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.405025+0000) 2019-10-13T09:33:43.408 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:43.403+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.405284+0000) 2019-10-13T09:33:43.408 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:43.404+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.405456+0000) 2019-10-13T09:33:43.409 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:43.406+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.407116+0000) 2019-10-13T09:33:43.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:43.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168182 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:43.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:43.807+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.807595+0000) 2019-10-13T09:33:44.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:44.685+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:44.686762+0000) 2019-10-13T09:33:48.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:48.410+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.412372+0000) 2019-10-13T09:33:48.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:48.411+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.412879+0000) 2019-10-13T09:33:48.415 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:48.412+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.413984+0000) 2019-10-13T09:33:48.415 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:48.412+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.413574+0000) 2019-10-13T09:33:48.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:48.412+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.413650+0000) 2019-10-13T09:33:48.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:48.412+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.413852+0000) 2019-10-13T09:33:48.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:48.412+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.414030+0000) 2019-10-13T09:33:48.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:48.415+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.417089+0000) 2019-10-13T09:33:48.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:48.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:48.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:48.485+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.486086+0000) 2019-10-13T09:33:48.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:48.824+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.824995+0000) 2019-10-13T09:33:48.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:48.824+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.824954+0000) 2019-10-13T09:33:48.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:48.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.824995+0000) 2019-10-13T09:33:52.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:52.928+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.929899+0000) 2019-10-13T09:33:52.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:52.928+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.929065+0000) 2019-10-13T09:33:53.197 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:33:53.197 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:53.201 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:33:53.201 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:53.423 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:53.420+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.421779+0000) 2019-10-13T09:33:53.424 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:53.421+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.423143+0000) 2019-10-13T09:33:53.424 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:53.421+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.422425+0000) 2019-10-13T09:33:53.424 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:53.421+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.422461+0000) 2019-10-13T09:33:53.425 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:53.421+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.422543+0000) 2019-10-13T09:33:53.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:53.421+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.422953+0000) 2019-10-13T09:33:53.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:53.421+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.423049+0000) 2019-10-13T09:33:53.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:53.423+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.425659+0000) 2019-10-13T09:33:53.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:53.427+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:53.810 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:53.808+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.809994+0000) 2019-10-13T09:33:54.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:54.686+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:54.686953+0000) 2019-10-13T09:33:58.431 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:33:58.429+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.431015+0000) 2019-10-13T09:33:58.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:58.428+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.430219+0000) 2019-10-13T09:33:58.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:58.428+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.430331+0000) 2019-10-13T09:33:58.432 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:33:58.429+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.430584+0000) 2019-10-13T09:33:58.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:58.429+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.430862+0000) 2019-10-13T09:33:58.433 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:33:58.429+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.431047+0000) 2019-10-13T09:33:58.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:58.429+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.431204+0000) 2019-10-13T09:33:58.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:33:58.432+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.433826+0000) 2019-10-13T09:33:58.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:33:58.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168268 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:33:58.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:33:58.485+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.486225+0000) 2019-10-13T09:33:58.826 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:33:58.824+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.825154+0000) 2019-10-13T09:33:58.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:33:58.824+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.825143+0000) 2019-10-13T09:33:58.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:33:58.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.825153+0000) 2019-10-13T09:34:02.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:02.929+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.930067+0000) 2019-10-13T09:34:02.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:02.928+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.929211+0000) 2019-10-13T09:34:03.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:03.437+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.438084+0000) 2019-10-13T09:34:03.440 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:03.438+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.438717+0000) 2019-10-13T09:34:03.440 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:03.438+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.438779+0000) 2019-10-13T09:34:03.440 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:03.438+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.438901+0000) 2019-10-13T09:34:03.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:03.438+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.439219+0000) 2019-10-13T09:34:03.441 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:03.439+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.440419+0000) 2019-10-13T09:34:03.441 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:03.439+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.439647+0000) 2019-10-13T09:34:03.442 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:03.440+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.441712+0000) 2019-10-13T09:34:03.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:03.444+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:03.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:03.811+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.812451+0000) 2019-10-13T09:34:04.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:04.686+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:04.687138+0000) 2019-10-13T09:34:08.447 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:08.446+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.446479+0000) 2019-10-13T09:34:08.448 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:08.446+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.446736+0000) 2019-10-13T09:34:08.449 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:08.446+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.447568+0000) 2019-10-13T09:34:08.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:08.446+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.447131+0000) 2019-10-13T09:34:08.449 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:08.447+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.447405+0000) 2019-10-13T09:34:08.449 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:08.447+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.447482+0000) 2019-10-13T09:34:08.449 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:08.447+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.447739+0000) 2019-10-13T09:34:08.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:08.450+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.451154+0000) 2019-10-13T09:34:08.454 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:08.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:08.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:08.486+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.486388+0000) 2019-10-13T09:34:08.826 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:08.824+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.825328+0000) 2019-10-13T09:34:08.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:08.824+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.825331+0000) 2019-10-13T09:34:08.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:08.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.825331+0000) 2019-10-13T09:34:12.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:12.929+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.929359+0000) 2019-10-13T09:34:12.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:12.929+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.930252+0000) 2019-10-13T09:34:13.455 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:13.454+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.455495+0000) 2019-10-13T09:34:13.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:13.454+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.454718+0000) 2019-10-13T09:34:13.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:13.454+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.454895+0000) 2019-10-13T09:34:13.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:13.454+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.454980+0000) 2019-10-13T09:34:13.457 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:13.454+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.455217+0000) 2019-10-13T09:34:13.457 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:13.454+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.455352+0000) 2019-10-13T09:34:13.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:13.455+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.455556+0000) 2019-10-13T09:34:13.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:13.457+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.458270+0000) 2019-10-13T09:34:13.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:13.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:13.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:13.814+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.814895+0000) 2019-10-13T09:34:14.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:14.687+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:14.687358+0000) 2019-10-13T09:34:18.464 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:18.461+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.462903+0000) 2019-10-13T09:34:18.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:18.461+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.463283+0000) 2019-10-13T09:34:18.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:18.463+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.464012+0000) 2019-10-13T09:34:18.465 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:18.462+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.463486+0000) 2019-10-13T09:34:18.465 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:18.462+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.463760+0000) 2019-10-13T09:34:18.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:18.462+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.464013+0000) 2019-10-13T09:34:18.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:18.462+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.464114+0000) 2019-10-13T09:34:18.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:18.465+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.466663+0000) 2019-10-13T09:34:18.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:18.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:18.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:18.485+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.486529+0000) 2019-10-13T09:34:18.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:18.824+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.825506+0000) 2019-10-13T09:34:18.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:18.824+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.825495+0000) 2019-10-13T09:34:18.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:18.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.825506+0000) 2019-10-13T09:34:22.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:22.930+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.930434+0000) 2019-10-13T09:34:22.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:22.928+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.929538+0000) 2019-10-13T09:34:23.365 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:34:23.366 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:23.370 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:34:23.370 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:23.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:23.470+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.471762+0000) 2019-10-13T09:34:23.473 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:23.470+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.472031+0000) 2019-10-13T09:34:23.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:23.472+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.472839+0000) 2019-10-13T09:34:23.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:23.470+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.472289+0000) 2019-10-13T09:34:23.474 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:23.471+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.472471+0000) 2019-10-13T09:34:23.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:23.471+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.472706+0000) 2019-10-13T09:34:23.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:23.471+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.472782+0000) 2019-10-13T09:34:23.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:23.475+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.475995+0000) 2019-10-13T09:34:23.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:23.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168414 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:23.817 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:23.816+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.817243+0000) 2019-10-13T09:34:24.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:24.686+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:24.687524+0000) 2019-10-13T09:34:28.480 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:28.477+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.478971+0000) 2019-10-13T09:34:28.480 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:28.477+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.479155+0000) 2019-10-13T09:34:28.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:28.477+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.479340+0000) 2019-10-13T09:34:28.481 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:28.478+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.479846+0000) 2019-10-13T09:34:28.481 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:28.478+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.480197+0000) 2019-10-13T09:34:28.481 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:28.478+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.480350+0000) 2019-10-13T09:34:28.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:28.480+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.481451+0000) 2019-10-13T09:34:28.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:28.481+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.482559+0000) 2019-10-13T09:34:28.486 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:28.484+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:28.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:28.485+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.486618+0000) 2019-10-13T09:34:28.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:28.824+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.825707+0000) 2019-10-13T09:34:28.827 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:28.824+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.825707+0000) 2019-10-13T09:34:28.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:28.824+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.825707+0000) 2019-10-13T09:34:32.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:32.930+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.930575+0000) 2019-10-13T09:34:32.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:32.928+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.929710+0000) 2019-10-13T09:34:33.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:33.485+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.487282+0000) 2019-10-13T09:34:33.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:33.486+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.488130+0000) 2019-10-13T09:34:33.489 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:33.487+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.488652+0000) 2019-10-13T09:34:33.490 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:33.488+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.489697+0000) 2019-10-13T09:34:33.490 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:33.487+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.488877+0000) 2019-10-13T09:34:33.490 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:33.487+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.489007+0000) 2019-10-13T09:34:33.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:33.489+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.490358+0000) 2019-10-13T09:34:33.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:33.488+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.489625+0000) 2019-10-13T09:34:33.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:33.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:33.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:33.819+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.819572+0000) 2019-10-13T09:34:34.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:34.686+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:34.687652+0000) 2019-10-13T09:34:38.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:38.486+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.486754+0000) 2019-10-13T09:34:38.497 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:38.495+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.497425+0000) 2019-10-13T09:34:38.498 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:38.495+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.496626+0000) 2019-10-13T09:34:38.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:38.495+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.496913+0000) 2019-10-13T09:34:38.498 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:38.495+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.497002+0000) 2019-10-13T09:34:38.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:38.495+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.497138+0000) 2019-10-13T09:34:38.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:38.495+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.497289+0000) 2019-10-13T09:34:38.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:38.496+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.497562+0000) 2019-10-13T09:34:38.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:38.498+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.500377+0000) 2019-10-13T09:34:38.504 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:38.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168501 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:38.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:38.825+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.825874+0000) 2019-10-13T09:34:38.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:38.825+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.825872+0000) 2019-10-13T09:34:38.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:38.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.825874+0000) 2019-10-13T09:34:42.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:42.929+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.930759+0000) 2019-10-13T09:34:42.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:42.929+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.929839+0000) 2019-10-13T09:34:43.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:43.503+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.505049+0000) 2019-10-13T09:34:43.507 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:43.503+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.505175+0000) 2019-10-13T09:34:43.507 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:43.504+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.505440+0000) 2019-10-13T09:34:43.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:43.504+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.505599+0000) 2019-10-13T09:34:43.508 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:43.504+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.506671+0000) 2019-10-13T09:34:43.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:43.504+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.505944+0000) 2019-10-13T09:34:43.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:43.504+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.506129+0000) 2019-10-13T09:34:43.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:43.506+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.508586+0000) 2019-10-13T09:34:43.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:43.510+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:43.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:43.821+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.821956+0000) 2019-10-13T09:34:44.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:44.687+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:44.687813+0000) 2019-10-13T09:34:48.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:48.486+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.486938+0000) 2019-10-13T09:34:48.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:48.513+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.514157+0000) 2019-10-13T09:34:48.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:48.513+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.514339+0000) 2019-10-13T09:34:48.516 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:48.514+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.514557+0000) 2019-10-13T09:34:48.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:48.514+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.514762+0000) 2019-10-13T09:34:48.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:48.514+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.514972+0000) 2019-10-13T09:34:48.517 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:48.514+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.516076+0000) 2019-10-13T09:34:48.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:48.515+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.515726+0000) 2019-10-13T09:34:48.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:48.516+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.518009+0000) 2019-10-13T09:34:48.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:48.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:48.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:48.825+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.826065+0000) 2019-10-13T09:34:48.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:48.825+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.826073+0000) 2019-10-13T09:34:48.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:48.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.826065+0000) 2019-10-13T09:34:52.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:52.929+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.930032+0000) 2019-10-13T09:34:52.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:52.930+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.930947+0000) 2019-10-13T09:34:53.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:53.522+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.522926+0000) 2019-10-13T09:34:53.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:53.522+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.523042+0000) 2019-10-13T09:34:53.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:53.523+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.523495+0000) 2019-10-13T09:34:53.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:53.523+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.523421+0000) 2019-10-13T09:34:53.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:53.524+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.525110+0000) 2019-10-13T09:34:53.526 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:53.523+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.523688+0000) 2019-10-13T09:34:53.526 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:53.523+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.523989+0000) 2019-10-13T09:34:53.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:53.525+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.526333+0000) 2019-10-13T09:34:53.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:53.529+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:53.572 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:34:53.572 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:53.576 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:34:53.576 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:53.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:53.823+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.824519+0000) 2019-10-13T09:34:54.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:54.687+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:54.688000+0000) 2019-10-13T09:34:58.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:58.486+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.487121+0000) 2019-10-13T09:34:58.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:34:58.531+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.531502+0000) 2019-10-13T09:34:58.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:58.531+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.531592+0000) 2019-10-13T09:34:58.533 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:34:58.531+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.531805+0000) 2019-10-13T09:34:58.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:34:58.531+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.531973+0000) 2019-10-13T09:34:58.534 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:34:58.532+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.533325+0000) 2019-10-13T09:34:58.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:58.532+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.532481+0000) 2019-10-13T09:34:58.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:58.532+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.532774+0000) 2019-10-13T09:34:58.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:34:58.534+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.535346+0000) 2019-10-13T09:34:58.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:34:58.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:34:58.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:34:58.825+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.826210+0000) 2019-10-13T09:34:58.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:34:58.825+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.826242+0000) 2019-10-13T09:34:58.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:34:58.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.826210+0000) 2019-10-13T09:35:02.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:02.930+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.931072+0000) 2019-10-13T09:35:02.931 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:02.930+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.930143+0000) 2019-10-13T09:35:03.541 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:03.538+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.539934+0000) 2019-10-13T09:35:03.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:03.538+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.540130+0000) 2019-10-13T09:35:03.541 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:03.538+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.540312+0000) 2019-10-13T09:35:03.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:03.540+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.541588+0000) 2019-10-13T09:35:03.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:03.539+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.540531+0000) 2019-10-13T09:35:03.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:03.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.540989+0000) 2019-10-13T09:35:03.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:03.539+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.541037+0000) 2019-10-13T09:35:03.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:03.543+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.544308+0000) 2019-10-13T09:35:03.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:03.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:03.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:03.826+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.826649+0000) 2019-10-13T09:35:04.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:04.687+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:04.688168+0000) 2019-10-13T09:35:08.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:08.486+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.487301+0000) 2019-10-13T09:35:08.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:08.547+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.548953+0000) 2019-10-13T09:35:08.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:08.547+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.549134+0000) 2019-10-13T09:35:08.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:08.547+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.549361+0000) 2019-10-13T09:35:08.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:08.549+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.550652+0000) 2019-10-13T09:35:08.552 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:08.548+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.549822+0000) 2019-10-13T09:35:08.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:08.548+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.549960+0000) 2019-10-13T09:35:08.552 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:08.548+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.550060+0000) 2019-10-13T09:35:08.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:08.551+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.552621+0000) 2019-10-13T09:35:08.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:08.554+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:08.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:08.825+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.826432+0000) 2019-10-13T09:35:08.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:08.825+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.826412+0000) 2019-10-13T09:35:08.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:08.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.826413+0000) 2019-10-13T09:35:12.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:12.930+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.931262+0000) 2019-10-13T09:35:12.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:12.929+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.930272+0000) 2019-10-13T09:35:13.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:13.555+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.557215+0000) 2019-10-13T09:35:13.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:13.556+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.557616+0000) 2019-10-13T09:35:13.559 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:13.556+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.557739+0000) 2019-10-13T09:35:13.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:13.556+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.558020+0000) 2019-10-13T09:35:13.560 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:13.558+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.559719+0000) 2019-10-13T09:35:13.560 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:13.557+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.558413+0000) 2019-10-13T09:35:13.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:13.557+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.558563+0000) 2019-10-13T09:35:13.561 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:13.560+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.560865+0000) 2019-10-13T09:35:13.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:13.563+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:13.828 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:13.828+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.828615+0000) 2019-10-13T09:35:14.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:14.687+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:14.688360+0000) 2019-10-13T09:35:18.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:18.486+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.487502+0000) 2019-10-13T09:35:18.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:18.564+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.565872+0000) 2019-10-13T09:35:18.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:18.564+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.565976+0000) 2019-10-13T09:35:18.568 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:18.564+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.566119+0000) 2019-10-13T09:35:18.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:18.565+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.566413+0000) 2019-10-13T09:35:18.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:18.565+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.566626+0000) 2019-10-13T09:35:18.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:18.567+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.567852+0000) 2019-10-13T09:35:18.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:18.565+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.567017+0000) 2019-10-13T09:35:18.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:18.568+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.569586+0000) 2019-10-13T09:35:18.573 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:18.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:18.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:18.825+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.826600+0000) 2019-10-13T09:35:18.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:18.825+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.826600+0000) 2019-10-13T09:35:18.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:18.825+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.826600+0000) 2019-10-13T09:35:22.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:22.931+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.931430+0000) 2019-10-13T09:35:22.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:22.929+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.930442+0000) 2019-10-13T09:35:23.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:23.573+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.575023+0000) 2019-10-13T09:35:23.577 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:23.573+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.575211+0000) 2019-10-13T09:35:23.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:23.573+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.575387+0000) 2019-10-13T09:35:23.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:23.573+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.574624+0000) 2019-10-13T09:35:23.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:23.574+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.575594+0000) 2019-10-13T09:35:23.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:23.576+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.576771+0000) 2019-10-13T09:35:23.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:23.576+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.577428+0000) 2019-10-13T09:35:23.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:23.574+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.575961+0000) 2019-10-13T09:35:23.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:23.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:23.718 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:35:23.719 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:23.723 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:35:23.723 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:23.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:23.830+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.830880+0000) 2019-10-13T09:35:24.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:24.687+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:24.688575+0000) 2019-10-13T09:35:28.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:28.487+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.487617+0000) 2019-10-13T09:35:28.585 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:28.582+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.583627+0000) 2019-10-13T09:35:28.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:28.582+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.583880+0000) 2019-10-13T09:35:28.585 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:28.582+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.583944+0000) 2019-10-13T09:35:28.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:28.584+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.585030+0000) 2019-10-13T09:35:28.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:28.582+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.584286+0000) 2019-10-13T09:35:28.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:28.582+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.584382+0000) 2019-10-13T09:35:28.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:28.583+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.584664+0000) 2019-10-13T09:35:28.587 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:28.586+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.587422+0000) 2019-10-13T09:35:28.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:28.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:28.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:28.826+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.826742+0000) 2019-10-13T09:35:28.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:28.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.826742+0000) 2019-10-13T09:35:28.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:28.826+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.826785+0000) 2019-10-13T09:35:32.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:32.930+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.931611+0000) 2019-10-13T09:35:32.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:32.930+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.930634+0000) 2019-10-13T09:35:33.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:33.590+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.591306+0000) 2019-10-13T09:35:33.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:33.590+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.592446+0000) 2019-10-13T09:35:33.594 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:33.591+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.591906+0000) 2019-10-13T09:35:33.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:33.591+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.592053+0000) 2019-10-13T09:35:33.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:33.591+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.592191+0000) 2019-10-13T09:35:33.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:33.592+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.592467+0000) 2019-10-13T09:35:33.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:33.592+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.592895+0000) 2019-10-13T09:35:33.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:33.593+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.595594+0000) 2019-10-13T09:35:33.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:33.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:33.833 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:33.832+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.833140+0000) 2019-10-13T09:35:34.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:34.688+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:34.688796+0000) 2019-10-13T09:35:38.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:38.487+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.487753+0000) 2019-10-13T09:35:38.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:38.599+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.600069+0000) 2019-10-13T09:35:38.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:38.599+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.601513+0000) 2019-10-13T09:35:38.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:38.600+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.600539+0000) 2019-10-13T09:35:38.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:38.600+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.600818+0000) 2019-10-13T09:35:38.602 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:38.600+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.600966+0000) 2019-10-13T09:35:38.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:38.600+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.601147+0000) 2019-10-13T09:35:38.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:38.600+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.601350+0000) 2019-10-13T09:35:38.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:38.602+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.604589+0000) 2019-10-13T09:35:38.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:38.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168849 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:38.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:38.826+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.826948+0000) 2019-10-13T09:35:38.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:38.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.826957+0000) 2019-10-13T09:35:38.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:38.826+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.826956+0000) 2019-10-13T09:35:42.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:42.930+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.931741+0000) 2019-10-13T09:35:42.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:42.930+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.930845+0000) 2019-10-13T09:35:43.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:43.607+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.608271+0000) 2019-10-13T09:35:43.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:43.608+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.608482+0000) 2019-10-13T09:35:43.610 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:43.608+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.608638+0000) 2019-10-13T09:35:43.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:43.608+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.608846+0000) 2019-10-13T09:35:43.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:43.608+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.610461+0000) 2019-10-13T09:35:43.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:43.608+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.608950+0000) 2019-10-13T09:35:43.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:43.608+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.609287+0000) 2019-10-13T09:35:43.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:43.610+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.612090+0000) 2019-10-13T09:35:43.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:43.615+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:43.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:43.834+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.835338+0000) 2019-10-13T09:35:44.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:44.688+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:44.688996+0000) 2019-10-13T09:35:48.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:48.487+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.487928+0000) 2019-10-13T09:35:48.618 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:48.617+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.617525+0000) 2019-10-13T09:35:48.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:48.617+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.617636+0000) 2019-10-13T09:35:48.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:48.617+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.617823+0000) 2019-10-13T09:35:48.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:48.618+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.619722+0000) 2019-10-13T09:35:48.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:48.617+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.617994+0000) 2019-10-13T09:35:48.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:48.618+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.618458+0000) 2019-10-13T09:35:48.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:48.618+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.618558+0000) 2019-10-13T09:35:48.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:48.621+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.621754+0000) 2019-10-13T09:35:48.625 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:48.624+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168906 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:48.828 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:48.826+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.827147+0000) 2019-10-13T09:35:48.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:48.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.827154+0000) 2019-10-13T09:35:48.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:48.826+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.827154+0000) 2019-10-13T09:35:52.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:52.931+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.931895+0000) 2019-10-13T09:35:52.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:52.930+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.931043+0000) 2019-10-13T09:35:53.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:53.624+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.626365+0000) 2019-10-13T09:35:53.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:53.626+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.627678+0000) 2019-10-13T09:35:53.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:53.625+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.626870+0000) 2019-10-13T09:35:53.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:53.625+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.626966+0000) 2019-10-13T09:35:53.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:53.625+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.627203+0000) 2019-10-13T09:35:53.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:53.626+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.627414+0000) 2019-10-13T09:35:53.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:53.626+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.627577+0000) 2019-10-13T09:35:53.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:53.629+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.630126+0000) 2019-10-13T09:35:53.634 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:53.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:53.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:53.837+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.837947+0000) 2019-10-13T09:35:53.873 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:35:53.873 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:53.877 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:35:53.877 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:54.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:54.688+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:54.689162+0000) 2019-10-13T09:35:58.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:58.487+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.488119+0000) 2019-10-13T09:35:58.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:58.633+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.634737+0000) 2019-10-13T09:35:58.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:35:58.633+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.634915+0000) 2019-10-13T09:35:58.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:58.633+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.635082+0000) 2019-10-13T09:35:58.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:35:58.635+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.636306+0000) 2019-10-13T09:35:58.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:58.633+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.635330+0000) 2019-10-13T09:35:58.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:35:58.634+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.635910+0000) 2019-10-13T09:35:58.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:35:58.634+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.635806+0000) 2019-10-13T09:35:58.641 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:35:58.637+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.638625+0000) 2019-10-13T09:35:58.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:35:58.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:35:58.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:35:58.826+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.827331+0000) 2019-10-13T09:35:58.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:35:58.826+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.827319+0000) 2019-10-13T09:35:58.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:35:58.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.827330+0000) 2019-10-13T09:36:02.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:02.931+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.932060+0000) 2019-10-13T09:36:02.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:02.930+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.931240+0000) 2019-10-13T09:36:03.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:03.642+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.644090+0000) 2019-10-13T09:36:03.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:03.643+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.644466+0000) 2019-10-13T09:36:03.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:03.643+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.644639+0000) 2019-10-13T09:36:03.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:03.645+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.645954+0000) 2019-10-13T09:36:03.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:03.643+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.645205+0000) 2019-10-13T09:36:03.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:03.644+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.645448+0000) 2019-10-13T09:36:03.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:03.644+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.645521+0000) 2019-10-13T09:36:03.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:03.647+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.648208+0000) 2019-10-13T09:36:03.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:03.650+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 168994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:03.840 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:03.839+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.840336+0000) 2019-10-13T09:36:04.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:04.688+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:04.689327+0000) 2019-10-13T09:36:08.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:08.487+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.488316+0000) 2019-10-13T09:36:08.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:08.652+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.653938+0000) 2019-10-13T09:36:08.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:08.652+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.654262+0000) 2019-10-13T09:36:08.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:08.655+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.655807+0000) 2019-10-13T09:36:08.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:08.652+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.654382+0000) 2019-10-13T09:36:08.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:08.653+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.654974+0000) 2019-10-13T09:36:08.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:08.653+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.655155+0000) 2019-10-13T09:36:08.657 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:08.654+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.655419+0000) 2019-10-13T09:36:08.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:08.657+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.657852+0000) 2019-10-13T09:36:08.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:08.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:08.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:08.826+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.827506+0000) 2019-10-13T09:36:08.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:08.826+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.827506+0000) 2019-10-13T09:36:08.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:08.826+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.827538+0000) 2019-10-13T09:36:12.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:12.932+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.932243+0000) 2019-10-13T09:36:12.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:12.930+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.931436+0000) 2019-10-13T09:36:13.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:13.661+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.662973+0000) 2019-10-13T09:36:13.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:13.661+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.663336+0000) 2019-10-13T09:36:13.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:13.663+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.664561+0000) 2019-10-13T09:36:13.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:13.662+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.663554+0000) 2019-10-13T09:36:13.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:13.662+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.663783+0000) 2019-10-13T09:36:13.666 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:13.662+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.663954+0000) 2019-10-13T09:36:13.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:13.662+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.664277+0000) 2019-10-13T09:36:13.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:13.666+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.667536+0000) 2019-10-13T09:36:13.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:13.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169053 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:13.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:13.842+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.842524+0000) 2019-10-13T09:36:14.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:14.689+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:14.689518+0000) 2019-10-13T09:36:18.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:18.488+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.488513+0000) 2019-10-13T09:36:18.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:18.671+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.671967+0000) 2019-10-13T09:36:18.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:18.671+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.672300+0000) 2019-10-13T09:36:18.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:18.672+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.672588+0000) 2019-10-13T09:36:18.674 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:18.672+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.672648+0000) 2019-10-13T09:36:18.675 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:18.673+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.674068+0000) 2019-10-13T09:36:18.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:18.672+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.673149+0000) 2019-10-13T09:36:18.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:18.672+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.673371+0000) 2019-10-13T09:36:18.676 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:18.675+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.675761+0000) 2019-10-13T09:36:18.679 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:18.678+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:18.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:18.827+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.827680+0000) 2019-10-13T09:36:18.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:18.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.827667+0000) 2019-10-13T09:36:18.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:18.827+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.827668+0000) 2019-10-13T09:36:22.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:22.931+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:22.932401+0000) 2019-10-13T09:36:22.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:22.931+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:22.931633+0000) 2019-10-13T09:36:23.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:23.679+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.680346+0000) 2019-10-13T09:36:23.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:23.680+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.680874+0000) 2019-10-13T09:36:23.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:23.679+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.681537+0000) 2019-10-13T09:36:23.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:23.680+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.681261+0000) 2019-10-13T09:36:23.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:23.680+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.681329+0000) 2019-10-13T09:36:23.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:23.681+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.681659+0000) 2019-10-13T09:36:23.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:23.681+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.681840+0000) 2019-10-13T09:36:23.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:23.682+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.683833+0000) 2019-10-13T09:36:23.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:23.688+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:23.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:23.843+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.844827+0000) 2019-10-13T09:36:24.035 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:36:24.035 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:24.040 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:36:24.040 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:24.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:24.689+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:24.689769+0000) 2019-10-13T09:36:28.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:28.488+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.488691+0000) 2019-10-13T09:36:28.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:28.689+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.690266+0000) 2019-10-13T09:36:28.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:28.689+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.690334+0000) 2019-10-13T09:36:28.692 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:28.690+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.690494+0000) 2019-10-13T09:36:28.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:28.690+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.692020+0000) 2019-10-13T09:36:28.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:28.691+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.691511+0000) 2019-10-13T09:36:28.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:28.691+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.691689+0000) 2019-10-13T09:36:28.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:28.691+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.691894+0000) 2019-10-13T09:36:28.694 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:28.692+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.694509+0000) 2019-10-13T09:36:28.698 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:28.697+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169138 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:28.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:28.827+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.827867+0000) 2019-10-13T09:36:28.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:28.827+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.827870+0000) 2019-10-13T09:36:28.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:28.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.827867+0000) 2019-10-13T09:36:32.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:32.931+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:32.931786+0000) 2019-10-13T09:36:32.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:32.931+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:32.932584+0000) 2019-10-13T09:36:33.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:33.698+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.699125+0000) 2019-10-13T09:36:33.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:33.699+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.699408+0000) 2019-10-13T09:36:33.701 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:33.699+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.699719+0000) 2019-10-13T09:36:33.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:33.699+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.700046+0000) 2019-10-13T09:36:33.702 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:33.699+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.701359+0000) 2019-10-13T09:36:33.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:33.699+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.700211+0000) 2019-10-13T09:36:33.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:33.700+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.700462+0000) 2019-10-13T09:36:33.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:33.700+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.702665+0000) 2019-10-13T09:36:33.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:33.706+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169169 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:33.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:33.846+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.847200+0000) 2019-10-13T09:36:34.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:34.688+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:34.689944+0000) 2019-10-13T09:36:38.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:38.487+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.488840+0000) 2019-10-13T09:36:38.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:38.707+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.708630+0000) 2019-10-13T09:36:38.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:38.707+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.708874+0000) 2019-10-13T09:36:38.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:38.707+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.708962+0000) 2019-10-13T09:36:38.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:38.709+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.710422+0000) 2019-10-13T09:36:38.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:38.710+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.711498+0000) 2019-10-13T09:36:38.728 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:38.707+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.709241+0000) 2019-10-13T09:36:38.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:38.708+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.709868+0000) 2019-10-13T09:36:38.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:38.708+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.709935+0000) 2019-10-13T09:36:38.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:38.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169198 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:38.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:38.827+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.828070+0000) 2019-10-13T09:36:38.842 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:38.827+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.828068+0000) 2019-10-13T09:36:38.842 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:38.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.828069+0000) 2019-10-13T09:36:42.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:42.932+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:42.932722+0000) 2019-10-13T09:36:42.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:42.931+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:42.931959+0000) 2019-10-13T09:36:43.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:43.715+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.717104+0000) 2019-10-13T09:36:43.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:43.718+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.718825+0000) 2019-10-13T09:36:43.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:43.716+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.717734+0000) 2019-10-13T09:36:43.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:43.716+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.718142+0000) 2019-10-13T09:36:43.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:43.717+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.718469+0000) 2019-10-13T09:36:43.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:43.717+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.718616+0000) 2019-10-13T09:36:43.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:43.717+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.718894+0000) 2019-10-13T09:36:43.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:43.720+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.721482+0000) 2019-10-13T09:36:43.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:43.723+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:43.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:43.848+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.849627+0000) 2019-10-13T09:36:44.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:44.688+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:44.690133+0000) 2019-10-13T09:36:48.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:48.488+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.489019+0000) 2019-10-13T09:36:48.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:48.725+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.726644+0000) 2019-10-13T09:36:48.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:48.727+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.728516+0000) 2019-10-13T09:36:48.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:48.725+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.727097+0000) 2019-10-13T09:36:48.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:48.725+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.727081+0000) 2019-10-13T09:36:48.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:48.726+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.727458+0000) 2019-10-13T09:36:48.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:48.726+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.727797+0000) 2019-10-13T09:36:48.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:48.726+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.728296+0000) 2019-10-13T09:36:48.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:48.730+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.730779+0000) 2019-10-13T09:36:48.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:48.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169256 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:48.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:48.827+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.828215+0000) 2019-10-13T09:36:48.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:48.827+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.828215+0000) 2019-10-13T09:36:48.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:48.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.828215+0000) 2019-10-13T09:36:52.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:52.932+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:52.932918+0000) 2019-10-13T09:36:52.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:52.931+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:52.932106+0000) 2019-10-13T09:36:53.736 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:53.733+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.735296+0000) 2019-10-13T09:36:53.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:53.734+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.735453+0000) 2019-10-13T09:36:53.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:53.736+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.736728+0000) 2019-10-13T09:36:53.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:53.734+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.736029+0000) 2019-10-13T09:36:53.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:53.734+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.736352+0000) 2019-10-13T09:36:53.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:53.735+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.736482+0000) 2019-10-13T09:36:53.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:53.735+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.736869+0000) 2019-10-13T09:36:53.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:53.737+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.738278+0000) 2019-10-13T09:36:53.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:53.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169287 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:53.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:53.851+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.851782+0000) 2019-10-13T09:36:54.156 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:36:54.156 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:54.161 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:36:54.161 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:54.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:54.689+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:54.690308+0000) 2019-10-13T09:36:58.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:58.488+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.489234+0000) 2019-10-13T09:36:58.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:36:58.744+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.745321+0000) 2019-10-13T09:36:58.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:36:58.743+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.744560+0000) 2019-10-13T09:36:58.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:58.743+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.744753+0000) 2019-10-13T09:36:58.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:36:58.743+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.744971+0000) 2019-10-13T09:36:58.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:58.743+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.745070+0000) 2019-10-13T09:36:58.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:58.743+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.745202+0000) 2019-10-13T09:36:58.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:36:58.743+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.745371+0000) 2019-10-13T09:36:58.747 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:36:58.746+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.747201+0000) 2019-10-13T09:36:58.752 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:36:58.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:36:58.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:36:58.827+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.828431+0000) 2019-10-13T09:36:58.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:36:58.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.828429+0000) 2019-10-13T09:36:58.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:36:58.827+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.828429+0000) 2019-10-13T09:37:02.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:02.932+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:02.933102+0000) 2019-10-13T09:37:02.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:02.931+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:02.932276+0000) 2019-10-13T09:37:03.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:03.752+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.753010+0000) 2019-10-13T09:37:03.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:03.752+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.753289+0000) 2019-10-13T09:37:03.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:03.753+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.753516+0000) 2019-10-13T09:37:03.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:03.753+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.753663+0000) 2019-10-13T09:37:03.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:03.753+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.753796+0000) 2019-10-13T09:37:03.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:03.753+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.754055+0000) 2019-10-13T09:37:03.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:03.754+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.754914+0000) 2019-10-13T09:37:03.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:03.755+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.756569+0000) 2019-10-13T09:37:03.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:03.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:03.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:03.853+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.853790+0000) 2019-10-13T09:37:04.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:04.690+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:04.690495+0000) 2019-10-13T09:37:08.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:08.489+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.489427+0000) 2019-10-13T09:37:08.761 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:08.759+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.759451+0000) 2019-10-13T09:37:08.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:08.759+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.759667+0000) 2019-10-13T09:37:08.761 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:08.759+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.759803+0000) 2019-10-13T09:37:08.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:08.759+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.759944+0000) 2019-10-13T09:37:08.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:08.760+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.761123+0000) 2019-10-13T09:37:08.762 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:08.759+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.760388+0000) 2019-10-13T09:37:08.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:08.760+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.760784+0000) 2019-10-13T09:37:08.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:08.762+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.763367+0000) 2019-10-13T09:37:08.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:08.766+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:08.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:08.828+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.828584+0000) 2019-10-13T09:37:08.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:08.828+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.828579+0000) 2019-10-13T09:37:08.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:08.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.828583+0000) 2019-10-13T09:37:12.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:12.933+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:12.933271+0000) 2019-10-13T09:37:12.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:12.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:12.932452+0000) 2019-10-13T09:37:13.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:13.768+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.768481+0000) 2019-10-13T09:37:13.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:13.767+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.769707+0000) 2019-10-13T09:37:13.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:13.768+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.768949+0000) 2019-10-13T09:37:13.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:13.768+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.769047+0000) 2019-10-13T09:37:13.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:13.768+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.769257+0000) 2019-10-13T09:37:13.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:13.769+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.769549+0000) 2019-10-13T09:37:13.772 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:13.769+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.769700+0000) 2019-10-13T09:37:13.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:13.770+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.772423+0000) 2019-10-13T09:37:13.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:13.775+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169405 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:13.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:13.854+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.855851+0000) 2019-10-13T09:37:14.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:14.690+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:14.690690+0000) 2019-10-13T09:37:18.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:18.489+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.489600+0000) 2019-10-13T09:37:18.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:18.776+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.777319+0000) 2019-10-13T09:37:18.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:18.777+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.777487+0000) 2019-10-13T09:37:18.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:18.777+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.777687+0000) 2019-10-13T09:37:18.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:18.777+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.779178+0000) 2019-10-13T09:37:18.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:18.777+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.778229+0000) 2019-10-13T09:37:18.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:18.777+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.778361+0000) 2019-10-13T09:37:18.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:18.778+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.778772+0000) 2019-10-13T09:37:18.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:18.779+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.781349+0000) 2019-10-13T09:37:18.785 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:18.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169434 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:18.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:18.828+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.828727+0000) 2019-10-13T09:37:18.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:18.828+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.828763+0000) 2019-10-13T09:37:18.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:18.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.828762+0000) 2019-10-13T09:37:22.933 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:22.931+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:22.933469+0000) 2019-10-13T09:37:22.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:22.931+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:22.932631+0000) 2019-10-13T09:37:23.787 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:23.785+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.786423+0000) 2019-10-13T09:37:23.788 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:23.786+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.787857+0000) 2019-10-13T09:37:23.789 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:23.785+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.787164+0000) 2019-10-13T09:37:23.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:23.785+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.787257+0000) 2019-10-13T09:37:23.789 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:23.786+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.787465+0000) 2019-10-13T09:37:23.789 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:23.786+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.787625+0000) 2019-10-13T09:37:23.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:23.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.787667+0000) 2019-10-13T09:37:23.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:23.788+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.790056+0000) 2019-10-13T09:37:23.794 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:23.791+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:23.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:23.857+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.858652+0000) 2019-10-13T09:37:24.276 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:37:24.276 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:24.280 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:37:24.280 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:24.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:24.689+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:24.690837+0000) 2019-10-13T09:37:28.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:28.488+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.489792+0000) 2019-10-13T09:37:28.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:28.793+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.795079+0000) 2019-10-13T09:37:28.797 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:28.793+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.795298+0000) 2019-10-13T09:37:28.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:28.794+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.795512+0000) 2019-10-13T09:37:28.797 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:28.794+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.795664+0000) 2019-10-13T09:37:28.797 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:28.794+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.795826+0000) 2019-10-13T09:37:28.798 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:28.796+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.797195+0000) 2019-10-13T09:37:28.798 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:28.795+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.796447+0000) 2019-10-13T09:37:28.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:28.799+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.799817+0000) 2019-10-13T09:37:28.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:28.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:28.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:28.828+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.828882+0000) 2019-10-13T09:37:28.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:28.828+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.828889+0000) 2019-10-13T09:37:28.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:28.828+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.828889+0000) 2019-10-13T09:37:32.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:32.932+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:32.933646+0000) 2019-10-13T09:37:32.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:32.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:32.932812+0000) 2019-10-13T09:37:33.805 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:33.802+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.803616+0000) 2019-10-13T09:37:33.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:33.802+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.803800+0000) 2019-10-13T09:37:33.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:33.802+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.803973+0000) 2019-10-13T09:37:33.806 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:33.804+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.805205+0000) 2019-10-13T09:37:33.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:33.802+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.804330+0000) 2019-10-13T09:37:33.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:33.803+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.804483+0000) 2019-10-13T09:37:33.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:33.804+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.805511+0000) 2019-10-13T09:37:33.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:33.807+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.808513+0000) 2019-10-13T09:37:33.812 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:33.809+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169523 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:33.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:33.860+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.861465+0000) 2019-10-13T09:37:34.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:34.690+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:34.691021+0000) 2019-10-13T09:37:38.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:38.489+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.489987+0000) 2019-10-13T09:37:38.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:38.811+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.812467+0000) 2019-10-13T09:37:38.814 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:38.811+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.812724+0000) 2019-10-13T09:37:38.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:38.811+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.812910+0000) 2019-10-13T09:37:38.815 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:38.813+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.814516+0000) 2019-10-13T09:37:38.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:38.811+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.813215+0000) 2019-10-13T09:37:38.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:38.811+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.813207+0000) 2019-10-13T09:37:38.816 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:38.812+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.813841+0000) 2019-10-13T09:37:38.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:38.815+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.815931+0000) 2019-10-13T09:37:38.820 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:38.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:38.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:38.827+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.828979+0000) 2019-10-13T09:37:38.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:38.827+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.828979+0000) 2019-10-13T09:37:38.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:38.827+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.829018+0000) 2019-10-13T09:37:42.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:42.933+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:42.933821+0000) 2019-10-13T09:37:42.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:42.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:42.932981+0000) 2019-10-13T09:37:43.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:43.819+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.820880+0000) 2019-10-13T09:37:43.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:43.819+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.821120+0000) 2019-10-13T09:37:43.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:43.822+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.823055+0000) 2019-10-13T09:37:43.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:43.820+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.821676+0000) 2019-10-13T09:37:43.824 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:43.820+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.821835+0000) 2019-10-13T09:37:43.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:43.820+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.822159+0000) 2019-10-13T09:37:43.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:43.820+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.822254+0000) 2019-10-13T09:37:43.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:43.823+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.824587+0000) 2019-10-13T09:37:43.828 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:43.826+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:43.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:43.863+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.863540+0000) 2019-10-13T09:37:44.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:44.690+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:44.691198+0000) 2019-10-13T09:37:48.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:48.489+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.490151+0000) 2019-10-13T09:37:48.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:48.828+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.829130+0000) 2019-10-13T09:37:48.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:48.828+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.829178+0000) 2019-10-13T09:37:48.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:48.829+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.829538+0000) 2019-10-13T09:37:48.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:48.830+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.831506+0000) 2019-10-13T09:37:48.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:48.829+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.830217+0000) 2019-10-13T09:37:48.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:48.829+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.830268+0000) 2019-10-13T09:37:48.833 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:48.830+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.830836+0000) 2019-10-13T09:37:48.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:48.830+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.831036+0000) 2019-10-13T09:37:48.833 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:48.832+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.832934+0000) 2019-10-13T09:37:48.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:48.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169610 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:52.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:52.933+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:52.933965+0000) 2019-10-13T09:37:52.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:52.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:52.933146+0000) 2019-10-13T09:37:53.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:53.838+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.838551+0000) 2019-10-13T09:37:53.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:53.838+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.838645+0000) 2019-10-13T09:37:53.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:53.838+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.838833+0000) 2019-10-13T09:37:53.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:53.838+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.838962+0000) 2019-10-13T09:37:53.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:53.839+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.840393+0000) 2019-10-13T09:37:53.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:53.839+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.839513+0000) 2019-10-13T09:37:53.842 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:53.839+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.839628+0000) 2019-10-13T09:37:53.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:53.841+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.842345+0000) 2019-10-13T09:37:53.847 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:53.845+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:37:53.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:53.864+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.865638+0000) 2019-10-13T09:37:54.405 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:37:54.406 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:54.410 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:37:54.410 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:54.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:54.691+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:54.691419+0000) 2019-10-13T09:37:55.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:55.579+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:55.579419+0000) 2019-10-13T09:37:55.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:55.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:55.579496+0000) 2019-10-13T09:37:58.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:58.490+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.490294+0000) 2019-10-13T09:37:58.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:58.829+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.829327+0000) 2019-10-13T09:37:58.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:37:58.847+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.848570+0000) 2019-10-13T09:37:58.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:37:58.847+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.847859+0000) 2019-10-13T09:37:58.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:37:58.847+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.848099+0000) 2019-10-13T09:37:58.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:37:58.848+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.848609+0000) 2019-10-13T09:37:58.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:37:58.847+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.848344+0000) 2019-10-13T09:37:58.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:37:58.848+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.848461+0000) 2019-10-13T09:37:58.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:37:58.848+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.848893+0000) 2019-10-13T09:37:58.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:37:58.850+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.851186+0000) 2019-10-13T09:37:58.856 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:37:58.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:02.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:02.933+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:02.934112+0000) 2019-10-13T09:38:02.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:02.933+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:02.933339+0000) 2019-10-13T09:38:03.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:03.856+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.857340+0000) 2019-10-13T09:38:03.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:03.857+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.858551+0000) 2019-10-13T09:38:03.860 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:03.857+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.858218+0000) 2019-10-13T09:38:03.860 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:03.857+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.858220+0000) 2019-10-13T09:38:03.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:03.858+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.858435+0000) 2019-10-13T09:38:03.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:03.858+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.858622+0000) 2019-10-13T09:38:03.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:03.858+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.858870+0000) 2019-10-13T09:38:03.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:03.860+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.860888+0000) 2019-10-13T09:38:03.865 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:03.864+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:03.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:03.866+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.867724+0000) 2019-10-13T09:38:04.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:04.691+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:04.691597+0000) 2019-10-13T09:38:05.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:05.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:05.579660+0000) 2019-10-13T09:38:05.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:05.579+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:05.579620+0000) 2019-10-13T09:38:08.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:08.489+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.490468+0000) 2019-10-13T09:38:08.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:08.828+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.829454+0000) 2019-10-13T09:38:08.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:08.864+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.865551+0000) 2019-10-13T09:38:08.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:08.864+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.865898+0000) 2019-10-13T09:38:08.867 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:08.864+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.866082+0000) 2019-10-13T09:38:08.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:08.864+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.866320+0000) 2019-10-13T09:38:08.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:08.864+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.866293+0000) 2019-10-13T09:38:08.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:08.865+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.867524+0000) 2019-10-13T09:38:08.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:08.865+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.866726+0000) 2019-10-13T09:38:08.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:08.868+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.869904+0000) 2019-10-13T09:38:08.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:08.869+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:12.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:12.932+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:12.934320+0000) 2019-10-13T09:38:12.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:12.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:12.933556+0000) 2019-10-13T09:38:13.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:13.869+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.870753+0000) 2019-10-13T09:38:13.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:13.871+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.872732+0000) 2019-10-13T09:38:13.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:13.872+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.874099+0000) 2019-10-13T09:38:13.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:13.871+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.873231+0000) 2019-10-13T09:38:13.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:13.872+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.873516+0000) 2019-10-13T09:38:13.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:13.871+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.873358+0000) 2019-10-13T09:38:13.875 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:13.872+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.873656+0000) 2019-10-13T09:38:13.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:13.872+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.873821+0000) 2019-10-13T09:38:13.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:13.875+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.877237+0000) 2019-10-13T09:38:13.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:13.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169760 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:14.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:14.690+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:14.691766+0000) 2019-10-13T09:38:15.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:15.579+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:15.579846+0000) 2019-10-13T09:38:15.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:15.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:15.579846+0000) 2019-10-13T09:38:18.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:18.489+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.490635+0000) 2019-10-13T09:38:18.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:18.828+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.829606+0000) 2019-10-13T09:38:18.883 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:18.880+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.881808+0000) 2019-10-13T09:38:18.884 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:18.881+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.883166+0000) 2019-10-13T09:38:18.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:18.881+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.882460+0000) 2019-10-13T09:38:18.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:18.881+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.882591+0000) 2019-10-13T09:38:18.884 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:18.881+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.882824+0000) 2019-10-13T09:38:18.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:18.881+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.883144+0000) 2019-10-13T09:38:18.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:18.881+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.883363+0000) 2019-10-13T09:38:18.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:18.883+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.885480+0000) 2019-10-13T09:38:18.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:18.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:22.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:22.933+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:22.934539+0000) 2019-10-13T09:38:22.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:22.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:22.933749+0000) 2019-10-13T09:38:23.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:23.872+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.873068+0000) 2019-10-13T09:38:23.891 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:23.888+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.890182+0000) 2019-10-13T09:38:23.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:23.890+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.891490+0000) 2019-10-13T09:38:23.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:23.889+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.890576+0000) 2019-10-13T09:38:23.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:23.889+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.890847+0000) 2019-10-13T09:38:23.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:23.889+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.890871+0000) 2019-10-13T09:38:23.893 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:23.889+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.891006+0000) 2019-10-13T09:38:23.893 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:23.889+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.891295+0000) 2019-10-13T09:38:23.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:23.892+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.893363+0000) 2019-10-13T09:38:23.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:23.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:24.546 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:38:24.546 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:24.551 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:38:24.551 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:24.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:24.691+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:24.691944+0000) 2019-10-13T09:38:25.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:25.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:25.579998+0000) 2019-10-13T09:38:25.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:25.579+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:25.579998+0000) 2019-10-13T09:38:28.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:28.490+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.490802+0000) 2019-10-13T09:38:28.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:28.829+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.829793+0000) 2019-10-13T09:38:28.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:28.898+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.898927+0000) 2019-10-13T09:38:28.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:28.896+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.898061+0000) 2019-10-13T09:38:28.900 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:28.896+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.898372+0000) 2019-10-13T09:38:28.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:28.897+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.898628+0000) 2019-10-13T09:38:28.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:28.897+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.898743+0000) 2019-10-13T09:38:28.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:28.897+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.898962+0000) 2019-10-13T09:38:28.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:28.897+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.899082+0000) 2019-10-13T09:38:28.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:28.901+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.902238+0000) 2019-10-13T09:38:28.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:28.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:32.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:32.933+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:32.934694+0000) 2019-10-13T09:38:32.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:32.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:32.933928+0000) 2019-10-13T09:38:33.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:33.874+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.875392+0000) 2019-10-13T09:38:33.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:33.906+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.907242+0000) 2019-10-13T09:38:33.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:33.907+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.907516+0000) 2019-10-13T09:38:33.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:33.907+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.908615+0000) 2019-10-13T09:38:33.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:33.907+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.907797+0000) 2019-10-13T09:38:33.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:33.907+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.908079+0000) 2019-10-13T09:38:33.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:33.907+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.908092+0000) 2019-10-13T09:38:33.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:33.908+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.908465+0000) 2019-10-13T09:38:33.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:33.909+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.909993+0000) 2019-10-13T09:38:33.915 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:33.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:34.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:34.691+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:34.692119+0000) 2019-10-13T09:38:35.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:35.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:35.580205+0000) 2019-10-13T09:38:35.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:35.579+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:35.580205+0000) 2019-10-13T09:38:38.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:38.490+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.490953+0000) 2019-10-13T09:38:38.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:38.829+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.829956+0000) 2019-10-13T09:38:38.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:38.915+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.915915+0000) 2019-10-13T09:38:38.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:38.916+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.917178+0000) 2019-10-13T09:38:38.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:38.916+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.916442+0000) 2019-10-13T09:38:38.918 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:38.915+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.916368+0000) 2019-10-13T09:38:38.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:38.916+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.916598+0000) 2019-10-13T09:38:38.919 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:38.916+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.916929+0000) 2019-10-13T09:38:38.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:38.916+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.917063+0000) 2019-10-13T09:38:38.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:38.919+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.919910+0000) 2019-10-13T09:38:38.924 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:38.922+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:42.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:42.934+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:42.934824+0000) 2019-10-13T09:38:42.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:42.933+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:42.934105+0000) 2019-10-13T09:38:43.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:43.877+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.877701+0000) 2019-10-13T09:38:43.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:43.924+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.925026+0000) 2019-10-13T09:38:43.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:43.925+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.925496+0000) 2019-10-13T09:38:43.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:43.925+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.925693+0000) 2019-10-13T09:38:43.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:43.926+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.927049+0000) 2019-10-13T09:38:43.927 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:43.925+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.926213+0000) 2019-10-13T09:38:43.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:43.926+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.926534+0000) 2019-10-13T09:38:43.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:43.926+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.926966+0000) 2019-10-13T09:38:43.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:43.929+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.930342+0000) 2019-10-13T09:38:43.933 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:43.932+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169935 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:44.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:44.692+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:44.692249+0000) 2019-10-13T09:38:45.581 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:45.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:45.580447+0000) 2019-10-13T09:38:45.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:45.580+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:45.580447+0000) 2019-10-13T09:38:48.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:48.491+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.491166+0000) 2019-10-13T09:38:48.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:48.830+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.830134+0000) 2019-10-13T09:38:48.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:48.933+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.933926+0000) 2019-10-13T09:38:48.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:48.934+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.935417+0000) 2019-10-13T09:38:48.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:48.934+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.934532+0000) 2019-10-13T09:38:48.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:48.934+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.934758+0000) 2019-10-13T09:38:48.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:48.934+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.934867+0000) 2019-10-13T09:38:48.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:48.934+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.934981+0000) 2019-10-13T09:38:48.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:48.935+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.935470+0000) 2019-10-13T09:38:48.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:48.937+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.938363+0000) 2019-10-13T09:38:48.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:48.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169962 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:52.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:52.934+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:52.934948+0000) 2019-10-13T09:38:52.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:52.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:52.934253+0000) 2019-10-13T09:38:53.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:53.879+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.880036+0000) 2019-10-13T09:38:53.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:53.943+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.943866+0000) 2019-10-13T09:38:53.945 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:53.941+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.943033+0000) 2019-10-13T09:38:53.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:53.941+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.943214+0000) 2019-10-13T09:38:53.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:53.942+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.943537+0000) 2019-10-13T09:38:53.945 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:53.942+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.943837+0000) 2019-10-13T09:38:53.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:53.942+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.943771+0000) 2019-10-13T09:38:53.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:53.942+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.943992+0000) 2019-10-13T09:38:53.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:53.946+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.947089+0000) 2019-10-13T09:38:53.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:53.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 169994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:38:54.676 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:38:54.677 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:54.681 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:38:54.681 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:54.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:54.691+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:54.692405+0000) 2019-10-13T09:38:55.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:55.579+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:55.580664+0000) 2019-10-13T09:38:55.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:55.579+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:55.580664+0000) 2019-10-13T09:38:58.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:58.490+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.491358+0000) 2019-10-13T09:38:58.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:58.828+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.830319+0000) 2019-10-13T09:38:58.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:38:58.952+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.953757+0000) 2019-10-13T09:38:58.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:38:58.951+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.952465+0000) 2019-10-13T09:38:58.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:38:58.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.952631+0000) 2019-10-13T09:38:58.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:38:58.951+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.952808+0000) 2019-10-13T09:38:58.955 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:38:58.951+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.953208+0000) 2019-10-13T09:38:58.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:38:58.952+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.953518+0000) 2019-10-13T09:38:58.955 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:38:58.952+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.953576+0000) 2019-10-13T09:38:58.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:38:58.954+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.956181+0000) 2019-10-13T09:38:58.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:38:58.957+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:02.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:02.933+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:02.935101+0000) 2019-10-13T09:39:02.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:02.932+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:02.934351+0000) 2019-10-13T09:39:03.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:03.881+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.882344+0000) 2019-10-13T09:39:03.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:03.960+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.961827+0000) 2019-10-13T09:39:03.963 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:03.959+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.960915+0000) 2019-10-13T09:39:03.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:03.959+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.961163+0000) 2019-10-13T09:39:03.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:03.959+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.961081+0000) 2019-10-13T09:39:03.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:03.959+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.961319+0000) 2019-10-13T09:39:03.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:03.960+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.961454+0000) 2019-10-13T09:39:03.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:03.960+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.961879+0000) 2019-10-13T09:39:03.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:03.962+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.964476+0000) 2019-10-13T09:39:03.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:03.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:04.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:04.691+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:04.692569+0000) 2019-10-13T09:39:05.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:05.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:05.580879+0000) 2019-10-13T09:39:05.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:05.580+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:05.580879+0000) 2019-10-13T09:39:08.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:08.490+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.491563+0000) 2019-10-13T09:39:08.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:08.829+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.830479+0000) 2019-10-13T09:39:08.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:08.968+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.969841+0000) 2019-10-13T09:39:08.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:08.967+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.969197+0000) 2019-10-13T09:39:08.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:08.967+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.969387+0000) 2019-10-13T09:39:08.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:08.968+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.969788+0000) 2019-10-13T09:39:08.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:08.968+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.969987+0000) 2019-10-13T09:39:08.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:08.968+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.970162+0000) 2019-10-13T09:39:08.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:08.968+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.970375+0000) 2019-10-13T09:39:08.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:08.970+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.972506+0000) 2019-10-13T09:39:08.977 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:08.975+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:12.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:12.934+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:12.935289+0000) 2019-10-13T09:39:12.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:12.933+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:12.934496+0000) 2019-10-13T09:39:13.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:13.884+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.884762+0000) 2019-10-13T09:39:13.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:13.976+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.977823+0000) 2019-10-13T09:39:13.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:13.976+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.977985+0000) 2019-10-13T09:39:13.979 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:13.976+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.978129+0000) 2019-10-13T09:39:13.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:13.976+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.978331+0000) 2019-10-13T09:39:13.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:13.979+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.979899+0000) 2019-10-13T09:39:13.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:13.979+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.980573+0000) 2019-10-13T09:39:13.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:13.977+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.978991+0000) 2019-10-13T09:39:13.981 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:13.977+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.979209+0000) 2019-10-13T09:39:13.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:13.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170110 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:14.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:14.692+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:14.692750+0000) 2019-10-13T09:39:15.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:15.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:15.581097+0000) 2019-10-13T09:39:15.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:15.580+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:15.581097+0000) 2019-10-13T09:39:18.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:18.491+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.491762+0000) 2019-10-13T09:39:18.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:18.830+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.830636+0000) 2019-10-13T09:39:18.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:18.985+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.986287+0000) 2019-10-13T09:39:18.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:18.986+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.986606+0000) 2019-10-13T09:39:18.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:18.986+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.987505+0000) 2019-10-13T09:39:18.989 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:18.986+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.986879+0000) 2019-10-13T09:39:18.989 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:18.986+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.987098+0000) 2019-10-13T09:39:18.989 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:18.987+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.987492+0000) 2019-10-13T09:39:18.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:18.986+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.987365+0000) 2019-10-13T09:39:18.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:18.990+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.990788+0000) 2019-10-13T09:39:18.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:18.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:22.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:22.934+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:22.935459+0000) 2019-10-13T09:39:22.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:22.934+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:22.934641+0000) 2019-10-13T09:39:23.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:23.886+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.887095+0000) 2019-10-13T09:39:23.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:23.994+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.994956+0000) 2019-10-13T09:39:23.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:23.994+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.995125+0000) 2019-10-13T09:39:23.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:23.994+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.995352+0000) 2019-10-13T09:39:23.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:23.996+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.996830+0000) 2019-10-13T09:39:23.998 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:23.995+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.995596+0000) 2019-10-13T09:39:23.998 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:23.995+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.996009+0000) 2019-10-13T09:39:23.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:23.995+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.996189+0000) 2019-10-13T09:39:23.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:23.998+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.999301+0000) 2019-10-13T09:39:24.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:24.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:24.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:24.692+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:24.692921+0000) 2019-10-13T09:39:24.763 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:39:24.763 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:24.767 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:39:24.767 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:25.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:25.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:25.581318+0000) 2019-10-13T09:39:25.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:25.581+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:25.581318+0000) 2019-10-13T09:39:28.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:28.491+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.491890+0000) 2019-10-13T09:39:28.832 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:28.830+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.830806+0000) 2019-10-13T09:39:29.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:29.003+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.003739+0000) 2019-10-13T09:39:29.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:29.003+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.003812+0000) 2019-10-13T09:39:29.006 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:29.003+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.004074+0000) 2019-10-13T09:39:29.006 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:29.003+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.004241+0000) 2019-10-13T09:39:29.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:29.004+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.005549+0000) 2019-10-13T09:39:29.007 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:29.004+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.004742+0000) 2019-10-13T09:39:29.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:29.004+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.004932+0000) 2019-10-13T09:39:29.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:29.006+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:29.007135+0000) 2019-10-13T09:39:29.011 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:29.010+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:32.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:32.935+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:32.935654+0000) 2019-10-13T09:39:32.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:32.934+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:32.934860+0000) 2019-10-13T09:39:33.898 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:33.889+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.889486+0000) 2019-10-13T09:39:34.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:34.011+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.012191+0000) 2019-10-13T09:39:34.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:34.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.013347+0000) 2019-10-13T09:39:34.026 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:34.012+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.012727+0000) 2019-10-13T09:39:34.026 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:34.012+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.012862+0000) 2019-10-13T09:39:34.026 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:34.014+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.015418+0000) 2019-10-13T09:39:34.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:34.012+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.013047+0000) 2019-10-13T09:39:34.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:34.012+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.013228+0000) 2019-10-13T09:39:34.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:34.013+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.013625+0000) 2019-10-13T09:39:34.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:34.019+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:34.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:34.692+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.693092+0000) 2019-10-13T09:39:35.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:35.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:35.581478+0000) 2019-10-13T09:39:35.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:35.581+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:35.581478+0000) 2019-10-13T09:39:38.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:38.491+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.492047+0000) 2019-10-13T09:39:38.832 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:38.830+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.831048+0000) 2019-10-13T09:39:39.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:39.020+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.022355+0000) 2019-10-13T09:39:39.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:39.021+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.022636+0000) 2019-10-13T09:39:39.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:39.023+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.024374+0000) 2019-10-13T09:39:39.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:39.021+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.023008+0000) 2019-10-13T09:39:39.025 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:39.021+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.023128+0000) 2019-10-13T09:39:39.025 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:39.022+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.023570+0000) 2019-10-13T09:39:39.026 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:39.024+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.025623+0000) 2019-10-13T09:39:39.026 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:39.022+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:39.024187+0000) 2019-10-13T09:39:39.029 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:39.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:42.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:42.935+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:42.935854+0000) 2019-10-13T09:39:42.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:42.933+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:42.935049+0000) 2019-10-13T09:39:43.892 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:43.891+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.891891+0000) 2019-10-13T09:39:44.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:44.030+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.031472+0000) 2019-10-13T09:39:44.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:44.029+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.030563+0000) 2019-10-13T09:39:44.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:44.029+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.030861+0000) 2019-10-13T09:39:44.033 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:44.029+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.031018+0000) 2019-10-13T09:39:44.033 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:44.029+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.031319+0000) 2019-10-13T09:39:44.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:44.030+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.031642+0000) 2019-10-13T09:39:44.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:44.030+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.032305+0000) 2019-10-13T09:39:44.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:44.033+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.034613+0000) 2019-10-13T09:39:44.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:44.035+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170283 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:44.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:44.691+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.693263+0000) 2019-10-13T09:39:45.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:45.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:45.581679+0000) 2019-10-13T09:39:45.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:45.580+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:45.581679+0000) 2019-10-13T09:39:48.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:48.491+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.492230+0000) 2019-10-13T09:39:48.832 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:48.830+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.831237+0000) 2019-10-13T09:39:49.040 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:49.037+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.038999+0000) 2019-10-13T09:39:49.040 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:49.037+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.039206+0000) 2019-10-13T09:39:49.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:49.038+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.040646+0000) 2019-10-13T09:39:49.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:49.038+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.040121+0000) 2019-10-13T09:39:49.042 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:49.038+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.040176+0000) 2019-10-13T09:39:49.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:49.038+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.040343+0000) 2019-10-13T09:39:49.042 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:49.039+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.040470+0000) 2019-10-13T09:39:49.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:49.041+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:49.042890+0000) 2019-10-13T09:39:49.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:49.045+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:52.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:52.935+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:52.936033+0000) 2019-10-13T09:39:52.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:52.934+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:52.935238+0000) 2019-10-13T09:39:53.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:53.893+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.894140+0000) 2019-10-13T09:39:54.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:54.046+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.048298+0000) 2019-10-13T09:39:54.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:54.047+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.049611+0000) 2019-10-13T09:39:54.050 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:54.047+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.048988+0000) 2019-10-13T09:39:54.051 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:54.047+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.049191+0000) 2019-10-13T09:39:54.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:54.048+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.049449+0000) 2019-10-13T09:39:54.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:54.048+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.049617+0000) 2019-10-13T09:39:54.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:54.048+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.050056+0000) 2019-10-13T09:39:54.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:54.051+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.053648+0000) 2019-10-13T09:39:54.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:54.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170342 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:39:54.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:54.692+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.693456+0000) 2019-10-13T09:39:54.887 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:39:54.887 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:54.892 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:39:54.893 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:55.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:55.580+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:55.581889+0000) 2019-10-13T09:39:55.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:55.580+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:55.581889+0000) 2019-10-13T09:39:58.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:58.491+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.492380+0000) 2019-10-13T09:39:58.832 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:58.830+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.831376+0000) 2019-10-13T09:39:59.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:39:59.056+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.058642+0000) 2019-10-13T09:39:59.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:39:59.056+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.057777+0000) 2019-10-13T09:39:59.060 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:39:59.056+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.057928+0000) 2019-10-13T09:39:59.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:39:59.056+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.058084+0000) 2019-10-13T09:39:59.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:39:59.057+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.058760+0000) 2019-10-13T09:39:59.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:39:59.057+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.058910+0000) 2019-10-13T09:39:59.061 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:39:59.057+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.059192+0000) 2019-10-13T09:39:59.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:39:59.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:59.062528+0000) 2019-10-13T09:39:59.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:39:59.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:02.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:02.935+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:02.936195+0000) 2019-10-13T09:40:02.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:02.934+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:02.935372+0000) 2019-10-13T09:40:03.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:03.895+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.896533+0000) 2019-10-13T09:40:04.067 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:04.064+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.066252+0000) 2019-10-13T09:40:04.068 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:04.065+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.066738+0000) 2019-10-13T09:40:04.069 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:04.065+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.066637+0000) 2019-10-13T09:40:04.069 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:04.065+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.067057+0000) 2019-10-13T09:40:04.069 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:04.065+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.067074+0000) 2019-10-13T09:40:04.070 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:04.066+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.067539+0000) 2019-10-13T09:40:04.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:04.067+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.068270+0000) 2019-10-13T09:40:04.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:04.068+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.069370+0000) 2019-10-13T09:40:04.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:04.071+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170402 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:04.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:04.693+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.693605+0000) 2019-10-13T09:40:05.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:05.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:05.582112+0000) 2019-10-13T09:40:05.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:05.581+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:05.582112+0000) 2019-10-13T09:40:08.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:08.492+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.492583+0000) 2019-10-13T09:40:08.832 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:08.831+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.831560+0000) 2019-10-13T09:40:09.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:09.073+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.073413+0000) 2019-10-13T09:40:09.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:09.073+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.073640+0000) 2019-10-13T09:40:09.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:09.074+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.075258+0000) 2019-10-13T09:40:09.076 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:09.073+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.073971+0000) 2019-10-13T09:40:09.076 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:09.073+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.074286+0000) 2019-10-13T09:40:09.076 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:09.074+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.074526+0000) 2019-10-13T09:40:09.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:09.074+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.074771+0000) 2019-10-13T09:40:09.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:09.076+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:09.077574+0000) 2019-10-13T09:40:09.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:09.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:12.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:12.935+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:12.936369+0000) 2019-10-13T09:40:12.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:12.935+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:12.935540+0000) 2019-10-13T09:40:13.899 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:13.898+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.898911+0000) 2019-10-13T09:40:14.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:14.081+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.081983+0000) 2019-10-13T09:40:14.083 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:14.081+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.082123+0000) 2019-10-13T09:40:14.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:14.082+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.082442+0000) 2019-10-13T09:40:14.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:14.083+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.084437+0000) 2019-10-13T09:40:14.085 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:14.082+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.082652+0000) 2019-10-13T09:40:14.085 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:14.082+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.082801+0000) 2019-10-13T09:40:14.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:14.082+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.082810+0000) 2019-10-13T09:40:14.086 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:14.085+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.086070+0000) 2019-10-13T09:40:14.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:14.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170464 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:14.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:14.693+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.693734+0000) 2019-10-13T09:40:15.583 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:15.582+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:15.582326+0000) 2019-10-13T09:40:15.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:15.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:15.582326+0000) 2019-10-13T09:40:18.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:18.492+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.492789+0000) 2019-10-13T09:40:18.832 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:18.831+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.831721+0000) 2019-10-13T09:40:19.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:19.088+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.089165+0000) 2019-10-13T09:40:19.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:19.088+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.089315+0000) 2019-10-13T09:40:19.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:19.089+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.089477+0000) 2019-10-13T09:40:19.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:19.089+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.089748+0000) 2019-10-13T09:40:19.092 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:19.089+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.089855+0000) 2019-10-13T09:40:19.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:19.090+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.091026+0000) 2019-10-13T09:40:19.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:19.089+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.090220+0000) 2019-10-13T09:40:19.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:19.092+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:19.093297+0000) 2019-10-13T09:40:19.097 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:19.095+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:22.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:22.936+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:22.936526+0000) 2019-10-13T09:40:22.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:22.934+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:22.935723+0000) 2019-10-13T09:40:23.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:23.901+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.901355+0000) 2019-10-13T09:40:24.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:24.097+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.098633+0000) 2019-10-13T09:40:24.099 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:24.096+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.097880+0000) 2019-10-13T09:40:24.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:24.096+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.098019+0000) 2019-10-13T09:40:24.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:24.097+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.098554+0000) 2019-10-13T09:40:24.100 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:24.097+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.098616+0000) 2019-10-13T09:40:24.100 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:24.097+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.098785+0000) 2019-10-13T09:40:24.101 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:24.097+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.099116+0000) 2019-10-13T09:40:24.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:24.100+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.101257+0000) 2019-10-13T09:40:24.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:24.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:24.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:24.692+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.693902+0000) 2019-10-13T09:40:25.002 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:40:25.002 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:25.007 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:40:25.007 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:25.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:25.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:25.582523+0000) 2019-10-13T09:40:25.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:25.581+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:25.582522+0000) 2019-10-13T09:40:28.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:28.492+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.492951+0000) 2019-10-13T09:40:28.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:28.830+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.831887+0000) 2019-10-13T09:40:29.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:29.105+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.106608+0000) 2019-10-13T09:40:29.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:29.105+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.106833+0000) 2019-10-13T09:40:29.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:29.107+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.108322+0000) 2019-10-13T09:40:29.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:29.105+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.107383+0000) 2019-10-13T09:40:29.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:29.106+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.107570+0000) 2019-10-13T09:40:29.110 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:29.106+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.107962+0000) 2019-10-13T09:40:29.110 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:29.106+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.108106+0000) 2019-10-13T09:40:29.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:29.109+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:29.110715+0000) 2019-10-13T09:40:29.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:29.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170559 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:32.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:32.936+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:32.936695+0000) 2019-10-13T09:40:32.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:32.934+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:32.935920+0000) 2019-10-13T09:40:33.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:33.903+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.903773+0000) 2019-10-13T09:40:34.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:34.116+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.117065+0000) 2019-10-13T09:40:34.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:34.114+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.116259+0000) 2019-10-13T09:40:34.118 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:34.115+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.116496+0000) 2019-10-13T09:40:34.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:34.115+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.116926+0000) 2019-10-13T09:40:34.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:34.115+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.117259+0000) 2019-10-13T09:40:34.119 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:34.115+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.117163+0000) 2019-10-13T09:40:34.119 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:34.116+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.117554+0000) 2019-10-13T09:40:34.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:34.118+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.119453+0000) 2019-10-13T09:40:34.124 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:34.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170588 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:34.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:34.693+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.694076+0000) 2019-10-13T09:40:35.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:35.581+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:35.582710+0000) 2019-10-13T09:40:35.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:35.581+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:35.582710+0000) 2019-10-13T09:40:38.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:38.492+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.493081+0000) 2019-10-13T09:40:38.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:38.831+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.832092+0000) 2019-10-13T09:40:39.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:39.123+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.124553+0000) 2019-10-13T09:40:39.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:39.123+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.124684+0000) 2019-10-13T09:40:39.127 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:39.123+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.125232+0000) 2019-10-13T09:40:39.127 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:39.124+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.125528+0000) 2019-10-13T09:40:39.127 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:39.124+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.125646+0000) 2019-10-13T09:40:39.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:39.124+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.125805+0000) 2019-10-13T09:40:39.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:39.126+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.127006+0000) 2019-10-13T09:40:39.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:39.127+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:39.128212+0000) 2019-10-13T09:40:39.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:39.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:42.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:42.935+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:42.936943+0000) 2019-10-13T09:40:42.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:42.935+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:42.936105+0000) 2019-10-13T09:40:43.906 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:43.905+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.906208+0000) 2019-10-13T09:40:44.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:44.132+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.133875+0000) 2019-10-13T09:40:44.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:44.132+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.134000+0000) 2019-10-13T09:40:44.136 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:44.132+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.134191+0000) 2019-10-13T09:40:44.136 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:44.132+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.134382+0000) 2019-10-13T09:40:44.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:44.133+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.135634+0000) 2019-10-13T09:40:44.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:44.133+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.134945+0000) 2019-10-13T09:40:44.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:44.133+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.135296+0000) 2019-10-13T09:40:44.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:44.135+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.137396+0000) 2019-10-13T09:40:44.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:44.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170649 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:44.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:44.693+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.694226+0000) 2019-10-13T09:40:45.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:45.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:45.582890+0000) 2019-10-13T09:40:45.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:45.582+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:45.582890+0000) 2019-10-13T09:40:48.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:48.492+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.493254+0000) 2019-10-13T09:40:48.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:48.831+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.832266+0000) 2019-10-13T09:40:49.143 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:49.140+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.142303+0000) 2019-10-13T09:40:49.144 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:49.141+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.142447+0000) 2019-10-13T09:40:49.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:49.141+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.142719+0000) 2019-10-13T09:40:49.144 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:49.141+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.142818+0000) 2019-10-13T09:40:49.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:49.142+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.144484+0000) 2019-10-13T09:40:49.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:49.141+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.143197+0000) 2019-10-13T09:40:49.145 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:49.142+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.143530+0000) 2019-10-13T09:40:49.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:49.145+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:49.146943+0000) 2019-10-13T09:40:49.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:49.147+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170683 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:52.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:52.936+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:52.937127+0000) 2019-10-13T09:40:52.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:52.935+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:52.936296+0000) 2019-10-13T09:40:53.908 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:53.907+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.908506+0000) 2019-10-13T09:40:54.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:54.149+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.149952+0000) 2019-10-13T09:40:54.152 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:54.149+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.150358+0000) 2019-10-13T09:40:54.152 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:54.150+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.150475+0000) 2019-10-13T09:40:54.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:54.150+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.150639+0000) 2019-10-13T09:40:54.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:54.150+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.150949+0000) 2019-10-13T09:40:54.153 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:54.150+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.151114+0000) 2019-10-13T09:40:54.153 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:54.151+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.151798+0000) 2019-10-13T09:40:54.153 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:54.152+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.153274+0000) 2019-10-13T09:40:54.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:54.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170712 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:40:54.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:54.694+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.694349+0000) 2019-10-13T09:40:55.132 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:40:55.132 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:55.137 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:40:55.137 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:55.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:55.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:55.583011+0000) 2019-10-13T09:40:55.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:55.582+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:55.583011+0000) 2019-10-13T09:40:58.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:58.493+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.493444+0000) 2019-10-13T09:40:58.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:58.832+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.832437+0000) 2019-10-13T09:40:59.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:40:59.156+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.157111+0000) 2019-10-13T09:40:59.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:40:59.158+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.158830+0000) 2019-10-13T09:40:59.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:40:59.157+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.157852+0000) 2019-10-13T09:40:59.160 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:40:59.157+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.157965+0000) 2019-10-13T09:40:59.160 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:40:59.157+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.158197+0000) 2019-10-13T09:40:59.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:40:59.158+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.158449+0000) 2019-10-13T09:40:59.160 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:40:59.158+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.158575+0000) 2019-10-13T09:40:59.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:40:59.159+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:59.160599+0000) 2019-10-13T09:40:59.165 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:40:59.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170744 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:02.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:02.936+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:02.937332+0000) 2019-10-13T09:41:02.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:02.936+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:02.936507+0000) 2019-10-13T09:41:03.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:03.910+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.910860+0000) 2019-10-13T09:41:04.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:04.165+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.165831+0000) 2019-10-13T09:41:04.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:04.165+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.165904+0000) 2019-10-13T09:41:04.167 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:04.165+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.166294+0000) 2019-10-13T09:41:04.168 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:04.166+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.167242+0000) 2019-10-13T09:41:04.168 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:04.166+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.166637+0000) 2019-10-13T09:41:04.169 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:04.166+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.166473+0000) 2019-10-13T09:41:04.169 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:04.166+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.166493+0000) 2019-10-13T09:41:04.169 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:04.168+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.168828+0000) 2019-10-13T09:41:04.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:04.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:04.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:04.694+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.694516+0000) 2019-10-13T09:41:05.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:05.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:05.583130+0000) 2019-10-13T09:41:05.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:05.583+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:05.583130+0000) 2019-10-13T09:41:08.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:08.492+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.493635+0000) 2019-10-13T09:41:08.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:08.831+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.832627+0000) 2019-10-13T09:41:09.174 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:09.171+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.173081+0000) 2019-10-13T09:41:09.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:09.171+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.173255+0000) 2019-10-13T09:41:09.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:09.172+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.173559+0000) 2019-10-13T09:41:09.175 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:09.172+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.173631+0000) 2019-10-13T09:41:09.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:09.173+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.174431+0000) 2019-10-13T09:41:09.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:09.172+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.173936+0000) 2019-10-13T09:41:09.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:09.173+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.174621+0000) 2019-10-13T09:41:09.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:09.176+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:09.177226+0000) 2019-10-13T09:41:09.181 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:09.179+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170807 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:12.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:12.937+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:12.937489+0000) 2019-10-13T09:41:12.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:12.935+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:12.936694+0000) 2019-10-13T09:41:13.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:13.912+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.913070+0000) 2019-10-13T09:41:14.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:14.182+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.183173+0000) 2019-10-13T09:41:14.184 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:14.180+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.182359+0000) 2019-10-13T09:41:14.184 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:14.181+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.182491+0000) 2019-10-13T09:41:14.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:14.181+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.182803+0000) 2019-10-13T09:41:14.184 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:14.181+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.182837+0000) 2019-10-13T09:41:14.185 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:14.182+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.183520+0000) 2019-10-13T09:41:14.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:14.182+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.183752+0000) 2019-10-13T09:41:14.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:14.186+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.187078+0000) 2019-10-13T09:41:14.190 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:14.188+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:14.695 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:14.693+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.694675+0000) 2019-10-13T09:41:15.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:15.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:15.583286+0000) 2019-10-13T09:41:15.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:15.582+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:15.583286+0000) 2019-10-13T09:41:18.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:18.493+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.493842+0000) 2019-10-13T09:41:18.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:18.832+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.832841+0000) 2019-10-13T09:41:19.192 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:19.191+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.192328+0000) 2019-10-13T09:41:19.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:19.190+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.191492+0000) 2019-10-13T09:41:19.194 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:19.190+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.191630+0000) 2019-10-13T09:41:19.194 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:19.190+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.191755+0000) 2019-10-13T09:41:19.194 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:19.190+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.191875+0000) 2019-10-13T09:41:19.194 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:19.190+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.192023+0000) 2019-10-13T09:41:19.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:19.190+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.192264+0000) 2019-10-13T09:41:19.195 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:19.194+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:19.195207+0000) 2019-10-13T09:41:19.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:19.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170868 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:22.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:22.937+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:22.937643+0000) 2019-10-13T09:41:22.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:22.936+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:22.936824+0000) 2019-10-13T09:41:23.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:23.915+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.915306+0000) 2019-10-13T09:41:24.200 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:24.197+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.199102+0000) 2019-10-13T09:41:24.200 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:24.197+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.199233+0000) 2019-10-13T09:41:24.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:24.198+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.199613+0000) 2019-10-13T09:41:24.201 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:24.199+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.200500+0000) 2019-10-13T09:41:24.202 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:24.198+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.199757+0000) 2019-10-13T09:41:24.202 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:24.198+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.199901+0000) 2019-10-13T09:41:24.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:24.198+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.200386+0000) 2019-10-13T09:41:24.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:24.202+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.203446+0000) 2019-10-13T09:41:24.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:24.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:24.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:24.694+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.694862+0000) 2019-10-13T09:41:25.238 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:41:25.238 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:25.243 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:41:25.243 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:25.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:25.582+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:25.583476+0000) 2019-10-13T09:41:25.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:25.582+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:25.583476+0000) 2019-10-13T09:41:28.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:28.493+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.494011+0000) 2019-10-13T09:41:28.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:28.832+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.833021+0000) 2019-10-13T09:41:29.208 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:29.205+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.206940+0000) 2019-10-13T09:41:29.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:29.205+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.207268+0000) 2019-10-13T09:41:29.209 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:29.207+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.208699+0000) 2019-10-13T09:41:29.210 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:29.206+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.208113+0000) 2019-10-13T09:41:29.210 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:29.206+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.208260+0000) 2019-10-13T09:41:29.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:29.206+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.208355+0000) 2019-10-13T09:41:29.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:29.207+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.208622+0000) 2019-10-13T09:41:29.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:29.210+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:29.211005+0000) 2019-10-13T09:41:29.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:29.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:32.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:32.936+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:32.937825+0000) 2019-10-13T09:41:32.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:32.936+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:32.936981+0000) 2019-10-13T09:41:33.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:33.916+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.917813+0000) 2019-10-13T09:41:34.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:34.214+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.215930+0000) 2019-10-13T09:41:34.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:34.214+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.216119+0000) 2019-10-13T09:41:34.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:34.215+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.216591+0000) 2019-10-13T09:41:34.218 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:34.215+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.216770+0000) 2019-10-13T09:41:34.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:34.215+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.217013+0000) 2019-10-13T09:41:34.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:34.215+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.217097+0000) 2019-10-13T09:41:34.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:34.216+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.218711+0000) 2019-10-13T09:41:34.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:34.218+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.220674+0000) 2019-10-13T09:41:34.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:34.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170960 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:34.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:34.694+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.695055+0000) 2019-10-13T09:41:35.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:35.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:35.583681+0000) 2019-10-13T09:41:35.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:35.583+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:35.583681+0000) 2019-10-13T09:41:38.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:38.493+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.494223+0000) 2019-10-13T09:41:38.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:38.832+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.833192+0000) 2019-10-13T09:41:39.225 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:39.223+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.224340+0000) 2019-10-13T09:41:39.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:39.224+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.224997+0000) 2019-10-13T09:41:39.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:39.224+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.225054+0000) 2019-10-13T09:41:39.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:39.224+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.226412+0000) 2019-10-13T09:41:39.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:39.224+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.225154+0000) 2019-10-13T09:41:39.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:39.225+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.225478+0000) 2019-10-13T09:41:39.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:39.225+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.225604+0000) 2019-10-13T09:41:39.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:39.226+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:39.227780+0000) 2019-10-13T09:41:39.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:39.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 170992 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:42.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:42.937+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:42.937975+0000) 2019-10-13T09:41:42.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:42.936+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:42.937158+0000) 2019-10-13T09:41:43.920 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:43.919+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.920054+0000) 2019-10-13T09:41:44.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:44.233+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.233568+0000) 2019-10-13T09:41:44.235 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:44.233+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.233763+0000) 2019-10-13T09:41:44.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:44.233+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.234022+0000) 2019-10-13T09:41:44.236 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:44.233+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.235628+0000) 2019-10-13T09:41:44.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:44.233+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.234177+0000) 2019-10-13T09:41:44.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:44.233+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.234291+0000) 2019-10-13T09:41:44.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:44.234+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.235198+0000) 2019-10-13T09:41:44.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:44.236+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.237980+0000) 2019-10-13T09:41:44.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:44.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:44.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:44.694+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.695231+0000) 2019-10-13T09:41:45.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:45.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:45.583866+0000) 2019-10-13T09:41:45.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:45.583+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:45.583866+0000) 2019-10-13T09:41:48.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:48.494+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.494422+0000) 2019-10-13T09:41:48.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:48.833+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.833378+0000) 2019-10-13T09:41:49.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:49.242+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.242970+0000) 2019-10-13T09:41:49.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:49.242+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.243253+0000) 2019-10-13T09:41:49.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:49.244+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.245193+0000) 2019-10-13T09:41:49.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:49.243+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.243575+0000) 2019-10-13T09:41:49.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:49.243+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.243779+0000) 2019-10-13T09:41:49.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:49.243+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.243946+0000) 2019-10-13T09:41:49.246 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:49.244+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.244498+0000) 2019-10-13T09:41:49.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:49.245+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:49.246014+0000) 2019-10-13T09:41:49.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:49.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:52.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:52.937+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:52.938130+0000) 2019-10-13T09:41:52.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:52.936+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:52.937314+0000) 2019-10-13T09:41:53.922 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:53.921+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.922429+0000) 2019-10-13T09:41:54.252 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:54.251+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.252264+0000) 2019-10-13T09:41:54.253 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:54.250+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.251462+0000) 2019-10-13T09:41:54.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:54.250+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.251571+0000) 2019-10-13T09:41:54.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:54.250+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.251707+0000) 2019-10-13T09:41:54.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:54.250+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.251908+0000) 2019-10-13T09:41:54.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:54.250+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.251968+0000) 2019-10-13T09:41:54.255 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:54.250+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.252107+0000) 2019-10-13T09:41:54.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:54.255+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.256140+0000) 2019-10-13T09:41:54.258 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:54.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171084 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:41:54.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:54.694+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.695379+0000) 2019-10-13T09:41:55.377 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:41:55.377 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:55.381 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:41:55.381 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:55.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:55.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:55.584087+0000) 2019-10-13T09:41:55.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:55.583+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:55.584086+0000) 2019-10-13T09:41:58.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:58.493+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.494602+0000) 2019-10-13T09:41:58.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:58.832+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.833541+0000) 2019-10-13T09:41:59.260 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:41:59.259+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.259757+0000) 2019-10-13T09:41:59.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:41:59.257+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.258892+0000) 2019-10-13T09:41:59.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:41:59.257+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.258932+0000) 2019-10-13T09:41:59.261 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:41:59.257+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.259044+0000) 2019-10-13T09:41:59.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:41:59.258+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.259490+0000) 2019-10-13T09:41:59.262 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:41:59.258+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.259645+0000) 2019-10-13T09:41:59.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:41:59.258+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.259800+0000) 2019-10-13T09:41:59.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:41:59.261+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:59.262442+0000) 2019-10-13T09:41:59.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:41:59.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:02.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:02.937+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:02.938335+0000) 2019-10-13T09:42:02.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:02.936+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:02.937490+0000) 2019-10-13T09:42:03.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:03.924+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.924857+0000) 2019-10-13T09:42:04.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:04.268+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.268992+0000) 2019-10-13T09:42:04.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:04.266+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.268369+0000) 2019-10-13T09:42:04.270 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:04.267+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.268898+0000) 2019-10-13T09:42:04.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:04.267+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.269177+0000) 2019-10-13T09:42:04.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:04.267+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.269299+0000) 2019-10-13T09:42:04.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:04.268+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.269454+0000) 2019-10-13T09:42:04.271 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:04.268+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.269546+0000) 2019-10-13T09:42:04.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:04.270+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.271275+0000) 2019-10-13T09:42:04.277 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:04.275+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:04.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:04.694+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.695532+0000) 2019-10-13T09:42:05.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:05.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:05.584297+0000) 2019-10-13T09:42:05.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:05.583+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:05.584297+0000) 2019-10-13T09:42:08.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:08.494+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.494743+0000) 2019-10-13T09:42:08.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:08.833+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.833709+0000) 2019-10-13T09:42:09.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:09.276+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.278238+0000) 2019-10-13T09:42:09.280 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:09.277+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.278572+0000) 2019-10-13T09:42:09.280 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:09.277+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.279084+0000) 2019-10-13T09:42:09.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:09.277+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.279333+0000) 2019-10-13T09:42:09.281 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:09.279+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.280700+0000) 2019-10-13T09:42:09.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:09.278+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.279745+0000) 2019-10-13T09:42:09.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:09.278+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.279981+0000) 2019-10-13T09:42:09.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:09.280+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:09.281515+0000) 2019-10-13T09:42:09.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:09.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171179 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:12.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:12.938+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:12.938511+0000) 2019-10-13T09:42:12.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:12.937+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:12.937680+0000) 2019-10-13T09:42:13.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:13.926+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.927194+0000) 2019-10-13T09:42:14.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:14.285+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.287071+0000) 2019-10-13T09:42:14.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:14.286+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.287419+0000) 2019-10-13T09:42:14.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:14.287+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.288652+0000) 2019-10-13T09:42:14.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:14.286+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.288030+0000) 2019-10-13T09:42:14.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:14.286+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.288316+0000) 2019-10-13T09:42:14.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:14.286+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.288224+0000) 2019-10-13T09:42:14.290 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:14.287+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.288447+0000) 2019-10-13T09:42:14.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:14.290+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.290820+0000) 2019-10-13T09:42:14.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:14.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:14.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:14.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.695704+0000) 2019-10-13T09:42:15.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:15.583+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:15.584542+0000) 2019-10-13T09:42:15.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:15.583+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:15.584542+0000) 2019-10-13T09:42:18.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:18.494+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.494913+0000) 2019-10-13T09:42:18.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:18.833+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.833860+0000) 2019-10-13T09:42:19.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:19.295+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.297129+0000) 2019-10-13T09:42:19.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:19.295+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.297382+0000) 2019-10-13T09:42:19.299 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:19.296+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.297546+0000) 2019-10-13T09:42:19.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:19.296+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.297739+0000) 2019-10-13T09:42:19.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:19.298+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.299169+0000) 2019-10-13T09:42:19.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:19.296+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.298198+0000) 2019-10-13T09:42:19.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:19.296+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.298309+0000) 2019-10-13T09:42:19.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:19.300+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:19.301324+0000) 2019-10-13T09:42:19.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:19.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171240 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:22.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:22.938+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:22.938648+0000) 2019-10-13T09:42:22.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:22.937+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:22.937829+0000) 2019-10-13T09:42:23.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:23.928+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.929121+0000) 2019-10-13T09:42:24.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:24.304+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.304983+0000) 2019-10-13T09:42:24.306 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:24.304+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.305151+0000) 2019-10-13T09:42:24.307 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:24.304+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.306547+0000) 2019-10-13T09:42:24.308 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:24.305+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.305930+0000) 2019-10-13T09:42:24.308 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:24.305+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.306098+0000) 2019-10-13T09:42:24.308 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:24.305+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.306253+0000) 2019-10-13T09:42:24.308 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:24.306+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.306510+0000) 2019-10-13T09:42:24.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:24.307+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.308919+0000) 2019-10-13T09:42:24.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:24.311+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:24.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:24.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.695881+0000) 2019-10-13T09:42:25.483 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:42:25.483 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:25.487 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:42:25.487 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:25.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:25.584+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:25.584757+0000) 2019-10-13T09:42:25.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:25.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:25.584758+0000) 2019-10-13T09:42:28.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:28.494+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.495131+0000) 2019-10-13T09:42:28.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:28.833+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.833953+0000) 2019-10-13T09:42:29.323 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:29.312+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.313303+0000) 2019-10-13T09:42:29.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:29.313+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.313551+0000) 2019-10-13T09:42:29.324 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:29.313+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.313717+0000) 2019-10-13T09:42:29.324 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:29.313+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.315220+0000) 2019-10-13T09:42:29.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:29.315+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.317703+0000) 2019-10-13T09:42:29.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:29.313+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.313724+0000) 2019-10-13T09:42:29.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:29.313+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.313970+0000) 2019-10-13T09:42:29.332 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:29.313+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:29.314227+0000) 2019-10-13T09:42:29.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:29.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171303 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:32.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:32.937+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:32.938824+0000) 2019-10-13T09:42:32.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:32.937+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:32.937977+0000) 2019-10-13T09:42:33.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:33.930+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.931426+0000) 2019-10-13T09:42:34.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:34.320+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.320827+0000) 2019-10-13T09:42:34.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:34.320+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.321130+0000) 2019-10-13T09:42:34.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:34.321+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.323027+0000) 2019-10-13T09:42:34.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:34.321+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.321891+0000) 2019-10-13T09:42:34.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:34.321+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.322100+0000) 2019-10-13T09:42:34.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:34.321+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.322257+0000) 2019-10-13T09:42:34.324 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:34.322+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.322607+0000) 2019-10-13T09:42:34.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:34.323+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.325067+0000) 2019-10-13T09:42:34.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:34.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:34.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:34.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.696064+0000) 2019-10-13T09:42:35.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:35.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:35.584992+0000) 2019-10-13T09:42:35.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:35.584+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:35.584991+0000) 2019-10-13T09:42:38.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:38.495+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.495309+0000) 2019-10-13T09:42:38.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:38.834+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.834122+0000) 2019-10-13T09:42:39.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:39.329+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.329697+0000) 2019-10-13T09:42:39.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:39.329+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.329918+0000) 2019-10-13T09:42:39.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:39.329+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.330302+0000) 2019-10-13T09:42:39.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:39.330+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.330495+0000) 2019-10-13T09:42:39.332 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:39.330+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.330771+0000) 2019-10-13T09:42:39.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:39.330+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.330827+0000) 2019-10-13T09:42:39.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:39.331+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.332031+0000) 2019-10-13T09:42:39.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:39.332+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:39.333290+0000) 2019-10-13T09:42:39.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:39.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171364 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:42.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:42.938+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:42.938979+0000) 2019-10-13T09:42:42.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:42.937+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:42.938159+0000) 2019-10-13T09:42:43.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:43.933+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.933909+0000) 2019-10-13T09:42:44.339 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:44.336+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.338016+0000) 2019-10-13T09:42:44.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:44.336+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.338147+0000) 2019-10-13T09:42:44.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:44.336+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.338216+0000) 2019-10-13T09:42:44.340 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:44.337+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.338503+0000) 2019-10-13T09:42:44.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:44.339+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.340027+0000) 2019-10-13T09:42:44.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:44.337+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.339028+0000) 2019-10-13T09:42:44.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:44.337+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.339107+0000) 2019-10-13T09:42:44.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:44.341+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.342003+0000) 2019-10-13T09:42:44.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:44.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:44.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:44.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.696198+0000) 2019-10-13T09:42:45.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:45.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:45.585210+0000) 2019-10-13T09:42:45.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:45.584+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:45.585210+0000) 2019-10-13T09:42:48.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:48.494+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.495497+0000) 2019-10-13T09:42:48.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:48.833+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.834274+0000) 2019-10-13T09:42:49.346 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:49.343+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.344910+0000) 2019-10-13T09:42:49.347 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:49.345+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.346165+0000) 2019-10-13T09:42:49.347 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:49.343+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.345358+0000) 2019-10-13T09:42:49.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:49.344+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.345559+0000) 2019-10-13T09:42:49.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:49.344+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.345684+0000) 2019-10-13T09:42:49.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:49.344+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.345807+0000) 2019-10-13T09:42:49.348 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:49.344+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.345967+0000) 2019-10-13T09:42:49.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:49.349+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:49.350218+0000) 2019-10-13T09:42:49.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:49.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:52.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:52.938+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:52.939153+0000) 2019-10-13T09:42:52.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:52.937+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:52.938284+0000) 2019-10-13T09:42:53.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:53.935+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.936227+0000) 2019-10-13T09:42:54.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:54.350+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.352027+0000) 2019-10-13T09:42:54.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:54.350+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.352299+0000) 2019-10-13T09:42:54.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:54.351+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.352526+0000) 2019-10-13T09:42:54.355 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:54.353+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.353829+0000) 2019-10-13T09:42:54.355 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:54.351+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.352959+0000) 2019-10-13T09:42:54.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:54.351+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.353068+0000) 2019-10-13T09:42:54.355 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:54.351+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.353322+0000) 2019-10-13T09:42:54.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:54.355+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.356056+0000) 2019-10-13T09:42:54.359 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:54.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:42:54.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:54.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.696342+0000) 2019-10-13T09:42:55.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:55.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:55.585369+0000) 2019-10-13T09:42:55.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:55.584+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:55.585369+0000) 2019-10-13T09:42:55.630 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:42:55.631 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:55.635 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:42:55.635 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:58.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:58.495+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.495672+0000) 2019-10-13T09:42:58.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:58.833+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.834481+0000) 2019-10-13T09:42:59.360 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:42:59.357+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.359373+0000) 2019-10-13T09:42:59.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:42:59.358+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.359530+0000) 2019-10-13T09:42:59.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:42:59.360+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.360913+0000) 2019-10-13T09:42:59.362 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:42:59.358+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.360239+0000) 2019-10-13T09:42:59.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:42:59.359+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.360452+0000) 2019-10-13T09:42:59.362 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:42:59.359+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.360723+0000) 2019-10-13T09:42:59.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:42:59.359+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.360960+0000) 2019-10-13T09:42:59.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:42:59.362+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:59.363203+0000) 2019-10-13T09:42:59.368 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:42:59.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171488 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:02.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:02.939+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:02.939345+0000) 2019-10-13T09:43:02.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:02.937+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:02.938466+0000) 2019-10-13T09:43:03.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:03.938+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.938308+0000) 2019-10-13T09:43:04.370 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:04.369+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.369838+0000) 2019-10-13T09:43:04.371 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:04.367+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.368832+0000) 2019-10-13T09:43:04.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:04.367+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.369063+0000) 2019-10-13T09:43:04.371 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:04.367+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.369274+0000) 2019-10-13T09:43:04.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:04.367+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.369381+0000) 2019-10-13T09:43:04.371 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:04.368+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.369534+0000) 2019-10-13T09:43:04.372 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:04.368+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.369576+0000) 2019-10-13T09:43:04.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:04.372+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.372937+0000) 2019-10-13T09:43:04.375 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:04.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:04.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:04.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.696493+0000) 2019-10-13T09:43:05.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:05.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:05.585556+0000) 2019-10-13T09:43:05.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:05.585+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:05.585556+0000) 2019-10-13T09:43:08.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:08.495+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.495827+0000) 2019-10-13T09:43:08.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:08.834+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.834672+0000) 2019-10-13T09:43:09.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:09.375+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.376163+0000) 2019-10-13T09:43:09.378 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:09.375+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.376331+0000) 2019-10-13T09:43:09.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:09.376+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.376515+0000) 2019-10-13T09:43:09.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:09.377+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.377989+0000) 2019-10-13T09:43:09.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:09.377+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.377458+0000) 2019-10-13T09:43:09.379 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:09.377+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.377721+0000) 2019-10-13T09:43:09.379 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:09.377+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.377850+0000) 2019-10-13T09:43:09.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:09.379+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:09.380492+0000) 2019-10-13T09:43:09.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:09.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:12.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:12.939+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:12.939523+0000) 2019-10-13T09:43:12.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:12.938+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:12.938624+0000) 2019-10-13T09:43:13.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:13.940+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.940746+0000) 2019-10-13T09:43:14.385 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:14.383+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.384368+0000) 2019-10-13T09:43:14.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:14.384+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.384888+0000) 2019-10-13T09:43:14.386 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:14.384+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.385031+0000) 2019-10-13T09:43:14.386 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:14.384+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.385214+0000) 2019-10-13T09:43:14.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:14.384+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.385339+0000) 2019-10-13T09:43:14.387 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:14.385+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.386385+0000) 2019-10-13T09:43:14.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:14.387+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.387745+0000) 2019-10-13T09:43:14.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:14.385+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.385758+0000) 2019-10-13T09:43:14.391 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:14.389+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:14.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:14.696+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.696646+0000) 2019-10-13T09:43:15.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:15.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:15.585710+0000) 2019-10-13T09:43:15.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:15.585+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:15.585710+0000) 2019-10-13T09:43:18.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:18.495+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.495980+0000) 2019-10-13T09:43:18.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:18.834+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.834854+0000) 2019-10-13T09:43:19.393 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:19.391+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.391554+0000) 2019-10-13T09:43:19.393 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:19.391+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.391829+0000) 2019-10-13T09:43:19.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:19.391+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.392075+0000) 2019-10-13T09:43:19.394 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:19.391+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.393666+0000) 2019-10-13T09:43:19.394 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:19.392+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.392724+0000) 2019-10-13T09:43:19.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:19.392+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.392859+0000) 2019-10-13T09:43:19.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:19.392+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.393171+0000) 2019-10-13T09:43:19.395 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:19.393+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:19.395621+0000) 2019-10-13T09:43:19.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:19.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:22.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:22.938+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:22.939711+0000) 2019-10-13T09:43:22.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:22.938+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:22.938781+0000) 2019-10-13T09:43:23.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:23.942+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.943127+0000) 2019-10-13T09:43:24.403 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:24.401+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.401463+0000) 2019-10-13T09:43:24.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:24.401+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.401788+0000) 2019-10-13T09:43:24.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:24.401+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.403144+0000) 2019-10-13T09:43:24.404 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:24.401+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.402211+0000) 2019-10-13T09:43:24.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:24.401+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.402387+0000) 2019-10-13T09:43:24.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:24.402+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.402630+0000) 2019-10-13T09:43:24.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:24.402+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.402795+0000) 2019-10-13T09:43:24.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:24.402+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.404461+0000) 2019-10-13T09:43:24.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:24.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:24.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:24.696+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.696752+0000) 2019-10-13T09:43:25.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:25.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:25.585894+0000) 2019-10-13T09:43:25.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:25.584+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:25.585894+0000) 2019-10-13T09:43:25.744 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:43:25.744 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:25.749 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:43:25.750 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:28.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:28.495+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.496170+0000) 2019-10-13T09:43:28.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:28.834+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.835062+0000) 2019-10-13T09:43:29.411 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:29.408+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.409881+0000) 2019-10-13T09:43:29.411 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:29.410+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.411475+0000) 2019-10-13T09:43:29.412 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:29.409+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.410647+0000) 2019-10-13T09:43:29.412 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:29.409+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.410783+0000) 2019-10-13T09:43:29.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:29.409+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.411049+0000) 2019-10-13T09:43:29.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:29.409+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.411267+0000) 2019-10-13T09:43:29.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:29.410+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.411824+0000) 2019-10-13T09:43:29.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:29.412+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:29.413716+0000) 2019-10-13T09:43:29.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:29.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:32.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:32.939+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:32.939823+0000) 2019-10-13T09:43:32.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:32.937+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:32.938972+0000) 2019-10-13T09:43:33.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:33.944+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.945430+0000) 2019-10-13T09:43:34.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:34.419+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.420217+0000) 2019-10-13T09:43:34.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:34.417+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.419257+0000) 2019-10-13T09:43:34.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:34.417+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.419388+0000) 2019-10-13T09:43:34.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:34.418+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.419505+0000) 2019-10-13T09:43:34.422 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:34.418+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.420101+0000) 2019-10-13T09:43:34.422 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:34.418+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.420149+0000) 2019-10-13T09:43:34.422 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:34.418+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.420335+0000) 2019-10-13T09:43:34.423 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:34.422+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.423588+0000) 2019-10-13T09:43:34.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:34.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:34.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:34.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.696903+0000) 2019-10-13T09:43:35.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:35.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:35.586064+0000) 2019-10-13T09:43:35.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:35.585+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:35.586064+0000) 2019-10-13T09:43:38.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:38.495+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.496345+0000) 2019-10-13T09:43:38.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:38.833+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.835267+0000) 2019-10-13T09:43:39.429 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:39.426+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.427562+0000) 2019-10-13T09:43:39.429 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:39.426+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.427779+0000) 2019-10-13T09:43:39.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:39.426+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.427964+0000) 2019-10-13T09:43:39.429 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:39.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.428118+0000) 2019-10-13T09:43:39.430 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:39.428+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.429350+0000) 2019-10-13T09:43:39.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:39.427+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.428477+0000) 2019-10-13T09:43:39.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:39.427+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.429358+0000) 2019-10-13T09:43:39.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:39.431+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:39.431773+0000) 2019-10-13T09:43:39.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:39.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171736 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:42.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:42.939+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:42.940034+0000) 2019-10-13T09:43:42.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:42.938+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:42.939149+0000) 2019-10-13T09:43:43.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:43.947+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.947669+0000) 2019-10-13T09:43:44.437 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:44.436+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.437677+0000) 2019-10-13T09:43:44.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:44.435+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.437046+0000) 2019-10-13T09:43:44.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:44.435+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.437209+0000) 2019-10-13T09:43:44.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:44.435+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.437370+0000) 2019-10-13T09:43:44.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:44.436+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.437602+0000) 2019-10-13T09:43:44.439 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:44.436+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.437935+0000) 2019-10-13T09:43:44.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:44.436+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.438105+0000) 2019-10-13T09:43:44.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:44.440+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.441115+0000) 2019-10-13T09:43:44.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:44.442+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:44.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:44.695+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.697089+0000) 2019-10-13T09:43:45.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:45.584+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:45.586314+0000) 2019-10-13T09:43:45.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:45.584+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:45.586314+0000) 2019-10-13T09:43:48.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:48.496+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.496558+0000) 2019-10-13T09:43:48.836 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:48.834+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.835489+0000) 2019-10-13T09:43:49.446 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:49.444+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.445679+0000) 2019-10-13T09:43:49.447 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:49.444+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.446154+0000) 2019-10-13T09:43:49.447 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:49.445+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.446467+0000) 2019-10-13T09:43:49.448 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:49.446+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.447469+0000) 2019-10-13T09:43:49.448 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:49.445+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.446935+0000) 2019-10-13T09:43:49.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:49.445+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.447089+0000) 2019-10-13T09:43:49.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:49.445+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.447310+0000) 2019-10-13T09:43:49.449 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:49.449+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:49.449785+0000) 2019-10-13T09:43:49.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:49.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:52.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:52.939+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:52.940222+0000) 2019-10-13T09:43:52.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:52.938+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:52.939269+0000) 2019-10-13T09:43:53.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:53.949+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.950185+0000) 2019-10-13T09:43:54.455 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:54.453+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.454142+0000) 2019-10-13T09:43:54.456 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:54.454+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.455454+0000) 2019-10-13T09:43:54.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:54.454+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.454507+0000) 2019-10-13T09:43:54.456 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:54.454+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.454659+0000) 2019-10-13T09:43:54.457 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:54.454+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.455064+0000) 2019-10-13T09:43:54.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:54.454+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.455373+0000) 2019-10-13T09:43:54.457 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:54.455+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.455895+0000) 2019-10-13T09:43:54.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:54.457+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.458425+0000) 2019-10-13T09:43:54.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:54.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:43:54.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:54.696+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.697267+0000) 2019-10-13T09:43:55.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:55.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:55.586511+0000) 2019-10-13T09:43:55.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:55.586+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:55.586510+0000) 2019-10-13T09:43:55.858 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:43:55.858 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:55.863 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:43:55.863 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:58.498 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:58.496+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.496792+0000) 2019-10-13T09:43:58.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:58.835+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.835705+0000) 2019-10-13T09:43:59.464 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:43:59.462+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.463155+0000) 2019-10-13T09:43:59.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:43:59.462+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.463347+0000) 2019-10-13T09:43:59.465 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:43:59.463+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.463456+0000) 2019-10-13T09:43:59.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:43:59.464+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.464746+0000) 2019-10-13T09:43:59.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:43:59.463+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.463741+0000) 2019-10-13T09:43:59.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:43:59.463+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.464000+0000) 2019-10-13T09:43:59.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:43:59.463+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.464237+0000) 2019-10-13T09:43:59.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:43:59.466+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:59.467372+0000) 2019-10-13T09:43:59.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:43:59.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171860 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:02.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:02.940+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:02.940403+0000) 2019-10-13T09:44:02.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:02.939+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:02.939438+0000) 2019-10-13T09:44:03.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:03.952+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.952627+0000) 2019-10-13T09:44:04.473 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:04.472+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.472837+0000) 2019-10-13T09:44:04.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:04.471+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.471965+0000) 2019-10-13T09:44:04.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:04.471+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.472359+0000) 2019-10-13T09:44:04.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:04.472+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.472708+0000) 2019-10-13T09:44:04.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:04.472+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.472885+0000) 2019-10-13T09:44:04.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:04.472+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.473084+0000) 2019-10-13T09:44:04.475 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:04.472+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.473173+0000) 2019-10-13T09:44:04.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:04.474+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.475370+0000) 2019-10-13T09:44:04.481 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:04.479+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171889 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:04.698 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:04.697+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.697459+0000) 2019-10-13T09:44:05.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:05.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:05.586699+0000) 2019-10-13T09:44:05.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:05.586+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:05.586699+0000) 2019-10-13T09:44:08.498 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:08.496+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.497013+0000) 2019-10-13T09:44:08.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:08.835+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.835889+0000) 2019-10-13T09:44:09.482 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:09.481+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.481438+0000) 2019-10-13T09:44:09.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:09.481+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.482786+0000) 2019-10-13T09:44:09.483 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:09.481+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.481948+0000) 2019-10-13T09:44:09.484 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:09.481+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.482251+0000) 2019-10-13T09:44:09.484 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:09.481+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.482249+0000) 2019-10-13T09:44:09.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:09.483+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.485628+0000) 2019-10-13T09:44:09.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:09.482+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.482906+0000) 2019-10-13T09:44:09.488 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:09.482+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:09.483081+0000) 2019-10-13T09:44:09.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:09.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:12.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:12.938+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:12.940550+0000) 2019-10-13T09:44:12.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:12.938+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:12.939631+0000) 2019-10-13T09:44:13.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:13.954+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.955040+0000) 2019-10-13T09:44:14.491 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:14.489+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.491482+0000) 2019-10-13T09:44:14.492 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:14.489+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.490981+0000) 2019-10-13T09:44:14.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:14.489+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.491117+0000) 2019-10-13T09:44:14.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:14.489+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.491204+0000) 2019-10-13T09:44:14.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:14.490+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.491534+0000) 2019-10-13T09:44:14.493 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:14.490+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.491628+0000) 2019-10-13T09:44:14.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:14.490+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.492167+0000) 2019-10-13T09:44:14.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:14.492+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.494583+0000) 2019-10-13T09:44:14.499 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:14.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171952 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:14.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:14.696+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.697627+0000) 2019-10-13T09:44:15.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:15.585+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:15.586898+0000) 2019-10-13T09:44:15.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:15.585+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:15.586898+0000) 2019-10-13T09:44:18.498 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:18.496+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.497191+0000) 2019-10-13T09:44:18.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:18.834+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.836041+0000) 2019-10-13T09:44:19.501 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:19.499+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.500887+0000) 2019-10-13T09:44:19.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:19.498+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.500309+0000) 2019-10-13T09:44:19.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:19.499+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.500574+0000) 2019-10-13T09:44:19.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:19.499+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.500807+0000) 2019-10-13T09:44:19.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:19.499+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.500871+0000) 2019-10-13T09:44:19.502 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:19.499+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.501005+0000) 2019-10-13T09:44:19.503 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:19.500+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.501505+0000) 2019-10-13T09:44:19.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:19.502+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:19.504489+0000) 2019-10-13T09:44:19.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:19.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 171984 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:22.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:22.939+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:22.940747+0000) 2019-10-13T09:44:22.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:22.939+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:22.939820+0000) 2019-10-13T09:44:23.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:23.956+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.957010+0000) 2019-10-13T09:44:24.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:24.507+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.509363+0000) 2019-10-13T09:44:24.511 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:24.508+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.509545+0000) 2019-10-13T09:44:24.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:24.508+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.509653+0000) 2019-10-13T09:44:24.512 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:24.510+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.511543+0000) 2019-10-13T09:44:24.512 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:24.508+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.509976+0000) 2019-10-13T09:44:24.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:24.508+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.509817+0000) 2019-10-13T09:44:24.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:24.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.510278+0000) 2019-10-13T09:44:24.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:24.513+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.513835+0000) 2019-10-13T09:44:24.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:24.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:24.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:24.697+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.697830+0000) 2019-10-13T09:44:25.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:25.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:25.587086+0000) 2019-10-13T09:44:25.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:25.586+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:25.587086+0000) 2019-10-13T09:44:25.996 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:44:25.996 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:26.000 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:44:26.001 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:28.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:28.496+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.497367+0000) 2019-10-13T09:44:28.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:28.835+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.836255+0000) 2019-10-13T09:44:29.526 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:29.517+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.518472+0000) 2019-10-13T09:44:29.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:29.517+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.518959+0000) 2019-10-13T09:44:29.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:29.517+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.519385+0000) 2019-10-13T09:44:29.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:29.518+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.519461+0000) 2019-10-13T09:44:29.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:29.519+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.519752+0000) 2019-10-13T09:44:29.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:29.520+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.521294+0000) 2019-10-13T09:44:29.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:29.518+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.519656+0000) 2019-10-13T09:44:29.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:29.518+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:29.519822+0000) 2019-10-13T09:44:29.534 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:29.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:32.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:32.940+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:32.940925+0000) 2019-10-13T09:44:32.941 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:32.939+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:32.940029+0000) 2019-10-13T09:44:33.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:33.958+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.959480+0000) 2019-10-13T09:44:34.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:34.525+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.527163+0000) 2019-10-13T09:44:34.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:34.525+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.527358+0000) 2019-10-13T09:44:34.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:34.526+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.527697+0000) 2019-10-13T09:44:34.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:34.526+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.527546+0000) 2019-10-13T09:44:34.530 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:34.528+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.529357+0000) 2019-10-13T09:44:34.530 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:34.526+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.528069+0000) 2019-10-13T09:44:34.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:34.527+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.528460+0000) 2019-10-13T09:44:34.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:34.531+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.531792+0000) 2019-10-13T09:44:34.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:34.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:34.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:34.697+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.698000+0000) 2019-10-13T09:44:35.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:35.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:35.587275+0000) 2019-10-13T09:44:35.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:35.586+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:35.587275+0000) 2019-10-13T09:44:38.498 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:38.497+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.497588+0000) 2019-10-13T09:44:38.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:38.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.836451+0000) 2019-10-13T09:44:39.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:39.535+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.536090+0000) 2019-10-13T09:44:39.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:39.536+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.536642+0000) 2019-10-13T09:44:39.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:39.535+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.536278+0000) 2019-10-13T09:44:39.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:39.536+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.536849+0000) 2019-10-13T09:44:39.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:39.536+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.536902+0000) 2019-10-13T09:44:39.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:39.536+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.537235+0000) 2019-10-13T09:44:39.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:39.537+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.538224+0000) 2019-10-13T09:44:39.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:39.539+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:39.539927+0000) 2019-10-13T09:44:39.544 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:39.542+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:42.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:42.940+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:42.941125+0000) 2019-10-13T09:44:42.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:42.939+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:42.940231+0000) 2019-10-13T09:44:43.962 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:43.961+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.961765+0000) 2019-10-13T09:44:44.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:44.545+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.545823+0000) 2019-10-13T09:44:44.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:44.544+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.544837+0000) 2019-10-13T09:44:44.547 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:44.544+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.545157+0000) 2019-10-13T09:44:44.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:44.545+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.545429+0000) 2019-10-13T09:44:44.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:44.545+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.545633+0000) 2019-10-13T09:44:44.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:44.545+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.545868+0000) 2019-10-13T09:44:44.548 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:44.545+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.545982+0000) 2019-10-13T09:44:44.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:44.548+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.548982+0000) 2019-10-13T09:44:44.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:44.551+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:44.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:44.697+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.698172+0000) 2019-10-13T09:44:45.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:45.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:45.587498+0000) 2019-10-13T09:44:45.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:45.587+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:45.587498+0000) 2019-10-13T09:44:48.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:48.497+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.497779+0000) 2019-10-13T09:44:48.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:48.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.836594+0000) 2019-10-13T09:44:49.555 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:49.554+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.554976+0000) 2019-10-13T09:44:49.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:49.553+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.554239+0000) 2019-10-13T09:44:49.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:49.554+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.554630+0000) 2019-10-13T09:44:49.557 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:49.554+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.554793+0000) 2019-10-13T09:44:49.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:49.554+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.555211+0000) 2019-10-13T09:44:49.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:49.554+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.555292+0000) 2019-10-13T09:44:49.558 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:49.555+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.555490+0000) 2019-10-13T09:44:49.559 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:49.556+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:49.557112+0000) 2019-10-13T09:44:49.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:49.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:52.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:52.940+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:52.940419+0000) 2019-10-13T09:44:52.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:52.940+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:52.941311+0000) 2019-10-13T09:44:53.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:53.964+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.964196+0000) 2019-10-13T09:44:54.564 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:54.563+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.564024+0000) 2019-10-13T09:44:54.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:54.562+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.563041+0000) 2019-10-13T09:44:54.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:54.563+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.563526+0000) 2019-10-13T09:44:54.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:54.563+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.563728+0000) 2019-10-13T09:44:54.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:54.563+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.563975+0000) 2019-10-13T09:44:54.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:54.563+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.564016+0000) 2019-10-13T09:44:54.566 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:54.563+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.564286+0000) 2019-10-13T09:44:54.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:54.566+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.566840+0000) 2019-10-13T09:44:54.571 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:54.569+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:44:54.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:54.698+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.698336+0000) 2019-10-13T09:44:55.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:55.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:55.587725+0000) 2019-10-13T09:44:55.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:55.587+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:55.587725+0000) 2019-10-13T09:44:56.120 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:44:56.120 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:56.171 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:44:56.172 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:58.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:58.496+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.497966+0000) 2019-10-13T09:44:58.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:58.835+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.836750+0000) 2019-10-13T09:44:59.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:44:59.570+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.571890+0000) 2019-10-13T09:44:59.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:44:59.570+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.572087+0000) 2019-10-13T09:44:59.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:44:59.570+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.572293+0000) 2019-10-13T09:44:59.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:44:59.571+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.572490+0000) 2019-10-13T09:44:59.575 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:44:59.571+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.573720+0000) 2019-10-13T09:44:59.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:44:59.573+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.574895+0000) 2019-10-13T09:44:59.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:44:59.571+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.572755+0000) 2019-10-13T09:44:59.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:44:59.571+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:59.573146+0000) 2019-10-13T09:44:59.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:44:59.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172232 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:02.942 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:02.939+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:02.941509+0000) 2019-10-13T09:45:02.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:02.939+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:02.940600+0000) 2019-10-13T09:45:03.966 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:03.965+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.966540+0000) 2019-10-13T09:45:04.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:04.578+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.580277+0000) 2019-10-13T09:45:04.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:04.579+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.580721+0000) 2019-10-13T09:45:04.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:04.579+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.580851+0000) 2019-10-13T09:45:04.583 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:04.580+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.582162+0000) 2019-10-13T09:45:04.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:04.579+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.581117+0000) 2019-10-13T09:45:04.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:04.579+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.581305+0000) 2019-10-13T09:45:04.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:04.580+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.581641+0000) 2019-10-13T09:45:04.584 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:04.582+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.584645+0000) 2019-10-13T09:45:04.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:04.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172261 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:04.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:04.697+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.698517+0000) 2019-10-13T09:45:05.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:05.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:05.587944+0000) 2019-10-13T09:45:05.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:05.587+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:05.587944+0000) 2019-10-13T09:45:08.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:08.497+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.498156+0000) 2019-10-13T09:45:08.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:08.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.836911+0000) 2019-10-13T09:45:09.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:09.587+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.589364+0000) 2019-10-13T09:45:09.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:09.588+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.589505+0000) 2019-10-13T09:45:09.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:09.588+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.589627+0000) 2019-10-13T09:45:09.592 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:09.588+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.589824+0000) 2019-10-13T09:45:09.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:09.588+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.590336+0000) 2019-10-13T09:45:09.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:09.589+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.591188+0000) 2019-10-13T09:45:09.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:09.589+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.590563+0000) 2019-10-13T09:45:09.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:09.591+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:09.593608+0000) 2019-10-13T09:45:09.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:09.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172295 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:12.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:12.940+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:12.941714+0000) 2019-10-13T09:45:12.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:12.940+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:12.940791+0000) 2019-10-13T09:45:13.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:13.968+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.968878+0000) 2019-10-13T09:45:14.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:14.597+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.599385+0000) 2019-10-13T09:45:14.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:14.598+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.599663+0000) 2019-10-13T09:45:14.602 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:14.598+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.599529+0000) 2019-10-13T09:45:14.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:14.598+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.599829+0000) 2019-10-13T09:45:14.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:14.598+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.600155+0000) 2019-10-13T09:45:14.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:14.599+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.600534+0000) 2019-10-13T09:45:14.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:14.601+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.601778+0000) 2019-10-13T09:45:14.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:14.602+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.603345+0000) 2019-10-13T09:45:14.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:14.605+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:14.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:14.698+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.698656+0000) 2019-10-13T09:45:15.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:15.586+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:15.588163+0000) 2019-10-13T09:45:15.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:15.586+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:15.588163+0000) 2019-10-13T09:45:18.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:18.497+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.498347+0000) 2019-10-13T09:45:18.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:18.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.837099+0000) 2019-10-13T09:45:19.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:19.607+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.608556+0000) 2019-10-13T09:45:19.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:19.607+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.608703+0000) 2019-10-13T09:45:19.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:19.609+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.610054+0000) 2019-10-13T09:45:19.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:19.607+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.609214+0000) 2019-10-13T09:45:19.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:19.608+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.609524+0000) 2019-10-13T09:45:19.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:19.608+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.609693+0000) 2019-10-13T09:45:19.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:19.608+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.609820+0000) 2019-10-13T09:45:19.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:19.612+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:19.613404+0000) 2019-10-13T09:45:19.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:19.614+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172356 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:22.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:22.941+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:22.941885+0000) 2019-10-13T09:45:22.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:22.940+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:22.940977+0000) 2019-10-13T09:45:23.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:23.970+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.971362+0000) 2019-10-13T09:45:24.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:24.617+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.618495+0000) 2019-10-13T09:45:24.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:24.617+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.617491+0000) 2019-10-13T09:45:24.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:24.617+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.617677+0000) 2019-10-13T09:45:24.620 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:24.617+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.617744+0000) 2019-10-13T09:45:24.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:24.617+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.617856+0000) 2019-10-13T09:45:24.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:24.617+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.618249+0000) 2019-10-13T09:45:24.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:24.617+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.618351+0000) 2019-10-13T09:45:24.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:24.619+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.620153+0000) 2019-10-13T09:45:24.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:24.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:24.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:24.698+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.698765+0000) 2019-10-13T09:45:25.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:25.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:25.588349+0000) 2019-10-13T09:45:25.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:25.587+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:25.588348+0000) 2019-10-13T09:45:26.348 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:45:26.349 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:26.355 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:45:26.356 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:28.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:28.498+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.498574+0000) 2019-10-13T09:45:28.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:28.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.837267+0000) 2019-10-13T09:45:29.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:29.623+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.623965+0000) 2019-10-13T09:45:29.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:29.623+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.624116+0000) 2019-10-13T09:45:29.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:29.624+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.625536+0000) 2019-10-13T09:45:29.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:29.624+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.624519+0000) 2019-10-13T09:45:29.627 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:29.624+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.624959+0000) 2019-10-13T09:45:29.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:29.624+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.625196+0000) 2019-10-13T09:45:29.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:29.625+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.625436+0000) 2019-10-13T09:45:29.628 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:29.627+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:29.627944+0000) 2019-10-13T09:45:29.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:29.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:32.942 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:32.941+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:32.942047+0000) 2019-10-13T09:45:32.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:32.940+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:32.941153+0000) 2019-10-13T09:45:33.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:33.973+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.973865+0000) 2019-10-13T09:45:34.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:34.632+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.632456+0000) 2019-10-13T09:45:34.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:34.632+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.632686+0000) 2019-10-13T09:45:34.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:34.632+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.632949+0000) 2019-10-13T09:45:34.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:34.633+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.633728+0000) 2019-10-13T09:45:34.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:34.632+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.633137+0000) 2019-10-13T09:45:34.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:34.633+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.633436+0000) 2019-10-13T09:45:34.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:34.633+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.633503+0000) 2019-10-13T09:45:34.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:34.636+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.636833+0000) 2019-10-13T09:45:34.639 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:34.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:34.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:34.698+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.698924+0000) 2019-10-13T09:45:35.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:35.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:35.588537+0000) 2019-10-13T09:45:35.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:35.588+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:35.588537+0000) 2019-10-13T09:45:38.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:38.498+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.498750+0000) 2019-10-13T09:45:38.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:38.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.837381+0000) 2019-10-13T09:45:39.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:39.640+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.641285+0000) 2019-10-13T09:45:39.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:39.640+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.640524+0000) 2019-10-13T09:45:39.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:39.640+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.640756+0000) 2019-10-13T09:45:39.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:39.640+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.641047+0000) 2019-10-13T09:45:39.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:39.640+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.641179+0000) 2019-10-13T09:45:39.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:39.640+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.641335+0000) 2019-10-13T09:45:39.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:39.641+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.641610+0000) 2019-10-13T09:45:39.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:39.642+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:39.643546+0000) 2019-10-13T09:45:39.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:39.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172480 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:42.942 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:42.941+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:42.942221+0000) 2019-10-13T09:45:42.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:42.940+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:42.941355+0000) 2019-10-13T09:45:43.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:43.976+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.976332+0000) 2019-10-13T09:45:44.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:44.649+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.650312+0000) 2019-10-13T09:45:44.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:44.648+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.649616+0000) 2019-10-13T09:45:44.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:44.648+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.649981+0000) 2019-10-13T09:45:44.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:44.648+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.650091+0000) 2019-10-13T09:45:44.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:44.649+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.650460+0000) 2019-10-13T09:45:44.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:44.649+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.650570+0000) 2019-10-13T09:45:44.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:44.649+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.650630+0000) 2019-10-13T09:45:44.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:44.652+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.653006+0000) 2019-10-13T09:45:44.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:44.655+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:44.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:44.697+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.699062+0000) 2019-10-13T09:45:45.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:45.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:45.588752+0000) 2019-10-13T09:45:45.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:45.587+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:45.588754+0000) 2019-10-13T09:45:48.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:48.498+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.498939+0000) 2019-10-13T09:45:48.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:48.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.837544+0000) 2019-10-13T09:45:49.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:49.657+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.659261+0000) 2019-10-13T09:45:49.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:49.658+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.659450+0000) 2019-10-13T09:45:49.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:49.658+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.659699+0000) 2019-10-13T09:45:49.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:49.658+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.659879+0000) 2019-10-13T09:45:49.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:49.658+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.660021+0000) 2019-10-13T09:45:49.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:49.659+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.660477+0000) 2019-10-13T09:45:49.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:49.659+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.660722+0000) 2019-10-13T09:45:49.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:49.662+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:49.663384+0000) 2019-10-13T09:45:49.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:49.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:52.942 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:52.940+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:52.942412+0000) 2019-10-13T09:45:52.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:52.940+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:52.941593+0000) 2019-10-13T09:45:53.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:53.977+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.978776+0000) 2019-10-13T09:45:54.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:54.666+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.667575+0000) 2019-10-13T09:45:54.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:54.666+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.667872+0000) 2019-10-13T09:45:54.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:54.666+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.667761+0000) 2019-10-13T09:45:54.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:54.667+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.668495+0000) 2019-10-13T09:45:54.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:54.667+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.668610+0000) 2019-10-13T09:45:54.670 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:54.667+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.668798+0000) 2019-10-13T09:45:54.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:54.667+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.669221+0000) 2019-10-13T09:45:54.671 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:54.669+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.671556+0000) 2019-10-13T09:45:54.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:54.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172572 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:45:54.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:54.698+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.699198+0000) 2019-10-13T09:45:55.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:55.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:55.588909+0000) 2019-10-13T09:45:55.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:55.588+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:55.588909+0000) 2019-10-13T09:45:56.503 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:45:56.503 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:56.507 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:45:56.508 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:58.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:58.498+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.499148+0000) 2019-10-13T09:45:58.838 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:58.837+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.837715+0000) 2019-10-13T09:45:59.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:45:59.675+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.676693+0000) 2019-10-13T09:45:59.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:45:59.676+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.678044+0000) 2019-10-13T09:45:59.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:45:59.675+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.677145+0000) 2019-10-13T09:45:59.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:45:59.676+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.677590+0000) 2019-10-13T09:45:59.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:45:59.676+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.677762+0000) 2019-10-13T09:45:59.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:45:59.676+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.677975+0000) 2019-10-13T09:45:59.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:45:59.676+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.678194+0000) 2019-10-13T09:45:59.681 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:45:59.680+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:59.681832+0000) 2019-10-13T09:45:59.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:45:59.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172604 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:02.942 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:02.941+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:02.942553+0000) 2019-10-13T09:46:02.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:02.941+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:02.941743+0000) 2019-10-13T09:46:03.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:03.980+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.981136+0000) 2019-10-13T09:46:04.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:04.684+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.685785+0000) 2019-10-13T09:46:04.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:04.684+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.685946+0000) 2019-10-13T09:46:04.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:04.684+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.686160+0000) 2019-10-13T09:46:04.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:04.687+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.687983+0000) 2019-10-13T09:46:04.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:04.685+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.686499+0000) 2019-10-13T09:46:04.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:04.685+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.687094+0000) 2019-10-13T09:46:04.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:04.685+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.687234+0000) 2019-10-13T09:46:04.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:04.689+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.690247+0000) 2019-10-13T09:46:04.694 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:04.691+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:04.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:04.697+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.699325+0000) 2019-10-13T09:46:05.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:05.587+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:05.589053+0000) 2019-10-13T09:46:05.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:05.587+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:05.589053+0000) 2019-10-13T09:46:08.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:08.498+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.499345+0000) 2019-10-13T09:46:08.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:08.837+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.837908+0000) 2019-10-13T09:46:09.696 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:09.694+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.694810+0000) 2019-10-13T09:46:09.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:09.695+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.696096+0000) 2019-10-13T09:46:09.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:09.694+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.695338+0000) 2019-10-13T09:46:09.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:09.695+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.695440+0000) 2019-10-13T09:46:09.697 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:09.695+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.695634+0000) 2019-10-13T09:46:09.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:09.695+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.695731+0000) 2019-10-13T09:46:09.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:09.695+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.695922+0000) 2019-10-13T09:46:09.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:09.698+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:09.699652+0000) 2019-10-13T09:46:09.703 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:09.701+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:12.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:12.942+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:12.942765+0000) 2019-10-13T09:46:12.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:12.941+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:12.941881+0000) 2019-10-13T09:46:13.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:13.982+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.983466+0000) 2019-10-13T09:46:14.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:14.699+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.699469+0000) 2019-10-13T09:46:14.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:14.703+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.703849+0000) 2019-10-13T09:46:14.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:14.704+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.705163+0000) 2019-10-13T09:46:14.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:14.703+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.704076+0000) 2019-10-13T09:46:14.706 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:14.703+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.704265+0000) 2019-10-13T09:46:14.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:14.703+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.704376+0000) 2019-10-13T09:46:14.707 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:14.704+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.704580+0000) 2019-10-13T09:46:14.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:14.704+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.704897+0000) 2019-10-13T09:46:14.708 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:14.707+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.708219+0000) 2019-10-13T09:46:14.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:14.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:15.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:15.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:15.589225+0000) 2019-10-13T09:46:15.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:15.588+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:15.589225+0000) 2019-10-13T09:46:18.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:18.499+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.499562+0000) 2019-10-13T09:46:18.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:18.837+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.838132+0000) 2019-10-13T09:46:19.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:19.712+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.712451+0000) 2019-10-13T09:46:19.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:19.713+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.713969+0000) 2019-10-13T09:46:19.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:19.714+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.715533+0000) 2019-10-13T09:46:19.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:19.712+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.712991+0000) 2019-10-13T09:46:19.887 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:19.712+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.713242+0000) 2019-10-13T09:46:19.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:19.712+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.713381+0000) 2019-10-13T09:46:19.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:19.713+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.713631+0000) 2019-10-13T09:46:19.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:19.713+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:19.713939+0000) 2019-10-13T09:46:19.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:19.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:22.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:22.942+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:22.942908+0000) 2019-10-13T09:46:22.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:22.941+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:22.942033+0000) 2019-10-13T09:46:23.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:23.985+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.985817+0000) 2019-10-13T09:46:24.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:24.699+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.699677+0000) 2019-10-13T09:46:24.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:24.721+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.722179+0000) 2019-10-13T09:46:24.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:24.722+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.722583+0000) 2019-10-13T09:46:24.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:24.723+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.724302+0000) 2019-10-13T09:46:24.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:24.722+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.722829+0000) 2019-10-13T09:46:24.725 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:24.722+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.723353+0000) 2019-10-13T09:46:24.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:24.723+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.723503+0000) 2019-10-13T09:46:24.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:24.723+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.723666+0000) 2019-10-13T09:46:24.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:24.724+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.725556+0000) 2019-10-13T09:46:24.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:24.728+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:25.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:25.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:25.589421+0000) 2019-10-13T09:46:25.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:25.589+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:25.589421+0000) 2019-10-13T09:46:26.658 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:46:26.658 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:26.663 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:46:26.663 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:28.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:28.498+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.499774+0000) 2019-10-13T09:46:28.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:28.836+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.838351+0000) 2019-10-13T09:46:29.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:29.729+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.731105+0000) 2019-10-13T09:46:29.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:29.730+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.731403+0000) 2019-10-13T09:46:29.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:29.730+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.731521+0000) 2019-10-13T09:46:29.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:29.732+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.733379+0000) 2019-10-13T09:46:29.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:29.730+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.732046+0000) 2019-10-13T09:46:29.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:29.730+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.732329+0000) 2019-10-13T09:46:29.734 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:29.731+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.732508+0000) 2019-10-13T09:46:29.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:29.734+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:29.735073+0000) 2019-10-13T09:46:29.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:29.736+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172791 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:32.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:32.942+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:32.943051+0000) 2019-10-13T09:46:32.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:32.940+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:32.942238+0000) 2019-10-13T09:46:33.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:33.987+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.988135+0000) 2019-10-13T09:46:34.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:34.698+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.699878+0000) 2019-10-13T09:46:34.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:34.738+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.740096+0000) 2019-10-13T09:46:34.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:34.738+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.740308+0000) 2019-10-13T09:46:34.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:34.739+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.740472+0000) 2019-10-13T09:46:34.742 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:34.741+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.742506+0000) 2019-10-13T09:46:34.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:34.739+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.740797+0000) 2019-10-13T09:46:34.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:34.740+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.741451+0000) 2019-10-13T09:46:34.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:34.740+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.741990+0000) 2019-10-13T09:46:34.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:34.743+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.744334+0000) 2019-10-13T09:46:34.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:34.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:35.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:35.588+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:35.589620+0000) 2019-10-13T09:46:35.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:35.588+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:35.589620+0000) 2019-10-13T09:46:38.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:38.499+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.499990+0000) 2019-10-13T09:46:38.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:38.837+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.838561+0000) 2019-10-13T09:46:39.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:39.749+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.749754+0000) 2019-10-13T09:46:39.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:39.747+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.748692+0000) 2019-10-13T09:46:39.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:39.747+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.748890+0000) 2019-10-13T09:46:39.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:39.748+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.749561+0000) 2019-10-13T09:46:39.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:39.748+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.749719+0000) 2019-10-13T09:46:39.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:39.748+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.749862+0000) 2019-10-13T09:46:39.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:39.748+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.750258+0000) 2019-10-13T09:46:39.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:39.752+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.752953+0000) 2019-10-13T09:46:39.756 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:39.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172852 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:42.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:42.942+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:42.943217+0000) 2019-10-13T09:46:42.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:42.941+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:42.942434+0000) 2019-10-13T09:46:43.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:43.989+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:43.990434+0000) 2019-10-13T09:46:44.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:44.699+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.700080+0000) 2019-10-13T09:46:44.758 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:44.756+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.758536+0000) 2019-10-13T09:46:44.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:44.756+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.757514+0000) 2019-10-13T09:46:44.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:44.756+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.757701+0000) 2019-10-13T09:46:44.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:44.756+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.757993+0000) 2019-10-13T09:46:44.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:44.756+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.758179+0000) 2019-10-13T09:46:44.760 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:44.756+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.758264+0000) 2019-10-13T09:46:44.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:44.757+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.758492+0000) 2019-10-13T09:46:44.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:44.760+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.761813+0000) 2019-10-13T09:46:44.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:44.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:45.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:45.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:45.589802+0000) 2019-10-13T09:46:45.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:45.589+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:45.589801+0000) 2019-10-13T09:46:48.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:48.498+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:48.500183+0000) 2019-10-13T09:46:48.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:48.838+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:48.838725+0000) 2019-10-13T09:46:49.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:49.764+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.766360+0000) 2019-10-13T09:46:49.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:49.765+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.766670+0000) 2019-10-13T09:46:49.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:49.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.768265+0000) 2019-10-13T09:46:49.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:49.765+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.766989+0000) 2019-10-13T09:46:49.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:49.765+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.767374+0000) 2019-10-13T09:46:49.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:49.766+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.767523+0000) 2019-10-13T09:46:49.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:49.766+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.767675+0000) 2019-10-13T09:46:49.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:49.769+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.771264+0000) 2019-10-13T09:46:49.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:49.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:52.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:52.942+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:52.943386+0000) 2019-10-13T09:46:52.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:52.941+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:52.942612+0000) 2019-10-13T09:46:53.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:53.991+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:53.992762+0000) 2019-10-13T09:46:54.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:54.699+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.700265+0000) 2019-10-13T09:46:54.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:54.773+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.775285+0000) 2019-10-13T09:46:54.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:54.775+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.776859+0000) 2019-10-13T09:46:54.778 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:54.774+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.775951+0000) 2019-10-13T09:46:54.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:54.774+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.775979+0000) 2019-10-13T09:46:54.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:54.774+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.776309+0000) 2019-10-13T09:46:54.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:54.774+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.776130+0000) 2019-10-13T09:46:54.778 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:54.775+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.776520+0000) 2019-10-13T09:46:54.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:54.777+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.779674+0000) 2019-10-13T09:46:54.785 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:54.781+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:46:55.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:55.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:55.589988+0000) 2019-10-13T09:46:55.591 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:55.589+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:55.589988+0000) 2019-10-13T09:46:56.806 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:46:56.806 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:56.811 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:46:56.811 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:58.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:58.499+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:58.500319+0000) 2019-10-13T09:46:58.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:58.838+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:58.838905+0000) 2019-10-13T09:46:59.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:46:59.784+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.785203+0000) 2019-10-13T09:46:59.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:46:59.783+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.783879+0000) 2019-10-13T09:46:59.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:46:59.783+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.784321+0000) 2019-10-13T09:46:59.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:46:59.783+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.784184+0000) 2019-10-13T09:46:59.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:46:59.784+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.784586+0000) 2019-10-13T09:46:59.787 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:46:59.784+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.784831+0000) 2019-10-13T09:46:59.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:46:59.784+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.785077+0000) 2019-10-13T09:46:59.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:46:59.787+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.787947+0000) 2019-10-13T09:46:59.791 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:46:59.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 172976 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:02.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:02.942+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:02.943558+0000) 2019-10-13T09:47:02.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:02.942+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:02.942762+0000) 2019-10-13T09:47:03.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:03.994+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:03.995056+0000) 2019-10-13T09:47:04.701 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:04.700+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.700471+0000) 2019-10-13T09:47:04.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:04.791+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.791479+0000) 2019-10-13T09:47:04.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:04.791+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.791619+0000) 2019-10-13T09:47:04.793 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:04.791+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.791982+0000) 2019-10-13T09:47:04.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:04.792+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.793582+0000) 2019-10-13T09:47:04.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:04.791+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.792346+0000) 2019-10-13T09:47:04.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:04.792+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.792570+0000) 2019-10-13T09:47:04.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:04.792+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.792648+0000) 2019-10-13T09:47:04.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:04.794+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.795101+0000) 2019-10-13T09:47:04.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:04.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:05.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:05.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:05.590194+0000) 2019-10-13T09:47:05.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:05.589+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:05.590194+0000) 2019-10-13T09:47:08.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:08.500+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:08.500530+0000) 2019-10-13T09:47:08.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:08.838+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:08.839069+0000) 2019-10-13T09:47:09.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:09.799+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.799796+0000) 2019-10-13T09:47:09.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:09.799+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.799603+0000) 2019-10-13T09:47:09.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:09.799+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.799928+0000) 2019-10-13T09:47:09.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:09.799+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.800072+0000) 2019-10-13T09:47:09.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:09.799+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.800344+0000) 2019-10-13T09:47:09.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:09.801+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.801790+0000) 2019-10-13T09:47:09.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:09.800+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.801120+0000) 2019-10-13T09:47:09.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:09.803+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.804025+0000) 2019-10-13T09:47:09.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:09.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173039 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:12.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:12.943+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:12.943705+0000) 2019-10-13T09:47:12.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:12.942+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:12.942898+0000) 2019-10-13T09:47:13.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:13.997+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:13.997432+0000) 2019-10-13T09:47:14.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:14.699+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.700659+0000) 2019-10-13T09:47:14.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:14.809+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.810479+0000) 2019-10-13T09:47:14.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:14.808+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.809693+0000) 2019-10-13T09:47:14.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:14.808+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.809879+0000) 2019-10-13T09:47:14.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:14.808+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.809645+0000) 2019-10-13T09:47:14.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:14.808+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.810268+0000) 2019-10-13T09:47:14.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:14.809+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.810586+0000) 2019-10-13T09:47:14.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:14.811+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.812287+0000) 2019-10-13T09:47:14.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:14.809+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.810841+0000) 2019-10-13T09:47:14.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:14.815+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:15.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:15.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:15.590356+0000) 2019-10-13T09:47:15.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:15.589+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:15.590356+0000) 2019-10-13T09:47:18.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:18.499+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:18.500706+0000) 2019-10-13T09:47:18.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:18.838+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:18.839241+0000) 2019-10-13T09:47:19.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:19.816+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.818165+0000) 2019-10-13T09:47:19.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:19.817+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.818592+0000) 2019-10-13T09:47:19.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:19.817+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.818796+0000) 2019-10-13T09:47:19.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:19.817+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.818909+0000) 2019-10-13T09:47:19.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:19.818+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.819504+0000) 2019-10-13T09:47:19.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:19.817+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.819380+0000) 2019-10-13T09:47:19.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:19.818+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.819603+0000) 2019-10-13T09:47:19.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:19.820+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.821725+0000) 2019-10-13T09:47:19.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:19.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173098 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:22.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:22.943+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:22.943911+0000) 2019-10-13T09:47:22.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:22.942+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:22.943096+0000) 2019-10-13T09:47:24.000 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:23.999+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:23.999994+0000) 2019-10-13T09:47:24.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:24.700+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.700805+0000) 2019-10-13T09:47:24.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:24.825+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.827271+0000) 2019-10-13T09:47:24.829 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:24.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.827467+0000) 2019-10-13T09:47:24.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:24.828+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.829376+0000) 2019-10-13T09:47:24.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:24.826+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.828333+0000) 2019-10-13T09:47:24.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:24.827+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.828486+0000) 2019-10-13T09:47:24.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:24.826+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.827847+0000) 2019-10-13T09:47:24.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:24.827+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.828833+0000) 2019-10-13T09:47:24.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:24.830+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.830816+0000) 2019-10-13T09:47:24.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:24.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173128 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:25.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:25.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:25.590515+0000) 2019-10-13T09:47:25.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:25.589+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:25.590514+0000) 2019-10-13T09:47:26.950 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:47:26.950 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:26.954 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:47:26.954 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:28.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:28.500+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:28.500872+0000) 2019-10-13T09:47:28.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:28.838+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:28.839394+0000) 2019-10-13T09:47:29.836 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:29.833+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.835324+0000) 2019-10-13T09:47:29.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:29.836+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.837512+0000) 2019-10-13T09:47:29.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:29.834+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.836208+0000) 2019-10-13T09:47:29.838 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:29.835+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.836480+0000) 2019-10-13T09:47:29.838 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:29.835+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.836643+0000) 2019-10-13T09:47:29.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:29.838+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.838916+0000) 2019-10-13T09:47:29.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:29.835+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.836946+0000) 2019-10-13T09:47:29.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:29.835+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.837335+0000) 2019-10-13T09:47:29.844 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:29.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173159 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:32.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:32.944+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:32.944095+0000) 2019-10-13T09:47:32.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:32.942+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:32.943261+0000) 2019-10-13T09:47:34.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:34.000+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.002255+0000) 2019-10-13T09:47:34.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:34.700+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.700918+0000) 2019-10-13T09:47:34.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:34.843+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.844722+0000) 2019-10-13T09:47:34.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:34.843+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.844841+0000) 2019-10-13T09:47:34.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:34.843+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.845105+0000) 2019-10-13T09:47:34.847 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:34.843+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.845230+0000) 2019-10-13T09:47:34.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:34.844+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.845527+0000) 2019-10-13T09:47:34.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:34.844+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.845584+0000) 2019-10-13T09:47:34.848 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:34.845+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.846894+0000) 2019-10-13T09:47:34.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:34.847+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.849113+0000) 2019-10-13T09:47:34.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:34.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173189 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:35.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:35.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:35.590689+0000) 2019-10-13T09:47:35.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:35.590+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:35.590695+0000) 2019-10-13T09:47:38.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:38.500+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:38.501043+0000) 2019-10-13T09:47:38.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:38.839+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:38.839567+0000) 2019-10-13T09:47:39.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:39.851+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.853286+0000) 2019-10-13T09:47:39.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:39.852+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.853476+0000) 2019-10-13T09:47:39.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:39.853+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.854908+0000) 2019-10-13T09:47:39.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:39.852+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.853934+0000) 2019-10-13T09:47:39.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:39.852+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.854365+0000) 2019-10-13T09:47:39.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:39.853+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.855173+0000) 2019-10-13T09:47:39.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:39.854+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.855419+0000) 2019-10-13T09:47:39.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:39.856+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.858398+0000) 2019-10-13T09:47:39.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:39.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173218 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:42.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:42.943+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:42.944314+0000) 2019-10-13T09:47:42.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:42.943+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:42.943432+0000) 2019-10-13T09:47:44.004 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:44.002+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.004528+0000) 2019-10-13T09:47:44.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:44.700+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.701141+0000) 2019-10-13T09:47:44.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:44.861+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.863434+0000) 2019-10-13T09:47:44.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:44.861+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.862322+0000) 2019-10-13T09:47:44.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:44.862+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.862469+0000) 2019-10-13T09:47:44.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:44.862+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.862655+0000) 2019-10-13T09:47:44.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:44.862+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.862909+0000) 2019-10-13T09:47:44.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:44.862+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.863105+0000) 2019-10-13T09:47:44.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:44.862+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.863297+0000) 2019-10-13T09:47:44.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:44.864+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.865884+0000) 2019-10-13T09:47:44.868 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:44.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173248 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:45.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:45.590+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:45.590840+0000) 2019-10-13T09:47:45.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:45.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:45.590866+0000) 2019-10-13T09:47:48.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:48.500+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:48.501235+0000) 2019-10-13T09:47:48.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:48.839+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:48.839756+0000) 2019-10-13T09:47:49.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:49.868+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.869074+0000) 2019-10-13T09:47:49.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:49.868+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.869165+0000) 2019-10-13T09:47:49.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:49.869+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.870549+0000) 2019-10-13T09:47:49.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:49.869+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.869894+0000) 2019-10-13T09:47:49.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:49.869+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.869627+0000) 2019-10-13T09:47:49.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:49.869+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.870201+0000) 2019-10-13T09:47:49.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:49.869+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.870320+0000) 2019-10-13T09:47:49.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:49.872+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.873058+0000) 2019-10-13T09:47:49.877 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:49.875+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173279 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:52.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:52.943+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:52.944447+0000) 2019-10-13T09:47:52.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:52.943+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:52.943586+0000) 2019-10-13T09:47:54.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:54.006+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.006791+0000) 2019-10-13T09:47:54.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:54.701+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.701297+0000) 2019-10-13T09:47:54.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:54.877+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.877786+0000) 2019-10-13T09:47:54.879 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:54.877+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.878193+0000) 2019-10-13T09:47:54.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:54.878+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.879124+0000) 2019-10-13T09:47:54.880 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:54.878+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.878393+0000) 2019-10-13T09:47:54.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:54.878+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.878757+0000) 2019-10-13T09:47:54.881 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:54.878+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.878927+0000) 2019-10-13T09:47:54.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:54.878+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.879114+0000) 2019-10-13T09:47:54.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:54.881+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.882301+0000) 2019-10-13T09:47:54.886 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:54.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:47:55.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:55.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:55.590996+0000) 2019-10-13T09:47:55.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:55.590+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:55.590993+0000) 2019-10-13T09:47:57.079 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:47:57.079 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:57.084 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:47:57.084 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:58.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:58.501+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:58.501470+0000) 2019-10-13T09:47:58.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:58.839+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:58.839926+0000) 2019-10-13T09:47:59.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:47:59.885+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.886625+0000) 2019-10-13T09:47:59.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:47:59.887+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.887954+0000) 2019-10-13T09:47:59.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:47:59.885+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.887145+0000) 2019-10-13T09:47:59.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:47:59.885+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.887293+0000) 2019-10-13T09:47:59.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:47:59.886+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.887673+0000) 2019-10-13T09:47:59.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:47:59.886+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.887899+0000) 2019-10-13T09:47:59.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:47:59.889+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.890256+0000) 2019-10-13T09:47:59.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:47:59.886+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.888176+0000) 2019-10-13T09:47:59.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:47:59.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173338 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:02.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:02.944+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:02.944616+0000) 2019-10-13T09:48:02.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:02.942+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:02.943782+0000) 2019-10-13T09:48:04.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:04.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.008804+0000) 2019-10-13T09:48:04.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:04.700+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.701493+0000) 2019-10-13T09:48:04.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:04.895+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.897064+0000) 2019-10-13T09:48:04.899 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:04.895+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.897234+0000) 2019-10-13T09:48:04.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:04.896+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.897613+0000) 2019-10-13T09:48:04.900 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:04.896+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.897724+0000) 2019-10-13T09:48:04.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:04.897+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.898659+0000) 2019-10-13T09:48:04.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:04.899+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.900334+0000) 2019-10-13T09:48:04.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:04.896+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.898129+0000) 2019-10-13T09:48:04.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:04.896+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.898320+0000) 2019-10-13T09:48:04.905 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:04.902+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:05.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:05.589+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:05.591197+0000) 2019-10-13T09:48:05.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:05.589+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:05.591178+0000) 2019-10-13T09:48:08.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:08.500+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:08.501648+0000) 2019-10-13T09:48:08.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:08.839+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:08.840113+0000) 2019-10-13T09:48:09.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:09.906+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.906943+0000) 2019-10-13T09:48:09.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:09.904+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.905909+0000) 2019-10-13T09:48:09.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:09.904+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.906102+0000) 2019-10-13T09:48:09.908 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:09.905+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.906461+0000) 2019-10-13T09:48:09.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:09.905+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.906778+0000) 2019-10-13T09:48:09.909 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:09.905+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.906961+0000) 2019-10-13T09:48:09.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:09.905+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.907230+0000) 2019-10-13T09:48:09.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:09.909+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.910659+0000) 2019-10-13T09:48:09.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:09.912+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:12.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:12.944+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:12.944831+0000) 2019-10-13T09:48:12.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:12.943+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:12.943963+0000) 2019-10-13T09:48:14.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:14.010+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.011093+0000) 2019-10-13T09:48:14.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:14.700+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.701677+0000) 2019-10-13T09:48:14.916 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:14.913+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.915133+0000) 2019-10-13T09:48:14.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:14.914+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.915424+0000) 2019-10-13T09:48:14.917 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:14.913+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.915348+0000) 2019-10-13T09:48:14.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:14.914+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.915670+0000) 2019-10-13T09:48:14.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:14.916+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.917306+0000) 2019-10-13T09:48:14.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:14.914+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.915927+0000) 2019-10-13T09:48:14.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:14.915+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.916423+0000) 2019-10-13T09:48:14.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:14.919+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.919935+0000) 2019-10-13T09:48:14.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:14.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:15.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:15.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:15.591354+0000) 2019-10-13T09:48:15.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:15.590+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:15.591348+0000) 2019-10-13T09:48:18.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:18.501+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:18.501822+0000) 2019-10-13T09:48:18.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:18.839+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:18.840271+0000) 2019-10-13T09:48:19.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:19.922+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.924251+0000) 2019-10-13T09:48:19.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:19.923+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.924709+0000) 2019-10-13T09:48:19.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:19.925+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.925743+0000) 2019-10-13T09:48:19.927 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:19.923+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.924929+0000) 2019-10-13T09:48:19.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:19.923+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.925122+0000) 2019-10-13T09:48:19.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:19.923+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.925301+0000) 2019-10-13T09:48:19.927 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:19.924+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.925603+0000) 2019-10-13T09:48:19.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:19.928+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.929143+0000) 2019-10-13T09:48:19.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:19.929+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:22.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:22.944+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:22.945037+0000) 2019-10-13T09:48:22.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:22.943+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:22.944070+0000) 2019-10-13T09:48:24.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:24.012+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.013438+0000) 2019-10-13T09:48:24.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:24.701+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.701848+0000) 2019-10-13T09:48:24.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:24.933+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.933975+0000) 2019-10-13T09:48:24.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:24.931+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.933003+0000) 2019-10-13T09:48:24.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:24.931+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.933177+0000) 2019-10-13T09:48:24.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:24.931+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.933208+0000) 2019-10-13T09:48:24.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:24.931+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.933276+0000) 2019-10-13T09:48:24.935 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:24.932+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.933430+0000) 2019-10-13T09:48:24.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:24.932+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.933708+0000) 2019-10-13T09:48:24.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:24.936+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.936872+0000) 2019-10-13T09:48:24.939 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:24.936+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173490 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:25.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:25.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:25.591516+0000) 2019-10-13T09:48:25.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:25.591+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:25.591516+0000) 2019-10-13T09:48:27.242 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:48:27.243 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:27.247 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:48:27.247 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:28.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:28.501+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:28.501991+0000) 2019-10-13T09:48:28.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:28.840+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:28.840440+0000) 2019-10-13T09:48:29.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:29.939+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.939872+0000) 2019-10-13T09:48:29.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:29.939+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.939963+0000) 2019-10-13T09:48:29.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:29.939+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.940073+0000) 2019-10-13T09:48:29.942 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:29.939+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.940387+0000) 2019-10-13T09:48:29.942 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:29.940+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.941826+0000) 2019-10-13T09:48:29.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:29.940+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.940739+0000) 2019-10-13T09:48:29.943 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:29.940+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.940895+0000) 2019-10-13T09:48:29.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:29.942+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.944008+0000) 2019-10-13T09:48:29.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:29.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:32.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:32.944+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:32.945219+0000) 2019-10-13T09:48:32.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:32.943+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:32.944262+0000) 2019-10-13T09:48:34.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:34.014+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.015865+0000) 2019-10-13T09:48:34.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:34.701+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.702036+0000) 2019-10-13T09:48:34.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:34.947+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.947994+0000) 2019-10-13T09:48:34.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:34.947+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.948339+0000) 2019-10-13T09:48:34.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:34.947+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.949650+0000) 2019-10-13T09:48:34.951 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:34.948+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.948735+0000) 2019-10-13T09:48:34.952 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:34.948+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.949066+0000) 2019-10-13T09:48:34.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:34.948+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.949231+0000) 2019-10-13T09:48:34.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:34.948+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.949322+0000) 2019-10-13T09:48:34.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:34.951+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.953087+0000) 2019-10-13T09:48:34.958 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:34.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:35.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:35.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:35.591683+0000) 2019-10-13T09:48:35.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:35.591+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:35.591688+0000) 2019-10-13T09:48:38.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:38.502+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:38.502163+0000) 2019-10-13T09:48:38.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:38.840+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:38.840619+0000) 2019-10-13T09:48:39.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:39.958+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.958506+0000) 2019-10-13T09:48:39.960 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:39.958+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.958758+0000) 2019-10-13T09:48:39.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:39.958+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.958980+0000) 2019-10-13T09:48:39.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:39.959+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.960484+0000) 2019-10-13T09:48:39.961 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:39.959+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.959439+0000) 2019-10-13T09:48:39.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:39.959+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.959578+0000) 2019-10-13T09:48:39.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:39.959+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.959921+0000) 2019-10-13T09:48:39.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:39.961+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.962501+0000) 2019-10-13T09:48:39.967 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:39.965+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:42.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:42.944+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:42.945342+0000) 2019-10-13T09:48:42.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:42.944+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:42.944436+0000) 2019-10-13T09:48:44.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:44.017+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.018194+0000) 2019-10-13T09:48:44.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:44.700+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.702189+0000) 2019-10-13T09:48:44.969 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:44.966+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.967772+0000) 2019-10-13T09:48:44.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:44.966+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.967965+0000) 2019-10-13T09:48:44.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:44.969+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.969925+0000) 2019-10-13T09:48:44.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:44.966+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.968277+0000) 2019-10-13T09:48:44.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:44.966+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.968228+0000) 2019-10-13T09:48:44.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:44.967+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.968826+0000) 2019-10-13T09:48:44.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:44.967+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.968973+0000) 2019-10-13T09:48:44.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:44.971+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.971735+0000) 2019-10-13T09:48:44.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:44.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:45.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:45.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:45.591872+0000) 2019-10-13T09:48:45.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:45.590+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:45.591875+0000) 2019-10-13T09:48:48.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:48.501+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:48.502341+0000) 2019-10-13T09:48:48.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:48.839+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:48.840769+0000) 2019-10-13T09:48:49.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:49.974+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.975849+0000) 2019-10-13T09:48:49.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:49.974+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.975907+0000) 2019-10-13T09:48:49.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:49.974+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.976084+0000) 2019-10-13T09:48:49.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:49.974+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.976248+0000) 2019-10-13T09:48:49.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:49.976+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.977619+0000) 2019-10-13T09:48:49.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:49.975+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.976548+0000) 2019-10-13T09:48:49.979 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:49.975+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.977011+0000) 2019-10-13T09:48:49.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:49.979+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.979985+0000) 2019-10-13T09:48:49.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:49.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:52.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:52.945+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:52.945506+0000) 2019-10-13T09:48:52.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:52.943+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:52.944563+0000) 2019-10-13T09:48:54.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:54.019+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.020686+0000) 2019-10-13T09:48:54.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:54.701+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.702344+0000) 2019-10-13T09:48:54.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:54.985+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.986462+0000) 2019-10-13T09:48:54.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:54.983+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.984748+0000) 2019-10-13T09:48:54.989 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:54.983+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.985065+0000) 2019-10-13T09:48:54.989 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:54.984+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.985450+0000) 2019-10-13T09:48:54.989 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:54.984+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.985676+0000) 2019-10-13T09:48:54.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:54.984+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.985861+0000) 2019-10-13T09:48:54.990 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:54.984+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.986095+0000) 2019-10-13T09:48:54.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:54.988+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.988770+0000) 2019-10-13T09:48:54.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:48:54.991+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:48:55.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:55.590+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:55.592018+0000) 2019-10-13T09:48:55.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:55.590+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:55.592049+0000) 2019-10-13T09:48:57.406 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:48:57.406 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:57.412 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:48:57.412 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:58.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:58.501+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:58.502581+0000) 2019-10-13T09:48:58.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:58.840+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:58.840929+0000) 2019-10-13T09:48:59.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:48:59.993+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.994872+0000) 2019-10-13T09:48:59.996 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:48:59.993+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.994984+0000) 2019-10-13T09:48:59.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:48:59.994+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.995423+0000) 2019-10-13T09:48:59.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:48:59.995+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.996327+0000) 2019-10-13T09:48:59.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:48:59.994+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.995817+0000) 2019-10-13T09:48:59.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:48:59.994+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.995972+0000) 2019-10-13T09:48:59.998 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:48:59.994+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.996240+0000) 2019-10-13T09:48:59.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:48:59.997+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.998568+0000) 2019-10-13T09:49:00.003 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:00.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:02.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:02.945+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:02.945690+0000) 2019-10-13T09:49:02.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:02.944+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:02.944762+0000) 2019-10-13T09:49:04.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:04.022+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.022996+0000) 2019-10-13T09:49:04.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:04.701+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.702520+0000) 2019-10-13T09:49:05.005 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:05.002+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.003687+0000) 2019-10-13T09:49:05.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:05.002+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.003836+0000) 2019-10-13T09:49:05.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:05.005+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.005800+0000) 2019-10-13T09:49:05.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:05.002+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.004211+0000) 2019-10-13T09:49:05.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:05.002+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.004262+0000) 2019-10-13T09:49:05.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:05.003+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.004426+0000) 2019-10-13T09:49:05.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:05.003+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.004772+0000) 2019-10-13T09:49:05.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:05.007+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.008228+0000) 2019-10-13T09:49:05.012 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:05.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173734 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:05.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:05.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.592169+0000) 2019-10-13T09:49:05.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:05.591+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:05.592219+0000) 2019-10-13T09:49:08.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:08.502+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:08.502778+0000) 2019-10-13T09:49:08.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:08.840+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:08.841085+0000) 2019-10-13T09:49:10.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:10.012+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.012772+0000) 2019-10-13T09:49:10.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:10.010+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.011786+0000) 2019-10-13T09:49:10.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:10.010+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.011975+0000) 2019-10-13T09:49:10.014 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:10.010+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.012326+0000) 2019-10-13T09:49:10.014 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:10.010+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.012244+0000) 2019-10-13T09:49:10.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:10.011+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.012576+0000) 2019-10-13T09:49:10.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:10.011+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.012741+0000) 2019-10-13T09:49:10.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:10.015+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:10.015762+0000) 2019-10-13T09:49:10.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:10.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:12.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:12.945+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:12.945885+0000) 2019-10-13T09:49:12.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:12.944+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:12.944951+0000) 2019-10-13T09:49:14.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:14.025+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.025255+0000) 2019-10-13T09:49:14.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:14.702+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.702677+0000) 2019-10-13T09:49:15.021 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:15.019+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.019670+0000) 2019-10-13T09:49:15.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:15.019+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.019864+0000) 2019-10-13T09:49:15.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:15.019+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.019999+0000) 2019-10-13T09:49:15.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:15.019+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.020290+0000) 2019-10-13T09:49:15.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:15.019+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.020322+0000) 2019-10-13T09:49:15.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:15.020+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.021691+0000) 2019-10-13T09:49:15.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:15.020+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.021043+0000) 2019-10-13T09:49:15.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:15.023+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.024225+0000) 2019-10-13T09:49:15.027 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:15.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173796 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:15.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:15.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.592361+0000) 2019-10-13T09:49:15.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:15.591+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:15.592362+0000) 2019-10-13T09:49:18.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:18.502+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:18.502976+0000) 2019-10-13T09:49:18.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:18.840+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:18.841257+0000) 2019-10-13T09:49:20.029 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:20.028+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.028463+0000) 2019-10-13T09:49:20.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:20.028+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.028607+0000) 2019-10-13T09:49:20.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:20.028+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.030272+0000) 2019-10-13T09:49:20.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:20.028+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.028937+0000) 2019-10-13T09:49:20.031 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:20.028+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.029263+0000) 2019-10-13T09:49:20.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:20.028+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.028489+0000) 2019-10-13T09:49:20.031 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:20.029+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.029415+0000) 2019-10-13T09:49:20.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:20.031+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:20.032945+0000) 2019-10-13T09:49:20.036 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:20.034+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:22.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:22.945+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:22.946066+0000) 2019-10-13T09:49:22.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:22.944+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:22.945160+0000) 2019-10-13T09:49:24.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:24.026+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.027747+0000) 2019-10-13T09:49:24.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:24.702+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.702823+0000) 2019-10-13T09:49:25.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:25.035+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.037431+0000) 2019-10-13T09:49:25.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:25.036+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.036506+0000) 2019-10-13T09:49:25.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:25.036+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.036899+0000) 2019-10-13T09:49:25.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:25.036+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.037039+0000) 2019-10-13T09:49:25.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:25.036+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.037350+0000) 2019-10-13T09:49:25.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:25.037+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.037858+0000) 2019-10-13T09:49:25.039 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:25.037+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.038004+0000) 2019-10-13T09:49:25.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:25.039+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.041075+0000) 2019-10-13T09:49:25.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:25.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:25.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:25.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.592564+0000) 2019-10-13T09:49:25.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:25.592+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:25.592558+0000) 2019-10-13T09:49:27.552 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:49:27.552 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:27.556 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:49:27.556 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:28.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:28.503+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:28.503143+0000) 2019-10-13T09:49:28.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:28.841+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:28.841449+0000) 2019-10-13T09:49:30.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:30.043+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.043929+0000) 2019-10-13T09:49:30.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:30.045+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.045749+0000) 2019-10-13T09:49:30.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:30.044+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.044416+0000) 2019-10-13T09:49:30.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:30.044+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.044833+0000) 2019-10-13T09:49:30.047 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:30.044+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.044855+0000) 2019-10-13T09:49:30.047 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:30.044+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.044949+0000) 2019-10-13T09:49:30.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:30.044+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.045336+0000) 2019-10-13T09:49:30.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:30.046+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:30.047600+0000) 2019-10-13T09:49:30.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:30.050+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:32.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:32.945+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:32.946267+0000) 2019-10-13T09:49:32.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:32.944+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:32.945318+0000) 2019-10-13T09:49:34.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:34.029+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.030102+0000) 2019-10-13T09:49:34.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:34.702+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.703008+0000) 2019-10-13T09:49:35.054 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:35.050+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.052241+0000) 2019-10-13T09:49:35.054 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:35.051+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.052422+0000) 2019-10-13T09:49:35.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:35.051+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.052583+0000) 2019-10-13T09:49:35.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:35.053+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.053846+0000) 2019-10-13T09:49:35.055 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:35.051+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.053189+0000) 2019-10-13T09:49:35.055 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:35.051+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.053278+0000) 2019-10-13T09:49:35.056 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:35.052+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.053546+0000) 2019-10-13T09:49:35.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:35.055+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.056189+0000) 2019-10-13T09:49:35.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:35.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:35.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:35.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.592725+0000) 2019-10-13T09:49:35.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:35.591+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:35.592722+0000) 2019-10-13T09:49:38.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:38.502+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:38.503260+0000) 2019-10-13T09:49:38.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:38.840+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:38.841625+0000) 2019-10-13T09:49:40.062 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:40.059+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.060937+0000) 2019-10-13T09:49:40.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:40.059+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.061163+0000) 2019-10-13T09:49:40.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:40.060+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.061412+0000) 2019-10-13T09:49:40.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:40.062+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.063100+0000) 2019-10-13T09:49:40.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:40.060+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.061667+0000) 2019-10-13T09:49:40.064 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:40.060+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.062113+0000) 2019-10-13T09:49:40.064 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:40.060+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.062180+0000) 2019-10-13T09:49:40.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:40.064+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:40.064770+0000) 2019-10-13T09:49:40.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:40.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:42.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:42.945+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:42.946468+0000) 2019-10-13T09:49:42.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:42.944+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:42.945521+0000) 2019-10-13T09:49:44.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:44.031+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.032521+0000) 2019-10-13T09:49:44.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:44.701+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.703238+0000) 2019-10-13T09:49:45.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:45.068+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.069903+0000) 2019-10-13T09:49:45.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:45.068+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.070209+0000) 2019-10-13T09:49:45.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:45.070+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.071174+0000) 2019-10-13T09:49:45.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:45.069+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.070892+0000) 2019-10-13T09:49:45.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:45.069+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.071054+0000) 2019-10-13T09:49:45.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:45.069+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.071130+0000) 2019-10-13T09:49:45.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:45.069+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.071242+0000) 2019-10-13T09:49:45.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:45.074+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.074788+0000) 2019-10-13T09:49:45.076 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:45.074+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 173980 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:45.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:45.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.592891+0000) 2019-10-13T09:49:45.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:45.592+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:45.592890+0000) 2019-10-13T09:49:48.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:48.502+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:48.503430+0000) 2019-10-13T09:49:48.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:48.841+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:48.841809+0000) 2019-10-13T09:49:50.079 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:50.075+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.077111+0000) 2019-10-13T09:49:50.079 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:50.075+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.077237+0000) 2019-10-13T09:49:50.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:50.076+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.077592+0000) 2019-10-13T09:49:50.080 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:50.076+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.077795+0000) 2019-10-13T09:49:50.080 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:50.076+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.078103+0000) 2019-10-13T09:49:50.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:50.076+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.078315+0000) 2019-10-13T09:49:50.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:50.077+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.078601+0000) 2019-10-13T09:49:50.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:50.080+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:50.081326+0000) 2019-10-13T09:49:50.084 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:50.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:52.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:52.945+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:52.946619+0000) 2019-10-13T09:49:52.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:52.945+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:52.945658+0000) 2019-10-13T09:49:54.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:54.034+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.034785+0000) 2019-10-13T09:49:54.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:54.702+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.703433+0000) 2019-10-13T09:49:55.086 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:55.083+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.085338+0000) 2019-10-13T09:49:55.087 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:49:55.084+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.085426+0000) 2019-10-13T09:49:55.087 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:55.084+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.085535+0000) 2019-10-13T09:49:55.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:49:55.086+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.087090+0000) 2019-10-13T09:49:55.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:55.084+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.086073+0000) 2019-10-13T09:49:55.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:49:55.084+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.086079+0000) 2019-10-13T09:49:55.088 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:55.084+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.086217+0000) 2019-10-13T09:49:55.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:49:55.089+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.089996+0000) 2019-10-13T09:49:55.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:49:55.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174040 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:49:55.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:49:55.591+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.593079+0000) 2019-10-13T09:49:55.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:49:55.591+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:55.593078+0000) 2019-10-13T09:49:57.643 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:49:57.643 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:57.647 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:49:57.648 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:58.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:49:58.503+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:58.503622+0000) 2019-10-13T09:49:58.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:49:58.841+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:58.842012+0000) 2019-10-13T09:50:00.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:00.094+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.095155+0000) 2019-10-13T09:50:00.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:00.093+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.094303+0000) 2019-10-13T09:50:00.096 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:00.094+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.094406+0000) 2019-10-13T09:50:00.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:00.094+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.094700+0000) 2019-10-13T09:50:00.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:00.094+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.095048+0000) 2019-10-13T09:50:00.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:00.094+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.095189+0000) 2019-10-13T09:50:00.097 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:00.095+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.095581+0000) 2019-10-13T09:50:00.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:00.097+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:00.097904+0000) 2019-10-13T09:50:00.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:00.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:02.946 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:02.946+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:02.946797+0000) 2019-10-13T09:50:02.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:02.945+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:02.945835+0000) 2019-10-13T09:50:04.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:04.037+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.037203+0000) 2019-10-13T09:50:04.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:04.703+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.703634+0000) 2019-10-13T09:50:05.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:05.103+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.104631+0000) 2019-10-13T09:50:05.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:05.102+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.103339+0000) 2019-10-13T09:50:05.105 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:05.103+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.103658+0000) 2019-10-13T09:50:05.106 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:05.103+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.104041+0000) 2019-10-13T09:50:05.106 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:05.103+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.104212+0000) 2019-10-13T09:50:05.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:05.103+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.104310+0000) 2019-10-13T09:50:05.106 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:05.104+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.104925+0000) 2019-10-13T09:50:05.107 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:05.107+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.107864+0000) 2019-10-13T09:50:05.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:05.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174102 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:05.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:05.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.593277+0000) 2019-10-13T09:50:05.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:05.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:05.593279+0000) 2019-10-13T09:50:08.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:08.503+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:08.503821+0000) 2019-10-13T09:50:08.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:08.841+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:08.842139+0000) 2019-10-13T09:50:10.114 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:10.112+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.112696+0000) 2019-10-13T09:50:10.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:10.112+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.112885+0000) 2019-10-13T09:50:10.114 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:10.112+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.113037+0000) 2019-10-13T09:50:10.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:10.112+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.114448+0000) 2019-10-13T09:50:10.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:10.113+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.114162+0000) 2019-10-13T09:50:10.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:10.113+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.114324+0000) 2019-10-13T09:50:10.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:10.114+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.114501+0000) 2019-10-13T09:50:10.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:10.115+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:10.117654+0000) 2019-10-13T09:50:10.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:10.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:12.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:12.946+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:12.946996+0000) 2019-10-13T09:50:12.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:12.945+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:12.946000+0000) 2019-10-13T09:50:14.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:14.038+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.039468+0000) 2019-10-13T09:50:14.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:14.703+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.703824+0000) 2019-10-13T09:50:15.123 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:15.121+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.121534+0000) 2019-10-13T09:50:15.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:15.121+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.121830+0000) 2019-10-13T09:50:15.124 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:15.121+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.122184+0000) 2019-10-13T09:50:15.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:15.121+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.122373+0000) 2019-10-13T09:50:15.125 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:15.122+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.123764+0000) 2019-10-13T09:50:15.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:15.122+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.122899+0000) 2019-10-13T09:50:15.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:15.122+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.123244+0000) 2019-10-13T09:50:15.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:15.124+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.126302+0000) 2019-10-13T09:50:15.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:15.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174162 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:15.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:15.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.593470+0000) 2019-10-13T09:50:15.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:15.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:15.593464+0000) 2019-10-13T09:50:18.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:18.503+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:18.504001+0000) 2019-10-13T09:50:18.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:18.841+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:18.842292+0000) 2019-10-13T09:50:20.132 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:20.129+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.130645+0000) 2019-10-13T09:50:20.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:20.130+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.132601+0000) 2019-10-13T09:50:20.133 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:20.129+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.131126+0000) 2019-10-13T09:50:20.133 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:20.129+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.131249+0000) 2019-10-13T09:50:20.133 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:20.130+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.131571+0000) 2019-10-13T09:50:20.133 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:20.130+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.131735+0000) 2019-10-13T09:50:20.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:20.130+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.131852+0000) 2019-10-13T09:50:20.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:20.132+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:20.134418+0000) 2019-10-13T09:50:20.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:20.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174191 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:22.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:22.946+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:22.947143+0000) 2019-10-13T09:50:22.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:22.945+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:22.946175+0000) 2019-10-13T09:50:24.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:24.041+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.041753+0000) 2019-10-13T09:50:24.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:24.703+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.704019+0000) 2019-10-13T09:50:25.141 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:25.138+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.139754+0000) 2019-10-13T09:50:25.141 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:25.138+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.140083+0000) 2019-10-13T09:50:25.142 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:25.140+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.141064+0000) 2019-10-13T09:50:25.142 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:25.138+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.140209+0000) 2019-10-13T09:50:25.142 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:25.139+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.140492+0000) 2019-10-13T09:50:25.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:25.139+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.140923+0000) 2019-10-13T09:50:25.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:25.139+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.141048+0000) 2019-10-13T09:50:25.143 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:25.142+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.143625+0000) 2019-10-13T09:50:25.149 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:25.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:25.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:25.592+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.593591+0000) 2019-10-13T09:50:25.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:25.592+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:25.593608+0000) 2019-10-13T09:50:27.766 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:50:27.766 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:27.770 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:50:27.770 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:28.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:28.502+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:28.504189+0000) 2019-10-13T09:50:28.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:28.841+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:28.842473+0000) 2019-10-13T09:50:30.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:30.150+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.151070+0000) 2019-10-13T09:50:30.152 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:30.148+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.150294+0000) 2019-10-13T09:50:30.152 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:30.149+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.150406+0000) 2019-10-13T09:50:30.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:30.149+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.150735+0000) 2019-10-13T09:50:30.152 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:30.149+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.150845+0000) 2019-10-13T09:50:30.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:30.149+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.151086+0000) 2019-10-13T09:50:30.153 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:30.152+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.153424+0000) 2019-10-13T09:50:30.153 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:30.149+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:30.151340+0000) 2019-10-13T09:50:30.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:30.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:32.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:32.946+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:32.947311+0000) 2019-10-13T09:50:32.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:32.945+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:32.946327+0000) 2019-10-13T09:50:34.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:34.043+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.044006+0000) 2019-10-13T09:50:34.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:34.703+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.704236+0000) 2019-10-13T09:50:35.160 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:35.157+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.159143+0000) 2019-10-13T09:50:35.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:35.158+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.159442+0000) 2019-10-13T09:50:35.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:35.158+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.159654+0000) 2019-10-13T09:50:35.162 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:35.159+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.160496+0000) 2019-10-13T09:50:35.162 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:35.158+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.159984+0000) 2019-10-13T09:50:35.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:35.158+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.160160+0000) 2019-10-13T09:50:35.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:35.159+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.160775+0000) 2019-10-13T09:50:35.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:35.163+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.163873+0000) 2019-10-13T09:50:35.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:35.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:35.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:35.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.593763+0000) 2019-10-13T09:50:35.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:35.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:35.593781+0000) 2019-10-13T09:50:38.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:38.503+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:38.504383+0000) 2019-10-13T09:50:38.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:38.842+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:38.842669+0000) 2019-10-13T09:50:40.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:40.167+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.168914+0000) 2019-10-13T09:50:40.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:40.167+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.169026+0000) 2019-10-13T09:50:40.171 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:40.167+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.169243+0000) 2019-10-13T09:50:40.171 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:40.167+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.169365+0000) 2019-10-13T09:50:40.171 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:40.168+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.169684+0000) 2019-10-13T09:50:40.172 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:40.170+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.170760+0000) 2019-10-13T09:50:40.172 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:40.171+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.171857+0000) 2019-10-13T09:50:40.172 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:40.168+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:40.169991+0000) 2019-10-13T09:50:40.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:40.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174313 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:42.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:42.947+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:42.947494+0000) 2019-10-13T09:50:42.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:42.946+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:42.946488+0000) 2019-10-13T09:50:44.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:44.046+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.046483+0000) 2019-10-13T09:50:44.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:44.704+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.704463+0000) 2019-10-13T09:50:45.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:45.177+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.178190+0000) 2019-10-13T09:50:45.180 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:45.178+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.178773+0000) 2019-10-13T09:50:45.180 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:45.178+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.179051+0000) 2019-10-13T09:50:45.180 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:45.178+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.179261+0000) 2019-10-13T09:50:45.181 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:45.179+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.180575+0000) 2019-10-13T09:50:45.181 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:45.179+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.179554+0000) 2019-10-13T09:50:45.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:45.179+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.179595+0000) 2019-10-13T09:50:45.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:45.180+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.181613+0000) 2019-10-13T09:50:45.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:45.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174348 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:45.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:45.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.593956+0000) 2019-10-13T09:50:45.595 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:45.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:45.593961+0000) 2019-10-13T09:50:48.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:48.504+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:48.504559+0000) 2019-10-13T09:50:48.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:48.842+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:48.842883+0000) 2019-10-13T09:50:50.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:50.186+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.187039+0000) 2019-10-13T09:50:50.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:50.186+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.187125+0000) 2019-10-13T09:50:50.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:50.187+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.187528+0000) 2019-10-13T09:50:50.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:50.188+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.189212+0000) 2019-10-13T09:50:50.190 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:50.187+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.187794+0000) 2019-10-13T09:50:50.191 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:50.187+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.188182+0000) 2019-10-13T09:50:50.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:50.188+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.188539+0000) 2019-10-13T09:50:50.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:50.190+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:50.190995+0000) 2019-10-13T09:50:50.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:50.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174375 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:52.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:52.946+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:52.946658+0000) 2019-10-13T09:50:52.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:52.947+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:52.947657+0000) 2019-10-13T09:50:54.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:54.048+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.048984+0000) 2019-10-13T09:50:54.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:54.704+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.704653+0000) 2019-10-13T09:50:55.197 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:55.195+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.196025+0000) 2019-10-13T09:50:55.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:55.195+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.196212+0000) 2019-10-13T09:50:55.198 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:50:55.196+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.196497+0000) 2019-10-13T09:50:55.198 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:50:55.196+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.197496+0000) 2019-10-13T09:50:55.199 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:55.196+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.196592+0000) 2019-10-13T09:50:55.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:55.196+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.196975+0000) 2019-10-13T09:50:55.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:50:55.196+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.197254+0000) 2019-10-13T09:50:55.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:50:55.200+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.200782+0000) 2019-10-13T09:50:55.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:50:55.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:50:55.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:50:55.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.594125+0000) 2019-10-13T09:50:55.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:50:55.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:55.594147+0000) 2019-10-13T09:50:57.855 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:50:57.855 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:57.859 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:50:57.859 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:58.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:50:58.504+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:58.504732+0000) 2019-10-13T09:50:58.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:50:58.842+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:58.843085+0000) 2019-10-13T09:51:00.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:00.204+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.205116+0000) 2019-10-13T09:51:00.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:00.205+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.205530+0000) 2019-10-13T09:51:00.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:00.205+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.205620+0000) 2019-10-13T09:51:00.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:00.205+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.205876+0000) 2019-10-13T09:51:00.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:00.205+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.206087+0000) 2019-10-13T09:51:00.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:00.205+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.207636+0000) 2019-10-13T09:51:00.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:00.205+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.206295+0000) 2019-10-13T09:51:00.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:00.207+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:00.208755+0000) 2019-10-13T09:51:00.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:00.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:02.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:02.946+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:02.947825+0000) 2019-10-13T09:51:02.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:02.945+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:02.946833+0000) 2019-10-13T09:51:04.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:04.049+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.051496+0000) 2019-10-13T09:51:04.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:04.703+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.704840+0000) 2019-10-13T09:51:05.215 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:05.212+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.213730+0000) 2019-10-13T09:51:05.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:05.212+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.213963+0000) 2019-10-13T09:51:05.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:05.213+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.215001+0000) 2019-10-13T09:51:05.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:05.212+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.214388+0000) 2019-10-13T09:51:05.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:05.213+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.214540+0000) 2019-10-13T09:51:05.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:05.213+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.214622+0000) 2019-10-13T09:51:05.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:05.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.214745+0000) 2019-10-13T09:51:05.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:05.215+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.217475+0000) 2019-10-13T09:51:05.220 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:05.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174470 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:05.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:05.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.594279+0000) 2019-10-13T09:51:05.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:05.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:05.594315+0000) 2019-10-13T09:51:08.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:08.503+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:08.504884+0000) 2019-10-13T09:51:08.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:08.842+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:08.843281+0000) 2019-10-13T09:51:10.222 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:10.220+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.222345+0000) 2019-10-13T09:51:10.223 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:10.220+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.221614+0000) 2019-10-13T09:51:10.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:10.220+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.221781+0000) 2019-10-13T09:51:10.223 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:10.220+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.221962+0000) 2019-10-13T09:51:10.224 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:10.221+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.222410+0000) 2019-10-13T09:51:10.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:10.220+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.222362+0000) 2019-10-13T09:51:10.224 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:10.221+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.222723+0000) 2019-10-13T09:51:10.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:10.224+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:10.226130+0000) 2019-10-13T09:51:10.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:10.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:12.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:12.947+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:12.948016+0000) 2019-10-13T09:51:12.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:12.946+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:12.947014+0000) 2019-10-13T09:51:14.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:14.053+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.053972+0000) 2019-10-13T09:51:14.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:14.704+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.705024+0000) 2019-10-13T09:51:15.231 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:15.229+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.230532+0000) 2019-10-13T09:51:15.233 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:15.231+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.232259+0000) 2019-10-13T09:51:15.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:15.229+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.231331+0000) 2019-10-13T09:51:15.233 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:15.229+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.230874+0000) 2019-10-13T09:51:15.233 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:15.230+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.231566+0000) 2019-10-13T09:51:15.233 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:15.230+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.231693+0000) 2019-10-13T09:51:15.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:15.230+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.232036+0000) 2019-10-13T09:51:15.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:15.232+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.233604+0000) 2019-10-13T09:51:15.239 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:15.236+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:15.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:15.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.594434+0000) 2019-10-13T09:51:15.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:15.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:15.594422+0000) 2019-10-13T09:51:18.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:18.504+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:18.505028+0000) 2019-10-13T09:51:18.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:18.842+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:18.843466+0000) 2019-10-13T09:51:20.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:20.238+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.239629+0000) 2019-10-13T09:51:20.241 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:20.240+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.241331+0000) 2019-10-13T09:51:20.242 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:20.238+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.239977+0000) 2019-10-13T09:51:20.242 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:20.239+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.240621+0000) 2019-10-13T09:51:20.242 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:20.239+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.240762+0000) 2019-10-13T09:51:20.243 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:20.239+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.240958+0000) 2019-10-13T09:51:20.243 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:20.240+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.241525+0000) 2019-10-13T09:51:20.243 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:20.242+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:20.243080+0000) 2019-10-13T09:51:20.247 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:20.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174557 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:22.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:22.947+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:22.948200+0000) 2019-10-13T09:51:22.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:22.946+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:22.947208+0000) 2019-10-13T09:51:24.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:24.055+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.056343+0000) 2019-10-13T09:51:24.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:24.704+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.705209+0000) 2019-10-13T09:51:25.250 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:25.247+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.248737+0000) 2019-10-13T09:51:25.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:25.247+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.249076+0000) 2019-10-13T09:51:25.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:25.247+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.249181+0000) 2019-10-13T09:51:25.251 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:25.249+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.250081+0000) 2019-10-13T09:51:25.251 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:25.248+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.249445+0000) 2019-10-13T09:51:25.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:25.248+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.250084+0000) 2019-10-13T09:51:25.252 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:25.248+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.250199+0000) 2019-10-13T09:51:25.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:25.252+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.253042+0000) 2019-10-13T09:51:25.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:25.254+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174592 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:25.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:25.593+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.594583+0000) 2019-10-13T09:51:25.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:25.593+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:25.594577+0000) 2019-10-13T09:51:27.935 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:51:27.935 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:27.939 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:51:27.939 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:28.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:28.504+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:28.505193+0000) 2019-10-13T09:51:28.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:28.843+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:28.843628+0000) 2019-10-13T09:51:30.259 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:30.257+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.258261+0000) 2019-10-13T09:51:30.260 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:30.258+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.259657+0000) 2019-10-13T09:51:30.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:30.258+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.258900+0000) 2019-10-13T09:51:30.261 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:30.259+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.259438+0000) 2019-10-13T09:51:30.261 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:30.259+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.259636+0000) 2019-10-13T09:51:30.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:30.259+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.259907+0000) 2019-10-13T09:51:30.261 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:30.259+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.260202+0000) 2019-10-13T09:51:30.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:30.261+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:30.261961+0000) 2019-10-13T09:51:30.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:30.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174619 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:32.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:32.948+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:32.948394+0000) 2019-10-13T09:51:32.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:32.947+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:32.947385+0000) 2019-10-13T09:51:34.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:34.058+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.058794+0000) 2019-10-13T09:51:34.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:34.705+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.705407+0000) 2019-10-13T09:51:35.268 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:35.267+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.268193+0000) 2019-10-13T09:51:35.269 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:35.266+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.267246+0000) 2019-10-13T09:51:35.269 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:35.267+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.267523+0000) 2019-10-13T09:51:35.270 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:35.267+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.267941+0000) 2019-10-13T09:51:35.270 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:35.267+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.268236+0000) 2019-10-13T09:51:35.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:35.267+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.268321+0000) 2019-10-13T09:51:35.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:35.268+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.268604+0000) 2019-10-13T09:51:35.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:35.270+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.271192+0000) 2019-10-13T09:51:35.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:35.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174652 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:35.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:35.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.594760+0000) 2019-10-13T09:51:35.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:35.594+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:35.594757+0000) 2019-10-13T09:51:38.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:38.505+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:38.505367+0000) 2019-10-13T09:51:38.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:38.843+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:38.843820+0000) 2019-10-13T09:51:40.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:40.273+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.274053+0000) 2019-10-13T09:51:40.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:40.274+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.274599+0000) 2019-10-13T09:51:40.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:40.274+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.274782+0000) 2019-10-13T09:51:40.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:40.274+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.275064+0000) 2019-10-13T09:51:40.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:40.274+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.275150+0000) 2019-10-13T09:51:40.277 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:40.275+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.276164+0000) 2019-10-13T09:51:40.277 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:40.274+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.275364+0000) 2019-10-13T09:51:40.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:40.276+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:40.277566+0000) 2019-10-13T09:51:40.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:40.279+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174679 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:42.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:42.948+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:42.948599+0000) 2019-10-13T09:51:42.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:42.947+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:42.947593+0000) 2019-10-13T09:51:44.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:44.061+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.061141+0000) 2019-10-13T09:51:44.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:44.705+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.705603+0000) 2019-10-13T09:51:45.283 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:45.282+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.283088+0000) 2019-10-13T09:51:45.284 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:45.281+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.282024+0000) 2019-10-13T09:51:45.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:45.281+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.282224+0000) 2019-10-13T09:51:45.284 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:45.281+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.282249+0000) 2019-10-13T09:51:45.285 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:45.282+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.282465+0000) 2019-10-13T09:51:45.285 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:45.282+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.282593+0000) 2019-10-13T09:51:45.285 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:45.282+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.283033+0000) 2019-10-13T09:51:45.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:45.284+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.285023+0000) 2019-10-13T09:51:45.289 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:45.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:45.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:45.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.594947+0000) 2019-10-13T09:51:45.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:45.594+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:45.594947+0000) 2019-10-13T09:51:48.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:48.504+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:48.505542+0000) 2019-10-13T09:51:48.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:48.842+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:48.843977+0000) 2019-10-13T09:51:50.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:50.289+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.291118+0000) 2019-10-13T09:51:50.292 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:50.288+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.290101+0000) 2019-10-13T09:51:50.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:50.289+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.290683+0000) 2019-10-13T09:51:50.292 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:50.289+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.291032+0000) 2019-10-13T09:51:50.293 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:50.289+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.291283+0000) 2019-10-13T09:51:50.293 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:50.290+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.291389+0000) 2019-10-13T09:51:50.293 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:50.290+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.291591+0000) 2019-10-13T09:51:50.294 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:50.291+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:50.293673+0000) 2019-10-13T09:51:50.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:50.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:52.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:52.947+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:52.948743+0000) 2019-10-13T09:51:52.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:52.946+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:52.947780+0000) 2019-10-13T09:51:54.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:54.062+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.063604+0000) 2019-10-13T09:51:54.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:54.704+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.705795+0000) 2019-10-13T09:51:55.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:51:55.298+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.299620+0000) 2019-10-13T09:51:55.301 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:55.298+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.299786+0000) 2019-10-13T09:51:55.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:55.298+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.299991+0000) 2019-10-13T09:51:55.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:51:55.299+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.301512+0000) 2019-10-13T09:51:55.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:55.299+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.300633+0000) 2019-10-13T09:51:55.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:51:55.299+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.300837+0000) 2019-10-13T09:51:55.303 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:55.299+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.300984+0000) 2019-10-13T09:51:55.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:51:55.301+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.303423+0000) 2019-10-13T09:51:55.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:51:55.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174774 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:51:55.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:51:55.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.595118+0000) 2019-10-13T09:51:55.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:51:55.594+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:55.595118+0000) 2019-10-13T09:51:58.023 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:51:58.024 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:58.028 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:51:58.028 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:58.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:51:58.504+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:58.505688+0000) 2019-10-13T09:51:58.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:51:58.843+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:58.844147+0000) 2019-10-13T09:52:00.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:00.308+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.310043+0000) 2019-10-13T09:52:00.311 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:00.307+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.309076+0000) 2019-10-13T09:52:00.311 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:00.308+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.309429+0000) 2019-10-13T09:52:00.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:00.308+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.309690+0000) 2019-10-13T09:52:00.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:00.308+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.309916+0000) 2019-10-13T09:52:00.312 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:00.308+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.310065+0000) 2019-10-13T09:52:00.312 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:00.308+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.310267+0000) 2019-10-13T09:52:00.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:00.310+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:00.312530+0000) 2019-10-13T09:52:00.317 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:00.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:02.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:02.948+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:02.948907+0000) 2019-10-13T09:52:02.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:02.947+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:02.947972+0000) 2019-10-13T09:52:04.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:04.065+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.065915+0000) 2019-10-13T09:52:04.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:04.705+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.705986+0000) 2019-10-13T09:52:05.319 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:05.316+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.318228+0000) 2019-10-13T09:52:05.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:05.317+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.318616+0000) 2019-10-13T09:52:05.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:05.317+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.319041+0000) 2019-10-13T09:52:05.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:05.317+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.319165+0000) 2019-10-13T09:52:05.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:05.317+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.319286+0000) 2019-10-13T09:52:05.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:05.318+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.319559+0000) 2019-10-13T09:52:05.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:05.320+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.320906+0000) 2019-10-13T09:52:05.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:05.322+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.323199+0000) 2019-10-13T09:52:05.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:05.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:05.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:05.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.595285+0000) 2019-10-13T09:52:05.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:05.594+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:05.595286+0000) 2019-10-13T09:52:08.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:08.505+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:08.505832+0000) 2019-10-13T09:52:08.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:08.843+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:08.844311+0000) 2019-10-13T09:52:10.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:10.326+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.328281+0000) 2019-10-13T09:52:10.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:10.326+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.328377+0000) 2019-10-13T09:52:10.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:10.327+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.328644+0000) 2019-10-13T09:52:10.330 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:10.327+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.328827+0000) 2019-10-13T09:52:10.331 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:10.329+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.330195+0000) 2019-10-13T09:52:10.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:10.330+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.331397+0000) 2019-10-13T09:52:10.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:10.327+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.329243+0000) 2019-10-13T09:52:10.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:10.328+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:10.329592+0000) 2019-10-13T09:52:10.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:10.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:12.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:12.948+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:12.949074+0000) 2019-10-13T09:52:12.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:12.947+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:12.948120+0000) 2019-10-13T09:52:14.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:14.067+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.068278+0000) 2019-10-13T09:52:14.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:14.705+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.706177+0000) 2019-10-13T09:52:15.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:15.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.338557+0000) 2019-10-13T09:52:15.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:15.337+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.337613+0000) 2019-10-13T09:52:15.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:15.337+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.337785+0000) 2019-10-13T09:52:15.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:15.337+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.337971+0000) 2019-10-13T09:52:15.342 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:15.337+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.338172+0000) 2019-10-13T09:52:15.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:15.338+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.338762+0000) 2019-10-13T09:52:15.342 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:15.338+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.338901+0000) 2019-10-13T09:52:15.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:15.341+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.341938+0000) 2019-10-13T09:52:15.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:15.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174896 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:15.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:15.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.595476+0000) 2019-10-13T09:52:15.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:15.595+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:15.595475+0000) 2019-10-13T09:52:18.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:18.505+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:18.505962+0000) 2019-10-13T09:52:18.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:18.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:18.844502+0000) 2019-10-13T09:52:20.348 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:20.346+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.346740+0000) 2019-10-13T09:52:20.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:20.347+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.348335+0000) 2019-10-13T09:52:20.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:20.346+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.347235+0000) 2019-10-13T09:52:20.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:20.347+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.347616+0000) 2019-10-13T09:52:20.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:20.347+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.347763+0000) 2019-10-13T09:52:20.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:20.347+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.347965+0000) 2019-10-13T09:52:20.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:20.347+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.348214+0000) 2019-10-13T09:52:20.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:20.350+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:20.351528+0000) 2019-10-13T09:52:20.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:20.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:22.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:22.948+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:22.949225+0000) 2019-10-13T09:52:22.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:22.948+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:22.948314+0000) 2019-10-13T09:52:24.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:24.070+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.070750+0000) 2019-10-13T09:52:24.707 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:24.706+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.706368+0000) 2019-10-13T09:52:25.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:25.355+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.355644+0000) 2019-10-13T09:52:25.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:25.355+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.356003+0000) 2019-10-13T09:52:25.357 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:25.355+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.356055+0000) 2019-10-13T09:52:25.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:25.356+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.356976+0000) 2019-10-13T09:52:25.358 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:25.355+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.356170+0000) 2019-10-13T09:52:25.359 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:25.355+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.356322+0000) 2019-10-13T09:52:25.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:25.356+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.356711+0000) 2019-10-13T09:52:25.360 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:25.359+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.360097+0000) 2019-10-13T09:52:25.362 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:25.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:25.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:25.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.595610+0000) 2019-10-13T09:52:25.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:25.595+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:25.595652+0000) 2019-10-13T09:52:28.119 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:52:28.120 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:28.124 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:52:28.124 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:28.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:28.505+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:28.506132+0000) 2019-10-13T09:52:28.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:28.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:28.844630+0000) 2019-10-13T09:52:30.364 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:30.364+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.364773+0000) 2019-10-13T09:52:30.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:30.363+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.363953+0000) 2019-10-13T09:52:30.366 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:30.363+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.363736+0000) 2019-10-13T09:52:30.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:30.363+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.364249+0000) 2019-10-13T09:52:30.366 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:30.364+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.364477+0000) 2019-10-13T09:52:30.366 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:30.364+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.364688+0000) 2019-10-13T09:52:30.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:30.364+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.364849+0000) 2019-10-13T09:52:30.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:30.366+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:30.367398+0000) 2019-10-13T09:52:30.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:30.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 174985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:32.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:32.949+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:32.949416+0000) 2019-10-13T09:52:32.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:32.947+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:32.948456+0000) 2019-10-13T09:52:34.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:34.072+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.072909+0000) 2019-10-13T09:52:34.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:34.705+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.706587+0000) 2019-10-13T09:52:35.374 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:35.371+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.372724+0000) 2019-10-13T09:52:35.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:35.371+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.372597+0000) 2019-10-13T09:52:35.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:35.373+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.374065+0000) 2019-10-13T09:52:35.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:35.371+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.373339+0000) 2019-10-13T09:52:35.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:35.371+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.373341+0000) 2019-10-13T09:52:35.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:35.372+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.373693+0000) 2019-10-13T09:52:35.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:35.372+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.373754+0000) 2019-10-13T09:52:35.376 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:35.376+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.376846+0000) 2019-10-13T09:52:35.381 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:35.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:35.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:35.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.595776+0000) 2019-10-13T09:52:35.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:35.594+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:35.595794+0000) 2019-10-13T09:52:38.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:38.505+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:38.506318+0000) 2019-10-13T09:52:38.846 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:38.843+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:38.844796+0000) 2019-10-13T09:52:40.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:40.380+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.382687+0000) 2019-10-13T09:52:40.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:40.380+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.381720+0000) 2019-10-13T09:52:40.383 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:40.380+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.381853+0000) 2019-10-13T09:52:40.384 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:40.380+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.382321+0000) 2019-10-13T09:52:40.384 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:40.381+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.382457+0000) 2019-10-13T09:52:40.385 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:40.381+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.382811+0000) 2019-10-13T09:52:40.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:40.381+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.382925+0000) 2019-10-13T09:52:40.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:40.384+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:40.386481+0000) 2019-10-13T09:52:40.390 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:40.387+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175045 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:42.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:42.947+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:42.948587+0000) 2019-10-13T09:52:42.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:42.947+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:42.949631+0000) 2019-10-13T09:52:44.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:44.073+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.075288+0000) 2019-10-13T09:52:44.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:44.705+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.706784+0000) 2019-10-13T09:52:45.392 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:45.389+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.390560+0000) 2019-10-13T09:52:45.392 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:45.389+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.390962+0000) 2019-10-13T09:52:45.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:45.389+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.390863+0000) 2019-10-13T09:52:45.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:45.389+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.391158+0000) 2019-10-13T09:52:45.393 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:45.389+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.391235+0000) 2019-10-13T09:52:45.394 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:45.391+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.392783+0000) 2019-10-13T09:52:45.394 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:45.390+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.391684+0000) 2019-10-13T09:52:45.395 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:45.393+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.394933+0000) 2019-10-13T09:52:45.399 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:45.396+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:45.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:45.594+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.595893+0000) 2019-10-13T09:52:45.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:45.594+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:45.595914+0000) 2019-10-13T09:52:48.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:48.505+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:48.506453+0000) 2019-10-13T09:52:48.846 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:48.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:48.844955+0000) 2019-10-13T09:52:50.401 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:50.398+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.399528+0000) 2019-10-13T09:52:50.401 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:50.398+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.399930+0000) 2019-10-13T09:52:50.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:50.398+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.400208+0000) 2019-10-13T09:52:50.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:50.399+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.400507+0000) 2019-10-13T09:52:50.402 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:50.399+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.400617+0000) 2019-10-13T09:52:50.402 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:50.399+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.400763+0000) 2019-10-13T09:52:50.403 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:50.400+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.402052+0000) 2019-10-13T09:52:50.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:50.402+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:50.404293+0000) 2019-10-13T09:52:50.408 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:50.406+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:52.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:52.948+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:52.948765+0000) 2019-10-13T09:52:52.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:52.949+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:52.949812+0000) 2019-10-13T09:52:54.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:54.076+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.077721+0000) 2019-10-13T09:52:54.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:54.706+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.706976+0000) 2019-10-13T09:52:55.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:55.407+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.409137+0000) 2019-10-13T09:52:55.411 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:52:55.407+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.409367+0000) 2019-10-13T09:52:55.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:55.408+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.409558+0000) 2019-10-13T09:52:55.411 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:52:55.408+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.409799+0000) 2019-10-13T09:52:55.412 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:52:55.410+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.411146+0000) 2019-10-13T09:52:55.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:55.408+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.410179+0000) 2019-10-13T09:52:55.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:55.409+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.410804+0000) 2019-10-13T09:52:55.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:52:55.412+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.413170+0000) 2019-10-13T09:52:55.418 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:52:55.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:52:55.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:52:55.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.596060+0000) 2019-10-13T09:52:55.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:52:55.595+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:55.596065+0000) 2019-10-13T09:52:58.227 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:52:58.227 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:58.231 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:52:58.231 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:58.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:52:58.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:58.506583+0000) 2019-10-13T09:52:58.846 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:52:58.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:58.845080+0000) 2019-10-13T09:53:00.421 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:00.417+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.419148+0000) 2019-10-13T09:53:00.421 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:00.417+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.419306+0000) 2019-10-13T09:53:00.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:00.418+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.419452+0000) 2019-10-13T09:53:00.421 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:00.418+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.419833+0000) 2019-10-13T09:53:00.421 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:00.418+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.419979+0000) 2019-10-13T09:53:00.422 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:00.420+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.420851+0000) 2019-10-13T09:53:00.422 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:00.418+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.420153+0000) 2019-10-13T09:53:00.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:00.423+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:00.424119+0000) 2019-10-13T09:53:00.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:00.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:02.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:02.949+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:02.949993+0000) 2019-10-13T09:53:02.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:02.948+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:02.948946+0000) 2019-10-13T09:53:04.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:04.079+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.079955+0000) 2019-10-13T09:53:04.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:04.706+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.707181+0000) 2019-10-13T09:53:05.429 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:05.427+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.427969+0000) 2019-10-13T09:53:05.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:05.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.428353+0000) 2019-10-13T09:53:05.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:05.429+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.429900+0000) 2019-10-13T09:53:05.432 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:05.428+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.428581+0000) 2019-10-13T09:53:05.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:05.428+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.428905+0000) 2019-10-13T09:53:05.432 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:05.428+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.429143+0000) 2019-10-13T09:53:05.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:05.429+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.429493+0000) 2019-10-13T09:53:05.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:05.431+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.432483+0000) 2019-10-13T09:53:05.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:05.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175202 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:05.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:05.595+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.596192+0000) 2019-10-13T09:53:05.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:05.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:05.596262+0000) 2019-10-13T09:53:08.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:08.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:08.506749+0000) 2019-10-13T09:53:08.846 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:08.845+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:08.845248+0000) 2019-10-13T09:53:10.440 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:10.437+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.438308+0000) 2019-10-13T09:53:10.440 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:10.438+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.438700+0000) 2019-10-13T09:53:10.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:10.438+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.438725+0000) 2019-10-13T09:53:10.440 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:10.438+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.438874+0000) 2019-10-13T09:53:10.441 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:10.438+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.439127+0000) 2019-10-13T09:53:10.441 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:10.439+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.440191+0000) 2019-10-13T09:53:10.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:10.439+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.439446+0000) 2019-10-13T09:53:10.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:10.442+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:10.443354+0000) 2019-10-13T09:53:10.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:10.444+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175229 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:12.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:12.949+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:12.950195+0000) 2019-10-13T09:53:12.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:12.948+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:12.949145+0000) 2019-10-13T09:53:14.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:14.082+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.082395+0000) 2019-10-13T09:53:14.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:14.707+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.707365+0000) 2019-10-13T09:53:15.447 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:15.445+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.446184+0000) 2019-10-13T09:53:15.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:15.446+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.446423+0000) 2019-10-13T09:53:15.448 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:15.446+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.446619+0000) 2019-10-13T09:53:15.449 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:15.447+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.448277+0000) 2019-10-13T09:53:15.449 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:15.446+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.446880+0000) 2019-10-13T09:53:15.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:15.446+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.447378+0000) 2019-10-13T09:53:15.449 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:15.447+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.447507+0000) 2019-10-13T09:53:15.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:15.449+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.450425+0000) 2019-10-13T09:53:15.454 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:15.452+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:15.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:15.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.596417+0000) 2019-10-13T09:53:15.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:15.596+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:15.596410+0000) 2019-10-13T09:53:18.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:18.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:18.506866+0000) 2019-10-13T09:53:18.846 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:18.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:18.845448+0000) 2019-10-13T09:53:20.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:20.453+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.455135+0000) 2019-10-13T09:53:20.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:20.454+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.455648+0000) 2019-10-13T09:53:20.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:20.454+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.455981+0000) 2019-10-13T09:53:20.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:20.454+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.456153+0000) 2019-10-13T09:53:20.458 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:20.456+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.457534+0000) 2019-10-13T09:53:20.458 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:20.455+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.456575+0000) 2019-10-13T09:53:20.458 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:20.455+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.456850+0000) 2019-10-13T09:53:20.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:20.458+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:20.458737+0000) 2019-10-13T09:53:20.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:20.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175289 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:22.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:22.950+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:22.950424+0000) 2019-10-13T09:53:22.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:22.948+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:22.949343+0000) 2019-10-13T09:53:24.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:24.084+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.084798+0000) 2019-10-13T09:53:24.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:24.706+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.707545+0000) 2019-10-13T09:53:25.465 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:25.462+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.464070+0000) 2019-10-13T09:53:25.465 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:25.462+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.464331+0000) 2019-10-13T09:53:25.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:25.463+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.464807+0000) 2019-10-13T09:53:25.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:25.463+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.465022+0000) 2019-10-13T09:53:25.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:25.463+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.465153+0000) 2019-10-13T09:53:25.467 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:25.463+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.465330+0000) 2019-10-13T09:53:25.467 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:25.465+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.466677+0000) 2019-10-13T09:53:25.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:25.467+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.467901+0000) 2019-10-13T09:53:25.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:25.469+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:25.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:25.595+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.596579+0000) 2019-10-13T09:53:25.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:25.595+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:25.596578+0000) 2019-10-13T09:53:28.316 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:53:28.316 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:28.321 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:53:28.321 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:28.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:28.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:28.507015+0000) 2019-10-13T09:53:28.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:28.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:28.845653+0000) 2019-10-13T09:53:30.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:30.472+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.473976+0000) 2019-10-13T09:53:30.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:30.471+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.473135+0000) 2019-10-13T09:53:30.475 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:30.472+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.473394+0000) 2019-10-13T09:53:30.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:30.472+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.473486+0000) 2019-10-13T09:53:30.475 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:30.472+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.473735+0000) 2019-10-13T09:53:30.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:30.472+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.473941+0000) 2019-10-13T09:53:30.476 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:30.472+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.474236+0000) 2019-10-13T09:53:30.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:30.475+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:30.476933+0000) 2019-10-13T09:53:30.481 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:30.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:32.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:32.949+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:32.950571+0000) 2019-10-13T09:53:32.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:32.948+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:32.949578+0000) 2019-10-13T09:53:34.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:34.086+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.087095+0000) 2019-10-13T09:53:34.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:34.706+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.707714+0000) 2019-10-13T09:53:35.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:35.481+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.482843+0000) 2019-10-13T09:53:35.483 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:35.480+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.481878+0000) 2019-10-13T09:53:35.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:35.480+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.482108+0000) 2019-10-13T09:53:35.484 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:35.480+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.482367+0000) 2019-10-13T09:53:35.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:35.482+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.484514+0000) 2019-10-13T09:53:35.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:35.481+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.482812+0000) 2019-10-13T09:53:35.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:35.481+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.483116+0000) 2019-10-13T09:53:35.485 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:35.482+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.483519+0000) 2019-10-13T09:53:35.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:35.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:35.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:35.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.596771+0000) 2019-10-13T09:53:35.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:35.596+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:35.596772+0000) 2019-10-13T09:53:38.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:38.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:38.507220+0000) 2019-10-13T09:53:38.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:38.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:38.845782+0000) 2019-10-13T09:53:40.493 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:40.490+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.491746+0000) 2019-10-13T09:53:40.493 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:40.490+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.492182+0000) 2019-10-13T09:53:40.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:40.490+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.492106+0000) 2019-10-13T09:53:40.494 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:40.491+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.493185+0000) 2019-10-13T09:53:40.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:40.491+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.492533+0000) 2019-10-13T09:53:40.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:40.491+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.492610+0000) 2019-10-13T09:53:40.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:40.491+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.492726+0000) 2019-10-13T09:53:40.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:40.493+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:40.495569+0000) 2019-10-13T09:53:40.499 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:40.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:42.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:42.949+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:42.949757+0000) 2019-10-13T09:53:42.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:42.949+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:42.950751+0000) 2019-10-13T09:53:44.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:44.088+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.089493+0000) 2019-10-13T09:53:44.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:44.707+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.707898+0000) 2019-10-13T09:53:45.500 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:45.497+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.499224+0000) 2019-10-13T09:53:45.501 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:45.498+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.499405+0000) 2019-10-13T09:53:45.501 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:45.498+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.499609+0000) 2019-10-13T09:53:45.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:45.500+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.501191+0000) 2019-10-13T09:53:45.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:45.499+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.500440+0000) 2019-10-13T09:53:45.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:45.499+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.500656+0000) 2019-10-13T09:53:45.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:45.498+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.500231+0000) 2019-10-13T09:53:45.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:45.503+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.504286+0000) 2019-10-13T09:53:45.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:45.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:45.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:45.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.596940+0000) 2019-10-13T09:53:45.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:45.596+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:45.596940+0000) 2019-10-13T09:53:48.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:48.507+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:48.507410+0000) 2019-10-13T09:53:48.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:48.845+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:48.845951+0000) 2019-10-13T09:53:50.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:50.509+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.509770+0000) 2019-10-13T09:53:50.511 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:50.508+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.508406+0000) 2019-10-13T09:53:50.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:50.508+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.508838+0000) 2019-10-13T09:53:50.511 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:50.508+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.509079+0000) 2019-10-13T09:53:50.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:50.508+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.509168+0000) 2019-10-13T09:53:50.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:50.508+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.509300+0000) 2019-10-13T09:53:50.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:50.509+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.509544+0000) 2019-10-13T09:53:50.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:50.511+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:50.511987+0000) 2019-10-13T09:53:50.516 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:50.514+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:52.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:52.950+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:52.950926+0000) 2019-10-13T09:53:52.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:52.949+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:52.949939+0000) 2019-10-13T09:53:54.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:54.091+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.092044+0000) 2019-10-13T09:53:54.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:54.707+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.708089+0000) 2019-10-13T09:53:55.518 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:55.516+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.516850+0000) 2019-10-13T09:53:55.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:55.516+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.517224+0000) 2019-10-13T09:53:55.519 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:53:55.517+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.518229+0000) 2019-10-13T09:53:55.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:55.517+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.517438+0000) 2019-10-13T09:53:55.520 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:55.517+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.517501+0000) 2019-10-13T09:53:55.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:53:55.517+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.517785+0000) 2019-10-13T09:53:55.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:53:55.518+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.518482+0000) 2019-10-13T09:53:55.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:53:55.521+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.522063+0000) 2019-10-13T09:53:55.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:53:55.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175506 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:53:55.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:53:55.596+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.597124+0000) 2019-10-13T09:53:55.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:53:55.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:55.597124+0000) 2019-10-13T09:53:58.437 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:53:58.438 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:58.442 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:53:58.442 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:58.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:53:58.507+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:58.507554+0000) 2019-10-13T09:53:58.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:53:58.845+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:58.846115+0000) 2019-10-13T09:54:00.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:00.525+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.526313+0000) 2019-10-13T09:54:00.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:00.526+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.526465+0000) 2019-10-13T09:54:00.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:00.526+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.526730+0000) 2019-10-13T09:54:00.529 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:00.527+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.527831+0000) 2019-10-13T09:54:00.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:00.526+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.527115+0000) 2019-10-13T09:54:00.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:00.526+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.527142+0000) 2019-10-13T09:54:00.529 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:00.526+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.527381+0000) 2019-10-13T09:54:00.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:00.529+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:00.530579+0000) 2019-10-13T09:54:00.534 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:00.532+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175533 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:02.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:02.949+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:02.950051+0000) 2019-10-13T09:54:02.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:02.950+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:02.951103+0000) 2019-10-13T09:54:04.094 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:04.094+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.094469+0000) 2019-10-13T09:54:04.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:04.706+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.708299+0000) 2019-10-13T09:54:05.537 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:05.533+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.535296+0000) 2019-10-13T09:54:05.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:05.534+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.535490+0000) 2019-10-13T09:54:05.537 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:05.534+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.535761+0000) 2019-10-13T09:54:05.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:05.534+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.535879+0000) 2019-10-13T09:54:05.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:05.534+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.536217+0000) 2019-10-13T09:54:05.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:05.534+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.536355+0000) 2019-10-13T09:54:05.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:05.536+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.537249+0000) 2019-10-13T09:54:05.539 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:05.537+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.538508+0000) 2019-10-13T09:54:05.543 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:05.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175568 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:05.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:05.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.597315+0000) 2019-10-13T09:54:05.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:05.596+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:05.597316+0000) 2019-10-13T09:54:08.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:08.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:08.507749+0000) 2019-10-13T09:54:08.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:08.844+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:08.846302+0000) 2019-10-13T09:54:10.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:10.542+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.544258+0000) 2019-10-13T09:54:10.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:10.542+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.544369+0000) 2019-10-13T09:54:10.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:10.543+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.544561+0000) 2019-10-13T09:54:10.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:10.543+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.544756+0000) 2019-10-13T09:54:10.547 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:10.543+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.545001+0000) 2019-10-13T09:54:10.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:10.543+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.545216+0000) 2019-10-13T09:54:10.547 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:10.545+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.546009+0000) 2019-10-13T09:54:10.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:10.546+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:10.547368+0000) 2019-10-13T09:54:10.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:10.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175595 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:12.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:12.950+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:12.951251+0000) 2019-10-13T09:54:12.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:12.948+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:12.950252+0000) 2019-10-13T09:54:14.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:14.096+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.096791+0000) 2019-10-13T09:54:14.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:14.707+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.708491+0000) 2019-10-13T09:54:15.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:15.551+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.552995+0000) 2019-10-13T09:54:15.554 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:15.551+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.553217+0000) 2019-10-13T09:54:15.555 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:15.554+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.554744+0000) 2019-10-13T09:54:15.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:15.552+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.553842+0000) 2019-10-13T09:54:15.556 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:15.552+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.554190+0000) 2019-10-13T09:54:15.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:15.552+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.554055+0000) 2019-10-13T09:54:15.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:15.553+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.554479+0000) 2019-10-13T09:54:15.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:15.556+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.556875+0000) 2019-10-13T09:54:15.561 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:15.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175628 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:15.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:15.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.597484+0000) 2019-10-13T09:54:15.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:15.596+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:15.597491+0000) 2019-10-13T09:54:18.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:18.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:18.507963+0000) 2019-10-13T09:54:18.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:18.845+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:18.846480+0000) 2019-10-13T09:54:20.563 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:20.562+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.563471+0000) 2019-10-13T09:54:20.564 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:20.561+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.562664+0000) 2019-10-13T09:54:20.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:20.561+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.562862+0000) 2019-10-13T09:54:20.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:20.561+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.562976+0000) 2019-10-13T09:54:20.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:20.561+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.563319+0000) 2019-10-13T09:54:20.565 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:20.562+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.563482+0000) 2019-10-13T09:54:20.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:20.562+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.563645+0000) 2019-10-13T09:54:20.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:20.565+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:20.565850+0000) 2019-10-13T09:54:20.571 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:20.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175655 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:22.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:22.950+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:22.951421+0000) 2019-10-13T09:54:22.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:22.949+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:22.950440+0000) 2019-10-13T09:54:24.099 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:24.098+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.099289+0000) 2019-10-13T09:54:24.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:24.708+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.708673+0000) 2019-10-13T09:54:25.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:25.570+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.571556+0000) 2019-10-13T09:54:25.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:25.570+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.571782+0000) 2019-10-13T09:54:25.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:25.570+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.572237+0000) 2019-10-13T09:54:25.574 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:25.571+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.572538+0000) 2019-10-13T09:54:25.575 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:25.571+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.573565+0000) 2019-10-13T09:54:25.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:25.571+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.572748+0000) 2019-10-13T09:54:25.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:25.571+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.573033+0000) 2019-10-13T09:54:25.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:25.574+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.576244+0000) 2019-10-13T09:54:25.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:25.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:25.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:25.596+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.597623+0000) 2019-10-13T09:54:25.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:25.596+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:25.597634+0000) 2019-10-13T09:54:28.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:28.506+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:28.508155+0000) 2019-10-13T09:54:28.526 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:54:28.526 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:28.530 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:54:28.530 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:28.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:28.846+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:28.846691+0000) 2019-10-13T09:54:30.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:30.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.580690+0000) 2019-10-13T09:54:30.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:30.579+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.581091+0000) 2019-10-13T09:54:30.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:30.579+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.581321+0000) 2019-10-13T09:54:30.586 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:30.580+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.581447+0000) 2019-10-13T09:54:30.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:30.580+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.581607+0000) 2019-10-13T09:54:30.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:30.580+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.581805+0000) 2019-10-13T09:54:30.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:30.580+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.581756+0000) 2019-10-13T09:54:30.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:30.582+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:30.584376+0000) 2019-10-13T09:54:30.589 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:30.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175717 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:32.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:32.950+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:32.951568+0000) 2019-10-13T09:54:32.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:32.950+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:32.950615+0000) 2019-10-13T09:54:34.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:34.100+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.101507+0000) 2019-10-13T09:54:34.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:34.708+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.708883+0000) 2019-10-13T09:54:35.591 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:35.589+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.590140+0000) 2019-10-13T09:54:35.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:35.589+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.590326+0000) 2019-10-13T09:54:35.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:35.590+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.590873+0000) 2019-10-13T09:54:35.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:35.590+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.591604+0000) 2019-10-13T09:54:35.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:35.590+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.591046+0000) 2019-10-13T09:54:35.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:35.590+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.591175+0000) 2019-10-13T09:54:35.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:35.591+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.591587+0000) 2019-10-13T09:54:35.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:35.593+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.594278+0000) 2019-10-13T09:54:35.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:35.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175750 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:35.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:35.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.597754+0000) 2019-10-13T09:54:35.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:35.597+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:35.597752+0000) 2019-10-13T09:54:38.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:38.508+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:38.508434+0000) 2019-10-13T09:54:38.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:38.846+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:38.846910+0000) 2019-10-13T09:54:40.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:40.598+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.598882+0000) 2019-10-13T09:54:40.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:40.598+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.598986+0000) 2019-10-13T09:54:40.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:40.598+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.599302+0000) 2019-10-13T09:54:40.602 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:40.599+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.599436+0000) 2019-10-13T09:54:40.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:40.599+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.599555+0000) 2019-10-13T09:54:40.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:40.599+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.599799+0000) 2019-10-13T09:54:40.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:40.600+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.600882+0000) 2019-10-13T09:54:40.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:40.602+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:40.603497+0000) 2019-10-13T09:54:40.605 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:40.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:42.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:42.951+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:42.951702+0000) 2019-10-13T09:54:42.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:42.950+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:42.950793+0000) 2019-10-13T09:54:44.104 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:44.103+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.103960+0000) 2019-10-13T09:54:44.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:44.708+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.709073+0000) 2019-10-13T09:54:45.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:45.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.597921+0000) 2019-10-13T09:54:45.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:45.597+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.597922+0000) 2019-10-13T09:54:45.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:45.606+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.606449+0000) 2019-10-13T09:54:45.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:45.606+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.606949+0000) 2019-10-13T09:54:45.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:45.606+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.607209+0000) 2019-10-13T09:54:45.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:45.607+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.607391+0000) 2019-10-13T09:54:45.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:45.607+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.607600+0000) 2019-10-13T09:54:45.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:45.607+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.607746+0000) 2019-10-13T09:54:45.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:45.608+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.609606+0000) 2019-10-13T09:54:45.610 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:45.609+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:45.610068+0000) 2019-10-13T09:54:45.615 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:45.613+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:48.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:48.508+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:48.508595+0000) 2019-10-13T09:54:48.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:48.847+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:48.847096+0000) 2019-10-13T09:54:50.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:50.614+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.615941+0000) 2019-10-13T09:54:50.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:50.615+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.616520+0000) 2019-10-13T09:54:50.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:50.615+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.616935+0000) 2019-10-13T09:54:50.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:50.615+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.617099+0000) 2019-10-13T09:54:50.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:50.615+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.617265+0000) 2019-10-13T09:54:50.619 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:50.616+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.617481+0000) 2019-10-13T09:54:50.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:50.618+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.618857+0000) 2019-10-13T09:54:50.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:50.618+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:50.619579+0000) 2019-10-13T09:54:50.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:50.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:52.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:52.951+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:52.951868+0000) 2019-10-13T09:54:52.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:52.949+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:52.950994+0000) 2019-10-13T09:54:54.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:54.105+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.106420+0000) 2019-10-13T09:54:54.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:54.708+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.709264+0000) 2019-10-13T09:54:55.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:55.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.598134+0000) 2019-10-13T09:54:55.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:55.597+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.598128+0000) 2019-10-13T09:54:55.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:55.623+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.625344+0000) 2019-10-13T09:54:55.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:54:55.624+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.625508+0000) 2019-10-13T09:54:55.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:55.624+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.625693+0000) 2019-10-13T09:54:55.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:54:55.624+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.625945+0000) 2019-10-13T09:54:55.627 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:54:55.625+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.626456+0000) 2019-10-13T09:54:55.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:54:55.626+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.627375+0000) 2019-10-13T09:54:55.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:54:55.625+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.627220+0000) 2019-10-13T09:54:55.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:54:55.629+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:55.630047+0000) 2019-10-13T09:54:55.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:54:55.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:54:58.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:54:58.507+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:58.508803+0000) 2019-10-13T09:54:58.665 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:54:58.665 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:58.670 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:54:58.670 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:58.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:54:58.845+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:58.847305+0000) 2019-10-13T09:55:00.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:00.633+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.634422+0000) 2019-10-13T09:55:00.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:00.633+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.634786+0000) 2019-10-13T09:55:00.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:00.633+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.635238+0000) 2019-10-13T09:55:00.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:00.634+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.635531+0000) 2019-10-13T09:55:00.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:00.634+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.635532+0000) 2019-10-13T09:55:00.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:00.634+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.635920+0000) 2019-10-13T09:55:00.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:00.636+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.636937+0000) 2019-10-13T09:55:00.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:00.637+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:00.638288+0000) 2019-10-13T09:55:00.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:00.640+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175899 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:02.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:02.951+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:02.952046+0000) 2019-10-13T09:55:02.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:02.950+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:02.951148+0000) 2019-10-13T09:55:04.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:04.108+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.108769+0000) 2019-10-13T09:55:04.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:04.708+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.709456+0000) 2019-10-13T09:55:05.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:05.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.598336+0000) 2019-10-13T09:55:05.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:05.597+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.598317+0000) 2019-10-13T09:55:05.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:05.642+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.643687+0000) 2019-10-13T09:55:05.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:05.642+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.643826+0000) 2019-10-13T09:55:05.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:05.642+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.644034+0000) 2019-10-13T09:55:05.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:05.642+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.644170+0000) 2019-10-13T09:55:05.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:05.642+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.644282+0000) 2019-10-13T09:55:05.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:05.644+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.645542+0000) 2019-10-13T09:55:05.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:05.643+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.644963+0000) 2019-10-13T09:55:05.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:05.647+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:05.648358+0000) 2019-10-13T09:55:05.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:05.649+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:08.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:08.508+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:08.508998+0000) 2019-10-13T09:55:08.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:08.846+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:08.847475+0000) 2019-10-13T09:55:10.654 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:10.653+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.654105+0000) 2019-10-13T09:55:10.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:10.651+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.653145+0000) 2019-10-13T09:55:10.655 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:10.651+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.653280+0000) 2019-10-13T09:55:10.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:10.652+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.653417+0000) 2019-10-13T09:55:10.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:10.652+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.654122+0000) 2019-10-13T09:55:10.656 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:10.652+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.654246+0000) 2019-10-13T09:55:10.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:10.653+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.654476+0000) 2019-10-13T09:55:10.656 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:10.655+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:10.656574+0000) 2019-10-13T09:55:10.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:10.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:12.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:12.951+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:12.952229+0000) 2019-10-13T09:55:12.952 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:12.950+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:12.951329+0000) 2019-10-13T09:55:14.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:14.110+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.111132+0000) 2019-10-13T09:55:14.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:14.709+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.709642+0000) 2019-10-13T09:55:15.599 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:15.598+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.598514+0000) 2019-10-13T09:55:15.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:15.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.598536+0000) 2019-10-13T09:55:15.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:15.660+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.662320+0000) 2019-10-13T09:55:15.664 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:15.661+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.662521+0000) 2019-10-13T09:55:15.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:15.661+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.662808+0000) 2019-10-13T09:55:15.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:15.663+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.664804+0000) 2019-10-13T09:55:15.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:15.661+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.663269+0000) 2019-10-13T09:55:15.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:15.662+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.663530+0000) 2019-10-13T09:55:15.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:15.662+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.663764+0000) 2019-10-13T09:55:15.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:15.664+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:15.666197+0000) 2019-10-13T09:55:15.670 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:15.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 175994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:18.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:18.508+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:18.509155+0000) 2019-10-13T09:55:18.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:18.847+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:18.847673+0000) 2019-10-13T09:55:20.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:20.670+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.671221+0000) 2019-10-13T09:55:20.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:20.671+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.671537+0000) 2019-10-13T09:55:20.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:20.671+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.673287+0000) 2019-10-13T09:55:20.676 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:20.671+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.671745+0000) 2019-10-13T09:55:20.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:20.670+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.671384+0000) 2019-10-13T09:55:20.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:20.671+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.671856+0000) 2019-10-13T09:55:20.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:20.671+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.672384+0000) 2019-10-13T09:55:20.677 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:20.673+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:20.675520+0000) 2019-10-13T09:55:20.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:20.678+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:22.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:22.951+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:22.952414+0000) 2019-10-13T09:55:22.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:22.951+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:22.951514+0000) 2019-10-13T09:55:24.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:24.112+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.113527+0000) 2019-10-13T09:55:24.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:24.709+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.709741+0000) 2019-10-13T09:55:25.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:25.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.598707+0000) 2019-10-13T09:55:25.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:25.598+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.598704+0000) 2019-10-13T09:55:25.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:25.680+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.680714+0000) 2019-10-13T09:55:25.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:25.680+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.681030+0000) 2019-10-13T09:55:25.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:25.680+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.681155+0000) 2019-10-13T09:55:25.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:25.681+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.682639+0000) 2019-10-13T09:55:25.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:25.681+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.681623+0000) 2019-10-13T09:55:25.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:25.681+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.681841+0000) 2019-10-13T09:55:25.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:25.681+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.682001+0000) 2019-10-13T09:55:25.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:25.683+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:25.684935+0000) 2019-10-13T09:55:25.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:25.688+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176056 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:28.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:28.509+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:28.509352+0000) 2019-10-13T09:55:28.838 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:55:28.838 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:28.842 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:55:28.842 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:28.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:28.847+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:28.847869+0000) 2019-10-13T09:55:30.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:30.690+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.690602+0000) 2019-10-13T09:55:30.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:30.690+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.691127+0000) 2019-10-13T09:55:30.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:30.690+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.691348+0000) 2019-10-13T09:55:30.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:30.691+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.691617+0000) 2019-10-13T09:55:30.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:30.692+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.692975+0000) 2019-10-13T09:55:30.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:30.691+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.692186+0000) 2019-10-13T09:55:30.694 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:30.691+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.691890+0000) 2019-10-13T09:55:30.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:30.693+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:30.694537+0000) 2019-10-13T09:55:30.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:30.697+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:32.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:32.951+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:32.952620+0000) 2019-10-13T09:55:32.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:32.951+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:32.951685+0000) 2019-10-13T09:55:34.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:34.115+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.115930+0000) 2019-10-13T09:55:34.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:34.709+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.709925+0000) 2019-10-13T09:55:35.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:35.597+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.598901+0000) 2019-10-13T09:55:35.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:35.597+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.598899+0000) 2019-10-13T09:55:35.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:35.700+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.701045+0000) 2019-10-13T09:55:35.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:35.698+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.700139+0000) 2019-10-13T09:55:35.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:35.699+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.700439+0000) 2019-10-13T09:55:35.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:35.699+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.700612+0000) 2019-10-13T09:55:35.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:35.699+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.700862+0000) 2019-10-13T09:55:35.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:35.699+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.701083+0000) 2019-10-13T09:55:35.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:35.700+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.701413+0000) 2019-10-13T09:55:35.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:35.703+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:35.704025+0000) 2019-10-13T09:55:35.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:35.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:38.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:38.508+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:38.509563+0000) 2019-10-13T09:55:38.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:38.847+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:38.848053+0000) 2019-10-13T09:55:40.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:40.709+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.709976+0000) 2019-10-13T09:55:40.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:40.707+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.709140+0000) 2019-10-13T09:55:40.712 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:40.707+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.709024+0000) 2019-10-13T09:55:40.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:40.707+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.709327+0000) 2019-10-13T09:55:40.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:40.708+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.709507+0000) 2019-10-13T09:55:40.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:40.708+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.709902+0000) 2019-10-13T09:55:40.713 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:40.708+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.710058+0000) 2019-10-13T09:55:40.713 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:40.712+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:40.713071+0000) 2019-10-13T09:55:40.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:40.714+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:42.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:42.952+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:42.952782+0000) 2019-10-13T09:55:42.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:42.950+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:42.951803+0000) 2019-10-13T09:55:44.118 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:44.117+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.118304+0000) 2019-10-13T09:55:44.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:44.709+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.710114+0000) 2019-10-13T09:55:45.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:45.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.599100+0000) 2019-10-13T09:55:45.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:45.598+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.599098+0000) 2019-10-13T09:55:45.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:45.716+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.718050+0000) 2019-10-13T09:55:45.720 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:45.718+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.719494+0000) 2019-10-13T09:55:45.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:45.717+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.718516+0000) 2019-10-13T09:55:45.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:45.717+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.718764+0000) 2019-10-13T09:55:45.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:45.717+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.718928+0000) 2019-10-13T09:55:45.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:45.717+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.719131+0000) 2019-10-13T09:55:45.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:45.718+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.719673+0000) 2019-10-13T09:55:45.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:45.721+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:45.722575+0000) 2019-10-13T09:55:45.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:45.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:48.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:48.508+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:48.509750+0000) 2019-10-13T09:55:48.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:48.847+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:48.848225+0000) 2019-10-13T09:55:50.729 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:50.726+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.727431+0000) 2019-10-13T09:55:50.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:50.726+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.727615+0000) 2019-10-13T09:55:50.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:50.726+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.727803+0000) 2019-10-13T09:55:50.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:50.726+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.728021+0000) 2019-10-13T09:55:50.730 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:50.729+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.729910+0000) 2019-10-13T09:55:50.730 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:50.727+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.728504+0000) 2019-10-13T09:55:50.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:50.727+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.729076+0000) 2019-10-13T09:55:50.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:50.730+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:50.731586+0000) 2019-10-13T09:55:50.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:50.733+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:52.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:52.952+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:52.952949+0000) 2019-10-13T09:55:52.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:52.951+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:52.951997+0000) 2019-10-13T09:55:54.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:54.120+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.120713+0000) 2019-10-13T09:55:54.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:54.708+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.710340+0000) 2019-10-13T09:55:55.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:55.598+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.599217+0000) 2019-10-13T09:55:55.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:55.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.599265+0000) 2019-10-13T09:55:55.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:55:55.736+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.737250+0000) 2019-10-13T09:55:55.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:55.734+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.736352+0000) 2019-10-13T09:55:55.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:55:55.735+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.736523+0000) 2019-10-13T09:55:55.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:55.735+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.736878+0000) 2019-10-13T09:55:55.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:55:55.735+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.737176+0000) 2019-10-13T09:55:55.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:55:55.735+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.737375+0000) 2019-10-13T09:55:55.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:55:55.736+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.737494+0000) 2019-10-13T09:55:55.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:55:55.739+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:55.740206+0000) 2019-10-13T09:55:55.744 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:55:55.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176238 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:55:58.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:55:58.509+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:58.509942+0000) 2019-10-13T09:55:58.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:55:58.847+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:58.848421+0000) 2019-10-13T09:55:58.977 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:55:58.977 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:58.982 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:55:58.982 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:00.746 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:00.743+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.744954+0000) 2019-10-13T09:56:00.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:00.743+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.745213+0000) 2019-10-13T09:56:00.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:00.744+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.745413+0000) 2019-10-13T09:56:00.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:00.745+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.746494+0000) 2019-10-13T09:56:00.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:00.744+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.745973+0000) 2019-10-13T09:56:00.748 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:00.744+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.746112+0000) 2019-10-13T09:56:00.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:00.744+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.746351+0000) 2019-10-13T09:56:00.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:00.748+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:00.749022+0000) 2019-10-13T09:56:00.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:00.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:02.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:02.952+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:02.953118+0000) 2019-10-13T09:56:02.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:02.951+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:02.952188+0000) 2019-10-13T09:56:04.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:04.121+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.123013+0000) 2019-10-13T09:56:04.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:04.710+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.710543+0000) 2019-10-13T09:56:05.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:05.599+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.599423+0000) 2019-10-13T09:56:05.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:05.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.599445+0000) 2019-10-13T09:56:05.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:05.753+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.754071+0000) 2019-10-13T09:56:05.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:05.753+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.754186+0000) 2019-10-13T09:56:05.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:05.754+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.754452+0000) 2019-10-13T09:56:05.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:05.754+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.754556+0000) 2019-10-13T09:56:05.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:05.754+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.754995+0000) 2019-10-13T09:56:05.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:05.754+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.755906+0000) 2019-10-13T09:56:05.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:05.754+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.755309+0000) 2019-10-13T09:56:05.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:05.756+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:05.758131+0000) 2019-10-13T09:56:05.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:05.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:08.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:08.509+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:08.510135+0000) 2019-10-13T09:56:08.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:08.848+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:08.848615+0000) 2019-10-13T09:56:10.764 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:10.762+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.764538+0000) 2019-10-13T09:56:10.765 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:10.763+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.763686+0000) 2019-10-13T09:56:10.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:10.763+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.763926+0000) 2019-10-13T09:56:10.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:10.763+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.764082+0000) 2019-10-13T09:56:10.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:10.763+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.764300+0000) 2019-10-13T09:56:10.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:10.764+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.764459+0000) 2019-10-13T09:56:10.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:10.764+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.764618+0000) 2019-10-13T09:56:10.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:10.765+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:10.767107+0000) 2019-10-13T09:56:10.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:10.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176327 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:12.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:12.952+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:12.953288+0000) 2019-10-13T09:56:12.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:12.951+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:12.952360+0000) 2019-10-13T09:56:14.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:14.124+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.125457+0000) 2019-10-13T09:56:14.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:14.710+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.710682+0000) 2019-10-13T09:56:15.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:15.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.599608+0000) 2019-10-13T09:56:15.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:15.599+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.599608+0000) 2019-10-13T09:56:15.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:15.772+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.774230+0000) 2019-10-13T09:56:15.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:15.772+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.773254+0000) 2019-10-13T09:56:15.775 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:15.773+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.773559+0000) 2019-10-13T09:56:15.775 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:15.773+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.773758+0000) 2019-10-13T09:56:15.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:15.773+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.773833+0000) 2019-10-13T09:56:15.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:15.773+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.774210+0000) 2019-10-13T09:56:15.776 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:15.773+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.774137+0000) 2019-10-13T09:56:15.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:15.774+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:15.776701+0000) 2019-10-13T09:56:15.781 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:15.780+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176360 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:18.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:18.509+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:18.510318+0000) 2019-10-13T09:56:18.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:18.848+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:18.848820+0000) 2019-10-13T09:56:20.783 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:20.781+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.782221+0000) 2019-10-13T09:56:20.784 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:20.782+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.783621+0000) 2019-10-13T09:56:20.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:20.782+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.782641+0000) 2019-10-13T09:56:20.784 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:20.782+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.783074+0000) 2019-10-13T09:56:20.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:20.782+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.783324+0000) 2019-10-13T09:56:20.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:20.782+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.783373+0000) 2019-10-13T09:56:20.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:20.783+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.783679+0000) 2019-10-13T09:56:20.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:20.785+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:20.786211+0000) 2019-10-13T09:56:20.790 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:20.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:22.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:22.952+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:22.953469+0000) 2019-10-13T09:56:22.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:22.951+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:22.952572+0000) 2019-10-13T09:56:24.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:24.127+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.127825+0000) 2019-10-13T09:56:24.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:24.709+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.710874+0000) 2019-10-13T09:56:25.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:25.598+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.599796+0000) 2019-10-13T09:56:25.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:25.598+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.599797+0000) 2019-10-13T09:56:25.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:25.791+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.792525+0000) 2019-10-13T09:56:25.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:25.790+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.791525+0000) 2019-10-13T09:56:25.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:25.790+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.792217+0000) 2019-10-13T09:56:25.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:25.791+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.792431+0000) 2019-10-13T09:56:25.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:25.791+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.792802+0000) 2019-10-13T09:56:25.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:25.791+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.792853+0000) 2019-10-13T09:56:25.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:25.791+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.793094+0000) 2019-10-13T09:56:25.796 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:25.795+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:25.796080+0000) 2019-10-13T09:56:25.800 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:25.797+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:28.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:28.509+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:28.510478+0000) 2019-10-13T09:56:28.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:28.848+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:28.849042+0000) 2019-10-13T09:56:29.090 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:56:29.091 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:29.095 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:56:29.096 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:30.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:30.799+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.800660+0000) 2019-10-13T09:56:30.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:30.799+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.800927+0000) 2019-10-13T09:56:30.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:30.801+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.802597+0000) 2019-10-13T09:56:30.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:30.799+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.801300+0000) 2019-10-13T09:56:30.803 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:30.800+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.801754+0000) 2019-10-13T09:56:30.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:30.800+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.802062+0000) 2019-10-13T09:56:30.804 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:30.800+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.802227+0000) 2019-10-13T09:56:30.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:30.804+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:30.805204+0000) 2019-10-13T09:56:30.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:30.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176449 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:32.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:32.953+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:32.953650+0000) 2019-10-13T09:56:32.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:32.951+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:32.952745+0000) 2019-10-13T09:56:34.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:34.129+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.130142+0000) 2019-10-13T09:56:34.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:34.710+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.711042+0000) 2019-10-13T09:56:35.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:35.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.600000+0000) 2019-10-13T09:56:35.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:35.599+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.600006+0000) 2019-10-13T09:56:35.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:35.810+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.811516+0000) 2019-10-13T09:56:35.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:35.809+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.810685+0000) 2019-10-13T09:56:35.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:35.809+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.810892+0000) 2019-10-13T09:56:35.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:35.809+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.811184+0000) 2019-10-13T09:56:35.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:35.810+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.811552+0000) 2019-10-13T09:56:35.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:35.810+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.811692+0000) 2019-10-13T09:56:35.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:35.810+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.811871+0000) 2019-10-13T09:56:35.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:35.814+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:35.815009+0000) 2019-10-13T09:56:35.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:35.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176482 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:38.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:38.510+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:38.510699+0000) 2019-10-13T09:56:38.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:38.848+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:38.849239+0000) 2019-10-13T09:56:40.820 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:40.820+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.820814+0000) 2019-10-13T09:56:40.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:40.818+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.819763+0000) 2019-10-13T09:56:40.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:40.818+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.820024+0000) 2019-10-13T09:56:40.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:40.818+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.820335+0000) 2019-10-13T09:56:40.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:40.819+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.820489+0000) 2019-10-13T09:56:40.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:40.819+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.820651+0000) 2019-10-13T09:56:40.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:40.819+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.820789+0000) 2019-10-13T09:56:40.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:40.822+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:40.823218+0000) 2019-10-13T09:56:40.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:40.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:42.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:42.953+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:42.953810+0000) 2019-10-13T09:56:42.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:42.952+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:42.952882+0000) 2019-10-13T09:56:44.132 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:44.132+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.132583+0000) 2019-10-13T09:56:44.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:44.710+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.711221+0000) 2019-10-13T09:56:45.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:45.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.600218+0000) 2019-10-13T09:56:45.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:45.599+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.600211+0000) 2019-10-13T09:56:45.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:45.828+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.829165+0000) 2019-10-13T09:56:45.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:45.826+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.828317+0000) 2019-10-13T09:56:45.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:45.827+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.828564+0000) 2019-10-13T09:56:45.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:45.827+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.829201+0000) 2019-10-13T09:56:45.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:45.827+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.829383+0000) 2019-10-13T09:56:45.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:45.828+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.829649+0000) 2019-10-13T09:56:45.831 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:45.828+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.829804+0000) 2019-10-13T09:56:45.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:45.831+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.832135+0000) 2019-10-13T09:56:45.836 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:45.834+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:48.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:48.510+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:48.510878+0000) 2019-10-13T09:56:48.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:48.848+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:48.849377+0000) 2019-10-13T09:56:50.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:50.837+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.837430+0000) 2019-10-13T09:56:50.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:50.837+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.837665+0000) 2019-10-13T09:56:50.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:50.837+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.837705+0000) 2019-10-13T09:56:50.839 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:50.837+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.837946+0000) 2019-10-13T09:56:50.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:50.838+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.839483+0000) 2019-10-13T09:56:50.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:50.837+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.838164+0000) 2019-10-13T09:56:50.840 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:50.838+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.838499+0000) 2019-10-13T09:56:50.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:50.839+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.840506+0000) 2019-10-13T09:56:50.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:50.844+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:52.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:52.953+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:52.953983+0000) 2019-10-13T09:56:52.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:52.952+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:52.952977+0000) 2019-10-13T09:56:54.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:54.134+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:54.135061+0000) 2019-10-13T09:56:54.712 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:54.711+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:54.711425+0000) 2019-10-13T09:56:55.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:55.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.600434+0000) 2019-10-13T09:56:55.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:55.600+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.600430+0000) 2019-10-13T09:56:55.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:56:55.847+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.848858+0000) 2019-10-13T09:56:55.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:56:55.846+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.846780+0000) 2019-10-13T09:56:55.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:56:55.846+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.846936+0000) 2019-10-13T09:56:55.850 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:55.846+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.847174+0000) 2019-10-13T09:56:55.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:55.847+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.847421+0000) 2019-10-13T09:56:55.850 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:56:55.847+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.847581+0000) 2019-10-13T09:56:55.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:56:55.847+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.847909+0000) 2019-10-13T09:56:55.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:56:55.849+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.851212+0000) 2019-10-13T09:56:55.855 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:56:55.853+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:56:58.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:56:58.510+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:58.511062+0000) 2019-10-13T09:56:58.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:56:58.849+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:58.849551+0000) 2019-10-13T09:56:59.179 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:56:59.180 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:59.184 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:56:59.184 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:00.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:00.855+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.855992+0000) 2019-10-13T09:57:00.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:00.855+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.856161+0000) 2019-10-13T09:57:00.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:00.856+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.856564+0000) 2019-10-13T09:57:00.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:00.856+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.856697+0000) 2019-10-13T09:57:00.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:00.856+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.856871+0000) 2019-10-13T09:57:00.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:00.857+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.857472+0000) 2019-10-13T09:57:00.859 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:00.856+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.858402+0000) 2019-10-13T09:57:00.860 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:00.859+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.860806+0000) 2019-10-13T09:57:00.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:00.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:02.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:02.953+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:02.954152+0000) 2019-10-13T09:57:02.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:02.953+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:02.953149+0000) 2019-10-13T09:57:04.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:04.136+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:04.137397+0000) 2019-10-13T09:57:04.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:04.711+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:04.711603+0000) 2019-10-13T09:57:05.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:05.600+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.600583+0000) 2019-10-13T09:57:05.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:05.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.600625+0000) 2019-10-13T09:57:05.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:05.864+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.865309+0000) 2019-10-13T09:57:05.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:05.865+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.865526+0000) 2019-10-13T09:57:05.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:05.865+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.865718+0000) 2019-10-13T09:57:05.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:05.866+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.867802+0000) 2019-10-13T09:57:05.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:05.865+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.866016+0000) 2019-10-13T09:57:05.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:05.866+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.866825+0000) 2019-10-13T09:57:05.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:05.866+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.866997+0000) 2019-10-13T09:57:05.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:05.867+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.869096+0000) 2019-10-13T09:57:05.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:05.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176672 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:08.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:08.509+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:08.511275+0000) 2019-10-13T09:57:08.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:08.848+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:08.849744+0000) 2019-10-13T09:57:10.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:10.874+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.874880+0000) 2019-10-13T09:57:10.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:10.872+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.873881+0000) 2019-10-13T09:57:10.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:10.872+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.874203+0000) 2019-10-13T09:57:10.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:10.872+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.874338+0000) 2019-10-13T09:57:10.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:10.873+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.874548+0000) 2019-10-13T09:57:10.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:10.873+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.874562+0000) 2019-10-13T09:57:10.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:10.873+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.874999+0000) 2019-10-13T09:57:10.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:10.876+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.877344+0000) 2019-10-13T09:57:10.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:10.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:12.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:12.953+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:12.954307+0000) 2019-10-13T09:57:12.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:12.952+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:12.953277+0000) 2019-10-13T09:57:14.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:14.139+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:14.139680+0000) 2019-10-13T09:57:14.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:14.710+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:14.711799+0000) 2019-10-13T09:57:15.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:15.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.600813+0000) 2019-10-13T09:57:15.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:15.599+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.600808+0000) 2019-10-13T09:57:15.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:15.880+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.881838+0000) 2019-10-13T09:57:15.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:15.880+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.882111+0000) 2019-10-13T09:57:15.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:15.880+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.882247+0000) 2019-10-13T09:57:15.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:15.881+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.882482+0000) 2019-10-13T09:57:15.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:15.881+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.882625+0000) 2019-10-13T09:57:15.884 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:15.881+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.882978+0000) 2019-10-13T09:57:15.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:15.883+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.884276+0000) 2019-10-13T09:57:15.886 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:15.885+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.886522+0000) 2019-10-13T09:57:15.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:15.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176734 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:18.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:18.510+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:18.511472+0000) 2019-10-13T09:57:18.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:18.849+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:18.849932+0000) 2019-10-13T09:57:20.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:20.889+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.890578+0000) 2019-10-13T09:57:20.892 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:20.889+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.890737+0000) 2019-10-13T09:57:20.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:20.889+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.890962+0000) 2019-10-13T09:57:20.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:20.889+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.891207+0000) 2019-10-13T09:57:20.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:20.891+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.892233+0000) 2019-10-13T09:57:20.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:20.890+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.891440+0000) 2019-10-13T09:57:20.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:20.890+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.891860+0000) 2019-10-13T09:57:20.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:20.894+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.895361+0000) 2019-10-13T09:57:20.897 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:20.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:22.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:22.954+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:22.954484+0000) 2019-10-13T09:57:22.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:22.952+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:22.953415+0000) 2019-10-13T09:57:24.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:24.141+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:24.141845+0000) 2019-10-13T09:57:24.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:24.711+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:24.712011+0000) 2019-10-13T09:57:25.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:25.599+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.601021+0000) 2019-10-13T09:57:25.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:25.599+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.601023+0000) 2019-10-13T09:57:25.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:25.898+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.898815+0000) 2019-10-13T09:57:25.900 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:25.896+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.897897+0000) 2019-10-13T09:57:25.901 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:25.897+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.898670+0000) 2019-10-13T09:57:25.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:25.897+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.898744+0000) 2019-10-13T09:57:25.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:25.897+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.898873+0000) 2019-10-13T09:57:25.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:25.897+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.899094+0000) 2019-10-13T09:57:25.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:25.897+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.899372+0000) 2019-10-13T09:57:25.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:25.900+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.900986+0000) 2019-10-13T09:57:25.907 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:25.904+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:28.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:28.511+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:28.511691+0000) 2019-10-13T09:57:28.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:28.849+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:28.850125+0000) 2019-10-13T09:57:29.338 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:57:29.338 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:29.342 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:57:29.342 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:30.908 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:30.905+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.907262+0000) 2019-10-13T09:57:30.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:30.906+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.907497+0000) 2019-10-13T09:57:30.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:30.906+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.907891+0000) 2019-10-13T09:57:30.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:30.907+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.908558+0000) 2019-10-13T09:57:30.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:30.907+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.908766+0000) 2019-10-13T09:57:30.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:30.909+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.910162+0000) 2019-10-13T09:57:30.911 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:30.907+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.909195+0000) 2019-10-13T09:57:30.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:30.911+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.912557+0000) 2019-10-13T09:57:30.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:30.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:32.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:32.953+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:32.954608+0000) 2019-10-13T09:57:32.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:32.953+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:32.953627+0000) 2019-10-13T09:57:34.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:34.144+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:34.144390+0000) 2019-10-13T09:57:34.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:34.711+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:34.712202+0000) 2019-10-13T09:57:35.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:35.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.601216+0000) 2019-10-13T09:57:35.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:35.600+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.601214+0000) 2019-10-13T09:57:35.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:35.916+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.916805+0000) 2019-10-13T09:57:35.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:35.916+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.917053+0000) 2019-10-13T09:57:35.919 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:35.916+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.917355+0000) 2019-10-13T09:57:35.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:35.917+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.918327+0000) 2019-10-13T09:57:35.920 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:35.917+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.917719+0000) 2019-10-13T09:57:35.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:35.917+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.918052+0000) 2019-10-13T09:57:35.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:35.918+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.918487+0000) 2019-10-13T09:57:35.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:35.919+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.920723+0000) 2019-10-13T09:57:35.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:35.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176860 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:38.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:38.511+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:38.511900+0000) 2019-10-13T09:57:38.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:38.849+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:38.850323+0000) 2019-10-13T09:57:40.927 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:40.925+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.925632+0000) 2019-10-13T09:57:40.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:40.925+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.925849+0000) 2019-10-13T09:57:40.927 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:40.925+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.926140+0000) 2019-10-13T09:57:40.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:40.927+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.927824+0000) 2019-10-13T09:57:40.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:40.925+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.926392+0000) 2019-10-13T09:57:40.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:40.926+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.926481+0000) 2019-10-13T09:57:40.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:40.926+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.926870+0000) 2019-10-13T09:57:40.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:40.928+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.929291+0000) 2019-10-13T09:57:40.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:40.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:42.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:42.954+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:42.954760+0000) 2019-10-13T09:57:42.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:42.953+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:42.953799+0000) 2019-10-13T09:57:44.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:44.146+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:44.146790+0000) 2019-10-13T09:57:44.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:44.712+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:44.712393+0000) 2019-10-13T09:57:45.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:45.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.601442+0000) 2019-10-13T09:57:45.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:45.601+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.601431+0000) 2019-10-13T09:57:45.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:45.932+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.933293+0000) 2019-10-13T09:57:45.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:45.933+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.933620+0000) 2019-10-13T09:57:45.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:45.933+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.933767+0000) 2019-10-13T09:57:45.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:45.933+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.935098+0000) 2019-10-13T09:57:45.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:45.933+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.934285+0000) 2019-10-13T09:57:45.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:45.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.934383+0000) 2019-10-13T09:57:45.937 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:45.934+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.934583+0000) 2019-10-13T09:57:45.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:45.935+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.937350+0000) 2019-10-13T09:57:45.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:45.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:48.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:48.511+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:48.512064+0000) 2019-10-13T09:57:48.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:48.850+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:48.850518+0000) 2019-10-13T09:57:50.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:50.942+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.943797+0000) 2019-10-13T09:57:50.945 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:50.942+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.942921+0000) 2019-10-13T09:57:50.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:50.942+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.942811+0000) 2019-10-13T09:57:50.945 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:50.942+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.943312+0000) 2019-10-13T09:57:50.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:50.943+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.943562+0000) 2019-10-13T09:57:50.946 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:50.943+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.944030+0000) 2019-10-13T09:57:50.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:50.943+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.944214+0000) 2019-10-13T09:57:50.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:50.944+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.946427+0000) 2019-10-13T09:57:50.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:50.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176952 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:52.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:52.954+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:52.954935+0000) 2019-10-13T09:57:52.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:52.952+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:52.954005+0000) 2019-10-13T09:57:54.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:54.148+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:54.149207+0000) 2019-10-13T09:57:54.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:54.711+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:54.712553+0000) 2019-10-13T09:57:55.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:55.600+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.601640+0000) 2019-10-13T09:57:55.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:55.600+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.601667+0000) 2019-10-13T09:57:55.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:55.950+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.951835+0000) 2019-10-13T09:57:55.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:55.950+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.951976+0000) 2019-10-13T09:57:55.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:57:55.950+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.952105+0000) 2019-10-13T09:57:55.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:57:55.950+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.952333+0000) 2019-10-13T09:57:55.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:57:55.950+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.952389+0000) 2019-10-13T09:57:55.954 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:57:55.952+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.953759+0000) 2019-10-13T09:57:55.955 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:57:55.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.952647+0000) 2019-10-13T09:57:55.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:57:55.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 176986 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:57:55.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:57:55.954+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.956244+0000) 2019-10-13T09:57:58.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:57:58.511+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:58.512274+0000) 2019-10-13T09:57:58.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:57:58.849+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:58.850718+0000) 2019-10-13T09:57:59.475 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:57:59.476 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:59.482 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:57:59.482 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:00.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:00.957+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.959090+0000) 2019-10-13T09:58:00.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:00.958+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.959408+0000) 2019-10-13T09:58:00.961 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:00.958+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.959558+0000) 2019-10-13T09:58:00.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:00.960+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.961571+0000) 2019-10-13T09:58:00.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:00.958+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.959775+0000) 2019-10-13T09:58:00.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:00.958+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.960072+0000) 2019-10-13T09:58:00.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:00.958+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.960193+0000) 2019-10-13T09:58:00.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:00.963+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.964663+0000) 2019-10-13T09:58:00.966 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:00.964+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:02.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:02.953+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:02.954168+0000) 2019-10-13T09:58:02.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:02.954+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:02.955114+0000) 2019-10-13T09:58:04.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:04.151+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:04.151681+0000) 2019-10-13T09:58:04.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:04.711+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:04.712746+0000) 2019-10-13T09:58:05.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:05.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.601838+0000) 2019-10-13T09:58:05.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:05.601+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.601875+0000) 2019-10-13T09:58:05.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:05.966+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.967564+0000) 2019-10-13T09:58:05.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:05.966+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.967944+0000) 2019-10-13T09:58:05.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:05.968+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.968963+0000) 2019-10-13T09:58:05.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:05.966+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.968169+0000) 2019-10-13T09:58:05.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:05.966+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.968302+0000) 2019-10-13T09:58:05.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:05.967+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.968531+0000) 2019-10-13T09:58:05.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:05.967+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.968614+0000) 2019-10-13T09:58:05.971 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:05.970+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.971587+0000) 2019-10-13T09:58:05.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:05.973+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:08.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:08.511+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:08.512482+0000) 2019-10-13T09:58:08.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:08.850+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:08.850904+0000) 2019-10-13T09:58:10.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:10.977+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.977953+0000) 2019-10-13T09:58:10.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:10.975+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.977126+0000) 2019-10-13T09:58:10.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:10.976+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.977519+0000) 2019-10-13T09:58:10.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:10.976+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.977610+0000) 2019-10-13T09:58:10.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:10.976+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.977833+0000) 2019-10-13T09:58:10.980 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:10.976+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.977933+0000) 2019-10-13T09:58:10.980 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:10.976+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.978343+0000) 2019-10-13T09:58:10.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:10.979+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.980266+0000) 2019-10-13T09:58:10.986 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:10.984+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:12.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:12.954+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:12.955296+0000) 2019-10-13T09:58:12.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:12.953+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:12.954356+0000) 2019-10-13T09:58:14.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:14.153+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:14.154008+0000) 2019-10-13T09:58:14.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:14.712+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:14.712915+0000) 2019-10-13T09:58:15.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:15.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.602028+0000) 2019-10-13T09:58:15.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:15.601+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.602063+0000) 2019-10-13T09:58:15.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:15.987+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.988337+0000) 2019-10-13T09:58:15.989 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:15.986+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.987445+0000) 2019-10-13T09:58:15.989 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:15.986+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.987560+0000) 2019-10-13T09:58:15.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:15.986+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.987703+0000) 2019-10-13T09:58:15.990 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:15.986+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.987893+0000) 2019-10-13T09:58:15.990 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:15.986+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.988115+0000) 2019-10-13T09:58:15.990 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:15.987+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.988431+0000) 2019-10-13T09:58:15.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:15.989+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.989882+0000) 2019-10-13T09:58:15.996 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:15.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:18.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:18.512+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:18.512666+0000) 2019-10-13T09:58:18.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:18.850+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:18.851071+0000) 2019-10-13T09:58:20.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:20.996+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.997175+0000) 2019-10-13T09:58:20.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:20.995+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.996354+0000) 2019-10-13T09:58:20.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:20.996+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.996536+0000) 2019-10-13T09:58:20.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:20.996+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.996783+0000) 2019-10-13T09:58:20.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:20.996+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.996947+0000) 2019-10-13T09:58:20.999 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:20.996+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.997310+0000) 2019-10-13T09:58:20.999 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:20.997+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.997487+0000) 2019-10-13T09:58:21.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:21.000+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:21.001469+0000) 2019-10-13T09:58:21.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:21.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:22.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:22.955+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:22.955469+0000) 2019-10-13T09:58:22.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:22.954+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:22.954546+0000) 2019-10-13T09:58:24.156 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:24.156+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:24.156319+0000) 2019-10-13T09:58:24.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:24.712+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:24.713125+0000) 2019-10-13T09:58:25.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:25.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.602221+0000) 2019-10-13T09:58:25.604 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:25.601+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.602224+0000) 2019-10-13T09:58:26.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:26.005+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.005752+0000) 2019-10-13T09:58:26.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:26.004+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.004843+0000) 2019-10-13T09:58:26.007 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:26.004+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.005097+0000) 2019-10-13T09:58:26.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:26.005+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.005630+0000) 2019-10-13T09:58:26.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:26.005+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.005994+0000) 2019-10-13T09:58:26.008 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:26.005+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.005901+0000) 2019-10-13T09:58:26.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:26.005+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.006160+0000) 2019-10-13T09:58:26.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:26.008+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:26.009360+0000) 2019-10-13T09:58:26.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:26.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177176 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:28.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:28.512+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:28.512859+0000) 2019-10-13T09:58:28.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:28.850+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:28.851215+0000) 2019-10-13T09:58:29.638 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:58:29.638 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:29.643 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:58:29.643 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:31.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:31.013+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.013888+0000) 2019-10-13T09:58:31.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:31.013+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.014117+0000) 2019-10-13T09:58:31.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:31.013+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.014315+0000) 2019-10-13T09:58:31.016 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:31.014+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.014454+0000) 2019-10-13T09:58:31.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:31.015+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.015977+0000) 2019-10-13T09:58:31.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:31.014+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.015185+0000) 2019-10-13T09:58:31.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:31.014+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.015340+0000) 2019-10-13T09:58:31.017 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:31.016+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.017184+0000) 2019-10-13T09:58:31.023 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:31.021+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177204 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:32.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:32.955+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:32.955612+0000) 2019-10-13T09:58:32.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:32.954+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:32.954675+0000) 2019-10-13T09:58:34.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:34.158+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:34.158632+0000) 2019-10-13T09:58:34.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:34.713+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:34.713271+0000) 2019-10-13T09:58:35.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:35.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.602454+0000) 2019-10-13T09:58:35.604 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:35.602+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.602454+0000) 2019-10-13T09:58:36.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:36.023+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.023667+0000) 2019-10-13T09:58:36.025 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:36.023+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.023799+0000) 2019-10-13T09:58:36.025 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:36.023+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.024020+0000) 2019-10-13T09:58:36.026 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:36.023+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.024301+0000) 2019-10-13T09:58:36.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:36.023+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.025272+0000) 2019-10-13T09:58:36.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:36.024+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.024521+0000) 2019-10-13T09:58:36.027 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:36.024+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.024736+0000) 2019-10-13T09:58:36.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:36.026+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:36.028479+0000) 2019-10-13T09:58:36.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:36.030+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177238 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:38.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:38.512+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:38.513041+0000) 2019-10-13T09:58:38.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:38.849+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:38.851404+0000) 2019-10-13T09:58:41.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:41.031+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.032884+0000) 2019-10-13T09:58:41.034 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:41.031+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.033042+0000) 2019-10-13T09:58:41.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:41.032+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.034485+0000) 2019-10-13T09:58:41.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:41.032+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.033449+0000) 2019-10-13T09:58:41.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:41.032+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.033854+0000) 2019-10-13T09:58:41.036 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:41.032+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.033919+0000) 2019-10-13T09:58:41.036 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:41.032+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.034101+0000) 2019-10-13T09:58:41.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:41.034+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.036538+0000) 2019-10-13T09:58:41.041 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:41.039+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177266 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:42.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:42.954+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:42.955781+0000) 2019-10-13T09:58:42.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:42.953+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:42.954864+0000) 2019-10-13T09:58:44.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:44.160+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:44.161067+0000) 2019-10-13T09:58:44.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:44.712+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:44.713478+0000) 2019-10-13T09:58:45.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:45.601+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.602637+0000) 2019-10-13T09:58:45.604 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:45.601+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.602637+0000) 2019-10-13T09:58:46.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:46.041+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.043599+0000) 2019-10-13T09:58:46.044 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:46.040+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.042246+0000) 2019-10-13T09:58:46.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:46.041+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.042639+0000) 2019-10-13T09:58:46.044 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:46.041+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.042883+0000) 2019-10-13T09:58:46.045 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:46.041+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.043123+0000) 2019-10-13T09:58:46.045 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:46.042+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.043393+0000) 2019-10-13T09:58:46.045 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:46.042+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.043559+0000) 2019-10-13T09:58:46.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:46.044+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:46.046428+0000) 2019-10-13T09:58:46.051 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:46.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177302 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:48.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:48.512+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:48.513230+0000) 2019-10-13T09:58:48.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:48.850+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:48.851567+0000) 2019-10-13T09:58:51.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:51.051+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.052672+0000) 2019-10-13T09:58:51.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:51.050+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.051826+0000) 2019-10-13T09:58:51.053 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:51.050+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.052167+0000) 2019-10-13T09:58:51.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:51.050+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.052335+0000) 2019-10-13T09:58:51.054 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:51.051+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.052703+0000) 2019-10-13T09:58:51.054 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:51.051+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.052661+0000) 2019-10-13T09:58:51.055 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:51.051+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.053026+0000) 2019-10-13T09:58:51.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:51.055+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.056301+0000) 2019-10-13T09:58:51.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:51.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177330 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:52.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:52.955+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:52.955963+0000) 2019-10-13T09:58:52.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:52.954+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:52.955025+0000) 2019-10-13T09:58:54.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:54.162+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:54.163342+0000) 2019-10-13T09:58:54.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:54.713+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:54.713690+0000) 2019-10-13T09:58:55.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:55.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.602828+0000) 2019-10-13T09:58:55.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:55.602+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.602826+0000) 2019-10-13T09:58:56.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:58:56.059+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.060926+0000) 2019-10-13T09:58:56.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:58:56.059+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.061131+0000) 2019-10-13T09:58:56.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:56.059+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.061209+0000) 2019-10-13T09:58:56.064 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:56.060+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.061870+0000) 2019-10-13T09:58:56.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:58:56.060+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.061969+0000) 2019-10-13T09:58:56.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:58:56.061+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.062711+0000) 2019-10-13T09:58:56.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:58:56.064+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.064940+0000) 2019-10-13T09:58:56.065 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:58:56.060+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:56.062085+0000) 2019-10-13T09:58:56.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:58:56.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177364 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:58:58.514 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:58:58.512+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:58.513446+0000) 2019-10-13T09:58:58.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:58:58.851+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:58.851741+0000) 2019-10-13T09:58:59.792 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:58:59.792 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:59.801 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:58:59.801 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:01.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:01.070+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.071173+0000) 2019-10-13T09:59:01.072 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:01.068+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.070227+0000) 2019-10-13T09:59:01.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:01.069+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.070814+0000) 2019-10-13T09:59:01.072 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:01.069+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.071038+0000) 2019-10-13T09:59:01.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:01.069+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.071305+0000) 2019-10-13T09:59:01.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:01.070+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.071564+0000) 2019-10-13T09:59:01.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:01.070+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.071885+0000) 2019-10-13T09:59:01.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:01.073+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.073865+0000) 2019-10-13T09:59:01.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:01.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177394 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:02.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:02.955+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:02.956129+0000) 2019-10-13T09:59:02.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:02.954+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:02.955219+0000) 2019-10-13T09:59:04.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:04.165+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:04.165816+0000) 2019-10-13T09:59:04.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:04.713+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:04.713882+0000) 2019-10-13T09:59:05.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:05.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.603038+0000) 2019-10-13T09:59:05.604 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:05.602+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.603037+0000) 2019-10-13T09:59:06.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:06.079+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.079537+0000) 2019-10-13T09:59:06.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:06.079+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.079773+0000) 2019-10-13T09:59:06.082 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:06.079+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.080142+0000) 2019-10-13T09:59:06.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:06.079+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.080296+0000) 2019-10-13T09:59:06.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:06.080+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.080471+0000) 2019-10-13T09:59:06.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:06.080+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.080714+0000) 2019-10-13T09:59:06.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:06.081+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.082097+0000) 2019-10-13T09:59:06.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:06.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:06.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:06.084+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:06.084750+0000) 2019-10-13T09:59:08.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:08.513+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:08.513632+0000) 2019-10-13T09:59:08.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:08.851+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:08.851908+0000) 2019-10-13T09:59:11.089 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:11.087+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.088283+0000) 2019-10-13T09:59:11.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:11.089+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.089935+0000) 2019-10-13T09:59:11.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:11.088+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.088896+0000) 2019-10-13T09:59:11.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:11.088+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.089063+0000) 2019-10-13T09:59:11.092 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:11.088+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.089214+0000) 2019-10-13T09:59:11.092 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:11.088+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.089374+0000) 2019-10-13T09:59:11.092 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:11.089+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.089637+0000) 2019-10-13T09:59:11.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:11.091+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.091910+0000) 2019-10-13T09:59:11.096 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:11.095+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:12.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:12.955+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:12.956288+0000) 2019-10-13T09:59:12.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:12.955+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:12.955415+0000) 2019-10-13T09:59:14.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:14.167+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:14.168157+0000) 2019-10-13T09:59:14.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:14.713+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:14.714061+0000) 2019-10-13T09:59:15.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:15.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.603234+0000) 2019-10-13T09:59:15.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:15.603+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.603234+0000) 2019-10-13T09:59:16.098 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:16.096+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.097102+0000) 2019-10-13T09:59:16.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:16.097+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.098428+0000) 2019-10-13T09:59:16.099 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:16.097+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.097591+0000) 2019-10-13T09:59:16.100 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:16.097+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.097796+0000) 2019-10-13T09:59:16.100 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:16.097+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.098182+0000) 2019-10-13T09:59:16.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:16.097+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.098288+0000) 2019-10-13T09:59:16.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:16.098+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.098512+0000) 2019-10-13T09:59:16.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:16.100+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:16.100740+0000) 2019-10-13T09:59:16.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:16.104+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177490 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:18.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:18.513+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:18.513820+0000) 2019-10-13T09:59:18.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:18.851+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:18.852116+0000) 2019-10-13T09:59:21.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:21.105+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.106311+0000) 2019-10-13T09:59:21.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:21.107+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.108155+0000) 2019-10-13T09:59:21.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:21.106+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.107016+0000) 2019-10-13T09:59:21.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:21.107+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.107451+0000) 2019-10-13T09:59:21.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:21.107+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.107534+0000) 2019-10-13T09:59:21.109 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:21.107+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.107719+0000) 2019-10-13T09:59:21.109 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:21.107+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.107899+0000) 2019-10-13T09:59:21.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:21.109+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.110603+0000) 2019-10-13T09:59:21.117 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:21.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177520 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:22.956 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:22.955+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:22.956482+0000) 2019-10-13T09:59:22.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:22.955+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:22.955591+0000) 2019-10-13T09:59:24.170 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:24.170+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:24.170411+0000) 2019-10-13T09:59:24.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:24.713+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:24.714248+0000) 2019-10-13T09:59:25.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:25.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.603430+0000) 2019-10-13T09:59:25.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:25.602+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.603431+0000) 2019-10-13T09:59:26.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:26.115+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.116610+0000) 2019-10-13T09:59:26.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:26.117+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.118077+0000) 2019-10-13T09:59:26.119 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:26.115+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.117082+0000) 2019-10-13T09:59:26.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:26.116+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.117610+0000) 2019-10-13T09:59:26.119 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:26.116+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.117869+0000) 2019-10-13T09:59:26.120 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:26.116+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.118081+0000) 2019-10-13T09:59:26.120 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:26.116+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.118327+0000) 2019-10-13T09:59:26.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:26.119+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:26.120112+0000) 2019-10-13T09:59:26.124 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:26.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:28.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:28.513+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:28.514005+0000) 2019-10-13T09:59:28.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:28.850+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:28.852246+0000) 2019-10-13T09:59:29.938 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T09:59:29.939 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:29.943 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T09:59:29.943 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:31.126 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:31.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.125370+0000) 2019-10-13T09:59:31.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:31.124+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.125807+0000) 2019-10-13T09:59:31.127 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:31.124+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.125732+0000) 2019-10-13T09:59:31.127 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:31.124+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.126096+0000) 2019-10-13T09:59:31.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:31.125+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.127308+0000) 2019-10-13T09:59:31.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:31.125+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.126629+0000) 2019-10-13T09:59:31.128 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:31.125+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.126988+0000) 2019-10-13T09:59:31.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:31.127+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.129272+0000) 2019-10-13T09:59:31.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:31.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177582 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:32.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:32.955+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:32.956668+0000) 2019-10-13T09:59:32.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:32.954+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:32.955756+0000) 2019-10-13T09:59:34.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:34.170+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:34.172712+0000) 2019-10-13T09:59:34.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:34.713+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:34.714438+0000) 2019-10-13T09:59:35.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:35.602+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.603656+0000) 2019-10-13T09:59:35.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:35.602+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.603653+0000) 2019-10-13T09:59:36.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:36.132+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.134225+0000) 2019-10-13T09:59:36.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:36.133+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.134448+0000) 2019-10-13T09:59:36.136 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:36.133+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.134663+0000) 2019-10-13T09:59:36.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:36.134+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.135962+0000) 2019-10-13T09:59:36.137 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:36.133+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.135002+0000) 2019-10-13T09:59:36.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:36.133+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.135318+0000) 2019-10-13T09:59:36.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:36.134+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.135497+0000) 2019-10-13T09:59:36.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:36.136+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:36.138408+0000) 2019-10-13T09:59:36.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:36.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177616 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:38.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:38.513+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:38.514115+0000) 2019-10-13T09:59:38.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:38.851+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:38.852429+0000) 2019-10-13T09:59:41.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:41.141+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.143221+0000) 2019-10-13T09:59:41.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:41.142+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.143499+0000) 2019-10-13T09:59:41.146 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:41.144+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.144950+0000) 2019-10-13T09:59:41.146 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:41.142+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.143981+0000) 2019-10-13T09:59:41.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:41.142+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.144134+0000) 2019-10-13T09:59:41.146 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:41.142+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.144334+0000) 2019-10-13T09:59:41.146 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:41.143+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.144870+0000) 2019-10-13T09:59:41.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:41.145+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.146049+0000) 2019-10-13T09:59:41.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:41.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:42.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:42.956+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:42.956822+0000) 2019-10-13T09:59:42.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:42.955+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:42.955932+0000) 2019-10-13T09:59:44.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:44.174+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:44.175132+0000) 2019-10-13T09:59:44.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:44.714+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:44.714631+0000) 2019-10-13T09:59:45.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:45.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.603874+0000) 2019-10-13T09:59:45.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:45.603+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.603874+0000) 2019-10-13T09:59:46.155 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:46.151+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.152445+0000) 2019-10-13T09:59:46.155 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:46.151+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.152840+0000) 2019-10-13T09:59:46.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:46.151+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.152965+0000) 2019-10-13T09:59:46.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:46.151+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.153141+0000) 2019-10-13T09:59:46.157 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:46.153+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.154578+0000) 2019-10-13T09:59:46.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:46.155+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.156023+0000) 2019-10-13T09:59:46.157 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:46.152+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.153466+0000) 2019-10-13T09:59:46.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:46.152+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:46.153678+0000) 2019-10-13T09:59:46.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:46.158+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:48.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:48.513+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:48.514231+0000) 2019-10-13T09:59:48.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:48.852+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:48.852607+0000) 2019-10-13T09:59:51.163 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:51.161+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.161610+0000) 2019-10-13T09:59:51.163 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:51.161+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.161885+0000) 2019-10-13T09:59:51.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:51.161+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.162112+0000) 2019-10-13T09:59:51.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:51.163+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.163829+0000) 2019-10-13T09:59:51.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:51.162+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.162461+0000) 2019-10-13T09:59:51.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:51.162+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.162814+0000) 2019-10-13T09:59:51.165 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:51.162+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.163181+0000) 2019-10-13T09:59:51.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:51.163+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.164605+0000) 2019-10-13T09:59:51.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:51.168+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177708 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:52.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:52.956+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:52.956970+0000) 2019-10-13T09:59:52.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:52.955+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:52.956086+0000) 2019-10-13T09:59:54.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:54.176+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:54.177248+0000) 2019-10-13T09:59:54.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:54.714+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:54.714839+0000) 2019-10-13T09:59:55.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:55.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.604072+0000) 2019-10-13T09:59:55.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:55.603+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.604069+0000) 2019-10-13T09:59:56.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T09:59:56.170+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.171089+0000) 2019-10-13T09:59:56.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T09:59:56.170+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.171303+0000) 2019-10-13T09:59:56.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:56.170+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.171297+0000) 2019-10-13T09:59:56.173 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T09:59:56.171+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.171631+0000) 2019-10-13T09:59:56.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T09:59:56.171+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.172603+0000) 2019-10-13T09:59:56.174 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T09:59:56.171+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.171889+0000) 2019-10-13T09:59:56.174 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:56.171+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.171994+0000) 2019-10-13T09:59:56.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T09:59:56.175+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:56.176242+0000) 2019-10-13T09:59:56.180 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T09:59:56.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T09:59:58.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T09:59:58.514+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:58.514408+0000) 2019-10-13T09:59:58.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T09:59:58.852+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:58.852801+0000) 2019-10-13T10:00:00.043 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:00:00.043 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:00.047 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:00:00.047 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:01.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:01.180+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.180939+0000) 2019-10-13T10:00:01.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:01.180+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.181084+0000) 2019-10-13T10:00:01.183 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:01.180+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.181299+0000) 2019-10-13T10:00:01.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:01.181+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.182522+0000) 2019-10-13T10:00:01.184 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:01.181+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.181591+0000) 2019-10-13T10:00:01.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:01.181+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.181765+0000) 2019-10-13T10:00:01.184 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:01.181+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.182120+0000) 2019-10-13T10:00:01.185 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:01.183+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.183859+0000) 2019-10-13T10:00:01.189 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:01.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177772 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:02.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:02.956+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:02.957094+0000) 2019-10-13T10:00:02.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:02.955+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:02.956263+0000) 2019-10-13T10:00:04.179 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:04.179+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:04.179629+0000) 2019-10-13T10:00:04.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:04.714+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:04.715031+0000) 2019-10-13T10:00:05.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:05.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.604245+0000) 2019-10-13T10:00:05.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:05.604+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.604244+0000) 2019-10-13T10:00:06.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:06.189+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.189896+0000) 2019-10-13T10:00:06.191 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:06.189+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.190048+0000) 2019-10-13T10:00:06.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:06.189+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.190284+0000) 2019-10-13T10:00:06.192 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:06.191+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.192176+0000) 2019-10-13T10:00:06.193 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:06.190+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.190535+0000) 2019-10-13T10:00:06.193 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:06.190+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.190657+0000) 2019-10-13T10:00:06.193 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:06.190+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.190833+0000) 2019-10-13T10:00:06.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:06.193+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:06.194592+0000) 2019-10-13T10:00:06.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:06.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:08.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:08.514+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:08.514595+0000) 2019-10-13T10:00:08.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:08.852+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:08.852972+0000) 2019-10-13T10:00:11.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:11.196+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.198264+0000) 2019-10-13T10:00:11.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:11.197+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.198490+0000) 2019-10-13T10:00:11.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:11.197+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.198681+0000) 2019-10-13T10:00:11.201 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:11.199+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.200384+0000) 2019-10-13T10:00:11.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:11.197+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.198947+0000) 2019-10-13T10:00:11.201 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:11.197+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.199372+0000) 2019-10-13T10:00:11.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:11.198+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.199545+0000) 2019-10-13T10:00:11.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:11.200+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.201277+0000) 2019-10-13T10:00:11.207 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:11.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177834 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:12.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:12.957+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:12.957276+0000) 2019-10-13T10:00:12.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:12.955+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:12.956444+0000) 2019-10-13T10:00:14.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:14.182+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:14.182146+0000) 2019-10-13T10:00:14.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:14.714+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:14.715229+0000) 2019-10-13T10:00:15.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:15.603+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.604445+0000) 2019-10-13T10:00:15.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:15.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.604446+0000) 2019-10-13T10:00:16.209 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:16.206+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.208348+0000) 2019-10-13T10:00:16.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:16.207+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.208557+0000) 2019-10-13T10:00:16.210 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:16.207+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.208720+0000) 2019-10-13T10:00:16.211 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:16.209+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.210373+0000) 2019-10-13T10:00:16.211 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:16.207+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.209116+0000) 2019-10-13T10:00:16.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:16.208+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.209574+0000) 2019-10-13T10:00:16.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:16.208+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.209947+0000) 2019-10-13T10:00:16.212 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:16.211+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:16.212437+0000) 2019-10-13T10:00:16.217 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:16.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177868 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:18.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:18.513+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:18.514797+0000) 2019-10-13T10:00:18.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:18.852+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:18.853151+0000) 2019-10-13T10:00:21.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:21.216+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.217855+0000) 2019-10-13T10:00:21.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:21.216+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.218165+0000) 2019-10-13T10:00:21.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:21.217+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.219143+0000) 2019-10-13T10:00:21.220 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:21.217+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.218543+0000) 2019-10-13T10:00:21.220 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:21.217+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.218863+0000) 2019-10-13T10:00:21.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:21.217+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.218980+0000) 2019-10-13T10:00:21.221 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:21.217+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.219083+0000) 2019-10-13T10:00:21.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:21.218+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.220610+0000) 2019-10-13T10:00:21.226 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:21.224+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:22.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:22.956+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:22.957460+0000) 2019-10-13T10:00:22.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:22.955+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:22.956658+0000) 2019-10-13T10:00:24.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:24.183+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:24.184665+0000) 2019-10-13T10:00:24.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:24.714+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:24.715445+0000) 2019-10-13T10:00:25.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:25.603+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.604628+0000) 2019-10-13T10:00:25.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:25.603+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.604627+0000) 2019-10-13T10:00:26.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:26.225+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.227014+0000) 2019-10-13T10:00:26.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:26.225+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.227255+0000) 2019-10-13T10:00:26.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:26.225+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.227310+0000) 2019-10-13T10:00:26.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:26.226+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.227626+0000) 2019-10-13T10:00:26.229 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:26.226+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.227751+0000) 2019-10-13T10:00:26.230 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:26.227+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.229368+0000) 2019-10-13T10:00:26.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:26.228+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.230078+0000) 2019-10-13T10:00:26.231 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:26.226+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:26.228350+0000) 2019-10-13T10:00:26.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:26.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:28.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:28.514+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:28.514969+0000) 2019-10-13T10:00:28.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:28.852+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:28.853340+0000) 2019-10-13T10:00:30.130 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:00:30.130 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:30.136 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:00:30.136 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:31.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:31.233+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.235134+0000) 2019-10-13T10:00:31.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:31.233+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.235339+0000) 2019-10-13T10:00:31.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:31.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.237331+0000) 2019-10-13T10:00:31.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:31.234+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.235603+0000) 2019-10-13T10:00:31.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:31.234+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.235767+0000) 2019-10-13T10:00:31.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:31.234+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.236357+0000) 2019-10-13T10:00:31.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:31.235+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.236519+0000) 2019-10-13T10:00:31.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:31.237+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.238538+0000) 2019-10-13T10:00:31.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:31.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177960 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:32.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:32.956+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:32.957614+0000) 2019-10-13T10:00:32.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:32.956+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:32.956819+0000) 2019-10-13T10:00:34.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:34.186+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:34.187037+0000) 2019-10-13T10:00:34.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:34.715+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:34.715644+0000) 2019-10-13T10:00:35.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:35.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.604793+0000) 2019-10-13T10:00:35.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:35.604+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.604792+0000) 2019-10-13T10:00:36.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:36.242+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.243740+0000) 2019-10-13T10:00:36.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:36.242+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.244301+0000) 2019-10-13T10:00:36.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:36.243+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.244524+0000) 2019-10-13T10:00:36.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:36.242+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.244117+0000) 2019-10-13T10:00:36.246 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:36.243+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.244877+0000) 2019-10-13T10:00:36.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:36.243+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.245049+0000) 2019-10-13T10:00:36.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:36.245+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.245796+0000) 2019-10-13T10:00:36.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:36.247+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:36.248077+0000) 2019-10-13T10:00:36.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:36.248+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 177994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:38.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:38.514+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:38.515162+0000) 2019-10-13T10:00:38.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:38.853+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:38.853525+0000) 2019-10-13T10:00:41.253 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:41.251+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.252115+0000) 2019-10-13T10:00:41.253 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:41.252+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.253411+0000) 2019-10-13T10:00:41.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:41.252+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.252950+0000) 2019-10-13T10:00:41.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:41.252+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.253128+0000) 2019-10-13T10:00:41.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:41.252+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.253327+0000) 2019-10-13T10:00:41.255 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:41.253+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.253440+0000) 2019-10-13T10:00:41.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:41.253+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.253963+0000) 2019-10-13T10:00:41.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:41.256+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.257132+0000) 2019-10-13T10:00:41.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:41.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178024 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:42.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:42.957+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:42.957766+0000) 2019-10-13T10:00:42.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:42.956+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:42.957003+0000) 2019-10-13T10:00:44.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:44.188+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:44.189380+0000) 2019-10-13T10:00:44.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:44.715+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:44.715856+0000) 2019-10-13T10:00:45.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:45.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.604982+0000) 2019-10-13T10:00:45.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:45.604+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.604982+0000) 2019-10-13T10:00:46.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:46.260+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.260485+0000) 2019-10-13T10:00:46.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:46.261+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.261814+0000) 2019-10-13T10:00:46.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:46.260+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.260939+0000) 2019-10-13T10:00:46.263 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:46.260+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.261058+0000) 2019-10-13T10:00:46.263 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:46.261+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.261394+0000) 2019-10-13T10:00:46.263 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:46.261+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.261495+0000) 2019-10-13T10:00:46.263 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:46.261+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.261648+0000) 2019-10-13T10:00:46.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:46.263+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:46.264625+0000) 2019-10-13T10:00:46.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:46.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178058 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:48.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:48.515+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:48.515353+0000) 2019-10-13T10:00:48.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:48.853+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:48.853710+0000) 2019-10-13T10:00:51.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:51.268+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.269369+0000) 2019-10-13T10:00:51.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:51.268+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.268615+0000) 2019-10-13T10:00:51.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:51.268+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.268697+0000) 2019-10-13T10:00:51.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:51.268+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.268845+0000) 2019-10-13T10:00:51.271 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:51.268+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.269082+0000) 2019-10-13T10:00:51.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:51.269+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.269541+0000) 2019-10-13T10:00:51.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:51.269+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.269698+0000) 2019-10-13T10:00:51.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:51.271+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.272656+0000) 2019-10-13T10:00:51.277 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:51.275+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178086 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:52.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:52.957+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:52.957914+0000) 2019-10-13T10:00:52.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:52.956+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:52.957145+0000) 2019-10-13T10:00:54.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:54.191+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:54.191714+0000) 2019-10-13T10:00:54.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:54.715+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:54.716043+0000) 2019-10-13T10:00:55.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:55.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.605193+0000) 2019-10-13T10:00:55.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:55.604+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.605193+0000) 2019-10-13T10:00:56.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:00:56.276+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.278100+0000) 2019-10-13T10:00:56.279 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:00:56.277+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.278456+0000) 2019-10-13T10:00:56.280 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:00:56.278+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.279399+0000) 2019-10-13T10:00:56.280 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:00:56.277+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.278663+0000) 2019-10-13T10:00:56.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:00:56.277+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.278763+0000) 2019-10-13T10:00:56.281 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:56.277+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.278949+0000) 2019-10-13T10:00:56.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:56.277+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.278575+0000) 2019-10-13T10:00:56.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:00:56.280+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:56.281015+0000) 2019-10-13T10:00:56.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:00:56.284+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178120 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:00:58.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:00:58.514+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:58.515567+0000) 2019-10-13T10:00:58.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:00:58.852+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:58.853884+0000) 2019-10-13T10:01:00.260 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:01:00.261 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:00.265 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:01:00.265 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:01.288 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:01.285+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.287305+0000) 2019-10-13T10:01:01.290 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:01.288+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.289002+0000) 2019-10-13T10:01:01.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:01.286+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.287905+0000) 2019-10-13T10:01:01.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:01.286+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.288165+0000) 2019-10-13T10:01:01.290 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:01.286+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.288023+0000) 2019-10-13T10:01:01.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:01.287+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.288452+0000) 2019-10-13T10:01:01.291 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:01.286+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.288242+0000) 2019-10-13T10:01:01.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:01.290+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.290974+0000) 2019-10-13T10:01:01.293 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:01.291+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178151 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:02.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:02.957+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:02.958094+0000) 2019-10-13T10:01:02.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:02.956+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:02.957308+0000) 2019-10-13T10:01:04.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:04.193+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:04.194000+0000) 2019-10-13T10:01:04.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:04.715+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:04.716206+0000) 2019-10-13T10:01:05.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:05.604+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.605381+0000) 2019-10-13T10:01:05.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:05.604+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.605381+0000) 2019-10-13T10:01:06.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:06.293+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.294639+0000) 2019-10-13T10:01:06.296 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:06.293+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.294890+0000) 2019-10-13T10:01:06.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:06.293+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.295020+0000) 2019-10-13T10:01:06.297 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:06.293+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.295261+0000) 2019-10-13T10:01:06.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:06.295+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.296299+0000) 2019-10-13T10:01:06.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:06.294+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.295466+0000) 2019-10-13T10:01:06.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:06.294+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.295723+0000) 2019-10-13T10:01:06.299 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:06.298+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:06.299228+0000) 2019-10-13T10:01:06.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:06.300+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:08.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:08.515+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:08.515757+0000) 2019-10-13T10:01:08.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:08.852+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:08.854082+0000) 2019-10-13T10:01:11.305 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:11.302+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.303794+0000) 2019-10-13T10:01:11.305 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:11.302+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.304046+0000) 2019-10-13T10:01:11.306 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:11.303+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.305431+0000) 2019-10-13T10:01:11.306 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:11.303+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.304632+0000) 2019-10-13T10:01:11.306 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:11.303+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.304424+0000) 2019-10-13T10:01:11.307 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:11.303+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.305065+0000) 2019-10-13T10:01:11.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:11.303+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.305263+0000) 2019-10-13T10:01:11.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:11.305+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.307398+0000) 2019-10-13T10:01:11.312 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:11.309+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178213 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:12.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:12.957+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:12.958305+0000) 2019-10-13T10:01:12.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:12.956+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:12.957525+0000) 2019-10-13T10:01:14.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:14.194+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:14.196472+0000) 2019-10-13T10:01:14.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:14.715+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:14.716361+0000) 2019-10-13T10:01:15.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:15.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.605599+0000) 2019-10-13T10:01:15.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:15.605+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.605603+0000) 2019-10-13T10:01:16.313 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:16.312+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.313757+0000) 2019-10-13T10:01:16.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:16.311+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.312772+0000) 2019-10-13T10:01:16.315 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:16.311+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.312989+0000) 2019-10-13T10:01:16.315 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:16.311+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.313267+0000) 2019-10-13T10:01:16.315 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:16.311+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.313364+0000) 2019-10-13T10:01:16.315 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:16.312+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.313742+0000) 2019-10-13T10:01:16.315 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:16.312+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.313917+0000) 2019-10-13T10:01:16.316 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:16.315+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:16.316801+0000) 2019-10-13T10:01:16.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:16.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178246 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:18.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:18.515+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:18.515909+0000) 2019-10-13T10:01:18.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:18.853+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:18.854266+0000) 2019-10-13T10:01:21.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:21.319+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.321061+0000) 2019-10-13T10:01:21.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:21.319+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.321322+0000) 2019-10-13T10:01:21.323 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:21.320+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.321477+0000) 2019-10-13T10:01:21.324 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:21.321+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.322761+0000) 2019-10-13T10:01:21.324 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:21.320+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.321872+0000) 2019-10-13T10:01:21.324 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:21.320+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.322012+0000) 2019-10-13T10:01:21.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:21.320+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.322134+0000) 2019-10-13T10:01:21.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:21.322+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.323939+0000) 2019-10-13T10:01:21.328 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:21.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:22.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:22.957+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:22.958484+0000) 2019-10-13T10:01:22.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:22.957+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:22.957627+0000) 2019-10-13T10:01:24.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:24.198+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:24.198780+0000) 2019-10-13T10:01:24.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:24.716+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:24.716527+0000) 2019-10-13T10:01:25.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:25.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.605772+0000) 2019-10-13T10:01:25.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:25.605+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.605813+0000) 2019-10-13T10:01:26.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:26.328+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.329000+0000) 2019-10-13T10:01:26.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:26.328+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.329234+0000) 2019-10-13T10:01:26.331 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:26.330+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.331143+0000) 2019-10-13T10:01:26.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:26.329+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.329564+0000) 2019-10-13T10:01:26.332 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:26.329+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.329690+0000) 2019-10-13T10:01:26.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:26.329+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.329846+0000) 2019-10-13T10:01:26.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:26.329+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.330137+0000) 2019-10-13T10:01:26.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:26.332+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:26.333564+0000) 2019-10-13T10:01:26.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:26.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:28.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:28.515+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:28.516080+0000) 2019-10-13T10:01:28.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:28.854+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:28.854424+0000) 2019-10-13T10:01:30.366 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:01:30.367 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:30.371 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:01:30.371 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:31.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:31.337+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.337873+0000) 2019-10-13T10:01:31.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:31.337+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.338188+0000) 2019-10-13T10:01:31.340 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:31.338+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.338475+0000) 2019-10-13T10:01:31.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:31.338+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.339159+0000) 2019-10-13T10:01:31.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:31.339+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.339394+0000) 2019-10-13T10:01:31.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:31.339+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.339531+0000) 2019-10-13T10:01:31.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:31.340+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.340936+0000) 2019-10-13T10:01:31.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:31.342+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.343014+0000) 2019-10-13T10:01:31.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:31.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:32.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:32.958+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:32.958638+0000) 2019-10-13T10:01:32.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:32.957+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:32.957758+0000) 2019-10-13T10:01:34.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:34.200+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:34.201134+0000) 2019-10-13T10:01:34.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:34.716+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:34.716694+0000) 2019-10-13T10:01:35.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:35.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.605954+0000) 2019-10-13T10:01:35.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:35.605+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.605975+0000) 2019-10-13T10:01:36.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:36.347+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.348008+0000) 2019-10-13T10:01:36.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:36.347+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.348298+0000) 2019-10-13T10:01:36.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:36.348+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.348871+0000) 2019-10-13T10:01:36.350 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:36.349+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.350236+0000) 2019-10-13T10:01:36.351 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:36.348+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.349213+0000) 2019-10-13T10:01:36.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:36.349+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.349494+0000) 2019-10-13T10:01:36.351 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:36.349+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.349840+0000) 2019-10-13T10:01:36.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:36.351+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:36.351790+0000) 2019-10-13T10:01:36.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:36.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:38.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:38.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:38.516289+0000) 2019-10-13T10:01:38.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:38.854+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:38.854590+0000) 2019-10-13T10:01:41.359 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:41.358+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.359018+0000) 2019-10-13T10:01:41.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:41.356+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.357996+0000) 2019-10-13T10:01:41.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:41.357+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.358422+0000) 2019-10-13T10:01:41.360 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:41.357+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.358467+0000) 2019-10-13T10:01:41.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:41.357+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.358656+0000) 2019-10-13T10:01:41.361 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:41.357+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.358802+0000) 2019-10-13T10:01:41.361 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:41.357+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.358964+0000) 2019-10-13T10:01:41.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:41.360+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.361435+0000) 2019-10-13T10:01:41.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:41.363+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:42.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:42.958+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:42.958821+0000) 2019-10-13T10:01:42.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:42.956+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:42.957906+0000) 2019-10-13T10:01:44.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:44.203+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:44.203595+0000) 2019-10-13T10:01:44.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:44.715+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:44.716875+0000) 2019-10-13T10:01:45.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:45.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.606117+0000) 2019-10-13T10:01:45.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:45.605+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.606155+0000) 2019-10-13T10:01:46.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:46.365+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.367277+0000) 2019-10-13T10:01:46.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:46.366+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.367468+0000) 2019-10-13T10:01:46.369 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:46.366+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.367580+0000) 2019-10-13T10:01:46.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:46.366+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.367759+0000) 2019-10-13T10:01:46.370 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:46.366+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.368022+0000) 2019-10-13T10:01:46.370 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:46.368+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.369260+0000) 2019-10-13T10:01:46.370 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:46.367+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.368459+0000) 2019-10-13T10:01:46.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:46.370+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:46.371501+0000) 2019-10-13T10:01:46.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:46.372+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:48.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:48.515+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:48.516482+0000) 2019-10-13T10:01:48.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:48.853+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:48.854803+0000) 2019-10-13T10:01:51.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:51.374+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.375692+0000) 2019-10-13T10:01:51.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:51.374+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.375976+0000) 2019-10-13T10:01:51.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:51.374+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.376098+0000) 2019-10-13T10:01:51.378 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:51.376+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.377584+0000) 2019-10-13T10:01:51.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:51.374+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.376265+0000) 2019-10-13T10:01:51.379 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:51.375+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.376660+0000) 2019-10-13T10:01:51.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:51.375+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.376985+0000) 2019-10-13T10:01:51.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:51.377+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.378685+0000) 2019-10-13T10:01:51.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:51.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178466 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:52.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:52.958+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:52.959014+0000) 2019-10-13T10:01:52.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:52.957+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:52.958107+0000) 2019-10-13T10:01:54.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:54.205+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:54.205936+0000) 2019-10-13T10:01:54.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:54.716+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:54.717064+0000) 2019-10-13T10:01:55.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:55.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.606263+0000) 2019-10-13T10:01:55.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:55.605+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.606308+0000) 2019-10-13T10:01:56.386 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:01:56.383+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.384908+0000) 2019-10-13T10:01:56.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:56.383+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.385163+0000) 2019-10-13T10:01:56.387 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:01:56.385+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.386336+0000) 2019-10-13T10:01:56.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:56.384+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.385444+0000) 2019-10-13T10:01:56.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:01:56.384+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.385597+0000) 2019-10-13T10:01:56.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:01:56.384+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.385892+0000) 2019-10-13T10:01:56.388 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:01:56.384+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.385996+0000) 2019-10-13T10:01:56.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:01:56.389+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:56.389859+0000) 2019-10-13T10:01:56.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:01:56.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178498 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:01:58.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:01:58.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:58.516676+0000) 2019-10-13T10:01:58.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:01:58.854+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:58.854991+0000) 2019-10-13T10:02:00.469 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:02:00.469 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:00.474 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:02:00.474 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:01.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:01.392+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.393518+0000) 2019-10-13T10:02:01.395 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:01.392+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.393682+0000) 2019-10-13T10:02:01.395 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:01.392+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.393881+0000) 2019-10-13T10:02:01.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:01.393+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.394939+0000) 2019-10-13T10:02:01.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:01.392+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.394150+0000) 2019-10-13T10:02:01.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:01.393+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.394550+0000) 2019-10-13T10:02:01.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:01.393+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.394761+0000) 2019-10-13T10:02:01.398 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:01.397+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.398874+0000) 2019-10-13T10:02:01.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:01.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:02.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:02.958+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:02.959193+0000) 2019-10-13T10:02:02.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:02.957+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:02.958293+0000) 2019-10-13T10:02:04.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:04.207+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:04.208197+0000) 2019-10-13T10:02:04.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:04.716+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:04.717231+0000) 2019-10-13T10:02:05.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:05.605+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.606435+0000) 2019-10-13T10:02:05.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:05.605+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.606464+0000) 2019-10-13T10:02:06.402 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:06.400+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.402068+0000) 2019-10-13T10:02:06.403 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:06.399+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.401152+0000) 2019-10-13T10:02:06.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:06.400+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.401664+0000) 2019-10-13T10:02:06.403 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:06.400+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.401875+0000) 2019-10-13T10:02:06.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:06.400+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.402077+0000) 2019-10-13T10:02:06.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:06.400+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.402213+0000) 2019-10-13T10:02:06.404 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:06.401+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.402508+0000) 2019-10-13T10:02:06.404 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:06.402+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:06.404255+0000) 2019-10-13T10:02:06.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:06.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:08.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:08.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:08.516860+0000) 2019-10-13T10:02:08.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:08.854+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:08.855190+0000) 2019-10-13T10:02:11.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:11.409+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.409978+0000) 2019-10-13T10:02:11.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:11.409+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.410110+0000) 2019-10-13T10:02:11.413 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:11.410+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.410530+0000) 2019-10-13T10:02:11.413 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:11.410+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.410611+0000) 2019-10-13T10:02:11.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:11.410+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.410732+0000) 2019-10-13T10:02:11.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:11.410+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.411099+0000) 2019-10-13T10:02:11.414 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:11.410+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.412019+0000) 2019-10-13T10:02:11.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:11.411+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.413472+0000) 2019-10-13T10:02:11.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:11.414+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178592 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:12.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:12.958+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:12.959378+0000) 2019-10-13T10:02:12.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:12.958+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:12.958473+0000) 2019-10-13T10:02:14.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:14.209+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:14.210649+0000) 2019-10-13T10:02:14.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:14.717+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:14.717420+0000) 2019-10-13T10:02:15.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:15.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.606646+0000) 2019-10-13T10:02:15.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:15.606+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.606650+0000) 2019-10-13T10:02:16.418 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:16.416+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.417036+0000) 2019-10-13T10:02:16.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:16.416+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.417180+0000) 2019-10-13T10:02:16.419 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:16.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.418534+0000) 2019-10-13T10:02:16.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:16.417+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.417562+0000) 2019-10-13T10:02:16.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:16.417+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.417835+0000) 2019-10-13T10:02:16.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:16.417+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.417889+0000) 2019-10-13T10:02:16.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:16.417+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.418333+0000) 2019-10-13T10:02:16.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:16.420+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:16.421505+0000) 2019-10-13T10:02:16.425 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:16.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178624 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:18.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:18.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:18.517050+0000) 2019-10-13T10:02:18.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:18.855+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:18.855374+0000) 2019-10-13T10:02:21.427 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:21.426+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.426415+0000) 2019-10-13T10:02:21.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:21.426+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.426831+0000) 2019-10-13T10:02:21.429 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:21.427+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.428140+0000) 2019-10-13T10:02:21.429 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:21.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.427177+0000) 2019-10-13T10:02:21.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:21.426+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.427300+0000) 2019-10-13T10:02:21.429 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:21.427+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.427479+0000) 2019-10-13T10:02:21.430 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:21.427+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.427688+0000) 2019-10-13T10:02:21.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:21.429+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.429931+0000) 2019-10-13T10:02:21.435 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:21.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178656 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:22.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:22.958+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:22.959547+0000) 2019-10-13T10:02:22.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:22.958+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:22.958629+0000) 2019-10-13T10:02:24.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:24.212+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:24.213046+0000) 2019-10-13T10:02:24.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:24.717+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:24.717568+0000) 2019-10-13T10:02:25.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:25.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.606864+0000) 2019-10-13T10:02:25.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:25.606+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.606870+0000) 2019-10-13T10:02:26.437 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:26.434+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.435771+0000) 2019-10-13T10:02:26.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:26.436+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.437128+0000) 2019-10-13T10:02:26.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:26.434+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.436318+0000) 2019-10-13T10:02:26.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:26.435+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.436699+0000) 2019-10-13T10:02:26.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:26.435+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.436442+0000) 2019-10-13T10:02:26.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:26.435+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.436740+0000) 2019-10-13T10:02:26.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:26.435+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.437028+0000) 2019-10-13T10:02:26.440 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:26.439+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:26.440089+0000) 2019-10-13T10:02:26.444 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:26.442+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:28.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:28.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:28.517218+0000) 2019-10-13T10:02:28.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:28.854+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:28.855575+0000) 2019-10-13T10:02:30.612 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:02:30.612 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:30.617 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:02:30.617 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:31.446 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:31.443+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.444715+0000) 2019-10-13T10:02:31.446 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:31.443+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.445070+0000) 2019-10-13T10:02:31.447 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:31.443+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.444977+0000) 2019-10-13T10:02:31.447 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:31.443+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.445157+0000) 2019-10-13T10:02:31.447 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:31.444+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.445434+0000) 2019-10-13T10:02:31.448 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:31.446+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.446878+0000) 2019-10-13T10:02:31.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:31.444+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.445580+0000) 2019-10-13T10:02:31.449 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:31.448+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.449154+0000) 2019-10-13T10:02:31.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:31.449+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:32.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:32.959+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:32.959703+0000) 2019-10-13T10:02:32.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:32.957+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:32.958809+0000) 2019-10-13T10:02:34.215 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:34.215+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:34.215544+0000) 2019-10-13T10:02:34.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:34.716+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:34.717761+0000) 2019-10-13T10:02:35.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:35.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.607064+0000) 2019-10-13T10:02:35.608 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:35.606+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.607070+0000) 2019-10-13T10:02:36.454 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:36.453+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.454549+0000) 2019-10-13T10:02:36.455 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:36.452+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.453530+0000) 2019-10-13T10:02:36.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:36.452+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.453706+0000) 2019-10-13T10:02:36.456 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:36.452+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.453846+0000) 2019-10-13T10:02:36.456 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:36.452+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.454032+0000) 2019-10-13T10:02:36.456 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:36.452+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.454230+0000) 2019-10-13T10:02:36.456 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:36.453+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.454417+0000) 2019-10-13T10:02:36.457 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:36.455+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:36.456150+0000) 2019-10-13T10:02:36.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:36.458+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178750 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:38.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:38.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:38.517404+0000) 2019-10-13T10:02:38.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:38.855+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:38.855796+0000) 2019-10-13T10:02:41.462 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:41.459+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.461254+0000) 2019-10-13T10:02:41.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:41.460+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.461730+0000) 2019-10-13T10:02:41.463 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:41.460+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.462030+0000) 2019-10-13T10:02:41.464 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:41.462+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.463370+0000) 2019-10-13T10:02:41.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:41.460+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.462360+0000) 2019-10-13T10:02:41.464 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:41.461+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.462477+0000) 2019-10-13T10:02:41.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:41.461+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.462683+0000) 2019-10-13T10:02:41.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:41.464+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.465414+0000) 2019-10-13T10:02:41.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:41.467+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:42.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:42.959+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:42.959843+0000) 2019-10-13T10:02:42.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:42.957+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:42.958961+0000) 2019-10-13T10:02:44.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:44.217+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:44.217917+0000) 2019-10-13T10:02:44.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:44.717+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:44.717984+0000) 2019-10-13T10:02:45.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:45.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.607274+0000) 2019-10-13T10:02:45.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:45.606+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.607294+0000) 2019-10-13T10:02:46.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:46.470+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.471590+0000) 2019-10-13T10:02:46.472 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:46.469+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.470764+0000) 2019-10-13T10:02:46.473 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:46.469+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.470845+0000) 2019-10-13T10:02:46.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:46.469+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.470953+0000) 2019-10-13T10:02:46.473 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:46.469+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.471182+0000) 2019-10-13T10:02:46.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:46.470+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.471528+0000) 2019-10-13T10:02:46.473 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:46.470+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.471660+0000) 2019-10-13T10:02:46.474 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:46.473+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:46.474520+0000) 2019-10-13T10:02:46.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:46.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178814 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:48.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:48.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:48.517533+0000) 2019-10-13T10:02:48.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:48.855+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:48.855929+0000) 2019-10-13T10:02:51.481 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:51.478+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.479940+0000) 2019-10-13T10:02:51.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:51.478+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.480122+0000) 2019-10-13T10:02:51.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:51.478+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.480295+0000) 2019-10-13T10:02:51.482 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:51.479+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.480514+0000) 2019-10-13T10:02:51.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:51.480+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.481837+0000) 2019-10-13T10:02:51.483 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:51.479+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.480874+0000) 2019-10-13T10:02:51.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:51.479+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.481075+0000) 2019-10-13T10:02:51.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:51.482+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.484548+0000) 2019-10-13T10:02:51.488 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:51.485+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:52.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:52.959+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:52.960033+0000) 2019-10-13T10:02:52.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:52.958+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:52.959139+0000) 2019-10-13T10:02:54.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:54.219+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:54.220258+0000) 2019-10-13T10:02:54.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:54.717+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:54.718172+0000) 2019-10-13T10:02:55.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:55.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.607445+0000) 2019-10-13T10:02:55.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:55.607+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.607446+0000) 2019-10-13T10:02:56.490 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:02:56.488+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.488812+0000) 2019-10-13T10:02:56.490 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:56.488+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.489029+0000) 2019-10-13T10:02:56.490 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:02:56.488+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.489252+0000) 2019-10-13T10:02:56.491 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:02:56.489+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.491362+0000) 2019-10-13T10:02:56.492 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:56.489+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.489489+0000) 2019-10-13T10:02:56.492 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:02:56.489+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.489760+0000) 2019-10-13T10:02:56.492 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:02:56.489+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.489999+0000) 2019-10-13T10:02:56.493 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:02:56.491+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:56.493610+0000) 2019-10-13T10:02:56.497 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:02:56.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178876 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:02:58.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:02:58.517+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:58.517686+0000) 2019-10-13T10:02:58.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:02:58.855+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:58.856122+0000) 2019-10-13T10:03:00.702 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:03:00.702 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:00.706 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:03:00.706 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:01.499 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:01.497+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.499251+0000) 2019-10-13T10:03:01.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:01.497+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.498317+0000) 2019-10-13T10:03:01.500 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:01.498+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.498538+0000) 2019-10-13T10:03:01.500 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:01.498+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.498650+0000) 2019-10-13T10:03:01.500 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:01.498+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.498863+0000) 2019-10-13T10:03:01.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:01.498+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.498921+0000) 2019-10-13T10:03:01.501 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:01.498+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.499148+0000) 2019-10-13T10:03:01.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:01.501+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.503331+0000) 2019-10-13T10:03:01.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:01.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178908 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:02.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:02.959+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:02.960232+0000) 2019-10-13T10:03:02.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:02.959+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:02.959333+0000) 2019-10-13T10:03:04.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:04.221+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:04.222549+0000) 2019-10-13T10:03:04.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:04.718+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:04.718357+0000) 2019-10-13T10:03:05.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:05.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.607670+0000) 2019-10-13T10:03:05.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:05.607+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.607668+0000) 2019-10-13T10:03:06.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:06.506+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.507035+0000) 2019-10-13T10:03:06.508 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:06.506+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.507201+0000) 2019-10-13T10:03:06.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:06.507+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.507508+0000) 2019-10-13T10:03:06.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:06.508+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.509130+0000) 2019-10-13T10:03:06.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:06.508+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.509245+0000) 2019-10-13T10:03:06.511 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:06.510+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.510870+0000) 2019-10-13T10:03:06.511 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:06.509+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.509541+0000) 2019-10-13T10:03:06.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:06.511+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:06.512636+0000) 2019-10-13T10:03:06.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:06.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:08.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:08.517+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:08.517845+0000) 2019-10-13T10:03:08.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:08.856+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:08.856320+0000) 2019-10-13T10:03:11.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:11.513+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.515310+0000) 2019-10-13T10:03:11.517 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:11.515+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.516710+0000) 2019-10-13T10:03:11.517 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:11.514+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.515950+0000) 2019-10-13T10:03:11.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:11.514+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.516139+0000) 2019-10-13T10:03:11.518 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:11.514+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.516380+0000) 2019-10-13T10:03:11.518 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:11.515+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.516553+0000) 2019-10-13T10:03:11.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:11.515+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.517060+0000) 2019-10-13T10:03:11.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:11.518+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.519451+0000) 2019-10-13T10:03:11.524 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:11.521+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 178970 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:12.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:12.959+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:12.960412+0000) 2019-10-13T10:03:12.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:12.958+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:12.959526+0000) 2019-10-13T10:03:14.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:14.224+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:14.224873+0000) 2019-10-13T10:03:14.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:14.717+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:14.718555+0000) 2019-10-13T10:03:15.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:15.606+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.607882+0000) 2019-10-13T10:03:15.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:15.606+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.607878+0000) 2019-10-13T10:03:16.525 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:16.523+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.524573+0000) 2019-10-13T10:03:16.526 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:16.525+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.526018+0000) 2019-10-13T10:03:16.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:16.523+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.525236+0000) 2019-10-13T10:03:16.527 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:16.523+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.525370+0000) 2019-10-13T10:03:16.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:16.524+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.525790+0000) 2019-10-13T10:03:16.527 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:16.524+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.525865+0000) 2019-10-13T10:03:16.527 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:16.524+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.525478+0000) 2019-10-13T10:03:16.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:16.527+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:16.528083+0000) 2019-10-13T10:03:16.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:16.531+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:18.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:18.516+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:18.518026+0000) 2019-10-13T10:03:18.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:18.855+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:18.856512+0000) 2019-10-13T10:03:21.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:21.532+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.533731+0000) 2019-10-13T10:03:21.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:21.532+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.534017+0000) 2019-10-13T10:03:21.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:21.532+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.534102+0000) 2019-10-13T10:03:21.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:21.532+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.534239+0000) 2019-10-13T10:03:21.536 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:21.534+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.535319+0000) 2019-10-13T10:03:21.537 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:21.533+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.534726+0000) 2019-10-13T10:03:21.537 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:21.533+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.534862+0000) 2019-10-13T10:03:21.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:21.537+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.537899+0000) 2019-10-13T10:03:21.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:21.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:22.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:22.960+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:22.960591+0000) 2019-10-13T10:03:22.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:22.958+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:22.959700+0000) 2019-10-13T10:03:24.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:24.226+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:24.227255+0000) 2019-10-13T10:03:24.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:24.717+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:24.718748+0000) 2019-10-13T10:03:25.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:25.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.608104+0000) 2019-10-13T10:03:25.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:25.607+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.608078+0000) 2019-10-13T10:03:26.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:26.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.540798+0000) 2019-10-13T10:03:26.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:26.539+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.540860+0000) 2019-10-13T10:03:26.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:26.539+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.541245+0000) 2019-10-13T10:03:26.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:26.540+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.541539+0000) 2019-10-13T10:03:26.543 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:26.541+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.542148+0000) 2019-10-13T10:03:26.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:26.540+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.541827+0000) 2019-10-13T10:03:26.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:26.541+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.542422+0000) 2019-10-13T10:03:26.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:26.543+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:26.544675+0000) 2019-10-13T10:03:26.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:26.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:28.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:28.517+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:28.518200+0000) 2019-10-13T10:03:28.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:28.856+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:28.856720+0000) 2019-10-13T10:03:30.790 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:03:30.790 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:30.794 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:03:30.794 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:31.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:31.548+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.550052+0000) 2019-10-13T10:03:31.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:31.548+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.550243+0000) 2019-10-13T10:03:31.552 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:31.548+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.550386+0000) 2019-10-13T10:03:31.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:31.549+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.550522+0000) 2019-10-13T10:03:31.553 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:31.551+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.552536+0000) 2019-10-13T10:03:31.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:31.549+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.550795+0000) 2019-10-13T10:03:31.553 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:31.550+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.551477+0000) 2019-10-13T10:03:31.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:31.554+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.555006+0000) 2019-10-13T10:03:31.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:31.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:32.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:32.960+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:32.960782+0000) 2019-10-13T10:03:32.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:32.959+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:32.959891+0000) 2019-10-13T10:03:34.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:34.229+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:34.229659+0000) 2019-10-13T10:03:34.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:34.718+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:34.718940+0000) 2019-10-13T10:03:35.609 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:35.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.608316+0000) 2019-10-13T10:03:35.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:35.607+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.608314+0000) 2019-10-13T10:03:36.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:36.556+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.558044+0000) 2019-10-13T10:03:36.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:36.556+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.558346+0000) 2019-10-13T10:03:36.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:36.557+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.558490+0000) 2019-10-13T10:03:36.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:36.557+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.558693+0000) 2019-10-13T10:03:36.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:36.557+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.558956+0000) 2019-10-13T10:03:36.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:36.557+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.559220+0000) 2019-10-13T10:03:36.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:36.559+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.560683+0000) 2019-10-13T10:03:36.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:36.561+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:36.562390+0000) 2019-10-13T10:03:36.566 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:36.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:38.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:38.517+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:38.518393+0000) 2019-10-13T10:03:38.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:38.856+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:38.856868+0000) 2019-10-13T10:03:41.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:41.566+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.567098+0000) 2019-10-13T10:03:41.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:41.566+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.567320+0000) 2019-10-13T10:03:41.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:41.567+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.567393+0000) 2019-10-13T10:03:41.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:41.567+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.567618+0000) 2019-10-13T10:03:41.570 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:41.567+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.569387+0000) 2019-10-13T10:03:41.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:41.567+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.567833+0000) 2019-10-13T10:03:41.570 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:41.567+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.567968+0000) 2019-10-13T10:03:41.571 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:41.570+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.571751+0000) 2019-10-13T10:03:41.575 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:41.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179159 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:42.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:42.959+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:42.960983+0000) 2019-10-13T10:03:42.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:42.959+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:42.960077+0000) 2019-10-13T10:03:44.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:44.231+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:44.231996+0000) 2019-10-13T10:03:44.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:44.718+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:44.719131+0000) 2019-10-13T10:03:45.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:45.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.608510+0000) 2019-10-13T10:03:45.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:45.608+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.608511+0000) 2019-10-13T10:03:46.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:46.575+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.576266+0000) 2019-10-13T10:03:46.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:46.576+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.576528+0000) 2019-10-13T10:03:46.578 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:46.576+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.576820+0000) 2019-10-13T10:03:46.579 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:46.576+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.578198+0000) 2019-10-13T10:03:46.579 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:46.576+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.577200+0000) 2019-10-13T10:03:46.579 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:46.577+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.577496+0000) 2019-10-13T10:03:46.579 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:46.577+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.577684+0000) 2019-10-13T10:03:46.580 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:46.578+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:46.580624+0000) 2019-10-13T10:03:46.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:46.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179191 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:48.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:48.518+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:48.518600+0000) 2019-10-13T10:03:48.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:48.856+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:48.857056+0000) 2019-10-13T10:03:51.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:51.584+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.585172+0000) 2019-10-13T10:03:51.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:51.584+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.585384+0000) 2019-10-13T10:03:51.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:51.585+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.585582+0000) 2019-10-13T10:03:51.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:51.585+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.585860+0000) 2019-10-13T10:03:51.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:51.585+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.586312+0000) 2019-10-13T10:03:51.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:51.586+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.586503+0000) 2019-10-13T10:03:51.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:51.585+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.586763+0000) 2019-10-13T10:03:51.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:51.587+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.589212+0000) 2019-10-13T10:03:51.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:51.592+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179221 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:52.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:52.960+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:52.961156+0000) 2019-10-13T10:03:52.961 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:52.960+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:52.960214+0000) 2019-10-13T10:03:54.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:54.233+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:54.234477+0000) 2019-10-13T10:03:54.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:54.719+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:54.719321+0000) 2019-10-13T10:03:55.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:55.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.608698+0000) 2019-10-13T10:03:55.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:55.608+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.608696+0000) 2019-10-13T10:03:56.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:03:56.593+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.594326+0000) 2019-10-13T10:03:56.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:03:56.594+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.595574+0000) 2019-10-13T10:03:56.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:03:56.594+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.594764+0000) 2019-10-13T10:03:56.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:56.594+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.595161+0000) 2019-10-13T10:03:56.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:03:56.594+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.595357+0000) 2019-10-13T10:03:56.597 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:03:56.595+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.595670+0000) 2019-10-13T10:03:56.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:56.595+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.595816+0000) 2019-10-13T10:03:56.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:03:56.597+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:56.598190+0000) 2019-10-13T10:03:56.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:03:56.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179253 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:03:58.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:03:58.517+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:58.518793+0000) 2019-10-13T10:03:58.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:03:58.856+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:58.857241+0000) 2019-10-13T10:04:00.896 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:04:00.897 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:00.901 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:04:00.901 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:01.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:01.602+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.603413+0000) 2019-10-13T10:04:01.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:01.602+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.603982+0000) 2019-10-13T10:04:01.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:01.604+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.605324+0000) 2019-10-13T10:04:01.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:01.603+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.604459+0000) 2019-10-13T10:04:01.607 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:01.603+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.604621+0000) 2019-10-13T10:04:01.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:01.603+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.604765+0000) 2019-10-13T10:04:01.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:01.603+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.605168+0000) 2019-10-13T10:04:01.608 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:01.607+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.608272+0000) 2019-10-13T10:04:01.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:01.609+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:02.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:02.960+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:02.961335+0000) 2019-10-13T10:04:02.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:02.959+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:02.960412+0000) 2019-10-13T10:04:04.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:04.236+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:04.236805+0000) 2019-10-13T10:04:04.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:04.718+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:04.719488+0000) 2019-10-13T10:04:05.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:05.607+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.608874+0000) 2019-10-13T10:04:05.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:05.607+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.608863+0000) 2019-10-13T10:04:06.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:06.611+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.613049+0000) 2019-10-13T10:04:06.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:06.611+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.613270+0000) 2019-10-13T10:04:06.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:06.612+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.613675+0000) 2019-10-13T10:04:06.615 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:06.612+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.613837+0000) 2019-10-13T10:04:06.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:06.612+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.614357+0000) 2019-10-13T10:04:06.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:06.615+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.615746+0000) 2019-10-13T10:04:06.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:06.615+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.616657+0000) 2019-10-13T10:04:06.617 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:06.613+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:06.614791+0000) 2019-10-13T10:04:06.621 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:06.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:08.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:08.518+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:08.518983+0000) 2019-10-13T10:04:08.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:08.856+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:08.857339+0000) 2019-10-13T10:04:11.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:11.620+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.622020+0000) 2019-10-13T10:04:11.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:11.620+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.622247+0000) 2019-10-13T10:04:11.625 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:11.623+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.624321+0000) 2019-10-13T10:04:11.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:11.621+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.622574+0000) 2019-10-13T10:04:11.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:11.621+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.622693+0000) 2019-10-13T10:04:11.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:11.621+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.622963+0000) 2019-10-13T10:04:11.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:11.622+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.623498+0000) 2019-10-13T10:04:11.626 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:11.625+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.626223+0000) 2019-10-13T10:04:11.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:11.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:12.961 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:12.961+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:12.961497+0000) 2019-10-13T10:04:12.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:12.959+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:12.960582+0000) 2019-10-13T10:04:14.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:14.238+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:14.239227+0000) 2019-10-13T10:04:14.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:14.719+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:14.719668+0000) 2019-10-13T10:04:15.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:15.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.609056+0000) 2019-10-13T10:04:15.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:15.608+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.609057+0000) 2019-10-13T10:04:16.637 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:16.632+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.633202+0000) 2019-10-13T10:04:16.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:16.634+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.635582+0000) 2019-10-13T10:04:16.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:16.630+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.631795+0000) 2019-10-13T10:04:16.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:16.630+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.632155+0000) 2019-10-13T10:04:16.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:16.631+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.632458+0000) 2019-10-13T10:04:16.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:16.631+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.632647+0000) 2019-10-13T10:04:16.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:16.631+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.632842+0000) 2019-10-13T10:04:16.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:16.631+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:16.633046+0000) 2019-10-13T10:04:16.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:16.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179379 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:18.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:18.518+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:18.519172+0000) 2019-10-13T10:04:18.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:18.856+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:18.857487+0000) 2019-10-13T10:04:21.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:21.640+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.641741+0000) 2019-10-13T10:04:21.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:21.640+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.641868+0000) 2019-10-13T10:04:21.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:21.640+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.642039+0000) 2019-10-13T10:04:21.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:21.643+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.643929+0000) 2019-10-13T10:04:21.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:21.641+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.642777+0000) 2019-10-13T10:04:21.645 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:21.641+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.643066+0000) 2019-10-13T10:04:21.645 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:21.641+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.643166+0000) 2019-10-13T10:04:21.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:21.645+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.646263+0000) 2019-10-13T10:04:21.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:21.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:22.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:22.961+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:22.961667+0000) 2019-10-13T10:04:22.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:22.960+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:22.960757+0000) 2019-10-13T10:04:24.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:24.241+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:24.241587+0000) 2019-10-13T10:04:24.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:24.719+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:24.719816+0000) 2019-10-13T10:04:25.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:25.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.609275+0000) 2019-10-13T10:04:25.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:25.608+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.609275+0000) 2019-10-13T10:04:26.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:26.651+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.652271+0000) 2019-10-13T10:04:26.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:26.650+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.651303+0000) 2019-10-13T10:04:26.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:26.651+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.651409+0000) 2019-10-13T10:04:26.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:26.651+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.651736+0000) 2019-10-13T10:04:26.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:26.651+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.651954+0000) 2019-10-13T10:04:26.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:26.651+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.652192+0000) 2019-10-13T10:04:26.656 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:26.654+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.654792+0000) 2019-10-13T10:04:26.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:26.652+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:26.652663+0000) 2019-10-13T10:04:26.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:26.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179443 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:28.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:28.518+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:28.519317+0000) 2019-10-13T10:04:28.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:28.857+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:28.857669+0000) 2019-10-13T10:04:31.056 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:04:31.056 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:31.061 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:04:31.061 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:31.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:31.659+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.659529+0000) 2019-10-13T10:04:31.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:31.660+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.661339+0000) 2019-10-13T10:04:31.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:31.659+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.659637+0000) 2019-10-13T10:04:31.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:31.659+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.660328+0000) 2019-10-13T10:04:31.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:31.660+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.660439+0000) 2019-10-13T10:04:31.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:31.660+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.660578+0000) 2019-10-13T10:04:31.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:31.660+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.660832+0000) 2019-10-13T10:04:31.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:31.663+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.664112+0000) 2019-10-13T10:04:31.668 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:31.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179473 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:32.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:32.960+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:32.961847+0000) 2019-10-13T10:04:32.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:32.960+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:32.960946+0000) 2019-10-13T10:04:34.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:34.242+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:34.243855+0000) 2019-10-13T10:04:34.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:34.719+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:34.719997+0000) 2019-10-13T10:04:35.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:35.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.609469+0000) 2019-10-13T10:04:35.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:35.609+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.609470+0000) 2019-10-13T10:04:36.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:36.667+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.668387+0000) 2019-10-13T10:04:36.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:36.668+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.668836+0000) 2019-10-13T10:04:36.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:36.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.669842+0000) 2019-10-13T10:04:36.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:36.668+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.669105+0000) 2019-10-13T10:04:36.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:36.668+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.669036+0000) 2019-10-13T10:04:36.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:36.669+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.669548+0000) 2019-10-13T10:04:36.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:36.669+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.669814+0000) 2019-10-13T10:04:36.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:36.670+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:36.672610+0000) 2019-10-13T10:04:36.677 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:36.675+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:38.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:38.519+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:38.519463+0000) 2019-10-13T10:04:38.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:38.857+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:38.857818+0000) 2019-10-13T10:04:41.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:41.677+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.679323+0000) 2019-10-13T10:04:41.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:41.677+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.678149+0000) 2019-10-13T10:04:41.680 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:41.678+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.678452+0000) 2019-10-13T10:04:41.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:41.678+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.678684+0000) 2019-10-13T10:04:41.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:41.678+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.678531+0000) 2019-10-13T10:04:41.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:41.678+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.679011+0000) 2019-10-13T10:04:41.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:41.678+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.679159+0000) 2019-10-13T10:04:41.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:41.679+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.681252+0000) 2019-10-13T10:04:41.686 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:41.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179537 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:42.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:42.961+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:42.962026+0000) 2019-10-13T10:04:42.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:42.960+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:42.961120+0000) 2019-10-13T10:04:44.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:44.245+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:44.246252+0000) 2019-10-13T10:04:44.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:44.719+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:44.720182+0000) 2019-10-13T10:04:45.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:45.608+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.609661+0000) 2019-10-13T10:04:45.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:45.608+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.609662+0000) 2019-10-13T10:04:46.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:46.685+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.686782+0000) 2019-10-13T10:04:46.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:46.687+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.688264+0000) 2019-10-13T10:04:46.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:46.685+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.687377+0000) 2019-10-13T10:04:46.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:46.686+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.687555+0000) 2019-10-13T10:04:46.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:46.686+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.687843+0000) 2019-10-13T10:04:46.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:46.686+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.687908+0000) 2019-10-13T10:04:46.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:46.686+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.688112+0000) 2019-10-13T10:04:46.691 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:46.690+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:46.691340+0000) 2019-10-13T10:04:46.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:46.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179569 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:48.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:48.518+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:48.519614+0000) 2019-10-13T10:04:48.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:48.857+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:48.857969+0000) 2019-10-13T10:04:51.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:51.694+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.695982+0000) 2019-10-13T10:04:51.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:51.694+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.696326+0000) 2019-10-13T10:04:51.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:51.697+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.697912+0000) 2019-10-13T10:04:51.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:51.695+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.696901+0000) 2019-10-13T10:04:51.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:51.695+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.697057+0000) 2019-10-13T10:04:51.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:51.695+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.697245+0000) 2019-10-13T10:04:51.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:51.696+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.697451+0000) 2019-10-13T10:04:51.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:51.698+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.699674+0000) 2019-10-13T10:04:51.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:51.702+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179599 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:52.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:52.961+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:52.962198+0000) 2019-10-13T10:04:52.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:52.960+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:52.961310+0000) 2019-10-13T10:04:54.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:54.248+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:54.248636+0000) 2019-10-13T10:04:54.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:54.719+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:54.720358+0000) 2019-10-13T10:04:55.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:55.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.609841+0000) 2019-10-13T10:04:55.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:55.609+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.609835+0000) 2019-10-13T10:04:56.707 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:56.704+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.705719+0000) 2019-10-13T10:04:56.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:04:56.706+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.707441+0000) 2019-10-13T10:04:56.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:04:56.704+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.706088+0000) 2019-10-13T10:04:56.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:04:56.704+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.706377+0000) 2019-10-13T10:04:56.708 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:56.705+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.706581+0000) 2019-10-13T10:04:56.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:04:56.705+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.706743+0000) 2019-10-13T10:04:56.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:04:56.705+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.707025+0000) 2019-10-13T10:04:56.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:04:56.708+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:56.709308+0000) 2019-10-13T10:04:56.714 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:04:56.711+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:04:58.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:04:58.519+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:58.519775+0000) 2019-10-13T10:04:58.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:04:58.857+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:58.858111+0000) 2019-10-13T10:05:01.203 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:05:01.203 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:01.207 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:05:01.207 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:01.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:01.713+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.714612+0000) 2019-10-13T10:05:01.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:01.713+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.714915+0000) 2019-10-13T10:05:01.717 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:01.713+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.715145+0000) 2019-10-13T10:05:01.717 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:01.713+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.715225+0000) 2019-10-13T10:05:01.717 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:01.713+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.715379+0000) 2019-10-13T10:05:01.718 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:01.716+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.716724+0000) 2019-10-13T10:05:01.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:01.717+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.718048+0000) 2019-10-13T10:05:01.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:01.714+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.715888+0000) 2019-10-13T10:05:01.723 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:01.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:02.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:02.961+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:02.962396+0000) 2019-10-13T10:05:02.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:02.960+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:02.961490+0000) 2019-10-13T10:05:04.251 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:04.250+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:04.250984+0000) 2019-10-13T10:05:04.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:04.720+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:04.720568+0000) 2019-10-13T10:05:05.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:05.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.610009+0000) 2019-10-13T10:05:05.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:05.609+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.610008+0000) 2019-10-13T10:05:06.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:06.722+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.723578+0000) 2019-10-13T10:05:06.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:06.722+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.723760+0000) 2019-10-13T10:05:06.725 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:06.722+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.723799+0000) 2019-10-13T10:05:06.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:06.725+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.725950+0000) 2019-10-13T10:05:06.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:06.722+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.724230+0000) 2019-10-13T10:05:06.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:06.722+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.724310+0000) 2019-10-13T10:05:06.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:06.723+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.724607+0000) 2019-10-13T10:05:06.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:06.726+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:06.727669+0000) 2019-10-13T10:05:06.730 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:06.728+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:08.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:08.519+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:08.519944+0000) 2019-10-13T10:05:08.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:08.857+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:08.858310+0000) 2019-10-13T10:05:11.732 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:11.730+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.731196+0000) 2019-10-13T10:05:11.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:11.730+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.731366+0000) 2019-10-13T10:05:11.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:11.731+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.731650+0000) 2019-10-13T10:05:11.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:11.732+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.733471+0000) 2019-10-13T10:05:11.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:11.731+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.731777+0000) 2019-10-13T10:05:11.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:11.731+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.732043+0000) 2019-10-13T10:05:11.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:11.732+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.732431+0000) 2019-10-13T10:05:11.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:11.735+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.736014+0000) 2019-10-13T10:05:11.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:11.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:12.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:12.962+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:12.962602+0000) 2019-10-13T10:05:12.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:12.961+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:12.961677+0000) 2019-10-13T10:05:14.253 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:14.252+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:14.253308+0000) 2019-10-13T10:05:14.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:14.720+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:14.720768+0000) 2019-10-13T10:05:15.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:15.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.610190+0000) 2019-10-13T10:05:15.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:15.609+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.610218+0000) 2019-10-13T10:05:16.741 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:16.739+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.740230+0000) 2019-10-13T10:05:16.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:16.740+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.740416+0000) 2019-10-13T10:05:16.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:16.740+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.740611+0000) 2019-10-13T10:05:16.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:16.741+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.742307+0000) 2019-10-13T10:05:16.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:16.740+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.740978+0000) 2019-10-13T10:05:16.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:16.740+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.741228+0000) 2019-10-13T10:05:16.743 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:16.741+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.741663+0000) 2019-10-13T10:05:16.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:16.743+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:16.744726+0000) 2019-10-13T10:05:16.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:16.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:18.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:18.519+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:18.520119+0000) 2019-10-13T10:05:18.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:18.858+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:18.858496+0000) 2019-10-13T10:05:21.750 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:21.748+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.748790+0000) 2019-10-13T10:05:21.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:21.748+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.748955+0000) 2019-10-13T10:05:21.750 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:21.748+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.749180+0000) 2019-10-13T10:05:21.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:21.748+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.749208+0000) 2019-10-13T10:05:21.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:21.748+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.749387+0000) 2019-10-13T10:05:21.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:21.749+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.749941+0000) 2019-10-13T10:05:21.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:21.751+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.751964+0000) 2019-10-13T10:05:21.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:21.752+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.753368+0000) 2019-10-13T10:05:21.757 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:21.755+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179789 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:22.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:22.962+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:22.962786+0000) 2019-10-13T10:05:22.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:22.961+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:22.961845+0000) 2019-10-13T10:05:24.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:24.254+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:24.255559+0000) 2019-10-13T10:05:24.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:24.720+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:24.720951+0000) 2019-10-13T10:05:25.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:25.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.610331+0000) 2019-10-13T10:05:25.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:25.609+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.610334+0000) 2019-10-13T10:05:26.758 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:26.757+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.757448+0000) 2019-10-13T10:05:26.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:26.757+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.757570+0000) 2019-10-13T10:05:26.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:26.757+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.758889+0000) 2019-10-13T10:05:26.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:26.757+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.758213+0000) 2019-10-13T10:05:26.760 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:26.758+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.758437+0000) 2019-10-13T10:05:26.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:26.758+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.758656+0000) 2019-10-13T10:05:26.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:26.758+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.759003+0000) 2019-10-13T10:05:26.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:26.759+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:26.761220+0000) 2019-10-13T10:05:26.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:26.764+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:28.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:28.519+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:28.520320+0000) 2019-10-13T10:05:28.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:28.857+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:28.858589+0000) 2019-10-13T10:05:31.341 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:05:31.341 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:31.345 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:05:31.345 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:31.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:31.765+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.766595+0000) 2019-10-13T10:05:31.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:31.765+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.766846+0000) 2019-10-13T10:05:31.768 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:31.765+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.767063+0000) 2019-10-13T10:05:31.769 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:31.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.768717+0000) 2019-10-13T10:05:31.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:31.766+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.767624+0000) 2019-10-13T10:05:31.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:31.766+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.767838+0000) 2019-10-13T10:05:31.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:31.767+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.768404+0000) 2019-10-13T10:05:31.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:31.769+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.771006+0000) 2019-10-13T10:05:31.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:31.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:32.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:32.961+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:32.962941+0000) 2019-10-13T10:05:32.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:32.961+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:32.962004+0000) 2019-10-13T10:05:34.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:34.257+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:34.257901+0000) 2019-10-13T10:05:34.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:34.720+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:34.721110+0000) 2019-10-13T10:05:35.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:35.609+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.610519+0000) 2019-10-13T10:05:35.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:35.609+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.610517+0000) 2019-10-13T10:05:36.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:36.774+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.775455+0000) 2019-10-13T10:05:36.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:36.774+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.775711+0000) 2019-10-13T10:05:36.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:36.776+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.776920+0000) 2019-10-13T10:05:36.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:36.774+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.776316+0000) 2019-10-13T10:05:36.778 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:36.775+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.776615+0000) 2019-10-13T10:05:36.778 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:36.775+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.776758+0000) 2019-10-13T10:05:36.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:36.775+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.776975+0000) 2019-10-13T10:05:36.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:36.778+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:36.779234+0000) 2019-10-13T10:05:36.784 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:36.781+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179883 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:38.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:38.519+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:38.520512+0000) 2019-10-13T10:05:38.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:38.857+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:38.858765+0000) 2019-10-13T10:05:41.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:41.782+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.784182+0000) 2019-10-13T10:05:41.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:41.783+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.784477+0000) 2019-10-13T10:05:41.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:41.785+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.785879+0000) 2019-10-13T10:05:41.787 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:41.783+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.784999+0000) 2019-10-13T10:05:41.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:41.783+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.785074+0000) 2019-10-13T10:05:41.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:41.783+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.785162+0000) 2019-10-13T10:05:41.787 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:41.783+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.785316+0000) 2019-10-13T10:05:41.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:41.788+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.788804+0000) 2019-10-13T10:05:41.791 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:41.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:42.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:42.962+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:42.963120+0000) 2019-10-13T10:05:42.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:42.961+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:42.962187+0000) 2019-10-13T10:05:44.260 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:44.259+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:44.260202+0000) 2019-10-13T10:05:44.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:44.720+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:44.721312+0000) 2019-10-13T10:05:45.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:45.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.610734+0000) 2019-10-13T10:05:45.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:45.610+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.610732+0000) 2019-10-13T10:05:46.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:46.790+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.792281+0000) 2019-10-13T10:05:46.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:46.791+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.792491+0000) 2019-10-13T10:05:46.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:46.791+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.792631+0000) 2019-10-13T10:05:46.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:46.791+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.792866+0000) 2019-10-13T10:05:46.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:46.794+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.795056+0000) 2019-10-13T10:05:46.795 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:46.792+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.793418+0000) 2019-10-13T10:05:46.795 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:46.792+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.793657+0000) 2019-10-13T10:05:46.796 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:46.796+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:46.796766+0000) 2019-10-13T10:05:46.801 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:46.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:48.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:48.519+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:48.520712+0000) 2019-10-13T10:05:48.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:48.858+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:48.858954+0000) 2019-10-13T10:05:51.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:51.800+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.801717+0000) 2019-10-13T10:05:51.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:51.802+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.803018+0000) 2019-10-13T10:05:51.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:51.800+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.802208+0000) 2019-10-13T10:05:51.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:51.801+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.802439+0000) 2019-10-13T10:05:51.804 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:51.801+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.802657+0000) 2019-10-13T10:05:51.805 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:51.801+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.802932+0000) 2019-10-13T10:05:51.805 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:51.801+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.803232+0000) 2019-10-13T10:05:51.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:51.804+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.805502+0000) 2019-10-13T10:05:51.810 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:51.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 179977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:52.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:52.962+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:52.963343+0000) 2019-10-13T10:05:52.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:52.961+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:52.962358+0000) 2019-10-13T10:05:54.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:54.262+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:54.262523+0000) 2019-10-13T10:05:54.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:54.721+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:54.721477+0000) 2019-10-13T10:05:55.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:55.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.610942+0000) 2019-10-13T10:05:55.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:55.610+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.610945+0000) 2019-10-13T10:05:56.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:05:56.811+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.811888+0000) 2019-10-13T10:05:56.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:56.810+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.810957+0000) 2019-10-13T10:05:56.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:05:56.810+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.811105+0000) 2019-10-13T10:05:56.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:56.810+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.811242+0000) 2019-10-13T10:05:56.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:05:56.811+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.811901+0000) 2019-10-13T10:05:56.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:05:56.811+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.811798+0000) 2019-10-13T10:05:56.814 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:05:56.811+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.812203+0000) 2019-10-13T10:05:56.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:05:56.813+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:56.814647+0000) 2019-10-13T10:05:56.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:05:56.818+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:05:58.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:05:58.520+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:58.520899+0000) 2019-10-13T10:05:58.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:05:58.858+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:58.859149+0000) 2019-10-13T10:06:01.486 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:06:01.486 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:01.491 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:06:01.491 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:01.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:01.819+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.820191+0000) 2019-10-13T10:06:01.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:01.820+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.820410+0000) 2019-10-13T10:06:01.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:01.820+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.820461+0000) 2019-10-13T10:06:01.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:01.820+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.820564+0000) 2019-10-13T10:06:01.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:01.820+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.820711+0000) 2019-10-13T10:06:01.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:01.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.821939+0000) 2019-10-13T10:06:01.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:01.822+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.823401+0000) 2019-10-13T10:06:01.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:01.820+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.821175+0000) 2019-10-13T10:06:01.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:01.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:02.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:02.963+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:02.963527+0000) 2019-10-13T10:06:02.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:02.962+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:02.962546+0000) 2019-10-13T10:06:04.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:04.264+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:04.264868+0000) 2019-10-13T10:06:04.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:04.721+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:04.721628+0000) 2019-10-13T10:06:05.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:05.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.611111+0000) 2019-10-13T10:06:05.612 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:05.610+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.611099+0000) 2019-10-13T10:06:06.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:06.827+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.828085+0000) 2019-10-13T10:06:06.829 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:06.827+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.828243+0000) 2019-10-13T10:06:06.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:06.828+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.829650+0000) 2019-10-13T10:06:06.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:06.828+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.828727+0000) 2019-10-13T10:06:06.831 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:06.828+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.828936+0000) 2019-10-13T10:06:06.831 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:06.828+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.828827+0000) 2019-10-13T10:06:06.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:06.828+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.829109+0000) 2019-10-13T10:06:06.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:06.831+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:06.832340+0000) 2019-10-13T10:06:06.836 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:06.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:08.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:08.520+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:08.521084+0000) 2019-10-13T10:06:08.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:08.858+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:08.859263+0000) 2019-10-13T10:06:11.838 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:11.836+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.837173+0000) 2019-10-13T10:06:11.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:11.836+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.837278+0000) 2019-10-13T10:06:11.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:11.837+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.837406+0000) 2019-10-13T10:06:11.839 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:11.837+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.837566+0000) 2019-10-13T10:06:11.840 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:11.838+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.839303+0000) 2019-10-13T10:06:11.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:11.837+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.837967+0000) 2019-10-13T10:06:11.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:11.837+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.838319+0000) 2019-10-13T10:06:11.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:11.840+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.841713+0000) 2019-10-13T10:06:11.846 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:11.844+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:12.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:12.963+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:12.963704+0000) 2019-10-13T10:06:12.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:12.962+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:12.962738+0000) 2019-10-13T10:06:14.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:14.266+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:14.267351+0000) 2019-10-13T10:06:14.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:14.720+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:14.721817+0000) 2019-10-13T10:06:15.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:15.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.611322+0000) 2019-10-13T10:06:15.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:15.610+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.611312+0000) 2019-10-13T10:06:16.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:16.845+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.846999+0000) 2019-10-13T10:06:16.848 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:16.845+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.847100+0000) 2019-10-13T10:06:16.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:16.845+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.847387+0000) 2019-10-13T10:06:16.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:16.846+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.848410+0000) 2019-10-13T10:06:16.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:16.846+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.847631+0000) 2019-10-13T10:06:16.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:16.846+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.847815+0000) 2019-10-13T10:06:16.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:16.846+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.847818+0000) 2019-10-13T10:06:16.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:16.849+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:16.851540+0000) 2019-10-13T10:06:16.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:16.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:18.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:18.520+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:18.521252+0000) 2019-10-13T10:06:18.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:18.858+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:18.859429+0000) 2019-10-13T10:06:21.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:21.853+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.855272+0000) 2019-10-13T10:06:21.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:21.853+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.855373+0000) 2019-10-13T10:06:21.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:21.854+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.855643+0000) 2019-10-13T10:06:21.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:21.855+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.856988+0000) 2019-10-13T10:06:21.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:21.854+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.856089+0000) 2019-10-13T10:06:21.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:21.854+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.856336+0000) 2019-10-13T10:06:21.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:21.855+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.856451+0000) 2019-10-13T10:06:21.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:21.857+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.859308+0000) 2019-10-13T10:06:21.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:21.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:22.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:22.963+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:22.963858+0000) 2019-10-13T10:06:22.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:22.962+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:22.962940+0000) 2019-10-13T10:06:24.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:24.268+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:24.269656+0000) 2019-10-13T10:06:24.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:24.721+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:24.721979+0000) 2019-10-13T10:06:25.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:25.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.611557+0000) 2019-10-13T10:06:25.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:25.610+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.611549+0000) 2019-10-13T10:06:26.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:26.863+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.864582+0000) 2019-10-13T10:06:26.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:26.863+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.864915+0000) 2019-10-13T10:06:26.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:26.863+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.865173+0000) 2019-10-13T10:06:26.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:26.864+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.865553+0000) 2019-10-13T10:06:26.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:26.865+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.867184+0000) 2019-10-13T10:06:26.867 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:26.864+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.865857+0000) 2019-10-13T10:06:26.868 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:26.864+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.866318+0000) 2019-10-13T10:06:26.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:26.866+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:26.868510+0000) 2019-10-13T10:06:26.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:26.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:28.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:28.520+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:28.521440+0000) 2019-10-13T10:06:28.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:28.858+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:28.859619+0000) 2019-10-13T10:06:31.567 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:06:31.567 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:31.571 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:06:31.571 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:31.876 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:31.872+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.874338+0000) 2019-10-13T10:06:31.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:31.873+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.874871+0000) 2019-10-13T10:06:31.876 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:31.873+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.874935+0000) 2019-10-13T10:06:31.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:31.875+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.876387+0000) 2019-10-13T10:06:31.877 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:31.873+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.875389+0000) 2019-10-13T10:06:31.878 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:31.874+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.875524+0000) 2019-10-13T10:06:31.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:31.874+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.875722+0000) 2019-10-13T10:06:31.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:31.877+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.878386+0000) 2019-10-13T10:06:31.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:31.880+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180229 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:32.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:32.963+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:32.963966+0000) 2019-10-13T10:06:32.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:32.962+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:32.963070+0000) 2019-10-13T10:06:34.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:34.271+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:34.272202+0000) 2019-10-13T10:06:34.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:34.721+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:34.722137+0000) 2019-10-13T10:06:35.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:35.610+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.611709+0000) 2019-10-13T10:06:35.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:35.610+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:35.611736+0000) 2019-10-13T10:06:36.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:36.884+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.885177+0000) 2019-10-13T10:06:36.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:36.882+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.883883+0000) 2019-10-13T10:06:36.886 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:36.882+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.884204+0000) 2019-10-13T10:06:36.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:36.883+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.884392+0000) 2019-10-13T10:06:36.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:36.883+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.884447+0000) 2019-10-13T10:06:36.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:36.883+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.884761+0000) 2019-10-13T10:06:36.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:36.883+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.885090+0000) 2019-10-13T10:06:36.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:36.887+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.887825+0000) 2019-10-13T10:06:36.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:36.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180261 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:38.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:38.521+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:38.521651+0000) 2019-10-13T10:06:38.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:38.859+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:38.859812+0000) 2019-10-13T10:06:41.894 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:41.893+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.894385+0000) 2019-10-13T10:06:41.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:41.893+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.893479+0000) 2019-10-13T10:06:41.895 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:41.893+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.893563+0000) 2019-10-13T10:06:41.896 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:41.893+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.893865+0000) 2019-10-13T10:06:41.896 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:41.893+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.893960+0000) 2019-10-13T10:06:41.896 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:41.893+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.894313+0000) 2019-10-13T10:06:41.896 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:41.894+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.894467+0000) 2019-10-13T10:06:41.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:41.895+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.896722+0000) 2019-10-13T10:06:41.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:41.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180293 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:42.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:42.963+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:42.964134+0000) 2019-10-13T10:06:42.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:42.962+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:42.963234+0000) 2019-10-13T10:06:44.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:44.274+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:44.274502+0000) 2019-10-13T10:06:44.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:44.721+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:44.722341+0000) 2019-10-13T10:06:45.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:45.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.611916+0000) 2019-10-13T10:06:45.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:45.611+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:45.611948+0000) 2019-10-13T10:06:46.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:46.901+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.901591+0000) 2019-10-13T10:06:46.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:46.901+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.901733+0000) 2019-10-13T10:06:46.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:46.902+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.903305+0000) 2019-10-13T10:06:46.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:46.901+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.902356+0000) 2019-10-13T10:06:46.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:46.902+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.902505+0000) 2019-10-13T10:06:46.904 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:46.902+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.902752+0000) 2019-10-13T10:06:46.905 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:46.902+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.902819+0000) 2019-10-13T10:06:46.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:46.904+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.905208+0000) 2019-10-13T10:06:46.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:46.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:48.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:48.521+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:48.521899+0000) 2019-10-13T10:06:48.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:48.859+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:48.859990+0000) 2019-10-13T10:06:51.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:51.910+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.911283+0000) 2019-10-13T10:06:51.913 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:51.911+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.911637+0000) 2019-10-13T10:06:51.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:51.911+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.911875+0000) 2019-10-13T10:06:51.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:51.911+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.912596+0000) 2019-10-13T10:06:51.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:51.911+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.912030+0000) 2019-10-13T10:06:51.914 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:51.912+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.912542+0000) 2019-10-13T10:06:51.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:51.912+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.912617+0000) 2019-10-13T10:06:51.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:51.915+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.916170+0000) 2019-10-13T10:06:51.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:51.918+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180355 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:52.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:52.964+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:52.964313+0000) 2019-10-13T10:06:52.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:52.963+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:52.963404+0000) 2019-10-13T10:06:54.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:54.276+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:54.276786+0000) 2019-10-13T10:06:54.723 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:54.722+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:54.722520+0000) 2019-10-13T10:06:55.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:55.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.612124+0000) 2019-10-13T10:06:55.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:55.612+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:55.612122+0000) 2019-10-13T10:06:56.921 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:56.920+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.920442+0000) 2019-10-13T10:06:56.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:06:56.921+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.921843+0000) 2019-10-13T10:06:56.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:06:56.920+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.920959+0000) 2019-10-13T10:06:56.923 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:06:56.920+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.920835+0000) 2019-10-13T10:06:56.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:56.920+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.921217+0000) 2019-10-13T10:06:56.923 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:06:56.920+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.921105+0000) 2019-10-13T10:06:56.923 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:06:56.921+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.921668+0000) 2019-10-13T10:06:56.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:06:56.923+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.924347+0000) 2019-10-13T10:06:56.929 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:06:56.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:06:58.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:06:58.522+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:58.522109+0000) 2019-10-13T10:06:58.861 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:06:58.860+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:58.860192+0000) 2019-10-13T10:07:01.729 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:07:01.729 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:01.734 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:07:01.734 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:01.931 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:01.928+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.929957+0000) 2019-10-13T10:07:01.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:01.930+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.931286+0000) 2019-10-13T10:07:01.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:01.928+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.930346+0000) 2019-10-13T10:07:01.932 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:01.929+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.930598+0000) 2019-10-13T10:07:01.932 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:01.929+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.930929+0000) 2019-10-13T10:07:01.933 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:01.929+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.931154+0000) 2019-10-13T10:07:01.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:01.930+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.931472+0000) 2019-10-13T10:07:01.934 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:01.933+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.934171+0000) 2019-10-13T10:07:01.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:01.935+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180419 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:02.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:02.964+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:02.964506+0000) 2019-10-13T10:07:02.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:02.962+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:02.963566+0000) 2019-10-13T10:07:04.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:04.279+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:04.279202+0000) 2019-10-13T10:07:04.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:04.721+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:04.722711+0000) 2019-10-13T10:07:05.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:05.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:05.612350+0000) 2019-10-13T10:07:05.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:05.611+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:05.612351+0000) 2019-10-13T10:07:06.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:06.936+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.937837+0000) 2019-10-13T10:07:06.940 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:06.936+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.938077+0000) 2019-10-13T10:07:06.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:06.936+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.938156+0000) 2019-10-13T10:07:06.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:06.936+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.938313+0000) 2019-10-13T10:07:06.941 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:06.937+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.938563+0000) 2019-10-13T10:07:06.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:06.938+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.940176+0000) 2019-10-13T10:07:06.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:06.937+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.938883+0000) 2019-10-13T10:07:06.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:06.940+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.942402+0000) 2019-10-13T10:07:06.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:06.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180451 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:08.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:08.520+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:08.522301+0000) 2019-10-13T10:07:08.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:08.859+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:08.860363+0000) 2019-10-13T10:07:11.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:11.944+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.946005+0000) 2019-10-13T10:07:11.948 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:11.945+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.946528+0000) 2019-10-13T10:07:11.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:11.946+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.947838+0000) 2019-10-13T10:07:11.949 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:11.945+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.946814+0000) 2019-10-13T10:07:11.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:11.945+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.946721+0000) 2019-10-13T10:07:11.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:11.945+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.947126+0000) 2019-10-13T10:07:11.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:11.945+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.947321+0000) 2019-10-13T10:07:11.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:11.947+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.949325+0000) 2019-10-13T10:07:11.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:11.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:12.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:12.963+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:12.964659+0000) 2019-10-13T10:07:12.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:12.963+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:12.963771+0000) 2019-10-13T10:07:14.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:14.280+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:14.281611+0000) 2019-10-13T10:07:14.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:14.722+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:14.722867+0000) 2019-10-13T10:07:15.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:15.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:15.612572+0000) 2019-10-13T10:07:15.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:15.611+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:15.612577+0000) 2019-10-13T10:07:16.956 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:16.953+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.955040+0000) 2019-10-13T10:07:16.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:16.953+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.955226+0000) 2019-10-13T10:07:16.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:16.953+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.955308+0000) 2019-10-13T10:07:16.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:16.954+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.955492+0000) 2019-10-13T10:07:16.957 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:16.954+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.955783+0000) 2019-10-13T10:07:16.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:16.955+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.957148+0000) 2019-10-13T10:07:16.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:16.954+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.956291+0000) 2019-10-13T10:07:16.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:16.957+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.959389+0000) 2019-10-13T10:07:16.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:16.961+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:18.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:18.521+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:18.522487+0000) 2019-10-13T10:07:18.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:18.859+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:18.860570+0000) 2019-10-13T10:07:21.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:21.964+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.965351+0000) 2019-10-13T10:07:21.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:21.963+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.964412+0000) 2019-10-13T10:07:21.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:21.963+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.964705+0000) 2019-10-13T10:07:21.966 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:21.963+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.965050+0000) 2019-10-13T10:07:21.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:21.963+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.965104+0000) 2019-10-13T10:07:21.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:21.964+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.965475+0000) 2019-10-13T10:07:21.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:21.964+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.965750+0000) 2019-10-13T10:07:21.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:21.968+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.968875+0000) 2019-10-13T10:07:21.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:21.970+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:22.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:22.964+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:22.964834+0000) 2019-10-13T10:07:22.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:22.963+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:22.963920+0000) 2019-10-13T10:07:24.284 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:24.283+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:24.283818+0000) 2019-10-13T10:07:24.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:24.722+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:24.723025+0000) 2019-10-13T10:07:25.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:25.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:25.612772+0000) 2019-10-13T10:07:25.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:25.612+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:25.612774+0000) 2019-10-13T10:07:26.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:26.974+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.974957+0000) 2019-10-13T10:07:26.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:26.973+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.974245+0000) 2019-10-13T10:07:26.976 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:26.973+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.974370+0000) 2019-10-13T10:07:26.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:26.974+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.974530+0000) 2019-10-13T10:07:26.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:26.974+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.974871+0000) 2019-10-13T10:07:26.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:26.974+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.975110+0000) 2019-10-13T10:07:26.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:26.974+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.975379+0000) 2019-10-13T10:07:26.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:26.976+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.977374+0000) 2019-10-13T10:07:26.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:26.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:28.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:28.522+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:28.522633+0000) 2019-10-13T10:07:28.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:28.860+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:28.860701+0000) 2019-10-13T10:07:31.883 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:07:31.883 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:31.887 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:07:31.887 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:31.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:31.983+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.984242+0000) 2019-10-13T10:07:31.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:31.982+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.983349+0000) 2019-10-13T10:07:31.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:31.982+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.983371+0000) 2019-10-13T10:07:31.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:31.983+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.983685+0000) 2019-10-13T10:07:31.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:31.983+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.983938+0000) 2019-10-13T10:07:31.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:31.984+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.984521+0000) 2019-10-13T10:07:31.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:31.984+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.984449+0000) 2019-10-13T10:07:31.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:31.986+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.987165+0000) 2019-10-13T10:07:31.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:31.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:32.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:32.964+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:32.965008+0000) 2019-10-13T10:07:32.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:32.963+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:32.964059+0000) 2019-10-13T10:07:34.286 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:34.285+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:34.285750+0000) 2019-10-13T10:07:34.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:34.722+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:34.723215+0000) 2019-10-13T10:07:35.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:35.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:35.612942+0000) 2019-10-13T10:07:35.614 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:35.612+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:35.612943+0000) 2019-10-13T10:07:36.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:36.992+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.992417+0000) 2019-10-13T10:07:36.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:36.992+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.992743+0000) 2019-10-13T10:07:36.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:36.993+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.993746+0000) 2019-10-13T10:07:36.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:36.992+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.992886+0000) 2019-10-13T10:07:36.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:36.992+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.993300+0000) 2019-10-13T10:07:36.995 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:36.993+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.993674+0000) 2019-10-13T10:07:36.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:36.993+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.993864+0000) 2019-10-13T10:07:36.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:36.995+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.996288+0000) 2019-10-13T10:07:37.001 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:36.999+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:38.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:38.522+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:38.522845+0000) 2019-10-13T10:07:38.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:38.860+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:38.860873+0000) 2019-10-13T10:07:42.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:42.002+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.002942+0000) 2019-10-13T10:07:42.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:42.001+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.001986+0000) 2019-10-13T10:07:42.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:42.001+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.002144+0000) 2019-10-13T10:07:42.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:42.002+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.002633+0000) 2019-10-13T10:07:42.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:42.002+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.002940+0000) 2019-10-13T10:07:42.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:42.002+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.003161+0000) 2019-10-13T10:07:42.005 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:42.003+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.003420+0000) 2019-10-13T10:07:42.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:42.005+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.006017+0000) 2019-10-13T10:07:42.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:42.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:42.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:42.964+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.965143+0000) 2019-10-13T10:07:42.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:42.964+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:42.964221+0000) 2019-10-13T10:07:44.288 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:44.287+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:44.288116+0000) 2019-10-13T10:07:44.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:44.723+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:44.723448+0000) 2019-10-13T10:07:45.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:45.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:45.613128+0000) 2019-10-13T10:07:45.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:45.612+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:45.613127+0000) 2019-10-13T10:07:47.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:47.011+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.012356+0000) 2019-10-13T10:07:47.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:47.009+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.011023+0000) 2019-10-13T10:07:47.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:47.010+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.012016+0000) 2019-10-13T10:07:47.013 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:47.010+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.012169+0000) 2019-10-13T10:07:47.014 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:47.010+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.012319+0000) 2019-10-13T10:07:47.014 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:47.011+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.012691+0000) 2019-10-13T10:07:47.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:47.011+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.012768+0000) 2019-10-13T10:07:47.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:47.013+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.014633+0000) 2019-10-13T10:07:47.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:47.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:48.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:48.522+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:48.523058+0000) 2019-10-13T10:07:48.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:48.860+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:48.861065+0000) 2019-10-13T10:07:52.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:52.019+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.020890+0000) 2019-10-13T10:07:52.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:52.021+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.022388+0000) 2019-10-13T10:07:52.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:52.020+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.021598+0000) 2019-10-13T10:07:52.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:52.020+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.021967+0000) 2019-10-13T10:07:52.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:52.020+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.021902+0000) 2019-10-13T10:07:52.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:52.020+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.022314+0000) 2019-10-13T10:07:52.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:52.020+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.022382+0000) 2019-10-13T10:07:52.026 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:52.025+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.026039+0000) 2019-10-13T10:07:52.030 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:52.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:52.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:52.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.965282+0000) 2019-10-13T10:07:52.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:52.963+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:52.964387+0000) 2019-10-13T10:07:54.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:54.289+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:54.290582+0000) 2019-10-13T10:07:54.725 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:54.722+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:54.723630+0000) 2019-10-13T10:07:55.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:55.611+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:55.613340+0000) 2019-10-13T10:07:55.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:55.611+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:55.613342+0000) 2019-10-13T10:07:57.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:07:57.030+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.031830+0000) 2019-10-13T10:07:57.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:07:57.029+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.030807+0000) 2019-10-13T10:07:57.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:07:57.029+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.030578+0000) 2019-10-13T10:07:57.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:57.029+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.031215+0000) 2019-10-13T10:07:57.033 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:07:57.030+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.031515+0000) 2019-10-13T10:07:57.033 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:57.030+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.031608+0000) 2019-10-13T10:07:57.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:07:57.030+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.031795+0000) 2019-10-13T10:07:57.034 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:07:57.033+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.034910+0000) 2019-10-13T10:07:57.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:07:57.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:07:58.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:07:58.522+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:58.523301+0000) 2019-10-13T10:07:58.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:07:58.860+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:58.861282+0000) 2019-10-13T10:08:01.970 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:08:01.970 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:01.975 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:08:01.975 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:02.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:02.039+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.040872+0000) 2019-10-13T10:08:02.041 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:02.038+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.039944+0000) 2019-10-13T10:08:02.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:02.038+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.040064+0000) 2019-10-13T10:08:02.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:02.038+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.040354+0000) 2019-10-13T10:08:02.042 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:02.039+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.040528+0000) 2019-10-13T10:08:02.042 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:02.039+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.040936+0000) 2019-10-13T10:08:02.043 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:02.039+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.041003+0000) 2019-10-13T10:08:02.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:02.042+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.044079+0000) 2019-10-13T10:08:02.047 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:02.045+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:02.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:02.964+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.965440+0000) 2019-10-13T10:08:02.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:02.963+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:02.964586+0000) 2019-10-13T10:08:04.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:04.292+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:04.292883+0000) 2019-10-13T10:08:04.725 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:04.723+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:04.723801+0000) 2019-10-13T10:08:05.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:05.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:05.613453+0000) 2019-10-13T10:08:05.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:05.612+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:05.613521+0000) 2019-10-13T10:08:07.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:07.047+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.049454+0000) 2019-10-13T10:08:07.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:07.046+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.048104+0000) 2019-10-13T10:08:07.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:07.047+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.048553+0000) 2019-10-13T10:08:07.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:07.047+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.048772+0000) 2019-10-13T10:08:07.053 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:07.047+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.049157+0000) 2019-10-13T10:08:07.053 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:07.047+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.049304+0000) 2019-10-13T10:08:07.053 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:07.048+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.049531+0000) 2019-10-13T10:08:07.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:07.051+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.053121+0000) 2019-10-13T10:08:07.056 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:07.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:08.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:08.522+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:08.523487+0000) 2019-10-13T10:08:08.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:08.860+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:08.861469+0000) 2019-10-13T10:08:12.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:12.055+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.056857+0000) 2019-10-13T10:08:12.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:12.055+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.057233+0000) 2019-10-13T10:08:12.059 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:12.056+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.057709+0000) 2019-10-13T10:08:12.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:12.056+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.057997+0000) 2019-10-13T10:08:12.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:12.057+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.058572+0000) 2019-10-13T10:08:12.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:12.056+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.058333+0000) 2019-10-13T10:08:12.060 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:12.057+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.058714+0000) 2019-10-13T10:08:12.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:12.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.060790+0000) 2019-10-13T10:08:12.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:12.064+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:12.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:12.964+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.965602+0000) 2019-10-13T10:08:12.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:12.964+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:12.964721+0000) 2019-10-13T10:08:14.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:14.294+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:14.295315+0000) 2019-10-13T10:08:14.725 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:14.723+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:14.723978+0000) 2019-10-13T10:08:15.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:15.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:15.613642+0000) 2019-10-13T10:08:15.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:15.613+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:15.613681+0000) 2019-10-13T10:08:17.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:17.067+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.067748+0000) 2019-10-13T10:08:17.069 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:17.066+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.066744+0000) 2019-10-13T10:08:17.069 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:17.066+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.067174+0000) 2019-10-13T10:08:17.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:17.066+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.067381+0000) 2019-10-13T10:08:17.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:17.067+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.067590+0000) 2019-10-13T10:08:17.070 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:17.067+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.067715+0000) 2019-10-13T10:08:17.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:17.067+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.067979+0000) 2019-10-13T10:08:17.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:17.069+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.070600+0000) 2019-10-13T10:08:17.076 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:17.074+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:18.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:18.523+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:18.523617+0000) 2019-10-13T10:08:18.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:18.861+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:18.861655+0000) 2019-10-13T10:08:22.077 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:22.075+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.076194+0000) 2019-10-13T10:08:22.078 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:22.075+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.076384+0000) 2019-10-13T10:08:22.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:22.077+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.077760+0000) 2019-10-13T10:08:22.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:22.076+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.076975+0000) 2019-10-13T10:08:22.080 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:22.076+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.077225+0000) 2019-10-13T10:08:22.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:22.077+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.077588+0000) 2019-10-13T10:08:22.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:22.077+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.077408+0000) 2019-10-13T10:08:22.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:22.079+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.080093+0000) 2019-10-13T10:08:22.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:22.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:22.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:22.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.965764+0000) 2019-10-13T10:08:22.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:22.964+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:22.964855+0000) 2019-10-13T10:08:24.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:24.297+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:24.297693+0000) 2019-10-13T10:08:24.725 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:24.723+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:24.724169+0000) 2019-10-13T10:08:25.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:25.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:25.613869+0000) 2019-10-13T10:08:25.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:25.613+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:25.613875+0000) 2019-10-13T10:08:27.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:27.086+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.086998+0000) 2019-10-13T10:08:27.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:27.086+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.087162+0000) 2019-10-13T10:08:27.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:27.087+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.088561+0000) 2019-10-13T10:08:27.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:27.087+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.087541+0000) 2019-10-13T10:08:27.090 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:27.087+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.087735+0000) 2019-10-13T10:08:27.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:27.087+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.087853+0000) 2019-10-13T10:08:27.090 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:27.088+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.088393+0000) 2019-10-13T10:08:27.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:27.090+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.091269+0000) 2019-10-13T10:08:27.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:27.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:28.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:28.523+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:28.523798+0000) 2019-10-13T10:08:28.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:28.861+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:28.861860+0000) 2019-10-13T10:08:32.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:32.095+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.095874+0000) 2019-10-13T10:08:32.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:32.093+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.095018+0000) 2019-10-13T10:08:32.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:32.093+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.095141+0000) 2019-10-13T10:08:32.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:32.094+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.095390+0000) 2019-10-13T10:08:32.097 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:32.094+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.095658+0000) 2019-10-13T10:08:32.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:32.094+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.096026+0000) 2019-10-13T10:08:32.098 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:32.094+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.096198+0000) 2019-10-13T10:08:32.099 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:32.098+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.099054+0000) 2019-10-13T10:08:32.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:32.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 180985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:32.108 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:08:32.108 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:32.111 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:08:32.112 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:32.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:32.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.965871+0000) 2019-10-13T10:08:32.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:32.963+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:32.965028+0000) 2019-10-13T10:08:34.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:34.299+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:34.299942+0000) 2019-10-13T10:08:34.725 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:34.723+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:34.724360+0000) 2019-10-13T10:08:35.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:35.612+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:35.614022+0000) 2019-10-13T10:08:35.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:35.612+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:35.614061+0000) 2019-10-13T10:08:37.105 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:37.102+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.104289+0000) 2019-10-13T10:08:37.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:37.104+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.105572+0000) 2019-10-13T10:08:37.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:37.103+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.104646+0000) 2019-10-13T10:08:37.107 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:37.103+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.104798+0000) 2019-10-13T10:08:37.107 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:37.103+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.105205+0000) 2019-10-13T10:08:37.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:37.103+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.105321+0000) 2019-10-13T10:08:37.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:37.104+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.105685+0000) 2019-10-13T10:08:37.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:37.107+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.108368+0000) 2019-10-13T10:08:37.112 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:37.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181020 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:38.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:38.523+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:38.523984+0000) 2019-10-13T10:08:38.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:38.861+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:38.862060+0000) 2019-10-13T10:08:42.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:42.113+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.114218+0000) 2019-10-13T10:08:42.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:42.112+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.113509+0000) 2019-10-13T10:08:42.115 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:42.112+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.113622+0000) 2019-10-13T10:08:42.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:42.112+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.114012+0000) 2019-10-13T10:08:42.116 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:42.112+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.114015+0000) 2019-10-13T10:08:42.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:42.112+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.114130+0000) 2019-10-13T10:08:42.116 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:42.112+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.114367+0000) 2019-10-13T10:08:42.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:42.117+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.117862+0000) 2019-10-13T10:08:42.122 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:42.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:42.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:42.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.966051+0000) 2019-10-13T10:08:42.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:42.964+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:42.965170+0000) 2019-10-13T10:08:44.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:44.302+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:44.302327+0000) 2019-10-13T10:08:44.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:44.723+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:44.724565+0000) 2019-10-13T10:08:45.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:45.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:45.614277+0000) 2019-10-13T10:08:45.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:45.613+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:45.614240+0000) 2019-10-13T10:08:47.124 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:47.121+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.122563+0000) 2019-10-13T10:08:47.124 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:47.121+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.122796+0000) 2019-10-13T10:08:47.125 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:47.122+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.124357+0000) 2019-10-13T10:08:47.125 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:47.121+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.123124+0000) 2019-10-13T10:08:47.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:47.121+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.123312+0000) 2019-10-13T10:08:47.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:47.122+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.123516+0000) 2019-10-13T10:08:47.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:47.122+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.123729+0000) 2019-10-13T10:08:47.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:47.124+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.126286+0000) 2019-10-13T10:08:47.131 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:47.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181082 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:48.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:48.523+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:48.524195+0000) 2019-10-13T10:08:48.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:48.861+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:48.862271+0000) 2019-10-13T10:08:52.133 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:52.130+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.131900+0000) 2019-10-13T10:08:52.133 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:52.130+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.132057+0000) 2019-10-13T10:08:52.134 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:52.130+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.132343+0000) 2019-10-13T10:08:52.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:52.131+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.133611+0000) 2019-10-13T10:08:52.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:52.131+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.132705+0000) 2019-10-13T10:08:52.135 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:52.131+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.132929+0000) 2019-10-13T10:08:52.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:52.131+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.133099+0000) 2019-10-13T10:08:52.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:52.133+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.135277+0000) 2019-10-13T10:08:52.140 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:52.137+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:52.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:52.964+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.965305+0000) 2019-10-13T10:08:52.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:52.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:52.966213+0000) 2019-10-13T10:08:54.304 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:54.303+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:54.304668+0000) 2019-10-13T10:08:54.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:54.724+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:54.724754+0000) 2019-10-13T10:08:55.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:55.613+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:55.614438+0000) 2019-10-13T10:08:55.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:55.613+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:55.614439+0000) 2019-10-13T10:08:57.142 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:08:57.139+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.140500+0000) 2019-10-13T10:08:57.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:08:57.139+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.140622+0000) 2019-10-13T10:08:57.142 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:08:57.139+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.140777+0000) 2019-10-13T10:08:57.142 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:57.139+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.140935+0000) 2019-10-13T10:08:57.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:08:57.140+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.142452+0000) 2019-10-13T10:08:57.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:08:57.140+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.141442+0000) 2019-10-13T10:08:57.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:57.140+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.141679+0000) 2019-10-13T10:08:57.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:08:57.142+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.144671+0000) 2019-10-13T10:08:57.148 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:08:57.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:08:58.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:08:58.524+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:58.524393+0000) 2019-10-13T10:08:58.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:08:58.862+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:58.862406+0000) 2019-10-13T10:09:02.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:02.148+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.148910+0000) 2019-10-13T10:09:02.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:02.149+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.150542+0000) 2019-10-13T10:09:02.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:02.149+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.149505+0000) 2019-10-13T10:09:02.151 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:02.149+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.149784+0000) 2019-10-13T10:09:02.151 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:02.149+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.149922+0000) 2019-10-13T10:09:02.152 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:02.149+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.150099+0000) 2019-10-13T10:09:02.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:02.150+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.150551+0000) 2019-10-13T10:09:02.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:02.152+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.152880+0000) 2019-10-13T10:09:02.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:02.157+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181175 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:02.279 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:09:02.280 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:02.284 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:09:02.284 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:02.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:02.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.966379+0000) 2019-10-13T10:09:02.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:02.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:02.965467+0000) 2019-10-13T10:09:04.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:04.306+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:04.307049+0000) 2019-10-13T10:09:04.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:04.724+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:04.724980+0000) 2019-10-13T10:09:05.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:05.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:05.614651+0000) 2019-10-13T10:09:05.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:05.614+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:05.614653+0000) 2019-10-13T10:09:07.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:07.159+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.160400+0000) 2019-10-13T10:09:07.161 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:07.159+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.159411+0000) 2019-10-13T10:09:07.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:07.159+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.159615+0000) 2019-10-13T10:09:07.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:07.159+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.159858+0000) 2019-10-13T10:09:07.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:07.159+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.159976+0000) 2019-10-13T10:09:07.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:07.159+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.160321+0000) 2019-10-13T10:09:07.162 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:07.160+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.160595+0000) 2019-10-13T10:09:07.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:07.162+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.163463+0000) 2019-10-13T10:09:07.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:07.166+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:08.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:08.524+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:08.524583+0000) 2019-10-13T10:09:08.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:08.862+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:08.862572+0000) 2019-10-13T10:09:12.170 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:12.168+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.169165+0000) 2019-10-13T10:09:12.171 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:12.168+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.169356+0000) 2019-10-13T10:09:12.171 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:12.168+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.169351+0000) 2019-10-13T10:09:12.171 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:12.169+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.169655+0000) 2019-10-13T10:09:12.172 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:12.169+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.170600+0000) 2019-10-13T10:09:12.172 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:12.171+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.172070+0000) 2019-10-13T10:09:12.172 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:12.169+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.169905+0000) 2019-10-13T10:09:12.173 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:12.169+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.170099+0000) 2019-10-13T10:09:12.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:12.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:12.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:12.966+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.966557+0000) 2019-10-13T10:09:12.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:12.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:12.965604+0000) 2019-10-13T10:09:14.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:14.308+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:14.309107+0000) 2019-10-13T10:09:14.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:14.725+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:14.725205+0000) 2019-10-13T10:09:15.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:15.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:15.614844+0000) 2019-10-13T10:09:15.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:15.614+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:15.614843+0000) 2019-10-13T10:09:17.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:17.176+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.178240+0000) 2019-10-13T10:09:17.180 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:17.177+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.178541+0000) 2019-10-13T10:09:17.180 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:17.177+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.178734+0000) 2019-10-13T10:09:17.181 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:17.178+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.179547+0000) 2019-10-13T10:09:17.181 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:17.177+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.178869+0000) 2019-10-13T10:09:17.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:17.177+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.179019+0000) 2019-10-13T10:09:17.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:17.178+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.179615+0000) 2019-10-13T10:09:17.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:17.180+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.181408+0000) 2019-10-13T10:09:17.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:17.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181272 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:18.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:18.523+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:18.524739+0000) 2019-10-13T10:09:18.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:18.861+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:18.862735+0000) 2019-10-13T10:09:22.188 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:22.187+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.188168+0000) 2019-10-13T10:09:22.189 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:22.186+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.187464+0000) 2019-10-13T10:09:22.189 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:22.186+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.187558+0000) 2019-10-13T10:09:22.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:22.186+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.187932+0000) 2019-10-13T10:09:22.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:22.186+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.188059+0000) 2019-10-13T10:09:22.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:22.186+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.188361+0000) 2019-10-13T10:09:22.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:22.187+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.188463+0000) 2019-10-13T10:09:22.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:22.190+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.191596+0000) 2019-10-13T10:09:22.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:22.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181301 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:22.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:22.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.966711+0000) 2019-10-13T10:09:22.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:22.964+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:22.965714+0000) 2019-10-13T10:09:24.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:24.311+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:24.311354+0000) 2019-10-13T10:09:24.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:24.724+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:24.725390+0000) 2019-10-13T10:09:25.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:25.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:25.615055+0000) 2019-10-13T10:09:25.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:25.614+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:25.615055+0000) 2019-10-13T10:09:27.198 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:27.195+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.196522+0000) 2019-10-13T10:09:27.198 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:27.195+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.196594+0000) 2019-10-13T10:09:27.198 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:27.195+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.196877+0000) 2019-10-13T10:09:27.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:27.195+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.197085+0000) 2019-10-13T10:09:27.199 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:27.197+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.197881+0000) 2019-10-13T10:09:27.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:27.195+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.197212+0000) 2019-10-13T10:09:27.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:27.196+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.197413+0000) 2019-10-13T10:09:27.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:27.199+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.200282+0000) 2019-10-13T10:09:27.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:27.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:28.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:28.524+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:28.524931+0000) 2019-10-13T10:09:28.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:28.862+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:28.862910+0000) 2019-10-13T10:09:32.206 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:32.203+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.205300+0000) 2019-10-13T10:09:32.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:32.204+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.205663+0000) 2019-10-13T10:09:32.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:32.204+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.205607+0000) 2019-10-13T10:09:32.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:32.206+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.206984+0000) 2019-10-13T10:09:32.208 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:32.204+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.206014+0000) 2019-10-13T10:09:32.208 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:32.205+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.206413+0000) 2019-10-13T10:09:32.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:32.204+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.206284+0000) 2019-10-13T10:09:32.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:32.209+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.210586+0000) 2019-10-13T10:09:32.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:32.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:32.442 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:09:32.442 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:32.446 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:09:32.447 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:32.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:32.966+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.966896+0000) 2019-10-13T10:09:32.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:32.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:32.965930+0000) 2019-10-13T10:09:34.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:34.313+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:34.313633+0000) 2019-10-13T10:09:34.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:34.724+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:34.725562+0000) 2019-10-13T10:09:35.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:35.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:35.615253+0000) 2019-10-13T10:09:35.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:35.614+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:35.615255+0000) 2019-10-13T10:09:37.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:37.213+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.214526+0000) 2019-10-13T10:09:37.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:37.212+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.214382+0000) 2019-10-13T10:09:37.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:37.215+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.216007+0000) 2019-10-13T10:09:37.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:37.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.215186+0000) 2019-10-13T10:09:37.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:37.213+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.215318+0000) 2019-10-13T10:09:37.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:37.214+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.215446+0000) 2019-10-13T10:09:37.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:37.214+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.215864+0000) 2019-10-13T10:09:37.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:37.217+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.217919+0000) 2019-10-13T10:09:37.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:37.220+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181394 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:38.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:38.524+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:38.525093+0000) 2019-10-13T10:09:38.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:38.862+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:38.863094+0000) 2019-10-13T10:09:42.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:42.222+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.223432+0000) 2019-10-13T10:09:42.226 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:42.223+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.224751+0000) 2019-10-13T10:09:42.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:42.222+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.224120+0000) 2019-10-13T10:09:42.226 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:42.222+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.224317+0000) 2019-10-13T10:09:42.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:42.223+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.224456+0000) 2019-10-13T10:09:42.227 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:42.223+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.224617+0000) 2019-10-13T10:09:42.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:42.223+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.224938+0000) 2019-10-13T10:09:42.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:42.225+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.227110+0000) 2019-10-13T10:09:42.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:42.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:42.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:42.965+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.967019+0000) 2019-10-13T10:09:42.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:42.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:42.966103+0000) 2019-10-13T10:09:44.316 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:44.315+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:44.316122+0000) 2019-10-13T10:09:44.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:44.725+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:44.725714+0000) 2019-10-13T10:09:45.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:45.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:45.615484+0000) 2019-10-13T10:09:45.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:45.615+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:45.615483+0000) 2019-10-13T10:09:47.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:47.232+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.232758+0000) 2019-10-13T10:09:47.234 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:47.232+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.233037+0000) 2019-10-13T10:09:47.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:47.232+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.234350+0000) 2019-10-13T10:09:47.235 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:47.233+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.233472+0000) 2019-10-13T10:09:47.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:47.233+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.233779+0000) 2019-10-13T10:09:47.236 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:47.233+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.233942+0000) 2019-10-13T10:09:47.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:47.233+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.234188+0000) 2019-10-13T10:09:47.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:47.235+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.236832+0000) 2019-10-13T10:09:47.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:47.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:48.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:48.524+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:48.525285+0000) 2019-10-13T10:09:48.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:48.862+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:48.863289+0000) 2019-10-13T10:09:52.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:52.242+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.242474+0000) 2019-10-13T10:09:52.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:52.242+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.242600+0000) 2019-10-13T10:09:52.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:52.242+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.242857+0000) 2019-10-13T10:09:52.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:52.243+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.244278+0000) 2019-10-13T10:09:52.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:52.242+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.243063+0000) 2019-10-13T10:09:52.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:52.242+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.243323+0000) 2019-10-13T10:09:52.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:52.243+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.243453+0000) 2019-10-13T10:09:52.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:52.246+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.246752+0000) 2019-10-13T10:09:52.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:52.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:52.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:52.966+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.967173+0000) 2019-10-13T10:09:52.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:52.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:52.966216+0000) 2019-10-13T10:09:54.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:54.317+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:54.318525+0000) 2019-10-13T10:09:54.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:54.725+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:54.725889+0000) 2019-10-13T10:09:55.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:55.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:55.615680+0000) 2019-10-13T10:09:55.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:55.615+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:55.615679+0000) 2019-10-13T10:09:57.252 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:57.250+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.251048+0000) 2019-10-13T10:09:57.253 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:09:57.251+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.251446+0000) 2019-10-13T10:09:57.254 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:09:57.252+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.253491+0000) 2019-10-13T10:09:57.254 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:57.251+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.251882+0000) 2019-10-13T10:09:57.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:09:57.251+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.252151+0000) 2019-10-13T10:09:57.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:09:57.251+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.252214+0000) 2019-10-13T10:09:57.255 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:09:57.252+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.252469+0000) 2019-10-13T10:09:57.255 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:09:57.254+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.255063+0000) 2019-10-13T10:09:57.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:09:57.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:09:58.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:09:58.525+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:58.525507+0000) 2019-10-13T10:09:58.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:09:58.863+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:58.863480+0000) 2019-10-13T10:10:02.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:02.259+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.260681+0000) 2019-10-13T10:10:02.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:02.259+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.260904+0000) 2019-10-13T10:10:02.263 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:02.262+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.262848+0000) 2019-10-13T10:10:02.263 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:02.260+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.261526+0000) 2019-10-13T10:10:02.264 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:02.260+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.261418+0000) 2019-10-13T10:10:02.264 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:02.260+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.261854+0000) 2019-10-13T10:10:02.264 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:02.260+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.262040+0000) 2019-10-13T10:10:02.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:02.263+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.264536+0000) 2019-10-13T10:10:02.269 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:02.267+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:02.572 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:10:02.572 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:02.577 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:10:02.577 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:02.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:02.966+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.967310+0000) 2019-10-13T10:10:02.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:02.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:02.966360+0000) 2019-10-13T10:10:04.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:04.320+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:04.320837+0000) 2019-10-13T10:10:04.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:04.725+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:04.726089+0000) 2019-10-13T10:10:05.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:05.614+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:05.615878+0000) 2019-10-13T10:10:05.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:05.614+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:05.615881+0000) 2019-10-13T10:10:07.272 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:07.268+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.270283+0000) 2019-10-13T10:10:07.272 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:07.269+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.270654+0000) 2019-10-13T10:10:07.272 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:07.269+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.270746+0000) 2019-10-13T10:10:07.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:07.271+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.272651+0000) 2019-10-13T10:10:07.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:07.269+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.271012+0000) 2019-10-13T10:10:07.274 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:07.269+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.271215+0000) 2019-10-13T10:10:07.274 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:07.270+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.271550+0000) 2019-10-13T10:10:07.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:07.274+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.274956+0000) 2019-10-13T10:10:07.279 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:07.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181576 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:08.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:08.524+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:08.525668+0000) 2019-10-13T10:10:08.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:08.862+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:08.863637+0000) 2019-10-13T10:10:12.281 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:12.280+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.280971+0000) 2019-10-13T10:10:12.282 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:12.278+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.280069+0000) 2019-10-13T10:10:12.283 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:12.278+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.280248+0000) 2019-10-13T10:10:12.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:12.279+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.280394+0000) 2019-10-13T10:10:12.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:12.279+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.280875+0000) 2019-10-13T10:10:12.284 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:12.279+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.280858+0000) 2019-10-13T10:10:12.284 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:12.279+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.281327+0000) 2019-10-13T10:10:12.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:12.283+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.283801+0000) 2019-10-13T10:10:12.289 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:12.286+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:12.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:12.967+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.967445+0000) 2019-10-13T10:10:12.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:12.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:12.966543+0000) 2019-10-13T10:10:14.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:14.322+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:14.323095+0000) 2019-10-13T10:10:14.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:14.725+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:14.726268+0000) 2019-10-13T10:10:15.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:15.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:15.616100+0000) 2019-10-13T10:10:15.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:15.615+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:15.616102+0000) 2019-10-13T10:10:17.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:17.287+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.289250+0000) 2019-10-13T10:10:17.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:17.288+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.289594+0000) 2019-10-13T10:10:17.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:17.288+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.289657+0000) 2019-10-13T10:10:17.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:17.288+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.290055+0000) 2019-10-13T10:10:17.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:17.290+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.290947+0000) 2019-10-13T10:10:17.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:17.292+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.293413+0000) 2019-10-13T10:10:17.305 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:17.289+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.290575+0000) 2019-10-13T10:10:17.306 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:17.289+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.290733+0000) 2019-10-13T10:10:17.306 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:17.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181638 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:18.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:18.525+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:18.525836+0000) 2019-10-13T10:10:18.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:18.863+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:18.863803+0000) 2019-10-13T10:10:22.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:22.299+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.300495+0000) 2019-10-13T10:10:22.301 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:22.298+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.299727+0000) 2019-10-13T10:10:22.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:22.298+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.299956+0000) 2019-10-13T10:10:22.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:22.298+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.300156+0000) 2019-10-13T10:10:22.302 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:22.299+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.300389+0000) 2019-10-13T10:10:22.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:22.299+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.300413+0000) 2019-10-13T10:10:22.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:22.299+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.300723+0000) 2019-10-13T10:10:22.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:22.301+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.302691+0000) 2019-10-13T10:10:22.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:22.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:22.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:22.967+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.967599+0000) 2019-10-13T10:10:22.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:22.966+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:22.966697+0000) 2019-10-13T10:10:24.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:24.325+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:24.325539+0000) 2019-10-13T10:10:24.727 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:24.725+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:24.726466+0000) 2019-10-13T10:10:25.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:25.615+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:25.616325+0000) 2019-10-13T10:10:25.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:25.615+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:25.616324+0000) 2019-10-13T10:10:27.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:27.309+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.309888+0000) 2019-10-13T10:10:27.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:27.307+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.308805+0000) 2019-10-13T10:10:27.311 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:27.307+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.308915+0000) 2019-10-13T10:10:27.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:27.307+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.309350+0000) 2019-10-13T10:10:27.311 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:27.308+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.309638+0000) 2019-10-13T10:10:27.311 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:27.308+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.309903+0000) 2019-10-13T10:10:27.312 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:27.308+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.310114+0000) 2019-10-13T10:10:27.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:27.311+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.312321+0000) 2019-10-13T10:10:27.317 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:27.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:28.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:28.525+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:28.526040+0000) 2019-10-13T10:10:28.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:28.863+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:28.864003+0000) 2019-10-13T10:10:32.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:32.317+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.319309+0000) 2019-10-13T10:10:32.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:32.318+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.318457+0000) 2019-10-13T10:10:32.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:32.318+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.318766+0000) 2019-10-13T10:10:32.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:32.318+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.319153+0000) 2019-10-13T10:10:32.321 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:32.318+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.319245+0000) 2019-10-13T10:10:32.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:32.321+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.322841+0000) 2019-10-13T10:10:32.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:32.318+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.319387+0000) 2019-10-13T10:10:32.324 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:32.319+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.319679+0000) 2019-10-13T10:10:32.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:32.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:32.710 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:10:32.710 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:32.723 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:10:32.723 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:32.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:32.966+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.967790+0000) 2019-10-13T10:10:32.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:32.966+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:32.966845+0000) 2019-10-13T10:10:34.328 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:34.326+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:34.327862+0000) 2019-10-13T10:10:34.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:34.726+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:34.726652+0000) 2019-10-13T10:10:35.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:35.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:35.616452+0000) 2019-10-13T10:10:35.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:35.616+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:35.616514+0000) 2019-10-13T10:10:37.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:37.327+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.328883+0000) 2019-10-13T10:10:37.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:37.327+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.328262+0000) 2019-10-13T10:10:37.330 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:37.328+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.328421+0000) 2019-10-13T10:10:37.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:37.328+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.328585+0000) 2019-10-13T10:10:37.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:37.328+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.328670+0000) 2019-10-13T10:10:37.330 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:37.328+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.328898+0000) 2019-10-13T10:10:37.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:37.328+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.329079+0000) 2019-10-13T10:10:37.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:37.329+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.331377+0000) 2019-10-13T10:10:37.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:37.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181760 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:38.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:38.525+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:38.526218+0000) 2019-10-13T10:10:38.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:38.863+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:38.864182+0000) 2019-10-13T10:10:42.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:42.336+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.337229+0000) 2019-10-13T10:10:42.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:42.337+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.337499+0000) 2019-10-13T10:10:42.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:42.336+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.338398+0000) 2019-10-13T10:10:42.340 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:42.337+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.337917+0000) 2019-10-13T10:10:42.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:42.337+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.338053+0000) 2019-10-13T10:10:42.340 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:42.337+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.338263+0000) 2019-10-13T10:10:42.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:42.338+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.338536+0000) 2019-10-13T10:10:42.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:42.340+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.341932+0000) 2019-10-13T10:10:42.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:42.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181789 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:42.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:42.967+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.967980+0000) 2019-10-13T10:10:42.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:42.966+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:42.967026+0000) 2019-10-13T10:10:44.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:44.329+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:44.330228+0000) 2019-10-13T10:10:44.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:44.726+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:44.726828+0000) 2019-10-13T10:10:45.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:45.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:45.616674+0000) 2019-10-13T10:10:45.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:45.616+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:45.616677+0000) 2019-10-13T10:10:47.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:47.346+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.346902+0000) 2019-10-13T10:10:47.348 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:47.346+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.347196+0000) 2019-10-13T10:10:47.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:47.347+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.347479+0000) 2019-10-13T10:10:47.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:47.347+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.348448+0000) 2019-10-13T10:10:47.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:47.347+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.347611+0000) 2019-10-13T10:10:47.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:47.347+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.347701+0000) 2019-10-13T10:10:47.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:47.347+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.347890+0000) 2019-10-13T10:10:47.350 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:47.349+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.350275+0000) 2019-10-13T10:10:47.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:47.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:48.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:48.525+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:48.526425+0000) 2019-10-13T10:10:48.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:48.863+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:48.864417+0000) 2019-10-13T10:10:52.356 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:52.355+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.356073+0000) 2019-10-13T10:10:52.358 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:52.354+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.355423+0000) 2019-10-13T10:10:52.358 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:52.354+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.355543+0000) 2019-10-13T10:10:52.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:52.354+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.355792+0000) 2019-10-13T10:10:52.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:52.354+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.355896+0000) 2019-10-13T10:10:52.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:52.354+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.356184+0000) 2019-10-13T10:10:52.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:52.354+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.356128+0000) 2019-10-13T10:10:52.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:52.357+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.358477+0000) 2019-10-13T10:10:52.364 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:52.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181849 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:52.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:52.967+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.968137+0000) 2019-10-13T10:10:52.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:52.966+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:52.967205+0000) 2019-10-13T10:10:54.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:54.331+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:54.332543+0000) 2019-10-13T10:10:54.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:54.726+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:54.727020+0000) 2019-10-13T10:10:55.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:55.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:55.616867+0000) 2019-10-13T10:10:55.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:55.616+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:55.616866+0000) 2019-10-13T10:10:57.366 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:10:57.363+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.364568+0000) 2019-10-13T10:10:57.366 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:10:57.362+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.364375+0000) 2019-10-13T10:10:57.366 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:57.363+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.364848+0000) 2019-10-13T10:10:57.367 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:10:57.363+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.364923+0000) 2019-10-13T10:10:57.367 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:10:57.365+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.366120+0000) 2019-10-13T10:10:57.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:57.364+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.365392+0000) 2019-10-13T10:10:57.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:10:57.364+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.365743+0000) 2019-10-13T10:10:57.369 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:10:57.368+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.369371+0000) 2019-10-13T10:10:57.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:10:57.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:10:58.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:10:58.525+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:58.526613+0000) 2019-10-13T10:10:58.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:10:58.863+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:58.864602+0000) 2019-10-13T10:11:02.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:02.372+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.374100+0000) 2019-10-13T10:11:02.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:02.372+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.374201+0000) 2019-10-13T10:11:02.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:02.373+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.374540+0000) 2019-10-13T10:11:02.376 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:02.373+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.375137+0000) 2019-10-13T10:11:02.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:02.373+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.375283+0000) 2019-10-13T10:11:02.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:02.375+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.376561+0000) 2019-10-13T10:11:02.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:02.374+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.376162+0000) 2019-10-13T10:11:02.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:02.378+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.379223+0000) 2019-10-13T10:11:02.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:02.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:02.814 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:11:02.814 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:02.818 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:11:02.818 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:02.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:02.967+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.968283+0000) 2019-10-13T10:11:02.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:02.965+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:02.967368+0000) 2019-10-13T10:11:04.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:04.334+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:04.334947+0000) 2019-10-13T10:11:04.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:04.726+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:04.727253+0000) 2019-10-13T10:11:05.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:05.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:05.617062+0000) 2019-10-13T10:11:05.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:05.616+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:05.617061+0000) 2019-10-13T10:11:07.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:07.382+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.383678+0000) 2019-10-13T10:11:07.385 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:07.382+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.384163+0000) 2019-10-13T10:11:07.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:07.383+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.384425+0000) 2019-10-13T10:11:07.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:07.383+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.384643+0000) 2019-10-13T10:11:07.387 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:07.385+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.386234+0000) 2019-10-13T10:11:07.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:07.383+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.384802+0000) 2019-10-13T10:11:07.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:07.383+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.385355+0000) 2019-10-13T10:11:07.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:07.387+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.388158+0000) 2019-10-13T10:11:07.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:07.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181942 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:08.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:08.526+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:08.526806+0000) 2019-10-13T10:11:08.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:08.864+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:08.864790+0000) 2019-10-13T10:11:12.395 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:12.392+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.393601+0000) 2019-10-13T10:11:12.395 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:12.392+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.393935+0000) 2019-10-13T10:11:12.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:12.394+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.395718+0000) 2019-10-13T10:11:12.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:12.392+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.394312+0000) 2019-10-13T10:11:12.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:12.392+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.394381+0000) 2019-10-13T10:11:12.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:12.393+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.394689+0000) 2019-10-13T10:11:12.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:12.393+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.394822+0000) 2019-10-13T10:11:12.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:12.397+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.397796+0000) 2019-10-13T10:11:12.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:12.398+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 181971 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:12.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:12.968+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.968459+0000) 2019-10-13T10:11:12.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:12.966+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:12.967532+0000) 2019-10-13T10:11:14.337 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:14.337+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:14.337304+0000) 2019-10-13T10:11:14.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:14.727+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:14.727461+0000) 2019-10-13T10:11:15.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:15.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:15.617271+0000) 2019-10-13T10:11:15.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:15.616+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:15.617273+0000) 2019-10-13T10:11:17.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:17.404+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.404997+0000) 2019-10-13T10:11:17.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:17.400+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.401194+0000) 2019-10-13T10:11:17.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:17.401+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.402226+0000) 2019-10-13T10:11:17.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:17.402+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.402411+0000) 2019-10-13T10:11:17.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:17.402+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.402621+0000) 2019-10-13T10:11:17.407 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:17.402+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.402816+0000) 2019-10-13T10:11:17.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:17.402+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.403004+0000) 2019-10-13T10:11:17.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:17.405+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.406446+0000) 2019-10-13T10:11:17.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:17.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:18.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:18.526+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:18.527028+0000) 2019-10-13T10:11:18.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:18.864+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:18.865003+0000) 2019-10-13T10:11:22.412 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:22.410+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.410886+0000) 2019-10-13T10:11:22.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:22.410+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.411084+0000) 2019-10-13T10:11:22.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:22.410+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.412311+0000) 2019-10-13T10:11:22.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:22.411+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.411748+0000) 2019-10-13T10:11:22.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:22.411+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.411855+0000) 2019-10-13T10:11:22.414 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:22.411+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.412009+0000) 2019-10-13T10:11:22.414 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:22.411+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.412356+0000) 2019-10-13T10:11:22.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:22.413+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.415507+0000) 2019-10-13T10:11:22.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:22.417+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182033 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:22.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:22.967+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.968652+0000) 2019-10-13T10:11:22.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:22.967+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:22.967702+0000) 2019-10-13T10:11:24.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:24.338+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:24.339724+0000) 2019-10-13T10:11:24.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:24.727+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:24.727661+0000) 2019-10-13T10:11:25.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:25.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:25.617476+0000) 2019-10-13T10:11:25.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:25.617+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:25.617475+0000) 2019-10-13T10:11:27.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:27.419+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.419788+0000) 2019-10-13T10:11:27.422 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:27.419+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.421423+0000) 2019-10-13T10:11:27.422 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:27.420+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.420427+0000) 2019-10-13T10:11:27.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:27.420+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.420649+0000) 2019-10-13T10:11:27.423 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:27.421+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.421392+0000) 2019-10-13T10:11:27.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:27.421+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.421454+0000) 2019-10-13T10:11:27.423 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:27.421+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.421795+0000) 2019-10-13T10:11:27.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:27.422+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.424226+0000) 2019-10-13T10:11:27.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:27.428+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:28.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:28.527+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:28.527186+0000) 2019-10-13T10:11:28.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:28.865+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:28.865199+0000) 2019-10-13T10:11:32.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:32.429+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.430273+0000) 2019-10-13T10:11:32.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:32.430+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.432094+0000) 2019-10-13T10:11:32.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:32.430+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.430817+0000) 2019-10-13T10:11:32.433 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:32.430+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.431219+0000) 2019-10-13T10:11:32.433 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:32.430+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.431278+0000) 2019-10-13T10:11:32.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:32.432+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.433763+0000) 2019-10-13T10:11:32.434 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:32.431+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.431515+0000) 2019-10-13T10:11:32.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:32.431+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.431699+0000) 2019-10-13T10:11:32.439 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:32.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:32.918 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:11:32.919 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:32.924 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:11:32.924 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:32.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:32.967+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.968791+0000) 2019-10-13T10:11:32.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:32.967+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:32.967849+0000) 2019-10-13T10:11:34.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:34.341+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:34.341999+0000) 2019-10-13T10:11:34.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:34.726+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:34.727843+0000) 2019-10-13T10:11:35.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:35.616+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:35.617694+0000) 2019-10-13T10:11:35.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:35.616+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:35.617695+0000) 2019-10-13T10:11:37.441 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:37.438+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.440167+0000) 2019-10-13T10:11:37.442 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:37.439+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.440489+0000) 2019-10-13T10:11:37.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:37.441+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.441822+0000) 2019-10-13T10:11:37.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:37.439+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.441019+0000) 2019-10-13T10:11:37.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:37.439+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.441177+0000) 2019-10-13T10:11:37.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:37.439+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.441311+0000) 2019-10-13T10:11:37.444 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:37.440+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.441466+0000) 2019-10-13T10:11:37.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:37.443+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.444152+0000) 2019-10-13T10:11:37.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:37.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:38.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:38.526+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:38.527421+0000) 2019-10-13T10:11:38.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:38.864+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:38.865352+0000) 2019-10-13T10:11:42.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:42.447+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.449148+0000) 2019-10-13T10:11:42.451 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:42.447+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.449260+0000) 2019-10-13T10:11:42.451 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:42.448+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.449391+0000) 2019-10-13T10:11:42.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:42.448+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.449566+0000) 2019-10-13T10:11:42.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:42.448+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.449864+0000) 2019-10-13T10:11:42.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:42.448+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.450071+0000) 2019-10-13T10:11:42.452 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:42.450+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.450898+0000) 2019-10-13T10:11:42.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:42.452+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.453475+0000) 2019-10-13T10:11:42.457 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:42.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182155 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:42.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:42.968+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.968951+0000) 2019-10-13T10:11:42.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:42.967+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:42.968008+0000) 2019-10-13T10:11:44.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:44.343+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:44.344319+0000) 2019-10-13T10:11:44.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:44.727+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:44.728026+0000) 2019-10-13T10:11:45.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:45.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:45.617912+0000) 2019-10-13T10:11:45.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:45.617+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:45.617912+0000) 2019-10-13T10:11:47.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:47.456+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.458101+0000) 2019-10-13T10:11:47.461 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:47.458+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.459423+0000) 2019-10-13T10:11:47.461 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:47.457+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.458700+0000) 2019-10-13T10:11:47.461 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:47.457+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.458875+0000) 2019-10-13T10:11:47.462 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:47.457+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.459137+0000) 2019-10-13T10:11:47.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:47.457+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.459257+0000) 2019-10-13T10:11:47.462 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:47.458+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.459887+0000) 2019-10-13T10:11:47.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:47.461+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.462152+0000) 2019-10-13T10:11:47.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:47.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:48.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:48.526+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:48.527601+0000) 2019-10-13T10:11:48.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:48.864+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:48.865525+0000) 2019-10-13T10:11:52.468 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:52.465+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.466823+0000) 2019-10-13T10:11:52.469 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:52.465+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.466960+0000) 2019-10-13T10:11:52.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:52.465+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.467100+0000) 2019-10-13T10:11:52.469 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:52.465+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.467362+0000) 2019-10-13T10:11:52.469 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:52.466+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.467542+0000) 2019-10-13T10:11:52.470 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:52.468+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.468876+0000) 2019-10-13T10:11:52.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:52.466+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.467983+0000) 2019-10-13T10:11:52.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:52.470+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.471158+0000) 2019-10-13T10:11:52.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:52.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:52.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:52.968+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.969087+0000) 2019-10-13T10:11:52.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:52.967+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:52.968141+0000) 2019-10-13T10:11:54.347 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:54.346+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:54.346726+0000) 2019-10-13T10:11:54.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:54.727+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:54.728233+0000) 2019-10-13T10:11:55.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:55.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:55.618104+0000) 2019-10-13T10:11:55.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:55.617+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:55.618104+0000) 2019-10-13T10:11:57.478 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:57.475+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.476417+0000) 2019-10-13T10:11:57.478 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:11:57.475+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.476674+0000) 2019-10-13T10:11:57.479 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:11:57.475+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.477049+0000) 2019-10-13T10:11:57.479 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:11:57.476+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.477444+0000) 2019-10-13T10:11:57.479 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:11:57.476+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.477529+0000) 2019-10-13T10:11:57.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:57.476+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.477685+0000) 2019-10-13T10:11:57.480 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:11:57.478+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.478976+0000) 2019-10-13T10:11:57.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:11:57.479+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.480097+0000) 2019-10-13T10:11:57.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:11:57.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182250 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:11:58.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:11:58.527+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:58.527769+0000) 2019-10-13T10:11:58.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:11:58.865+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:58.865707+0000) 2019-10-13T10:12:02.486 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:02.484+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.484794+0000) 2019-10-13T10:12:02.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:02.484+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.484925+0000) 2019-10-13T10:12:02.487 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:02.485+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.485401+0000) 2019-10-13T10:12:02.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:02.485+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.485496+0000) 2019-10-13T10:12:02.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:02.485+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.485769+0000) 2019-10-13T10:12:02.487 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:02.485+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.486167+0000) 2019-10-13T10:12:02.488 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:02.486+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.486944+0000) 2019-10-13T10:12:02.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:02.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.489549+0000) 2019-10-13T10:12:02.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:02.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182277 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:02.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:02.967+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.968309+0000) 2019-10-13T10:12:02.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:02.969+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:02.969256+0000) 2019-10-13T10:12:03.039 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:12:03.039 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:03.043 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:12:03.043 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:04.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:04.348+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:04.348771+0000) 2019-10-13T10:12:04.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:04.728+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:04.728441+0000) 2019-10-13T10:12:05.619 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:05.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:05.618312+0000) 2019-10-13T10:12:05.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:05.617+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:05.618315+0000) 2019-10-13T10:12:07.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:07.492+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.493035+0000) 2019-10-13T10:12:07.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:07.492+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.493258+0000) 2019-10-13T10:12:07.495 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:07.493+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.493518+0000) 2019-10-13T10:12:07.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:07.494+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.495070+0000) 2019-10-13T10:12:07.496 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:07.493+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.494018+0000) 2019-10-13T10:12:07.496 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:07.493+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.494282+0000) 2019-10-13T10:12:07.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:07.494+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.494485+0000) 2019-10-13T10:12:07.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:07.496+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.497129+0000) 2019-10-13T10:12:07.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:07.500+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:08.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:08.527+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:08.527934+0000) 2019-10-13T10:12:08.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:08.865+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:08.865923+0000) 2019-10-13T10:12:12.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:12.502+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.502449+0000) 2019-10-13T10:12:12.504 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:12.502+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.502638+0000) 2019-10-13T10:12:12.505 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:12.502+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.503880+0000) 2019-10-13T10:12:12.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:12.502+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.503194+0000) 2019-10-13T10:12:12.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:12.502+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.503314+0000) 2019-10-13T10:12:12.505 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:12.503+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.503677+0000) 2019-10-13T10:12:12.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:12.503+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.503826+0000) 2019-10-13T10:12:12.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:12.504+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.506202+0000) 2019-10-13T10:12:12.512 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:12.510+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:12.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:12.968+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.969415+0000) 2019-10-13T10:12:12.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:12.968+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:12.968453+0000) 2019-10-13T10:12:14.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:14.350+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:14.351105+0000) 2019-10-13T10:12:14.730 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:14.728+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:14.728646+0000) 2019-10-13T10:12:15.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:15.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:15.618506+0000) 2019-10-13T10:12:15.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:15.618+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:15.618552+0000) 2019-10-13T10:12:17.514 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:17.512+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.512528+0000) 2019-10-13T10:12:17.514 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:17.512+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.513163+0000) 2019-10-13T10:12:17.515 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:17.512+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.514098+0000) 2019-10-13T10:12:17.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:17.512+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.513376+0000) 2019-10-13T10:12:17.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:17.513+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.513534+0000) 2019-10-13T10:12:17.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:17.513+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.513822+0000) 2019-10-13T10:12:17.516 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:17.513+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.514197+0000) 2019-10-13T10:12:17.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:17.515+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.516961+0000) 2019-10-13T10:12:17.521 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:17.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182372 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:18.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:18.528+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:18.528110+0000) 2019-10-13T10:12:18.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:18.864+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:18.866048+0000) 2019-10-13T10:12:22.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:22.520+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.522230+0000) 2019-10-13T10:12:22.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:22.521+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.522406+0000) 2019-10-13T10:12:22.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:22.522+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.524423+0000) 2019-10-13T10:12:22.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:22.521+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.522790+0000) 2019-10-13T10:12:22.525 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:22.521+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.523019+0000) 2019-10-13T10:12:22.525 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:22.522+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.523466+0000) 2019-10-13T10:12:22.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:22.522+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.523629+0000) 2019-10-13T10:12:22.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:22.524+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.526319+0000) 2019-10-13T10:12:22.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:22.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:22.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:22.968+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.969561+0000) 2019-10-13T10:12:22.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:22.967+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:22.968613+0000) 2019-10-13T10:12:24.353 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:24.352+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:24.353452+0000) 2019-10-13T10:12:24.730 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:24.727+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:24.728777+0000) 2019-10-13T10:12:25.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:25.617+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:25.618697+0000) 2019-10-13T10:12:25.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:25.617+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:25.618697+0000) 2019-10-13T10:12:27.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:27.530+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.532041+0000) 2019-10-13T10:12:27.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:27.530+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.532147+0000) 2019-10-13T10:12:27.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:27.531+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.532763+0000) 2019-10-13T10:12:27.535 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:27.531+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.532848+0000) 2019-10-13T10:12:27.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:27.531+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.533051+0000) 2019-10-13T10:12:27.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:27.533+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.533821+0000) 2019-10-13T10:12:27.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:27.531+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.533262+0000) 2019-10-13T10:12:27.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:27.535+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.535806+0000) 2019-10-13T10:12:27.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:27.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182432 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:28.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:28.526+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:28.528302+0000) 2019-10-13T10:12:28.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:28.865+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:28.866173+0000) 2019-10-13T10:12:32.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:32.540+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.541489+0000) 2019-10-13T10:12:32.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:32.540+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.542268+0000) 2019-10-13T10:12:32.544 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:32.542+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.543195+0000) 2019-10-13T10:12:32.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:32.541+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.542540+0000) 2019-10-13T10:12:32.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:32.541+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.542733+0000) 2019-10-13T10:12:32.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:32.541+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.543107+0000) 2019-10-13T10:12:32.545 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:32.541+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.543311+0000) 2019-10-13T10:12:32.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:32.544+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.545491+0000) 2019-10-13T10:12:32.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:32.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182459 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:32.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:32.969+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.969724+0000) 2019-10-13T10:12:32.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:32.968+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:32.968788+0000) 2019-10-13T10:12:33.203 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:12:33.203 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:33.207 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:12:33.207 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:34.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:34.355+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:34.355816+0000) 2019-10-13T10:12:34.730 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:34.728+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:34.728954+0000) 2019-10-13T10:12:35.620 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:35.618+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:35.618877+0000) 2019-10-13T10:12:35.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:35.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:35.618909+0000) 2019-10-13T10:12:37.554 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:37.550+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.551583+0000) 2019-10-13T10:12:37.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:37.550+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.551733+0000) 2019-10-13T10:12:37.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:37.550+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.551932+0000) 2019-10-13T10:12:37.554 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:37.550+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.552251+0000) 2019-10-13T10:12:37.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:37.551+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.552553+0000) 2019-10-13T10:12:37.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:37.551+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.552660+0000) 2019-10-13T10:12:37.555 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:37.552+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.553512+0000) 2019-10-13T10:12:37.555 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:37.553+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.554620+0000) 2019-10-13T10:12:37.560 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:37.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:38.529 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:38.527+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:38.528472+0000) 2019-10-13T10:12:38.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:38.865+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:38.866346+0000) 2019-10-13T10:12:42.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:42.559+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.560521+0000) 2019-10-13T10:12:42.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:42.559+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.560748+0000) 2019-10-13T10:12:42.563 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:42.561+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.562516+0000) 2019-10-13T10:12:42.563 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:42.559+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.561217+0000) 2019-10-13T10:12:42.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:42.560+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.561617+0000) 2019-10-13T10:12:42.563 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:42.560+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.561721+0000) 2019-10-13T10:12:42.564 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:42.560+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.561965+0000) 2019-10-13T10:12:42.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:42.563+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.564460+0000) 2019-10-13T10:12:42.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:42.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:42.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:42.969+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.969929+0000) 2019-10-13T10:12:42.970 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:42.968+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:42.968961+0000) 2019-10-13T10:12:44.358 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:44.357+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:44.358251+0000) 2019-10-13T10:12:44.730 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:44.728+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:44.729170+0000) 2019-10-13T10:12:45.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:45.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:45.619090+0000) 2019-10-13T10:12:45.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:45.618+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:45.619074+0000) 2019-10-13T10:12:47.570 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:47.569+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.569513+0000) 2019-10-13T10:12:47.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:47.570+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.570828+0000) 2019-10-13T10:12:47.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:47.569+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.570019+0000) 2019-10-13T10:12:47.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:47.569+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.570320+0000) 2019-10-13T10:12:47.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:47.570+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.570581+0000) 2019-10-13T10:12:47.572 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:47.570+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.570828+0000) 2019-10-13T10:12:47.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:47.570+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.571353+0000) 2019-10-13T10:12:47.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:47.573+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.573948+0000) 2019-10-13T10:12:47.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:47.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:48.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:48.528+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:48.528640+0000) 2019-10-13T10:12:48.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:48.866+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:48.866548+0000) 2019-10-13T10:12:52.580 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:52.579+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.580043+0000) 2019-10-13T10:12:52.581 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:52.578+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.579104+0000) 2019-10-13T10:12:52.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:52.578+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.579230+0000) 2019-10-13T10:12:52.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:52.579+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.579441+0000) 2019-10-13T10:12:52.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:52.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.579883+0000) 2019-10-13T10:12:52.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:52.579+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.580065+0000) 2019-10-13T10:12:52.582 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:52.581+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.581958+0000) 2019-10-13T10:12:52.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:52.580+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.580497+0000) 2019-10-13T10:12:52.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:52.586+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:52.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:52.969+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.970088+0000) 2019-10-13T10:12:52.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:52.968+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:52.969081+0000) 2019-10-13T10:12:54.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:54.360+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:54.360619+0000) 2019-10-13T10:12:54.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:54.729+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:54.729325+0000) 2019-10-13T10:12:55.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:55.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:55.619288+0000) 2019-10-13T10:12:55.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:55.618+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:55.619289+0000) 2019-10-13T10:12:57.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:12:57.588+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.588898+0000) 2019-10-13T10:12:57.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:57.588+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.589089+0000) 2019-10-13T10:12:57.591 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:12:57.589+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.590344+0000) 2019-10-13T10:12:57.592 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:12:57.589+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.589414+0000) 2019-10-13T10:12:57.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:12:57.589+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.589555+0000) 2019-10-13T10:12:57.592 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:57.589+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.590030+0000) 2019-10-13T10:12:57.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:12:57.590+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.590446+0000) 2019-10-13T10:12:57.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:12:57.592+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.592833+0000) 2019-10-13T10:12:57.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:12:57.596+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182616 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:12:58.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:12:58.528+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:58.528811+0000) 2019-10-13T10:12:58.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:12:58.866+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:58.866743+0000) 2019-10-13T10:13:02.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:02.598+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.599799+0000) 2019-10-13T10:13:02.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:02.598+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.598915+0000) 2019-10-13T10:13:02.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:02.598+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.599032+0000) 2019-10-13T10:13:02.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:02.598+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.599295+0000) 2019-10-13T10:13:02.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:02.599+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.599482+0000) 2019-10-13T10:13:02.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:02.599+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.599756+0000) 2019-10-13T10:13:02.602 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:02.599+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.599888+0000) 2019-10-13T10:13:02.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:02.601+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.602801+0000) 2019-10-13T10:13:02.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:02.605+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:02.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:02.969+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.970243+0000) 2019-10-13T10:13:02.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:02.969+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:02.969251+0000) 2019-10-13T10:13:03.307 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:13:03.308 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:03.312 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:13:03.312 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:04.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:04.361+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:04.362665+0000) 2019-10-13T10:13:04.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:04.728+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:04.729497+0000) 2019-10-13T10:13:05.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:05.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:05.619492+0000) 2019-10-13T10:13:05.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:05.618+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:05.619491+0000) 2019-10-13T10:13:07.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:07.606+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.607442+0000) 2019-10-13T10:13:07.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:07.607+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.608748+0000) 2019-10-13T10:13:07.617 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:07.606+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.608131+0000) 2019-10-13T10:13:07.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:07.607+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.608534+0000) 2019-10-13T10:13:07.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:07.607+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.608662+0000) 2019-10-13T10:13:07.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:07.610+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.612515+0000) 2019-10-13T10:13:07.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:07.607+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.608978+0000) 2019-10-13T10:13:07.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:07.608+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.609497+0000) 2019-10-13T10:13:07.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:07.614+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:08.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:08.527+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:08.529026+0000) 2019-10-13T10:13:08.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:08.865+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:08.866950+0000) 2019-10-13T10:13:12.618 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:12.616+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.618279+0000) 2019-10-13T10:13:12.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:12.615+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.617338+0000) 2019-10-13T10:13:12.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:12.616+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.617557+0000) 2019-10-13T10:13:12.619 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:12.616+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.617812+0000) 2019-10-13T10:13:12.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:12.616+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.618197+0000) 2019-10-13T10:13:12.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:12.616+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.618333+0000) 2019-10-13T10:13:12.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:12.617+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.618542+0000) 2019-10-13T10:13:12.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:12.619+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.621336+0000) 2019-10-13T10:13:12.626 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:12.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:12.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:12.969+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.970400+0000) 2019-10-13T10:13:12.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:12.968+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:12.969416+0000) 2019-10-13T10:13:14.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:14.364+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:14.365043+0000) 2019-10-13T10:13:14.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:14.728+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:14.729668+0000) 2019-10-13T10:13:15.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:15.618+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:15.619682+0000) 2019-10-13T10:13:15.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:15.618+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:15.619696+0000) 2019-10-13T10:13:17.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:17.627+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.628065+0000) 2019-10-13T10:13:17.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:17.625+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.626800+0000) 2019-10-13T10:13:17.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:17.625+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.627164+0000) 2019-10-13T10:13:17.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:17.630+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.631265+0000) 2019-10-13T10:13:17.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:17.626+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.627790+0000) 2019-10-13T10:13:17.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:17.625+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.627356+0000) 2019-10-13T10:13:17.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:17.626+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.628085+0000) 2019-10-13T10:13:17.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:17.626+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.628229+0000) 2019-10-13T10:13:17.637 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:17.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:18.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:18.528+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:18.529190+0000) 2019-10-13T10:13:18.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:18.866+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:18.867144+0000) 2019-10-13T10:13:22.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:22.635+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.636908+0000) 2019-10-13T10:13:22.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:22.635+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.637130+0000) 2019-10-13T10:13:22.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:22.635+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.637218+0000) 2019-10-13T10:13:22.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:22.637+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.637978+0000) 2019-10-13T10:13:22.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:22.636+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.637825+0000) 2019-10-13T10:13:22.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:22.636+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.637996+0000) 2019-10-13T10:13:22.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:22.636+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.638113+0000) 2019-10-13T10:13:22.641 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:22.640+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.641074+0000) 2019-10-13T10:13:22.646 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:22.643+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:22.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:22.969+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.970557+0000) 2019-10-13T10:13:22.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:22.968+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:22.969585+0000) 2019-10-13T10:13:24.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:24.366+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:24.367518+0000) 2019-10-13T10:13:24.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:24.729+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:24.729863+0000) 2019-10-13T10:13:25.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:25.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:25.619900+0000) 2019-10-13T10:13:25.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:25.619+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:25.619899+0000) 2019-10-13T10:13:27.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:27.645+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.646408+0000) 2019-10-13T10:13:27.648 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:27.647+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.647863+0000) 2019-10-13T10:13:27.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:27.645+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.647137+0000) 2019-10-13T10:13:27.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:27.646+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.647450+0000) 2019-10-13T10:13:27.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:27.645+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.647370+0000) 2019-10-13T10:13:27.649 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:27.646+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.647724+0000) 2019-10-13T10:13:27.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:27.646+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.648044+0000) 2019-10-13T10:13:27.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:27.650+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.651098+0000) 2019-10-13T10:13:27.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:27.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182798 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:28.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:28.528+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:28.529333+0000) 2019-10-13T10:13:28.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:28.866+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:28.867360+0000) 2019-10-13T10:13:32.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:32.655+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.655858+0000) 2019-10-13T10:13:32.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:32.656+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.657144+0000) 2019-10-13T10:13:32.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:32.655+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.656191+0000) 2019-10-13T10:13:32.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:32.656+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.656661+0000) 2019-10-13T10:13:32.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:32.656+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.656538+0000) 2019-10-13T10:13:32.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:32.656+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.656912+0000) 2019-10-13T10:13:32.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:32.657+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.657443+0000) 2019-10-13T10:13:32.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:32.658+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.659587+0000) 2019-10-13T10:13:32.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:32.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:32.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:32.969+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.969763+0000) 2019-10-13T10:13:32.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:32.970+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:32.970719+0000) 2019-10-13T10:13:33.478 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:13:33.478 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:33.483 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:13:33.483 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:34.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:34.369+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:34.369825+0000) 2019-10-13T10:13:34.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:34.729+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:34.730065+0000) 2019-10-13T10:13:35.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:35.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:35.620108+0000) 2019-10-13T10:13:35.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:35.619+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:35.620108+0000) 2019-10-13T10:13:37.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:37.664+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.665379+0000) 2019-10-13T10:13:37.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:37.665+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.665546+0000) 2019-10-13T10:13:37.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:37.665+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.665871+0000) 2019-10-13T10:13:37.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:37.666+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.667108+0000) 2019-10-13T10:13:37.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:37.665+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.666192+0000) 2019-10-13T10:13:37.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:37.666+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.666464+0000) 2019-10-13T10:13:37.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:37.666+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.666719+0000) 2019-10-13T10:13:37.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:37.669+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.670282+0000) 2019-10-13T10:13:37.675 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:37.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182860 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:38.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:38.529+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:38.529528+0000) 2019-10-13T10:13:38.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:38.867+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:38.867556+0000) 2019-10-13T10:13:42.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:42.674+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.675317+0000) 2019-10-13T10:13:42.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:42.675+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.675567+0000) 2019-10-13T10:13:42.677 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:42.675+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.675801+0000) 2019-10-13T10:13:42.677 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:42.675+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.675883+0000) 2019-10-13T10:13:42.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:42.676+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.676737+0000) 2019-10-13T10:13:42.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:42.675+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.676067+0000) 2019-10-13T10:13:42.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:42.676+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.676611+0000) 2019-10-13T10:13:42.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:42.678+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.679338+0000) 2019-10-13T10:13:42.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:42.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:42.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:42.970+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.970883+0000) 2019-10-13T10:13:42.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:42.969+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:42.969986+0000) 2019-10-13T10:13:44.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:44.372+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:44.372225+0000) 2019-10-13T10:13:44.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:44.730+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:44.730232+0000) 2019-10-13T10:13:45.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:45.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:45.620363+0000) 2019-10-13T10:13:45.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:45.619+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:45.620363+0000) 2019-10-13T10:13:47.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:47.684+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.684611+0000) 2019-10-13T10:13:47.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:47.684+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.684851+0000) 2019-10-13T10:13:47.686 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:47.684+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.684987+0000) 2019-10-13T10:13:47.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:47.685+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.686558+0000) 2019-10-13T10:13:47.687 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:47.685+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.685637+0000) 2019-10-13T10:13:47.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:47.685+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.685909+0000) 2019-10-13T10:13:47.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:47.685+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.685973+0000) 2019-10-13T10:13:47.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:47.688+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.689480+0000) 2019-10-13T10:13:47.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:47.691+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182920 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:48.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:48.529+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:48.529666+0000) 2019-10-13T10:13:48.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:48.867+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:48.867778+0000) 2019-10-13T10:13:52.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:52.692+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.693545+0000) 2019-10-13T10:13:52.695 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:52.692+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.693644+0000) 2019-10-13T10:13:52.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:52.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.693892+0000) 2019-10-13T10:13:52.696 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:52.692+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.693993+0000) 2019-10-13T10:13:52.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:52.692+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.694145+0000) 2019-10-13T10:13:52.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:52.692+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.694290+0000) 2019-10-13T10:13:52.697 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:52.694+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.695600+0000) 2019-10-13T10:13:52.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:52.695+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.696435+0000) 2019-10-13T10:13:52.702 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:52.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:52.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:52.971+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.971045+0000) 2019-10-13T10:13:52.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:52.969+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:52.970146+0000) 2019-10-13T10:13:54.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:54.373+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:54.374656+0000) 2019-10-13T10:13:54.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:54.729+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:54.730442+0000) 2019-10-13T10:13:55.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:55.619+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:55.620552+0000) 2019-10-13T10:13:55.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:55.619+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:55.620551+0000) 2019-10-13T10:13:57.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:13:57.701+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.703176+0000) 2019-10-13T10:13:57.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:13:57.701+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.702781+0000) 2019-10-13T10:13:57.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:13:57.701+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.702740+0000) 2019-10-13T10:13:57.706 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:57.702+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.703464+0000) 2019-10-13T10:13:57.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:13:57.702+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.703412+0000) 2019-10-13T10:13:57.706 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:57.702+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.703587+0000) 2019-10-13T10:13:57.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:13:57.703+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.704902+0000) 2019-10-13T10:13:57.707 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:13:57.705+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.707136+0000) 2019-10-13T10:13:57.710 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:13:57.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 182982 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:13:58.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:13:58.528+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:58.529808+0000) 2019-10-13T10:13:58.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:13:58.866+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:58.867966+0000) 2019-10-13T10:14:02.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:02.710+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.712360+0000) 2019-10-13T10:14:02.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:02.709+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.711048+0000) 2019-10-13T10:14:02.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:02.710+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.711439+0000) 2019-10-13T10:14:02.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:02.710+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.711660+0000) 2019-10-13T10:14:02.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:02.710+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.712050+0000) 2019-10-13T10:14:02.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:02.710+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.712277+0000) 2019-10-13T10:14:02.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:02.711+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.712425+0000) 2019-10-13T10:14:02.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:02.713+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.715391+0000) 2019-10-13T10:14:02.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:02.717+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183009 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:02.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:02.970+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.971183+0000) 2019-10-13T10:14:02.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:02.969+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:02.970318+0000) 2019-10-13T10:14:03.632 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:14:03.632 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:03.636 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:14:03.637 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:04.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:04.376+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:04.376946+0000) 2019-10-13T10:14:04.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:04.729+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:04.730663+0000) 2019-10-13T10:14:05.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:05.620+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:05.620748+0000) 2019-10-13T10:14:05.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:05.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:05.620749+0000) 2019-10-13T10:14:07.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:07.718+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.720346+0000) 2019-10-13T10:14:07.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:07.719+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.720827+0000) 2019-10-13T10:14:07.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:07.719+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.721086+0000) 2019-10-13T10:14:07.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:07.721+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.722625+0000) 2019-10-13T10:14:07.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:07.720+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.721668+0000) 2019-10-13T10:14:07.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:07.720+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.721838+0000) 2019-10-13T10:14:07.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:07.720+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.722165+0000) 2019-10-13T10:14:07.724 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:07.723+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.724270+0000) 2019-10-13T10:14:07.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:07.726+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183042 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:08.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:08.528+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:08.529996+0000) 2019-10-13T10:14:08.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:08.867+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:08.868196+0000) 2019-10-13T10:14:12.731 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:12.728+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.730040+0000) 2019-10-13T10:14:12.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:12.729+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.730417+0000) 2019-10-13T10:14:12.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:12.730+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.731553+0000) 2019-10-13T10:14:12.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:12.730+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.731675+0000) 2019-10-13T10:14:12.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:12.730+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.731871+0000) 2019-10-13T10:14:12.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:12.732+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.733657+0000) 2019-10-13T10:14:12.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:12.730+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.732009+0000) 2019-10-13T10:14:12.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:12.733+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.734334+0000) 2019-10-13T10:14:12.740 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:12.737+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183069 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:12.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:12.970+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.971265+0000) 2019-10-13T10:14:12.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:12.969+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:12.970468+0000) 2019-10-13T10:14:14.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:14.378+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:14.379175+0000) 2019-10-13T10:14:14.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:14.729+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:14.730848+0000) 2019-10-13T10:14:15.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:15.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:15.620945+0000) 2019-10-13T10:14:15.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:15.620+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:15.620945+0000) 2019-10-13T10:14:17.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:17.739+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.740691+0000) 2019-10-13T10:14:17.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:17.739+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.740853+0000) 2019-10-13T10:14:17.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:17.739+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.741015+0000) 2019-10-13T10:14:17.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:17.741+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.742424+0000) 2019-10-13T10:14:17.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:17.740+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.741855+0000) 2019-10-13T10:14:17.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:17.740+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.742215+0000) 2019-10-13T10:14:17.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:17.741+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.742440+0000) 2019-10-13T10:14:17.745 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:17.744+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.744917+0000) 2019-10-13T10:14:17.749 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:17.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:18.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:18.529+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:18.530165+0000) 2019-10-13T10:14:18.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:18.868+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:18.868409+0000) 2019-10-13T10:14:22.751 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:22.750+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.751365+0000) 2019-10-13T10:14:22.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:22.750+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.750505+0000) 2019-10-13T10:14:22.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:22.750+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.750625+0000) 2019-10-13T10:14:22.753 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:22.750+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.750826+0000) 2019-10-13T10:14:22.753 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:22.750+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.751090+0000) 2019-10-13T10:14:22.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:22.750+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.751222+0000) 2019-10-13T10:14:22.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:22.751+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.751409+0000) 2019-10-13T10:14:22.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:22.754+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.754798+0000) 2019-10-13T10:14:22.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:22.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183131 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:22.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:22.970+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.971404+0000) 2019-10-13T10:14:22.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:22.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:22.970630+0000) 2019-10-13T10:14:24.381 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:24.380+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:24.381417+0000) 2019-10-13T10:14:24.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:24.730+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:24.731071+0000) 2019-10-13T10:14:25.622 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:25.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:25.621144+0000) 2019-10-13T10:14:25.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:25.620+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:25.621139+0000) 2019-10-13T10:14:27.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:27.759+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.759626+0000) 2019-10-13T10:14:27.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:27.759+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.759974+0000) 2019-10-13T10:14:27.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:27.760+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.761404+0000) 2019-10-13T10:14:27.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:27.759+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.760203+0000) 2019-10-13T10:14:27.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:27.760+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.760499+0000) 2019-10-13T10:14:27.763 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:27.760+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.760751+0000) 2019-10-13T10:14:27.763 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:27.760+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.761103+0000) 2019-10-13T10:14:27.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:27.762+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.762920+0000) 2019-10-13T10:14:27.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:27.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:28.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:28.530+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:28.530365+0000) 2019-10-13T10:14:28.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:28.868+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:28.868598+0000) 2019-10-13T10:14:32.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:32.769+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.769431+0000) 2019-10-13T10:14:32.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:32.769+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.769749+0000) 2019-10-13T10:14:32.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:32.769+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.769889+0000) 2019-10-13T10:14:32.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:32.769+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.770262+0000) 2019-10-13T10:14:32.773 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:32.769+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.770385+0000) 2019-10-13T10:14:32.773 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:32.770+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.770566+0000) 2019-10-13T10:14:32.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:32.770+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.770745+0000) 2019-10-13T10:14:32.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:32.771+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.772588+0000) 2019-10-13T10:14:32.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:32.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183190 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:32.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:32.971+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.971560+0000) 2019-10-13T10:14:32.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:32.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:32.970724+0000) 2019-10-13T10:14:33.743 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:14:33.743 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:33.748 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:14:33.748 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:34.385 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:34.383+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:34.383747+0000) 2019-10-13T10:14:34.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:34.731+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:34.731264+0000) 2019-10-13T10:14:35.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:35.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:35.621378+0000) 2019-10-13T10:14:35.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:35.620+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:35.621379+0000) 2019-10-13T10:14:37.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:37.777+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.778585+0000) 2019-10-13T10:14:37.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:37.777+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.778745+0000) 2019-10-13T10:14:37.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:37.777+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.779119+0000) 2019-10-13T10:14:37.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:37.777+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.779205+0000) 2019-10-13T10:14:37.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:37.779+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.780534+0000) 2019-10-13T10:14:37.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:37.778+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.779584+0000) 2019-10-13T10:14:37.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:37.778+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.779821+0000) 2019-10-13T10:14:37.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:37.782+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.783167+0000) 2019-10-13T10:14:37.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:37.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:38.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:38.529+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:38.530521+0000) 2019-10-13T10:14:38.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:38.867+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:38.868787+0000) 2019-10-13T10:14:42.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:42.786+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.787821+0000) 2019-10-13T10:14:42.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:42.786+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.787971+0000) 2019-10-13T10:14:42.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:42.786+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.788163+0000) 2019-10-13T10:14:42.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:42.787+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.788791+0000) 2019-10-13T10:14:42.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:42.787+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.789043+0000) 2019-10-13T10:14:42.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:42.787+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.789167+0000) 2019-10-13T10:14:42.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:42.789+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.790622+0000) 2019-10-13T10:14:42.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:42.791+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.792203+0000) 2019-10-13T10:14:42.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:42.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:42.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:42.971+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.971700+0000) 2019-10-13T10:14:42.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:42.969+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:42.970840+0000) 2019-10-13T10:14:44.386 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:44.385+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:44.386098+0000) 2019-10-13T10:14:44.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:44.730+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:44.731459+0000) 2019-10-13T10:14:45.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:45.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:45.621610+0000) 2019-10-13T10:14:45.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:45.620+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:45.621611+0000) 2019-10-13T10:14:47.799 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:47.796+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.797935+0000) 2019-10-13T10:14:47.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:47.796+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.798062+0000) 2019-10-13T10:14:47.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:47.796+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.798273+0000) 2019-10-13T10:14:47.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:47.797+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.799541+0000) 2019-10-13T10:14:47.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:47.797+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.798663+0000) 2019-10-13T10:14:47.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:47.797+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.799009+0000) 2019-10-13T10:14:47.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:47.797+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.799358+0000) 2019-10-13T10:14:47.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:47.799+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.800995+0000) 2019-10-13T10:14:47.806 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:47.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:48.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:48.529+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:48.530731+0000) 2019-10-13T10:14:48.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:48.867+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:48.868909+0000) 2019-10-13T10:14:52.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:52.805+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.806969+0000) 2019-10-13T10:14:52.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:52.805+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.807177+0000) 2019-10-13T10:14:52.809 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:52.806+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.807651+0000) 2019-10-13T10:14:52.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:52.806+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.807876+0000) 2019-10-13T10:14:52.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:52.806+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.808163+0000) 2019-10-13T10:14:52.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:52.807+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.808505+0000) 2019-10-13T10:14:52.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:52.807+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.809472+0000) 2019-10-13T10:14:52.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:52.809+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.810741+0000) 2019-10-13T10:14:52.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:52.813+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183310 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:52.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:52.970+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.971844+0000) 2019-10-13T10:14:52.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:52.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:52.970956+0000) 2019-10-13T10:14:54.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:54.387+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:54.388422+0000) 2019-10-13T10:14:54.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:54.730+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:54.731613+0000) 2019-10-13T10:14:55.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:55.620+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:55.621801+0000) 2019-10-13T10:14:55.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:55.620+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:55.621817+0000) 2019-10-13T10:14:57.818 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:57.815+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.816531+0000) 2019-10-13T10:14:57.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:14:57.815+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.816707+0000) 2019-10-13T10:14:57.818 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:14:57.815+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.816938+0000) 2019-10-13T10:14:57.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:14:57.817+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.819161+0000) 2019-10-13T10:14:57.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:14:57.815+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.817251+0000) 2019-10-13T10:14:57.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:14:57.816+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.817426+0000) 2019-10-13T10:14:57.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:57.816+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.817670+0000) 2019-10-13T10:14:57.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:14:57.818+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.820356+0000) 2019-10-13T10:14:57.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:14:57.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:14:58.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:14:58.530+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:58.530881+0000) 2019-10-13T10:14:58.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:14:58.867+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:58.869090+0000) 2019-10-13T10:15:02.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:02.824+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.826266+0000) 2019-10-13T10:15:02.828 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:02.825+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.826607+0000) 2019-10-13T10:15:02.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:02.826+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.827526+0000) 2019-10-13T10:15:02.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:02.825+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.826827+0000) 2019-10-13T10:15:02.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:02.825+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.827017+0000) 2019-10-13T10:15:02.829 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:02.825+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.827314+0000) 2019-10-13T10:15:02.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:02.826+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.827447+0000) 2019-10-13T10:15:02.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:02.830+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.830964+0000) 2019-10-13T10:15:02.834 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:02.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:02.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:02.971+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.971992+0000) 2019-10-13T10:15:02.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:02.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:02.971118+0000) 2019-10-13T10:15:03.873 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:15:03.873 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:03.877 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:15:03.878 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:04.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:04.390+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:04.390767+0000) 2019-10-13T10:15:04.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:04.731+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:04.731759+0000) 2019-10-13T10:15:05.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:05.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:05.622009+0000) 2019-10-13T10:15:05.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:05.621+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:05.622030+0000) 2019-10-13T10:15:07.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:07.834+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.835277+0000) 2019-10-13T10:15:07.837 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:07.835+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.835460+0000) 2019-10-13T10:15:07.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:07.835+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.835916+0000) 2019-10-13T10:15:07.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:07.836+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.837336+0000) 2019-10-13T10:15:07.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:07.836+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.836520+0000) 2019-10-13T10:15:07.838 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:07.836+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.836630+0000) 2019-10-13T10:15:07.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:07.836+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.837088+0000) 2019-10-13T10:15:07.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:07.838+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.839679+0000) 2019-10-13T10:15:07.845 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:07.843+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:08.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:08.530+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:08.531033+0000) 2019-10-13T10:15:08.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:08.868+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:08.869241+0000) 2019-10-13T10:15:12.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:12.845+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.846287+0000) 2019-10-13T10:15:12.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:12.845+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.845779+0000) 2019-10-13T10:15:12.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:12.845+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.845938+0000) 2019-10-13T10:15:12.847 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:12.845+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.846184+0000) 2019-10-13T10:15:12.848 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:12.845+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.846296+0000) 2019-10-13T10:15:12.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:12.846+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.846459+0000) 2019-10-13T10:15:12.848 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:12.846+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.846831+0000) 2019-10-13T10:15:12.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:12.848+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.849622+0000) 2019-10-13T10:15:12.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:12.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:12.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:12.971+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.972134+0000) 2019-10-13T10:15:12.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:12.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:12.971250+0000) 2019-10-13T10:15:14.393 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:14.392+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:14.393069+0000) 2019-10-13T10:15:14.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:14.731+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:14.731928+0000) 2019-10-13T10:15:15.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:15.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:15.622181+0000) 2019-10-13T10:15:15.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:15.621+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:15.622168+0000) 2019-10-13T10:15:17.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:17.854+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.854495+0000) 2019-10-13T10:15:17.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:17.855+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.856096+0000) 2019-10-13T10:15:17.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:17.855+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.855394+0000) 2019-10-13T10:15:17.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:17.855+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.855508+0000) 2019-10-13T10:15:17.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:17.855+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.855733+0000) 2019-10-13T10:15:17.857 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:17.855+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.855923+0000) 2019-10-13T10:15:17.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:17.855+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.856356+0000) 2019-10-13T10:15:17.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:17.858+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.859476+0000) 2019-10-13T10:15:17.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:17.862+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:18.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:18.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:18.531210+0000) 2019-10-13T10:15:18.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:18.869+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:18.869405+0000) 2019-10-13T10:15:22.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:22.863+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.864767+0000) 2019-10-13T10:15:22.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:22.863+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.865012+0000) 2019-10-13T10:15:22.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:22.864+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.865447+0000) 2019-10-13T10:15:22.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:22.865+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.866315+0000) 2019-10-13T10:15:22.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:22.864+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.866003+0000) 2019-10-13T10:15:22.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:22.864+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.866241+0000) 2019-10-13T10:15:22.868 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:22.865+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.866460+0000) 2019-10-13T10:15:22.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:22.869+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.869921+0000) 2019-10-13T10:15:22.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:22.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:22.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:22.972+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.972322+0000) 2019-10-13T10:15:22.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:22.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:22.971411+0000) 2019-10-13T10:15:24.395 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:24.394+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:24.395036+0000) 2019-10-13T10:15:24.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:24.731+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:24.732114+0000) 2019-10-13T10:15:25.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:25.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:25.622339+0000) 2019-10-13T10:15:25.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:25.621+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:25.622325+0000) 2019-10-13T10:15:27.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:27.873+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.874429+0000) 2019-10-13T10:15:27.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:27.873+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.874594+0000) 2019-10-13T10:15:27.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:27.873+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.875208+0000) 2019-10-13T10:15:27.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:27.873+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.875307+0000) 2019-10-13T10:15:27.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:27.876+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.876798+0000) 2019-10-13T10:15:27.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:27.874+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.876062+0000) 2019-10-13T10:15:27.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:27.875+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.876787+0000) 2019-10-13T10:15:27.879 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:27.878+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.879139+0000) 2019-10-13T10:15:27.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:27.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:28.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:28.530+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:28.531356+0000) 2019-10-13T10:15:28.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:28.868+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:28.869551+0000) 2019-10-13T10:15:32.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:32.885+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.885988+0000) 2019-10-13T10:15:32.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:32.883+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.885169+0000) 2019-10-13T10:15:32.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:32.883+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.885333+0000) 2019-10-13T10:15:32.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:32.884+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.885532+0000) 2019-10-13T10:15:32.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:32.884+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.885715+0000) 2019-10-13T10:15:32.888 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:32.884+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.886036+0000) 2019-10-13T10:15:32.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:32.884+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.886297+0000) 2019-10-13T10:15:32.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:32.888+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.889040+0000) 2019-10-13T10:15:32.893 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:32.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183550 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:32.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:32.972+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.972472+0000) 2019-10-13T10:15:32.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:32.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:32.971581+0000) 2019-10-13T10:15:34.162 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:15:34.162 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:34.167 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:15:34.167 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:34.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:34.397+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:34.397360+0000) 2019-10-13T10:15:34.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:34.731+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:34.732278+0000) 2019-10-13T10:15:35.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:35.621+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:35.622516+0000) 2019-10-13T10:15:35.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:35.621+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:35.622517+0000) 2019-10-13T10:15:37.896 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:37.893+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.894430+0000) 2019-10-13T10:15:37.896 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:37.892+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.894288+0000) 2019-10-13T10:15:37.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:37.893+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.894508+0000) 2019-10-13T10:15:37.897 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:37.893+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.894709+0000) 2019-10-13T10:15:37.897 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:37.893+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.895004+0000) 2019-10-13T10:15:37.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:37.893+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.895212+0000) 2019-10-13T10:15:37.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:37.894+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.895904+0000) 2019-10-13T10:15:37.898 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:37.896+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.898600+0000) 2019-10-13T10:15:37.902 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:37.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:38.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:38.530+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:38.531526+0000) 2019-10-13T10:15:38.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:38.869+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:38.869715+0000) 2019-10-13T10:15:42.904 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:42.901+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.902993+0000) 2019-10-13T10:15:42.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:42.901+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.903255+0000) 2019-10-13T10:15:42.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:42.902+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.904487+0000) 2019-10-13T10:15:42.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:42.902+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.903615+0000) 2019-10-13T10:15:42.905 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:42.902+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.904035+0000) 2019-10-13T10:15:42.906 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:42.902+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.904113+0000) 2019-10-13T10:15:42.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:42.903+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.904523+0000) 2019-10-13T10:15:42.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:42.905+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.906998+0000) 2019-10-13T10:15:42.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:42.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:42.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:42.971+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.972660+0000) 2019-10-13T10:15:42.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:42.971+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:42.971748+0000) 2019-10-13T10:15:44.400 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:44.398+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:44.399843+0000) 2019-10-13T10:15:44.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:44.731+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:44.732477+0000) 2019-10-13T10:15:45.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:45.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:45.622669+0000) 2019-10-13T10:15:45.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:45.622+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:45.622669+0000) 2019-10-13T10:15:47.914 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:47.910+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.912372+0000) 2019-10-13T10:15:47.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:47.911+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.912706+0000) 2019-10-13T10:15:47.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:47.911+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.912859+0000) 2019-10-13T10:15:47.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:47.911+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.913009+0000) 2019-10-13T10:15:47.915 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:47.912+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.914293+0000) 2019-10-13T10:15:47.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:47.912+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.913675+0000) 2019-10-13T10:15:47.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:47.912+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.913894+0000) 2019-10-13T10:15:47.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:47.914+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.915878+0000) 2019-10-13T10:15:47.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:47.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:48.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:48.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:48.531695+0000) 2019-10-13T10:15:48.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:48.869+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:48.869856+0000) 2019-10-13T10:15:52.924 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:52.923+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.923853+0000) 2019-10-13T10:15:52.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:52.922+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.922917+0000) 2019-10-13T10:15:52.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:52.922+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.923092+0000) 2019-10-13T10:15:52.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:52.923+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.923825+0000) 2019-10-13T10:15:52.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:52.923+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.923935+0000) 2019-10-13T10:15:52.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:52.923+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.924373+0000) 2019-10-13T10:15:52.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:52.924+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.924751+0000) 2019-10-13T10:15:52.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:52.925+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.926677+0000) 2019-10-13T10:15:52.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:52.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:52.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:52.972+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.972817+0000) 2019-10-13T10:15:52.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:52.971+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:52.971880+0000) 2019-10-13T10:15:54.402 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:54.401+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:54.402096+0000) 2019-10-13T10:15:54.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:54.732+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:54.732632+0000) 2019-10-13T10:15:55.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:55.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:55.622870+0000) 2019-10-13T10:15:55.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:55.622+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:55.622870+0000) 2019-10-13T10:15:57.934 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:15:57.932+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.932737+0000) 2019-10-13T10:15:57.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:15:57.932+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.932899+0000) 2019-10-13T10:15:57.935 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:57.932+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.933032+0000) 2019-10-13T10:15:57.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:15:57.932+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.933282+0000) 2019-10-13T10:15:57.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:15:57.933+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.934500+0000) 2019-10-13T10:15:57.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:15:57.934+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.935367+0000) 2019-10-13T10:15:57.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:15:57.933+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.933792+0000) 2019-10-13T10:15:57.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:57.933+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.933965+0000) 2019-10-13T10:15:57.941 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:15:57.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:15:58.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:15:58.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:58.531892+0000) 2019-10-13T10:15:58.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:15:58.869+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:58.870012+0000) 2019-10-13T10:16:02.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:02.942+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.942860+0000) 2019-10-13T10:16:03.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:02.941+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.941886+0000) 2019-10-13T10:16:03.191 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:02.941+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.942006+0000) 2019-10-13T10:16:03.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:02.942+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.942411+0000) 2019-10-13T10:16:03.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:02.942+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.942594+0000) 2019-10-13T10:16:03.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:02.942+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.942887+0000) 2019-10-13T10:16:03.191 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:02.971+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.972015+0000) 2019-10-13T10:16:03.192 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:02.942+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.943085+0000) 2019-10-13T10:16:03.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:02.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:03.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:02.945+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.946508+0000) 2019-10-13T10:16:03.192 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:02.972+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:02.972965+0000) 2019-10-13T10:16:04.333 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:16:04.333 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:04.338 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:16:04.338 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:04.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:04.403+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:04.404447+0000) 2019-10-13T10:16:04.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:04.732+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:04.732803+0000) 2019-10-13T10:16:05.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:05.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:05.623011+0000) 2019-10-13T10:16:05.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:05.622+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:05.623045+0000) 2019-10-13T10:16:07.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:07.952+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.952895+0000) 2019-10-13T10:16:07.954 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:07.951+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.951966+0000) 2019-10-13T10:16:07.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:07.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.952276+0000) 2019-10-13T10:16:07.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:07.951+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.952384+0000) 2019-10-13T10:16:07.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:07.952+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.952549+0000) 2019-10-13T10:16:07.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:07.952+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.952749+0000) 2019-10-13T10:16:07.955 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:07.952+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.952986+0000) 2019-10-13T10:16:07.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:07.955+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.956030+0000) 2019-10-13T10:16:07.961 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:07.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:08.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:08.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:08.532061+0000) 2019-10-13T10:16:08.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:08.869+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:08.870221+0000) 2019-10-13T10:16:12.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:12.960+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.961719+0000) 2019-10-13T10:16:12.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:12.960+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.961968+0000) 2019-10-13T10:16:12.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:12.960+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.962152+0000) 2019-10-13T10:16:12.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:12.962+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.963590+0000) 2019-10-13T10:16:12.964 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:12.961+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.962698+0000) 2019-10-13T10:16:12.965 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:12.961+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.963064+0000) 2019-10-13T10:16:12.965 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:12.961+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.963276+0000) 2019-10-13T10:16:12.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:12.966+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.966877+0000) 2019-10-13T10:16:12.971 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:12.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:12.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:12.972+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.973088+0000) 2019-10-13T10:16:12.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:12.970+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.972106+0000) 2019-10-13T10:16:14.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:14.406+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:14.406685+0000) 2019-10-13T10:16:14.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:14.732+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:14.733010+0000) 2019-10-13T10:16:15.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:15.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:15.623184+0000) 2019-10-13T10:16:15.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:15.622+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:15.623201+0000) 2019-10-13T10:16:17.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:17.970+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.971537+0000) 2019-10-13T10:16:17.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:17.970+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.971702+0000) 2019-10-13T10:16:17.973 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:17.970+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.971921+0000) 2019-10-13T10:16:17.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:17.972+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.973340+0000) 2019-10-13T10:16:17.974 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:17.971+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.972437+0000) 2019-10-13T10:16:17.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:17.971+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.972658+0000) 2019-10-13T10:16:17.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:17.971+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.972955+0000) 2019-10-13T10:16:17.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:17.976+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.977098+0000) 2019-10-13T10:16:17.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:17.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:18.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:18.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:18.532228+0000) 2019-10-13T10:16:18.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:18.869+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:18.870407+0000) 2019-10-13T10:16:22.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:22.973+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.973243+0000) 2019-10-13T10:16:22.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:22.971+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.972788+0000) 2019-10-13T10:16:22.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:22.979+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.981372+0000) 2019-10-13T10:16:22.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:22.980+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.981703+0000) 2019-10-13T10:16:22.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:22.980+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.981595+0000) 2019-10-13T10:16:22.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:22.982+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.983453+0000) 2019-10-13T10:16:22.984 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:22.980+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.982123+0000) 2019-10-13T10:16:22.984 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:22.980+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.981936+0000) 2019-10-13T10:16:22.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:22.981+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.982574+0000) 2019-10-13T10:16:22.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:22.983+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.984370+0000) 2019-10-13T10:16:22.990 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:22.987+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183851 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:24.409 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:24.408+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:24.408649+0000) 2019-10-13T10:16:24.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:24.732+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:24.733203+0000) 2019-10-13T10:16:25.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:25.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:25.623357+0000) 2019-10-13T10:16:25.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:25.622+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:25.623362+0000) 2019-10-13T10:16:27.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:27.989+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.991113+0000) 2019-10-13T10:16:27.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:27.989+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.991358+0000) 2019-10-13T10:16:27.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:27.991+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.992947+0000) 2019-10-13T10:16:27.994 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:27.990+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.991715+0000) 2019-10-13T10:16:27.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:27.990+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.991699+0000) 2019-10-13T10:16:27.994 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:27.990+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.992150+0000) 2019-10-13T10:16:27.994 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:27.990+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.992323+0000) 2019-10-13T10:16:27.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:27.992+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.994246+0000) 2019-10-13T10:16:27.999 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:27.996+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183884 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:28.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:28.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:28.532408+0000) 2019-10-13T10:16:28.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:28.869+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:28.870549+0000) 2019-10-13T10:16:32.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:32.972+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.973420+0000) 2019-10-13T10:16:32.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:32.972+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.972941+0000) 2019-10-13T10:16:33.001 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:32.998+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.999463+0000) 2019-10-13T10:16:33.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:32.998+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.999711+0000) 2019-10-13T10:16:33.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:32.999+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:33.001277+0000) 2019-10-13T10:16:33.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:32.998+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:33.000284+0000) 2019-10-13T10:16:33.002 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:32.999+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:33.000600+0000) 2019-10-13T10:16:33.002 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:32.999+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:33.000751+0000) 2019-10-13T10:16:33.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:32.999+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:33.000985+0000) 2019-10-13T10:16:33.003 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:33.001+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:33.003202+0000) 2019-10-13T10:16:33.008 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:33.005+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183910 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:34.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:34.410+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:34.411015+0000) 2019-10-13T10:16:34.512 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:16:34.513 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:34.517 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:16:34.517 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:34.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:34.732+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:34.733440+0000) 2019-10-13T10:16:35.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:35.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:35.623515+0000) 2019-10-13T10:16:35.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:35.623+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:35.623551+0000) 2019-10-13T10:16:38.011 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:38.008+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.009319+0000) 2019-10-13T10:16:38.011 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:38.008+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.009374+0000) 2019-10-13T10:16:38.011 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:38.009+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.009538+0000) 2019-10-13T10:16:38.012 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:38.009+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.009755+0000) 2019-10-13T10:16:38.012 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:38.009+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.009982+0000) 2019-10-13T10:16:38.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:38.009+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.011218+0000) 2019-10-13T10:16:38.013 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:38.010+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.012658+0000) 2019-10-13T10:16:38.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:38.010+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.010487+0000) 2019-10-13T10:16:38.018 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:38.016+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:38.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:38.532+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.532599+0000) 2019-10-13T10:16:38.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:38.870+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.870721+0000) 2019-10-13T10:16:42.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:42.972+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.973564+0000) 2019-10-13T10:16:42.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:42.972+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.973108+0000) 2019-10-13T10:16:43.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:43.018+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.019596+0000) 2019-10-13T10:16:43.020 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:43.018+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.018761+0000) 2019-10-13T10:16:43.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:43.019+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.019405+0000) 2019-10-13T10:16:43.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:43.019+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.019564+0000) 2019-10-13T10:16:43.021 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:43.019+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.019741+0000) 2019-10-13T10:16:43.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:43.019+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.019906+0000) 2019-10-13T10:16:43.022 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:43.019+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.020170+0000) 2019-10-13T10:16:43.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:43.021+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:43.022252+0000) 2019-10-13T10:16:43.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:43.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 183971 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:44.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:44.412+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:44.413468+0000) 2019-10-13T10:16:44.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:44.733+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:44.733620+0000) 2019-10-13T10:16:45.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:45.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:45.623684+0000) 2019-10-13T10:16:45.625 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:45.623+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:45.623696+0000) 2019-10-13T10:16:48.030 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:48.028+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.028412+0000) 2019-10-13T10:16:48.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:48.028+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.028623+0000) 2019-10-13T10:16:48.030 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:48.028+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.028741+0000) 2019-10-13T10:16:48.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:48.029+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.030174+0000) 2019-10-13T10:16:48.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:48.028+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.029287+0000) 2019-10-13T10:16:48.032 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:48.029+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.029671+0000) 2019-10-13T10:16:48.032 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:48.029+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.030042+0000) 2019-10-13T10:16:48.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:48.031+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.032054+0000) 2019-10-13T10:16:48.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:48.035+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:48.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:48.532+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.532746+0000) 2019-10-13T10:16:48.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:48.870+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.870864+0000) 2019-10-13T10:16:52.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:52.973+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.973761+0000) 2019-10-13T10:16:52.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:52.972+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.973241+0000) 2019-10-13T10:16:53.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:53.038+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.038840+0000) 2019-10-13T10:16:53.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:53.037+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.037942+0000) 2019-10-13T10:16:53.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:53.037+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.038105+0000) 2019-10-13T10:16:53.040 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:53.038+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.038439+0000) 2019-10-13T10:16:53.041 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:53.038+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.038513+0000) 2019-10-13T10:16:53.041 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:53.038+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.038693+0000) 2019-10-13T10:16:53.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:53.038+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.038715+0000) 2019-10-13T10:16:53.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:53.041+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:53.042097+0000) 2019-10-13T10:16:53.046 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:53.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184030 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:54.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:54.415+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:54.415819+0000) 2019-10-13T10:16:54.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:54.732+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:54.733804+0000) 2019-10-13T10:16:55.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:55.622+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:55.623872+0000) 2019-10-13T10:16:55.628 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:55.622+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:55.623873+0000) 2019-10-13T10:16:58.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:16:58.045+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.046594+0000) 2019-10-13T10:16:58.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:16:58.047+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.048022+0000) 2019-10-13T10:16:58.049 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:16:58.045+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.046890+0000) 2019-10-13T10:16:58.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:58.045+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.047250+0000) 2019-10-13T10:16:58.049 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:16:58.046+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.047553+0000) 2019-10-13T10:16:58.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:16:58.046+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.047756+0000) 2019-10-13T10:16:58.050 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:58.046+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.047964+0000) 2019-10-13T10:16:58.050 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:16:58.049+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.050428+0000) 2019-10-13T10:16:58.055 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:16:58.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:16:58.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:16:58.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.532910+0000) 2019-10-13T10:16:58.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:16:58.870+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.871028+0000) 2019-10-13T10:17:02.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:02.973+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.973959+0000) 2019-10-13T10:17:02.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:02.972+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.973384+0000) 2019-10-13T10:17:03.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:03.056+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.057319+0000) 2019-10-13T10:17:03.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:03.055+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.056664+0000) 2019-10-13T10:17:03.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:03.055+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.056537+0000) 2019-10-13T10:17:03.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:03.055+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.056895+0000) 2019-10-13T10:17:03.059 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:03.055+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.057124+0000) 2019-10-13T10:17:03.059 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:03.056+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.057408+0000) 2019-10-13T10:17:03.059 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:03.056+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.057713+0000) 2019-10-13T10:17:03.060 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:03.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:03.060745+0000) 2019-10-13T10:17:03.065 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:03.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:04.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:04.417+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:04.418169+0000) 2019-10-13T10:17:04.714 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:17:04.714 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:04.718 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:17:04.718 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:04.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:04.732+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:04.733988+0000) 2019-10-13T10:17:05.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:05.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:05.624087+0000) 2019-10-13T10:17:05.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:05.623+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:05.624088+0000) 2019-10-13T10:17:08.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:08.065+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.066654+0000) 2019-10-13T10:17:08.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:08.064+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.065751+0000) 2019-10-13T10:17:08.068 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:08.064+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.065932+0000) 2019-10-13T10:17:08.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:08.064+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.066107+0000) 2019-10-13T10:17:08.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:08.065+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.066443+0000) 2019-10-13T10:17:08.069 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:08.065+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.066648+0000) 2019-10-13T10:17:08.069 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:08.065+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.066744+0000) 2019-10-13T10:17:08.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:08.069+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.070527+0000) 2019-10-13T10:17:08.075 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:08.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184124 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:08.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:08.531+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.533092+0000) 2019-10-13T10:17:08.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:08.870+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.871152+0000) 2019-10-13T10:17:12.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:12.972+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.973490+0000) 2019-10-13T10:17:12.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:12.974+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.974160+0000) 2019-10-13T10:17:13.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:13.075+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.076583+0000) 2019-10-13T10:17:13.077 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:13.074+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.075920+0000) 2019-10-13T10:17:13.078 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:13.074+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.076009+0000) 2019-10-13T10:17:13.078 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:13.074+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.076172+0000) 2019-10-13T10:17:13.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:13.074+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.076370+0000) 2019-10-13T10:17:13.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:13.075+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.076676+0000) 2019-10-13T10:17:13.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:13.078+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.078837+0000) 2019-10-13T10:17:13.079 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:13.075+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:13.077155+0000) 2019-10-13T10:17:13.084 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:13.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:14.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:14.420+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:14.420482+0000) 2019-10-13T10:17:14.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:14.733+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:14.734166+0000) 2019-10-13T10:17:15.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:15.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:15.624259+0000) 2019-10-13T10:17:15.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:15.623+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:15.624266+0000) 2019-10-13T10:17:18.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:18.084+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.085455+0000) 2019-10-13T10:17:18.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:18.084+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.085592+0000) 2019-10-13T10:17:18.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:18.085+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.086843+0000) 2019-10-13T10:17:18.088 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:18.084+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.086038+0000) 2019-10-13T10:17:18.088 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:18.085+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.086405+0000) 2019-10-13T10:17:18.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:18.085+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.086672+0000) 2019-10-13T10:17:18.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:18.085+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.086947+0000) 2019-10-13T10:17:18.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:18.087+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.089469+0000) 2019-10-13T10:17:18.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:18.091+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:18.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:18.532+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.533280+0000) 2019-10-13T10:17:18.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:18.870+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.871329+0000) 2019-10-13T10:17:22.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:22.973+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.974337+0000) 2019-10-13T10:17:22.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:22.973+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.973640+0000) 2019-10-13T10:17:23.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:23.094+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.094823+0000) 2019-10-13T10:17:23.096 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:23.094+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.094948+0000) 2019-10-13T10:17:23.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:23.094+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.096404+0000) 2019-10-13T10:17:23.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:23.095+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.095421+0000) 2019-10-13T10:17:23.099 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:23.095+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.095652+0000) 2019-10-13T10:17:23.100 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:23.095+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.095572+0000) 2019-10-13T10:17:23.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:23.095+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.095835+0000) 2019-10-13T10:17:23.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:23.096+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:23.097867+0000) 2019-10-13T10:17:23.103 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:23.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:24.423 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:24.421+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:24.422666+0000) 2019-10-13T10:17:24.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:24.733+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:24.734343+0000) 2019-10-13T10:17:25.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:25.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:25.624439+0000) 2019-10-13T10:17:25.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:25.624+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:25.624435+0000) 2019-10-13T10:17:28.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:28.103+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.105313+0000) 2019-10-13T10:17:28.106 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:28.104+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.104815+0000) 2019-10-13T10:17:28.106 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:28.104+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.104714+0000) 2019-10-13T10:17:28.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:28.104+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.104930+0000) 2019-10-13T10:17:28.107 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:28.104+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.105128+0000) 2019-10-13T10:17:28.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:28.104+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.105264+0000) 2019-10-13T10:17:28.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:28.105+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.105601+0000) 2019-10-13T10:17:28.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:28.106+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.107795+0000) 2019-10-13T10:17:28.114 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:28.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:28.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:28.533+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.533461+0000) 2019-10-13T10:17:28.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:28.871+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.871519+0000) 2019-10-13T10:17:32.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:32.973+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.974505+0000) 2019-10-13T10:17:32.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:32.973+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.973716+0000) 2019-10-13T10:17:33.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:33.115+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.115972+0000) 2019-10-13T10:17:33.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:33.114+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.114740+0000) 2019-10-13T10:17:33.117 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:33.114+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.115043+0000) 2019-10-13T10:17:33.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:33.115+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.115510+0000) 2019-10-13T10:17:33.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:33.115+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.115485+0000) 2019-10-13T10:17:33.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:33.115+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.115709+0000) 2019-10-13T10:17:33.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:33.115+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.115843+0000) 2019-10-13T10:17:33.118 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:33.117+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:33.118400+0000) 2019-10-13T10:17:33.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:33.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:34.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:34.424+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:34.425025+0000) 2019-10-13T10:17:34.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:34.734+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:34.734538+0000) 2019-10-13T10:17:34.802 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:17:34.802 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:34.807 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:17:34.807 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:35.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:35.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:35.624626+0000) 2019-10-13T10:17:35.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:35.624+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:35.624625+0000) 2019-10-13T10:17:38.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:38.123+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.123638+0000) 2019-10-13T10:17:38.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:38.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.124005+0000) 2019-10-13T10:17:38.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:38.123+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.124247+0000) 2019-10-13T10:17:38.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:38.124+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.125212+0000) 2019-10-13T10:17:38.126 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:38.124+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.124644+0000) 2019-10-13T10:17:38.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:38.124+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.124743+0000) 2019-10-13T10:17:38.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:38.124+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.125009+0000) 2019-10-13T10:17:38.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:38.127+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.128067+0000) 2019-10-13T10:17:38.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:38.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:38.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:38.533+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.533583+0000) 2019-10-13T10:17:38.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:38.871+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.871686+0000) 2019-10-13T10:17:42.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:42.972+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.973845+0000) 2019-10-13T10:17:42.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:42.974+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.974653+0000) 2019-10-13T10:17:43.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:43.131+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.133260+0000) 2019-10-13T10:17:43.135 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:43.133+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.134456+0000) 2019-10-13T10:17:43.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:43.132+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.133702+0000) 2019-10-13T10:17:43.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:43.132+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.133780+0000) 2019-10-13T10:17:43.136 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:43.132+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.133928+0000) 2019-10-13T10:17:43.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:43.132+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.134066+0000) 2019-10-13T10:17:43.136 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:43.132+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.134180+0000) 2019-10-13T10:17:43.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:43.137+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:43.137846+0000) 2019-10-13T10:17:43.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:43.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:44.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:44.426+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:44.427400+0000) 2019-10-13T10:17:44.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:44.733+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:44.734686+0000) 2019-10-13T10:17:45.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:45.623+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:45.624819+0000) 2019-10-13T10:17:45.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:45.623+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:45.624826+0000) 2019-10-13T10:17:48.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:48.140+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.142288+0000) 2019-10-13T10:17:48.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:48.141+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.142531+0000) 2019-10-13T10:17:48.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:48.141+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.142687+0000) 2019-10-13T10:17:48.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:48.142+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.143696+0000) 2019-10-13T10:17:48.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:48.141+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.142961+0000) 2019-10-13T10:17:48.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:48.141+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.143082+0000) 2019-10-13T10:17:48.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:48.141+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.143223+0000) 2019-10-13T10:17:48.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:48.145+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.146063+0000) 2019-10-13T10:17:48.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:48.146+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184362 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:48.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:48.532+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.533697+0000) 2019-10-13T10:17:48.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:48.871+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.871883+0000) 2019-10-13T10:17:52.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:52.974+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.974809+0000) 2019-10-13T10:17:52.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:52.973+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.974006+0000) 2019-10-13T10:17:53.151 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:53.148+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.149575+0000) 2019-10-13T10:17:53.151 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:53.148+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.149775+0000) 2019-10-13T10:17:53.152 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:53.150+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.150971+0000) 2019-10-13T10:17:53.152 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:53.148+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.150335+0000) 2019-10-13T10:17:53.152 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:53.149+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.150481+0000) 2019-10-13T10:17:53.153 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:53.149+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.150724+0000) 2019-10-13T10:17:53.153 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:53.149+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.151061+0000) 2019-10-13T10:17:53.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:53.153+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:53.154119+0000) 2019-10-13T10:17:53.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:53.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184389 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:54.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:54.429+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:54.429820+0000) 2019-10-13T10:17:54.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:54.734+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:54.734870+0000) 2019-10-13T10:17:55.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:55.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:55.625020+0000) 2019-10-13T10:17:55.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:55.624+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:55.625026+0000) 2019-10-13T10:17:58.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:17:58.157+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.158868+0000) 2019-10-13T10:17:58.161 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:58.157+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.159177+0000) 2019-10-13T10:17:58.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:17:58.157+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.159312+0000) 2019-10-13T10:17:58.163 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:17:58.158+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.159486+0000) 2019-10-13T10:17:58.163 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:17:58.158+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.159605+0000) 2019-10-13T10:17:58.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:58.158+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.160026+0000) 2019-10-13T10:17:58.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:17:58.160+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.160858+0000) 2019-10-13T10:17:58.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:17:58.161+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.162271+0000) 2019-10-13T10:17:58.167 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:17:58.164+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:17:58.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:17:58.533+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.533850+0000) 2019-10-13T10:17:58.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:17:58.871+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.872044+0000) 2019-10-13T10:18:02.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:02.974+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.974990+0000) 2019-10-13T10:18:02.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:02.973+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.974228+0000) 2019-10-13T10:18:03.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:03.166+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.168382+0000) 2019-10-13T10:18:03.170 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:03.167+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.168590+0000) 2019-10-13T10:18:03.171 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:03.169+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.169848+0000) 2019-10-13T10:18:03.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:03.167+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.168900+0000) 2019-10-13T10:18:03.171 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:03.167+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.169006+0000) 2019-10-13T10:18:03.171 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:03.167+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.169281+0000) 2019-10-13T10:18:03.171 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:03.168+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.169481+0000) 2019-10-13T10:18:03.172 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:03.170+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:03.171563+0000) 2019-10-13T10:18:03.177 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:03.174+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184447 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:04.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:04.431+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:04.432114+0000) 2019-10-13T10:18:04.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:04.734+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:04.735051+0000) 2019-10-13T10:18:04.882 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:18:04.882 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:04.887 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:18:04.887 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:05.626 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:05.624+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:05.625224+0000) 2019-10-13T10:18:05.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:05.624+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:05.625223+0000) 2019-10-13T10:18:08.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:08.177+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.177575+0000) 2019-10-13T10:18:08.179 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:08.177+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.177757+0000) 2019-10-13T10:18:08.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:08.177+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.178039+0000) 2019-10-13T10:18:08.180 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:08.177+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.178195+0000) 2019-10-13T10:18:08.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:08.178+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.179502+0000) 2019-10-13T10:18:08.181 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:08.178+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.178729+0000) 2019-10-13T10:18:08.181 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:08.178+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.179039+0000) 2019-10-13T10:18:08.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:08.180+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.181158+0000) 2019-10-13T10:18:08.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:08.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184480 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:08.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:08.533+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.533989+0000) 2019-10-13T10:18:08.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:08.871+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.872206+0000) 2019-10-13T10:18:12.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:12.974+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.975163+0000) 2019-10-13T10:18:12.975 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:12.973+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.974349+0000) 2019-10-13T10:18:13.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:13.187+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.187936+0000) 2019-10-13T10:18:13.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:13.187+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.189186+0000) 2019-10-13T10:18:13.190 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:13.187+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.188276+0000) 2019-10-13T10:18:13.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:13.188+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.188654+0000) 2019-10-13T10:18:13.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:13.188+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.188534+0000) 2019-10-13T10:18:13.191 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:13.188+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.189203+0000) 2019-10-13T10:18:13.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:13.189+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.189584+0000) 2019-10-13T10:18:13.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:13.190+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:13.192124+0000) 2019-10-13T10:18:13.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:13.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:14.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:14.433+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:14.434396+0000) 2019-10-13T10:18:14.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:14.734+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:14.735204+0000) 2019-10-13T10:18:15.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:15.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:15.625448+0000) 2019-10-13T10:18:15.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:15.625+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:15.625453+0000) 2019-10-13T10:18:18.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:18.197+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.198897+0000) 2019-10-13T10:18:18.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:18.197+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.197490+0000) 2019-10-13T10:18:18.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:18.197+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.197924+0000) 2019-10-13T10:18:18.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:18.197+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.198124+0000) 2019-10-13T10:18:18.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:18.197+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.198174+0000) 2019-10-13T10:18:18.202 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:18.197+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.198266+0000) 2019-10-13T10:18:18.202 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:18.198+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.198621+0000) 2019-10-13T10:18:18.202 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:18.199+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.200903+0000) 2019-10-13T10:18:18.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:18.204+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184540 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:18.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:18.533+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.534159+0000) 2019-10-13T10:18:18.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:18.872+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.872350+0000) 2019-10-13T10:18:22.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:22.974+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.975364+0000) 2019-10-13T10:18:22.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:22.974+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.974510+0000) 2019-10-13T10:18:23.208 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:23.206+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.206789+0000) 2019-10-13T10:18:23.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:23.206+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.207032+0000) 2019-10-13T10:18:23.209 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:23.207+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.208388+0000) 2019-10-13T10:18:23.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:23.207+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.207669+0000) 2019-10-13T10:18:23.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:23.207+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.207698+0000) 2019-10-13T10:18:23.210 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:23.207+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.208023+0000) 2019-10-13T10:18:23.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:23.207+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.208292+0000) 2019-10-13T10:18:23.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:23.209+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:23.210660+0000) 2019-10-13T10:18:23.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:23.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184567 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:24.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:24.435+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:24.436648+0000) 2019-10-13T10:18:24.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:24.735+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:24.735405+0000) 2019-10-13T10:18:25.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:25.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:25.625670+0000) 2019-10-13T10:18:25.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:25.625+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:25.625665+0000) 2019-10-13T10:18:28.218 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:28.214+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.216277+0000) 2019-10-13T10:18:28.219 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:28.215+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.216700+0000) 2019-10-13T10:18:28.219 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:28.215+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.216831+0000) 2019-10-13T10:18:28.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:28.215+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.216959+0000) 2019-10-13T10:18:28.219 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:28.215+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.217293+0000) 2019-10-13T10:18:28.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:28.216+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.217411+0000) 2019-10-13T10:18:28.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:28.217+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.218698+0000) 2019-10-13T10:18:28.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:28.220+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.220965+0000) 2019-10-13T10:18:28.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:28.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184600 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:28.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:28.533+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.534289+0000) 2019-10-13T10:18:28.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:28.871+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.872516+0000) 2019-10-13T10:18:32.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:32.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.975569+0000) 2019-10-13T10:18:32.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:32.973+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.974677+0000) 2019-10-13T10:18:33.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:33.223+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.225136+0000) 2019-10-13T10:18:33.227 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:33.223+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.225330+0000) 2019-10-13T10:18:33.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:33.224+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.225654+0000) 2019-10-13T10:18:33.228 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:33.226+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.227216+0000) 2019-10-13T10:18:33.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:33.224+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.226032+0000) 2019-10-13T10:18:33.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:33.225+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.226502+0000) 2019-10-13T10:18:33.228 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:33.225+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.226595+0000) 2019-10-13T10:18:33.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:33.228+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:33.229163+0000) 2019-10-13T10:18:33.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:33.230+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184630 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:34.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:34.438+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:34.438923+0000) 2019-10-13T10:18:34.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:34.734+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:34.735596+0000) 2019-10-13T10:18:35.014 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:18:35.014 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:35.018 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:18:35.019 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:35.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:35.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:35.625881+0000) 2019-10-13T10:18:35.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:35.625+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:35.625865+0000) 2019-10-13T10:18:38.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:38.232+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.233633+0000) 2019-10-13T10:18:38.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:38.232+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.233820+0000) 2019-10-13T10:18:38.236 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:38.234+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.235484+0000) 2019-10-13T10:18:38.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:38.232+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.234184+0000) 2019-10-13T10:18:38.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:38.233+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.234627+0000) 2019-10-13T10:18:38.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:38.233+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.235026+0000) 2019-10-13T10:18:38.237 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:38.233+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.235225+0000) 2019-10-13T10:18:38.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:38.236+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.237537+0000) 2019-10-13T10:18:38.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:38.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184662 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:38.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:38.533+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.534460+0000) 2019-10-13T10:18:38.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:38.871+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.872684+0000) 2019-10-13T10:18:42.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:42.974+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.974857+0000) 2019-10-13T10:18:42.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:42.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.975771+0000) 2019-10-13T10:18:43.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:43.241+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.243332+0000) 2019-10-13T10:18:43.245 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:43.242+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.243566+0000) 2019-10-13T10:18:43.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:43.242+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.244021+0000) 2019-10-13T10:18:43.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:43.242+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.244270+0000) 2019-10-13T10:18:43.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:43.242+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.244338+0000) 2019-10-13T10:18:43.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:43.243+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.244591+0000) 2019-10-13T10:18:43.246 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:43.245+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.245954+0000) 2019-10-13T10:18:43.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:43.246+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:43.246937+0000) 2019-10-13T10:18:43.253 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:43.250+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184690 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:44.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:44.440+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:44.441280+0000) 2019-10-13T10:18:44.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:44.735+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:44.735787+0000) 2019-10-13T10:18:45.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:45.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:45.626033+0000) 2019-10-13T10:18:45.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:45.625+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:45.626071+0000) 2019-10-13T10:18:48.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:48.254+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.255129+0000) 2019-10-13T10:18:48.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:48.252+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.254323+0000) 2019-10-13T10:18:48.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:48.253+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.254523+0000) 2019-10-13T10:18:48.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:48.253+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.254728+0000) 2019-10-13T10:18:48.257 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:48.253+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.254844+0000) 2019-10-13T10:18:48.257 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:48.253+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.254997+0000) 2019-10-13T10:18:48.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:48.253+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.255314+0000) 2019-10-13T10:18:48.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:48.257+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.258625+0000) 2019-10-13T10:18:48.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:48.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184722 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:48.536 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:48.534+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.534605+0000) 2019-10-13T10:18:48.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:48.872+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.872850+0000) 2019-10-13T10:18:52.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:52.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.975905+0000) 2019-10-13T10:18:52.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:52.974+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.975057+0000) 2019-10-13T10:18:53.264 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:53.263+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.263894+0000) 2019-10-13T10:18:53.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:53.261+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.263373+0000) 2019-10-13T10:18:53.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:53.262+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.263590+0000) 2019-10-13T10:18:53.265 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:53.262+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.263808+0000) 2019-10-13T10:18:53.265 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:53.262+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.264063+0000) 2019-10-13T10:18:53.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:53.262+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.264003+0000) 2019-10-13T10:18:53.266 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:53.263+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.264602+0000) 2019-10-13T10:18:53.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:53.265+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:53.266609+0000) 2019-10-13T10:18:53.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:53.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:54.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:54.443+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:54.443755+0000) 2019-10-13T10:18:54.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:54.735+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:54.735979+0000) 2019-10-13T10:18:55.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:55.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:55.626232+0000) 2019-10-13T10:18:55.628 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:55.625+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:55.626232+0000) 2019-10-13T10:18:58.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:18:58.272+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.273341+0000) 2019-10-13T10:18:58.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:58.272+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.272798+0000) 2019-10-13T10:18:58.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:18:58.272+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.273021+0000) 2019-10-13T10:18:58.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:58.272+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.273098+0000) 2019-10-13T10:18:58.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:18:58.273+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.273438+0000) 2019-10-13T10:18:58.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:18:58.273+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.273734+0000) 2019-10-13T10:18:58.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:18:58.273+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.273865+0000) 2019-10-13T10:18:58.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:18:58.274+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.275696+0000) 2019-10-13T10:18:58.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:18:58.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184784 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:18:58.536 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:18:58.534+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.534799+0000) 2019-10-13T10:18:58.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:18:58.872+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.873010+0000) 2019-10-13T10:19:02.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:02.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.976047+0000) 2019-10-13T10:19:02.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:02.974+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.975258+0000) 2019-10-13T10:19:03.284 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:03.282+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.282957+0000) 2019-10-13T10:19:03.285 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:03.282+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.284399+0000) 2019-10-13T10:19:03.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:03.283+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.283486+0000) 2019-10-13T10:19:03.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:03.283+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.283521+0000) 2019-10-13T10:19:03.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:03.283+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.283712+0000) 2019-10-13T10:19:03.287 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:03.283+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.283904+0000) 2019-10-13T10:19:03.287 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:03.283+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.284092+0000) 2019-10-13T10:19:03.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:03.284+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:03.286392+0000) 2019-10-13T10:19:03.291 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:03.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:04.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:04.444+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:04.446109+0000) 2019-10-13T10:19:04.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:04.735+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:04.736166+0000) 2019-10-13T10:19:05.153 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:19:05.153 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:05.157 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:19:05.157 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:05.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:05.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:05.626437+0000) 2019-10-13T10:19:05.628 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:05.626+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:05.626434+0000) 2019-10-13T10:19:08.293 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:08.291+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.291968+0000) 2019-10-13T10:19:08.293 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:08.291+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.292240+0000) 2019-10-13T10:19:08.294 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:08.291+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.293493+0000) 2019-10-13T10:19:08.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:08.292+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.292720+0000) 2019-10-13T10:19:08.295 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:08.292+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.292844+0000) 2019-10-13T10:19:08.295 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:08.292+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.293112+0000) 2019-10-13T10:19:08.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:08.293+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.293724+0000) 2019-10-13T10:19:08.296 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:08.294+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.296230+0000) 2019-10-13T10:19:08.301 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:08.299+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184844 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:08.536 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:08.534+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.534966+0000) 2019-10-13T10:19:08.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:08.872+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.873192+0000) 2019-10-13T10:19:12.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:12.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.976205+0000) 2019-10-13T10:19:12.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:12.974+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.975408+0000) 2019-10-13T10:19:13.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:13.299+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.300957+0000) 2019-10-13T10:19:13.303 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:13.300+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.302220+0000) 2019-10-13T10:19:13.303 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:13.300+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.302143+0000) 2019-10-13T10:19:13.304 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:13.301+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.302490+0000) 2019-10-13T10:19:13.304 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:13.302+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.303934+0000) 2019-10-13T10:19:13.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:13.301+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.302710+0000) 2019-10-13T10:19:13.305 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:13.301+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.302934+0000) 2019-10-13T10:19:13.305 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:13.304+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:13.305902+0000) 2019-10-13T10:19:13.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:13.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184874 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:14.448 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:14.447+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:14.448350+0000) 2019-10-13T10:19:14.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:14.735+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:14.736367+0000) 2019-10-13T10:19:15.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:15.625+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:15.626652+0000) 2019-10-13T10:19:15.628 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:15.625+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:15.626634+0000) 2019-10-13T10:19:18.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:18.307+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.309042+0000) 2019-10-13T10:19:18.311 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:18.309+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.310409+0000) 2019-10-13T10:19:18.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:18.308+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.309718+0000) 2019-10-13T10:19:18.311 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:18.308+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.309873+0000) 2019-10-13T10:19:18.312 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:18.308+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.310249+0000) 2019-10-13T10:19:18.312 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:18.309+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.310439+0000) 2019-10-13T10:19:18.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:18.309+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.310671+0000) 2019-10-13T10:19:18.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:18.312+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.313425+0000) 2019-10-13T10:19:18.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:18.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184906 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:18.536 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:18.534+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.535162+0000) 2019-10-13T10:19:18.874 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:18.872+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.873373+0000) 2019-10-13T10:19:22.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:22.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.976396+0000) 2019-10-13T10:19:22.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:22.974+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.975555+0000) 2019-10-13T10:19:23.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:23.319+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.319962+0000) 2019-10-13T10:19:23.321 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:23.317+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.319040+0000) 2019-10-13T10:19:23.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:23.318+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.319411+0000) 2019-10-13T10:19:23.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:23.318+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.319514+0000) 2019-10-13T10:19:23.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:23.318+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.319731+0000) 2019-10-13T10:19:23.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:23.321+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.321933+0000) 2019-10-13T10:19:23.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:23.318+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.319951+0000) 2019-10-13T10:19:23.323 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:23.318+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:23.319821+0000) 2019-10-13T10:19:23.328 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:23.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:24.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:24.450+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:24.450783+0000) 2019-10-13T10:19:24.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:24.735+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:24.736591+0000) 2019-10-13T10:19:25.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:25.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:25.626839+0000) 2019-10-13T10:19:25.628 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:25.626+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:25.626842+0000) 2019-10-13T10:19:28.329 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:28.328+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.329485+0000) 2019-10-13T10:19:28.330 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:28.327+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.328826+0000) 2019-10-13T10:19:28.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:28.327+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.329027+0000) 2019-10-13T10:19:28.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:28.327+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.329254+0000) 2019-10-13T10:19:28.331 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:28.327+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.329315+0000) 2019-10-13T10:19:28.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:28.328+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.329435+0000) 2019-10-13T10:19:28.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:28.328+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.329724+0000) 2019-10-13T10:19:28.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:28.331+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.332674+0000) 2019-10-13T10:19:28.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:28.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:28.536 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:28.534+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.535335+0000) 2019-10-13T10:19:28.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:28.872+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.873559+0000) 2019-10-13T10:19:32.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:32.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.976596+0000) 2019-10-13T10:19:32.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:32.975+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.975760+0000) 2019-10-13T10:19:33.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:33.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.338622+0000) 2019-10-13T10:19:33.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:33.336+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.337735+0000) 2019-10-13T10:19:33.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:33.336+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.337995+0000) 2019-10-13T10:19:33.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:33.336+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.338073+0000) 2019-10-13T10:19:33.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:33.336+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.338261+0000) 2019-10-13T10:19:33.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:33.337+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.338454+0000) 2019-10-13T10:19:33.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:33.337+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.338455+0000) 2019-10-13T10:19:33.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:33.342+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:33.342819+0000) 2019-10-13T10:19:33.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:33.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 184996 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:34.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:34.452+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:34.453101+0000) 2019-10-13T10:19:34.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:34.735+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:34.736762+0000) 2019-10-13T10:19:35.267 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:19:35.267 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:35.272 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:19:35.272 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:35.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:35.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:35.627054+0000) 2019-10-13T10:19:35.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:35.626+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:35.627059+0000) 2019-10-13T10:19:38.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:38.345+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.347106+0000) 2019-10-13T10:19:38.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:38.345+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.347345+0000) 2019-10-13T10:19:38.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:38.346+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.347604+0000) 2019-10-13T10:19:38.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:38.346+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.348212+0000) 2019-10-13T10:19:38.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:38.348+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.348942+0000) 2019-10-13T10:19:38.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:38.347+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.348472+0000) 2019-10-13T10:19:38.352 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:38.347+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.348998+0000) 2019-10-13T10:19:38.353 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:38.351+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.351882+0000) 2019-10-13T10:19:38.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:38.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:38.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:38.535+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.535494+0000) 2019-10-13T10:19:38.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:38.873+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.873721+0000) 2019-10-13T10:19:42.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:42.976+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.976747+0000) 2019-10-13T10:19:42.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:42.975+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.975942+0000) 2019-10-13T10:19:43.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:43.356+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.356960+0000) 2019-10-13T10:19:43.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:43.356+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.357244+0000) 2019-10-13T10:19:43.360 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:43.357+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.357500+0000) 2019-10-13T10:19:43.361 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:43.357+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.357664+0000) 2019-10-13T10:19:43.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:43.357+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.358469+0000) 2019-10-13T10:19:43.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:43.360+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.361171+0000) 2019-10-13T10:19:43.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:43.357+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.357880+0000) 2019-10-13T10:19:43.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:43.358+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:43.358468+0000) 2019-10-13T10:19:43.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:43.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185056 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:44.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:44.455+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:44.455441+0000) 2019-10-13T10:19:44.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:44.736+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:44.736948+0000) 2019-10-13T10:19:45.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:45.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:45.627256+0000) 2019-10-13T10:19:45.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:45.626+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:45.627256+0000) 2019-10-13T10:19:48.366 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:48.365+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.366117+0000) 2019-10-13T10:19:48.367 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:48.365+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.365408+0000) 2019-10-13T10:19:48.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:48.365+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.365507+0000) 2019-10-13T10:19:48.367 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:48.365+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.365620+0000) 2019-10-13T10:19:48.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:48.365+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.365876+0000) 2019-10-13T10:19:48.368 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:48.365+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.365951+0000) 2019-10-13T10:19:48.368 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:48.365+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.366258+0000) 2019-10-13T10:19:48.368 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:48.367+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.368433+0000) 2019-10-13T10:19:48.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:48.372+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185088 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:48.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:48.535+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.535668+0000) 2019-10-13T10:19:48.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:48.873+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.873922+0000) 2019-10-13T10:19:52.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:52.975+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.976913+0000) 2019-10-13T10:19:52.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:52.975+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.976101+0000) 2019-10-13T10:19:53.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:53.374+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.374833+0000) 2019-10-13T10:19:53.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:53.374+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.375181+0000) 2019-10-13T10:19:53.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:53.375+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.376866+0000) 2019-10-13T10:19:53.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:53.375+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.375422+0000) 2019-10-13T10:19:53.378 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:53.375+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.375559+0000) 2019-10-13T10:19:53.378 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:53.375+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.376104+0000) 2019-10-13T10:19:53.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:53.375+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.375723+0000) 2019-10-13T10:19:53.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:53.376+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:53.378182+0000) 2019-10-13T10:19:53.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:53.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185118 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:54.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:54.456+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:54.457906+0000) 2019-10-13T10:19:54.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:54.737+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:54.737156+0000) 2019-10-13T10:19:55.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:55.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:55.627453+0000) 2019-10-13T10:19:55.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:55.627+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:55.627454+0000) 2019-10-13T10:19:58.385 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:19:58.382+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.383794+0000) 2019-10-13T10:19:58.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:58.382+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.384017+0000) 2019-10-13T10:19:58.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:19:58.382+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.384271+0000) 2019-10-13T10:19:58.386 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:19:58.383+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.385688+0000) 2019-10-13T10:19:58.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:19:58.382+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.384366+0000) 2019-10-13T10:19:58.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:19:58.383+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.384467+0000) 2019-10-13T10:19:58.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:58.383+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.384930+0000) 2019-10-13T10:19:58.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:19:58.387+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.388774+0000) 2019-10-13T10:19:58.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:19:58.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:19:58.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:19:58.534+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.535855+0000) 2019-10-13T10:19:58.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:19:58.872+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.874092+0000) 2019-10-13T10:20:02.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:02.976+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.977099+0000) 2019-10-13T10:20:02.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:02.974+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.976286+0000) 2019-10-13T10:20:03.395 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:03.392+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.393730+0000) 2019-10-13T10:20:03.395 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:03.392+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.394119+0000) 2019-10-13T10:20:03.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:03.393+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.395565+0000) 2019-10-13T10:20:03.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:03.392+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.394286+0000) 2019-10-13T10:20:03.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:03.393+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.394727+0000) 2019-10-13T10:20:03.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:03.393+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.394877+0000) 2019-10-13T10:20:03.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:03.393+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.394922+0000) 2019-10-13T10:20:03.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:03.395+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:03.397191+0000) 2019-10-13T10:20:03.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:03.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:04.468 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:04.459+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:04.460262+0000) 2019-10-13T10:20:04.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:04.736+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:04.737312+0000) 2019-10-13T10:20:05.367 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:20:05.368 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:05.376 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:20:05.376 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:05.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:05.626+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:05.627583+0000) 2019-10-13T10:20:05.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:05.626+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:05.627644+0000) 2019-10-13T10:20:08.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:08.401+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.403256+0000) 2019-10-13T10:20:08.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:08.402+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.403477+0000) 2019-10-13T10:20:08.405 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:08.402+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.403553+0000) 2019-10-13T10:20:08.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:08.402+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.403868+0000) 2019-10-13T10:20:08.406 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:08.404+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.405525+0000) 2019-10-13T10:20:08.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:08.402+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.404323+0000) 2019-10-13T10:20:08.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:08.402+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.403954+0000) 2019-10-13T10:20:08.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:08.407+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.407836+0000) 2019-10-13T10:20:08.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:08.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185210 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:08.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:08.535+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.536031+0000) 2019-10-13T10:20:08.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:08.873+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.874257+0000) 2019-10-13T10:20:12.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:12.976+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.977261+0000) 2019-10-13T10:20:12.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:12.975+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.976467+0000) 2019-10-13T10:20:13.414 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:13.411+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.412689+0000) 2019-10-13T10:20:13.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:13.411+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.412994+0000) 2019-10-13T10:20:13.415 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:13.413+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.414525+0000) 2019-10-13T10:20:13.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:13.411+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.413227+0000) 2019-10-13T10:20:13.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:13.412+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.413535+0000) 2019-10-13T10:20:13.416 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:13.412+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.413760+0000) 2019-10-13T10:20:13.416 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:13.412+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.414048+0000) 2019-10-13T10:20:13.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:13.416+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:13.417714+0000) 2019-10-13T10:20:13.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:13.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185240 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:14.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:14.462+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:14.462593+0000) 2019-10-13T10:20:14.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:14.736+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:14.737500+0000) 2019-10-13T10:20:15.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:15.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:15.627803+0000) 2019-10-13T10:20:15.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:15.627+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:15.627802+0000) 2019-10-13T10:20:18.424 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:18.421+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.422647+0000) 2019-10-13T10:20:18.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:18.421+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.422773+0000) 2019-10-13T10:20:18.424 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:18.421+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.422996+0000) 2019-10-13T10:20:18.425 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:18.424+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.424930+0000) 2019-10-13T10:20:18.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:18.421+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.423158+0000) 2019-10-13T10:20:18.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:18.422+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.423743+0000) 2019-10-13T10:20:18.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:18.422+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.423983+0000) 2019-10-13T10:20:18.426 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:18.425+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.426721+0000) 2019-10-13T10:20:18.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:18.428+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185272 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:18.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:18.535+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.536196+0000) 2019-10-13T10:20:18.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:18.873+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.874426+0000) 2019-10-13T10:20:22.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:22.977+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.977425+0000) 2019-10-13T10:20:22.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:22.976+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.976648+0000) 2019-10-13T10:20:23.433 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:23.430+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.432029+0000) 2019-10-13T10:20:23.434 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:23.430+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.432293+0000) 2019-10-13T10:20:23.434 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:23.433+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.434064+0000) 2019-10-13T10:20:23.435 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:23.431+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.432681+0000) 2019-10-13T10:20:23.435 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:23.431+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.433134+0000) 2019-10-13T10:20:23.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:23.431+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.433202+0000) 2019-10-13T10:20:23.435 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:23.432+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.433407+0000) 2019-10-13T10:20:23.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:23.435+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:23.435880+0000) 2019-10-13T10:20:23.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:23.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185300 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:24.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:24.465+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:24.465658+0000) 2019-10-13T10:20:24.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:24.737+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:24.737692+0000) 2019-10-13T10:20:25.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:25.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:25.628020+0000) 2019-10-13T10:20:25.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:25.627+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:25.628021+0000) 2019-10-13T10:20:28.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:28.441+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.441807+0000) 2019-10-13T10:20:28.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:28.441+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.442071+0000) 2019-10-13T10:20:28.444 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:28.441+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.442288+0000) 2019-10-13T10:20:28.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:28.442+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.442562+0000) 2019-10-13T10:20:28.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:28.443+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.444155+0000) 2019-10-13T10:20:28.445 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:28.442+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.442409+0000) 2019-10-13T10:20:28.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:28.442+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.442942+0000) 2019-10-13T10:20:28.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:28.445+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.446147+0000) 2019-10-13T10:20:28.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:28.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185332 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:28.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:28.535+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.536339+0000) 2019-10-13T10:20:28.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:28.874+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.874594+0000) 2019-10-13T10:20:32.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:32.976+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.977625+0000) 2019-10-13T10:20:32.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:32.976+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.976830+0000) 2019-10-13T10:20:33.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:33.451+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.451968+0000) 2019-10-13T10:20:33.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:33.451+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.452253+0000) 2019-10-13T10:20:33.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:33.452+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.452498+0000) 2019-10-13T10:20:33.454 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:33.452+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.452880+0000) 2019-10-13T10:20:33.454 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:33.452+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.453049+0000) 2019-10-13T10:20:33.455 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:33.453+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.454215+0000) 2019-10-13T10:20:33.455 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:33.453+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.453461+0000) 2019-10-13T10:20:33.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:33.454+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:33.455553+0000) 2019-10-13T10:20:33.461 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:33.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185362 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:34.468 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:34.467+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:34.467761+0000) 2019-10-13T10:20:34.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:34.737+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:34.737865+0000) 2019-10-13T10:20:35.486 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:20:35.486 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:35.491 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:20:35.491 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:35.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:35.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:35.628217+0000) 2019-10-13T10:20:35.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:35.627+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:35.628216+0000) 2019-10-13T10:20:38.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:38.461+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.462227+0000) 2019-10-13T10:20:38.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:38.462+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.462651+0000) 2019-10-13T10:20:38.464 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:38.462+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.462535+0000) 2019-10-13T10:20:38.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:38.462+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.462954+0000) 2019-10-13T10:20:38.466 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:38.463+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.463855+0000) 2019-10-13T10:20:38.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:38.464+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.465426+0000) 2019-10-13T10:20:38.467 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:38.462+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.463118+0000) 2019-10-13T10:20:38.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:38.462+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.463248+0000) 2019-10-13T10:20:38.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:38.470+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185394 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:38.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:38.536+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.536458+0000) 2019-10-13T10:20:38.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:38.874+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.874786+0000) 2019-10-13T10:20:42.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:42.976+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.977791+0000) 2019-10-13T10:20:42.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:42.976+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.977010+0000) 2019-10-13T10:20:43.473 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:43.471+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.473482+0000) 2019-10-13T10:20:43.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:43.472+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.472599+0000) 2019-10-13T10:20:43.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:43.472+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.472823+0000) 2019-10-13T10:20:43.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:43.472+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.473093+0000) 2019-10-13T10:20:43.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:43.472+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.473327+0000) 2019-10-13T10:20:43.475 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:43.473+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.473561+0000) 2019-10-13T10:20:43.475 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:43.473+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.473632+0000) 2019-10-13T10:20:43.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:43.473+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:43.475179+0000) 2019-10-13T10:20:43.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:43.480+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:44.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:44.469+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:44.470163+0000) 2019-10-13T10:20:44.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:44.737+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:44.738033+0000) 2019-10-13T10:20:45.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:45.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:45.628381+0000) 2019-10-13T10:20:45.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:45.627+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:45.628381+0000) 2019-10-13T10:20:48.484 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:48.481+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.483314+0000) 2019-10-13T10:20:48.484 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:48.480+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.482130+0000) 2019-10-13T10:20:48.484 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:48.480+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.482364+0000) 2019-10-13T10:20:48.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:48.481+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.482575+0000) 2019-10-13T10:20:48.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:48.481+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.482816+0000) 2019-10-13T10:20:48.485 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:48.481+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.483078+0000) 2019-10-13T10:20:48.485 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:48.482+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.483638+0000) 2019-10-13T10:20:48.486 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:48.484+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.486483+0000) 2019-10-13T10:20:48.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:48.486+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:48.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:48.535+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.536624+0000) 2019-10-13T10:20:48.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:48.873+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.874889+0000) 2019-10-13T10:20:52.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:52.977+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.977966+0000) 2019-10-13T10:20:52.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:52.976+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.977176+0000) 2019-10-13T10:20:53.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:53.488+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.490061+0000) 2019-10-13T10:20:53.491 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:53.488+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.490213+0000) 2019-10-13T10:20:53.492 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:53.489+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.490456+0000) 2019-10-13T10:20:53.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:53.490+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.492028+0000) 2019-10-13T10:20:53.493 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:53.489+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.490597+0000) 2019-10-13T10:20:53.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:53.489+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.490756+0000) 2019-10-13T10:20:53.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:53.489+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.491172+0000) 2019-10-13T10:20:53.493 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:53.491+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:53.493097+0000) 2019-10-13T10:20:53.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:53.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:54.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:54.471+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:54.472463+0000) 2019-10-13T10:20:54.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:54.736+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:54.738232+0000) 2019-10-13T10:20:55.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:55.627+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:55.628521+0000) 2019-10-13T10:20:55.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:55.627+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:55.628520+0000) 2019-10-13T10:20:58.500 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:20:58.497+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.499002+0000) 2019-10-13T10:20:58.500 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:58.497+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.499078+0000) 2019-10-13T10:20:58.501 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:20:58.499+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.500288+0000) 2019-10-13T10:20:58.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:20:58.498+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.499721+0000) 2019-10-13T10:20:58.502 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:20:58.498+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.499851+0000) 2019-10-13T10:20:58.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:58.498+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.499971+0000) 2019-10-13T10:20:58.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:20:58.498+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.500113+0000) 2019-10-13T10:20:58.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:20:58.501+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.502610+0000) 2019-10-13T10:20:58.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:20:58.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:20:58.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:20:58.536+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.536758+0000) 2019-10-13T10:20:58.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:20:58.874+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.875039+0000) 2019-10-13T10:21:02.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:02.977+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.978172+0000) 2019-10-13T10:21:02.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:02.976+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.977351+0000) 2019-10-13T10:21:03.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:03.505+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.506820+0000) 2019-10-13T10:21:03.508 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:03.505+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.506963+0000) 2019-10-13T10:21:03.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:03.507+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.508157+0000) 2019-10-13T10:21:03.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:03.505+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.507264+0000) 2019-10-13T10:21:03.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:03.505+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.507345+0000) 2019-10-13T10:21:03.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:03.506+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.508058+0000) 2019-10-13T10:21:03.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:03.506+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.508168+0000) 2019-10-13T10:21:03.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:03.510+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:03.510754+0000) 2019-10-13T10:21:03.516 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:03.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:04.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:04.474+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:04.475059+0000) 2019-10-13T10:21:04.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:04.737+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:04.738446+0000) 2019-10-13T10:21:05.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:05.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:05.628693+0000) 2019-10-13T10:21:05.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:05.628+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:05.628693+0000) 2019-10-13T10:21:05.651 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:21:05.651 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:05.655 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:21:05.655 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:08.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:08.517+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.517917+0000) 2019-10-13T10:21:08.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:08.515+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.517012+0000) 2019-10-13T10:21:08.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:08.515+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.517169+0000) 2019-10-13T10:21:08.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:08.515+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.517273+0000) 2019-10-13T10:21:08.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:08.516+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.517636+0000) 2019-10-13T10:21:08.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:08.516+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.518073+0000) 2019-10-13T10:21:08.520 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:08.516+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.518291+0000) 2019-10-13T10:21:08.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:08.519+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.520631+0000) 2019-10-13T10:21:08.526 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:08.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185576 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:08.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:08.536+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.536862+0000) 2019-10-13T10:21:08.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:08.874+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.875206+0000) 2019-10-13T10:21:12.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:12.977+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.978359+0000) 2019-10-13T10:21:12.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:12.977+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.977531+0000) 2019-10-13T10:21:13.528 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:13.526+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.526565+0000) 2019-10-13T10:21:13.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:13.526+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.526815+0000) 2019-10-13T10:21:13.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:13.526+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.527028+0000) 2019-10-13T10:21:13.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:13.526+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.526966+0000) 2019-10-13T10:21:13.530 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:13.527+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.528266+0000) 2019-10-13T10:21:13.530 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:13.527+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.527461+0000) 2019-10-13T10:21:13.530 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:13.527+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.527713+0000) 2019-10-13T10:21:13.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:13.529+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:13.530492+0000) 2019-10-13T10:21:13.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:13.534+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185606 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:14.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:14.477+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:14.477461+0000) 2019-10-13T10:21:14.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:14.738+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:14.738629+0000) 2019-10-13T10:21:15.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:15.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:15.628875+0000) 2019-10-13T10:21:15.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:15.628+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:15.628874+0000) 2019-10-13T10:21:18.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:18.535+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.536156+0000) 2019-10-13T10:21:18.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:18.536+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.536974+0000) 2019-10-13T10:21:18.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:18.535+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.536351+0000) 2019-10-13T10:21:18.538 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:18.536+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.536603+0000) 2019-10-13T10:21:18.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:18.536+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.536824+0000) 2019-10-13T10:21:18.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:18.538+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.538763+0000) 2019-10-13T10:21:18.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:18.536+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.537235+0000) 2019-10-13T10:21:18.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:18.537+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.537484+0000) 2019-10-13T10:21:18.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:18.539+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.540152+0000) 2019-10-13T10:21:18.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:18.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185638 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:18.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:18.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.875417+0000) 2019-10-13T10:21:22.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:22.978+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.978542+0000) 2019-10-13T10:21:22.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:22.977+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.977711+0000) 2019-10-13T10:21:23.547 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:23.545+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.545966+0000) 2019-10-13T10:21:23.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:23.545+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.546186+0000) 2019-10-13T10:21:23.548 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:23.546+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.547314+0000) 2019-10-13T10:21:23.548 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:23.546+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.546624+0000) 2019-10-13T10:21:23.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:23.546+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.546781+0000) 2019-10-13T10:21:23.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:23.547+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.547568+0000) 2019-10-13T10:21:23.549 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:23.547+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.547810+0000) 2019-10-13T10:21:23.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:23.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185667 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:23.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:23.549+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:23.550284+0000) 2019-10-13T10:21:24.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:24.479+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:24.480005+0000) 2019-10-13T10:21:24.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:24.738+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:24.738810+0000) 2019-10-13T10:21:25.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:25.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:25.629086+0000) 2019-10-13T10:21:25.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:25.628+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:25.629086+0000) 2019-10-13T10:21:28.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:28.537+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.537113+0000) 2019-10-13T10:21:28.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:28.555+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.555817+0000) 2019-10-13T10:21:28.557 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:28.555+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.555959+0000) 2019-10-13T10:21:28.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:28.555+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.556088+0000) 2019-10-13T10:21:28.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:28.555+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.556224+0000) 2019-10-13T10:21:28.558 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:28.556+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.557484+0000) 2019-10-13T10:21:28.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:28.556+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.556830+0000) 2019-10-13T10:21:28.559 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:28.556+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.556948+0000) 2019-10-13T10:21:28.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:28.559+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.560503+0000) 2019-10-13T10:21:28.564 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:28.562+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:28.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:28.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.875593+0000) 2019-10-13T10:21:32.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:32.977+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.978722+0000) 2019-10-13T10:21:32.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:32.976+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.977893+0000) 2019-10-13T10:21:33.566 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:33.563+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.565704+0000) 2019-10-13T10:21:33.567 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:33.563+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.564811+0000) 2019-10-13T10:21:33.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:33.563+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.564958+0000) 2019-10-13T10:21:33.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:33.564+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.565492+0000) 2019-10-13T10:21:33.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:33.564+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.565638+0000) 2019-10-13T10:21:33.568 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:33.564+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.566023+0000) 2019-10-13T10:21:33.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:33.564+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.566273+0000) 2019-10-13T10:21:33.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:33.567+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:33.568906+0000) 2019-10-13T10:21:33.573 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:33.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185729 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:34.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:34.481+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:34.482190+0000) 2019-10-13T10:21:34.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:34.738+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:34.738959+0000) 2019-10-13T10:21:35.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:35.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:35.629255+0000) 2019-10-13T10:21:35.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:35.628+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:35.629256+0000) 2019-10-13T10:21:35.822 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:21:35.822 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:35.827 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:21:35.828 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:38.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:38.536+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.537304+0000) 2019-10-13T10:21:38.575 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:38.572+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.574363+0000) 2019-10-13T10:21:38.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:38.573+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.574666+0000) 2019-10-13T10:21:38.577 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:38.573+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.575623+0000) 2019-10-13T10:21:38.577 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:38.573+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.574841+0000) 2019-10-13T10:21:38.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:38.573+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.575049+0000) 2019-10-13T10:21:38.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:38.574+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.575714+0000) 2019-10-13T10:21:38.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:38.574+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.576009+0000) 2019-10-13T10:21:38.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:38.575+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.577419+0000) 2019-10-13T10:21:38.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:38.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185760 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:38.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:38.874+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.875708+0000) 2019-10-13T10:21:42.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:42.978+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.978951+0000) 2019-10-13T10:21:42.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:42.977+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.978096+0000) 2019-10-13T10:21:43.585 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:43.582+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.584087+0000) 2019-10-13T10:21:43.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:43.583+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.584453+0000) 2019-10-13T10:21:43.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:43.583+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.584590+0000) 2019-10-13T10:21:43.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:43.584+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.586094+0000) 2019-10-13T10:21:43.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:43.583+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.585183+0000) 2019-10-13T10:21:43.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:43.583+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.585332+0000) 2019-10-13T10:21:43.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:43.584+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.585807+0000) 2019-10-13T10:21:43.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:43.586+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:43.588129+0000) 2019-10-13T10:21:43.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:43.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:44.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:44.483+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:44.484419+0000) 2019-10-13T10:21:44.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:44.737+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:44.739156+0000) 2019-10-13T10:21:45.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:45.628+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:45.629451+0000) 2019-10-13T10:21:45.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:45.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:45.629451+0000) 2019-10-13T10:21:48.538 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:48.536+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.537450+0000) 2019-10-13T10:21:48.596 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:48.592+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.594011+0000) 2019-10-13T10:21:48.596 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:48.593+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.594463+0000) 2019-10-13T10:21:48.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:48.593+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.594604+0000) 2019-10-13T10:21:48.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:48.593+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.594715+0000) 2019-10-13T10:21:48.597 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:48.593+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.595141+0000) 2019-10-13T10:21:48.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:48.595+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.596059+0000) 2019-10-13T10:21:48.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:48.593+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.595386+0000) 2019-10-13T10:21:48.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:48.597+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.598053+0000) 2019-10-13T10:21:48.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:48.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185820 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:48.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:48.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.875907+0000) 2019-10-13T10:21:52.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:52.978+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.979154+0000) 2019-10-13T10:21:52.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:52.977+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.978309+0000) 2019-10-13T10:21:53.603 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:53.600+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.602146+0000) 2019-10-13T10:21:53.603 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:53.601+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.602440+0000) 2019-10-13T10:21:53.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:53.603+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.603884+0000) 2019-10-13T10:21:53.604 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:53.601+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.602609+0000) 2019-10-13T10:21:53.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:53.601+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.603133+0000) 2019-10-13T10:21:53.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:53.602+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.603739+0000) 2019-10-13T10:21:53.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:53.602+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.604122+0000) 2019-10-13T10:21:53.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:53.605+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:53.606120+0000) 2019-10-13T10:21:53.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:53.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185852 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:54.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:54.485+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:54.486667+0000) 2019-10-13T10:21:54.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:54.738+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:54.739348+0000) 2019-10-13T10:21:55.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:55.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:55.629664+0000) 2019-10-13T10:21:55.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:55.629+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:55.629664+0000) 2019-10-13T10:21:58.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:58.537+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.537647+0000) 2019-10-13T10:21:58.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:21:58.612+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.612774+0000) 2019-10-13T10:21:58.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:21:58.611+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.612057+0000) 2019-10-13T10:21:58.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:58.611+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.612209+0000) 2019-10-13T10:21:58.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:21:58.612+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.613100+0000) 2019-10-13T10:21:58.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:21:58.612+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.613241+0000) 2019-10-13T10:21:58.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:21:58.613+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.613453+0000) 2019-10-13T10:21:58.616 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:21:58.613+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.613623+0000) 2019-10-13T10:21:58.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:21:58.615+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.615802+0000) 2019-10-13T10:21:58.621 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:21:58.619+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:21:58.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:21:58.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.876072+0000) 2019-10-13T10:22:02.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:02.978+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.979368+0000) 2019-10-13T10:22:02.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:02.978+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.978500+0000) 2019-10-13T10:22:03.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:03.621+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.621528+0000) 2019-10-13T10:22:03.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:03.622+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.622780+0000) 2019-10-13T10:22:03.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:03.621+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.621996+0000) 2019-10-13T10:22:03.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:03.621+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.622239+0000) 2019-10-13T10:22:03.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:03.622+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.622436+0000) 2019-10-13T10:22:03.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:03.622+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.622639+0000) 2019-10-13T10:22:03.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:03.622+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.623246+0000) 2019-10-13T10:22:03.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:03.624+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:03.625336+0000) 2019-10-13T10:22:03.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:03.628+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:04.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:04.488+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:04.488980+0000) 2019-10-13T10:22:04.740 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:04.739+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:04.739552+0000) 2019-10-13T10:22:05.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:05.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:05.629834+0000) 2019-10-13T10:22:05.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:05.629+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:05.629842+0000) 2019-10-13T10:22:05.994 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:22:05.995 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:05.999 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:22:05.999 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:08.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:08.537+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.537888+0000) 2019-10-13T10:22:08.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:08.630+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.630768+0000) 2019-10-13T10:22:08.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:08.630+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.630981+0000) 2019-10-13T10:22:08.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:08.630+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.631148+0000) 2019-10-13T10:22:08.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:08.630+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.631336+0000) 2019-10-13T10:22:08.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:08.632+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.632844+0000) 2019-10-13T10:22:08.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:08.631+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.631594+0000) 2019-10-13T10:22:08.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:08.631+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.631767+0000) 2019-10-13T10:22:08.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:08.634+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.635137+0000) 2019-10-13T10:22:08.639 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:08.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185942 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:08.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:08.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.876218+0000) 2019-10-13T10:22:12.979 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:12.979+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.979537+0000) 2019-10-13T10:22:12.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:12.978+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.978672+0000) 2019-10-13T10:22:13.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:13.640+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.640454+0000) 2019-10-13T10:22:13.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:13.640+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.640647+0000) 2019-10-13T10:22:13.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:13.640+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.640904+0000) 2019-10-13T10:22:13.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:13.640+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.641070+0000) 2019-10-13T10:22:13.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:13.642+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.643015+0000) 2019-10-13T10:22:13.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:13.640+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.641354+0000) 2019-10-13T10:22:13.644 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:13.641+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.641941+0000) 2019-10-13T10:22:13.645 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:13.644+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:13.645009+0000) 2019-10-13T10:22:13.649 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:13.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 185974 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:14.491 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:14.491+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:14.491535+0000) 2019-10-13T10:22:14.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:14.739+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:14.739745+0000) 2019-10-13T10:22:15.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:15.629+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:15.630008+0000) 2019-10-13T10:22:15.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:15.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:15.630014+0000) 2019-10-13T10:22:18.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:18.537+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.538075+0000) 2019-10-13T10:22:18.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:18.649+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.650565+0000) 2019-10-13T10:22:18.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:18.648+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.649720+0000) 2019-10-13T10:22:18.651 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:18.648+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.649947+0000) 2019-10-13T10:22:18.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:18.648+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.650216+0000) 2019-10-13T10:22:18.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:18.648+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.650347+0000) 2019-10-13T10:22:18.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:18.649+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.650849+0000) 2019-10-13T10:22:18.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:18.649+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.651009+0000) 2019-10-13T10:22:18.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:18.652+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.653213+0000) 2019-10-13T10:22:18.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:18.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186004 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:18.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:18.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.876382+0000) 2019-10-13T10:22:22.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:22.979+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.979719+0000) 2019-10-13T10:22:22.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:22.977+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.978865+0000) 2019-10-13T10:22:23.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:23.659+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.660386+0000) 2019-10-13T10:22:23.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:23.658+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.659542+0000) 2019-10-13T10:22:23.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:23.658+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.659660+0000) 2019-10-13T10:22:23.662 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:23.658+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.659945+0000) 2019-10-13T10:22:23.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:23.658+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.660185+0000) 2019-10-13T10:22:23.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:23.659+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.660418+0000) 2019-10-13T10:22:23.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:23.659+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.660541+0000) 2019-10-13T10:22:23.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:23.661+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:23.662355+0000) 2019-10-13T10:22:23.668 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:23.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:24.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:24.492+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:24.493902+0000) 2019-10-13T10:22:24.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:24.739+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:24.739928+0000) 2019-10-13T10:22:25.631 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:25.628+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:25.630242+0000) 2019-10-13T10:22:25.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:25.628+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:25.630242+0000) 2019-10-13T10:22:28.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:28.536+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.538279+0000) 2019-10-13T10:22:28.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:28.667+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.669356+0000) 2019-10-13T10:22:28.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:28.668+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.669686+0000) 2019-10-13T10:22:28.672 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:28.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.670625+0000) 2019-10-13T10:22:28.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:28.668+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.669943+0000) 2019-10-13T10:22:28.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:28.668+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.670142+0000) 2019-10-13T10:22:28.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:28.670+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.672536+0000) 2019-10-13T10:22:28.674 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:28.668+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.670293+0000) 2019-10-13T10:22:28.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:28.669+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.670513+0000) 2019-10-13T10:22:28.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:28.675+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:28.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:28.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.876516+0000) 2019-10-13T10:22:32.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:32.979+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.979920+0000) 2019-10-13T10:22:32.980 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:32.978+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.979057+0000) 2019-10-13T10:22:33.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:33.677+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.679138+0000) 2019-10-13T10:22:33.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:33.678+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.679462+0000) 2019-10-13T10:22:33.681 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:33.678+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.680335+0000) 2019-10-13T10:22:33.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:33.678+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.679882+0000) 2019-10-13T10:22:33.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:33.678+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.680006+0000) 2019-10-13T10:22:33.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:33.678+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.680225+0000) 2019-10-13T10:22:33.682 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:33.679+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.680547+0000) 2019-10-13T10:22:33.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:33.682+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:33.684295+0000) 2019-10-13T10:22:33.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:33.685+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186096 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:34.496 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:34.495+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:34.496181+0000) 2019-10-13T10:22:34.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:34.739+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:34.740125+0000) 2019-10-13T10:22:35.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:35.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:35.630439+0000) 2019-10-13T10:22:35.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:35.629+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:35.630440+0000) 2019-10-13T10:22:36.177 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:22:36.177 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:36.184 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:22:36.184 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:38.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:38.537+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.538469+0000) 2019-10-13T10:22:38.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:38.686+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.688217+0000) 2019-10-13T10:22:38.690 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:38.687+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.688485+0000) 2019-10-13T10:22:38.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:38.687+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.688670+0000) 2019-10-13T10:22:38.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:38.689+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.690653+0000) 2019-10-13T10:22:38.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:38.687+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.689171+0000) 2019-10-13T10:22:38.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:38.687+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.689339+0000) 2019-10-13T10:22:38.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:38.688+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.689446+0000) 2019-10-13T10:22:38.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:38.691+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.692171+0000) 2019-10-13T10:22:38.697 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:38.694+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:38.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:38.876+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.876691+0000) 2019-10-13T10:22:42.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:42.978+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.979271+0000) 2019-10-13T10:22:42.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:42.979+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.980091+0000) 2019-10-13T10:22:43.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:43.697+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.697884+0000) 2019-10-13T10:22:43.711 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:43.697+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.698205+0000) 2019-10-13T10:22:43.711 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:43.698+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.698403+0000) 2019-10-13T10:22:43.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:43.698+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.698565+0000) 2019-10-13T10:22:43.712 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:43.699+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.699892+0000) 2019-10-13T10:22:43.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:43.700+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.700859+0000) 2019-10-13T10:22:43.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:43.698+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.698746+0000) 2019-10-13T10:22:43.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:43.698+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:43.699228+0000) 2019-10-13T10:22:43.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:43.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:44.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:44.497+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:44.498610+0000) 2019-10-13T10:22:44.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:44.739+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:44.740309+0000) 2019-10-13T10:22:45.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:45.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:45.630622+0000) 2019-10-13T10:22:45.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:45.630+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:45.630627+0000) 2019-10-13T10:22:48.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:48.538+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.538668+0000) 2019-10-13T10:22:48.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:48.707+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.707904+0000) 2019-10-13T10:22:48.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:48.708+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.708402+0000) 2019-10-13T10:22:48.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:48.708+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.708394+0000) 2019-10-13T10:22:48.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:48.708+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.708563+0000) 2019-10-13T10:22:48.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:48.708+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.708873+0000) 2019-10-13T10:22:48.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:48.709+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.709829+0000) 2019-10-13T10:22:48.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:48.708+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.709259+0000) 2019-10-13T10:22:48.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:48.711+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.712213+0000) 2019-10-13T10:22:48.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:48.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186185 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:48.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:48.876+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.876846+0000) 2019-10-13T10:22:52.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:52.979+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.979477+0000) 2019-10-13T10:22:52.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:52.979+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.980306+0000) 2019-10-13T10:22:53.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:53.717+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.717427+0000) 2019-10-13T10:22:53.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:53.718+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.719048+0000) 2019-10-13T10:22:53.720 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:53.717+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.717870+0000) 2019-10-13T10:22:53.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:53.717+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.718133+0000) 2019-10-13T10:22:53.720 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:53.717+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.718314+0000) 2019-10-13T10:22:53.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:53.718+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.718590+0000) 2019-10-13T10:22:53.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:53.718+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.718651+0000) 2019-10-13T10:22:53.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:53.720+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:53.720863+0000) 2019-10-13T10:22:53.726 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:53.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:54.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:54.500+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:54.501090+0000) 2019-10-13T10:22:54.741 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:54.740+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:54.740501+0000) 2019-10-13T10:22:55.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:55.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:55.630786+0000) 2019-10-13T10:22:55.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:55.630+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:55.630786+0000) 2019-10-13T10:22:58.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:58.538+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.538831+0000) 2019-10-13T10:22:58.726 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:22:58.724+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.724506+0000) 2019-10-13T10:22:58.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:22:58.724+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.724663+0000) 2019-10-13T10:22:58.728 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:22:58.725+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.725884+0000) 2019-10-13T10:22:58.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:58.724+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.725294+0000) 2019-10-13T10:22:58.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:22:58.725+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.725587+0000) 2019-10-13T10:22:58.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:22:58.725+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.725614+0000) 2019-10-13T10:22:58.729 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:22:58.725+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.726142+0000) 2019-10-13T10:22:58.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:22:58.728+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.729176+0000) 2019-10-13T10:22:58.740 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:22:58.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186246 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:22:58.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:22:58.876+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.876982+0000) 2019-10-13T10:23:02.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:02.979+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.980503+0000) 2019-10-13T10:23:02.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:02.978+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.979644+0000) 2019-10-13T10:23:03.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:03.731+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.732952+0000) 2019-10-13T10:23:03.734 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:03.731+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.733302+0000) 2019-10-13T10:23:03.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:03.734+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.734831+0000) 2019-10-13T10:23:03.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:03.732+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.733508+0000) 2019-10-13T10:23:03.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:03.732+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.733637+0000) 2019-10-13T10:23:03.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:03.732+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.733992+0000) 2019-10-13T10:23:03.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:03.732+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.734079+0000) 2019-10-13T10:23:03.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:03.735+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:03.736425+0000) 2019-10-13T10:23:03.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:03.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:04.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:04.503+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:04.503503+0000) 2019-10-13T10:23:04.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:04.739+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:04.740696+0000) 2019-10-13T10:23:05.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:05.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:05.630981+0000) 2019-10-13T10:23:05.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:05.630+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:05.630981+0000) 2019-10-13T10:23:06.370 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:23:06.370 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:06.376 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:23:06.376 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:08.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:08.538+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.539006+0000) 2019-10-13T10:23:08.748 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:08.743+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.744209+0000) 2019-10-13T10:23:08.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:08.741+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.742691+0000) 2019-10-13T10:23:08.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:08.741+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.742879+0000) 2019-10-13T10:23:08.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:08.741+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.743337+0000) 2019-10-13T10:23:08.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:08.742+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.743511+0000) 2019-10-13T10:23:08.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:08.742+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.743629+0000) 2019-10-13T10:23:08.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:08.742+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.743767+0000) 2019-10-13T10:23:08.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:08.746+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.747445+0000) 2019-10-13T10:23:08.811 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:08.748+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:08.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:08.876+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.877124+0000) 2019-10-13T10:23:12.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:12.980+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.980687+0000) 2019-10-13T10:23:12.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:12.979+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.979807+0000) 2019-10-13T10:23:13.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:13.750+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.751740+0000) 2019-10-13T10:23:13.753 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:13.750+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.751873+0000) 2019-10-13T10:23:13.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:13.750+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.752072+0000) 2019-10-13T10:23:13.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:13.753+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.754019+0000) 2019-10-13T10:23:13.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:13.751+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.752595+0000) 2019-10-13T10:23:13.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:13.751+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.753294+0000) 2019-10-13T10:23:13.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:13.752+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.753521+0000) 2019-10-13T10:23:13.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:13.755+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:13.756011+0000) 2019-10-13T10:23:13.760 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:13.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:14.509 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:14.505+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:14.505668+0000) 2019-10-13T10:23:14.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:14.739+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:14.740898+0000) 2019-10-13T10:23:15.632 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:15.629+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:15.631175+0000) 2019-10-13T10:23:15.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:15.629+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:15.631172+0000) 2019-10-13T10:23:18.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:18.537+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.539228+0000) 2019-10-13T10:23:18.763 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:18.760+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.762292+0000) 2019-10-13T10:23:18.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:18.759+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.761059+0000) 2019-10-13T10:23:18.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:18.759+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.761366+0000) 2019-10-13T10:23:18.764 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:18.760+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.761694+0000) 2019-10-13T10:23:18.764 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:18.760+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.761763+0000) 2019-10-13T10:23:18.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:18.760+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.761897+0000) 2019-10-13T10:23:18.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:18.760+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.762168+0000) 2019-10-13T10:23:18.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:18.763+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.765321+0000) 2019-10-13T10:23:18.815 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:18.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:18.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:18.875+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.877294+0000) 2019-10-13T10:23:22.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:22.980+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.980881+0000) 2019-10-13T10:23:22.981 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:22.979+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.979958+0000) 2019-10-13T10:23:23.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:23.770+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.771767+0000) 2019-10-13T10:23:23.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:23.769+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.770588+0000) 2019-10-13T10:23:23.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:23.769+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.771132+0000) 2019-10-13T10:23:23.773 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:23.769+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.771323+0000) 2019-10-13T10:23:23.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:23.770+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.771453+0000) 2019-10-13T10:23:23.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:23.770+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.771578+0000) 2019-10-13T10:23:23.774 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:23.770+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.771825+0000) 2019-10-13T10:23:23.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:23.772+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:23.774127+0000) 2019-10-13T10:23:23.779 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:23.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:24.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:24.507+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:24.508001+0000) 2019-10-13T10:23:24.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:24.740+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:24.741074+0000) 2019-10-13T10:23:25.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:25.630+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:25.631317+0000) 2019-10-13T10:23:25.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:25.630+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:25.631331+0000) 2019-10-13T10:23:28.540 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:28.538+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.539431+0000) 2019-10-13T10:23:28.781 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:28.779+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.781223+0000) 2019-10-13T10:23:28.782 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:28.779+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.780801+0000) 2019-10-13T10:23:28.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:28.779+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.780901+0000) 2019-10-13T10:23:28.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:28.779+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.781065+0000) 2019-10-13T10:23:28.783 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:28.779+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.781333+0000) 2019-10-13T10:23:28.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:28.780+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.781391+0000) 2019-10-13T10:23:28.783 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:28.780+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.781501+0000) 2019-10-13T10:23:28.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:28.781+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.783212+0000) 2019-10-13T10:23:28.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:28.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:28.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:28.877+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.877435+0000) 2019-10-13T10:23:32.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:32.980+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.981066+0000) 2019-10-13T10:23:32.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:32.979+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.980160+0000) 2019-10-13T10:23:33.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:33.789+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.790596+0000) 2019-10-13T10:23:33.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:33.789+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.789785+0000) 2019-10-13T10:23:33.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:33.789+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.790148+0000) 2019-10-13T10:23:33.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:33.789+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.790256+0000) 2019-10-13T10:23:33.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:33.790+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.790490+0000) 2019-10-13T10:23:33.793 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:33.790+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.790749+0000) 2019-10-13T10:23:33.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:33.790+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.790794+0000) 2019-10-13T10:23:33.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:33.792+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:33.793003+0000) 2019-10-13T10:23:33.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:33.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:34.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:34.509+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:34.510268+0000) 2019-10-13T10:23:34.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:34.740+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:34.741234+0000) 2019-10-13T10:23:35.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:35.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:35.631497+0000) 2019-10-13T10:23:35.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:35.631+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:35.631493+0000) 2019-10-13T10:23:36.524 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:23:36.524 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:36.529 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:23:36.529 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:38.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:38.539+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.539618+0000) 2019-10-13T10:23:38.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:38.798+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.799064+0000) 2019-10-13T10:23:38.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:38.799+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.800310+0000) 2019-10-13T10:23:38.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:38.799+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.799682+0000) 2019-10-13T10:23:38.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:38.799+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.799875+0000) 2019-10-13T10:23:38.802 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:38.799+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.799965+0000) 2019-10-13T10:23:38.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:38.799+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.800291+0000) 2019-10-13T10:23:38.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:38.800+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.800620+0000) 2019-10-13T10:23:38.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:38.802+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.802790+0000) 2019-10-13T10:23:38.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:38.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186488 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:38.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:38.877+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.877577+0000) 2019-10-13T10:23:42.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:42.980+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.981247+0000) 2019-10-13T10:23:42.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:42.980+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.980352+0000) 2019-10-13T10:23:43.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:43.809+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.810452+0000) 2019-10-13T10:23:43.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:43.809+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.809808+0000) 2019-10-13T10:23:43.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:43.809+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.809903+0000) 2019-10-13T10:23:43.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:43.809+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.810032+0000) 2019-10-13T10:23:43.814 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:43.809+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.810224+0000) 2019-10-13T10:23:43.814 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:43.809+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.810338+0000) 2019-10-13T10:23:43.814 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:43.810+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.810825+0000) 2019-10-13T10:23:43.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:43.813+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:43.814216+0000) 2019-10-13T10:23:43.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:43.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:44.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:44.512+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:44.512565+0000) 2019-10-13T10:23:44.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:44.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:44.741440+0000) 2019-10-13T10:23:45.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:45.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:45.631734+0000) 2019-10-13T10:23:45.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:45.631+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:45.631734+0000) 2019-10-13T10:23:48.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:48.538+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.539779+0000) 2019-10-13T10:23:48.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:48.817+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.818969+0000) 2019-10-13T10:23:48.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:48.817+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.819138+0000) 2019-10-13T10:23:48.821 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:48.817+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.819350+0000) 2019-10-13T10:23:48.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:48.818+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.819582+0000) 2019-10-13T10:23:48.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:48.820+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.820786+0000) 2019-10-13T10:23:48.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:48.818+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.820071+0000) 2019-10-13T10:23:48.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:48.818+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.820211+0000) 2019-10-13T10:23:48.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:48.822+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.823059+0000) 2019-10-13T10:23:48.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:48.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:48.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:48.876+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.877772+0000) 2019-10-13T10:23:52.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:52.980+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.981389+0000) 2019-10-13T10:23:52.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:52.979+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.980512+0000) 2019-10-13T10:23:53.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:53.826+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.828285+0000) 2019-10-13T10:23:53.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:53.828+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.829655+0000) 2019-10-13T10:23:53.831 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:53.827+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.829136+0000) 2019-10-13T10:23:53.831 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:53.827+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.829370+0000) 2019-10-13T10:23:53.831 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:53.828+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.829507+0000) 2019-10-13T10:23:53.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:53.828+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.829763+0000) 2019-10-13T10:23:53.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:53.828+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.829816+0000) 2019-10-13T10:23:53.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:53.831+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:53.831895+0000) 2019-10-13T10:23:53.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:53.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:54.515 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:54.514+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:54.514884+0000) 2019-10-13T10:23:54.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:54.740+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:54.741607+0000) 2019-10-13T10:23:55.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:55.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:55.631931+0000) 2019-10-13T10:23:55.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:55.631+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:55.631933+0000) 2019-10-13T10:23:58.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:58.539+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.539942+0000) 2019-10-13T10:23:58.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:23:58.836+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.838060+0000) 2019-10-13T10:23:58.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:58.837+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.838942+0000) 2019-10-13T10:23:58.840 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:23:58.837+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.839052+0000) 2019-10-13T10:23:58.841 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:23:58.837+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.839183+0000) 2019-10-13T10:23:58.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:23:58.839+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.840423+0000) 2019-10-13T10:23:58.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:23:58.838+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.839527+0000) 2019-10-13T10:23:58.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:23:58.838+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.839831+0000) 2019-10-13T10:23:58.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:23:58.840+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.841662+0000) 2019-10-13T10:23:58.847 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:23:58.844+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186610 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:23:58.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:23:58.876+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.877954+0000) 2019-10-13T10:24:02.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:02.980+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.981577+0000) 2019-10-13T10:24:02.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:02.979+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.980690+0000) 2019-10-13T10:24:03.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:03.846+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.848070+0000) 2019-10-13T10:24:03.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:03.846+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.848171+0000) 2019-10-13T10:24:03.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:03.848+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.849362+0000) 2019-10-13T10:24:03.851 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:03.847+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.848738+0000) 2019-10-13T10:24:03.851 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:03.847+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.848994+0000) 2019-10-13T10:24:03.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:03.847+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.849195+0000) 2019-10-13T10:24:03.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:03.848+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.849869+0000) 2019-10-13T10:24:03.853 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:03.851+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:03.852679+0000) 2019-10-13T10:24:03.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:03.854+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:04.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:04.516+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:04.516928+0000) 2019-10-13T10:24:04.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:04.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:04.741802+0000) 2019-10-13T10:24:05.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:05.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:05.632133+0000) 2019-10-13T10:24:05.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:05.631+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:05.632132+0000) 2019-10-13T10:24:06.700 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:24:06.701 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:06.728 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:24:06.729 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:08.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:08.538+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.540161+0000) 2019-10-13T10:24:08.859 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:08.856+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.857609+0000) 2019-10-13T10:24:08.859 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:08.856+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.857847+0000) 2019-10-13T10:24:08.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:08.857+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.859284+0000) 2019-10-13T10:24:08.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:08.856+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.858051+0000) 2019-10-13T10:24:08.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:08.857+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.858579+0000) 2019-10-13T10:24:08.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:08.857+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.858791+0000) 2019-10-13T10:24:08.862 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:08.857+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.859049+0000) 2019-10-13T10:24:08.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:08.859+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.861667+0000) 2019-10-13T10:24:08.866 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:08.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:08.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:08.876+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.878121+0000) 2019-10-13T10:24:12.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:12.980+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.981746+0000) 2019-10-13T10:24:12.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:12.980+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.980889+0000) 2019-10-13T10:24:13.868 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:13.866+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.868184+0000) 2019-10-13T10:24:13.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:13.865+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.867324+0000) 2019-10-13T10:24:13.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:13.866+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.867477+0000) 2019-10-13T10:24:13.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:13.866+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.867792+0000) 2019-10-13T10:24:13.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:13.866+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.867971+0000) 2019-10-13T10:24:13.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:13.867+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.868443+0000) 2019-10-13T10:24:13.870 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:13.866+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.868289+0000) 2019-10-13T10:24:13.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:13.869+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:13.871173+0000) 2019-10-13T10:24:13.876 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:13.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186702 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:14.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:14.518+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:14.519207+0000) 2019-10-13T10:24:14.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:14.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:14.741974+0000) 2019-10-13T10:24:15.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:15.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:15.632328+0000) 2019-10-13T10:24:15.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:15.631+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:15.632328+0000) 2019-10-13T10:24:18.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:18.539+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.540356+0000) 2019-10-13T10:24:18.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:18.875+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.877477+0000) 2019-10-13T10:24:18.878 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:18.876+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.876614+0000) 2019-10-13T10:24:18.878 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:18.876+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.876789+0000) 2019-10-13T10:24:18.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:18.876+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.876909+0000) 2019-10-13T10:24:18.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:18.876+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.877309+0000) 2019-10-13T10:24:18.879 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:18.877+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.877708+0000) 2019-10-13T10:24:18.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:18.877+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.877934+0000) 2019-10-13T10:24:18.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:18.877+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.878231+0000) 2019-10-13T10:24:18.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:18.878+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.880647+0000) 2019-10-13T10:24:18.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:18.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186732 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:22.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:22.981+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.981951+0000) 2019-10-13T10:24:22.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:22.980+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.981096+0000) 2019-10-13T10:24:23.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:23.886+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.887419+0000) 2019-10-13T10:24:23.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:23.886+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.886551+0000) 2019-10-13T10:24:23.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:23.886+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.886776+0000) 2019-10-13T10:24:23.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:23.886+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.886968+0000) 2019-10-13T10:24:23.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:23.887+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.887502+0000) 2019-10-13T10:24:23.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:23.887+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.887813+0000) 2019-10-13T10:24:23.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:23.887+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.888151+0000) 2019-10-13T10:24:23.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:23.890+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:23.891637+0000) 2019-10-13T10:24:23.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:23.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:24.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:24.520+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:24.521519+0000) 2019-10-13T10:24:24.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:24.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:24.742154+0000) 2019-10-13T10:24:25.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:25.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:25.632575+0000) 2019-10-13T10:24:25.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:25.632+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:25.632573+0000) 2019-10-13T10:24:28.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:28.540+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.540563+0000) 2019-10-13T10:24:28.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:28.878+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.878383+0000) 2019-10-13T10:24:28.898 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:28.896+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.896545+0000) 2019-10-13T10:24:28.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:28.897+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.898439+0000) 2019-10-13T10:24:28.899 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:28.896+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.896975+0000) 2019-10-13T10:24:28.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:28.896+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.897185+0000) 2019-10-13T10:24:28.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:28.897+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.897515+0000) 2019-10-13T10:24:28.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:28.897+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.897773+0000) 2019-10-13T10:24:28.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:28.897+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.898094+0000) 2019-10-13T10:24:28.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:28.900+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.901109+0000) 2019-10-13T10:24:28.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:28.904+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:32.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:32.981+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.982132+0000) 2019-10-13T10:24:32.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:32.981+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.981280+0000) 2019-10-13T10:24:33.907 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:33.904+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.906388+0000) 2019-10-13T10:24:33.908 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:33.905+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.906513+0000) 2019-10-13T10:24:33.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:33.907+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.908543+0000) 2019-10-13T10:24:33.909 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:33.905+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.906823+0000) 2019-10-13T10:24:33.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:33.905+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.907135+0000) 2019-10-13T10:24:33.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:33.906+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.907753+0000) 2019-10-13T10:24:33.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:33.906+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.907795+0000) 2019-10-13T10:24:33.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:33.909+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:33.910118+0000) 2019-10-13T10:24:33.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:33.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:34.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:34.523+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:34.523782+0000) 2019-10-13T10:24:34.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:34.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:34.742324+0000) 2019-10-13T10:24:35.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:35.631+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:35.632758+0000) 2019-10-13T10:24:35.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:35.631+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:35.632708+0000) 2019-10-13T10:24:36.893 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:24:36.894 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:36.899 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:24:36.899 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:38.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:38.539+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.540740+0000) 2019-10-13T10:24:38.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:38.877+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.878606+0000) 2019-10-13T10:24:38.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:38.915+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.916802+0000) 2019-10-13T10:24:38.919 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:38.917+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.918354+0000) 2019-10-13T10:24:38.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:38.915+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.917166+0000) 2019-10-13T10:24:38.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:38.916+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.917406+0000) 2019-10-13T10:24:38.919 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:38.916+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.917539+0000) 2019-10-13T10:24:38.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:38.916+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.918017+0000) 2019-10-13T10:24:38.920 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:38.916+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.918078+0000) 2019-10-13T10:24:38.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:38.920+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.920961+0000) 2019-10-13T10:24:38.926 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:38.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:42.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:42.980+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.981485+0000) 2019-10-13T10:24:42.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:42.981+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.982314+0000) 2019-10-13T10:24:43.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:43.925+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.926762+0000) 2019-10-13T10:24:43.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:43.927+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.927903+0000) 2019-10-13T10:24:43.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:43.925+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.927135+0000) 2019-10-13T10:24:43.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:43.925+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.927216+0000) 2019-10-13T10:24:43.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:43.925+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.927329+0000) 2019-10-13T10:24:43.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:43.926+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.927575+0000) 2019-10-13T10:24:43.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:43.926+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.928249+0000) 2019-10-13T10:24:43.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:43.930+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:43.930991+0000) 2019-10-13T10:24:43.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:43.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186884 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:44.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:44.525+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:44.526134+0000) 2019-10-13T10:24:44.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:44.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:44.742518+0000) 2019-10-13T10:24:45.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:45.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:45.632891+0000) 2019-10-13T10:24:45.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:45.632+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:45.632874+0000) 2019-10-13T10:24:48.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:48.539+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.540956+0000) 2019-10-13T10:24:48.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:48.878+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.878822+0000) 2019-10-13T10:24:48.937 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:48.934+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.936315+0000) 2019-10-13T10:24:48.938 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:48.936+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.937477+0000) 2019-10-13T10:24:48.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:48.935+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.936820+0000) 2019-10-13T10:24:48.939 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:48.935+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.937006+0000) 2019-10-13T10:24:48.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:48.935+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.937062+0000) 2019-10-13T10:24:48.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:48.935+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.937381+0000) 2019-10-13T10:24:48.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:48.936+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.937551+0000) 2019-10-13T10:24:48.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:48.940+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.940879+0000) 2019-10-13T10:24:48.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:48.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186914 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:52.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:52.982+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.982493+0000) 2019-10-13T10:24:52.983 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:52.981+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.981703+0000) 2019-10-13T10:24:53.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:53.944+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.945476+0000) 2019-10-13T10:24:53.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:53.944+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.945516+0000) 2019-10-13T10:24:53.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:53.946+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.947185+0000) 2019-10-13T10:24:53.948 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:53.944+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.946022+0000) 2019-10-13T10:24:53.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:53.944+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.946312+0000) 2019-10-13T10:24:53.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:53.945+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.946501+0000) 2019-10-13T10:24:53.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:53.945+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.946644+0000) 2019-10-13T10:24:53.957 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:53.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186946 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:24:53.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:53.948+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:53.949318+0000) 2019-10-13T10:24:54.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:54.528+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:54.528350+0000) 2019-10-13T10:24:54.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:54.742+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:54.742668+0000) 2019-10-13T10:24:55.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:55.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:55.633060+0000) 2019-10-13T10:24:55.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:55.632+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:55.633059+0000) 2019-10-13T10:24:58.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:58.540+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.541145+0000) 2019-10-13T10:24:58.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:58.878+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.879028+0000) 2019-10-13T10:24:58.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:24:58.953+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.955362+0000) 2019-10-13T10:24:58.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:24:58.953+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.954537+0000) 2019-10-13T10:24:58.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:24:58.953+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.954726+0000) 2019-10-13T10:24:58.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:24:58.953+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.955025+0000) 2019-10-13T10:24:58.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:24:58.953+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.955276+0000) 2019-10-13T10:24:58.957 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:24:58.953+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.955294+0000) 2019-10-13T10:24:58.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:24:58.954+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.955926+0000) 2019-10-13T10:24:58.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:24:58.957+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.958885+0000) 2019-10-13T10:24:58.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:24:58.961+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 186976 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:02.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:02.981+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.982663+0000) 2019-10-13T10:25:02.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:02.981+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.981868+0000) 2019-10-13T10:25:03.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:03.964+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.964801+0000) 2019-10-13T10:25:03.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:03.964+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.964955+0000) 2019-10-13T10:25:03.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:03.963+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.965708+0000) 2019-10-13T10:25:03.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:03.965+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.965456+0000) 2019-10-13T10:25:03.967 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:03.965+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.965929+0000) 2019-10-13T10:25:03.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:03.965+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.965969+0000) 2019-10-13T10:25:03.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:03.965+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.966120+0000) 2019-10-13T10:25:03.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:03.966+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:03.968348+0000) 2019-10-13T10:25:03.978 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:03.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:04.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:04.529+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:04.530634+0000) 2019-10-13T10:25:04.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:04.742+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:04.742837+0000) 2019-10-13T10:25:05.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:05.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:05.633225+0000) 2019-10-13T10:25:05.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:05.632+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:05.633225+0000) 2019-10-13T10:25:07.063 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:25:07.063 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:07.067 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:25:07.067 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:08.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:08.541+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.541308+0000) 2019-10-13T10:25:08.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:08.878+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.879218+0000) 2019-10-13T10:25:08.975 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:08.973+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.974175+0000) 2019-10-13T10:25:08.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:08.974+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.974583+0000) 2019-10-13T10:25:08.976 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:08.974+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.975061+0000) 2019-10-13T10:25:08.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:08.974+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.975092+0000) 2019-10-13T10:25:08.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:08.975+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.975415+0000) 2019-10-13T10:25:08.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:08.974+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.976260+0000) 2019-10-13T10:25:08.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:08.975+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.977261+0000) 2019-10-13T10:25:08.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:08.975+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.975892+0000) 2019-10-13T10:25:08.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:08.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:12.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:12.982+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.982839+0000) 2019-10-13T10:25:12.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:12.981+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.982039+0000) 2019-10-13T10:25:13.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:13.982+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.983302+0000) 2019-10-13T10:25:13.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:13.983+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.983561+0000) 2019-10-13T10:25:13.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:13.983+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.983733+0000) 2019-10-13T10:25:13.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:13.983+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.983886+0000) 2019-10-13T10:25:13.986 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:13.983+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.984136+0000) 2019-10-13T10:25:13.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:13.984+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.984506+0000) 2019-10-13T10:25:13.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:13.985+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.985782+0000) 2019-10-13T10:25:13.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:13.986+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:13.986776+0000) 2019-10-13T10:25:13.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:13.990+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:14.533 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:14.532+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:14.532985+0000) 2019-10-13T10:25:14.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:14.742+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:14.743012+0000) 2019-10-13T10:25:15.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:15.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:15.633385+0000) 2019-10-13T10:25:15.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:15.633+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:15.633385+0000) 2019-10-13T10:25:18.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:18.541+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.541472+0000) 2019-10-13T10:25:18.880 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:18.879+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.879405+0000) 2019-10-13T10:25:18.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:18.992+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.993114+0000) 2019-10-13T10:25:18.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:18.991+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.992354+0000) 2019-10-13T10:25:18.995 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:18.992+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.992767+0000) 2019-10-13T10:25:18.996 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:18.992+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.992990+0000) 2019-10-13T10:25:18.996 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:18.992+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.993143+0000) 2019-10-13T10:25:18.997 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:18.992+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.993265+0000) 2019-10-13T10:25:18.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:18.992+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.993314+0000) 2019-10-13T10:25:18.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:18.994+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.995662+0000) 2019-10-13T10:25:19.001 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:19.000+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:22.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:22.982+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.983020+0000) 2019-10-13T10:25:22.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:22.981+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.982192+0000) 2019-10-13T10:25:24.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:24.001+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.003286+0000) 2019-10-13T10:25:24.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:24.002+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.003495+0000) 2019-10-13T10:25:24.005 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:24.002+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.003744+0000) 2019-10-13T10:25:24.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:24.004+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.004963+0000) 2019-10-13T10:25:24.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:24.002+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.004141+0000) 2019-10-13T10:25:24.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:24.002+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.004381+0000) 2019-10-13T10:25:24.006 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:24.003+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.004746+0000) 2019-10-13T10:25:24.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:24.006+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.007646+0000) 2019-10-13T10:25:24.014 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:24.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:24.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:24.534+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.535366+0000) 2019-10-13T10:25:24.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:24.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.743216+0000) 2019-10-13T10:25:25.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:25.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:25.633581+0000) 2019-10-13T10:25:25.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:25.632+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:25.633587+0000) 2019-10-13T10:25:28.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:28.540+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:28.541659+0000) 2019-10-13T10:25:28.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:28.878+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:28.879568+0000) 2019-10-13T10:25:29.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:29.010+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.011634+0000) 2019-10-13T10:25:29.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:29.010+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.011839+0000) 2019-10-13T10:25:29.013 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:29.010+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.012084+0000) 2019-10-13T10:25:29.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:29.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.013535+0000) 2019-10-13T10:25:29.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:29.011+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.012578+0000) 2019-10-13T10:25:29.015 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:29.011+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.012735+0000) 2019-10-13T10:25:29.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:29.011+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.012889+0000) 2019-10-13T10:25:29.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:29.014+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.015371+0000) 2019-10-13T10:25:29.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:29.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:32.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:32.982+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.983225+0000) 2019-10-13T10:25:32.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:32.981+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.982373+0000) 2019-10-13T10:25:34.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:34.019+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.021160+0000) 2019-10-13T10:25:34.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:34.019+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.021372+0000) 2019-10-13T10:25:34.029 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:34.020+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.021518+0000) 2019-10-13T10:25:34.029 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:34.020+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.022051+0000) 2019-10-13T10:25:34.030 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:34.020+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.022213+0000) 2019-10-13T10:25:34.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:34.023+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.023988+0000) 2019-10-13T10:25:34.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:34.023+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.024158+0000) 2019-10-13T10:25:34.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:34.021+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.022407+0000) 2019-10-13T10:25:34.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:34.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:34.538 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:34.537+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.537854+0000) 2019-10-13T10:25:34.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:34.741+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.743366+0000) 2019-10-13T10:25:35.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:35.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:35.633832+0000) 2019-10-13T10:25:35.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:35.632+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:35.633805+0000) 2019-10-13T10:25:37.218 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:25:37.219 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:37.223 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:25:37.223 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:38.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:38.541+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:38.541794+0000) 2019-10-13T10:25:38.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:38.879+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:38.879756+0000) 2019-10-13T10:25:39.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:39.029+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.031083+0000) 2019-10-13T10:25:39.033 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:39.029+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.031208+0000) 2019-10-13T10:25:39.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:39.029+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.031299+0000) 2019-10-13T10:25:39.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:39.030+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.031605+0000) 2019-10-13T10:25:39.033 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:39.030+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.031756+0000) 2019-10-13T10:25:39.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:39.032+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.033479+0000) 2019-10-13T10:25:39.036 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:39.033+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.034162+0000) 2019-10-13T10:25:39.036 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:39.030+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.031959+0000) 2019-10-13T10:25:39.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:39.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:42.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:42.983+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.983428+0000) 2019-10-13T10:25:42.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:42.982+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:42.982575+0000) 2019-10-13T10:25:44.041 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:44.038+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.040076+0000) 2019-10-13T10:25:44.042 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:44.039+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.040513+0000) 2019-10-13T10:25:44.042 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:44.039+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.040785+0000) 2019-10-13T10:25:44.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:44.039+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.040969+0000) 2019-10-13T10:25:44.042 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:44.039+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.041179+0000) 2019-10-13T10:25:44.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:44.041+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.042500+0000) 2019-10-13T10:25:44.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:44.042+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.043307+0000) 2019-10-13T10:25:44.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:44.039+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.041323+0000) 2019-10-13T10:25:44.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:44.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:44.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:44.540+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.540237+0000) 2019-10-13T10:25:44.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:44.743+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.743546+0000) 2019-10-13T10:25:45.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:45.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:45.634007+0000) 2019-10-13T10:25:45.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:45.633+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:45.633996+0000) 2019-10-13T10:25:48.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:48.541+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.542000+0000) 2019-10-13T10:25:48.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:48.879+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.879906+0000) 2019-10-13T10:25:49.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:49.048+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.050410+0000) 2019-10-13T10:25:49.051 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:49.049+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.049542+0000) 2019-10-13T10:25:49.052 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:49.049+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.050038+0000) 2019-10-13T10:25:49.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:49.049+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.050108+0000) 2019-10-13T10:25:49.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:49.049+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.049855+0000) 2019-10-13T10:25:49.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:49.050+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.050464+0000) 2019-10-13T10:25:49.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:49.050+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.050617+0000) 2019-10-13T10:25:49.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:49.051+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.053304+0000) 2019-10-13T10:25:49.058 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:49.056+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:52.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:52.982+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.983587+0000) 2019-10-13T10:25:52.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:52.982+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:52.982752+0000) 2019-10-13T10:25:54.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:54.058+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.060346+0000) 2019-10-13T10:25:54.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:54.058+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.058868+0000) 2019-10-13T10:25:54.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:54.058+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.058978+0000) 2019-10-13T10:25:54.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:54.058+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.059208+0000) 2019-10-13T10:25:54.062 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:54.059+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.059549+0000) 2019-10-13T10:25:54.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:54.059+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.059826+0000) 2019-10-13T10:25:54.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:54.059+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.059984+0000) 2019-10-13T10:25:54.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:54.061+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.062857+0000) 2019-10-13T10:25:54.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:54.066+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:25:54.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:54.541+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.543439+0000) 2019-10-13T10:25:54.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:54.743+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.743717+0000) 2019-10-13T10:25:55.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:55.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:55.634217+0000) 2019-10-13T10:25:55.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:55.633+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:55.634217+0000) 2019-10-13T10:25:58.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:58.542+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.542198+0000) 2019-10-13T10:25:58.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:58.879+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.880077+0000) 2019-10-13T10:25:59.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:25:59.068+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.068729+0000) 2019-10-13T10:25:59.070 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:25:59.068+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.069101+0000) 2019-10-13T10:25:59.071 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:25:59.068+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.068922+0000) 2019-10-13T10:25:59.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:25:59.068+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.070582+0000) 2019-10-13T10:25:59.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:25:59.069+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.069450+0000) 2019-10-13T10:25:59.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:25:59.069+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.069929+0000) 2019-10-13T10:25:59.072 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:25:59.069+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.070096+0000) 2019-10-13T10:25:59.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:25:59.071+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.073338+0000) 2019-10-13T10:25:59.078 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:25:59.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:02.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:02.983+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.983758+0000) 2019-10-13T10:26:02.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:02.982+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:02.982880+0000) 2019-10-13T10:26:04.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:04.078+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.078580+0000) 2019-10-13T10:26:04.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:04.078+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.078729+0000) 2019-10-13T10:26:04.080 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:04.078+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.079056+0000) 2019-10-13T10:26:04.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:04.078+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.078978+0000) 2019-10-13T10:26:04.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:04.079+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.080479+0000) 2019-10-13T10:26:04.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:04.078+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.079316+0000) 2019-10-13T10:26:04.082 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:04.079+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.079814+0000) 2019-10-13T10:26:04.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:04.082+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.082775+0000) 2019-10-13T10:26:04.087 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:04.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:04.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:04.545+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.545912+0000) 2019-10-13T10:26:04.752 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:04.743+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.743891+0000) 2019-10-13T10:26:05.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:05.632+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:05.634340+0000) 2019-10-13T10:26:05.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:05.632+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:05.634343+0000) 2019-10-13T10:26:07.407 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:26:07.408 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:07.412 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:26:07.412 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:08.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:08.541+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.542374+0000) 2019-10-13T10:26:08.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:08.879+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.880259+0000) 2019-10-13T10:26:09.089 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:09.086+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.087818+0000) 2019-10-13T10:26:09.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:09.086+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.088273+0000) 2019-10-13T10:26:09.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:09.087+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.088456+0000) 2019-10-13T10:26:09.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:09.087+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.088587+0000) 2019-10-13T10:26:09.090 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:09.087+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.088473+0000) 2019-10-13T10:26:09.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:09.087+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.088803+0000) 2019-10-13T10:26:09.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:09.089+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.089764+0000) 2019-10-13T10:26:09.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:09.090+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.091294+0000) 2019-10-13T10:26:09.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:09.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:12.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:12.983+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.983942+0000) 2019-10-13T10:26:12.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:12.982+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:12.983028+0000) 2019-10-13T10:26:14.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:14.094+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.095973+0000) 2019-10-13T10:26:14.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:14.094+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.096244+0000) 2019-10-13T10:26:14.098 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:14.094+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.096373+0000) 2019-10-13T10:26:14.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:14.095+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.096565+0000) 2019-10-13T10:26:14.099 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:14.095+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.096766+0000) 2019-10-13T10:26:14.099 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:14.095+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.097187+0000) 2019-10-13T10:26:14.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:14.097+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.097897+0000) 2019-10-13T10:26:14.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:14.099+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.100503+0000) 2019-10-13T10:26:14.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:14.099+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:14.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:14.547+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.548361+0000) 2019-10-13T10:26:14.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:14.743+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.744041+0000) 2019-10-13T10:26:15.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:15.633+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:15.634510+0000) 2019-10-13T10:26:15.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:15.633+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:15.634510+0000) 2019-10-13T10:26:18.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:18.541+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.542529+0000) 2019-10-13T10:26:18.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:18.879+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.880466+0000) 2019-10-13T10:26:19.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:19.103+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.104425+0000) 2019-10-13T10:26:19.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:19.102+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.103596+0000) 2019-10-13T10:26:19.107 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:19.102+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.103720+0000) 2019-10-13T10:26:19.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:19.102+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.103889+0000) 2019-10-13T10:26:19.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:19.102+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.104175+0000) 2019-10-13T10:26:19.112 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:19.103+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.104449+0000) 2019-10-13T10:26:19.112 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:19.103+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.104540+0000) 2019-10-13T10:26:19.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:19.107+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.107965+0000) 2019-10-13T10:26:19.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:19.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:22.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:22.983+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.984140+0000) 2019-10-13T10:26:22.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:22.981+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:22.983249+0000) 2019-10-13T10:26:24.113 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:24.110+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.112246+0000) 2019-10-13T10:26:24.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:24.111+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.112498+0000) 2019-10-13T10:26:24.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:24.112+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.113439+0000) 2019-10-13T10:26:24.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:24.111+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.112680+0000) 2019-10-13T10:26:24.115 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:24.111+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.112792+0000) 2019-10-13T10:26:24.115 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:24.112+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.113421+0000) 2019-10-13T10:26:24.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:24.112+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.113709+0000) 2019-10-13T10:26:24.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:24.115+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.116638+0000) 2019-10-13T10:26:24.121 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:24.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:24.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:24.550+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.550859+0000) 2019-10-13T10:26:24.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:24.743+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.744214+0000) 2019-10-13T10:26:25.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:25.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:25.634681+0000) 2019-10-13T10:26:25.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:25.634+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:25.634681+0000) 2019-10-13T10:26:28.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:28.542+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.542709+0000) 2019-10-13T10:26:28.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:28.880+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.880660+0000) 2019-10-13T10:26:29.123 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:29.120+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.122095+0000) 2019-10-13T10:26:29.124 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:29.120+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.122229+0000) 2019-10-13T10:26:29.124 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:29.121+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.122413+0000) 2019-10-13T10:26:29.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:29.121+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.122729+0000) 2019-10-13T10:26:29.125 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:29.123+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.123809+0000) 2019-10-13T10:26:29.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:29.121+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.123117+0000) 2019-10-13T10:26:29.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:29.121+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.122884+0000) 2019-10-13T10:26:29.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:29.125+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.126335+0000) 2019-10-13T10:26:29.131 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:29.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:32.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:32.984+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.984312+0000) 2019-10-13T10:26:32.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:32.982+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:32.983418+0000) 2019-10-13T10:26:34.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:34.132+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.133155+0000) 2019-10-13T10:26:34.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:34.131+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.132239+0000) 2019-10-13T10:26:34.134 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:34.132+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.132543+0000) 2019-10-13T10:26:34.134 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:34.132+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.132775+0000) 2019-10-13T10:26:34.135 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:34.132+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.133007+0000) 2019-10-13T10:26:34.135 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:34.132+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.132927+0000) 2019-10-13T10:26:34.135 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:34.132+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.133210+0000) 2019-10-13T10:26:34.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:34.135+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.136467+0000) 2019-10-13T10:26:34.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:34.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:34.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:34.552+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.553152+0000) 2019-10-13T10:26:34.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:34.744+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.744391+0000) 2019-10-13T10:26:35.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:35.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:35.634869+0000) 2019-10-13T10:26:35.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:35.634+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:35.634869+0000) 2019-10-13T10:26:37.596 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:26:37.597 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:37.601 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:26:37.601 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:38.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:38.542+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.542873+0000) 2019-10-13T10:26:38.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:38.880+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.880838+0000) 2019-10-13T10:26:39.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:39.140+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.141066+0000) 2019-10-13T10:26:39.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:39.139+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.140250+0000) 2019-10-13T10:26:39.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:39.140+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.140422+0000) 2019-10-13T10:26:39.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:39.140+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.140791+0000) 2019-10-13T10:26:39.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:39.140+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.141189+0000) 2019-10-13T10:26:39.144 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:39.141+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.141582+0000) 2019-10-13T10:26:39.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:39.141+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.141857+0000) 2019-10-13T10:26:39.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:39.144+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:39.144912+0000) 2019-10-13T10:26:39.150 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:39.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:42.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:42.983+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.984471+0000) 2019-10-13T10:26:42.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:42.983+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:42.983594+0000) 2019-10-13T10:26:44.152 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:44.150+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.151142+0000) 2019-10-13T10:26:44.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:44.150+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.151325+0000) 2019-10-13T10:26:44.153 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:44.150+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.152488+0000) 2019-10-13T10:26:44.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:44.151+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.151706+0000) 2019-10-13T10:26:44.154 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:44.151+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.151977+0000) 2019-10-13T10:26:44.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:44.151+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.152131+0000) 2019-10-13T10:26:44.154 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:44.151+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.152258+0000) 2019-10-13T10:26:44.155 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:44.153+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.155671+0000) 2019-10-13T10:26:44.159 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:44.157+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:44.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:44.555+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.556154+0000) 2019-10-13T10:26:44.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:44.744+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.744584+0000) 2019-10-13T10:26:45.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:45.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:45.635019+0000) 2019-10-13T10:26:45.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:45.634+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:45.635015+0000) 2019-10-13T10:26:48.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:48.542+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.543038+0000) 2019-10-13T10:26:48.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:48.880+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.881036+0000) 2019-10-13T10:26:49.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:49.159+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.160318+0000) 2019-10-13T10:26:49.162 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:49.160+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.160568+0000) 2019-10-13T10:26:49.162 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:49.160+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.160706+0000) 2019-10-13T10:26:49.163 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:49.159+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.161407+0000) 2019-10-13T10:26:49.163 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:49.160+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.161050+0000) 2019-10-13T10:26:49.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:49.160+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.161198+0000) 2019-10-13T10:26:49.163 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:49.161+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.161741+0000) 2019-10-13T10:26:49.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:49.162+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:49.164524+0000) 2019-10-13T10:26:49.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:49.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:52.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:52.983+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.984638+0000) 2019-10-13T10:26:52.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:52.982+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:52.983804+0000) 2019-10-13T10:26:54.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:54.168+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.169602+0000) 2019-10-13T10:26:54.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:54.168+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.169728+0000) 2019-10-13T10:26:54.172 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:54.168+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.169866+0000) 2019-10-13T10:26:54.172 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:54.168+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.170144+0000) 2019-10-13T10:26:54.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:54.169+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.170637+0000) 2019-10-13T10:26:54.173 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:54.169+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.170797+0000) 2019-10-13T10:26:54.173 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:54.170+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.171481+0000) 2019-10-13T10:26:54.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:54.173+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.174093+0000) 2019-10-13T10:26:54.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:54.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:26:54.559 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:54.557+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.558586+0000) 2019-10-13T10:26:54.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:54.743+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.744776+0000) 2019-10-13T10:26:55.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:55.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:55.635202+0000) 2019-10-13T10:26:55.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:55.634+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:55.635203+0000) 2019-10-13T10:26:58.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:58.542+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.543246+0000) 2019-10-13T10:26:58.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:58.880+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.881195+0000) 2019-10-13T10:26:59.182 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:26:59.177+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.179054+0000) 2019-10-13T10:26:59.182 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:26:59.177+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.179346+0000) 2019-10-13T10:26:59.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:26:59.177+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.179376+0000) 2019-10-13T10:26:59.182 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:26:59.178+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.179521+0000) 2019-10-13T10:26:59.183 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:26:59.180+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.180989+0000) 2019-10-13T10:26:59.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:26:59.182+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.183084+0000) 2019-10-13T10:26:59.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:26:59.178+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.179637+0000) 2019-10-13T10:26:59.184 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:26:59.178+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:59.180260+0000) 2019-10-13T10:26:59.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:26:59.183+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:02.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:02.984+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.984795+0000) 2019-10-13T10:27:02.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:02.983+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:02.983997+0000) 2019-10-13T10:27:04.187 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:04.186+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.187428+0000) 2019-10-13T10:27:04.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:04.185+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.186719+0000) 2019-10-13T10:27:04.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:04.185+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.186829+0000) 2019-10-13T10:27:04.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:04.185+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.187232+0000) 2019-10-13T10:27:04.189 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:04.185+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.187149+0000) 2019-10-13T10:27:04.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:04.186+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.187755+0000) 2019-10-13T10:27:04.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:04.186+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.187881+0000) 2019-10-13T10:27:04.193 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:04.189+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.190518+0000) 2019-10-13T10:27:04.196 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:04.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:04.561 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:04.560+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.560747+0000) 2019-10-13T10:27:04.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:04.744+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.744974+0000) 2019-10-13T10:27:05.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:05.634+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:05.635412+0000) 2019-10-13T10:27:05.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:05.634+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:05.635412+0000) 2019-10-13T10:27:07.763 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:27:07.763 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:07.775 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:27:07.776 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:08.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:08.542+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.543392+0000) 2019-10-13T10:27:08.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:08.880+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.881378+0000) 2019-10-13T10:27:09.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:09.195+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.196798+0000) 2019-10-13T10:27:09.198 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:09.195+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.196894+0000) 2019-10-13T10:27:09.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:09.195+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.197023+0000) 2019-10-13T10:27:09.199 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:09.195+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.197184+0000) 2019-10-13T10:27:09.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:09.196+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.197487+0000) 2019-10-13T10:27:09.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:09.197+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.198374+0000) 2019-10-13T10:27:09.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:09.199+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.199808+0000) 2019-10-13T10:27:09.201 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:09.196+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:09.197910+0000) 2019-10-13T10:27:09.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:09.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:12.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:12.984+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.984962+0000) 2019-10-13T10:27:12.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:12.983+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:12.984161+0000) 2019-10-13T10:27:14.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:14.206+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.207714+0000) 2019-10-13T10:27:14.209 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:14.204+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.206328+0000) 2019-10-13T10:27:14.209 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:14.204+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.206352+0000) 2019-10-13T10:27:14.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:14.205+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.206909+0000) 2019-10-13T10:27:14.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:14.205+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.207013+0000) 2019-10-13T10:27:14.209 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:14.205+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.207318+0000) 2019-10-13T10:27:14.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:14.206+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.207883+0000) 2019-10-13T10:27:14.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:14.209+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.210637+0000) 2019-10-13T10:27:14.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:14.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:14.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:14.562+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.562945+0000) 2019-10-13T10:27:14.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:14.744+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.745126+0000) 2019-10-13T10:27:15.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:15.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:15.635614+0000) 2019-10-13T10:27:15.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:15.635+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:15.635614+0000) 2019-10-13T10:27:18.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:18.543+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.543565+0000) 2019-10-13T10:27:18.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:18.881+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.881600+0000) 2019-10-13T10:27:19.218 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:19.215+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.216063+0000) 2019-10-13T10:27:19.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:19.215+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.216196+0000) 2019-10-13T10:27:19.219 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:19.216+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.216411+0000) 2019-10-13T10:27:19.219 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:19.216+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.216792+0000) 2019-10-13T10:27:19.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:19.216+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.217099+0000) 2019-10-13T10:27:19.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:19.217+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.217937+0000) 2019-10-13T10:27:19.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:19.217+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.217556+0000) 2019-10-13T10:27:19.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:19.219+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:19.220347+0000) 2019-10-13T10:27:19.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:19.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187823 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:22.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:22.984+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.985143+0000) 2019-10-13T10:27:22.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:22.983+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:22.984301+0000) 2019-10-13T10:27:24.226 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:24.225+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.226477+0000) 2019-10-13T10:27:24.228 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:24.225+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.225613+0000) 2019-10-13T10:27:24.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:24.225+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.225722+0000) 2019-10-13T10:27:24.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:24.225+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.225815+0000) 2019-10-13T10:27:24.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:24.225+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.226029+0000) 2019-10-13T10:27:24.229 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:24.226+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.226418+0000) 2019-10-13T10:27:24.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:24.225+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.226335+0000) 2019-10-13T10:27:24.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:24.228+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.229257+0000) 2019-10-13T10:27:24.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:24.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:24.565 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:24.565+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.565389+0000) 2019-10-13T10:27:24.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:24.744+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.745295+0000) 2019-10-13T10:27:25.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:25.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:25.635805+0000) 2019-10-13T10:27:25.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:25.635+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:25.635804+0000) 2019-10-13T10:27:28.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:28.543+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.543757+0000) 2019-10-13T10:27:28.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:28.881+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.881754+0000) 2019-10-13T10:27:29.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:29.232+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.233271+0000) 2019-10-13T10:27:29.235 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:29.233+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.233421+0000) 2019-10-13T10:27:29.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:29.233+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.233789+0000) 2019-10-13T10:27:29.236 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:29.233+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.234009+0000) 2019-10-13T10:27:29.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:29.233+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.234113+0000) 2019-10-13T10:27:29.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:29.233+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.234337+0000) 2019-10-13T10:27:29.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:29.234+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.235687+0000) 2019-10-13T10:27:29.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:29.236+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:29.236743+0000) 2019-10-13T10:27:29.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:29.240+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:32.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:32.984+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.985314+0000) 2019-10-13T10:27:32.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:32.984+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:32.984480+0000) 2019-10-13T10:27:34.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:34.242+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.243015+0000) 2019-10-13T10:27:34.245 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:34.242+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.243135+0000) 2019-10-13T10:27:34.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:34.242+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.243339+0000) 2019-10-13T10:27:34.246 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:34.243+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.245107+0000) 2019-10-13T10:27:34.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:34.243+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.243695+0000) 2019-10-13T10:27:34.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:34.243+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.243927+0000) 2019-10-13T10:27:34.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:34.244+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.244413+0000) 2019-10-13T10:27:34.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:34.245+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.247286+0000) 2019-10-13T10:27:34.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:34.251+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:34.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:34.566+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.567499+0000) 2019-10-13T10:27:34.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:34.745+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.745465+0000) 2019-10-13T10:27:35.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:35.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:35.636011+0000) 2019-10-13T10:27:35.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:35.635+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:35.636010+0000) 2019-10-13T10:27:37.976 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:27:37.976 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:37.980 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:27:37.981 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:38.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:38.542+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.543945+0000) 2019-10-13T10:27:38.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:38.880+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.881924+0000) 2019-10-13T10:27:39.255 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:39.251+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.253323+0000) 2019-10-13T10:27:39.255 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:39.252+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.253533+0000) 2019-10-13T10:27:39.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:39.252+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.253703+0000) 2019-10-13T10:27:39.256 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:39.253+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.255297+0000) 2019-10-13T10:27:39.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:39.252+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.254041+0000) 2019-10-13T10:27:39.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:39.253+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.254594+0000) 2019-10-13T10:27:39.257 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:39.253+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.254721+0000) 2019-10-13T10:27:39.258 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:39.256+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:39.258443+0000) 2019-10-13T10:27:39.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:39.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:42.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:42.984+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.985499+0000) 2019-10-13T10:27:42.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:42.983+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:42.984677+0000) 2019-10-13T10:27:44.264 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:44.261+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.262627+0000) 2019-10-13T10:27:44.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:44.261+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.263195+0000) 2019-10-13T10:27:44.264 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:44.261+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.263337+0000) 2019-10-13T10:27:44.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:44.263+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.264856+0000) 2019-10-13T10:27:44.265 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:44.262+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.263942+0000) 2019-10-13T10:27:44.266 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:44.262+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.264046+0000) 2019-10-13T10:27:44.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:44.262+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.263468+0000) 2019-10-13T10:27:44.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:44.264+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.266231+0000) 2019-10-13T10:27:44.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:44.268+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 187975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:44.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:44.568+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.569714+0000) 2019-10-13T10:27:44.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:44.744+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.745638+0000) 2019-10-13T10:27:45.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:45.635+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:45.636175+0000) 2019-10-13T10:27:45.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:45.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:45.636209+0000) 2019-10-13T10:27:48.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:48.543+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.544125+0000) 2019-10-13T10:27:48.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:48.881+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.882099+0000) 2019-10-13T10:27:49.274 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:49.270+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.272148+0000) 2019-10-13T10:27:49.274 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:49.270+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.272382+0000) 2019-10-13T10:27:49.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:49.271+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.272502+0000) 2019-10-13T10:27:49.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:49.271+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.272783+0000) 2019-10-13T10:27:49.276 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:49.273+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.274016+0000) 2019-10-13T10:27:49.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:49.274+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.275245+0000) 2019-10-13T10:27:49.276 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:49.271+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.273238+0000) 2019-10-13T10:27:49.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:49.272+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:49.273567+0000) 2019-10-13T10:27:49.280 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:49.278+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188007 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:52.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:52.984+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.985684+0000) 2019-10-13T10:27:52.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:52.983+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:52.984845+0000) 2019-10-13T10:27:54.282 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:54.279+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.281379+0000) 2019-10-13T10:27:54.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:54.280+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.281574+0000) 2019-10-13T10:27:54.284 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:54.282+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.282780+0000) 2019-10-13T10:27:54.284 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:54.280+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.282264+0000) 2019-10-13T10:27:54.284 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:54.281+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.282414+0000) 2019-10-13T10:27:54.285 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:54.281+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.282605+0000) 2019-10-13T10:27:54.285 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:54.281+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.283013+0000) 2019-10-13T10:27:54.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:54.284+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.285374+0000) 2019-10-13T10:27:54.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:54.287+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:27:54.572 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:54.571+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.571764+0000) 2019-10-13T10:27:54.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:54.744+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.745773+0000) 2019-10-13T10:27:55.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:55.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:55.636374+0000) 2019-10-13T10:27:55.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:55.635+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:55.636379+0000) 2019-10-13T10:27:58.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:58.543+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.544315+0000) 2019-10-13T10:27:58.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:58.881+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.882311+0000) 2019-10-13T10:27:59.292 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:27:59.289+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.290971+0000) 2019-10-13T10:27:59.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:27:59.289+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.291089+0000) 2019-10-13T10:27:59.293 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:27:59.289+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.291263+0000) 2019-10-13T10:27:59.293 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:27:59.290+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.291554+0000) 2019-10-13T10:27:59.293 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:27:59.290+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.291610+0000) 2019-10-13T10:27:59.294 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:27:59.292+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.293253+0000) 2019-10-13T10:27:59.294 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:27:59.290+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.291832+0000) 2019-10-13T10:27:59.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:27:59.294+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:59.295207+0000) 2019-10-13T10:27:59.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:27:59.297+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188067 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:02.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:02.985+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:02.985834+0000) 2019-10-13T10:28:02.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:02.984+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:02.985002+0000) 2019-10-13T10:28:04.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:04.298+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.299879+0000) 2019-10-13T10:28:04.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:04.298+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.300060+0000) 2019-10-13T10:28:04.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:04.301+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.302436+0000) 2019-10-13T10:28:04.303 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:04.299+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.301196+0000) 2019-10-13T10:28:04.303 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:04.299+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.301302+0000) 2019-10-13T10:28:04.303 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:04.300+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.301997+0000) 2019-10-13T10:28:04.304 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:04.300+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.302353+0000) 2019-10-13T10:28:04.304 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:04.303+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.304576+0000) 2019-10-13T10:28:04.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:04.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:04.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:04.573+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.574078+0000) 2019-10-13T10:28:04.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:04.745+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.745951+0000) 2019-10-13T10:28:05.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:05.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:05.636598+0000) 2019-10-13T10:28:05.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:05.636+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:05.636582+0000) 2019-10-13T10:28:08.163 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:28:08.163 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:08.167 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:28:08.167 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:08.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:08.544+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.544511+0000) 2019-10-13T10:28:08.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:08.882+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.882438+0000) 2019-10-13T10:28:09.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:09.309+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.310198+0000) 2019-10-13T10:28:09.312 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:09.310+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.310621+0000) 2019-10-13T10:28:09.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:09.310+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.311189+0000) 2019-10-13T10:28:09.312 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:09.310+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.311307+0000) 2019-10-13T10:28:09.313 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:09.311+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.312448+0000) 2019-10-13T10:28:09.313 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:09.311+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.311676+0000) 2019-10-13T10:28:09.313 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:09.311+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.311852+0000) 2019-10-13T10:28:09.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:09.312+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:09.313477+0000) 2019-10-13T10:28:09.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:09.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188129 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:12.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:12.985+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:12.986015+0000) 2019-10-13T10:28:12.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:12.984+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:12.985162+0000) 2019-10-13T10:28:14.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:14.320+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.320425+0000) 2019-10-13T10:28:14.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:14.320+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.320559+0000) 2019-10-13T10:28:14.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:14.320+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.320988+0000) 2019-10-13T10:28:14.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:14.321+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.322168+0000) 2019-10-13T10:28:14.323 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:14.321+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.321480+0000) 2019-10-13T10:28:14.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:14.321+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.321764+0000) 2019-10-13T10:28:14.323 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:14.321+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.321916+0000) 2019-10-13T10:28:14.324 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:14.322+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.323245+0000) 2019-10-13T10:28:14.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:14.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188159 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:14.577 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:14.575+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.576594+0000) 2019-10-13T10:28:14.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:14.745+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.746130+0000) 2019-10-13T10:28:15.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:15.636+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:15.636759+0000) 2019-10-13T10:28:15.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:15.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:15.636806+0000) 2019-10-13T10:28:18.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:18.544+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.544727+0000) 2019-10-13T10:28:18.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:18.882+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.882576+0000) 2019-10-13T10:28:19.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:19.329+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.330160+0000) 2019-10-13T10:28:19.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:19.329+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.330261+0000) 2019-10-13T10:28:19.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:19.330+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.330599+0000) 2019-10-13T10:28:19.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:19.330+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.330757+0000) 2019-10-13T10:28:19.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:19.330+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.331433+0000) 2019-10-13T10:28:19.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:19.332+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.333048+0000) 2019-10-13T10:28:19.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:19.330+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.330925+0000) 2019-10-13T10:28:19.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:19.330+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:19.331040+0000) 2019-10-13T10:28:19.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:19.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188189 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:22.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:22.985+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:22.986206+0000) 2019-10-13T10:28:22.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:22.985+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:22.985292+0000) 2019-10-13T10:28:24.340 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:24.338+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.340156+0000) 2019-10-13T10:28:24.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:24.337+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.339388+0000) 2019-10-13T10:28:24.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:24.338+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.339869+0000) 2019-10-13T10:28:24.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:24.338+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.339975+0000) 2019-10-13T10:28:24.342 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:24.338+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.340163+0000) 2019-10-13T10:28:24.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:24.338+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.340328+0000) 2019-10-13T10:28:24.342 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:24.339+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.340775+0000) 2019-10-13T10:28:24.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:24.340+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.342685+0000) 2019-10-13T10:28:24.348 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:24.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:24.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:24.578+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.579115+0000) 2019-10-13T10:28:24.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:24.745+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.746280+0000) 2019-10-13T10:28:25.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:25.635+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:25.636913+0000) 2019-10-13T10:28:25.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:25.635+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:25.636940+0000) 2019-10-13T10:28:28.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:28.543+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.544912+0000) 2019-10-13T10:28:28.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:28.881+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.882747+0000) 2019-10-13T10:28:29.353 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:29.348+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.350538+0000) 2019-10-13T10:28:29.353 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:29.347+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.349198+0000) 2019-10-13T10:28:29.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:29.348+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.349641+0000) 2019-10-13T10:28:29.354 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:29.348+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.349767+0000) 2019-10-13T10:28:29.354 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:29.348+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.349798+0000) 2019-10-13T10:28:29.354 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:29.348+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.350090+0000) 2019-10-13T10:28:29.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:29.348+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.350313+0000) 2019-10-13T10:28:29.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:29.351+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:29.352968+0000) 2019-10-13T10:28:29.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:29.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:32.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:32.985+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:32.986393+0000) 2019-10-13T10:28:32.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:32.984+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:32.985459+0000) 2019-10-13T10:28:34.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:34.356+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.357541+0000) 2019-10-13T10:28:34.359 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:34.356+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.357770+0000) 2019-10-13T10:28:34.360 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:34.356+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.357811+0000) 2019-10-13T10:28:34.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:34.356+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.358067+0000) 2019-10-13T10:28:34.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:34.356+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.358166+0000) 2019-10-13T10:28:34.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:34.358+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.359721+0000) 2019-10-13T10:28:34.361 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:34.356+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.358299+0000) 2019-10-13T10:28:34.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:34.360+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.362576+0000) 2019-10-13T10:28:34.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:34.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:34.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:34.580+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.581384+0000) 2019-10-13T10:28:34.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:34.745+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.746424+0000) 2019-10-13T10:28:35.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:35.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:35.637112+0000) 2019-10-13T10:28:35.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:35.636+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:35.637112+0000) 2019-10-13T10:28:38.360 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:28:38.361 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:38.365 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:28:38.365 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:38.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:38.544+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.545077+0000) 2019-10-13T10:28:38.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:38.882+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.882935+0000) 2019-10-13T10:28:39.367 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:39.364+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.365801+0000) 2019-10-13T10:28:39.367 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:39.364+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.365887+0000) 2019-10-13T10:28:39.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:39.366+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.367601+0000) 2019-10-13T10:28:39.368 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:39.364+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.366290+0000) 2019-10-13T10:28:39.368 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:39.365+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.366488+0000) 2019-10-13T10:28:39.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:39.365+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.366778+0000) 2019-10-13T10:28:39.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:39.365+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.367020+0000) 2019-10-13T10:28:39.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:39.369+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:39.370431+0000) 2019-10-13T10:28:39.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:39.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188311 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:42.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:42.985+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:42.986610+0000) 2019-10-13T10:28:42.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:42.985+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:42.985664+0000) 2019-10-13T10:28:44.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:44.373+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.374640+0000) 2019-10-13T10:28:44.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:44.373+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.374792+0000) 2019-10-13T10:28:44.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:44.375+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.375994+0000) 2019-10-13T10:28:44.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:44.373+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.375160+0000) 2019-10-13T10:28:44.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:44.373+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.375324+0000) 2019-10-13T10:28:44.378 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:44.374+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.375621+0000) 2019-10-13T10:28:44.378 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:44.374+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.375754+0000) 2019-10-13T10:28:44.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:44.377+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.378227+0000) 2019-10-13T10:28:44.382 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:44.379+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188341 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:44.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:44.582+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.583526+0000) 2019-10-13T10:28:44.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:44.746+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.746573+0000) 2019-10-13T10:28:45.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:45.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:45.637274+0000) 2019-10-13T10:28:45.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:45.636+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:45.637274+0000) 2019-10-13T10:28:48.546 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:48.544+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.545298+0000) 2019-10-13T10:28:48.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:48.882+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.883068+0000) 2019-10-13T10:28:49.384 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:49.381+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.382482+0000) 2019-10-13T10:28:49.384 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:49.381+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.382653+0000) 2019-10-13T10:28:49.384 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:49.381+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.382811+0000) 2019-10-13T10:28:49.385 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:49.383+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.384149+0000) 2019-10-13T10:28:49.385 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:49.381+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.383178+0000) 2019-10-13T10:28:49.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:49.381+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.383232+0000) 2019-10-13T10:28:49.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:49.382+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.383493+0000) 2019-10-13T10:28:49.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:49.386+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:49.386821+0000) 2019-10-13T10:28:49.389 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:49.387+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188373 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:52.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:52.986+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:52.986807+0000) 2019-10-13T10:28:52.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:52.985+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:52.985862+0000) 2019-10-13T10:28:54.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:54.390+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.391045+0000) 2019-10-13T10:28:54.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:54.389+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.390307+0000) 2019-10-13T10:28:54.393 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:54.390+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.390647+0000) 2019-10-13T10:28:54.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:54.390+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.391004+0000) 2019-10-13T10:28:54.393 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:54.390+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.391144+0000) 2019-10-13T10:28:54.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:54.390+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.391299+0000) 2019-10-13T10:28:54.394 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:54.391+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.391426+0000) 2019-10-13T10:28:54.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:54.393+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.393938+0000) 2019-10-13T10:28:54.399 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:54.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:28:54.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:54.585+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.585606+0000) 2019-10-13T10:28:54.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:54.746+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.746746+0000) 2019-10-13T10:28:55.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:55.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:55.637494+0000) 2019-10-13T10:28:55.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:55.637+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:55.637479+0000) 2019-10-13T10:28:58.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:58.545+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.545470+0000) 2019-10-13T10:28:58.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:58.882+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.883235+0000) 2019-10-13T10:28:59.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:28:59.400+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.400860+0000) 2019-10-13T10:28:59.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:28:59.399+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.400329+0000) 2019-10-13T10:28:59.405 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:28:59.400+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.400588+0000) 2019-10-13T10:28:59.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:28:59.400+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.400760+0000) 2019-10-13T10:28:59.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:28:59.400+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.401086+0000) 2019-10-13T10:28:59.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:28:59.401+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.401425+0000) 2019-10-13T10:28:59.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:28:59.402+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.403128+0000) 2019-10-13T10:28:59.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:28:59.401+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:59.401574+0000) 2019-10-13T10:28:59.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:28:59.408+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188434 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:02.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:02.986+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:02.986955+0000) 2019-10-13T10:29:02.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:02.985+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:02.986069+0000) 2019-10-13T10:29:04.412 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:04.410+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.411165+0000) 2019-10-13T10:29:04.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:04.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.412283+0000) 2019-10-13T10:29:04.413 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:04.411+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.411493+0000) 2019-10-13T10:29:04.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:04.411+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.411627+0000) 2019-10-13T10:29:04.414 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:04.411+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.411739+0000) 2019-10-13T10:29:04.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:04.411+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.412287+0000) 2019-10-13T10:29:04.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:04.412+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.412706+0000) 2019-10-13T10:29:04.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:04.414+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.415603+0000) 2019-10-13T10:29:04.419 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:04.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:04.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:04.587+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.587646+0000) 2019-10-13T10:29:04.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:04.746+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.746926+0000) 2019-10-13T10:29:05.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:05.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:05.637678+0000) 2019-10-13T10:29:05.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:05.637+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:05.637670+0000) 2019-10-13T10:29:08.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:08.545+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.545670+0000) 2019-10-13T10:29:08.590 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:29:08.591 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:08.595 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:29:08.595 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:08.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:08.883+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.883436+0000) 2019-10-13T10:29:09.422 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:09.419+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.420571+0000) 2019-10-13T10:29:09.422 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:09.419+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.420631+0000) 2019-10-13T10:29:09.422 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:09.419+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.421002+0000) 2019-10-13T10:29:09.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:09.421+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.422017+0000) 2019-10-13T10:29:09.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:09.419+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.421358+0000) 2019-10-13T10:29:09.423 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:09.420+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.421507+0000) 2019-10-13T10:29:09.424 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:09.420+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.421900+0000) 2019-10-13T10:29:09.424 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:09.422+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:09.423544+0000) 2019-10-13T10:29:09.429 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:09.426+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:12.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:12.985+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:12.987105+0000) 2019-10-13T10:29:12.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:12.984+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:12.986274+0000) 2019-10-13T10:29:14.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:14.428+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.430158+0000) 2019-10-13T10:29:14.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:14.430+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.431859+0000) 2019-10-13T10:29:14.432 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:14.429+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.430435+0000) 2019-10-13T10:29:14.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:14.429+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.430548+0000) 2019-10-13T10:29:14.433 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:14.429+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.430749+0000) 2019-10-13T10:29:14.433 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:14.429+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.430989+0000) 2019-10-13T10:29:14.433 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:14.429+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.431069+0000) 2019-10-13T10:29:14.434 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:14.431+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.433387+0000) 2019-10-13T10:29:14.438 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:14.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:14.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:14.588+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.589959+0000) 2019-10-13T10:29:14.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:14.745+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.747117+0000) 2019-10-13T10:29:15.639 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:15.636+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:15.637805+0000) 2019-10-13T10:29:15.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:15.636+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:15.637840+0000) 2019-10-13T10:29:18.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:18.544+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.545849+0000) 2019-10-13T10:29:18.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:18.882+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.883635+0000) 2019-10-13T10:29:19.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:19.437+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.438849+0000) 2019-10-13T10:29:19.440 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:19.437+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.439067+0000) 2019-10-13T10:29:19.441 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:19.438+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.440319+0000) 2019-10-13T10:29:19.441 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:19.438+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.439584+0000) 2019-10-13T10:29:19.442 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:19.438+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.440003+0000) 2019-10-13T10:29:19.442 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:19.438+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.440325+0000) 2019-10-13T10:29:19.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:19.438+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.440360+0000) 2019-10-13T10:29:19.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:19.441+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:19.443055+0000) 2019-10-13T10:29:19.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:19.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188556 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:22.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:22.986+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:22.987271+0000) 2019-10-13T10:29:22.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:22.985+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:22.986431+0000) 2019-10-13T10:29:24.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:24.446+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.448320+0000) 2019-10-13T10:29:24.450 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:24.447+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.448531+0000) 2019-10-13T10:29:24.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:24.447+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.448741+0000) 2019-10-13T10:29:24.451 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:24.447+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.448923+0000) 2019-10-13T10:29:24.451 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:24.447+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.449056+0000) 2019-10-13T10:29:24.451 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:24.447+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.449079+0000) 2019-10-13T10:29:24.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:24.448+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.450486+0000) 2019-10-13T10:29:24.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:24.451+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.453311+0000) 2019-10-13T10:29:24.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:24.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:24.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:24.591+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.592219+0000) 2019-10-13T10:29:24.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:24.746+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.747290+0000) 2019-10-13T10:29:25.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:25.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:25.637991+0000) 2019-10-13T10:29:25.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:25.637+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:25.637987+0000) 2019-10-13T10:29:28.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:28.545+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.546012+0000) 2019-10-13T10:29:28.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:28.882+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.883811+0000) 2019-10-13T10:29:29.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:29.455+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.456596+0000) 2019-10-13T10:29:29.458 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:29.455+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.456819+0000) 2019-10-13T10:29:29.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:29.457+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.458651+0000) 2019-10-13T10:29:29.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:29.455+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.457158+0000) 2019-10-13T10:29:29.460 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:29.455+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.457047+0000) 2019-10-13T10:29:29.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:29.456+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.457575+0000) 2019-10-13T10:29:29.460 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:29.456+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.457886+0000) 2019-10-13T10:29:29.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:29.461+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:29.462104+0000) 2019-10-13T10:29:29.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:29.462+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188618 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:32.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:32.986+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:32.987507+0000) 2019-10-13T10:29:32.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:32.986+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:32.986643+0000) 2019-10-13T10:29:34.467 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:34.464+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.465900+0000) 2019-10-13T10:29:34.467 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:34.464+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.466264+0000) 2019-10-13T10:29:34.468 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:34.465+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.466419+0000) 2019-10-13T10:29:34.468 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:34.465+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.466552+0000) 2019-10-13T10:29:34.469 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:34.467+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.467970+0000) 2019-10-13T10:29:34.469 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:34.465+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.467189+0000) 2019-10-13T10:29:34.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:34.466+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.467491+0000) 2019-10-13T10:29:34.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:34.469+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.470436+0000) 2019-10-13T10:29:34.475 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:34.472+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:34.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:34.593+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.594141+0000) 2019-10-13T10:29:34.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:34.746+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.747484+0000) 2019-10-13T10:29:35.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:35.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:35.638204+0000) 2019-10-13T10:29:35.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:35.637+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:35.638206+0000) 2019-10-13T10:29:38.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:38.545+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.546164+0000) 2019-10-13T10:29:38.763 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:29:38.764 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:38.768 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:29:38.768 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:38.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:38.883+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.883995+0000) 2019-10-13T10:29:39.477 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:39.476+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.476991+0000) 2019-10-13T10:29:39.478 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:39.475+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.475694+0000) 2019-10-13T10:29:39.478 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:39.475+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.476212+0000) 2019-10-13T10:29:39.478 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:39.476+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.476478+0000) 2019-10-13T10:29:39.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:39.476+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.476645+0000) 2019-10-13T10:29:39.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:39.477+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.478701+0000) 2019-10-13T10:29:39.479 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:39.476+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.476985+0000) 2019-10-13T10:29:39.479 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:39.476+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:39.477209+0000) 2019-10-13T10:29:39.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:39.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:42.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:42.987+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:42.987705+0000) 2019-10-13T10:29:42.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:42.986+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:42.986899+0000) 2019-10-13T10:29:44.486 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:44.485+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.485492+0000) 2019-10-13T10:29:44.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:44.485+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.485802+0000) 2019-10-13T10:29:44.488 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:44.485+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.486608+0000) 2019-10-13T10:29:44.488 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:44.485+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.486033+0000) 2019-10-13T10:29:44.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:44.485+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.486198+0000) 2019-10-13T10:29:44.488 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:44.486+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.486433+0000) 2019-10-13T10:29:44.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:44.486+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.486538+0000) 2019-10-13T10:29:44.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:44.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.489688+0000) 2019-10-13T10:29:44.494 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:44.492+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:44.596 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:44.595+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.596308+0000) 2019-10-13T10:29:44.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:44.747+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.747664+0000) 2019-10-13T10:29:45.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:45.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:45.638332+0000) 2019-10-13T10:29:45.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:45.638+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:45.638382+0000) 2019-10-13T10:29:48.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:48.545+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.546373+0000) 2019-10-13T10:29:48.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:48.884+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.884177+0000) 2019-10-13T10:29:49.496 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:49.494+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.495174+0000) 2019-10-13T10:29:49.497 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:49.495+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.495618+0000) 2019-10-13T10:29:49.497 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:49.496+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.497041+0000) 2019-10-13T10:29:49.498 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:49.495+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.496109+0000) 2019-10-13T10:29:49.498 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:49.495+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.496190+0000) 2019-10-13T10:29:49.498 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:49.495+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.496332+0000) 2019-10-13T10:29:49.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:49.496+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.496466+0000) 2019-10-13T10:29:49.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:49.498+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:49.498845+0000) 2019-10-13T10:29:49.504 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:49.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188740 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:52.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:52.987+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:52.987936+0000) 2019-10-13T10:29:52.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:52.987+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:52.987109+0000) 2019-10-13T10:29:54.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:54.504+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.504659+0000) 2019-10-13T10:29:54.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:54.505+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.505881+0000) 2019-10-13T10:29:54.507 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:54.504+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.505359+0000) 2019-10-13T10:29:54.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:54.504+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.505214+0000) 2019-10-13T10:29:54.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:54.505+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.505591+0000) 2019-10-13T10:29:54.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:54.505+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.505759+0000) 2019-10-13T10:29:54.508 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:54.505+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.505780+0000) 2019-10-13T10:29:54.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:54.507+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.508126+0000) 2019-10-13T10:29:54.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:54.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:29:54.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:54.598+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.598558+0000) 2019-10-13T10:29:54.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:54.747+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.747809+0000) 2019-10-13T10:29:55.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:55.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:55.638542+0000) 2019-10-13T10:29:55.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:55.637+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:55.638543+0000) 2019-10-13T10:29:58.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:58.545+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.546559+0000) 2019-10-13T10:29:58.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:58.883+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.884369+0000) 2019-10-13T10:29:59.515 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:29:59.512+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.514134+0000) 2019-10-13T10:29:59.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:29:59.512+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.514277+0000) 2019-10-13T10:29:59.516 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:29:59.512+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.514384+0000) 2019-10-13T10:29:59.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:29:59.513+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.514540+0000) 2019-10-13T10:29:59.517 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:29:59.515+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.516012+0000) 2019-10-13T10:29:59.517 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:29:59.513+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.514820+0000) 2019-10-13T10:29:59.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:29:59.513+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.514995+0000) 2019-10-13T10:29:59.518 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:29:59.517+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:59.517856+0000) 2019-10-13T10:29:59.521 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:29:59.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:02.988 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:02.986+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:02.987291+0000) 2019-10-13T10:30:02.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:02.988+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:02.988171+0000) 2019-10-13T10:30:04.523 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:04.520+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.521870+0000) 2019-10-13T10:30:04.524 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:04.520+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.521982+0000) 2019-10-13T10:30:04.524 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:04.520+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.522277+0000) 2019-10-13T10:30:04.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:04.521+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.522526+0000) 2019-10-13T10:30:04.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:04.521+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.523691+0000) 2019-10-13T10:30:04.525 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:04.521+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.522921+0000) 2019-10-13T10:30:04.525 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:04.522+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.523818+0000) 2019-10-13T10:30:04.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:04.523+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.525385+0000) 2019-10-13T10:30:04.531 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:04.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:04.601 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:04.599+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.600845+0000) 2019-10-13T10:30:04.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:04.747+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.747972+0000) 2019-10-13T10:30:05.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:05.637+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:05.638675+0000) 2019-10-13T10:30:05.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:05.637+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:05.638710+0000) 2019-10-13T10:30:08.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:08.546+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.546729+0000) 2019-10-13T10:30:08.885 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:08.883+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.884548+0000) 2019-10-13T10:30:08.935 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:30:08.936 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:08.941 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:30:08.942 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:09.532 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:09.529+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.531030+0000) 2019-10-13T10:30:09.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:09.529+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.531207+0000) 2019-10-13T10:30:09.534 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:09.531+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.533173+0000) 2019-10-13T10:30:09.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:09.531+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.533277+0000) 2019-10-13T10:30:09.535 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:09.532+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.533410+0000) 2019-10-13T10:30:09.535 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:09.532+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.534306+0000) 2019-10-13T10:30:09.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:09.532+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.534701+0000) 2019-10-13T10:30:09.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:09.532+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:09.534023+0000) 2019-10-13T10:30:09.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:09.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:12.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:12.986+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:12.988344+0000) 2019-10-13T10:30:12.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:12.986+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:12.987475+0000) 2019-10-13T10:30:14.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:14.538+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.540425+0000) 2019-10-13T10:30:14.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:14.538+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.539678+0000) 2019-10-13T10:30:14.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:14.539+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.540467+0000) 2019-10-13T10:30:14.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:14.539+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.540521+0000) 2019-10-13T10:30:14.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:14.539+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.540693+0000) 2019-10-13T10:30:14.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:14.539+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.540863+0000) 2019-10-13T10:30:14.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:14.539+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.541054+0000) 2019-10-13T10:30:14.543 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:14.540+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.542428+0000) 2019-10-13T10:30:14.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:14.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188891 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:14.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:14.602+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.602921+0000) 2019-10-13T10:30:14.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:14.747+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.748140+0000) 2019-10-13T10:30:15.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:15.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:15.638861+0000) 2019-10-13T10:30:15.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:15.638+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:15.638902+0000) 2019-10-13T10:30:18.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:18.546+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.546893+0000) 2019-10-13T10:30:18.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:18.884+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.884679+0000) 2019-10-13T10:30:19.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:19.547+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.548893+0000) 2019-10-13T10:30:19.550 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:19.547+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.549094+0000) 2019-10-13T10:30:19.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:19.549+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.550198+0000) 2019-10-13T10:30:19.551 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:19.548+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.549473+0000) 2019-10-13T10:30:19.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:19.548+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.549600+0000) 2019-10-13T10:30:19.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:19.548+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.549849+0000) 2019-10-13T10:30:19.552 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:19.548+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.550268+0000) 2019-10-13T10:30:19.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:19.552+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:19.552942+0000) 2019-10-13T10:30:19.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:19.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188922 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:22.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:22.987+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:22.988512+0000) 2019-10-13T10:30:22.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:22.987+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:22.987683+0000) 2019-10-13T10:30:24.559 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:24.557+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.558180+0000) 2019-10-13T10:30:24.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:24.557+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.558333+0000) 2019-10-13T10:30:24.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:24.558+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.558480+0000) 2019-10-13T10:30:24.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:24.558+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.559543+0000) 2019-10-13T10:30:24.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:24.558+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.558931+0000) 2019-10-13T10:30:24.561 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:24.558+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.559088+0000) 2019-10-13T10:30:24.561 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:24.558+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.559184+0000) 2019-10-13T10:30:24.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:24.561+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.561787+0000) 2019-10-13T10:30:24.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:24.563+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:24.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:24.604+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.605021+0000) 2019-10-13T10:30:24.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:24.747+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.748281+0000) 2019-10-13T10:30:25.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:25.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:25.639047+0000) 2019-10-13T10:30:25.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:25.638+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:25.639106+0000) 2019-10-13T10:30:28.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:28.546+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.547035+0000) 2019-10-13T10:30:28.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:28.884+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.884897+0000) 2019-10-13T10:30:29.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:29.565+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.566228+0000) 2019-10-13T10:30:29.568 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:29.565+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.566371+0000) 2019-10-13T10:30:29.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:29.566+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.566485+0000) 2019-10-13T10:30:29.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:29.567+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.568118+0000) 2019-10-13T10:30:29.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:29.566+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.566925+0000) 2019-10-13T10:30:29.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:29.566+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.567246+0000) 2019-10-13T10:30:29.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:29.567+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.567444+0000) 2019-10-13T10:30:29.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:29.569+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:29.569904+0000) 2019-10-13T10:30:29.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:29.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 188984 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:32.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:32.988+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:32.988694+0000) 2019-10-13T10:30:32.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:32.987+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:32.987885+0000) 2019-10-13T10:30:34.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:34.575+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.575706+0000) 2019-10-13T10:30:34.577 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:34.575+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.576007+0000) 2019-10-13T10:30:34.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:34.576+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.577133+0000) 2019-10-13T10:30:34.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:34.575+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.576293+0000) 2019-10-13T10:30:34.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:34.575+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.576384+0000) 2019-10-13T10:30:34.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:34.576+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.576763+0000) 2019-10-13T10:30:34.579 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:34.576+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.576871+0000) 2019-10-13T10:30:34.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:34.578+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.578930+0000) 2019-10-13T10:30:34.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:34.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:34.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:34.606+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.607036+0000) 2019-10-13T10:30:34.749 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:34.748+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.748437+0000) 2019-10-13T10:30:35.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:35.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:35.639255+0000) 2019-10-13T10:30:35.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:35.639+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:35.639290+0000) 2019-10-13T10:30:38.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:38.547+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.547207+0000) 2019-10-13T10:30:38.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:38.884+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.885079+0000) 2019-10-13T10:30:39.101 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:30:39.101 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:39.106 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:30:39.106 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:39.586 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:39.585+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.586089+0000) 2019-10-13T10:30:39.587 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:39.584+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.585287+0000) 2019-10-13T10:30:39.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:39.585+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.585625+0000) 2019-10-13T10:30:39.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:39.585+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.585731+0000) 2019-10-13T10:30:39.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:39.585+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.586092+0000) 2019-10-13T10:30:39.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:39.586+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.586665+0000) 2019-10-13T10:30:39.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:39.586+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.586799+0000) 2019-10-13T10:30:39.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:39.588+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:39.589321+0000) 2019-10-13T10:30:39.593 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:39.592+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:42.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:42.987+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:42.988077+0000) 2019-10-13T10:30:42.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:42.988+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:42.988890+0000) 2019-10-13T10:30:44.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:44.592+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.594331+0000) 2019-10-13T10:30:44.596 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:44.593+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.594607+0000) 2019-10-13T10:30:44.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:44.593+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.594690+0000) 2019-10-13T10:30:44.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:44.595+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.595792+0000) 2019-10-13T10:30:44.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:44.593+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.594946+0000) 2019-10-13T10:30:44.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:44.594+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.595482+0000) 2019-10-13T10:30:44.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:44.593+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.595230+0000) 2019-10-13T10:30:44.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:44.598+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.598795+0000) 2019-10-13T10:30:44.603 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:44.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189074 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:44.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:44.608+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.609128+0000) 2019-10-13T10:30:44.750 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:44.747+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.748607+0000) 2019-10-13T10:30:45.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:45.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:45.639483+0000) 2019-10-13T10:30:45.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:45.638+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:45.639488+0000) 2019-10-13T10:30:48.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:48.546+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.547418+0000) 2019-10-13T10:30:48.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:48.883+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.885234+0000) 2019-10-13T10:30:49.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:49.602+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.604106+0000) 2019-10-13T10:30:49.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:49.603+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.605012+0000) 2019-10-13T10:30:49.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:49.603+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.605114+0000) 2019-10-13T10:30:49.607 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:49.604+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.605438+0000) 2019-10-13T10:30:49.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:49.604+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.605780+0000) 2019-10-13T10:30:49.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:49.604+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.605920+0000) 2019-10-13T10:30:49.608 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:49.606+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.607050+0000) 2019-10-13T10:30:49.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:49.608+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:49.609264+0000) 2019-10-13T10:30:49.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:49.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189106 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:52.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:52.988+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:52.989049+0000) 2019-10-13T10:30:52.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:52.987+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:52.988263+0000) 2019-10-13T10:30:54.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:54.611+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.611435+0000) 2019-10-13T10:30:54.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:54.612+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.614153+0000) 2019-10-13T10:30:54.615 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:54.612+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.614202+0000) 2019-10-13T10:30:54.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:54.613+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.614821+0000) 2019-10-13T10:30:54.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:54.615+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.616111+0000) 2019-10-13T10:30:54.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:54.614+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.615617+0000) 2019-10-13T10:30:54.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:54.614+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.615926+0000) 2019-10-13T10:30:54.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:54.614+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.616380+0000) 2019-10-13T10:30:54.619 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:54.618+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.619041+0000) 2019-10-13T10:30:54.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:54.622+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189136 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:30:54.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:54.748+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.748777+0000) 2019-10-13T10:30:55.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:55.638+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:55.639679+0000) 2019-10-13T10:30:55.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:55.638+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:55.639680+0000) 2019-10-13T10:30:58.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:58.546+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.547542+0000) 2019-10-13T10:30:58.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:58.884+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.885413+0000) 2019-10-13T10:30:59.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:30:59.623+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.625189+0000) 2019-10-13T10:30:59.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:30:59.623+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.625335+0000) 2019-10-13T10:30:59.627 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:30:59.624+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.626195+0000) 2019-10-13T10:30:59.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:30:59.624+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.625686+0000) 2019-10-13T10:30:59.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:30:59.624+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.625854+0000) 2019-10-13T10:30:59.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:30:59.624+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.625982+0000) 2019-10-13T10:30:59.628 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:30:59.624+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.626286+0000) 2019-10-13T10:30:59.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:30:59.626+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:59.627791+0000) 2019-10-13T10:30:59.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:30:59.630+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:02.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:02.988+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:02.989216+0000) 2019-10-13T10:31:02.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:02.987+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:02.988466+0000) 2019-10-13T10:31:04.614 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:04.612+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.613573+0000) 2019-10-13T10:31:04.635 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:04.632+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.633739+0000) 2019-10-13T10:31:04.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:04.632+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.633864+0000) 2019-10-13T10:31:04.635 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:04.632+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.634119+0000) 2019-10-13T10:31:04.636 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:04.634+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.635895+0000) 2019-10-13T10:31:04.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:04.633+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.634558+0000) 2019-10-13T10:31:04.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:04.633+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.634778+0000) 2019-10-13T10:31:04.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:04.633+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.635177+0000) 2019-10-13T10:31:04.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:04.636+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.638083+0000) 2019-10-13T10:31:04.642 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:04.639+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:04.750 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:04.748+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.748987+0000) 2019-10-13T10:31:05.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:05.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:05.639857+0000) 2019-10-13T10:31:05.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:05.639+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:05.639857+0000) 2019-10-13T10:31:08.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:08.547+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.547751+0000) 2019-10-13T10:31:08.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:08.885+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.885617+0000) 2019-10-13T10:31:09.256 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:31:09.256 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:09.260 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:31:09.261 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:09.644 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:09.642+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.643148+0000) 2019-10-13T10:31:09.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:09.644+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.644821+0000) 2019-10-13T10:31:09.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:09.643+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.643514+0000) 2019-10-13T10:31:09.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:09.642+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.643343+0000) 2019-10-13T10:31:09.646 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:09.643+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.643689+0000) 2019-10-13T10:31:09.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:09.643+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.644282+0000) 2019-10-13T10:31:09.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:09.644+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.645653+0000) 2019-10-13T10:31:09.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:09.643+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:09.644297+0000) 2019-10-13T10:31:09.652 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:09.650+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189228 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:12.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:12.988+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:12.989402+0000) 2019-10-13T10:31:12.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:12.988+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:12.988650+0000) 2019-10-13T10:31:14.616 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:14.615+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.615834+0000) 2019-10-13T10:31:14.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:14.652+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.653225+0000) 2019-10-13T10:31:14.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:14.652+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.653379+0000) 2019-10-13T10:31:14.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:14.653+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.653657+0000) 2019-10-13T10:31:14.656 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:14.654+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.655131+0000) 2019-10-13T10:31:14.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:14.653+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.654084+0000) 2019-10-13T10:31:14.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:14.654+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.654500+0000) 2019-10-13T10:31:14.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:14.653+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.653957+0000) 2019-10-13T10:31:14.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:14.655+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.656429+0000) 2019-10-13T10:31:14.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:14.660+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:14.750 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:14.748+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.749188+0000) 2019-10-13T10:31:15.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:15.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:15.640080+0000) 2019-10-13T10:31:15.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:15.639+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:15.640081+0000) 2019-10-13T10:31:18.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:18.547+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.547913+0000) 2019-10-13T10:31:18.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:18.885+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.885803+0000) 2019-10-13T10:31:19.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:19.662+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.662945+0000) 2019-10-13T10:31:19.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:19.662+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.663212+0000) 2019-10-13T10:31:19.665 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:19.663+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.664361+0000) 2019-10-13T10:31:19.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:19.663+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.663996+0000) 2019-10-13T10:31:19.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:19.663+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.664306+0000) 2019-10-13T10:31:19.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:19.664+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.664491+0000) 2019-10-13T10:31:19.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:19.665+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.666318+0000) 2019-10-13T10:31:19.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:19.664+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:19.664694+0000) 2019-10-13T10:31:19.673 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:19.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:22.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:22.989+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:22.989584+0000) 2019-10-13T10:31:22.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:22.988+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:22.988825+0000) 2019-10-13T10:31:24.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:24.617+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.618186+0000) 2019-10-13T10:31:24.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:24.672+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.672536+0000) 2019-10-13T10:31:24.676 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:24.672+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.672785+0000) 2019-10-13T10:31:24.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:24.672+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.673006+0000) 2019-10-13T10:31:24.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:24.673+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.673510+0000) 2019-10-13T10:31:24.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:24.673+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.673814+0000) 2019-10-13T10:31:24.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:24.673+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.674532+0000) 2019-10-13T10:31:24.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:24.675+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.676598+0000) 2019-10-13T10:31:24.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:24.673+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.674188+0000) 2019-10-13T10:31:24.681 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:24.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189318 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:24.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:24.749+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.749414+0000) 2019-10-13T10:31:25.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:25.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:25.640256+0000) 2019-10-13T10:31:25.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:25.640+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:25.640255+0000) 2019-10-13T10:31:28.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:28.546+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.548095+0000) 2019-10-13T10:31:28.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:28.885+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.885978+0000) 2019-10-13T10:31:29.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:29.682+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.683013+0000) 2019-10-13T10:31:29.684 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:29.680+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.682337+0000) 2019-10-13T10:31:29.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:29.681+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.682551+0000) 2019-10-13T10:31:29.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:29.681+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.682649+0000) 2019-10-13T10:31:29.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:29.681+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.682828+0000) 2019-10-13T10:31:29.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:29.681+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.683052+0000) 2019-10-13T10:31:29.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:29.681+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.683284+0000) 2019-10-13T10:31:29.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:29.684+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:29.685348+0000) 2019-10-13T10:31:29.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:29.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:32.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:32.989+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:32.989763+0000) 2019-10-13T10:31:32.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:32.988+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:32.989025+0000) 2019-10-13T10:31:34.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:34.619+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.620360+0000) 2019-10-13T10:31:34.693 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:34.692+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.693014+0000) 2019-10-13T10:31:34.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:34.690+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.692335+0000) 2019-10-13T10:31:34.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:34.691+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.692509+0000) 2019-10-13T10:31:34.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:34.691+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.692777+0000) 2019-10-13T10:31:34.695 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:34.691+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.693095+0000) 2019-10-13T10:31:34.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:34.691+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.693255+0000) 2019-10-13T10:31:34.695 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:34.691+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.693373+0000) 2019-10-13T10:31:34.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:34.695+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.696428+0000) 2019-10-13T10:31:34.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:34.698+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189380 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:34.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:34.748+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.749552+0000) 2019-10-13T10:31:35.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:35.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:35.640433+0000) 2019-10-13T10:31:35.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:35.639+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:35.640434+0000) 2019-10-13T10:31:38.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:38.546+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.548305+0000) 2019-10-13T10:31:38.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:38.885+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.886149+0000) 2019-10-13T10:31:39.427 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:31:39.428 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:39.432 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:31:39.432 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:39.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:39.700+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.701681+0000) 2019-10-13T10:31:39.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:39.702+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.702775+0000) 2019-10-13T10:31:39.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:39.701+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.702501+0000) 2019-10-13T10:31:39.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:39.701+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.702612+0000) 2019-10-13T10:31:39.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:39.701+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.702752+0000) 2019-10-13T10:31:39.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:39.701+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.702973+0000) 2019-10-13T10:31:39.705 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:39.701+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.703053+0000) 2019-10-13T10:31:39.706 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:39.704+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:39.705421+0000) 2019-10-13T10:31:39.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:39.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:42.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:42.989+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:42.989952+0000) 2019-10-13T10:31:42.990 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:42.988+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:42.989219+0000) 2019-10-13T10:31:44.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:44.622+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.622661+0000) 2019-10-13T10:31:44.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:44.710+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.711602+0000) 2019-10-13T10:31:44.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:44.712+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.712763+0000) 2019-10-13T10:31:44.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:44.710+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.711927+0000) 2019-10-13T10:31:44.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:44.710+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.712024+0000) 2019-10-13T10:31:44.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:44.710+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.712129+0000) 2019-10-13T10:31:44.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:44.710+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.712361+0000) 2019-10-13T10:31:44.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:44.711+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.712458+0000) 2019-10-13T10:31:44.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:44.714+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.715265+0000) 2019-10-13T10:31:44.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:44.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189440 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:44.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:44.749+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.749702+0000) 2019-10-13T10:31:45.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:45.639+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:45.640604+0000) 2019-10-13T10:31:45.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:45.639+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:45.640605+0000) 2019-10-13T10:31:48.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:48.547+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.548527+0000) 2019-10-13T10:31:48.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:48.885+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.886322+0000) 2019-10-13T10:31:49.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:49.718+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.719493+0000) 2019-10-13T10:31:49.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:49.718+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.720099+0000) 2019-10-13T10:31:49.722 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:49.719+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.721363+0000) 2019-10-13T10:31:49.722 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:49.719+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.720616+0000) 2019-10-13T10:31:49.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:49.719+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.720830+0000) 2019-10-13T10:31:49.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:49.719+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.721335+0000) 2019-10-13T10:31:49.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:49.719+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.720654+0000) 2019-10-13T10:31:49.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:49.722+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:49.723734+0000) 2019-10-13T10:31:49.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:49.726+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189474 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:52.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:52.989+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:52.990125+0000) 2019-10-13T10:31:52.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:52.988+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:52.989346+0000) 2019-10-13T10:31:54.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:54.624+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.624918+0000) 2019-10-13T10:31:54.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:54.729+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.729788+0000) 2019-10-13T10:31:54.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:54.729+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.730193+0000) 2019-10-13T10:31:54.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:54.730+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.730537+0000) 2019-10-13T10:31:54.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:54.730+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.730391+0000) 2019-10-13T10:31:54.732 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:54.730+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.731798+0000) 2019-10-13T10:31:54.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:54.731+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.731481+0000) 2019-10-13T10:31:54.733 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:54.731+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.731728+0000) 2019-10-13T10:31:54.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:54.732+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.733823+0000) 2019-10-13T10:31:54.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:54.737+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:31:54.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:54.749+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.749865+0000) 2019-10-13T10:31:55.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:55.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:55.640788+0000) 2019-10-13T10:31:55.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:55.640+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:55.640771+0000) 2019-10-13T10:31:58.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:58.548+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.548719+0000) 2019-10-13T10:31:58.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:58.886+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.886453+0000) 2019-10-13T10:31:59.740 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:31:59.738+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.740488+0000) 2019-10-13T10:31:59.741 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:31:59.739+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.739840+0000) 2019-10-13T10:31:59.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:31:59.739+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.740000+0000) 2019-10-13T10:31:59.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:31:59.739+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.740128+0000) 2019-10-13T10:31:59.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:31:59.739+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.740315+0000) 2019-10-13T10:31:59.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:31:59.740+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.740925+0000) 2019-10-13T10:31:59.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:31:59.740+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.741151+0000) 2019-10-13T10:31:59.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:31:59.742+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:59.743847+0000) 2019-10-13T10:31:59.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:31:59.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189534 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:02.990 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:02.989+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:02.990345+0000) 2019-10-13T10:32:02.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:02.989+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:02.989530+0000) 2019-10-13T10:32:04.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:04.626+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.627241+0000) 2019-10-13T10:32:04.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:04.748+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.749280+0000) 2019-10-13T10:32:04.751 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:04.749+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.749643+0000) 2019-10-13T10:32:04.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:04.749+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.749875+0000) 2019-10-13T10:32:04.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:04.749+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.750039+0000) 2019-10-13T10:32:04.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:04.750+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.751380+0000) 2019-10-13T10:32:04.752 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:04.750+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.751126+0000) 2019-10-13T10:32:04.752 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:04.750+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.751206+0000) 2019-10-13T10:32:04.753 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:04.753+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.753743+0000) 2019-10-13T10:32:04.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:04.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:05.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:05.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:05.640970+0000) 2019-10-13T10:32:05.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:05.640+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:05.640967+0000) 2019-10-13T10:32:08.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:08.548+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.548927+0000) 2019-10-13T10:32:08.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:08.886+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.886651+0000) 2019-10-13T10:32:09.573 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:32:09.573 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:09.577 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:32:09.577 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:09.761 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:09.759+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.759782+0000) 2019-10-13T10:32:09.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:09.759+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.760358+0000) 2019-10-13T10:32:09.761 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:09.760+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.761393+0000) 2019-10-13T10:32:09.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:09.760+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.760786+0000) 2019-10-13T10:32:09.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:09.760+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.760691+0000) 2019-10-13T10:32:09.762 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:09.760+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.760884+0000) 2019-10-13T10:32:09.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:09.760+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.761017+0000) 2019-10-13T10:32:09.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:09.762+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:09.763144+0000) 2019-10-13T10:32:09.769 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:09.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189596 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:11.501 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:11.499+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:11.500221+0000) 2019-10-13T10:32:12.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:12.990+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:12.990539+0000) 2019-10-13T10:32:12.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:12.988+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:12.989706+0000) 2019-10-13T10:32:14.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:14.629+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.629587+0000) 2019-10-13T10:32:14.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:14.768+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.769515+0000) 2019-10-13T10:32:14.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:14.768+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.769681+0000) 2019-10-13T10:32:14.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:14.769+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.770462+0000) 2019-10-13T10:32:14.772 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:14.769+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.770529+0000) 2019-10-13T10:32:14.772 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:14.769+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.770834+0000) 2019-10-13T10:32:14.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:14.769+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.770919+0000) 2019-10-13T10:32:14.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:14.771+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.772078+0000) 2019-10-13T10:32:14.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:14.773+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.773781+0000) 2019-10-13T10:32:14.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:14.775+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189624 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:15.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:15.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:15.641163+0000) 2019-10-13T10:32:15.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:15.640+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:15.641136+0000) 2019-10-13T10:32:18.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:18.548+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.549122+0000) 2019-10-13T10:32:18.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:18.885+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.886814+0000) 2019-10-13T10:32:19.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:19.777+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.779151+0000) 2019-10-13T10:32:19.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:19.777+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.779358+0000) 2019-10-13T10:32:19.781 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:19.778+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.779936+0000) 2019-10-13T10:32:19.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:19.778+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.780298+0000) 2019-10-13T10:32:19.782 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:19.779+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.780557+0000) 2019-10-13T10:32:19.783 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:19.780+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.781673+0000) 2019-10-13T10:32:19.783 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:19.779+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.780841+0000) 2019-10-13T10:32:19.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:19.783+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:19.784098+0000) 2019-10-13T10:32:19.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:19.786+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189656 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:21.501 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:21.499+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:21.500425+0000) 2019-10-13T10:32:22.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:22.990+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:22.990722+0000) 2019-10-13T10:32:22.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:22.989+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:22.989957+0000) 2019-10-13T10:32:24.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:24.631+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.631965+0000) 2019-10-13T10:32:24.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:24.787+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.789092+0000) 2019-10-13T10:32:24.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:24.787+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.789216+0000) 2019-10-13T10:32:24.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:24.788+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.789417+0000) 2019-10-13T10:32:24.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:24.788+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.789708+0000) 2019-10-13T10:32:24.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:24.789+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.790553+0000) 2019-10-13T10:32:24.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:24.790+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.791681+0000) 2019-10-13T10:32:24.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:24.788+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.789897+0000) 2019-10-13T10:32:24.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:24.788+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.789997+0000) 2019-10-13T10:32:24.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:24.794+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:25.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:25.640+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:25.641292+0000) 2019-10-13T10:32:25.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:25.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:25.641298+0000) 2019-10-13T10:32:28.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:28.548+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.549328+0000) 2019-10-13T10:32:28.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:28.886+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.886991+0000) 2019-10-13T10:32:29.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:29.796+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.797470+0000) 2019-10-13T10:32:29.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:29.797+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.798691+0000) 2019-10-13T10:32:29.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:29.796+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.798284+0000) 2019-10-13T10:32:29.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:29.797+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.798523+0000) 2019-10-13T10:32:29.800 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:29.797+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.798727+0000) 2019-10-13T10:32:29.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:29.797+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.798801+0000) 2019-10-13T10:32:29.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:29.797+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.798929+0000) 2019-10-13T10:32:29.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:29.802+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:29.802878+0000) 2019-10-13T10:32:29.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:29.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:31.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:31.499+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:31.500637+0000) 2019-10-13T10:32:32.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:32.990+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:32.990920+0000) 2019-10-13T10:32:32.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:32.989+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:32.990139+0000) 2019-10-13T10:32:34.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:34.634+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.634166+0000) 2019-10-13T10:32:34.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:34.805+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.807227+0000) 2019-10-13T10:32:34.809 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:34.806+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.807509+0000) 2019-10-13T10:32:34.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:34.808+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.809064+0000) 2019-10-13T10:32:34.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:34.807+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.808525+0000) 2019-10-13T10:32:34.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:34.807+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.808583+0000) 2019-10-13T10:32:34.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:34.807+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.809041+0000) 2019-10-13T10:32:34.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:34.808+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.809436+0000) 2019-10-13T10:32:34.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:34.811+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.811899+0000) 2019-10-13T10:32:34.817 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:34.814+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:35.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:35.640+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:35.641462+0000) 2019-10-13T10:32:35.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:35.640+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:35.641465+0000) 2019-10-13T10:32:38.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:38.549+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.549498+0000) 2019-10-13T10:32:38.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:38.886+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.887183+0000) 2019-10-13T10:32:39.719 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:32:39.719 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:39.723 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:32:39.724 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:39.819 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:39.816+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.818063+0000) 2019-10-13T10:32:39.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:39.816+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.818270+0000) 2019-10-13T10:32:39.820 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:39.817+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.818417+0000) 2019-10-13T10:32:39.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:39.817+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.818608+0000) 2019-10-13T10:32:39.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:39.818+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.819898+0000) 2019-10-13T10:32:39.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:39.817+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.818652+0000) 2019-10-13T10:32:39.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:39.817+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.819216+0000) 2019-10-13T10:32:39.822 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:39.820+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:39.821852+0000) 2019-10-13T10:32:39.826 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:39.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189778 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:41.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:41.500+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:41.500873+0000) 2019-10-13T10:32:42.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:42.990+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:42.991087+0000) 2019-10-13T10:32:42.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:42.989+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:42.990325+0000) 2019-10-13T10:32:44.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:44.635+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.636607+0000) 2019-10-13T10:32:44.828 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:44.826+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.826559+0000) 2019-10-13T10:32:44.829 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:44.826+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.828005+0000) 2019-10-13T10:32:44.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:44.826+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.827248+0000) 2019-10-13T10:32:44.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:44.827+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.827407+0000) 2019-10-13T10:32:44.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:44.827+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.827803+0000) 2019-10-13T10:32:44.830 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:44.827+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.829555+0000) 2019-10-13T10:32:44.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:44.827+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.828164+0000) 2019-10-13T10:32:44.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:44.827+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.827927+0000) 2019-10-13T10:32:44.836 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:44.834+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:45.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:45.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:45.641645+0000) 2019-10-13T10:32:45.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:45.641+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:45.641646+0000) 2019-10-13T10:32:48.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:48.549+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.549680+0000) 2019-10-13T10:32:48.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:48.886+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.887328+0000) 2019-10-13T10:32:49.838 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:49.835+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.836179+0000) 2019-10-13T10:32:49.838 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:49.836+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.836612+0000) 2019-10-13T10:32:49.838 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:49.836+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.837033+0000) 2019-10-13T10:32:49.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:49.836+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.837167+0000) 2019-10-13T10:32:49.839 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:49.836+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.837825+0000) 2019-10-13T10:32:49.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:49.836+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.837323+0000) 2019-10-13T10:32:49.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:49.837+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.837507+0000) 2019-10-13T10:32:49.840 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:49.839+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:49.840023+0000) 2019-10-13T10:32:49.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:49.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189840 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:51.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:51.500+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:51.501064+0000) 2019-10-13T10:32:52.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:52.990+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:52.991276+0000) 2019-10-13T10:32:52.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:52.990+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:52.990533+0000) 2019-10-13T10:32:54.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:54.638+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.639057+0000) 2019-10-13T10:32:54.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:54.843+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.844388+0000) 2019-10-13T10:32:54.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:54.843+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.844282+0000) 2019-10-13T10:32:54.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:54.844+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.844755+0000) 2019-10-13T10:32:54.847 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:54.844+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.845604+0000) 2019-10-13T10:32:54.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:54.844+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.845031+0000) 2019-10-13T10:32:54.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:54.844+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.845297+0000) 2019-10-13T10:32:54.847 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:54.844+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.845347+0000) 2019-10-13T10:32:54.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:54.846+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.847648+0000) 2019-10-13T10:32:54.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:54.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189868 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:32:55.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:55.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:55.641790+0000) 2019-10-13T10:32:55.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:55.641+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:55.641790+0000) 2019-10-13T10:32:58.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:58.548+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.549847+0000) 2019-10-13T10:32:58.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:58.886+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.887494+0000) 2019-10-13T10:32:59.854 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:32:59.851+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.853360+0000) 2019-10-13T10:32:59.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:32:59.854+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.854856+0000) 2019-10-13T10:32:59.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:32:59.852+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.853802+0000) 2019-10-13T10:32:59.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:32:59.852+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.854049+0000) 2019-10-13T10:32:59.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:32:59.852+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.854315+0000) 2019-10-13T10:32:59.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:32:59.853+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.854559+0000) 2019-10-13T10:32:59.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:32:59.853+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.854795+0000) 2019-10-13T10:32:59.857 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:32:59.856+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:59.857311+0000) 2019-10-13T10:32:59.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:32:59.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189900 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:01.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:01.500+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:01.501267+0000) 2019-10-13T10:33:02.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:02.990+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:02.991430+0000) 2019-10-13T10:33:02.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:02.989+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:02.990732+0000) 2019-10-13T10:33:04.641 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:04.640+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.641307+0000) 2019-10-13T10:33:04.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:04.862+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.863675+0000) 2019-10-13T10:33:04.865 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:04.861+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.863182+0000) 2019-10-13T10:33:04.865 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:04.861+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.863385+0000) 2019-10-13T10:33:04.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:04.862+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.863458+0000) 2019-10-13T10:33:04.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:04.862+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.863755+0000) 2019-10-13T10:33:04.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:04.862+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.863959+0000) 2019-10-13T10:33:04.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:04.862+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.863932+0000) 2019-10-13T10:33:04.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:04.864+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.865555+0000) 2019-10-13T10:33:04.870 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:04.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189929 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:05.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:05.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:05.641962+0000) 2019-10-13T10:33:05.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:05.641+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:05.641963+0000) 2019-10-13T10:33:08.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:08.549+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.550035+0000) 2019-10-13T10:33:08.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:08.886+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.887684+0000) 2019-10-13T10:33:09.862 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:33:09.862 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:09.866 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:33:09.867 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:09.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:09.871+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.872155+0000) 2019-10-13T10:33:09.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:09.870+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.871393+0000) 2019-10-13T10:33:09.873 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:09.870+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.871846+0000) 2019-10-13T10:33:09.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:09.870+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.871711+0000) 2019-10-13T10:33:09.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:09.870+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.872096+0000) 2019-10-13T10:33:09.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:09.871+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.872588+0000) 2019-10-13T10:33:09.874 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:09.871+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.872712+0000) 2019-10-13T10:33:09.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:09.874+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:09.875710+0000) 2019-10-13T10:33:09.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:09.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:11.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:11.500+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:11.501441+0000) 2019-10-13T10:33:12.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:12.991+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:12.991611+0000) 2019-10-13T10:33:12.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:12.990+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:12.990925+0000) 2019-10-13T10:33:14.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:14.643+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.643643+0000) 2019-10-13T10:33:14.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:14.881+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.882192+0000) 2019-10-13T10:33:14.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:14.880+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.881509+0000) 2019-10-13T10:33:14.883 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:14.880+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.881805+0000) 2019-10-13T10:33:14.884 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:14.880+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.881939+0000) 2019-10-13T10:33:14.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:14.880+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.882133+0000) 2019-10-13T10:33:14.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:14.880+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.882258+0000) 2019-10-13T10:33:14.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:14.881+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.882630+0000) 2019-10-13T10:33:14.886 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:14.886+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.886789+0000) 2019-10-13T10:33:14.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:14.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 189992 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:15.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:15.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:15.642130+0000) 2019-10-13T10:33:15.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:15.641+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:15.642138+0000) 2019-10-13T10:33:18.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:18.549+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.550207+0000) 2019-10-13T10:33:18.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:18.887+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.887877+0000) 2019-10-13T10:33:19.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:19.891+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.891765+0000) 2019-10-13T10:33:19.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:19.889+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.891019+0000) 2019-10-13T10:33:19.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:19.889+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.891207+0000) 2019-10-13T10:33:19.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:19.890+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.891421+0000) 2019-10-13T10:33:19.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:19.890+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.891450+0000) 2019-10-13T10:33:19.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:19.890+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.891634+0000) 2019-10-13T10:33:19.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:19.890+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.891993+0000) 2019-10-13T10:33:19.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:19.895+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:19.895817+0000) 2019-10-13T10:33:19.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:19.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:21.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:21.501+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:21.501648+0000) 2019-10-13T10:33:22.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:22.991+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:22.991796+0000) 2019-10-13T10:33:22.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:22.990+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:22.991087+0000) 2019-10-13T10:33:24.646 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:24.645+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.646060+0000) 2019-10-13T10:33:24.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:24.900+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.900939+0000) 2019-10-13T10:33:24.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:24.898+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.900151+0000) 2019-10-13T10:33:24.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:24.898+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.900264+0000) 2019-10-13T10:33:24.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:24.899+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.900535+0000) 2019-10-13T10:33:24.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:24.899+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.900909+0000) 2019-10-13T10:33:24.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:24.899+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.901059+0000) 2019-10-13T10:33:24.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:24.899+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.901336+0000) 2019-10-13T10:33:24.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:24.903+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.903808+0000) 2019-10-13T10:33:24.909 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:24.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:25.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:25.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:25.642304+0000) 2019-10-13T10:33:25.644 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:25.641+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:25.642305+0000) 2019-10-13T10:33:28.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:28.550+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.550436+0000) 2019-10-13T10:33:28.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:28.887+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.888011+0000) 2019-10-13T10:33:29.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:29.909+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.910825+0000) 2019-10-13T10:33:29.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:29.909+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.910183+0000) 2019-10-13T10:33:29.912 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:29.909+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.910373+0000) 2019-10-13T10:33:29.912 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:29.910+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.910749+0000) 2019-10-13T10:33:29.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:29.910+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.910928+0000) 2019-10-13T10:33:29.913 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:29.910+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.911068+0000) 2019-10-13T10:33:29.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:29.911+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.913137+0000) 2019-10-13T10:33:29.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:29.910+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:29.911360+0000) 2019-10-13T10:33:29.919 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:29.917+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:31.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:31.501+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:31.501852+0000) 2019-10-13T10:33:32.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:32.991+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:32.991975+0000) 2019-10-13T10:33:32.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:32.990+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:32.991256+0000) 2019-10-13T10:33:34.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:34.647+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.648371+0000) 2019-10-13T10:33:34.921 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:34.919+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.919709+0000) 2019-10-13T10:33:34.921 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:34.919+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.919853+0000) 2019-10-13T10:33:34.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:34.919+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.920127+0000) 2019-10-13T10:33:34.922 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:34.919+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.921193+0000) 2019-10-13T10:33:34.922 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:34.920+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.920566+0000) 2019-10-13T10:33:34.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:34.920+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.921014+0000) 2019-10-13T10:33:34.923 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:34.920+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.921082+0000) 2019-10-13T10:33:34.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:34.920+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.922694+0000) 2019-10-13T10:33:34.928 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:34.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:35.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:35.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:35.642496+0000) 2019-10-13T10:33:35.644 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:35.642+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:35.642496+0000) 2019-10-13T10:33:38.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:38.550+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.550593+0000) 2019-10-13T10:33:38.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:38.888+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.888163+0000) 2019-10-13T10:33:39.930 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:39.928+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.929058+0000) 2019-10-13T10:33:39.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:39.929+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.929403+0000) 2019-10-13T10:33:39.931 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:39.929+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.929571+0000) 2019-10-13T10:33:39.932 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:39.929+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.931463+0000) 2019-10-13T10:33:39.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:39.929+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.929906+0000) 2019-10-13T10:33:39.932 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:39.929+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.930030+0000) 2019-10-13T10:33:39.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:39.929+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.930311+0000) 2019-10-13T10:33:39.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:39.930+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:39.932260+0000) 2019-10-13T10:33:39.938 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:39.936+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190142 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:40.037 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:33:40.037 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:40.041 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:33:40.041 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:41.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:41.501+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:41.502065+0000) 2019-10-13T10:33:42.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:42.991+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:42.992127+0000) 2019-10-13T10:33:42.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:42.991+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:42.991381+0000) 2019-10-13T10:33:44.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:44.649+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.650642+0000) 2019-10-13T10:33:44.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:44.936+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.938326+0000) 2019-10-13T10:33:44.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:44.937+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.938889+0000) 2019-10-13T10:33:44.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:44.937+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.939027+0000) 2019-10-13T10:33:44.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:44.937+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.939216+0000) 2019-10-13T10:33:44.941 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:44.937+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.939308+0000) 2019-10-13T10:33:44.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:44.938+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.939662+0000) 2019-10-13T10:33:44.942 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:44.939+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.940459+0000) 2019-10-13T10:33:44.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:44.941+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.941765+0000) 2019-10-13T10:33:44.946 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:44.943+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190171 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:45.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:45.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:45.642659+0000) 2019-10-13T10:33:45.644 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:45.641+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:45.642661+0000) 2019-10-13T10:33:48.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:48.549+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.550771+0000) 2019-10-13T10:33:48.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:48.887+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.888290+0000) 2019-10-13T10:33:49.948 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:49.947+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.947833+0000) 2019-10-13T10:33:49.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:49.945+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.947014+0000) 2019-10-13T10:33:49.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:49.945+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.947162+0000) 2019-10-13T10:33:49.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:49.946+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.947625+0000) 2019-10-13T10:33:49.950 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:49.946+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.947679+0000) 2019-10-13T10:33:49.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:49.946+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.948192+0000) 2019-10-13T10:33:49.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:49.947+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.948459+0000) 2019-10-13T10:33:49.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:49.950+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:49.950925+0000) 2019-10-13T10:33:49.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:49.953+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:51.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:51.501+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:51.502244+0000) 2019-10-13T10:33:52.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:52.991+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:52.992280+0000) 2019-10-13T10:33:52.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:52.990+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:52.991543+0000) 2019-10-13T10:33:54.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:54.652+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.653119+0000) 2019-10-13T10:33:54.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:54.954+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.956323+0000) 2019-10-13T10:33:54.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:54.955+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.956482+0000) 2019-10-13T10:33:54.959 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:54.957+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.957946+0000) 2019-10-13T10:33:54.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:54.955+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.956885+0000) 2019-10-13T10:33:54.959 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:54.955+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.957327+0000) 2019-10-13T10:33:54.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:54.955+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.957365+0000) 2019-10-13T10:33:54.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:54.956+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.957726+0000) 2019-10-13T10:33:54.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:54.958+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.959411+0000) 2019-10-13T10:33:54.965 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:54.962+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190232 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:33:55.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:55.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:55.642839+0000) 2019-10-13T10:33:55.644 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:55.642+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:55.642844+0000) 2019-10-13T10:33:58.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:58.550+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.550963+0000) 2019-10-13T10:33:58.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:58.887+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.888463+0000) 2019-10-13T10:33:59.967 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:33:59.964+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.965705+0000) 2019-10-13T10:33:59.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:33:59.964+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.966097+0000) 2019-10-13T10:33:59.968 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:33:59.964+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.966291+0000) 2019-10-13T10:33:59.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:33:59.965+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.966527+0000) 2019-10-13T10:33:59.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:33:59.965+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.966600+0000) 2019-10-13T10:33:59.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:33:59.966+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.967538+0000) 2019-10-13T10:33:59.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:33:59.967+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.968315+0000) 2019-10-13T10:33:59.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:33:59.968+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:59.969053+0000) 2019-10-13T10:33:59.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:33:59.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:01.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:01.501+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:01.502412+0000) 2019-10-13T10:34:02.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:02.992+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:02.992452+0000) 2019-10-13T10:34:02.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:02.991+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:02.991725+0000) 2019-10-13T10:34:04.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:04.655+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.655349+0000) 2019-10-13T10:34:04.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:04.974+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.975702+0000) 2019-10-13T10:34:04.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:04.974+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.975823+0000) 2019-10-13T10:34:04.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:04.976+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.977458+0000) 2019-10-13T10:34:04.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:04.974+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.976307+0000) 2019-10-13T10:34:04.978 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:04.975+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.976783+0000) 2019-10-13T10:34:04.979 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:04.975+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.977041+0000) 2019-10-13T10:34:04.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:04.976+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.977465+0000) 2019-10-13T10:34:04.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:04.978+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.979127+0000) 2019-10-13T10:34:04.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:04.982+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:05.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:05.641+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:05.643031+0000) 2019-10-13T10:34:05.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:05.641+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:05.643031+0000) 2019-10-13T10:34:08.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:08.550+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.551139+0000) 2019-10-13T10:34:08.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:08.888+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.888619+0000) 2019-10-13T10:34:09.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:09.984+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.986043+0000) 2019-10-13T10:34:09.988 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:09.984+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.986342+0000) 2019-10-13T10:34:09.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:09.984+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.986372+0000) 2019-10-13T10:34:09.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:09.985+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.986672+0000) 2019-10-13T10:34:09.988 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:09.985+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.986846+0000) 2019-10-13T10:34:09.989 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:09.987+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.988332+0000) 2019-10-13T10:34:09.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:09.986+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.987565+0000) 2019-10-13T10:34:09.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:09.989+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:09.990381+0000) 2019-10-13T10:34:09.995 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:09.992+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190323 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:10.198 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:34:10.198 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:10.202 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:34:10.202 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:11.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:11.502+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:11.502593+0000) 2019-10-13T10:34:12.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:12.992+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:12.992619+0000) 2019-10-13T10:34:12.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:12.991+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:12.991918+0000) 2019-10-13T10:34:14.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:14.657+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.657399+0000) 2019-10-13T10:34:14.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:14.995+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.995661+0000) 2019-10-13T10:34:14.997 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:14.995+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.995901+0000) 2019-10-13T10:34:14.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:14.995+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.995983+0000) 2019-10-13T10:34:14.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:14.996+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.997205+0000) 2019-10-13T10:34:14.998 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:14.995+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.996365+0000) 2019-10-13T10:34:14.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:14.996+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.996751+0000) 2019-10-13T10:34:14.999 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:14.996+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.996709+0000) 2019-10-13T10:34:14.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:14.997+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.998647+0000) 2019-10-13T10:34:15.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:15.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190353 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:15.644 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:15.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:15.643198+0000) 2019-10-13T10:34:15.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:15.642+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:15.643217+0000) 2019-10-13T10:34:18.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:18.551+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.551353+0000) 2019-10-13T10:34:18.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:18.888+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.888748+0000) 2019-10-13T10:34:20.006 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:20.004+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.004859+0000) 2019-10-13T10:34:20.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:20.004+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.005139+0000) 2019-10-13T10:34:20.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:20.005+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.006916+0000) 2019-10-13T10:34:20.007 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:20.005+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.005500+0000) 2019-10-13T10:34:20.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:20.005+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.005700+0000) 2019-10-13T10:34:20.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:20.005+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.005841+0000) 2019-10-13T10:34:20.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:20.005+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.006349+0000) 2019-10-13T10:34:20.008 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:20.006+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.008369+0000) 2019-10-13T10:34:20.014 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:20.012+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190383 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:21.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:21.502+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:21.502774+0000) 2019-10-13T10:34:22.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:22.991+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:22.992831+0000) 2019-10-13T10:34:22.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:22.991+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:22.992116+0000) 2019-10-13T10:34:24.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:24.657+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:24.659709+0000) 2019-10-13T10:34:25.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:25.014+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.014667+0000) 2019-10-13T10:34:25.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:25.014+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.014886+0000) 2019-10-13T10:34:25.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:25.014+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.015260+0000) 2019-10-13T10:34:25.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:25.015+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.015825+0000) 2019-10-13T10:34:25.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:25.015+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.015851+0000) 2019-10-13T10:34:25.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:25.015+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.016042+0000) 2019-10-13T10:34:25.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:25.014+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.016324+0000) 2019-10-13T10:34:25.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:25.016+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.018286+0000) 2019-10-13T10:34:25.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:25.022+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190413 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:25.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:25.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.643408+0000) 2019-10-13T10:34:25.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:25.643+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:25.643414+0000) 2019-10-13T10:34:28.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:28.551+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.551549+0000) 2019-10-13T10:34:28.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:28.888+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.888939+0000) 2019-10-13T10:34:30.026 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:30.024+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.025141+0000) 2019-10-13T10:34:30.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:30.024+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.025279+0000) 2019-10-13T10:34:30.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:30.024+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.026390+0000) 2019-10-13T10:34:30.028 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:30.025+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.025738+0000) 2019-10-13T10:34:30.028 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:30.025+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.025657+0000) 2019-10-13T10:34:30.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:30.025+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.026024+0000) 2019-10-13T10:34:30.028 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:30.025+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.026072+0000) 2019-10-13T10:34:30.029 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:30.026+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.027908+0000) 2019-10-13T10:34:30.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:30.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190445 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:31.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:31.501+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:31.502958+0000) 2019-10-13T10:34:32.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:32.992+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:32.993016+0000) 2019-10-13T10:34:32.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:32.990+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:32.992229+0000) 2019-10-13T10:34:34.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:34.661+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:34.662105+0000) 2019-10-13T10:34:35.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:35.035+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.036105+0000) 2019-10-13T10:34:35.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:35.033+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.035050+0000) 2019-10-13T10:34:35.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:35.033+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.035251+0000) 2019-10-13T10:34:35.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:35.034+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.035472+0000) 2019-10-13T10:34:35.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:35.034+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.035791+0000) 2019-10-13T10:34:35.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:35.034+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.035810+0000) 2019-10-13T10:34:35.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:35.034+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.036068+0000) 2019-10-13T10:34:35.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:35.037+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.037737+0000) 2019-10-13T10:34:35.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:35.041+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190475 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:35.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:35.642+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.643612+0000) 2019-10-13T10:34:35.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:35.642+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:35.643613+0000) 2019-10-13T10:34:38.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:38.550+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.551746+0000) 2019-10-13T10:34:38.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:38.888+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.889125+0000) 2019-10-13T10:34:40.045 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:40.042+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.044050+0000) 2019-10-13T10:34:40.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:40.042+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.044126+0000) 2019-10-13T10:34:40.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:40.043+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.044461+0000) 2019-10-13T10:34:40.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:40.043+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.044427+0000) 2019-10-13T10:34:40.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:40.043+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.044725+0000) 2019-10-13T10:34:40.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:40.043+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.044833+0000) 2019-10-13T10:34:40.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:40.045+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.045897+0000) 2019-10-13T10:34:40.048 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:40.047+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.048281+0000) 2019-10-13T10:34:40.052 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:40.049+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190505 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:40.362 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:34:40.362 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:40.367 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:34:40.367 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:41.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:41.502+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:41.503132+0000) 2019-10-13T10:34:42.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:42.991+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:42.992425+0000) 2019-10-13T10:34:42.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:42.992+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:42.993199+0000) 2019-10-13T10:34:44.665 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:44.663+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.664384+0000) 2019-10-13T10:34:45.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:45.050+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.052179+0000) 2019-10-13T10:34:45.054 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:45.050+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.052374+0000) 2019-10-13T10:34:45.054 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:45.051+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.052581+0000) 2019-10-13T10:34:45.054 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:45.051+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.052645+0000) 2019-10-13T10:34:45.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:45.053+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.054139+0000) 2019-10-13T10:34:45.055 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:45.051+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.053003+0000) 2019-10-13T10:34:45.055 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:45.051+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.053132+0000) 2019-10-13T10:34:45.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:45.054+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.055623+0000) 2019-10-13T10:34:45.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:45.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190535 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:45.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:45.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.643794+0000) 2019-10-13T10:34:45.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:45.643+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:45.643793+0000) 2019-10-13T10:34:48.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:48.551+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.551931+0000) 2019-10-13T10:34:48.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:48.888+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.889281+0000) 2019-10-13T10:34:50.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:50.059+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.060590+0000) 2019-10-13T10:34:50.062 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:50.059+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.060902+0000) 2019-10-13T10:34:50.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:50.061+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.061987+0000) 2019-10-13T10:34:50.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:50.059+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.061278+0000) 2019-10-13T10:34:50.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:50.060+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.061466+0000) 2019-10-13T10:34:50.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:50.060+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.061588+0000) 2019-10-13T10:34:50.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:50.060+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.061838+0000) 2019-10-13T10:34:50.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:50.063+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.063868+0000) 2019-10-13T10:34:50.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:50.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190567 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:51.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:51.502+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:51.503307+0000) 2019-10-13T10:34:52.993 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:52.993+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:52.993428+0000) 2019-10-13T10:34:52.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:52.992+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:52.992587+0000) 2019-10-13T10:34:54.667 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:54.666+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.666836+0000) 2019-10-13T10:34:55.071 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:55.068+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.070033+0000) 2019-10-13T10:34:55.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:55.068+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.070221+0000) 2019-10-13T10:34:55.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:34:55.070+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.071639+0000) 2019-10-13T10:34:55.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:55.069+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.070554+0000) 2019-10-13T10:34:55.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:55.069+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.071234+0000) 2019-10-13T10:34:55.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:34:55.070+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.071476+0000) 2019-10-13T10:34:55.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:34:55.070+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.071483+0000) 2019-10-13T10:34:55.073 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:34:55.072+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.073653+0000) 2019-10-13T10:34:55.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:34:55.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190597 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:34:55.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:34:55.643+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.643921+0000) 2019-10-13T10:34:55.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:34:55.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:55.643961+0000) 2019-10-13T10:34:58.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:34:58.551+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.552104+0000) 2019-10-13T10:34:58.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:34:58.889+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.889454+0000) 2019-10-13T10:35:00.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:00.080+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.081320+0000) 2019-10-13T10:35:00.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:00.080+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.080567+0000) 2019-10-13T10:35:00.083 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:00.080+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.080969+0000) 2019-10-13T10:35:00.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:00.080+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.081003+0000) 2019-10-13T10:35:00.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:00.080+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.081162+0000) 2019-10-13T10:35:00.083 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:00.081+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.081565+0000) 2019-10-13T10:35:00.083 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:00.081+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.081686+0000) 2019-10-13T10:35:00.084 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:00.082+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.083231+0000) 2019-10-13T10:35:00.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:00.087+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:01.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:01.503+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:01.503503+0000) 2019-10-13T10:35:02.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:02.993+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:02.993606+0000) 2019-10-13T10:35:02.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:02.992+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:02.992774+0000) 2019-10-13T10:35:04.669 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:04.668+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.669338+0000) 2019-10-13T10:35:05.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:05.089+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.090011+0000) 2019-10-13T10:35:05.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:05.090+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.091383+0000) 2019-10-13T10:35:05.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:05.090+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.090613+0000) 2019-10-13T10:35:05.092 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:05.090+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.090830+0000) 2019-10-13T10:35:05.093 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:05.090+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.090892+0000) 2019-10-13T10:35:05.093 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:05.090+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.091103+0000) 2019-10-13T10:35:05.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:05.092+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.093086+0000) 2019-10-13T10:35:05.094 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:05.091+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.091509+0000) 2019-10-13T10:35:05.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:05.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190657 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:05.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:05.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.644130+0000) 2019-10-13T10:35:05.645 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:05.643+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:05.644126+0000) 2019-10-13T10:35:08.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:08.552+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.552276+0000) 2019-10-13T10:35:08.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:08.889+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.889641+0000) 2019-10-13T10:35:10.101 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:10.099+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.099449+0000) 2019-10-13T10:35:10.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:10.099+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.099623+0000) 2019-10-13T10:35:10.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:10.099+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.099745+0000) 2019-10-13T10:35:10.101 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:10.099+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.099931+0000) 2019-10-13T10:35:10.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:10.100+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.101266+0000) 2019-10-13T10:35:10.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:10.099+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.100173+0000) 2019-10-13T10:35:10.103 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:10.100+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.100656+0000) 2019-10-13T10:35:10.104 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:10.103+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.103871+0000) 2019-10-13T10:35:10.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:10.105+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190689 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:10.482 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:35:10.483 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:10.487 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:35:10.487 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:11.505 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:11.503+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:11.503713+0000) 2019-10-13T10:35:12.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:12.992+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:12.993775+0000) 2019-10-13T10:35:12.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:12.992+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:12.992952+0000) 2019-10-13T10:35:14.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:14.670+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.671605+0000) 2019-10-13T10:35:15.109 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:15.107+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.107584+0000) 2019-10-13T10:35:15.109 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:15.107+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.107744+0000) 2019-10-13T10:35:15.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:15.107+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.108906+0000) 2019-10-13T10:35:15.110 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:15.107+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.108178+0000) 2019-10-13T10:35:15.110 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:15.107+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.108334+0000) 2019-10-13T10:35:15.110 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:15.108+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.108741+0000) 2019-10-13T10:35:15.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:15.108+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.108892+0000) 2019-10-13T10:35:15.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:15.110+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.112199+0000) 2019-10-13T10:35:15.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:15.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190719 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:15.645 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:15.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.644327+0000) 2019-10-13T10:35:15.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:15.644+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:15.644306+0000) 2019-10-13T10:35:18.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:18.551+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.552485+0000) 2019-10-13T10:35:18.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:18.888+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.889887+0000) 2019-10-13T10:35:20.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:20.116+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.117543+0000) 2019-10-13T10:35:20.119 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:20.116+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.117423+0000) 2019-10-13T10:35:20.120 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:20.116+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.117938+0000) 2019-10-13T10:35:20.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:20.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.118176+0000) 2019-10-13T10:35:20.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:20.116+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.118226+0000) 2019-10-13T10:35:20.120 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:20.117+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.118422+0000) 2019-10-13T10:35:20.121 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:20.116+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.118609+0000) 2019-10-13T10:35:20.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:20.119+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.120870+0000) 2019-10-13T10:35:20.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:20.123+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190749 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:21.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:21.502+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:21.503905+0000) 2019-10-13T10:35:22.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:22.993+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:22.993968+0000) 2019-10-13T10:35:22.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:22.992+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:22.993137+0000) 2019-10-13T10:35:24.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:24.673+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.674151+0000) 2019-10-13T10:35:25.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:25.125+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.126587+0000) 2019-10-13T10:35:25.129 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:25.125+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.126720+0000) 2019-10-13T10:35:25.129 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:25.125+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.127143+0000) 2019-10-13T10:35:25.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:25.126+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.127491+0000) 2019-10-13T10:35:25.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:25.126+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.127875+0000) 2019-10-13T10:35:25.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:25.128+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.128989+0000) 2019-10-13T10:35:25.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:25.129+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.130635+0000) 2019-10-13T10:35:25.131 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:25.126+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.128176+0000) 2019-10-13T10:35:25.135 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:25.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190779 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:25.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:25.643+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.644480+0000) 2019-10-13T10:35:25.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:25.643+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:25.644480+0000) 2019-10-13T10:35:28.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:28.551+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.552691+0000) 2019-10-13T10:35:28.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:28.889+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.890038+0000) 2019-10-13T10:35:30.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:30.134+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.135789+0000) 2019-10-13T10:35:30.137 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:30.134+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.135603+0000) 2019-10-13T10:35:30.138 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:30.134+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.136207+0000) 2019-10-13T10:35:30.138 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:30.134+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.136140+0000) 2019-10-13T10:35:30.138 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:30.136+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.137390+0000) 2019-10-13T10:35:30.139 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:30.135+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.136535+0000) 2019-10-13T10:35:30.139 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:30.135+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.136610+0000) 2019-10-13T10:35:30.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:30.139+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.139820+0000) 2019-10-13T10:35:30.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:30.141+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190811 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:31.505 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:31.503+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:31.504102+0000) 2019-10-13T10:35:32.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:32.992+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:32.993336+0000) 2019-10-13T10:35:32.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:32.993+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:32.994147+0000) 2019-10-13T10:35:34.677 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:34.676+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.676676+0000) 2019-10-13T10:35:35.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:35.142+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.144183+0000) 2019-10-13T10:35:35.146 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:35.142+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.144361+0000) 2019-10-13T10:35:35.146 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:35.143+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.144490+0000) 2019-10-13T10:35:35.146 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:35.143+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.144840+0000) 2019-10-13T10:35:35.147 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:35.145+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.146165+0000) 2019-10-13T10:35:35.147 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:35.143+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.145013+0000) 2019-10-13T10:35:35.147 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:35.143+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.145300+0000) 2019-10-13T10:35:35.148 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:35.147+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.147869+0000) 2019-10-13T10:35:35.153 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:35.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:35.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:35.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.644655+0000) 2019-10-13T10:35:35.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:35.644+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:35.644655+0000) 2019-10-13T10:35:38.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:38.551+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.552857+0000) 2019-10-13T10:35:38.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:38.889+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.890189+0000) 2019-10-13T10:35:40.157 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:40.153+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.154721+0000) 2019-10-13T10:35:40.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:40.153+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.154985+0000) 2019-10-13T10:35:40.157 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:40.153+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.155198+0000) 2019-10-13T10:35:40.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:40.153+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.155265+0000) 2019-10-13T10:35:40.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:40.153+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.155381+0000) 2019-10-13T10:35:40.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:40.154+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.155519+0000) 2019-10-13T10:35:40.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:40.156+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.156922+0000) 2019-10-13T10:35:40.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:40.156+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.157652+0000) 2019-10-13T10:35:40.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:40.161+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:40.645 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:35:40.646 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:40.651 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:35:40.651 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:41.505 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:41.503+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:41.504268+0000) 2019-10-13T10:35:42.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:42.993+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:42.994330+0000) 2019-10-13T10:35:42.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:42.993+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:42.993507+0000) 2019-10-13T10:35:44.679 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:44.678+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.679183+0000) 2019-10-13T10:35:45.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:45.164+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.165382+0000) 2019-10-13T10:35:45.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:45.164+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.164615+0000) 2019-10-13T10:35:45.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:45.164+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.164869+0000) 2019-10-13T10:35:45.167 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:45.164+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.165083+0000) 2019-10-13T10:35:45.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:45.164+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.165187+0000) 2019-10-13T10:35:45.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:45.165+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.165483+0000) 2019-10-13T10:35:45.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:45.165+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.165745+0000) 2019-10-13T10:35:45.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:45.167+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.167951+0000) 2019-10-13T10:35:45.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:45.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:45.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:45.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.644825+0000) 2019-10-13T10:35:45.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:45.644+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:45.644825+0000) 2019-10-13T10:35:48.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:48.552+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.553041+0000) 2019-10-13T10:35:48.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:48.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.890374+0000) 2019-10-13T10:35:50.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:50.173+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.174261+0000) 2019-10-13T10:35:50.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:50.174+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.174458+0000) 2019-10-13T10:35:50.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:50.174+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.174697+0000) 2019-10-13T10:35:50.177 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:50.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.175535+0000) 2019-10-13T10:35:50.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:50.174+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.174998+0000) 2019-10-13T10:35:50.177 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:50.174+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.175285+0000) 2019-10-13T10:35:50.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:50.174+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.175317+0000) 2019-10-13T10:35:50.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:50.177+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.177985+0000) 2019-10-13T10:35:50.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:50.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190933 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:51.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:51.504+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:51.504447+0000) 2019-10-13T10:35:52.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:52.994+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:52.994522+0000) 2019-10-13T10:35:52.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:52.993+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:52.993632+0000) 2019-10-13T10:35:54.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:54.681+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.681502+0000) 2019-10-13T10:35:55.185 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:35:55.184+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.184899+0000) 2019-10-13T10:35:55.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:35:55.184+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.184428+0000) 2019-10-13T10:35:55.186 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:55.184+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.184636+0000) 2019-10-13T10:35:55.187 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:35:55.184+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.184839+0000) 2019-10-13T10:35:55.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:55.184+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.185036+0000) 2019-10-13T10:35:55.187 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:55.184+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.185188+0000) 2019-10-13T10:35:55.187 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:55.184+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.185373+0000) 2019-10-13T10:35:55.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:35:55.185+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.186564+0000) 2019-10-13T10:35:55.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:35:55.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:35:55.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:35:55.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.645002+0000) 2019-10-13T10:35:55.646 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:35:55.644+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:55.644996+0000) 2019-10-13T10:35:58.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:35:58.553+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.553258+0000) 2019-10-13T10:35:58.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:35:58.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.890549+0000) 2019-10-13T10:36:00.195 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:00.193+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.193749+0000) 2019-10-13T10:36:00.196 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:00.194+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.195033+0000) 2019-10-13T10:36:00.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:00.193+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.194360+0000) 2019-10-13T10:36:00.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:00.194+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.194594+0000) 2019-10-13T10:36:00.197 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:00.194+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.194663+0000) 2019-10-13T10:36:00.197 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:00.194+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.194790+0000) 2019-10-13T10:36:00.197 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:00.194+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.194919+0000) 2019-10-13T10:36:00.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:00.197+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.198052+0000) 2019-10-13T10:36:00.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:00.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 190993 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:01.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:01.504+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:01.504626+0000) 2019-10-13T10:36:02.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:02.993+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:02.994706+0000) 2019-10-13T10:36:02.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:02.992+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:02.993826+0000) 2019-10-13T10:36:04.684 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:04.682+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.683985+0000) 2019-10-13T10:36:05.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:05.203+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.205416+0000) 2019-10-13T10:36:05.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:05.203+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.204732+0000) 2019-10-13T10:36:05.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:05.203+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.204824+0000) 2019-10-13T10:36:05.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:05.203+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.205067+0000) 2019-10-13T10:36:05.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:05.203+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.205282+0000) 2019-10-13T10:36:05.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:05.204+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.205543+0000) 2019-10-13T10:36:05.208 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:05.203+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.205328+0000) 2019-10-13T10:36:05.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:05.205+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.207409+0000) 2019-10-13T10:36:05.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:05.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:05.646 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:05.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.645187+0000) 2019-10-13T10:36:05.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:05.644+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:05.645187+0000) 2019-10-13T10:36:08.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:08.552+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.553435+0000) 2019-10-13T10:36:08.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:08.889+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.890765+0000) 2019-10-13T10:36:10.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:10.212+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.214045+0000) 2019-10-13T10:36:10.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:10.212+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.214188+0000) 2019-10-13T10:36:10.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:10.213+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.214518+0000) 2019-10-13T10:36:10.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:10.213+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.215326+0000) 2019-10-13T10:36:10.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:10.213+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.214772+0000) 2019-10-13T10:36:10.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:10.213+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.214978+0000) 2019-10-13T10:36:10.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:10.213+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.215187+0000) 2019-10-13T10:36:10.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:10.215+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.217621+0000) 2019-10-13T10:36:10.222 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:10.219+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:10.808 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:36:10.808 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:10.813 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:36:10.813 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:11.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:11.503+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:11.504816+0000) 2019-10-13T10:36:12.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:12.994+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:12.994904+0000) 2019-10-13T10:36:12.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:12.993+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:12.994005+0000) 2019-10-13T10:36:14.686 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:14.685+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.686396+0000) 2019-10-13T10:36:15.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:15.221+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.223035+0000) 2019-10-13T10:36:15.224 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:15.221+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.223205+0000) 2019-10-13T10:36:15.225 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:15.222+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.224290+0000) 2019-10-13T10:36:15.225 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:15.222+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.223641+0000) 2019-10-13T10:36:15.226 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:15.222+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.224011+0000) 2019-10-13T10:36:15.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:15.223+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.224391+0000) 2019-10-13T10:36:15.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:15.222+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.224288+0000) 2019-10-13T10:36:15.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:15.225+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.227254+0000) 2019-10-13T10:36:15.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:15.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:15.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:15.644+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.645360+0000) 2019-10-13T10:36:15.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:15.644+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:15.645358+0000) 2019-10-13T10:36:18.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:18.552+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.553650+0000) 2019-10-13T10:36:18.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:18.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.890949+0000) 2019-10-13T10:36:20.234 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:20.233+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.233855+0000) 2019-10-13T10:36:20.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:20.231+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.233088+0000) 2019-10-13T10:36:20.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:20.231+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.233226+0000) 2019-10-13T10:36:20.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:20.232+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.233539+0000) 2019-10-13T10:36:20.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:20.232+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.233586+0000) 2019-10-13T10:36:20.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:20.232+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.234030+0000) 2019-10-13T10:36:20.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:20.232+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.234191+0000) 2019-10-13T10:36:20.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:20.236+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.236811+0000) 2019-10-13T10:36:20.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:20.239+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191115 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:21.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:21.504+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:21.505065+0000) 2019-10-13T10:36:22.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:22.994+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:22.995077+0000) 2019-10-13T10:36:22.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:22.993+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:22.994185+0000) 2019-10-13T10:36:24.689 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:24.688+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.688780+0000) 2019-10-13T10:36:25.244 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:25.243+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.243870+0000) 2019-10-13T10:36:25.245 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:25.241+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.243137+0000) 2019-10-13T10:36:25.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:25.241+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.243238+0000) 2019-10-13T10:36:25.245 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:25.242+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.243462+0000) 2019-10-13T10:36:25.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:25.242+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.243461+0000) 2019-10-13T10:36:25.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:25.242+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.243606+0000) 2019-10-13T10:36:25.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:25.242+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.244058+0000) 2019-10-13T10:36:25.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:25.246+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.247636+0000) 2019-10-13T10:36:25.252 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:25.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:25.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:25.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.645554+0000) 2019-10-13T10:36:25.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:25.645+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:25.645555+0000) 2019-10-13T10:36:28.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:28.553+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.553834+0000) 2019-10-13T10:36:28.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:28.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.891151+0000) 2019-10-13T10:36:30.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:30.252+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.252893+0000) 2019-10-13T10:36:30.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:30.252+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.253195+0000) 2019-10-13T10:36:30.255 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:30.253+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.253393+0000) 2019-10-13T10:36:30.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:30.253+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.254722+0000) 2019-10-13T10:36:30.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:30.253+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.253567+0000) 2019-10-13T10:36:30.256 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:30.253+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.253971+0000) 2019-10-13T10:36:30.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:30.253+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.254177+0000) 2019-10-13T10:36:30.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:30.254+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.255656+0000) 2019-10-13T10:36:30.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:30.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191177 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:31.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:31.504+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:31.505259+0000) 2019-10-13T10:36:32.995 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:32.993+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:32.994310+0000) 2019-10-13T10:36:32.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:32.994+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:32.995239+0000) 2019-10-13T10:36:34.691 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:34.690+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.691184+0000) 2019-10-13T10:36:35.264 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:35.262+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.262547+0000) 2019-10-13T10:36:35.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:35.262+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.262631+0000) 2019-10-13T10:36:35.264 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:35.262+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.262977+0000) 2019-10-13T10:36:35.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:35.263+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.263756+0000) 2019-10-13T10:36:35.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:35.263+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.263636+0000) 2019-10-13T10:36:35.266 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:35.263+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.263737+0000) 2019-10-13T10:36:35.266 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:35.263+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.263994+0000) 2019-10-13T10:36:35.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:35.264+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.265426+0000) 2019-10-13T10:36:35.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:35.269+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191207 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:35.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:35.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.645717+0000) 2019-10-13T10:36:35.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:35.645+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:35.645706+0000) 2019-10-13T10:36:38.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:38.553+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.553992+0000) 2019-10-13T10:36:38.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:38.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.891355+0000) 2019-10-13T10:36:40.273 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:40.272+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.272777+0000) 2019-10-13T10:36:40.274 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:40.271+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.272091+0000) 2019-10-13T10:36:40.274 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:40.271+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.272312+0000) 2019-10-13T10:36:40.274 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:40.272+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.272678+0000) 2019-10-13T10:36:40.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:40.272+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.272802+0000) 2019-10-13T10:36:40.275 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:40.272+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.272995+0000) 2019-10-13T10:36:40.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:40.272+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.273155+0000) 2019-10-13T10:36:40.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:40.274+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.275263+0000) 2019-10-13T10:36:40.281 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:40.279+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:40.943 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:36:40.944 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:40.948 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:36:40.948 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:41.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:41.505+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:41.505440+0000) 2019-10-13T10:36:42.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:42.994+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:42.995395+0000) 2019-10-13T10:36:42.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:42.994+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:42.994486+0000) 2019-10-13T10:36:44.694 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:44.692+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.693472+0000) 2019-10-13T10:36:45.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:45.281+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.282258+0000) 2019-10-13T10:36:45.283 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:45.281+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.281772+0000) 2019-10-13T10:36:45.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:45.281+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.281876+0000) 2019-10-13T10:36:45.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:45.281+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.282320+0000) 2019-10-13T10:36:45.284 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:45.282+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.282522+0000) 2019-10-13T10:36:45.284 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:45.282+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.282712+0000) 2019-10-13T10:36:45.284 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:45.282+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.282821+0000) 2019-10-13T10:36:45.286 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:45.285+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.285953+0000) 2019-10-13T10:36:45.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:45.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:45.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:45.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.645901+0000) 2019-10-13T10:36:45.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:45.645+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:45.645903+0000) 2019-10-13T10:36:48.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:48.553+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.554140+0000) 2019-10-13T10:36:48.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:48.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.891519+0000) 2019-10-13T10:36:50.292 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:50.289+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.290898+0000) 2019-10-13T10:36:50.293 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:50.289+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.291123+0000) 2019-10-13T10:36:50.293 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:50.290+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.291422+0000) 2019-10-13T10:36:50.293 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:50.290+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.291553+0000) 2019-10-13T10:36:50.294 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:50.291+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.292177+0000) 2019-10-13T10:36:50.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:50.290+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.291703+0000) 2019-10-13T10:36:50.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:50.290+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.292003+0000) 2019-10-13T10:36:50.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:50.294+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.294749+0000) 2019-10-13T10:36:50.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:50.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:51.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:51.504+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:51.505639+0000) 2019-10-13T10:36:52.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:52.993+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:52.994649+0000) 2019-10-13T10:36:52.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:52.994+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:52.995594+0000) 2019-10-13T10:36:54.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:54.694+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.695804+0000) 2019-10-13T10:36:55.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:55.297+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.298580+0000) 2019-10-13T10:36:55.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:36:55.298+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.299846+0000) 2019-10-13T10:36:55.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:36:55.297+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.299273+0000) 2019-10-13T10:36:55.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:55.297+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.299304+0000) 2019-10-13T10:36:55.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:55.298+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.299483+0000) 2019-10-13T10:36:55.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:36:55.298+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.299791+0000) 2019-10-13T10:36:55.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:55.298+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.299971+0000) 2019-10-13T10:36:55.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:36:55.300+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.302704+0000) 2019-10-13T10:36:55.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:36:55.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:36:55.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:36:55.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.646051+0000) 2019-10-13T10:36:55.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:36:55.645+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:55.646051+0000) 2019-10-13T10:36:58.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:36:58.552+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.554340+0000) 2019-10-13T10:36:58.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:36:58.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.891662+0000) 2019-10-13T10:37:00.309 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:00.307+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.309347+0000) 2019-10-13T10:37:00.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:00.307+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.308608+0000) 2019-10-13T10:37:00.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:00.307+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.308832+0000) 2019-10-13T10:37:00.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:00.307+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.308927+0000) 2019-10-13T10:37:00.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:00.307+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.309151+0000) 2019-10-13T10:37:00.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:00.308+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.309471+0000) 2019-10-13T10:37:00.311 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:00.308+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.309521+0000) 2019-10-13T10:37:00.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:00.311+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.313183+0000) 2019-10-13T10:37:00.317 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:00.314+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191359 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:01.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:01.504+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:01.505786+0000) 2019-10-13T10:37:02.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:02.994+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:02.995765+0000) 2019-10-13T10:37:02.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:02.994+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:02.994828+0000) 2019-10-13T10:37:04.698 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:04.697+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.697888+0000) 2019-10-13T10:37:05.319 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:05.316+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.318207+0000) 2019-10-13T10:37:05.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:05.317+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.319389+0000) 2019-10-13T10:37:05.321 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:05.317+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.318694+0000) 2019-10-13T10:37:05.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:05.317+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.318818+0000) 2019-10-13T10:37:05.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:05.317+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.319046+0000) 2019-10-13T10:37:05.321 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:05.317+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.319274+0000) 2019-10-13T10:37:05.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:05.318+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.319459+0000) 2019-10-13T10:37:05.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:05.321+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.322844+0000) 2019-10-13T10:37:05.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:05.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:05.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:05.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.646207+0000) 2019-10-13T10:37:05.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:05.645+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:05.646207+0000) 2019-10-13T10:37:08.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:08.553+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.554515+0000) 2019-10-13T10:37:08.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:08.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.891779+0000) 2019-10-13T10:37:10.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:10.326+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.328336+0000) 2019-10-13T10:37:10.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:10.326+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.328386+0000) 2019-10-13T10:37:10.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:10.327+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.329100+0000) 2019-10-13T10:37:10.332 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:10.328+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.329599+0000) 2019-10-13T10:37:10.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:10.328+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.329403+0000) 2019-10-13T10:37:10.332 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:10.328+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.329726+0000) 2019-10-13T10:37:10.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:10.329+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.329803+0000) 2019-10-13T10:37:10.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:10.331+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.332276+0000) 2019-10-13T10:37:10.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:10.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191421 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:11.166 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:37:11.167 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:11.172 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:37:11.172 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:11.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:11.505+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:11.505993+0000) 2019-10-13T10:37:12.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:12.994+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:12.994960+0000) 2019-10-13T10:37:12.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:12.995+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:12.995927+0000) 2019-10-13T10:37:14.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:14.699+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.700184+0000) 2019-10-13T10:37:15.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:15.336+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.337574+0000) 2019-10-13T10:37:15.339 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:15.336+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.337998+0000) 2019-10-13T10:37:15.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:15.336+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.338248+0000) 2019-10-13T10:37:15.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:15.337+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.338891+0000) 2019-10-13T10:37:15.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:15.337+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.339168+0000) 2019-10-13T10:37:15.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:15.339+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.340538+0000) 2019-10-13T10:37:15.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:15.338+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.339395+0000) 2019-10-13T10:37:15.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:15.342+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.343489+0000) 2019-10-13T10:37:15.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:15.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191451 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:15.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:15.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.646381+0000) 2019-10-13T10:37:15.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:15.646+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:15.646381+0000) 2019-10-13T10:37:18.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:18.554+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.554689+0000) 2019-10-13T10:37:18.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:18.891+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.891960+0000) 2019-10-13T10:37:20.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:20.347+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.348065+0000) 2019-10-13T10:37:20.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:20.347+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.348308+0000) 2019-10-13T10:37:20.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:20.348+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.348545+0000) 2019-10-13T10:37:20.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:20.348+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.348483+0000) 2019-10-13T10:37:20.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:20.349+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.350293+0000) 2019-10-13T10:37:20.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:20.348+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.349072+0000) 2019-10-13T10:37:20.351 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:20.348+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.349232+0000) 2019-10-13T10:37:20.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:20.351+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.351765+0000) 2019-10-13T10:37:20.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:20.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191481 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:21.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:21.505+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:21.506157+0000) 2019-10-13T10:37:22.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:22.994+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:22.995159+0000) 2019-10-13T10:37:22.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:22.995+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:22.996133+0000) 2019-10-13T10:37:24.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:24.701+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.702597+0000) 2019-10-13T10:37:25.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:25.357+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.358163+0000) 2019-10-13T10:37:25.360 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:25.357+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.358224+0000) 2019-10-13T10:37:25.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:25.358+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.358493+0000) 2019-10-13T10:37:25.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:25.358+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.359641+0000) 2019-10-13T10:37:25.361 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:25.358+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.359046+0000) 2019-10-13T10:37:25.361 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:25.358+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.359171+0000) 2019-10-13T10:37:25.361 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:25.359+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.359500+0000) 2019-10-13T10:37:25.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:25.360+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.360873+0000) 2019-10-13T10:37:25.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:25.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191513 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:25.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:25.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.646561+0000) 2019-10-13T10:37:25.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:25.646+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:25.646559+0000) 2019-10-13T10:37:28.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:28.554+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.554877+0000) 2019-10-13T10:37:28.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:28.892+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.892153+0000) 2019-10-13T10:37:30.369 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:30.368+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.369026+0000) 2019-10-13T10:37:30.371 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:30.367+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.368293+0000) 2019-10-13T10:37:30.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:30.368+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.368951+0000) 2019-10-13T10:37:30.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:30.368+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.368872+0000) 2019-10-13T10:37:30.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:30.368+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.369189+0000) 2019-10-13T10:37:30.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:30.369+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.369550+0000) 2019-10-13T10:37:30.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:30.369+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.369763+0000) 2019-10-13T10:37:30.373 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:30.371+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.372404+0000) 2019-10-13T10:37:30.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:30.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:31.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:31.506+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:31.506323+0000) 2019-10-13T10:37:32.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:32.994+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:32.995371+0000) 2019-10-13T10:37:32.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:32.996+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:32.996307+0000) 2019-10-13T10:37:34.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:34.704+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.704868+0000) 2019-10-13T10:37:35.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:35.376+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.377672+0000) 2019-10-13T10:37:35.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:35.376+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.377827+0000) 2019-10-13T10:37:35.380 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:35.376+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.378012+0000) 2019-10-13T10:37:35.380 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:35.376+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.378136+0000) 2019-10-13T10:37:35.380 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:35.376+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.378288+0000) 2019-10-13T10:37:35.381 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:35.378+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.378774+0000) 2019-10-13T10:37:35.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:35.377+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.378834+0000) 2019-10-13T10:37:35.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:35.381+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.382264+0000) 2019-10-13T10:37:35.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:35.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:35.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:35.645+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.646751+0000) 2019-10-13T10:37:35.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:35.645+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:35.646751+0000) 2019-10-13T10:37:38.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:38.553+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.555050+0000) 2019-10-13T10:37:38.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:38.890+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.892283+0000) 2019-10-13T10:37:40.389 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:40.385+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.387240+0000) 2019-10-13T10:37:40.389 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:40.386+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.387406+0000) 2019-10-13T10:37:40.390 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:40.387+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.388629+0000) 2019-10-13T10:37:40.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:40.386+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.388074+0000) 2019-10-13T10:37:40.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:40.387+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.388590+0000) 2019-10-13T10:37:40.390 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:40.387+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.388902+0000) 2019-10-13T10:37:40.391 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:40.387+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.389097+0000) 2019-10-13T10:37:40.391 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:40.390+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.391052+0000) 2019-10-13T10:37:40.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:40.394+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:41.321 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:37:41.321 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:41.326 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:37:41.326 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:41.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:41.505+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:41.506472+0000) 2019-10-13T10:37:42.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:42.996+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:42.996513+0000) 2019-10-13T10:37:42.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:42.994+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:42.995570+0000) 2019-10-13T10:37:44.707 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:44.706+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.707410+0000) 2019-10-13T10:37:45.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:45.396+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.398162+0000) 2019-10-13T10:37:45.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:45.397+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.398455+0000) 2019-10-13T10:37:45.400 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:45.397+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.398432+0000) 2019-10-13T10:37:45.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:45.397+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.398792+0000) 2019-10-13T10:37:45.401 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:45.398+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.400191+0000) 2019-10-13T10:37:45.401 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:45.397+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.399102+0000) 2019-10-13T10:37:45.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:45.397+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.399379+0000) 2019-10-13T10:37:45.402 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:45.400+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.401990+0000) 2019-10-13T10:37:45.406 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:45.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:45.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:45.646+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.646921+0000) 2019-10-13T10:37:45.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:45.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:45.646921+0000) 2019-10-13T10:37:48.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:48.554+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.555236+0000) 2019-10-13T10:37:48.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:48.891+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.892419+0000) 2019-10-13T10:37:50.409 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:50.405+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.407171+0000) 2019-10-13T10:37:50.409 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:50.406+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.407642+0000) 2019-10-13T10:37:50.409 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:50.406+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.407965+0000) 2019-10-13T10:37:50.409 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:50.406+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.408098+0000) 2019-10-13T10:37:50.410 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:50.407+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.409015+0000) 2019-10-13T10:37:50.411 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:50.407+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.408776+0000) 2019-10-13T10:37:50.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:50.407+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.409000+0000) 2019-10-13T10:37:50.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:50.409+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.411614+0000) 2019-10-13T10:37:50.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:50.414+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191664 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:51.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:51.505+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:51.506634+0000) 2019-10-13T10:37:52.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:52.995+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:52.995724+0000) 2019-10-13T10:37:52.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:52.995+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:52.996723+0000) 2019-10-13T10:37:54.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:54.708+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.709736+0000) 2019-10-13T10:37:55.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:55.415+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.417068+0000) 2019-10-13T10:37:55.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:37:55.415+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.416981+0000) 2019-10-13T10:37:55.419 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:55.415+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.417208+0000) 2019-10-13T10:37:55.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:37:55.416+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.417517+0000) 2019-10-13T10:37:55.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:55.416+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.417493+0000) 2019-10-13T10:37:55.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:37:55.416+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.418607+0000) 2019-10-13T10:37:55.420 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:55.416+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.418056+0000) 2019-10-13T10:37:55.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:37:55.419+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.420862+0000) 2019-10-13T10:37:55.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:37:55.421+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191694 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:37:55.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:37:55.646+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.647018+0000) 2019-10-13T10:37:55.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:37:55.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:55.647084+0000) 2019-10-13T10:37:58.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:37:58.554+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:58.555452+0000) 2019-10-13T10:37:58.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:37:58.892+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:58.892593+0000) 2019-10-13T10:38:00.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:00.423+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.425078+0000) 2019-10-13T10:38:00.427 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:00.425+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.426712+0000) 2019-10-13T10:38:00.427 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:00.424+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.425447+0000) 2019-10-13T10:38:00.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:00.423+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.425348+0000) 2019-10-13T10:38:00.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:00.424+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.425681+0000) 2019-10-13T10:38:00.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:00.424+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.426143+0000) 2019-10-13T10:38:00.428 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:00.424+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.426389+0000) 2019-10-13T10:38:00.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:00.428+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.429481+0000) 2019-10-13T10:38:00.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:00.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191724 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:01.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:01.506+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:01.506806+0000) 2019-10-13T10:38:02.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:02.995+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:02.995864+0000) 2019-10-13T10:38:02.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:02.996+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:02.996938+0000) 2019-10-13T10:38:04.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:04.711+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.712051+0000) 2019-10-13T10:38:05.436 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:05.435+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.435824+0000) 2019-10-13T10:38:05.437 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:05.434+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.435239+0000) 2019-10-13T10:38:05.437 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:05.435+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.435561+0000) 2019-10-13T10:38:05.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:05.435+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.435699+0000) 2019-10-13T10:38:05.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:05.435+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.435817+0000) 2019-10-13T10:38:05.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:05.435+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.436216+0000) 2019-10-13T10:38:05.438 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:05.436+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.436529+0000) 2019-10-13T10:38:05.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:05.438+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.438991+0000) 2019-10-13T10:38:05.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:05.442+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:05.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:05.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.647220+0000) 2019-10-13T10:38:05.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:05.646+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:05.647202+0000) 2019-10-13T10:38:08.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:08.555+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:08.555612+0000) 2019-10-13T10:38:08.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:08.892+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:08.892751+0000) 2019-10-13T10:38:10.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:10.443+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.444191+0000) 2019-10-13T10:38:10.445 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:10.444+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.444460+0000) 2019-10-13T10:38:10.446 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:10.445+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.445898+0000) 2019-10-13T10:38:10.447 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:10.444+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.445043+0000) 2019-10-13T10:38:10.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:10.444+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.445213+0000) 2019-10-13T10:38:10.447 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:10.444+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.445245+0000) 2019-10-13T10:38:10.447 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:10.445+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.445409+0000) 2019-10-13T10:38:10.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:10.447+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.447844+0000) 2019-10-13T10:38:10.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:10.451+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:11.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:11.506+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:11.506996+0000) 2019-10-13T10:38:11.543 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:38:11.544 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:11.548 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:38:11.548 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:12.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:12.995+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:12.996032+0000) 2019-10-13T10:38:12.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:12.996+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:12.997133+0000) 2019-10-13T10:38:14.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:14.713+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.714354+0000) 2019-10-13T10:38:15.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:15.453+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.453780+0000) 2019-10-13T10:38:15.456 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:15.453+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.453939+0000) 2019-10-13T10:38:15.456 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:15.453+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.454113+0000) 2019-10-13T10:38:15.457 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:15.453+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.454318+0000) 2019-10-13T10:38:15.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:15.454+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.454537+0000) 2019-10-13T10:38:15.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:15.454+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.454768+0000) 2019-10-13T10:38:15.458 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:15.454+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.455274+0000) 2019-10-13T10:38:15.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:15.456+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.456786+0000) 2019-10-13T10:38:15.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:15.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191816 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:15.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:15.647+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.647410+0000) 2019-10-13T10:38:15.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:15.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:15.647416+0000) 2019-10-13T10:38:18.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:18.555+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:18.555802+0000) 2019-10-13T10:38:18.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:18.892+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:18.892938+0000) 2019-10-13T10:38:20.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:20.460+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.461996+0000) 2019-10-13T10:38:20.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:20.462+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.463445+0000) 2019-10-13T10:38:20.465 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:20.461+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.462415+0000) 2019-10-13T10:38:20.465 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:20.461+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.462655+0000) 2019-10-13T10:38:20.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:20.461+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.462851+0000) 2019-10-13T10:38:20.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:20.461+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.462923+0000) 2019-10-13T10:38:20.466 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:20.461+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.463253+0000) 2019-10-13T10:38:20.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:20.465+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.465787+0000) 2019-10-13T10:38:20.470 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:20.467+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191846 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:21.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:21.506+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:21.507201+0000) 2019-10-13T10:38:22.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:22.994+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:22.996188+0000) 2019-10-13T10:38:22.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:22.997+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:22.997308+0000) 2019-10-13T10:38:24.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:24.716+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.716393+0000) 2019-10-13T10:38:25.472 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:25.471+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.471773+0000) 2019-10-13T10:38:25.472 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:25.469+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.471051+0000) 2019-10-13T10:38:25.473 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:25.470+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.471768+0000) 2019-10-13T10:38:25.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:25.470+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.471926+0000) 2019-10-13T10:38:25.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:25.470+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.472253+0000) 2019-10-13T10:38:25.474 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:25.470+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.472273+0000) 2019-10-13T10:38:25.474 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:25.471+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.472403+0000) 2019-10-13T10:38:25.474 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:25.473+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.474321+0000) 2019-10-13T10:38:25.480 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:25.477+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:25.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:25.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.647574+0000) 2019-10-13T10:38:25.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:25.646+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:25.647576+0000) 2019-10-13T10:38:28.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:28.555+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:28.555991+0000) 2019-10-13T10:38:28.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:28.892+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:28.893110+0000) 2019-10-13T10:38:30.482 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:30.480+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.481663+0000) 2019-10-13T10:38:30.483 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:30.479+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.480910+0000) 2019-10-13T10:38:30.483 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:30.479+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.481091+0000) 2019-10-13T10:38:30.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:30.479+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.481307+0000) 2019-10-13T10:38:30.483 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:30.480+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.481500+0000) 2019-10-13T10:38:30.483 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:30.480+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.481659+0000) 2019-10-13T10:38:30.484 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:30.480+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.481790+0000) 2019-10-13T10:38:30.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:30.482+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.483568+0000) 2019-10-13T10:38:30.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:30.487+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191908 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:31.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:31.506+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:31.507376+0000) 2019-10-13T10:38:32.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:32.995+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:32.996339+0000) 2019-10-13T10:38:32.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:32.997+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:32.997486+0000) 2019-10-13T10:38:34.719 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:34.718+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.718697+0000) 2019-10-13T10:38:35.492 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:35.489+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.490847+0000) 2019-10-13T10:38:35.492 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:35.489+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.491179+0000) 2019-10-13T10:38:35.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:35.490+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.491406+0000) 2019-10-13T10:38:35.494 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:35.490+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.492547+0000) 2019-10-13T10:38:35.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:35.490+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.491810+0000) 2019-10-13T10:38:35.494 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:35.490+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.492113+0000) 2019-10-13T10:38:35.494 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:35.490+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.491905+0000) 2019-10-13T10:38:35.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:35.493+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.495419+0000) 2019-10-13T10:38:35.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:35.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191938 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:35.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:35.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.647759+0000) 2019-10-13T10:38:35.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:35.647+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:35.647755+0000) 2019-10-13T10:38:38.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:38.555+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:38.556179+0000) 2019-10-13T10:38:38.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:38.892+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:38.893329+0000) 2019-10-13T10:38:40.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:40.498+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.500378+0000) 2019-10-13T10:38:40.502 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:40.499+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.500519+0000) 2019-10-13T10:38:40.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:40.499+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.500808+0000) 2019-10-13T10:38:40.503 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:40.500+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.502370+0000) 2019-10-13T10:38:40.503 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:40.500+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.501563+0000) 2019-10-13T10:38:40.503 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:40.500+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.501704+0000) 2019-10-13T10:38:40.504 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:40.500+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.501912+0000) 2019-10-13T10:38:40.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:40.502+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.504035+0000) 2019-10-13T10:38:40.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:40.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 191968 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:41.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:41.507+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:41.507579+0000) 2019-10-13T10:38:41.750 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:38:41.750 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:41.755 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:38:41.755 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:42.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:42.995+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:42.996512+0000) 2019-10-13T10:38:42.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:42.996+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:42.997639+0000) 2019-10-13T10:38:44.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:44.720+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.721071+0000) 2019-10-13T10:38:45.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:45.507+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.509347+0000) 2019-10-13T10:38:45.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:45.508+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.509712+0000) 2019-10-13T10:38:45.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:45.508+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.509951+0000) 2019-10-13T10:38:45.512 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:45.509+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.511034+0000) 2019-10-13T10:38:45.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:45.509+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.510711+0000) 2019-10-13T10:38:45.512 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:45.509+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.510583+0000) 2019-10-13T10:38:45.512 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:45.509+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.510438+0000) 2019-10-13T10:38:45.513 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:45.511+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.513244+0000) 2019-10-13T10:38:45.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:45.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:45.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:45.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.647941+0000) 2019-10-13T10:38:45.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:45.647+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:45.647945+0000) 2019-10-13T10:38:48.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:48.555+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:48.556307+0000) 2019-10-13T10:38:48.894 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:48.893+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:48.893458+0000) 2019-10-13T10:38:50.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:50.518+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.518944+0000) 2019-10-13T10:38:50.521 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:50.518+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.519093+0000) 2019-10-13T10:38:50.522 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:50.520+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.520793+0000) 2019-10-13T10:38:50.522 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:50.519+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.519641+0000) 2019-10-13T10:38:50.522 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:50.519+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.519719+0000) 2019-10-13T10:38:50.523 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:50.519+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.520068+0000) 2019-10-13T10:38:50.523 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:50.519+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.520244+0000) 2019-10-13T10:38:50.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:50.521+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.522215+0000) 2019-10-13T10:38:50.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:50.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192030 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:51.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:51.507+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:51.507768+0000) 2019-10-13T10:38:52.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:52.996+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:52.996674+0000) 2019-10-13T10:38:52.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:52.997+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:52.997807+0000) 2019-10-13T10:38:54.724 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:54.723+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.723762+0000) 2019-10-13T10:38:55.530 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:55.527+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.528166+0000) 2019-10-13T10:38:55.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:38:55.527+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.528336+0000) 2019-10-13T10:38:55.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:55.528+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.528462+0000) 2019-10-13T10:38:55.531 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:55.528+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.528763+0000) 2019-10-13T10:38:55.531 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:38:55.528+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.529022+0000) 2019-10-13T10:38:55.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:55.528+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.529206+0000) 2019-10-13T10:38:55.532 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:38:55.529+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.529993+0000) 2019-10-13T10:38:55.533 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:38:55.531+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.532140+0000) 2019-10-13T10:38:55.535 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:38:55.534+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:38:55.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:38:55.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.648096+0000) 2019-10-13T10:38:55.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:38:55.647+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:55.648057+0000) 2019-10-13T10:38:58.557 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:38:58.556+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:58.556462+0000) 2019-10-13T10:38:58.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:38:58.893+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:58.893641+0000) 2019-10-13T10:39:00.538 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:00.535+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.536326+0000) 2019-10-13T10:39:00.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:00.536+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.536493+0000) 2019-10-13T10:39:00.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:00.536+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.536828+0000) 2019-10-13T10:39:00.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:00.536+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.536900+0000) 2019-10-13T10:39:00.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:00.536+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.537155+0000) 2019-10-13T10:39:00.540 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:00.537+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.537685+0000) 2019-10-13T10:39:00.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:00.537+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.538293+0000) 2019-10-13T10:39:00.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:00.540+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.540851+0000) 2019-10-13T10:39:00.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:00.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192090 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:01.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:01.507+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:01.507942+0000) 2019-10-13T10:39:02.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:02.996+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:02.996841+0000) 2019-10-13T10:39:02.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:02.997+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:02.997951+0000) 2019-10-13T10:39:04.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:04.725+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.726032+0000) 2019-10-13T10:39:05.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:05.545+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.546298+0000) 2019-10-13T10:39:05.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:05.546+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.546484+0000) 2019-10-13T10:39:05.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:05.546+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.546627+0000) 2019-10-13T10:39:05.548 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:05.546+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.546834+0000) 2019-10-13T10:39:05.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:05.546+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.547046+0000) 2019-10-13T10:39:05.549 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:05.547+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.548164+0000) 2019-10-13T10:39:05.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:05.548+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.549287+0000) 2019-10-13T10:39:05.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:05.546+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.547376+0000) 2019-10-13T10:39:05.555 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:05.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192122 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:05.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:05.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.648262+0000) 2019-10-13T10:39:05.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:05.648+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:05.648262+0000) 2019-10-13T10:39:08.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:08.555+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:08.556679+0000) 2019-10-13T10:39:08.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:08.892+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:08.893841+0000) 2019-10-13T10:39:10.557 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:10.554+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.555664+0000) 2019-10-13T10:39:10.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:10.554+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.555936+0000) 2019-10-13T10:39:10.558 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:10.554+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.556008+0000) 2019-10-13T10:39:10.558 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:10.554+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.556117+0000) 2019-10-13T10:39:10.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:10.554+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.556199+0000) 2019-10-13T10:39:10.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:10.554+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.556390+0000) 2019-10-13T10:39:10.559 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:10.556+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.557468+0000) 2019-10-13T10:39:10.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:10.559+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.559816+0000) 2019-10-13T10:39:10.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:10.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:11.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:11.507+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:11.508082+0000) 2019-10-13T10:39:11.893 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:39:11.893 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:11.897 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:39:11.897 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:12.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:12.996+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:12.997016+0000) 2019-10-13T10:39:12.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:12.997+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:12.998101+0000) 2019-10-13T10:39:14.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:14.728+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.728360+0000) 2019-10-13T10:39:15.564 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:15.561+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.563364+0000) 2019-10-13T10:39:15.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:15.562+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.563980+0000) 2019-10-13T10:39:15.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:15.562+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.564125+0000) 2019-10-13T10:39:15.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:15.562+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.564246+0000) 2019-10-13T10:39:15.566 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:15.563+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.564394+0000) 2019-10-13T10:39:15.566 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:15.564+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.565608+0000) 2019-10-13T10:39:15.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:15.563+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.565124+0000) 2019-10-13T10:39:15.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:15.566+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.567541+0000) 2019-10-13T10:39:15.572 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:15.570+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192182 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:15.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:15.646+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.648381+0000) 2019-10-13T10:39:15.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:15.647+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:15.648393+0000) 2019-10-13T10:39:18.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:18.556+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:18.556895+0000) 2019-10-13T10:39:18.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:18.893+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:18.894014+0000) 2019-10-13T10:39:20.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:20.571+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.573308+0000) 2019-10-13T10:39:20.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:20.572+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.573587+0000) 2019-10-13T10:39:20.576 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:20.573+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.574695+0000) 2019-10-13T10:39:20.576 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:20.572+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.573933+0000) 2019-10-13T10:39:20.576 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:20.572+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.574141+0000) 2019-10-13T10:39:20.576 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:20.572+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.574259+0000) 2019-10-13T10:39:20.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:20.573+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.574469+0000) 2019-10-13T10:39:20.577 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:20.576+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.577468+0000) 2019-10-13T10:39:20.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:20.580+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192212 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:21.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:21.506+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:21.508234+0000) 2019-10-13T10:39:22.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:22.996+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:22.997188+0000) 2019-10-13T10:39:22.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:22.998+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:22.998266+0000) 2019-10-13T10:39:24.731 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:24.730+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.730864+0000) 2019-10-13T10:39:25.585 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:25.583+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.584513+0000) 2019-10-13T10:39:25.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:25.582+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.583722+0000) 2019-10-13T10:39:25.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:25.582+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.583977+0000) 2019-10-13T10:39:25.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:25.582+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.584078+0000) 2019-10-13T10:39:25.586 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:25.583+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.584622+0000) 2019-10-13T10:39:25.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:25.583+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.584713+0000) 2019-10-13T10:39:25.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:25.583+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.584837+0000) 2019-10-13T10:39:25.587 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:25.585+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.586332+0000) 2019-10-13T10:39:25.592 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:25.589+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:25.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:25.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.648579+0000) 2019-10-13T10:39:25.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:25.647+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:25.648579+0000) 2019-10-13T10:39:28.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:28.556+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:28.557063+0000) 2019-10-13T10:39:28.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:28.893+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:28.894157+0000) 2019-10-13T10:39:30.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:30.591+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.592767+0000) 2019-10-13T10:39:30.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:30.591+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.593091+0000) 2019-10-13T10:39:30.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:30.592+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.593568+0000) 2019-10-13T10:39:30.595 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:30.592+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.593651+0000) 2019-10-13T10:39:30.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:30.592+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.594143+0000) 2019-10-13T10:39:30.596 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:30.593+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.594932+0000) 2019-10-13T10:39:30.596 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:30.592+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.594370+0000) 2019-10-13T10:39:30.596 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:30.594+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.596227+0000) 2019-10-13T10:39:30.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:30.599+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192274 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:31.509 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:31.507+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:31.508418+0000) 2019-10-13T10:39:32.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:32.996+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:32.997370+0000) 2019-10-13T10:39:32.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:32.997+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:32.998447+0000) 2019-10-13T10:39:34.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:34.732+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.733406+0000) 2019-10-13T10:39:35.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:35.602+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.602426+0000) 2019-10-13T10:39:35.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:35.601+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.603545+0000) 2019-10-13T10:39:35.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:35.602+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.602731+0000) 2019-10-13T10:39:35.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:35.602+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.602963+0000) 2019-10-13T10:39:35.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:35.602+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.603064+0000) 2019-10-13T10:39:35.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:35.603+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.603616+0000) 2019-10-13T10:39:35.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:35.603+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.603686+0000) 2019-10-13T10:39:35.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:35.604+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.606441+0000) 2019-10-13T10:39:35.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:35.609+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:35.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:35.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.648689+0000) 2019-10-13T10:39:35.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:35.648+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:35.648727+0000) 2019-10-13T10:39:38.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:38.556+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:38.557214+0000) 2019-10-13T10:39:38.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:38.894+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:38.894327+0000) 2019-10-13T10:39:40.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:40.611+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.611942+0000) 2019-10-13T10:39:40.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:40.611+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.612150+0000) 2019-10-13T10:39:40.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:40.612+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.612420+0000) 2019-10-13T10:39:40.614 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:40.612+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.613180+0000) 2019-10-13T10:39:40.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:40.612+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.612842+0000) 2019-10-13T10:39:40.615 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:40.612+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.613105+0000) 2019-10-13T10:39:40.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:40.612+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.613251+0000) 2019-10-13T10:39:40.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:40.613+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.614695+0000) 2019-10-13T10:39:40.621 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:40.619+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:41.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:41.508+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:41.508580+0000) 2019-10-13T10:39:41.994 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:39:41.994 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:41.998 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:39:41.999 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:42.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:42.997+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:42.997576+0000) 2019-10-13T10:39:42.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:42.997+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:42.998674+0000) 2019-10-13T10:39:44.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:44.735+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.735715+0000) 2019-10-13T10:39:45.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:45.621+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.622597+0000) 2019-10-13T10:39:45.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:45.621+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.621984+0000) 2019-10-13T10:39:45.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:45.621+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.622261+0000) 2019-10-13T10:39:45.624 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:45.622+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.622459+0000) 2019-10-13T10:39:45.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:45.622+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.622608+0000) 2019-10-13T10:39:45.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:45.622+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.622802+0000) 2019-10-13T10:39:45.625 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:45.622+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.623051+0000) 2019-10-13T10:39:45.626 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:45.625+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.626484+0000) 2019-10-13T10:39:45.631 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:45.629+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:45.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:45.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.648796+0000) 2019-10-13T10:39:45.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:45.648+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:45.648885+0000) 2019-10-13T10:39:48.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:48.557+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:48.557373+0000) 2019-10-13T10:39:48.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:48.894+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:48.894527+0000) 2019-10-13T10:39:50.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:50.631+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.632377+0000) 2019-10-13T10:39:50.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:50.631+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.631879+0000) 2019-10-13T10:39:50.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:50.631+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.631956+0000) 2019-10-13T10:39:50.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:50.631+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.632043+0000) 2019-10-13T10:39:50.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:50.631+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.632235+0000) 2019-10-13T10:39:50.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:50.632+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.632622+0000) 2019-10-13T10:39:50.635 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:50.632+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.632537+0000) 2019-10-13T10:39:50.636 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:50.635+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.635757+0000) 2019-10-13T10:39:50.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:50.638+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:51.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:51.508+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:51.508728+0000) 2019-10-13T10:39:52.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:52.996+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:52.997755+0000) 2019-10-13T10:39:52.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:52.998+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:52.998877+0000) 2019-10-13T10:39:54.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:54.737+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.738055+0000) 2019-10-13T10:39:55.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:39:55.638+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.640362+0000) 2019-10-13T10:39:55.642 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:55.639+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.640593+0000) 2019-10-13T10:39:55.642 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:55.639+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.640979+0000) 2019-10-13T10:39:55.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:55.639+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.641069+0000) 2019-10-13T10:39:55.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:39:55.639+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.641217+0000) 2019-10-13T10:39:55.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:55.639+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.641298+0000) 2019-10-13T10:39:55.643 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:39:55.641+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.641799+0000) 2019-10-13T10:39:55.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:39:55.642+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.643268+0000) 2019-10-13T10:39:55.647 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:39:55.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:39:55.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:39:55.647+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.648909+0000) 2019-10-13T10:39:55.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:39:55.647+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:55.648976+0000) 2019-10-13T10:39:58.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:39:58.556+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:58.557609+0000) 2019-10-13T10:39:58.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:39:58.893+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:58.894700+0000) 2019-10-13T10:40:00.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:00.648+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.649296+0000) 2019-10-13T10:40:00.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:00.647+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.648855+0000) 2019-10-13T10:40:00.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:00.647+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.649151+0000) 2019-10-13T10:40:00.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:00.647+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.649255+0000) 2019-10-13T10:40:00.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:00.648+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.649450+0000) 2019-10-13T10:40:00.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:00.648+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.649755+0000) 2019-10-13T10:40:00.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:00.648+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.650033+0000) 2019-10-13T10:40:00.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:00.650+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.651464+0000) 2019-10-13T10:40:00.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:00.656+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:01.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:01.507+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:01.508929+0000) 2019-10-13T10:40:02.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:02.997+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:02.997924+0000) 2019-10-13T10:40:02.999 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:02.998+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:02.999083+0000) 2019-10-13T10:40:04.741 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:04.740+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.740541+0000) 2019-10-13T10:40:05.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:05.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.649133+0000) 2019-10-13T10:40:05.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:05.648+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.649205+0000) 2019-10-13T10:40:05.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:05.658+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.659410+0000) 2019-10-13T10:40:05.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:05.658+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.659556+0000) 2019-10-13T10:40:05.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:05.658+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.659672+0000) 2019-10-13T10:40:05.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:05.660+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.660833+0000) 2019-10-13T10:40:05.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:05.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.660159+0000) 2019-10-13T10:40:05.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:05.658+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.660321+0000) 2019-10-13T10:40:05.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:05.659+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.660745+0000) 2019-10-13T10:40:05.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:05.661+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:05.662225+0000) 2019-10-13T10:40:05.668 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:05.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192489 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:08.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:08.556+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:08.557814+0000) 2019-10-13T10:40:08.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:08.893+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:08.894857+0000) 2019-10-13T10:40:10.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:10.667+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.668982+0000) 2019-10-13T10:40:10.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:10.669+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.670034+0000) 2019-10-13T10:40:10.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:10.668+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.669508+0000) 2019-10-13T10:40:10.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:10.667+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.669387+0000) 2019-10-13T10:40:10.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:10.668+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.669696+0000) 2019-10-13T10:40:10.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:10.668+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.669952+0000) 2019-10-13T10:40:10.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:10.668+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.670249+0000) 2019-10-13T10:40:10.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:10.672+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.672978+0000) 2019-10-13T10:40:10.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:10.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:11.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:11.507+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:11.509125+0000) 2019-10-13T10:40:12.099 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:40:12.099 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:12.110 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:40:12.110 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:12.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:12.997+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:12.998088+0000) 2019-10-13T10:40:13.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:12.999+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:12.999316+0000) 2019-10-13T10:40:14.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:14.742+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.743037+0000) 2019-10-13T10:40:15.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:15.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.649282+0000) 2019-10-13T10:40:15.650 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:15.648+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.649361+0000) 2019-10-13T10:40:15.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:15.679+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.679838+0000) 2019-10-13T10:40:15.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:15.677+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.679336+0000) 2019-10-13T10:40:15.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:15.678+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.679460+0000) 2019-10-13T10:40:15.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:15.678+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.679880+0000) 2019-10-13T10:40:15.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:15.678+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.679812+0000) 2019-10-13T10:40:15.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:15.678+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.679977+0000) 2019-10-13T10:40:15.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:15.678+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.680109+0000) 2019-10-13T10:40:15.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:15.681+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:15.682392+0000) 2019-10-13T10:40:15.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:15.684+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192549 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:18.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:18.557+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:18.557995+0000) 2019-10-13T10:40:18.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:18.894+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:18.895042+0000) 2019-10-13T10:40:20.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:20.686+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.688101+0000) 2019-10-13T10:40:20.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:20.687+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.687521+0000) 2019-10-13T10:40:20.691 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:20.687+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.687417+0000) 2019-10-13T10:40:20.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:20.687+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.687939+0000) 2019-10-13T10:40:20.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:20.687+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.688027+0000) 2019-10-13T10:40:20.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:20.687+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.688336+0000) 2019-10-13T10:40:20.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:20.688+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.688705+0000) 2019-10-13T10:40:20.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:20.690+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.692189+0000) 2019-10-13T10:40:20.697 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:20.695+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:21.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:21.508+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:21.509299+0000) 2019-10-13T10:40:22.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:22.997+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:22.998281+0000) 2019-10-13T10:40:23.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:22.998+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:22.999560+0000) 2019-10-13T10:40:24.745 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:24.744+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.745377+0000) 2019-10-13T10:40:25.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:25.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.649455+0000) 2019-10-13T10:40:25.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:25.649+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.649517+0000) 2019-10-13T10:40:25.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:25.697+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.697422+0000) 2019-10-13T10:40:25.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:25.696+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.698516+0000) 2019-10-13T10:40:25.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:25.697+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.697750+0000) 2019-10-13T10:40:25.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:25.697+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.698004+0000) 2019-10-13T10:40:25.700 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:25.697+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.698210+0000) 2019-10-13T10:40:25.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:25.698+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.698502+0000) 2019-10-13T10:40:25.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:25.698+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.698623+0000) 2019-10-13T10:40:25.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:25.700+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:25.701932+0000) 2019-10-13T10:40:25.706 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:25.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:28.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:28.557+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:28.558208+0000) 2019-10-13T10:40:28.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:28.895+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:28.895230+0000) 2019-10-13T10:40:30.708 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:30.706+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.707905+0000) 2019-10-13T10:40:30.709 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:30.706+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.707181+0000) 2019-10-13T10:40:30.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:30.707+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.707486+0000) 2019-10-13T10:40:30.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:30.707+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.707600+0000) 2019-10-13T10:40:30.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:30.707+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.707985+0000) 2019-10-13T10:40:30.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:30.707+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.708143+0000) 2019-10-13T10:40:30.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:30.708+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.708469+0000) 2019-10-13T10:40:30.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:30.709+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.710488+0000) 2019-10-13T10:40:30.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:30.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:31.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:31.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:31.509463+0000) 2019-10-13T10:40:32.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:32.998+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:32.998461+0000) 2019-10-13T10:40:33.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:32.999+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:32.999774+0000) 2019-10-13T10:40:34.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:34.746+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.747657+0000) 2019-10-13T10:40:35.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:35.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.649589+0000) 2019-10-13T10:40:35.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:35.649+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.649682+0000) 2019-10-13T10:40:35.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:35.716+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.716961+0000) 2019-10-13T10:40:35.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:35.716+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.717251+0000) 2019-10-13T10:40:35.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:35.716+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.717345+0000) 2019-10-13T10:40:35.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:35.717+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.717971+0000) 2019-10-13T10:40:35.719 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:35.717+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.717700+0000) 2019-10-13T10:40:35.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:35.717+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.718009+0000) 2019-10-13T10:40:35.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:35.717+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.718138+0000) 2019-10-13T10:40:35.720 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:35.719+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:35.720249+0000) 2019-10-13T10:40:35.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:35.724+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:38.559 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:38.557+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:38.558408+0000) 2019-10-13T10:40:38.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:38.894+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:38.895437+0000) 2019-10-13T10:40:40.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:40.726+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.727020+0000) 2019-10-13T10:40:40.728 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:40.724+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.726277+0000) 2019-10-13T10:40:40.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:40.725+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.726661+0000) 2019-10-13T10:40:40.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:40.725+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.726802+0000) 2019-10-13T10:40:40.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:40.725+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.726946+0000) 2019-10-13T10:40:40.729 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:40.725+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.727179+0000) 2019-10-13T10:40:40.729 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:40.725+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.727308+0000) 2019-10-13T10:40:40.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:40.729+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.730415+0000) 2019-10-13T10:40:40.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:40.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:41.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:41.508+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:41.509611+0000) 2019-10-13T10:40:42.264 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:40:42.264 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:42.268 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:40:42.268 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:43.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:42.997+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:42.998637+0000) 2019-10-13T10:40:43.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:42.999+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:43.000017+0000) 2019-10-13T10:40:44.750 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:44.749+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.749966+0000) 2019-10-13T10:40:45.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:45.648+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.649755+0000) 2019-10-13T10:40:45.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:45.648+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.649833+0000) 2019-10-13T10:40:45.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:45.735+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.736629+0000) 2019-10-13T10:40:45.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:45.734+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.736015+0000) 2019-10-13T10:40:45.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:45.734+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.736134+0000) 2019-10-13T10:40:45.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:45.734+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.736239+0000) 2019-10-13T10:40:45.738 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:45.735+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.736483+0000) 2019-10-13T10:40:45.738 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:45.735+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.736755+0000) 2019-10-13T10:40:45.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:45.735+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.737006+0000) 2019-10-13T10:40:45.739 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:45.737+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:45.738648+0000) 2019-10-13T10:40:45.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:45.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:48.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:48.557+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:48.558576+0000) 2019-10-13T10:40:48.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:48.894+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:48.895607+0000) 2019-10-13T10:40:50.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:50.742+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.744225+0000) 2019-10-13T10:40:50.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:50.742+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.744348+0000) 2019-10-13T10:40:50.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:50.745+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.746094+0000) 2019-10-13T10:40:50.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:50.743+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.744814+0000) 2019-10-13T10:40:50.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:50.743+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.744985+0000) 2019-10-13T10:40:50.747 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:50.743+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.745287+0000) 2019-10-13T10:40:50.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:50.744+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.745508+0000) 2019-10-13T10:40:50.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:50.746+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.747515+0000) 2019-10-13T10:40:50.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:50.750+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:51.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:51.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:51.509777+0000) 2019-10-13T10:40:53.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:52.998+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:52.998809+0000) 2019-10-13T10:40:53.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:52.999+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:53.000200+0000) 2019-10-13T10:40:54.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:54.751+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.752233+0000) 2019-10-13T10:40:55.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:55.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.649958+0000) 2019-10-13T10:40:55.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:55.649+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.649988+0000) 2019-10-13T10:40:55.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:40:55.754+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.754941+0000) 2019-10-13T10:40:55.755 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:55.752+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.753911+0000) 2019-10-13T10:40:55.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:40:55.752+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.754264+0000) 2019-10-13T10:40:55.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:55.752+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.754364+0000) 2019-10-13T10:40:55.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:40:55.753+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.754530+0000) 2019-10-13T10:40:55.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:40:55.753+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.754821+0000) 2019-10-13T10:40:55.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:40:55.753+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.755034+0000) 2019-10-13T10:40:55.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:40:55.757+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:55.757844+0000) 2019-10-13T10:40:55.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:40:55.760+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192792 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:40:58.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:40:58.558+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:58.558768+0000) 2019-10-13T10:40:58.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:40:58.895+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:58.895816+0000) 2019-10-13T10:41:00.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:00.763+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.764654+0000) 2019-10-13T10:41:00.766 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:00.762+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.764008+0000) 2019-10-13T10:41:00.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:00.762+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.764173+0000) 2019-10-13T10:41:00.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:00.762+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.764055+0000) 2019-10-13T10:41:00.766 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:00.763+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.764652+0000) 2019-10-13T10:41:00.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:00.763+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.764752+0000) 2019-10-13T10:41:00.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:00.763+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.765106+0000) 2019-10-13T10:41:00.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:00.766+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.767384+0000) 2019-10-13T10:41:00.773 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:00.770+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:01.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:01.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:01.509954+0000) 2019-10-13T10:41:03.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:02.998+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:02.998993+0000) 2019-10-13T10:41:03.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:03.000+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:03.000405+0000) 2019-10-13T10:41:04.755 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:04.754+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.754805+0000) 2019-10-13T10:41:05.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:05.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.650152+0000) 2019-10-13T10:41:05.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:05.649+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.650184+0000) 2019-10-13T10:41:05.774 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:05.773+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.774342+0000) 2019-10-13T10:41:05.775 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:05.773+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.773914+0000) 2019-10-13T10:41:05.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:05.773+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.774338+0000) 2019-10-13T10:41:05.776 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:05.773+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.774372+0000) 2019-10-13T10:41:05.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:05.774+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.774792+0000) 2019-10-13T10:41:05.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:05.774+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.774935+0000) 2019-10-13T10:41:05.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:05.774+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.775028+0000) 2019-10-13T10:41:05.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:05.775+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:05.776551+0000) 2019-10-13T10:41:05.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:05.781+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:08.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:08.558+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:08.558983+0000) 2019-10-13T10:41:08.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:08.895+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:08.896013+0000) 2019-10-13T10:41:10.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:10.782+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.782981+0000) 2019-10-13T10:41:10.784 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:10.782+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.783216+0000) 2019-10-13T10:41:10.785 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:10.783+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.783419+0000) 2019-10-13T10:41:10.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:10.782+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.784533+0000) 2019-10-13T10:41:10.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:10.783+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.783890+0000) 2019-10-13T10:41:10.786 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:10.783+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.783989+0000) 2019-10-13T10:41:10.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:10.783+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.784356+0000) 2019-10-13T10:41:10.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:10.785+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.786769+0000) 2019-10-13T10:41:10.791 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:10.789+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:11.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:11.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:11.510100+0000) 2019-10-13T10:41:12.434 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:41:12.435 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:12.439 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:41:12.439 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:13.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:12.999+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:13.000623+0000) 2019-10-13T10:41:13.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:12.998+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:12.999135+0000) 2019-10-13T10:41:14.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:14.756+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.757197+0000) 2019-10-13T10:41:15.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:15.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.650372+0000) 2019-10-13T10:41:15.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:15.650+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.650389+0000) 2019-10-13T10:41:15.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:15.790+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.792563+0000) 2019-10-13T10:41:15.793 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:15.791+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.792023+0000) 2019-10-13T10:41:15.793 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:15.791+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.792291+0000) 2019-10-13T10:41:15.794 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:15.791+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.792194+0000) 2019-10-13T10:41:15.794 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:15.792+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.792650+0000) 2019-10-13T10:41:15.794 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:15.792+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.792933+0000) 2019-10-13T10:41:15.794 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:15.792+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.793343+0000) 2019-10-13T10:41:15.796 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:15.794+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:15.796085+0000) 2019-10-13T10:41:15.800 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:15.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:18.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:18.559+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:18.559181+0000) 2019-10-13T10:41:18.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:18.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:18.896146+0000) 2019-10-13T10:41:20.803 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:20.800+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.801354+0000) 2019-10-13T10:41:20.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:20.800+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.802377+0000) 2019-10-13T10:41:20.804 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:20.801+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.801871+0000) 2019-10-13T10:41:20.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:20.801+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.801978+0000) 2019-10-13T10:41:20.804 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:20.801+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.802231+0000) 2019-10-13T10:41:20.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:20.801+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.802305+0000) 2019-10-13T10:41:20.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:20.802+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.802434+0000) 2019-10-13T10:41:20.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:20.802+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.804668+0000) 2019-10-13T10:41:20.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:20.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192941 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:21.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:21.510+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:21.510276+0000) 2019-10-13T10:41:23.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:22.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:22.999328+0000) 2019-10-13T10:41:23.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:23.000+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:23.000818+0000) 2019-10-13T10:41:24.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:24.758+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.759536+0000) 2019-10-13T10:41:25.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:25.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.650599+0000) 2019-10-13T10:41:25.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:25.649+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.650601+0000) 2019-10-13T10:41:25.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:25.808+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.810129+0000) 2019-10-13T10:41:25.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:25.808+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.810207+0000) 2019-10-13T10:41:25.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:25.809+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.810675+0000) 2019-10-13T10:41:25.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:25.809+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.810734+0000) 2019-10-13T10:41:25.812 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:25.809+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.810950+0000) 2019-10-13T10:41:25.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:25.810+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.811484+0000) 2019-10-13T10:41:25.813 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:25.810+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.811618+0000) 2019-10-13T10:41:25.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:25.813+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:25.814438+0000) 2019-10-13T10:41:25.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:25.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 192973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:28.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:28.558+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:28.559357+0000) 2019-10-13T10:41:28.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:28.895+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:28.896247+0000) 2019-10-13T10:41:30.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:30.820+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.821020+0000) 2019-10-13T10:41:30.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:30.818+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.820290+0000) 2019-10-13T10:41:30.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:30.819+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.820460+0000) 2019-10-13T10:41:30.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:30.819+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.820627+0000) 2019-10-13T10:41:30.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:30.819+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.820739+0000) 2019-10-13T10:41:30.823 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:30.819+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.821116+0000) 2019-10-13T10:41:30.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:30.820+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.821402+0000) 2019-10-13T10:41:30.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:30.822+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.822998+0000) 2019-10-13T10:41:30.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:30.826+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193002 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:31.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:31.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:31.510410+0000) 2019-10-13T10:41:33.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:32.998+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:32.999494+0000) 2019-10-13T10:41:33.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:33.000+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:33.001006+0000) 2019-10-13T10:41:34.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:34.761+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.761902+0000) 2019-10-13T10:41:35.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:35.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.650796+0000) 2019-10-13T10:41:35.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:35.650+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.650793+0000) 2019-10-13T10:41:35.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:35.830+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.831054+0000) 2019-10-13T10:41:35.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:35.829+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.830436+0000) 2019-10-13T10:41:35.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:35.829+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.830630+0000) 2019-10-13T10:41:35.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:35.829+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.830980+0000) 2019-10-13T10:41:35.833 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:35.829+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.831114+0000) 2019-10-13T10:41:35.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:35.829+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.831209+0000) 2019-10-13T10:41:35.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:35.830+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.831885+0000) 2019-10-13T10:41:35.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:35.833+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:35.834685+0000) 2019-10-13T10:41:35.839 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:35.837+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:38.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:38.558+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:38.559589+0000) 2019-10-13T10:41:38.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:38.895+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:38.896416+0000) 2019-10-13T10:41:40.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:40.839+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.840472+0000) 2019-10-13T10:41:40.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:40.839+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.840828+0000) 2019-10-13T10:41:40.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:40.841+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.842151+0000) 2019-10-13T10:41:40.843 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:40.839+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.841057+0000) 2019-10-13T10:41:40.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:40.840+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.841417+0000) 2019-10-13T10:41:40.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:40.840+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.841642+0000) 2019-10-13T10:41:40.843 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:40.840+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.841832+0000) 2019-10-13T10:41:40.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:40.842+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.843671+0000) 2019-10-13T10:41:40.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:40.846+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193061 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:41.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:41.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:41.510576+0000) 2019-10-13T10:41:42.531 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:41:42.531 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:42.535 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:41:42.535 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:43.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:42.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:42.999668+0000) 2019-10-13T10:41:43.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:43.000+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:43.001191+0000) 2019-10-13T10:41:44.764 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:44.763+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.764291+0000) 2019-10-13T10:41:45.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:45.649+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.651004+0000) 2019-10-13T10:41:45.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:45.649+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.650990+0000) 2019-10-13T10:41:45.851 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:45.848+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.849607+0000) 2019-10-13T10:41:45.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:45.848+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.849906+0000) 2019-10-13T10:41:45.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:45.848+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.850087+0000) 2019-10-13T10:41:45.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:45.850+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.851233+0000) 2019-10-13T10:41:45.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:45.848+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.850375+0000) 2019-10-13T10:41:45.853 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:45.849+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.850525+0000) 2019-10-13T10:41:45.853 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:45.849+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.851092+0000) 2019-10-13T10:41:45.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:45.852+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:45.853696+0000) 2019-10-13T10:41:45.859 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:45.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:48.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:48.559+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:48.559804+0000) 2019-10-13T10:41:48.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:48.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:48.896564+0000) 2019-10-13T10:41:50.860 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:50.858+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.859588+0000) 2019-10-13T10:41:50.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:50.860+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.860815+0000) 2019-10-13T10:41:50.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:50.858+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.860242+0000) 2019-10-13T10:41:50.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:50.858+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.860387+0000) 2019-10-13T10:41:50.862 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:50.859+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.860501+0000) 2019-10-13T10:41:50.862 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:50.859+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.860613+0000) 2019-10-13T10:41:50.863 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:50.859+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.860918+0000) 2019-10-13T10:41:50.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:50.862+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.863216+0000) 2019-10-13T10:41:50.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:50.867+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193122 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:51.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:51.510+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:51.510719+0000) 2019-10-13T10:41:53.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:52.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:52.999849+0000) 2019-10-13T10:41:53.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:53.000+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:53.001403+0000) 2019-10-13T10:41:54.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:54.766+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.766769+0000) 2019-10-13T10:41:55.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:55.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.651194+0000) 2019-10-13T10:41:55.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:55.650+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.651194+0000) 2019-10-13T10:41:55.871 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:41:55.869+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.870027+0000) 2019-10-13T10:41:55.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:41:55.870+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.871210+0000) 2019-10-13T10:41:55.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:41:55.870+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.870445+0000) 2019-10-13T10:41:55.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:55.870+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.870528+0000) 2019-10-13T10:41:55.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:55.870+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.870707+0000) 2019-10-13T10:41:55.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:41:55.870+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.870880+0000) 2019-10-13T10:41:55.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:41:55.870+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.871165+0000) 2019-10-13T10:41:55.876 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:41:55.873+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:55.874029+0000) 2019-10-13T10:41:55.879 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:41:55.877+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193154 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:41:58.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:41:58.559+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:58.559991+0000) 2019-10-13T10:41:58.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:41:58.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:58.896730+0000) 2019-10-13T10:42:00.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:00.879+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.879835+0000) 2019-10-13T10:42:00.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:00.879+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.879946+0000) 2019-10-13T10:42:00.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:00.879+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.880115+0000) 2019-10-13T10:42:00.882 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:00.879+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.880363+0000) 2019-10-13T10:42:00.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:00.879+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.881434+0000) 2019-10-13T10:42:00.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:00.880+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.882715+0000) 2019-10-13T10:42:00.883 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:00.880+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.880675+0000) 2019-10-13T10:42:00.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:00.880+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.880871+0000) 2019-10-13T10:42:00.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:00.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193181 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:01.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:01.510+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:01.510890+0000) 2019-10-13T10:42:03.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:02.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:03.000030+0000) 2019-10-13T10:42:03.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:03.000+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:03.001624+0000) 2019-10-13T10:42:04.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:04.768+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.769342+0000) 2019-10-13T10:42:05.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:05.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.651387+0000) 2019-10-13T10:42:05.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:05.651+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.651392+0000) 2019-10-13T10:42:05.890 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:05.888+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.889266+0000) 2019-10-13T10:42:05.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:05.889+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.890835+0000) 2019-10-13T10:42:05.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:05.888+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.889370+0000) 2019-10-13T10:42:05.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:05.889+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.889941+0000) 2019-10-13T10:42:05.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:05.889+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.889737+0000) 2019-10-13T10:42:05.892 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:05.889+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.890118+0000) 2019-10-13T10:42:05.892 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:05.889+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.890235+0000) 2019-10-13T10:42:05.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:05.892+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:05.894348+0000) 2019-10-13T10:42:05.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:05.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:08.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:08.559+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:08.560224+0000) 2019-10-13T10:42:08.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:08.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:08.896900+0000) 2019-10-13T10:42:10.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:10.897+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.898726+0000) 2019-10-13T10:42:10.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:10.897+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.898869+0000) 2019-10-13T10:42:10.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:10.897+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.899156+0000) 2019-10-13T10:42:10.901 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:10.897+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.899054+0000) 2019-10-13T10:42:10.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:10.899+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.900856+0000) 2019-10-13T10:42:10.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:10.898+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.899640+0000) 2019-10-13T10:42:10.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:10.898+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.900119+0000) 2019-10-13T10:42:10.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:10.901+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.903254+0000) 2019-10-13T10:42:10.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:10.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:11.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:11.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:11.511060+0000) 2019-10-13T10:42:12.702 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:42:12.702 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:12.706 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:42:12.706 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:13.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:12.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:13.000234+0000) 2019-10-13T10:42:13.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:13.001+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:13.001830+0000) 2019-10-13T10:42:14.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:14.771+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.771807+0000) 2019-10-13T10:42:15.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:15.650+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.651614+0000) 2019-10-13T10:42:15.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:15.650+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.651604+0000) 2019-10-13T10:42:15.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:15.905+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.906799+0000) 2019-10-13T10:42:15.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:15.907+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.908291+0000) 2019-10-13T10:42:15.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:15.906+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.907619+0000) 2019-10-13T10:42:15.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:15.906+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.907832+0000) 2019-10-13T10:42:15.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:15.906+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.907956+0000) 2019-10-13T10:42:15.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:15.906+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.908120+0000) 2019-10-13T10:42:15.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:15.906+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.908361+0000) 2019-10-13T10:42:15.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:15.909+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:15.910425+0000) 2019-10-13T10:42:15.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:15.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:18.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:18.559+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:18.560426+0000) 2019-10-13T10:42:18.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:18.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:18.897047+0000) 2019-10-13T10:42:20.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:20.915+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.916876+0000) 2019-10-13T10:42:20.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:20.915+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.917026+0000) 2019-10-13T10:42:20.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:20.915+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.917186+0000) 2019-10-13T10:42:20.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:20.915+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.917292+0000) 2019-10-13T10:42:20.919 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:20.916+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.917505+0000) 2019-10-13T10:42:20.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:20.918+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.918921+0000) 2019-10-13T10:42:20.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:20.916+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.917718+0000) 2019-10-13T10:42:20.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:20.920+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.921421+0000) 2019-10-13T10:42:20.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:20.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193305 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:21.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:21.509+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:21.511244+0000) 2019-10-13T10:42:23.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:22.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:23.000401+0000) 2019-10-13T10:42:23.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:23.001+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:23.002023+0000) 2019-10-13T10:42:24.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:24.773+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.774036+0000) 2019-10-13T10:42:25.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:25.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.651785+0000) 2019-10-13T10:42:25.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:25.651+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.651783+0000) 2019-10-13T10:42:25.927 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:25.924+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.926346+0000) 2019-10-13T10:42:25.928 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:25.926+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.927433+0000) 2019-10-13T10:42:25.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:25.925+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.926882+0000) 2019-10-13T10:42:25.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:25.925+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.927210+0000) 2019-10-13T10:42:25.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:25.925+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.927299+0000) 2019-10-13T10:42:25.929 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:25.926+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.927603+0000) 2019-10-13T10:42:25.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:25.926+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.927733+0000) 2019-10-13T10:42:25.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:25.929+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:25.930093+0000) 2019-10-13T10:42:25.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:25.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193341 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:28.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:28.560+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:28.560621+0000) 2019-10-13T10:42:28.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:28.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:28.897173+0000) 2019-10-13T10:42:30.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:30.935+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.936630+0000) 2019-10-13T10:42:30.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:30.935+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.936953+0000) 2019-10-13T10:42:30.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:30.937+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.938159+0000) 2019-10-13T10:42:30.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:30.936+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.937404+0000) 2019-10-13T10:42:30.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:30.936+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.937499+0000) 2019-10-13T10:42:30.939 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:30.936+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.937896+0000) 2019-10-13T10:42:30.940 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:30.936+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.938178+0000) 2019-10-13T10:42:30.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:30.940+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.940783+0000) 2019-10-13T10:42:30.946 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:30.943+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:31.512 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:31.510+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:31.511384+0000) 2019-10-13T10:42:33.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:32.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:33.000550+0000) 2019-10-13T10:42:33.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:33.001+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:33.002197+0000) 2019-10-13T10:42:34.776 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:34.775+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.776341+0000) 2019-10-13T10:42:35.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:35.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.651987+0000) 2019-10-13T10:42:35.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:35.651+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.651982+0000) 2019-10-13T10:42:35.948 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:35.945+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.947073+0000) 2019-10-13T10:42:35.949 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:35.945+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.946999+0000) 2019-10-13T10:42:35.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:35.945+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.947300+0000) 2019-10-13T10:42:35.949 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:35.946+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.947516+0000) 2019-10-13T10:42:35.950 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:35.947+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.948035+0000) 2019-10-13T10:42:35.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:35.949+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.949771+0000) 2019-10-13T10:42:35.950 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:35.946+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.947825+0000) 2019-10-13T10:42:35.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:35.946+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:35.948032+0000) 2019-10-13T10:42:35.957 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:35.954+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193402 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:38.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:38.560+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:38.560835+0000) 2019-10-13T10:42:38.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:38.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:38.897354+0000) 2019-10-13T10:42:40.957 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:40.956+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.957136+0000) 2019-10-13T10:42:40.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:40.956+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.956461+0000) 2019-10-13T10:42:40.958 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:40.956+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.956545+0000) 2019-10-13T10:42:40.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:40.956+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.956674+0000) 2019-10-13T10:42:40.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:40.956+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.956746+0000) 2019-10-13T10:42:40.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:40.956+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.956925+0000) 2019-10-13T10:42:40.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:40.956+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.957262+0000) 2019-10-13T10:42:40.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:40.959+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.960197+0000) 2019-10-13T10:42:40.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:40.962+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:41.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:41.511+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:41.511553+0000) 2019-10-13T10:42:42.872 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:42:42.873 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:42.877 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:42:42.877 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:43.001 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:43.000+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:43.000686+0000) 2019-10-13T10:42:43.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:43.001+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:43.002385+0000) 2019-10-13T10:42:44.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:44.778+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.778679+0000) 2019-10-13T10:42:45.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:45.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.652185+0000) 2019-10-13T10:42:45.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:45.651+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.652185+0000) 2019-10-13T10:42:45.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:45.964+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.964512+0000) 2019-10-13T10:42:45.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:45.964+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.964681+0000) 2019-10-13T10:42:45.967 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:45.965+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.965763+0000) 2019-10-13T10:42:45.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:45.965+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.965591+0000) 2019-10-13T10:42:45.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:45.965+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.965719+0000) 2019-10-13T10:42:45.967 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:45.965+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.965864+0000) 2019-10-13T10:42:45.967 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:45.965+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.966146+0000) 2019-10-13T10:42:45.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:45.967+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:45.967916+0000) 2019-10-13T10:42:45.974 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:45.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193465 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:48.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:48.560+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:48.561044+0000) 2019-10-13T10:42:48.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:48.897+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:48.897530+0000) 2019-10-13T10:42:50.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:50.974+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.974472+0000) 2019-10-13T10:42:50.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:50.973+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.975718+0000) 2019-10-13T10:42:50.976 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:50.974+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.975079+0000) 2019-10-13T10:42:50.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:50.974+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.975147+0000) 2019-10-13T10:42:50.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:50.974+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.975352+0000) 2019-10-13T10:42:50.977 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:50.975+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.975688+0000) 2019-10-13T10:42:50.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:50.975+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.976014+0000) 2019-10-13T10:42:50.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:50.977+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.978819+0000) 2019-10-13T10:42:50.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:50.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193492 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:51.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:51.511+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:51.511726+0000) 2019-10-13T10:42:53.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:53.000+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:53.000850+0000) 2019-10-13T10:42:53.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:53.001+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:53.002576+0000) 2019-10-13T10:42:54.781 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:54.779+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.781064+0000) 2019-10-13T10:42:55.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:55.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.652378+0000) 2019-10-13T10:42:55.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:55.651+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.652378+0000) 2019-10-13T10:42:55.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:42:55.982+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.983606+0000) 2019-10-13T10:42:55.985 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:42:55.982+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.983840+0000) 2019-10-13T10:42:55.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:42:55.982+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.983993+0000) 2019-10-13T10:42:55.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:55.982+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.984170+0000) 2019-10-13T10:42:55.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:55.982+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.984280+0000) 2019-10-13T10:42:55.986 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:42:55.984+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.985757+0000) 2019-10-13T10:42:55.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:42:55.983+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.984708+0000) 2019-10-13T10:42:55.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:42:55.985+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:55.987416+0000) 2019-10-13T10:42:55.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:42:55.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:42:58.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:42:58.560+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:58.561215+0000) 2019-10-13T10:42:58.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:42:58.896+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:58.897638+0000) 2019-10-13T10:43:00.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:00.990+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.991558+0000) 2019-10-13T10:43:00.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:00.990+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.991867+0000) 2019-10-13T10:43:00.994 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:00.990+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.992277+0000) 2019-10-13T10:43:00.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:00.990+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.992345+0000) 2019-10-13T10:43:00.994 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:00.991+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.992618+0000) 2019-10-13T10:43:00.994 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:00.991+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.993067+0000) 2019-10-13T10:43:00.995 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:00.991+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.993698+0000) 2019-10-13T10:43:00.996 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:00.994+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.996195+0000) 2019-10-13T10:43:01.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:00.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:01.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:01.510+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:01.511893+0000) 2019-10-13T10:43:03.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:03.000+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:03.001034+0000) 2019-10-13T10:43:03.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:03.001+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:03.002763+0000) 2019-10-13T10:43:04.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:04.782+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.783622+0000) 2019-10-13T10:43:05.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:05.651+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:05.652568+0000) 2019-10-13T10:43:05.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:05.651+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:05.652569+0000) 2019-10-13T10:43:06.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:06.000+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.001504+0000) 2019-10-13T10:43:06.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:06.000+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.001715+0000) 2019-10-13T10:43:06.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:06.000+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.002188+0000) 2019-10-13T10:43:06.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:06.000+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.002367+0000) 2019-10-13T10:43:06.004 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:06.001+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.002772+0000) 2019-10-13T10:43:06.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:06.001+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.003118+0000) 2019-10-13T10:43:06.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:06.002+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.003515+0000) 2019-10-13T10:43:06.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:06.004+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:06.005550+0000) 2019-10-13T10:43:06.011 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:06.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:08.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:08.560+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:08.561373+0000) 2019-10-13T10:43:08.899 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:08.897+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:08.897807+0000) 2019-10-13T10:43:11.013 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:11.009+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.011371+0000) 2019-10-13T10:43:11.013 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:11.010+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.011687+0000) 2019-10-13T10:43:11.013 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:11.010+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.011764+0000) 2019-10-13T10:43:11.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:11.012+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.012963+0000) 2019-10-13T10:43:11.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:11.011+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.012454+0000) 2019-10-13T10:43:11.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:11.011+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.012740+0000) 2019-10-13T10:43:11.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:11.011+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.012930+0000) 2019-10-13T10:43:11.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:11.014+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.015280+0000) 2019-10-13T10:43:11.021 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:11.019+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193616 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:11.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:11.511+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.512068+0000) 2019-10-13T10:43:12.976 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:43:12.976 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:12.984 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:43:12.984 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:13.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:12.999+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:13.001194+0000) 2019-10-13T10:43:13.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:13.002+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:13.002904+0000) 2019-10-13T10:43:14.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:14.785+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.786108+0000) 2019-10-13T10:43:15.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:15.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:15.652793+0000) 2019-10-13T10:43:15.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:15.652+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:15.652792+0000) 2019-10-13T10:43:16.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:16.022+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.023382+0000) 2019-10-13T10:43:16.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:16.021+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.022677+0000) 2019-10-13T10:43:16.024 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:16.021+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.022883+0000) 2019-10-13T10:43:16.024 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:16.021+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.023295+0000) 2019-10-13T10:43:16.025 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:16.022+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.023413+0000) 2019-10-13T10:43:16.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:16.022+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.023920+0000) 2019-10-13T10:43:16.025 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:16.022+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.023532+0000) 2019-10-13T10:43:16.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:16.026+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:16.027442+0000) 2019-10-13T10:43:16.032 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:16.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193650 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:18.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:18.561+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:18.561631+0000) 2019-10-13T10:43:18.899 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:18.897+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:18.897996+0000) 2019-10-13T10:43:21.034 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:21.030+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.032132+0000) 2019-10-13T10:43:21.034 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:21.031+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.032428+0000) 2019-10-13T10:43:21.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:21.031+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.032722+0000) 2019-10-13T10:43:21.034 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:21.031+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.032908+0000) 2019-10-13T10:43:21.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:21.033+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.033955+0000) 2019-10-13T10:43:21.035 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:21.031+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.033221+0000) 2019-10-13T10:43:21.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:21.032+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.033421+0000) 2019-10-13T10:43:21.036 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:21.034+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.035284+0000) 2019-10-13T10:43:21.040 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:21.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193677 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:21.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:21.511+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.512215+0000) 2019-10-13T10:43:23.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:23.000+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:23.001380+0000) 2019-10-13T10:43:23.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:23.002+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:23.003078+0000) 2019-10-13T10:43:24.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:24.788+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.788785+0000) 2019-10-13T10:43:25.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:25.652+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:25.652961+0000) 2019-10-13T10:43:25.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:25.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:25.653012+0000) 2019-10-13T10:43:26.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:26.041+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.041794+0000) 2019-10-13T10:43:26.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:26.043+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.044523+0000) 2019-10-13T10:43:26.046 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:26.040+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.040734+0000) 2019-10-13T10:43:26.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:26.040+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.041032+0000) 2019-10-13T10:43:26.046 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:26.040+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.041243+0000) 2019-10-13T10:43:26.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:26.040+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.041350+0000) 2019-10-13T10:43:26.047 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:26.041+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.041542+0000) 2019-10-13T10:43:26.047 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:26.041+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:26.041707+0000) 2019-10-13T10:43:26.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:26.047+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:28.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:28.561+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:28.561842+0000) 2019-10-13T10:43:28.899 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:28.897+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:28.898152+0000) 2019-10-13T10:43:31.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:31.049+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.049981+0000) 2019-10-13T10:43:31.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:31.049+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.050168+0000) 2019-10-13T10:43:31.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:31.049+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.050367+0000) 2019-10-13T10:43:31.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:31.051+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.052324+0000) 2019-10-13T10:43:31.053 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:31.050+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.050868+0000) 2019-10-13T10:43:31.053 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:31.050+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.051057+0000) 2019-10-13T10:43:31.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:31.050+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.051333+0000) 2019-10-13T10:43:31.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:31.053+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.054407+0000) 2019-10-13T10:43:31.059 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:31.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193740 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:31.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:31.512+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.512388+0000) 2019-10-13T10:43:33.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:33.001+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:33.001576+0000) 2019-10-13T10:43:33.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:33.003+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:33.003301+0000) 2019-10-13T10:43:34.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:34.791+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.791308+0000) 2019-10-13T10:43:35.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:35.652+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:35.653126+0000) 2019-10-13T10:43:35.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:35.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:35.653171+0000) 2019-10-13T10:43:36.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:36.059+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.060375+0000) 2019-10-13T10:43:36.061 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:36.058+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.059389+0000) 2019-10-13T10:43:36.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:36.059+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.059838+0000) 2019-10-13T10:43:36.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:36.059+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.060019+0000) 2019-10-13T10:43:36.062 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:36.059+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.060156+0000) 2019-10-13T10:43:36.062 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:36.059+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.060261+0000) 2019-10-13T10:43:36.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:36.060+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.060849+0000) 2019-10-13T10:43:36.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:36.062+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:36.063722+0000) 2019-10-13T10:43:36.069 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:36.067+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193774 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:38.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:38.561+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:38.562055+0000) 2019-10-13T10:43:38.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:38.898+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:38.898328+0000) 2019-10-13T10:43:41.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:41.068+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.070065+0000) 2019-10-13T10:43:41.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:41.070+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.071666+0000) 2019-10-13T10:43:41.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:41.069+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.070636+0000) 2019-10-13T10:43:41.072 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:41.069+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.070948+0000) 2019-10-13T10:43:41.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:41.069+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.071073+0000) 2019-10-13T10:43:41.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:41.069+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.071201+0000) 2019-10-13T10:43:41.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:41.068+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.070319+0000) 2019-10-13T10:43:41.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:41.073+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.074176+0000) 2019-10-13T10:43:41.080 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:41.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:41.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:41.511+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.512581+0000) 2019-10-13T10:43:43.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:43.000+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:43.001765+0000) 2019-10-13T10:43:43.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:43.002+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:43.003521+0000) 2019-10-13T10:43:43.156 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:43:43.157 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:43.161 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:43:43.161 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:44.794 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:44.792+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.793901+0000) 2019-10-13T10:43:45.654 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:45.652+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.653319+0000) 2019-10-13T10:43:45.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:45.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.653338+0000) 2019-10-13T10:43:46.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:46.078+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.079493+0000) 2019-10-13T10:43:46.081 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:46.078+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.079581+0000) 2019-10-13T10:43:46.081 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:46.078+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.079825+0000) 2019-10-13T10:43:46.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:46.078+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.079717+0000) 2019-10-13T10:43:46.082 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:46.078+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.080073+0000) 2019-10-13T10:43:46.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:46.079+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.081347+0000) 2019-10-13T10:43:46.082 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:46.079+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.080763+0000) 2019-10-13T10:43:46.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:46.081+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:46.083017+0000) 2019-10-13T10:43:46.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:46.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:48.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:48.561+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:48.562237+0000) 2019-10-13T10:43:48.899 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:48.897+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:48.898518+0000) 2019-10-13T10:43:51.091 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:51.088+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.089555+0000) 2019-10-13T10:43:51.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:51.088+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.089417+0000) 2019-10-13T10:43:51.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:51.088+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.089750+0000) 2019-10-13T10:43:51.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:51.088+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.090160+0000) 2019-10-13T10:43:51.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:51.089+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.091254+0000) 2019-10-13T10:43:51.092 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:51.089+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.090580+0000) 2019-10-13T10:43:51.093 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:51.089+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.090783+0000) 2019-10-13T10:43:51.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:51.090+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.092524+0000) 2019-10-13T10:43:51.098 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:51.095+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:51.514 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:51.511+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.512786+0000) 2019-10-13T10:43:53.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:53.001+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:53.001914+0000) 2019-10-13T10:43:53.004 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:53.002+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:53.003700+0000) 2019-10-13T10:43:54.796 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:54.795+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.796172+0000) 2019-10-13T10:43:55.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:55.652+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.653554+0000) 2019-10-13T10:43:55.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:55.652+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.653535+0000) 2019-10-13T10:43:56.100 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:56.097+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.098955+0000) 2019-10-13T10:43:56.100 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:43:56.097+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.099229+0000) 2019-10-13T10:43:56.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:43:56.097+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.099386+0000) 2019-10-13T10:43:56.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:43:56.100+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.100926+0000) 2019-10-13T10:43:56.102 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:43:56.098+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.099824+0000) 2019-10-13T10:43:56.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:43:56.099+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.100451+0000) 2019-10-13T10:43:56.102 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:56.099+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.100539+0000) 2019-10-13T10:43:56.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:43:56.101+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:56.102490+0000) 2019-10-13T10:43:56.109 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:43:56.106+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:43:58.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:43:58.561+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:58.562446+0000) 2019-10-13T10:43:58.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:43:58.898+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:58.898707+0000) 2019-10-13T10:44:01.111 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:01.108+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.109476+0000) 2019-10-13T10:44:01.111 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:01.108+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.109760+0000) 2019-10-13T10:44:01.111 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:01.108+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.109815+0000) 2019-10-13T10:44:01.111 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:01.108+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.109984+0000) 2019-10-13T10:44:01.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:01.110+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.111655+0000) 2019-10-13T10:44:01.112 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:01.109+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.110451+0000) 2019-10-13T10:44:01.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:01.109+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.110670+0000) 2019-10-13T10:44:01.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:01.112+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.113174+0000) 2019-10-13T10:44:01.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:01.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193925 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:01.514 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:01.512+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.512955+0000) 2019-10-13T10:44:03.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:03.001+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:03.002088+0000) 2019-10-13T10:44:03.004 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:03.003+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:03.003908+0000) 2019-10-13T10:44:04.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:04.797+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.798223+0000) 2019-10-13T10:44:05.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:05.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.653780+0000) 2019-10-13T10:44:05.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:05.653+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.653753+0000) 2019-10-13T10:44:06.120 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:06.117+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.118836+0000) 2019-10-13T10:44:06.121 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:06.117+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.118989+0000) 2019-10-13T10:44:06.121 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:06.118+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.119531+0000) 2019-10-13T10:44:06.121 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:06.118+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.119621+0000) 2019-10-13T10:44:06.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:06.119+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.120038+0000) 2019-10-13T10:44:06.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:06.118+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.119781+0000) 2019-10-13T10:44:06.122 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:06.118+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.120093+0000) 2019-10-13T10:44:06.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:06.122+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:06.123086+0000) 2019-10-13T10:44:06.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:06.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193961 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:08.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:08.562+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:08.562634+0000) 2019-10-13T10:44:08.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:08.898+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:08.898901+0000) 2019-10-13T10:44:11.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:11.128+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.128708+0000) 2019-10-13T10:44:11.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:11.128+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.128823+0000) 2019-10-13T10:44:11.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:11.128+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.129153+0000) 2019-10-13T10:44:11.131 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:11.129+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.130637+0000) 2019-10-13T10:44:11.131 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:11.128+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.129235+0000) 2019-10-13T10:44:11.131 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:11.128+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.129368+0000) 2019-10-13T10:44:11.132 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:11.129+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.129915+0000) 2019-10-13T10:44:11.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:11.132+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.133670+0000) 2019-10-13T10:44:11.137 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:11.136+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 193988 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:11.514 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:11.512+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.513130+0000) 2019-10-13T10:44:13.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:13.001+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:13.002305+0000) 2019-10-13T10:44:13.004 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:13.003+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:13.004110+0000) 2019-10-13T10:44:13.337 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:44:13.337 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:13.342 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:44:13.342 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:14.801 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:14.800+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.800513+0000) 2019-10-13T10:44:15.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:15.653+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.653905+0000) 2019-10-13T10:44:15.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:15.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.653941+0000) 2019-10-13T10:44:16.139 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:16.137+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.138031+0000) 2019-10-13T10:44:16.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:16.137+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.138139+0000) 2019-10-13T10:44:16.140 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:16.139+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.139902+0000) 2019-10-13T10:44:16.141 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:16.138+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.139275+0000) 2019-10-13T10:44:16.141 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:16.138+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.139180+0000) 2019-10-13T10:44:16.141 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:16.139+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.139441+0000) 2019-10-13T10:44:16.141 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:16.139+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.139926+0000) 2019-10-13T10:44:16.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:16.141+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:16.142675+0000) 2019-10-13T10:44:16.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:16.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194022 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:18.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:18.562+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:18.562770+0000) 2019-10-13T10:44:18.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:18.898+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:18.899085+0000) 2019-10-13T10:44:21.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:21.147+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.147622+0000) 2019-10-13T10:44:21.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:21.147+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.147958+0000) 2019-10-13T10:44:21.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:21.147+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.148078+0000) 2019-10-13T10:44:21.150 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:21.147+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.148170+0000) 2019-10-13T10:44:21.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:21.147+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.148290+0000) 2019-10-13T10:44:21.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:21.148+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.148485+0000) 2019-10-13T10:44:21.150 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:21.147+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.148670+0000) 2019-10-13T10:44:21.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:21.150+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.151267+0000) 2019-10-13T10:44:21.155 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:21.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:21.514 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:21.513+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.513304+0000) 2019-10-13T10:44:23.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:23.002+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:23.002486+0000) 2019-10-13T10:44:23.004 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:23.004+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:23.004307+0000) 2019-10-13T10:44:24.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:24.802+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.803012+0000) 2019-10-13T10:44:25.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:25.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.654075+0000) 2019-10-13T10:44:25.655 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:25.654+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.654068+0000) 2019-10-13T10:44:26.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:26.155+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.155494+0000) 2019-10-13T10:44:26.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:26.155+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.155821+0000) 2019-10-13T10:44:26.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:26.156+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.156836+0000) 2019-10-13T10:44:26.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:26.155+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.156035+0000) 2019-10-13T10:44:26.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:26.156+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.156440+0000) 2019-10-13T10:44:26.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:26.156+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.156522+0000) 2019-10-13T10:44:26.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:26.156+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.156582+0000) 2019-10-13T10:44:26.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:26.158+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:26.158887+0000) 2019-10-13T10:44:26.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:26.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:28.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:28.561+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:28.562944+0000) 2019-10-13T10:44:28.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:28.897+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:28.899215+0000) 2019-10-13T10:44:31.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:31.163+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.164496+0000) 2019-10-13T10:44:31.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:31.163+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.164735+0000) 2019-10-13T10:44:31.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:31.163+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.164870+0000) 2019-10-13T10:44:31.167 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:31.164+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.165481+0000) 2019-10-13T10:44:31.167 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:31.163+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.165285+0000) 2019-10-13T10:44:31.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:31.164+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.165403+0000) 2019-10-13T10:44:31.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:31.164+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.165597+0000) 2019-10-13T10:44:31.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:31.166+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.167347+0000) 2019-10-13T10:44:31.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:31.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:31.514 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:31.512+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.513477+0000) 2019-10-13T10:44:33.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:33.001+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:33.002642+0000) 2019-10-13T10:44:33.004 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:33.004+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:33.004501+0000) 2019-10-13T10:44:34.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:34.804+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.805364+0000) 2019-10-13T10:44:35.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:35.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.654254+0000) 2019-10-13T10:44:35.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:35.653+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.654254+0000) 2019-10-13T10:44:36.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:36.172+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.174141+0000) 2019-10-13T10:44:36.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:36.172+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.174326+0000) 2019-10-13T10:44:36.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:36.173+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.174576+0000) 2019-10-13T10:44:36.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:36.173+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.174633+0000) 2019-10-13T10:44:36.177 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:36.173+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.175669+0000) 2019-10-13T10:44:36.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:36.174+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.176604+0000) 2019-10-13T10:44:36.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:36.173+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.175159+0000) 2019-10-13T10:44:36.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:36.174+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:36.175443+0000) 2019-10-13T10:44:36.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:36.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194146 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:38.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:38.562+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:38.563140+0000) 2019-10-13T10:44:38.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:38.898+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:38.899380+0000) 2019-10-13T10:44:41.188 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:41.184+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.186312+0000) 2019-10-13T10:44:41.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:41.182+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.184162+0000) 2019-10-13T10:44:41.188 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:41.182+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.184367+0000) 2019-10-13T10:44:41.189 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:41.183+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.184476+0000) 2019-10-13T10:44:41.189 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:41.183+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.184707+0000) 2019-10-13T10:44:41.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:41.183+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.185063+0000) 2019-10-13T10:44:41.189 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:41.183+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.185216+0000) 2019-10-13T10:44:41.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:41.185+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.187601+0000) 2019-10-13T10:44:41.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:41.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194174 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:41.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:41.512+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.513657+0000) 2019-10-13T10:44:43.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:43.001+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:43.002844+0000) 2019-10-13T10:44:43.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:43.002+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:43.004691+0000) 2019-10-13T10:44:43.524 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:44:43.524 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:43.532 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:44:43.533 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:44.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:44.806+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.807671+0000) 2019-10-13T10:44:45.655 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:45.653+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.654419+0000) 2019-10-13T10:44:45.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:45.653+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.654461+0000) 2019-10-13T10:44:46.195 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:46.192+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.194250+0000) 2019-10-13T10:44:46.196 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:46.193+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.195616+0000) 2019-10-13T10:44:46.196 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:46.193+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.194956+0000) 2019-10-13T10:44:46.197 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:46.194+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.195556+0000) 2019-10-13T10:44:46.197 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:46.194+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.195729+0000) 2019-10-13T10:44:46.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:46.194+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.196032+0000) 2019-10-13T10:44:46.198 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:46.194+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.196194+0000) 2019-10-13T10:44:46.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:46.196+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:46.198064+0000) 2019-10-13T10:44:46.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:46.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:48.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:48.562+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:48.563335+0000) 2019-10-13T10:44:48.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:48.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:48.899555+0000) 2019-10-13T10:44:51.206 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:51.203+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.204885+0000) 2019-10-13T10:44:51.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:51.203+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.205203+0000) 2019-10-13T10:44:51.206 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:51.203+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.205334+0000) 2019-10-13T10:44:51.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:51.205+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.206399+0000) 2019-10-13T10:44:51.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:51.204+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.205642+0000) 2019-10-13T10:44:51.208 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:51.204+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.205874+0000) 2019-10-13T10:44:51.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:51.204+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.206019+0000) 2019-10-13T10:44:51.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:51.206+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.207531+0000) 2019-10-13T10:44:51.214 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:51.211+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194236 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:51.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:51.513+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.513829+0000) 2019-10-13T10:44:53.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:53.002+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:53.002968+0000) 2019-10-13T10:44:53.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:53.004+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:53.004899+0000) 2019-10-13T10:44:54.810 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:54.809+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.810273+0000) 2019-10-13T10:44:55.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:55.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.654598+0000) 2019-10-13T10:44:55.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:55.654+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.654619+0000) 2019-10-13T10:44:56.215 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:44:56.214+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.214484+0000) 2019-10-13T10:44:56.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:44:56.215+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.216001+0000) 2019-10-13T10:44:56.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:56.214+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.215206+0000) 2019-10-13T10:44:56.217 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:44:56.214+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.215372+0000) 2019-10-13T10:44:56.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:44:56.215+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.215545+0000) 2019-10-13T10:44:56.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:44:56.215+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.215790+0000) 2019-10-13T10:44:56.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:44:56.217+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.217921+0000) 2019-10-13T10:44:56.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:56.215+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:56.216058+0000) 2019-10-13T10:44:56.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:44:56.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:44:58.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:44:58.563+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:58.563511+0000) 2019-10-13T10:44:58.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:44:58.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:58.899648+0000) 2019-10-13T10:45:01.225 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:01.223+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.224343+0000) 2019-10-13T10:45:01.225 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:01.224+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.224678+0000) 2019-10-13T10:45:01.226 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:01.224+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.225687+0000) 2019-10-13T10:45:01.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:01.224+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.224989+0000) 2019-10-13T10:45:01.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:01.224+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.225224+0000) 2019-10-13T10:45:01.227 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:01.225+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.225393+0000) 2019-10-13T10:45:01.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:01.225+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.225586+0000) 2019-10-13T10:45:01.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:01.227+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.228512+0000) 2019-10-13T10:45:01.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:01.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:01.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:01.513+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.514027+0000) 2019-10-13T10:45:03.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:03.002+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:03.003101+0000) 2019-10-13T10:45:03.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:03.004+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:03.005089+0000) 2019-10-13T10:45:04.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:04.812+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.812599+0000) 2019-10-13T10:45:05.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:05.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.654803+0000) 2019-10-13T10:45:05.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:05.654+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.654811+0000) 2019-10-13T10:45:06.234 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:06.232+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.232997+0000) 2019-10-13T10:45:06.234 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:06.232+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.233047+0000) 2019-10-13T10:45:06.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:06.233+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.234468+0000) 2019-10-13T10:45:06.235 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:06.233+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.233872+0000) 2019-10-13T10:45:06.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:06.233+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.234088+0000) 2019-10-13T10:45:06.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:06.233+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.234180+0000) 2019-10-13T10:45:06.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:06.233+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.234335+0000) 2019-10-13T10:45:06.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:06.235+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:06.236330+0000) 2019-10-13T10:45:06.242 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:06.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:08.564 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:08.563+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:08.563685+0000) 2019-10-13T10:45:08.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:08.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:08.899816+0000) 2019-10-13T10:45:11.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:11.242+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.242704+0000) 2019-10-13T10:45:11.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:11.242+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.242924+0000) 2019-10-13T10:45:11.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:11.242+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.243064+0000) 2019-10-13T10:45:11.245 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:11.243+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.244123+0000) 2019-10-13T10:45:11.245 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:11.242+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.243358+0000) 2019-10-13T10:45:11.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:11.243+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.244140+0000) 2019-10-13T10:45:11.246 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:11.243+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.244344+0000) 2019-10-13T10:45:11.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:11.246+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.246981+0000) 2019-10-13T10:45:11.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:11.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194360 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:11.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:11.514+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.514193+0000) 2019-10-13T10:45:13.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:13.002+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:13.003245+0000) 2019-10-13T10:45:13.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:13.005+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:13.005290+0000) 2019-10-13T10:45:13.853 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:45:13.854 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:13.858 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:45:13.858 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:14.815 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:14.814+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.815005+0000) 2019-10-13T10:45:15.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:15.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.655033+0000) 2019-10-13T10:45:15.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:15.654+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.655032+0000) 2019-10-13T10:45:16.252 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:16.249+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.251227+0000) 2019-10-13T10:45:16.253 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:16.250+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.251706+0000) 2019-10-13T10:45:16.253 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:16.250+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.251902+0000) 2019-10-13T10:45:16.253 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:16.250+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.252049+0000) 2019-10-13T10:45:16.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:16.250+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.252081+0000) 2019-10-13T10:45:16.254 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:16.251+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.252459+0000) 2019-10-13T10:45:16.254 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:16.252+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.253141+0000) 2019-10-13T10:45:16.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:16.253+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:16.254438+0000) 2019-10-13T10:45:16.260 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:16.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194394 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:18.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:18.562+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:18.563888+0000) 2019-10-13T10:45:18.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:18.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:18.900010+0000) 2019-10-13T10:45:21.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:21.259+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.260715+0000) 2019-10-13T10:45:21.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:21.259+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.260767+0000) 2019-10-13T10:45:21.262 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:21.259+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.261052+0000) 2019-10-13T10:45:21.263 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:21.261+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.262170+0000) 2019-10-13T10:45:21.263 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:21.259+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.261388+0000) 2019-10-13T10:45:21.264 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:21.260+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.261518+0000) 2019-10-13T10:45:21.264 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:21.260+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.261655+0000) 2019-10-13T10:45:21.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:21.264+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.264788+0000) 2019-10-13T10:45:21.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:21.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194423 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:21.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:21.513+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.514363+0000) 2019-10-13T10:45:23.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:23.002+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:23.003417+0000) 2019-10-13T10:45:23.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:23.005+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:23.005474+0000) 2019-10-13T10:45:24.817 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:24.816+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.817537+0000) 2019-10-13T10:45:25.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:25.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.655239+0000) 2019-10-13T10:45:25.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:25.654+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.655239+0000) 2019-10-13T10:45:26.270 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:26.267+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.268884+0000) 2019-10-13T10:45:26.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:26.268+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.269424+0000) 2019-10-13T10:45:26.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:26.268+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.269544+0000) 2019-10-13T10:45:26.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:26.268+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.269805+0000) 2019-10-13T10:45:26.271 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:26.268+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.269952+0000) 2019-10-13T10:45:26.272 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:26.269+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.271211+0000) 2019-10-13T10:45:26.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:26.270+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.271922+0000) 2019-10-13T10:45:26.272 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:26.268+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:26.270165+0000) 2019-10-13T10:45:26.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:26.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:28.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:28.563+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:28.564045+0000) 2019-10-13T10:45:28.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:28.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:28.900147+0000) 2019-10-13T10:45:31.280 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:31.278+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.279393+0000) 2019-10-13T10:45:31.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:31.278+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.279723+0000) 2019-10-13T10:45:31.281 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:31.278+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.279841+0000) 2019-10-13T10:45:31.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:31.279+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.281180+0000) 2019-10-13T10:45:31.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:31.278+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.280174+0000) 2019-10-13T10:45:31.282 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:31.279+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.280572+0000) 2019-10-13T10:45:31.282 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:31.279+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.280833+0000) 2019-10-13T10:45:31.283 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:31.280+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.282435+0000) 2019-10-13T10:45:31.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:31.286+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:31.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:31.513+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.514552+0000) 2019-10-13T10:45:33.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:33.003+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:33.003618+0000) 2019-10-13T10:45:33.005 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:33.003+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:33.005659+0000) 2019-10-13T10:45:34.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:34.819+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.820084+0000) 2019-10-13T10:45:35.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:35.654+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.655442+0000) 2019-10-13T10:45:35.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:35.654+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.655439+0000) 2019-10-13T10:45:36.290 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:36.287+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.289246+0000) 2019-10-13T10:45:36.291 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:36.288+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.289753+0000) 2019-10-13T10:45:36.291 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:36.288+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.290033+0000) 2019-10-13T10:45:36.292 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:36.289+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.291602+0000) 2019-10-13T10:45:36.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:36.289+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.290439+0000) 2019-10-13T10:45:36.292 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:36.289+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.290865+0000) 2019-10-13T10:45:36.292 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:36.289+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.291151+0000) 2019-10-13T10:45:36.293 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:36.290+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:36.292422+0000) 2019-10-13T10:45:36.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:36.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194518 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:38.565 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:38.563+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:38.564214+0000) 2019-10-13T10:45:38.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:38.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:38.900313+0000) 2019-10-13T10:45:41.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:41.298+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.299373+0000) 2019-10-13T10:45:41.301 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:41.299+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.299603+0000) 2019-10-13T10:45:41.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:41.299+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.299782+0000) 2019-10-13T10:45:41.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:41.299+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.299947+0000) 2019-10-13T10:45:41.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:41.300+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.301389+0000) 2019-10-13T10:45:41.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:41.301+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.302168+0000) 2019-10-13T10:45:41.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:41.299+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.300150+0000) 2019-10-13T10:45:41.303 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:41.300+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.300440+0000) 2019-10-13T10:45:41.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:41.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194547 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:41.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:41.514+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.514727+0000) 2019-10-13T10:45:43.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:43.003+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:43.003821+0000) 2019-10-13T10:45:43.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:43.005+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:43.005859+0000) 2019-10-13T10:45:43.940 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:45:43.940 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:43.944 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:45:43.945 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:44.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:44.821+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.822528+0000) 2019-10-13T10:45:45.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:45.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.655630+0000) 2019-10-13T10:45:45.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:45.655+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.655628+0000) 2019-10-13T10:45:46.310 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:46.308+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.308823+0000) 2019-10-13T10:45:46.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:46.308+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.309148+0000) 2019-10-13T10:45:46.311 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:46.309+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.310104+0000) 2019-10-13T10:45:46.312 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:46.309+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.309724+0000) 2019-10-13T10:45:46.312 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:46.309+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.310024+0000) 2019-10-13T10:45:46.312 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:46.309+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.309877+0000) 2019-10-13T10:45:46.312 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:46.309+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.310200+0000) 2019-10-13T10:45:46.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:46.312+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:46.313233+0000) 2019-10-13T10:45:46.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:46.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:48.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:48.563+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:48.564383+0000) 2019-10-13T10:45:48.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:48.900+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:48.900485+0000) 2019-10-13T10:45:51.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:51.318+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.318795+0000) 2019-10-13T10:45:51.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:51.319+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.319816+0000) 2019-10-13T10:45:51.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:51.318+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.319278+0000) 2019-10-13T10:45:51.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:51.318+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.319117+0000) 2019-10-13T10:45:51.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:51.319+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.319553+0000) 2019-10-13T10:45:51.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:51.319+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.319688+0000) 2019-10-13T10:45:51.323 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:51.319+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.319993+0000) 2019-10-13T10:45:51.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:51.320+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.321665+0000) 2019-10-13T10:45:51.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:51.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:51.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:51.514+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.514877+0000) 2019-10-13T10:45:53.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:53.003+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:53.004015+0000) 2019-10-13T10:45:53.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:53.005+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:53.006042+0000) 2019-10-13T10:45:54.825 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:54.824+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.824926+0000) 2019-10-13T10:45:55.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:55.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.655846+0000) 2019-10-13T10:45:55.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:55.655+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.655844+0000) 2019-10-13T10:45:56.329 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:45:56.327+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.328021+0000) 2019-10-13T10:45:56.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:56.327+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.328280+0000) 2019-10-13T10:45:56.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:45:56.328+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.329334+0000) 2019-10-13T10:45:56.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:45:56.328+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.328845+0000) 2019-10-13T10:45:56.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:45:56.328+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.329007+0000) 2019-10-13T10:45:56.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:56.329+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.329407+0000) 2019-10-13T10:45:56.331 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:45:56.329+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.329555+0000) 2019-10-13T10:45:56.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:45:56.330+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:56.331621+0000) 2019-10-13T10:45:56.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:45:56.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194642 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:45:58.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:45:58.563+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:58.564606+0000) 2019-10-13T10:45:58.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:45:58.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:58.900672+0000) 2019-10-13T10:46:01.339 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:01.336+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.338300+0000) 2019-10-13T10:46:01.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:01.337+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.338522+0000) 2019-10-13T10:46:01.340 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:01.338+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.339674+0000) 2019-10-13T10:46:01.341 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:01.337+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.338901+0000) 2019-10-13T10:46:01.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:01.337+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.339072+0000) 2019-10-13T10:46:01.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:01.337+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.339179+0000) 2019-10-13T10:46:01.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:01.337+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.339389+0000) 2019-10-13T10:46:01.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:01.340+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.341313+0000) 2019-10-13T10:46:01.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:01.345+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:01.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:01.514+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.515046+0000) 2019-10-13T10:46:03.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:03.003+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:03.004235+0000) 2019-10-13T10:46:03.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:03.005+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:03.006247+0000) 2019-10-13T10:46:04.827 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:04.827+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.827322+0000) 2019-10-13T10:46:05.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:05.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.656043+0000) 2019-10-13T10:46:05.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:05.655+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.656045+0000) 2019-10-13T10:46:06.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:06.346+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.348266+0000) 2019-10-13T10:46:06.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:06.347+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.348622+0000) 2019-10-13T10:46:06.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:06.347+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.348734+0000) 2019-10-13T10:46:06.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:06.349+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.350527+0000) 2019-10-13T10:46:06.351 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:06.347+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.349068+0000) 2019-10-13T10:46:06.351 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:06.348+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.349454+0000) 2019-10-13T10:46:06.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:06.348+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.349845+0000) 2019-10-13T10:46:06.352 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:06.351+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:06.352385+0000) 2019-10-13T10:46:06.357 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:06.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:08.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:08.563+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:08.564790+0000) 2019-10-13T10:46:08.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:08.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:08.900848+0000) 2019-10-13T10:46:11.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:11.356+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.358166+0000) 2019-10-13T10:46:11.359 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:11.356+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.358291+0000) 2019-10-13T10:46:11.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:11.357+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.358494+0000) 2019-10-13T10:46:11.360 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:11.359+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.359879+0000) 2019-10-13T10:46:11.361 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:11.357+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.358843+0000) 2019-10-13T10:46:11.361 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:11.357+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.359138+0000) 2019-10-13T10:46:11.361 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:11.358+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.359493+0000) 2019-10-13T10:46:11.362 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:11.361+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.362304+0000) 2019-10-13T10:46:11.367 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:11.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194732 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:11.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:11.514+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.515185+0000) 2019-10-13T10:46:13.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:13.003+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:13.004439+0000) 2019-10-13T10:46:13.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:13.006+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:13.006443+0000) 2019-10-13T10:46:14.094 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:46:14.095 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:14.099 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:46:14.099 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:14.830 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:14.829+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.829627+0000) 2019-10-13T10:46:15.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:15.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.656229+0000) 2019-10-13T10:46:15.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:15.655+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.656222+0000) 2019-10-13T10:46:16.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:16.367+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.368468+0000) 2019-10-13T10:46:16.370 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:16.367+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.368650+0000) 2019-10-13T10:46:16.370 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:16.368+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.370320+0000) 2019-10-13T10:46:16.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:16.367+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.368998+0000) 2019-10-13T10:46:16.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:16.367+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.369097+0000) 2019-10-13T10:46:16.371 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:16.368+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.369667+0000) 2019-10-13T10:46:16.371 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:16.368+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.369962+0000) 2019-10-13T10:46:16.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:16.370+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:16.371837+0000) 2019-10-13T10:46:16.377 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:16.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:18.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:18.564+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:18.564978+0000) 2019-10-13T10:46:18.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:18.899+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:18.901043+0000) 2019-10-13T10:46:21.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:21.377+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.378580+0000) 2019-10-13T10:46:21.381 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:21.377+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.378779+0000) 2019-10-13T10:46:21.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:21.377+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.379211+0000) 2019-10-13T10:46:21.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:21.377+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.379351+0000) 2019-10-13T10:46:21.382 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:21.378+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.379541+0000) 2019-10-13T10:46:21.382 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:21.378+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.379707+0000) 2019-10-13T10:46:21.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:21.377+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.379285+0000) 2019-10-13T10:46:21.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:21.382+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.383824+0000) 2019-10-13T10:46:21.387 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:21.385+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:21.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:21.513+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.515381+0000) 2019-10-13T10:46:23.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:23.004+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:23.004570+0000) 2019-10-13T10:46:23.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:23.005+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:23.006639+0000) 2019-10-13T10:46:24.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:24.831+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.832101+0000) 2019-10-13T10:46:25.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:25.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.656386+0000) 2019-10-13T10:46:25.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:25.655+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.656345+0000) 2019-10-13T10:46:26.389 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:26.387+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.388304+0000) 2019-10-13T10:46:26.390 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:26.388+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.388977+0000) 2019-10-13T10:46:26.390 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:26.388+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.389222+0000) 2019-10-13T10:46:26.390 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:26.389+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.389389+0000) 2019-10-13T10:46:26.392 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:26.388+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.390436+0000) 2019-10-13T10:46:26.392 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:26.389+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.390764+0000) 2019-10-13T10:46:26.392 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:26.389+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.389883+0000) 2019-10-13T10:46:26.392 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:26.388+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:26.388522+0000) 2019-10-13T10:46:26.397 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:26.395+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:28.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:28.564+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:28.565209+0000) 2019-10-13T10:46:28.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:28.900+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:28.901256+0000) 2019-10-13T10:46:31.399 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:31.397+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.397987+0000) 2019-10-13T10:46:31.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:31.397+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.398081+0000) 2019-10-13T10:46:31.400 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:31.397+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.398190+0000) 2019-10-13T10:46:31.400 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:31.398+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.399230+0000) 2019-10-13T10:46:31.401 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:31.398+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.398946+0000) 2019-10-13T10:46:31.401 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:31.398+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.398973+0000) 2019-10-13T10:46:31.401 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:31.398+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.399320+0000) 2019-10-13T10:46:31.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:31.400+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.401516+0000) 2019-10-13T10:46:31.406 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:31.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194856 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:31.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:31.515+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.515565+0000) 2019-10-13T10:46:33.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:33.004+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:33.004735+0000) 2019-10-13T10:46:33.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:33.006+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:33.006843+0000) 2019-10-13T10:46:34.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:34.833+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.834537+0000) 2019-10-13T10:46:35.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:35.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.656596+0000) 2019-10-13T10:46:35.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:35.656+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.656592+0000) 2019-10-13T10:46:36.408 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:36.406+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.407265+0000) 2019-10-13T10:46:36.408 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:36.406+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.407362+0000) 2019-10-13T10:46:36.409 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:36.408+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.408921+0000) 2019-10-13T10:46:36.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:36.408+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.408461+0000) 2019-10-13T10:46:36.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:36.408+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.408916+0000) 2019-10-13T10:46:36.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:36.408+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.409092+0000) 2019-10-13T10:46:36.411 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:36.408+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.409245+0000) 2019-10-13T10:46:36.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:36.410+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:36.411671+0000) 2019-10-13T10:46:36.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:36.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:38.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:38.565+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:38.565413+0000) 2019-10-13T10:46:38.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:38.901+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:38.901430+0000) 2019-10-13T10:46:41.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:41.416+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.417246+0000) 2019-10-13T10:46:41.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:41.417+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.417394+0000) 2019-10-13T10:46:41.419 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:41.417+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.417722+0000) 2019-10-13T10:46:41.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:41.418+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.418762+0000) 2019-10-13T10:46:41.420 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:41.417+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.417988+0000) 2019-10-13T10:46:41.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:41.417+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.418102+0000) 2019-10-13T10:46:41.420 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:41.418+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.418537+0000) 2019-10-13T10:46:41.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:41.421+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.421779+0000) 2019-10-13T10:46:41.426 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:41.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:41.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:41.515+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.515752+0000) 2019-10-13T10:46:43.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:43.004+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:43.004935+0000) 2019-10-13T10:46:43.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:43.006+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:43.007066+0000) 2019-10-13T10:46:44.249 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:46:44.249 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:44.253 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:46:44.253 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:44.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:44.836+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.837005+0000) 2019-10-13T10:46:45.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:45.655+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.656800+0000) 2019-10-13T10:46:45.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:45.655+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.656785+0000) 2019-10-13T10:46:46.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:46.425+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.427282+0000) 2019-10-13T10:46:46.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:46.426+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.427654+0000) 2019-10-13T10:46:46.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:46.426+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.427868+0000) 2019-10-13T10:46:46.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:46.426+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.428141+0000) 2019-10-13T10:46:46.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:46.426+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.428266+0000) 2019-10-13T10:46:46.432 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:46.427+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.428393+0000) 2019-10-13T10:46:46.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:46.427+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.427768+0000) 2019-10-13T10:46:46.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:46.429+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:46.430439+0000) 2019-10-13T10:46:46.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:46.433+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194953 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:48.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:48.564+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:48.565566+0000) 2019-10-13T10:46:48.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:48.900+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:48.901601+0000) 2019-10-13T10:46:51.438 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:51.435+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.436844+0000) 2019-10-13T10:46:51.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:51.435+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.437073+0000) 2019-10-13T10:46:51.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:51.435+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.436707+0000) 2019-10-13T10:46:51.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:51.435+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.437222+0000) 2019-10-13T10:46:51.440 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:51.437+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.438231+0000) 2019-10-13T10:46:51.440 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:51.438+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.439442+0000) 2019-10-13T10:46:51.440 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:51.436+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.437867+0000) 2019-10-13T10:46:51.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:51.436+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.438019+0000) 2019-10-13T10:46:51.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:51.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 194980 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:51.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:51.514+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.515860+0000) 2019-10-13T10:46:53.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:53.004+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:53.005128+0000) 2019-10-13T10:46:53.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:53.006+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:53.007223+0000) 2019-10-13T10:46:54.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:54.839+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.839393+0000) 2019-10-13T10:46:55.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:55.656+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.657001+0000) 2019-10-13T10:46:55.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:55.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.657026+0000) 2019-10-13T10:46:56.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:56.445+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.446450+0000) 2019-10-13T10:46:56.448 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:46:56.445+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.446954+0000) 2019-10-13T10:46:56.448 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:46:56.445+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.447145+0000) 2019-10-13T10:46:56.448 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:46:56.445+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.447345+0000) 2019-10-13T10:46:56.449 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:46:56.447+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.448431+0000) 2019-10-13T10:46:56.449 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:46:56.446+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.447812+0000) 2019-10-13T10:46:56.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:56.446+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.447879+0000) 2019-10-13T10:46:56.451 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:46:56.450+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:56.451127+0000) 2019-10-13T10:46:56.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:46:56.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:46:58.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:46:58.565+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:58.565724+0000) 2019-10-13T10:46:58.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:46:58.901+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:58.901792+0000) 2019-10-13T10:47:01.457 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:01.455+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.456596+0000) 2019-10-13T10:47:01.458 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:01.456+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.457676+0000) 2019-10-13T10:47:01.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:01.455+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.457128+0000) 2019-10-13T10:47:01.459 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:01.455+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.457313+0000) 2019-10-13T10:47:01.459 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:01.456+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.457433+0000) 2019-10-13T10:47:01.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:01.456+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.457626+0000) 2019-10-13T10:47:01.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:01.456+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.457748+0000) 2019-10-13T10:47:01.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:01.459+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.459950+0000) 2019-10-13T10:47:01.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:01.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195043 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:01.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:01.515+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.516035+0000) 2019-10-13T10:47:03.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:03.007+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:03.007440+0000) 2019-10-13T10:47:03.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:03.004+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:03.005326+0000) 2019-10-13T10:47:04.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:04.841+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.841929+0000) 2019-10-13T10:47:05.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:05.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.657214+0000) 2019-10-13T10:47:05.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:05.656+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.657209+0000) 2019-10-13T10:47:06.467 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:06.466+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.467537+0000) 2019-10-13T10:47:06.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:06.465+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.467033+0000) 2019-10-13T10:47:06.469 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:06.465+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.467224+0000) 2019-10-13T10:47:06.470 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:06.465+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.467256+0000) 2019-10-13T10:47:06.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:06.466+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.467535+0000) 2019-10-13T10:47:06.470 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:06.466+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.467750+0000) 2019-10-13T10:47:06.470 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:06.466+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.468116+0000) 2019-10-13T10:47:06.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:06.469+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:06.470714+0000) 2019-10-13T10:47:06.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:06.473+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:08.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:08.565+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:08.565905+0000) 2019-10-13T10:47:08.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:08.901+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:08.901997+0000) 2019-10-13T10:47:11.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:11.475+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.476542+0000) 2019-10-13T10:47:11.478 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:11.475+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.477052+0000) 2019-10-13T10:47:11.479 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:11.476+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.478046+0000) 2019-10-13T10:47:11.479 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:11.475+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.477267+0000) 2019-10-13T10:47:11.479 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:11.476+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.477505+0000) 2019-10-13T10:47:11.479 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:11.476+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.477700+0000) 2019-10-13T10:47:11.480 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:11.476+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.477994+0000) 2019-10-13T10:47:11.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:11.478+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.480457+0000) 2019-10-13T10:47:11.486 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:11.484+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:11.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:11.515+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.516148+0000) 2019-10-13T10:47:13.006 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:13.005+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:13.005492+0000) 2019-10-13T10:47:13.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:13.006+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:13.007639+0000) 2019-10-13T10:47:14.387 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:47:14.387 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:14.392 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:47:14.392 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:14.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:14.842+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.844333+0000) 2019-10-13T10:47:15.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:15.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.657430+0000) 2019-10-13T10:47:15.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:15.657+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.657418+0000) 2019-10-13T10:47:16.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:16.485+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.485976+0000) 2019-10-13T10:47:16.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:16.485+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.486148+0000) 2019-10-13T10:47:16.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:16.485+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.486308+0000) 2019-10-13T10:47:16.489 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:16.485+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.487260+0000) 2019-10-13T10:47:16.489 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:16.486+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.487055+0000) 2019-10-13T10:47:16.489 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:16.486+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.487327+0000) 2019-10-13T10:47:16.489 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:16.486+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.487327+0000) 2019-10-13T10:47:16.490 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:16.488+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:16.490634+0000) 2019-10-13T10:47:16.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:16.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195138 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:18.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:18.565+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:18.566091+0000) 2019-10-13T10:47:18.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:18.901+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:18.902167+0000) 2019-10-13T10:47:21.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:21.493+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.493981+0000) 2019-10-13T10:47:21.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:21.493+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.493700+0000) 2019-10-13T10:47:21.496 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:21.493+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.494389+0000) 2019-10-13T10:47:21.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:21.494+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.494606+0000) 2019-10-13T10:47:21.497 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:21.494+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.494975+0000) 2019-10-13T10:47:21.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:21.494+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.495262+0000) 2019-10-13T10:47:21.497 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:21.495+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.495491+0000) 2019-10-13T10:47:21.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:21.497+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.497909+0000) 2019-10-13T10:47:21.503 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:21.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:21.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:21.515+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.516280+0000) 2019-10-13T10:47:23.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:23.005+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:23.005668+0000) 2019-10-13T10:47:23.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:23.007+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:23.007849+0000) 2019-10-13T10:47:24.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:24.846+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.846823+0000) 2019-10-13T10:47:25.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:25.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.657629+0000) 2019-10-13T10:47:25.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:25.657+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.657610+0000) 2019-10-13T10:47:26.505 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:26.504+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.504832+0000) 2019-10-13T10:47:26.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:26.503+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.504293+0000) 2019-10-13T10:47:26.505 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:26.504+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.504617+0000) 2019-10-13T10:47:26.506 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:26.504+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.504848+0000) 2019-10-13T10:47:26.506 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:26.504+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.504753+0000) 2019-10-13T10:47:26.507 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:26.505+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.505409+0000) 2019-10-13T10:47:26.507 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:26.505+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.505492+0000) 2019-10-13T10:47:26.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:26.507+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:26.508122+0000) 2019-10-13T10:47:26.513 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:26.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195201 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:28.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:28.566+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:28.566288+0000) 2019-10-13T10:47:28.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:28.901+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:28.902301+0000) 2019-10-13T10:47:31.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:31.512+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.513735+0000) 2019-10-13T10:47:31.515 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:31.512+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.513998+0000) 2019-10-13T10:47:31.515 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:31.512+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.514225+0000) 2019-10-13T10:47:31.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:31.512+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.514316+0000) 2019-10-13T10:47:31.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:31.513+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.514450+0000) 2019-10-13T10:47:31.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:31.513+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.514558+0000) 2019-10-13T10:47:31.517 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:31.515+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.516709+0000) 2019-10-13T10:47:31.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:31.516+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.516839+0000) 2019-10-13T10:47:31.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:31.514+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.516382+0000) 2019-10-13T10:47:31.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:31.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195228 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:33.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:33.004+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:33.005851+0000) 2019-10-13T10:47:33.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:33.007+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:33.008049+0000) 2019-10-13T10:47:34.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:34.848+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.849166+0000) 2019-10-13T10:47:35.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:35.656+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.657820+0000) 2019-10-13T10:47:35.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:35.656+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.657815+0000) 2019-10-13T10:47:36.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:36.522+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.523468+0000) 2019-10-13T10:47:36.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:36.522+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.523820+0000) 2019-10-13T10:47:36.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:36.522+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.523713+0000) 2019-10-13T10:47:36.526 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:36.524+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.524811+0000) 2019-10-13T10:47:36.526 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:36.523+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.524513+0000) 2019-10-13T10:47:36.526 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:36.523+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.524863+0000) 2019-10-13T10:47:36.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:36.523+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.525011+0000) 2019-10-13T10:47:36.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:36.526+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:36.527613+0000) 2019-10-13T10:47:36.532 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:36.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:38.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:38.565+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:38.566389+0000) 2019-10-13T10:47:38.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:38.901+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:38.902434+0000) 2019-10-13T10:47:41.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:41.515+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.516511+0000) 2019-10-13T10:47:41.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:41.532+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.533810+0000) 2019-10-13T10:47:41.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:41.534+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.534868+0000) 2019-10-13T10:47:41.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:41.532+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.534153+0000) 2019-10-13T10:47:41.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:41.533+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.534512+0000) 2019-10-13T10:47:41.536 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:41.533+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.534638+0000) 2019-10-13T10:47:41.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:41.533+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.534924+0000) 2019-10-13T10:47:41.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:41.533+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.535033+0000) 2019-10-13T10:47:41.537 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:41.535+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.536642+0000) 2019-10-13T10:47:41.543 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:41.540+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:43.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:43.005+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:43.006036+0000) 2019-10-13T10:47:43.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:43.007+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:43.008226+0000) 2019-10-13T10:47:44.538 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:47:44.538 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:44.543 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:47:44.543 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:44.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:44.851+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:44.851544+0000) 2019-10-13T10:47:45.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:45.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.657976+0000) 2019-10-13T10:47:45.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:45.657+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.658011+0000) 2019-10-13T10:47:46.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:46.541+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.543069+0000) 2019-10-13T10:47:46.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:46.543+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.544144+0000) 2019-10-13T10:47:46.545 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:46.542+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.543547+0000) 2019-10-13T10:47:46.545 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:46.542+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.543667+0000) 2019-10-13T10:47:46.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:46.542+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.543855+0000) 2019-10-13T10:47:46.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:46.542+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.543810+0000) 2019-10-13T10:47:46.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:46.542+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.544296+0000) 2019-10-13T10:47:46.547 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:46.546+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:46.547019+0000) 2019-10-13T10:47:46.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:46.547+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195325 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:48.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:48.565+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:48.566534+0000) 2019-10-13T10:47:48.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:48.901+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:48.902599+0000) 2019-10-13T10:47:51.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:51.516+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.516664+0000) 2019-10-13T10:47:51.551 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:51.549+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.550471+0000) 2019-10-13T10:47:51.554 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:51.550+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.551580+0000) 2019-10-13T10:47:51.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:51.549+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.550896+0000) 2019-10-13T10:47:51.554 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:51.549+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.551022+0000) 2019-10-13T10:47:51.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:51.549+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.551138+0000) 2019-10-13T10:47:51.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:51.550+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.551645+0000) 2019-10-13T10:47:51.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:51.550+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.551878+0000) 2019-10-13T10:47:51.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:51.553+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.554221+0000) 2019-10-13T10:47:51.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:51.556+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195352 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:53.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:53.005+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:53.006222+0000) 2019-10-13T10:47:53.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:53.008+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:53.008390+0000) 2019-10-13T10:47:54.855 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:54.854+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:54.854571+0000) 2019-10-13T10:47:55.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:55.657+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.658171+0000) 2019-10-13T10:47:55.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:55.657+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.658169+0000) 2019-10-13T10:47:56.560 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:47:56.559+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.560519+0000) 2019-10-13T10:47:56.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:47:56.558+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.559807+0000) 2019-10-13T10:47:56.562 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:47:56.558+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.560044+0000) 2019-10-13T10:47:56.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:56.558+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.560315+0000) 2019-10-13T10:47:56.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:56.559+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.560640+0000) 2019-10-13T10:47:56.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:47:56.559+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.560917+0000) 2019-10-13T10:47:56.563 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:47:56.559+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.560989+0000) 2019-10-13T10:47:56.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:47:56.563+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:56.564186+0000) 2019-10-13T10:47:56.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:47:56.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195386 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:47:58.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:47:58.566+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:58.566748+0000) 2019-10-13T10:47:58.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:47:58.902+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:58.902785+0000) 2019-10-13T10:48:01.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:01.516+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.516849+0000) 2019-10-13T10:48:01.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:01.569+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.570761+0000) 2019-10-13T10:48:01.571 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:01.569+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.570253+0000) 2019-10-13T10:48:01.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:01.570+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.570532+0000) 2019-10-13T10:48:01.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:01.570+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.570679+0000) 2019-10-13T10:48:01.572 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:01.570+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.570904+0000) 2019-10-13T10:48:01.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:01.570+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.571086+0000) 2019-10-13T10:48:01.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:01.570+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.571342+0000) 2019-10-13T10:48:01.573 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:01.571+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.573085+0000) 2019-10-13T10:48:01.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:01.578+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:03.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:03.006+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:03.006424+0000) 2019-10-13T10:48:03.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:03.007+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:03.008542+0000) 2019-10-13T10:48:04.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:04.856+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:04.857670+0000) 2019-10-13T10:48:05.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:05.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.658387+0000) 2019-10-13T10:48:05.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:05.658+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.658386+0000) 2019-10-13T10:48:06.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:06.580+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.582251+0000) 2019-10-13T10:48:06.584 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:06.579+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.580278+0000) 2019-10-13T10:48:06.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:06.580+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.580435+0000) 2019-10-13T10:48:06.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:06.580+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.580733+0000) 2019-10-13T10:48:06.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:06.580+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.581081+0000) 2019-10-13T10:48:06.585 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:06.580+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.581379+0000) 2019-10-13T10:48:06.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:06.581+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.581698+0000) 2019-10-13T10:48:06.587 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:06.582+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:06.584209+0000) 2019-10-13T10:48:06.598 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:06.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195449 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:08.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:08.566+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:08.566958+0000) 2019-10-13T10:48:08.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:08.902+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:08.902979+0000) 2019-10-13T10:48:11.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:11.516+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.517069+0000) 2019-10-13T10:48:11.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:11.590+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.591378+0000) 2019-10-13T10:48:11.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:11.590+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.590450+0000) 2019-10-13T10:48:11.592 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:11.590+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.590856+0000) 2019-10-13T10:48:11.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:11.590+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.590961+0000) 2019-10-13T10:48:11.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:11.590+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.591121+0000) 2019-10-13T10:48:11.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:11.591+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.591543+0000) 2019-10-13T10:48:11.593 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:11.591+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.592094+0000) 2019-10-13T10:48:11.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:11.594+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.594799+0000) 2019-10-13T10:48:11.600 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:11.598+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195476 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:13.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:13.006+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:13.006585+0000) 2019-10-13T10:48:13.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:13.008+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:13.008744+0000) 2019-10-13T10:48:14.696 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:48:14.696 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:14.700 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:48:14.701 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:14.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:14.859+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:14.860706+0000) 2019-10-13T10:48:15.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:15.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.658645+0000) 2019-10-13T10:48:15.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:15.658+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.658645+0000) 2019-10-13T10:48:16.601 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:16.599+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.600714+0000) 2019-10-13T10:48:16.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:16.601+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.602030+0000) 2019-10-13T10:48:16.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:16.599+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.601319+0000) 2019-10-13T10:48:16.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:16.600+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.601767+0000) 2019-10-13T10:48:16.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:16.600+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.601870+0000) 2019-10-13T10:48:16.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:16.603+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.603736+0000) 2019-10-13T10:48:16.604 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:16.600+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.602170+0000) 2019-10-13T10:48:16.604 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:16.600+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:16.602287+0000) 2019-10-13T10:48:16.610 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:16.607+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:18.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:18.566+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:18.567133+0000) 2019-10-13T10:48:18.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:18.902+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:18.903159+0000) 2019-10-13T10:48:21.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:21.516+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.517264+0000) 2019-10-13T10:48:21.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:21.609+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.610605+0000) 2019-10-13T10:48:21.613 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:21.611+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.612336+0000) 2019-10-13T10:48:21.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:21.609+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.610975+0000) 2019-10-13T10:48:21.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:21.609+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.611274+0000) 2019-10-13T10:48:21.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:21.610+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.611724+0000) 2019-10-13T10:48:21.613 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:21.610+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.611797+0000) 2019-10-13T10:48:21.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:21.610+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.611977+0000) 2019-10-13T10:48:21.614 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:21.613+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.613799+0000) 2019-10-13T10:48:21.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:21.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:23.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:23.005+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:23.006783+0000) 2019-10-13T10:48:23.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:23.008+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:23.008929+0000) 2019-10-13T10:48:24.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:24.862+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:24.863140+0000) 2019-10-13T10:48:25.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:25.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.658845+0000) 2019-10-13T10:48:25.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:25.658+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.658845+0000) 2019-10-13T10:48:26.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:26.619+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.620664+0000) 2019-10-13T10:48:26.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:26.619+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.620776+0000) 2019-10-13T10:48:26.622 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:26.619+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.620983+0000) 2019-10-13T10:48:26.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:26.619+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.621284+0000) 2019-10-13T10:48:26.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:26.621+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.622003+0000) 2019-10-13T10:48:26.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:26.620+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.621891+0000) 2019-10-13T10:48:26.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:26.620+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.622106+0000) 2019-10-13T10:48:26.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:26.624+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:26.625690+0000) 2019-10-13T10:48:26.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:26.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:28.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:28.566+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:28.567275+0000) 2019-10-13T10:48:28.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:28.902+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:28.903361+0000) 2019-10-13T10:48:31.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:31.516+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.517459+0000) 2019-10-13T10:48:31.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:31.629+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.630787+0000) 2019-10-13T10:48:31.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:31.629+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.630934+0000) 2019-10-13T10:48:31.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:31.629+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.631140+0000) 2019-10-13T10:48:31.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:31.629+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.631386+0000) 2019-10-13T10:48:31.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:31.630+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.631794+0000) 2019-10-13T10:48:31.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:31.630+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.632068+0000) 2019-10-13T10:48:31.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:31.630+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.631357+0000) 2019-10-13T10:48:31.634 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:31.632+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.633618+0000) 2019-10-13T10:48:31.640 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:31.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195600 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:33.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:33.005+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:33.007002+0000) 2019-10-13T10:48:33.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:33.008+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:33.009144+0000) 2019-10-13T10:48:34.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:34.865+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:34.865738+0000) 2019-10-13T10:48:35.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:35.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.659042+0000) 2019-10-13T10:48:35.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:35.658+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.659042+0000) 2019-10-13T10:48:36.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:36.638+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.640322+0000) 2019-10-13T10:48:36.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:36.639+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.640503+0000) 2019-10-13T10:48:36.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:36.640+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.641483+0000) 2019-10-13T10:48:36.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:36.639+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.641050+0000) 2019-10-13T10:48:36.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:36.639+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.641145+0000) 2019-10-13T10:48:36.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:36.639+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.641326+0000) 2019-10-13T10:48:36.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:36.640+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.641474+0000) 2019-10-13T10:48:36.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:36.642+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:36.643682+0000) 2019-10-13T10:48:36.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:36.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:38.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:38.566+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:38.567445+0000) 2019-10-13T10:48:38.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:38.902+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:38.903508+0000) 2019-10-13T10:48:41.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:41.517+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.517646+0000) 2019-10-13T10:48:41.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:41.646+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.648383+0000) 2019-10-13T10:48:41.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:41.647+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.648573+0000) 2019-10-13T10:48:41.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:41.647+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.648720+0000) 2019-10-13T10:48:41.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:41.647+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.648896+0000) 2019-10-13T10:48:41.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:41.649+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.649951+0000) 2019-10-13T10:48:41.651 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:41.647+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.649319+0000) 2019-10-13T10:48:41.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:41.648+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.649451+0000) 2019-10-13T10:48:41.652 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:41.651+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.652642+0000) 2019-10-13T10:48:41.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:41.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:43.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:43.006+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:43.007190+0000) 2019-10-13T10:48:43.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:43.009+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:43.009347+0000) 2019-10-13T10:48:44.842 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:48:44.843 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:44.847 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:48:44.847 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:44.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:44.867+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:44.868106+0000) 2019-10-13T10:48:45.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:45.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.659197+0000) 2019-10-13T10:48:45.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:45.658+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.659197+0000) 2019-10-13T10:48:46.657 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:46.656+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.656635+0000) 2019-10-13T10:48:46.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:46.656+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.657019+0000) 2019-10-13T10:48:46.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:46.656+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.657261+0000) 2019-10-13T10:48:46.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:46.657+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.657545+0000) 2019-10-13T10:48:46.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:46.657+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.657789+0000) 2019-10-13T10:48:46.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:46.657+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.658275+0000) 2019-10-13T10:48:46.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:46.657+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.657962+0000) 2019-10-13T10:48:46.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:46.659+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:46.660686+0000) 2019-10-13T10:48:46.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:46.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195697 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:48.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:48.567+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:48.567638+0000) 2019-10-13T10:48:48.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:48.903+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:48.903693+0000) 2019-10-13T10:48:51.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:51.517+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.517838+0000) 2019-10-13T10:48:51.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:51.665+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.666361+0000) 2019-10-13T10:48:51.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:51.665+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.667452+0000) 2019-10-13T10:48:51.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:51.666+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.666920+0000) 2019-10-13T10:48:51.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:51.666+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.667074+0000) 2019-10-13T10:48:51.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:51.666+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.667354+0000) 2019-10-13T10:48:51.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:51.667+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.667649+0000) 2019-10-13T10:48:51.669 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:51.667+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.667866+0000) 2019-10-13T10:48:51.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:51.668+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.670309+0000) 2019-10-13T10:48:51.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:51.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195724 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:53.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:53.007+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:53.007377+0000) 2019-10-13T10:48:53.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:53.007+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:53.009544+0000) 2019-10-13T10:48:54.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:54.869+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:54.870389+0000) 2019-10-13T10:48:55.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:55.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.659372+0000) 2019-10-13T10:48:55.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:55.659+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.659373+0000) 2019-10-13T10:48:56.677 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:48:56.675+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.677638+0000) 2019-10-13T10:48:56.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:48:56.676+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.677143+0000) 2019-10-13T10:48:56.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:56.676+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.677274+0000) 2019-10-13T10:48:56.679 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:48:56.677+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.677698+0000) 2019-10-13T10:48:56.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:48:56.677+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.677773+0000) 2019-10-13T10:48:56.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:48:56.677+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.677938+0000) 2019-10-13T10:48:56.680 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:56.677+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.678151+0000) 2019-10-13T10:48:56.681 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:48:56.678+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:56.680425+0000) 2019-10-13T10:48:56.686 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:48:56.685+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195758 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:48:58.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:48:58.567+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:58.567831+0000) 2019-10-13T10:48:58.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:48:58.903+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:58.903848+0000) 2019-10-13T10:49:01.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:01.517+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.518054+0000) 2019-10-13T10:49:01.689 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:01.687+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.687566+0000) 2019-10-13T10:49:01.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:01.687+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.687741+0000) 2019-10-13T10:49:01.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:01.687+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.687976+0000) 2019-10-13T10:49:01.690 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:01.687+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.688125+0000) 2019-10-13T10:49:01.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:01.687+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.688314+0000) 2019-10-13T10:49:01.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:01.688+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.688592+0000) 2019-10-13T10:49:01.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:01.688+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.689852+0000) 2019-10-13T10:49:01.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:01.690+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.692435+0000) 2019-10-13T10:49:01.699 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:01.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195787 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:03.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:03.006+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:03.007530+0000) 2019-10-13T10:49:03.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:03.009+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:03.009726+0000) 2019-10-13T10:49:04.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:04.871+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:04.872702+0000) 2019-10-13T10:49:05.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:05.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.659504+0000) 2019-10-13T10:49:05.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:05.658+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.659548+0000) 2019-10-13T10:49:06.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:06.696+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.698223+0000) 2019-10-13T10:49:06.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:06.698+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.699397+0000) 2019-10-13T10:49:06.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:06.697+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.698678+0000) 2019-10-13T10:49:06.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:06.697+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.698797+0000) 2019-10-13T10:49:06.701 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:06.697+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.698978+0000) 2019-10-13T10:49:06.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:06.697+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.699197+0000) 2019-10-13T10:49:06.702 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:06.697+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.699337+0000) 2019-10-13T10:49:06.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:06.702+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:06.702897+0000) 2019-10-13T10:49:06.708 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:06.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:08.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:08.567+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:08.568007+0000) 2019-10-13T10:49:08.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:08.902+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:08.904058+0000) 2019-10-13T10:49:11.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:11.517+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.518249+0000) 2019-10-13T10:49:11.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:11.707+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.709058+0000) 2019-10-13T10:49:11.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:11.707+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.709151+0000) 2019-10-13T10:49:11.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:11.707+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.709294+0000) 2019-10-13T10:49:11.711 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:11.709+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.710581+0000) 2019-10-13T10:49:11.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:11.708+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.710009+0000) 2019-10-13T10:49:11.712 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:11.708+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.710206+0000) 2019-10-13T10:49:11.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:11.708+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.709422+0000) 2019-10-13T10:49:11.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:11.711+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.711741+0000) 2019-10-13T10:49:11.720 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:11.715+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195848 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:13.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:13.006+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:13.007689+0000) 2019-10-13T10:49:13.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:13.009+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:13.009943+0000) 2019-10-13T10:49:14.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:14.874+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:14.875113+0000) 2019-10-13T10:49:14.980 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:49:14.981 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:14.985 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:49:14.985 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:15.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:15.658+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.659663+0000) 2019-10-13T10:49:15.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:15.658+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.659672+0000) 2019-10-13T10:49:16.719 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:16.717+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.718597+0000) 2019-10-13T10:49:16.720 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:16.719+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.719884+0000) 2019-10-13T10:49:16.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:16.717+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.719172+0000) 2019-10-13T10:49:16.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:16.718+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.719559+0000) 2019-10-13T10:49:16.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:16.718+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.719654+0000) 2019-10-13T10:49:16.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:16.718+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.719858+0000) 2019-10-13T10:49:16.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:16.718+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.720342+0000) 2019-10-13T10:49:16.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:16.721+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:16.722250+0000) 2019-10-13T10:49:16.727 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:16.725+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195882 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:18.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:18.567+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:18.568199+0000) 2019-10-13T10:49:18.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:18.903+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:18.904298+0000) 2019-10-13T10:49:21.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:21.517+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.518432+0000) 2019-10-13T10:49:21.729 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:21.728+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.729530+0000) 2019-10-13T10:49:21.730 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:21.727+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.728869+0000) 2019-10-13T10:49:21.731 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:21.727+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.728950+0000) 2019-10-13T10:49:21.731 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:21.727+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.729106+0000) 2019-10-13T10:49:21.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:21.727+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.729308+0000) 2019-10-13T10:49:21.731 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:21.728+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.729478+0000) 2019-10-13T10:49:21.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:21.728+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.729920+0000) 2019-10-13T10:49:21.732 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:21.730+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.731459+0000) 2019-10-13T10:49:21.737 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:21.735+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195911 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:23.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:23.007+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:23.007903+0000) 2019-10-13T10:49:23.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:23.009+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:23.010100+0000) 2019-10-13T10:49:24.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:24.877+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:24.877517+0000) 2019-10-13T10:49:25.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:25.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.659823+0000) 2019-10-13T10:49:25.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:25.659+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.659822+0000) 2019-10-13T10:49:26.739 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:26.736+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.738098+0000) 2019-10-13T10:49:26.739 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:26.736+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.738145+0000) 2019-10-13T10:49:26.740 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:26.737+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.739103+0000) 2019-10-13T10:49:26.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:26.737+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.739298+0000) 2019-10-13T10:49:26.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:26.738+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.739513+0000) 2019-10-13T10:49:26.741 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:26.739+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.740566+0000) 2019-10-13T10:49:26.741 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:26.738+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.740066+0000) 2019-10-13T10:49:26.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:26.742+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:26.742805+0000) 2019-10-13T10:49:26.748 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:26.745+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195945 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:28.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:28.567+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:28.568389+0000) 2019-10-13T10:49:28.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:28.903+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:28.904439+0000) 2019-10-13T10:49:31.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:31.518+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.518631+0000) 2019-10-13T10:49:31.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:31.748+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.748766+0000) 2019-10-13T10:49:31.750 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:31.748+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.748594+0000) 2019-10-13T10:49:31.751 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:31.749+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.749402+0000) 2019-10-13T10:49:31.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:31.749+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.749760+0000) 2019-10-13T10:49:31.751 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:31.749+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.749976+0000) 2019-10-13T10:49:31.751 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:31.750+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.750920+0000) 2019-10-13T10:49:31.752 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:31.751+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.751945+0000) 2019-10-13T10:49:31.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:31.753+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.754235+0000) 2019-10-13T10:49:31.758 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:31.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 195973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:33.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:33.007+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:33.008114+0000) 2019-10-13T10:49:33.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:33.010+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:33.010280+0000) 2019-10-13T10:49:34.880 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:34.879+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:34.879916+0000) 2019-10-13T10:49:35.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:35.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.660037+0000) 2019-10-13T10:49:35.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:35.659+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.660028+0000) 2019-10-13T10:49:36.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:36.759+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.760290+0000) 2019-10-13T10:49:36.761 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:36.759+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.759673+0000) 2019-10-13T10:49:36.761 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:36.759+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.760118+0000) 2019-10-13T10:49:36.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:36.759+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.760270+0000) 2019-10-13T10:49:36.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:36.759+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.760186+0000) 2019-10-13T10:49:36.762 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:36.760+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.760470+0000) 2019-10-13T10:49:36.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:36.760+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.760753+0000) 2019-10-13T10:49:36.763 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:36.762+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:36.763489+0000) 2019-10-13T10:49:36.768 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:36.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:38.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:38.568+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:38.568562+0000) 2019-10-13T10:49:38.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:38.904+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:38.904609+0000) 2019-10-13T10:49:41.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:41.518+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.518824+0000) 2019-10-13T10:49:41.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:41.769+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.769449+0000) 2019-10-13T10:49:41.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:41.769+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.769734+0000) 2019-10-13T10:49:41.771 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:41.769+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.769908+0000) 2019-10-13T10:49:41.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:41.769+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.770071+0000) 2019-10-13T10:49:41.772 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:41.768+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.770432+0000) 2019-10-13T10:49:41.772 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:41.770+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.770732+0000) 2019-10-13T10:49:41.772 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:41.770+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.770577+0000) 2019-10-13T10:49:41.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:41.772+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.773916+0000) 2019-10-13T10:49:41.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:41.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:43.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:43.008+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:43.008289+0000) 2019-10-13T10:49:43.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:43.009+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:43.010459+0000) 2019-10-13T10:49:44.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:44.881+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:44.882428+0000) 2019-10-13T10:49:45.134 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:49:45.134 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:45.137 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:49:45.138 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:45.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:45.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.660249+0000) 2019-10-13T10:49:45.661 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:45.660+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.660203+0000) 2019-10-13T10:49:46.780 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:46.778+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.779125+0000) 2019-10-13T10:49:46.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:46.778+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.779352+0000) 2019-10-13T10:49:46.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:46.779+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.779605+0000) 2019-10-13T10:49:46.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:46.778+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.780347+0000) 2019-10-13T10:49:46.782 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:46.779+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.779762+0000) 2019-10-13T10:49:46.783 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:46.779+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.780032+0000) 2019-10-13T10:49:46.783 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:46.779+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.780215+0000) 2019-10-13T10:49:46.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:46.781+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:46.783495+0000) 2019-10-13T10:49:46.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:46.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:48.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:48.567+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:48.568743+0000) 2019-10-13T10:49:48.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:48.903+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:48.904799+0000) 2019-10-13T10:49:51.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:51.518+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.519018+0000) 2019-10-13T10:49:51.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:51.788+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.789571+0000) 2019-10-13T10:49:51.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:51.788+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.789601+0000) 2019-10-13T10:49:51.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:51.788+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.789742+0000) 2019-10-13T10:49:51.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:51.788+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.790200+0000) 2019-10-13T10:49:51.792 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:51.788+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.790096+0000) 2019-10-13T10:49:51.792 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:51.789+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.790464+0000) 2019-10-13T10:49:51.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:51.788+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.790714+0000) 2019-10-13T10:49:51.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:51.791+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.792895+0000) 2019-10-13T10:49:51.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:51.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196099 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:53.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:53.007+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:53.008454+0000) 2019-10-13T10:49:53.010 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:53.009+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:53.010662+0000) 2019-10-13T10:49:54.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:54.884+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:54.884795+0000) 2019-10-13T10:49:55.661 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:55.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.660422+0000) 2019-10-13T10:49:55.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:55.659+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.660391+0000) 2019-10-13T10:49:56.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:49:56.796+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.798290+0000) 2019-10-13T10:49:56.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:49:56.798+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.799270+0000) 2019-10-13T10:49:56.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:49:56.797+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.798871+0000) 2019-10-13T10:49:56.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:56.797+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.799014+0000) 2019-10-13T10:49:56.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:49:56.797+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.799228+0000) 2019-10-13T10:49:56.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:56.797+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.799330+0000) 2019-10-13T10:49:56.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:49:56.798+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.799719+0000) 2019-10-13T10:49:56.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:49:56.801+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:56.802448+0000) 2019-10-13T10:49:56.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:49:56.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196132 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:49:58.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:49:58.568+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:58.568933+0000) 2019-10-13T10:49:58.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:49:58.903+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:58.905007+0000) 2019-10-13T10:50:01.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:01.518+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.519211+0000) 2019-10-13T10:50:01.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:01.806+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.807996+0000) 2019-10-13T10:50:01.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:01.808+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.809258+0000) 2019-10-13T10:50:01.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:01.807+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.808536+0000) 2019-10-13T10:50:01.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:01.807+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.808629+0000) 2019-10-13T10:50:01.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:01.807+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.808724+0000) 2019-10-13T10:50:01.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:01.807+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.808858+0000) 2019-10-13T10:50:01.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:01.807+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.809076+0000) 2019-10-13T10:50:01.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:01.810+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.811061+0000) 2019-10-13T10:50:01.815 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:01.812+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:03.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:03.007+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:03.008625+0000) 2019-10-13T10:50:03.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:03.010+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:03.010845+0000) 2019-10-13T10:50:04.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:04.886+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:04.887204+0000) 2019-10-13T10:50:05.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:05.659+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.660580+0000) 2019-10-13T10:50:05.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:05.659+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.660615+0000) 2019-10-13T10:50:06.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:06.816+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.816841+0000) 2019-10-13T10:50:06.819 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:06.815+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.816476+0000) 2019-10-13T10:50:06.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:06.814+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.816367+0000) 2019-10-13T10:50:06.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:06.815+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.816784+0000) 2019-10-13T10:50:06.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:06.815+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.816755+0000) 2019-10-13T10:50:06.819 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:06.815+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.817154+0000) 2019-10-13T10:50:06.820 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:06.815+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.817242+0000) 2019-10-13T10:50:06.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:06.818+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:06.819148+0000) 2019-10-13T10:50:06.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:06.822+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196196 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:08.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:08.568+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:08.569149+0000) 2019-10-13T10:50:08.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:08.904+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:08.905195+0000) 2019-10-13T10:50:11.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:11.518+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.519444+0000) 2019-10-13T10:50:11.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:11.824+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.825765+0000) 2019-10-13T10:50:11.827 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:11.824+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.825890+0000) 2019-10-13T10:50:11.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:11.826+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.827479+0000) 2019-10-13T10:50:11.828 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:11.824+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.826254+0000) 2019-10-13T10:50:11.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:11.825+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.826469+0000) 2019-10-13T10:50:11.829 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:11.825+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.826818+0000) 2019-10-13T10:50:11.829 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:11.824+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.826321+0000) 2019-10-13T10:50:11.829 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:11.828+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.829097+0000) 2019-10-13T10:50:11.833 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:11.830+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:13.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:13.007+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:13.008806+0000) 2019-10-13T10:50:13.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:13.010+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:13.011022+0000) 2019-10-13T10:50:14.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:14.888+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:14.889610+0000) 2019-10-13T10:50:15.279 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:50:15.279 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:15.284 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:50:15.284 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:15.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:15.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.660759+0000) 2019-10-13T10:50:15.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:15.660+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.660726+0000) 2019-10-13T10:50:16.835 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:16.833+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.833775+0000) 2019-10-13T10:50:16.835 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:16.833+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.833890+0000) 2019-10-13T10:50:16.835 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:16.833+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.834195+0000) 2019-10-13T10:50:16.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:16.833+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.834033+0000) 2019-10-13T10:50:16.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:16.834+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.835623+0000) 2019-10-13T10:50:16.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:16.834+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.834596+0000) 2019-10-13T10:50:16.837 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:16.834+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.834822+0000) 2019-10-13T10:50:16.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:16.836+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:16.837680+0000) 2019-10-13T10:50:16.842 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:16.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:18.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:18.568+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:18.569335+0000) 2019-10-13T10:50:18.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:18.905+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:18.905376+0000) 2019-10-13T10:50:21.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:21.519+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.519654+0000) 2019-10-13T10:50:21.844 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:21.842+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.843213+0000) 2019-10-13T10:50:21.844 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:21.842+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.843364+0000) 2019-10-13T10:50:21.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:21.843+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.844379+0000) 2019-10-13T10:50:21.845 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:21.843+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.843827+0000) 2019-10-13T10:50:21.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:21.843+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.843963+0000) 2019-10-13T10:50:21.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:21.843+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.844121+0000) 2019-10-13T10:50:21.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:21.844+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.844576+0000) 2019-10-13T10:50:21.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:21.846+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.847379+0000) 2019-10-13T10:50:21.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:21.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196289 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:23.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:23.008+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:23.008990+0000) 2019-10-13T10:50:23.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:23.011+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:23.011213+0000) 2019-10-13T10:50:24.892 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:24.891+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:24.892067+0000) 2019-10-13T10:50:25.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:25.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.660900+0000) 2019-10-13T10:50:25.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:25.660+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.660920+0000) 2019-10-13T10:50:26.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:26.852+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.853339+0000) 2019-10-13T10:50:26.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:26.853+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.853810+0000) 2019-10-13T10:50:26.855 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:26.853+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.853994+0000) 2019-10-13T10:50:26.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:26.853+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.854134+0000) 2019-10-13T10:50:26.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:26.853+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.854525+0000) 2019-10-13T10:50:26.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:26.853+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.854290+0000) 2019-10-13T10:50:26.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:26.854+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.854552+0000) 2019-10-13T10:50:26.857 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:26.855+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:26.856560+0000) 2019-10-13T10:50:26.863 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:26.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196322 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:28.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:28.569+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:28.569554+0000) 2019-10-13T10:50:28.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:28.905+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:28.905570+0000) 2019-10-13T10:50:31.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:31.519+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.519846+0000) 2019-10-13T10:50:31.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:31.863+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.863554+0000) 2019-10-13T10:50:31.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:31.863+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.863689+0000) 2019-10-13T10:50:31.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:31.863+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.864789+0000) 2019-10-13T10:50:31.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:31.863+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.864330+0000) 2019-10-13T10:50:31.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:31.864+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.864451+0000) 2019-10-13T10:50:31.866 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:31.864+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.864611+0000) 2019-10-13T10:50:31.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:31.864+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.864792+0000) 2019-10-13T10:50:31.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:31.865+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.867221+0000) 2019-10-13T10:50:31.873 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:31.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196351 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:33.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:33.009+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:33.009171+0000) 2019-10-13T10:50:33.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:33.010+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:33.011418+0000) 2019-10-13T10:50:34.894 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:34.893+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:34.894482+0000) 2019-10-13T10:50:35.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:35.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.661087+0000) 2019-10-13T10:50:35.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:35.660+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.661108+0000) 2019-10-13T10:50:36.875 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:36.872+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.873965+0000) 2019-10-13T10:50:36.875 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:36.872+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.874360+0000) 2019-10-13T10:50:36.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:36.873+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.875590+0000) 2019-10-13T10:50:36.876 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:36.873+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.874547+0000) 2019-10-13T10:50:36.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:36.873+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.874673+0000) 2019-10-13T10:50:36.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:36.873+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.874930+0000) 2019-10-13T10:50:36.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:36.873+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.875100+0000) 2019-10-13T10:50:36.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:36.876+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:36.878424+0000) 2019-10-13T10:50:36.883 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:36.881+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196384 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:38.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:38.568+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:38.569730+0000) 2019-10-13T10:50:38.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:38.904+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:38.905731+0000) 2019-10-13T10:50:41.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:41.519+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.520057+0000) 2019-10-13T10:50:41.885 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:41.882+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.884219+0000) 2019-10-13T10:50:41.885 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:41.883+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.884453+0000) 2019-10-13T10:50:41.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:41.884+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.885969+0000) 2019-10-13T10:50:41.887 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:41.883+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.884814+0000) 2019-10-13T10:50:41.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:41.883+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.884942+0000) 2019-10-13T10:50:41.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:41.883+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.885380+0000) 2019-10-13T10:50:41.887 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:41.884+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.885447+0000) 2019-10-13T10:50:41.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:41.885+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.887542+0000) 2019-10-13T10:50:41.893 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:41.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:43.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:43.008+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:43.009321+0000) 2019-10-13T10:50:43.011 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:43.010+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:43.011607+0000) 2019-10-13T10:50:44.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:44.896+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:44.896879+0000) 2019-10-13T10:50:45.432 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:50:45.433 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:45.437 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:50:45.437 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:45.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:45.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.661270+0000) 2019-10-13T10:50:45.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:45.660+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.661301+0000) 2019-10-13T10:50:46.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:46.892+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.893966+0000) 2019-10-13T10:50:46.895 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:46.892+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.894115+0000) 2019-10-13T10:50:46.895 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:46.892+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.894323+0000) 2019-10-13T10:50:46.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:46.894+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.895705+0000) 2019-10-13T10:50:46.897 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:46.893+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.894653+0000) 2019-10-13T10:50:46.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:46.893+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.895344+0000) 2019-10-13T10:50:46.897 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:46.893+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.895228+0000) 2019-10-13T10:50:46.898 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:46.897+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:46.897998+0000) 2019-10-13T10:50:46.903 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:46.900+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:48.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:48.569+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:48.569890+0000) 2019-10-13T10:50:48.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:48.904+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:48.905843+0000) 2019-10-13T10:50:51.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:51.519+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.520248+0000) 2019-10-13T10:50:51.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:51.902+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.904097+0000) 2019-10-13T10:50:51.906 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:51.902+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.904352+0000) 2019-10-13T10:50:51.906 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:51.903+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.904473+0000) 2019-10-13T10:50:51.906 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:51.903+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.904646+0000) 2019-10-13T10:50:51.906 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:51.903+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.905022+0000) 2019-10-13T10:50:51.907 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:51.905+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.906258+0000) 2019-10-13T10:50:51.907 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:51.904+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.905714+0000) 2019-10-13T10:50:51.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:51.906+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.907520+0000) 2019-10-13T10:50:51.913 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:51.911+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:53.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:53.008+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:53.009506+0000) 2019-10-13T10:50:53.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:53.011+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:53.011786+0000) 2019-10-13T10:50:54.900 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:54.899+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:54.899966+0000) 2019-10-13T10:50:55.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:55.660+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.661476+0000) 2019-10-13T10:50:55.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:55.660+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.661508+0000) 2019-10-13T10:50:56.917 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:50:56.913+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.915314+0000) 2019-10-13T10:50:56.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:50:56.914+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.915413+0000) 2019-10-13T10:50:56.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:56.914+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.915476+0000) 2019-10-13T10:50:56.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:50:56.914+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.915724+0000) 2019-10-13T10:50:56.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:50:56.914+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.915879+0000) 2019-10-13T10:50:56.918 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:56.914+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.916097+0000) 2019-10-13T10:50:56.918 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:50:56.915+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.916085+0000) 2019-10-13T10:50:56.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:50:56.917+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:56.918458+0000) 2019-10-13T10:50:56.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:50:56.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196510 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:50:58.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:50:58.569+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:58.570060+0000) 2019-10-13T10:50:58.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:50:58.905+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:58.905981+0000) 2019-10-13T10:51:01.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:01.519+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.520447+0000) 2019-10-13T10:51:01.923 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:01.920+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.922092+0000) 2019-10-13T10:51:01.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:01.920+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.922326+0000) 2019-10-13T10:51:01.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:01.921+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.922639+0000) 2019-10-13T10:51:01.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:01.921+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.922795+0000) 2019-10-13T10:51:01.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:01.922+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.923289+0000) 2019-10-13T10:51:01.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:01.922+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.923549+0000) 2019-10-13T10:51:01.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:01.922+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.923727+0000) 2019-10-13T10:51:01.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:01.925+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.926395+0000) 2019-10-13T10:51:01.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:01.929+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196541 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:03.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:03.009+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:03.009670+0000) 2019-10-13T10:51:03.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:03.011+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:03.011984+0000) 2019-10-13T10:51:04.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:04.901+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:04.902349+0000) 2019-10-13T10:51:05.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:05.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.661670+0000) 2019-10-13T10:51:05.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:05.661+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.661695+0000) 2019-10-13T10:51:06.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:06.931+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.932241+0000) 2019-10-13T10:51:06.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:06.932+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.933355+0000) 2019-10-13T10:51:06.934 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:06.932+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.932819+0000) 2019-10-13T10:51:06.934 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:06.932+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.932873+0000) 2019-10-13T10:51:06.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:06.932+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.933070+0000) 2019-10-13T10:51:06.935 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:06.932+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.933247+0000) 2019-10-13T10:51:06.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:06.932+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.933356+0000) 2019-10-13T10:51:06.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:06.934+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:06.935348+0000) 2019-10-13T10:51:06.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:06.939+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196574 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:08.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:08.569+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:08.570283+0000) 2019-10-13T10:51:08.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:08.905+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:08.906144+0000) 2019-10-13T10:51:11.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:11.520+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.520640+0000) 2019-10-13T10:51:11.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:11.940+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.941013+0000) 2019-10-13T10:51:11.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:11.941+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.942118+0000) 2019-10-13T10:51:11.943 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:11.941+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.941487+0000) 2019-10-13T10:51:11.943 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:11.941+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.941561+0000) 2019-10-13T10:51:11.943 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:11.941+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.942015+0000) 2019-10-13T10:51:11.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:11.941+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.941747+0000) 2019-10-13T10:51:11.944 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:11.941+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.942277+0000) 2019-10-13T10:51:11.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:11.944+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.945267+0000) 2019-10-13T10:51:11.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:11.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:13.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:13.009+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:13.009855+0000) 2019-10-13T10:51:13.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:13.011+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:13.012184+0000) 2019-10-13T10:51:14.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:14.904+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:14.904602+0000) 2019-10-13T10:51:15.588 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:51:15.589 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:15.593 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:51:15.593 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:15.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:15.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.661900+0000) 2019-10-13T10:51:15.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:15.661+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.661900+0000) 2019-10-13T10:51:16.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:16.950+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.950912+0000) 2019-10-13T10:51:16.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:16.951+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.952186+0000) 2019-10-13T10:51:16.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:16.951+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.951595+0000) 2019-10-13T10:51:16.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:16.951+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.951765+0000) 2019-10-13T10:51:16.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:16.951+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.952042+0000) 2019-10-13T10:51:16.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:16.951+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.952251+0000) 2019-10-13T10:51:16.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:16.950+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.951084+0000) 2019-10-13T10:51:16.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:16.954+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:16.955091+0000) 2019-10-13T10:51:16.961 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:16.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:18.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:18.570+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:18.570497+0000) 2019-10-13T10:51:18.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:18.906+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:18.906292+0000) 2019-10-13T10:51:21.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:21.519+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.520834+0000) 2019-10-13T10:51:21.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:21.960+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.961424+0000) 2019-10-13T10:51:21.963 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:21.960+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.961574+0000) 2019-10-13T10:51:21.963 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:21.960+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.961796+0000) 2019-10-13T10:51:21.964 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:21.962+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.963323+0000) 2019-10-13T10:51:21.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:21.960+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.962060+0000) 2019-10-13T10:51:21.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:21.960+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.962183+0000) 2019-10-13T10:51:21.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:21.960+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.962240+0000) 2019-10-13T10:51:21.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:21.964+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.965573+0000) 2019-10-13T10:51:21.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:21.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196669 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:23.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:23.009+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:23.010031+0000) 2019-10-13T10:51:23.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:23.011+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:23.012362+0000) 2019-10-13T10:51:24.907 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:24.906+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:24.907143+0000) 2019-10-13T10:51:25.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:25.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.662041+0000) 2019-10-13T10:51:25.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:25.661+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.662031+0000) 2019-10-13T10:51:26.970 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:26.967+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.969353+0000) 2019-10-13T10:51:26.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:26.968+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.969484+0000) 2019-10-13T10:51:26.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:26.968+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.970594+0000) 2019-10-13T10:51:26.972 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:26.968+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.970087+0000) 2019-10-13T10:51:26.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:26.968+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.970139+0000) 2019-10-13T10:51:26.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:26.968+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.970325+0000) 2019-10-13T10:51:26.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:26.969+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.970462+0000) 2019-10-13T10:51:26.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:26.971+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:26.973341+0000) 2019-10-13T10:51:26.978 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:26.976+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196700 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:28.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:28.569+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:28.570707+0000) 2019-10-13T10:51:28.907 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:28.905+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:28.906500+0000) 2019-10-13T10:51:31.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:31.520+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.520965+0000) 2019-10-13T10:51:31.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:31.978+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.980665+0000) 2019-10-13T10:51:31.982 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:31.978+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.980048+0000) 2019-10-13T10:51:31.982 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:31.978+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.980200+0000) 2019-10-13T10:51:31.982 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:31.978+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.980352+0000) 2019-10-13T10:51:31.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:31.979+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.980524+0000) 2019-10-13T10:51:31.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:31.979+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.980665+0000) 2019-10-13T10:51:31.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:31.979+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.980793+0000) 2019-10-13T10:51:31.983 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:31.980+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.982267+0000) 2019-10-13T10:51:31.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:31.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:33.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:33.009+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:33.010197+0000) 2019-10-13T10:51:33.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:33.011+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:33.012483+0000) 2019-10-13T10:51:34.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:34.909+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:34.910126+0000) 2019-10-13T10:51:35.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:35.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.662272+0000) 2019-10-13T10:51:35.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:35.661+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.662271+0000) 2019-10-13T10:51:36.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:36.987+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.989048+0000) 2019-10-13T10:51:36.990 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:36.987+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.989185+0000) 2019-10-13T10:51:36.991 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:36.987+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.989280+0000) 2019-10-13T10:51:36.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:36.988+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.989551+0000) 2019-10-13T10:51:36.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:36.989+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.990537+0000) 2019-10-13T10:51:36.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:36.988+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.989902+0000) 2019-10-13T10:51:36.992 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:36.988+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.989940+0000) 2019-10-13T10:51:36.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:36.992+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:36.993512+0000) 2019-10-13T10:51:36.997 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:36.994+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196762 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:38.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:38.570+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:38.570921+0000) 2019-10-13T10:51:38.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:38.906+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:38.906670+0000) 2019-10-13T10:51:41.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:41.520+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.521121+0000) 2019-10-13T10:51:41.999 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:41.996+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.997672+0000) 2019-10-13T10:51:41.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:41.996+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.997554+0000) 2019-10-13T10:51:41.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:41.996+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.997903+0000) 2019-10-13T10:51:42.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:41.996+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.998152+0000) 2019-10-13T10:51:42.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:41.996+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.998267+0000) 2019-10-13T10:51:42.000 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:41.998+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.999417+0000) 2019-10-13T10:51:42.001 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:41.997+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.999021+0000) 2019-10-13T10:51:42.001 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:42.001+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:42.001765+0000) 2019-10-13T10:51:42.007 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:42.005+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:43.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:43.009+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:43.010363+0000) 2019-10-13T10:51:43.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:43.011+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:43.012680+0000) 2019-10-13T10:51:44.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:44.911+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:44.912443+0000) 2019-10-13T10:51:45.663 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:45.661+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.662454+0000) 2019-10-13T10:51:45.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:45.661+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.662451+0000) 2019-10-13T10:51:45.694 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:51:45.694 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:45.698 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:51:45.699 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:47.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:47.008+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.008938+0000) 2019-10-13T10:51:47.010 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:47.006+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.008375+0000) 2019-10-13T10:51:47.010 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:47.007+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.008524+0000) 2019-10-13T10:51:47.010 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:47.007+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.008759+0000) 2019-10-13T10:51:47.010 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:47.007+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.009129+0000) 2019-10-13T10:51:47.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:47.010+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.010985+0000) 2019-10-13T10:51:47.011 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:47.007+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.009239+0000) 2019-10-13T10:51:47.011 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:47.007+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.009223+0000) 2019-10-13T10:51:47.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:47.014+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:48.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:48.570+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:48.571099+0000) 2019-10-13T10:51:48.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:48.906+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:48.906859+0000) 2019-10-13T10:51:51.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:51.520+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:51.521323+0000) 2019-10-13T10:51:52.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:52.018+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.018849+0000) 2019-10-13T10:51:52.019 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:52.017+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.018167+0000) 2019-10-13T10:51:52.020 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:52.017+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.018305+0000) 2019-10-13T10:51:52.020 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:52.018+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.018563+0000) 2019-10-13T10:51:52.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:52.018+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.018967+0000) 2019-10-13T10:51:52.021 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:52.018+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.018877+0000) 2019-10-13T10:51:52.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:52.018+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.019257+0000) 2019-10-13T10:51:52.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:52.020+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:52.021512+0000) 2019-10-13T10:51:52.027 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:52.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:53.011 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:53.010+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:53.010560+0000) 2019-10-13T10:51:53.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:53.012+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:53.012884+0000) 2019-10-13T10:51:54.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:54.914+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:54.914823+0000) 2019-10-13T10:51:55.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:55.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.662645+0000) 2019-10-13T10:51:55.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:55.662+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.662645+0000) 2019-10-13T10:51:57.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:51:57.028+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.028403+0000) 2019-10-13T10:51:57.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:51:57.028+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.029565+0000) 2019-10-13T10:51:57.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:51:57.028+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.028924+0000) 2019-10-13T10:51:57.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:57.028+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.029325+0000) 2019-10-13T10:51:57.031 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:51:57.029+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.029514+0000) 2019-10-13T10:51:57.031 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:57.029+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.029654+0000) 2019-10-13T10:51:57.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:51:57.029+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.029834+0000) 2019-10-13T10:51:57.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:51:57.030+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.031536+0000) 2019-10-13T10:51:57.037 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:51:57.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:51:58.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:51:58.571+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:58.571330+0000) 2019-10-13T10:51:58.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:51:58.906+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:58.907071+0000) 2019-10-13T10:52:01.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:01.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:01.521457+0000) 2019-10-13T10:52:02.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:02.038+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.038438+0000) 2019-10-13T10:52:02.039 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:02.038+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.038594+0000) 2019-10-13T10:52:02.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:02.039+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.040152+0000) 2019-10-13T10:52:02.041 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:02.038+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.039092+0000) 2019-10-13T10:52:02.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:02.038+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.039371+0000) 2019-10-13T10:52:02.041 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:02.039+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.039490+0000) 2019-10-13T10:52:02.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:02.039+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.039725+0000) 2019-10-13T10:52:02.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:02.041+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:02.041856+0000) 2019-10-13T10:52:02.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:02.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:03.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:03.010+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:03.010724+0000) 2019-10-13T10:52:03.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:03.012+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:03.013043+0000) 2019-10-13T10:52:04.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:04.916+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:04.917144+0000) 2019-10-13T10:52:05.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:05.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.662842+0000) 2019-10-13T10:52:05.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:05.662+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.662841+0000) 2019-10-13T10:52:07.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:07.044+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.046352+0000) 2019-10-13T10:52:07.048 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:07.045+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.046847+0000) 2019-10-13T10:52:07.048 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:07.045+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.047028+0000) 2019-10-13T10:52:07.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:07.045+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.047297+0000) 2019-10-13T10:52:07.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:07.046+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.047512+0000) 2019-10-13T10:52:07.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:07.048+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.048727+0000) 2019-10-13T10:52:07.049 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:07.046+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.047700+0000) 2019-10-13T10:52:07.050 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:07.049+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.049743+0000) 2019-10-13T10:52:07.055 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:07.053+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196952 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:08.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:08.570+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:08.571515+0000) 2019-10-13T10:52:08.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:08.905+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:08.907224+0000) 2019-10-13T10:52:11.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:11.520+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:11.521640+0000) 2019-10-13T10:52:12.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:12.054+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.056069+0000) 2019-10-13T10:52:12.057 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:12.054+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.056287+0000) 2019-10-13T10:52:12.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:12.056+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.057549+0000) 2019-10-13T10:52:12.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:12.055+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.056549+0000) 2019-10-13T10:52:12.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:12.055+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.056882+0000) 2019-10-13T10:52:12.059 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:12.055+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.057350+0000) 2019-10-13T10:52:12.059 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:12.056+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.057516+0000) 2019-10-13T10:52:12.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:12.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:12.060824+0000) 2019-10-13T10:52:12.067 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:12.063+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 196983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:13.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:13.009+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:13.010888+0000) 2019-10-13T10:52:13.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:13.013+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:13.013201+0000) 2019-10-13T10:52:14.919 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:14.918+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:14.919421+0000) 2019-10-13T10:52:15.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:15.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.663033+0000) 2019-10-13T10:52:15.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:15.662+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.663033+0000) 2019-10-13T10:52:15.839 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:52:15.839 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:15.843 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:52:15.843 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:17.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:17.064+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.066308+0000) 2019-10-13T10:52:17.068 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:17.065+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.066488+0000) 2019-10-13T10:52:17.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:17.066+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.068030+0000) 2019-10-13T10:52:17.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:17.065+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.067019+0000) 2019-10-13T10:52:17.069 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:17.066+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.068092+0000) 2019-10-13T10:52:17.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:17.066+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.068247+0000) 2019-10-13T10:52:17.069 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:17.066+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.067688+0000) 2019-10-13T10:52:17.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:17.068+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.070361+0000) 2019-10-13T10:52:17.075 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:17.073+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:18.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:18.570+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:18.571697+0000) 2019-10-13T10:52:18.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:18.906+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:18.907404+0000) 2019-10-13T10:52:21.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:21.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:21.521826+0000) 2019-10-13T10:52:22.077 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:22.074+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.076037+0000) 2019-10-13T10:52:22.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:22.075+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.077551+0000) 2019-10-13T10:52:22.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:22.075+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.076568+0000) 2019-10-13T10:52:22.078 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:22.075+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.076890+0000) 2019-10-13T10:52:22.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:22.075+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.077037+0000) 2019-10-13T10:52:22.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:22.075+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.077185+0000) 2019-10-13T10:52:22.079 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:22.075+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.077321+0000) 2019-10-13T10:52:22.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:22.077+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:22.079098+0000) 2019-10-13T10:52:22.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:22.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197047 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:23.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:23.010+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:23.011032+0000) 2019-10-13T10:52:23.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:23.012+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:23.013368+0000) 2019-10-13T10:52:24.922 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:24.920+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:24.921766+0000) 2019-10-13T10:52:25.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:25.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.663225+0000) 2019-10-13T10:52:25.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:25.662+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.663225+0000) 2019-10-13T10:52:27.088 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:27.085+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.087371+0000) 2019-10-13T10:52:27.089 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:27.085+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.087364+0000) 2019-10-13T10:52:27.089 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:27.086+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.087645+0000) 2019-10-13T10:52:27.089 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:27.086+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.087946+0000) 2019-10-13T10:52:27.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:27.086+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.088612+0000) 2019-10-13T10:52:27.090 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:27.087+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.088567+0000) 2019-10-13T10:52:27.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:27.087+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.088714+0000) 2019-10-13T10:52:27.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:27.090+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.091024+0000) 2019-10-13T10:52:27.097 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:27.094+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:28.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:28.570+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:28.571883+0000) 2019-10-13T10:52:28.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:28.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:28.907581+0000) 2019-10-13T10:52:31.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:31.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.522059+0000) 2019-10-13T10:52:32.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:32.097+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.098611+0000) 2019-10-13T10:52:32.099 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:32.096+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.098208+0000) 2019-10-13T10:52:32.099 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:32.097+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.098570+0000) 2019-10-13T10:52:32.100 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:32.097+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.098879+0000) 2019-10-13T10:52:32.100 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:32.097+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.099062+0000) 2019-10-13T10:52:32.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:32.097+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.099210+0000) 2019-10-13T10:52:32.101 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:32.096+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.098116+0000) 2019-10-13T10:52:32.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:32.100+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:32.100870+0000) 2019-10-13T10:52:32.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:32.104+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197109 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:33.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:33.010+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:33.011199+0000) 2019-10-13T10:52:33.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:33.012+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:33.013511+0000) 2019-10-13T10:52:34.924 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:34.923+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:34.924050+0000) 2019-10-13T10:52:35.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:35.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.663434+0000) 2019-10-13T10:52:35.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:35.663+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:35.663432+0000) 2019-10-13T10:52:37.109 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:37.107+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.107625+0000) 2019-10-13T10:52:37.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:37.107+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.107855+0000) 2019-10-13T10:52:37.109 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:37.107+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.107937+0000) 2019-10-13T10:52:37.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:37.108+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.109213+0000) 2019-10-13T10:52:37.110 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:37.108+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.108561+0000) 2019-10-13T10:52:37.110 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:37.108+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.108848+0000) 2019-10-13T10:52:37.111 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:37.108+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.109263+0000) 2019-10-13T10:52:37.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:37.110+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.111348+0000) 2019-10-13T10:52:37.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:37.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197140 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:38.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:38.571+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:38.572058+0000) 2019-10-13T10:52:38.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:38.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:38.907761+0000) 2019-10-13T10:52:41.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:41.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.522226+0000) 2019-10-13T10:52:42.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:42.117+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.118619+0000) 2019-10-13T10:52:42.119 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:42.117+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.117948+0000) 2019-10-13T10:52:42.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:42.117+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.118104+0000) 2019-10-13T10:52:42.120 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:42.117+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.118266+0000) 2019-10-13T10:52:42.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:42.118+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.118825+0000) 2019-10-13T10:52:42.120 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:42.118+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.119155+0000) 2019-10-13T10:52:42.121 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:42.118+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.119262+0000) 2019-10-13T10:52:42.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:42.120+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:42.121518+0000) 2019-10-13T10:52:42.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:42.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197173 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:43.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:43.010+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:43.011333+0000) 2019-10-13T10:52:43.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:43.012+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:43.013662+0000) 2019-10-13T10:52:44.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:44.926+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:44.926442+0000) 2019-10-13T10:52:45.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:45.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.663643+0000) 2019-10-13T10:52:45.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:45.663+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:45.663638+0000) 2019-10-13T10:52:45.985 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:52:45.985 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:45.989 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:52:45.989 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:47.128 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:47.126+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.127163+0000) 2019-10-13T10:52:47.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:47.126+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.127300+0000) 2019-10-13T10:52:47.129 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:47.127+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.127480+0000) 2019-10-13T10:52:47.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:47.127+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.128643+0000) 2019-10-13T10:52:47.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:47.127+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.128097+0000) 2019-10-13T10:52:47.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:47.127+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.128357+0000) 2019-10-13T10:52:47.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:47.128+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.128565+0000) 2019-10-13T10:52:47.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:47.130+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.131610+0000) 2019-10-13T10:52:47.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:47.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197204 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:48.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:48.572+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:48.572228+0000) 2019-10-13T10:52:48.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:48.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:48.907934+0000) 2019-10-13T10:52:51.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:51.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.522383+0000) 2019-10-13T10:52:52.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:52.136+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.136982+0000) 2019-10-13T10:52:52.138 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:52.136+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.137100+0000) 2019-10-13T10:52:52.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:52.136+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.137205+0000) 2019-10-13T10:52:52.139 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:52.136+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.137334+0000) 2019-10-13T10:52:52.139 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:52.138+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.138815+0000) 2019-10-13T10:52:52.140 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:52.137+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.137831+0000) 2019-10-13T10:52:52.140 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:52.137+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.137820+0000) 2019-10-13T10:52:52.141 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:52.140+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:52.141209+0000) 2019-10-13T10:52:52.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:52.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:53.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:53.010+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:53.011476+0000) 2019-10-13T10:52:53.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:53.013+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:53.013802+0000) 2019-10-13T10:52:54.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:54.928+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:54.928830+0000) 2019-10-13T10:52:55.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:55.662+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.663859+0000) 2019-10-13T10:52:55.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:55.662+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:55.663859+0000) 2019-10-13T10:52:57.147 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:52:57.146+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.147500+0000) 2019-10-13T10:52:57.148 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:57.145+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.147081+0000) 2019-10-13T10:52:57.149 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:52:57.145+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.147235+0000) 2019-10-13T10:52:57.149 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:52:57.145+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.146873+0000) 2019-10-13T10:52:57.149 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:52:57.146+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.147491+0000) 2019-10-13T10:52:57.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:52:57.146+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.147725+0000) 2019-10-13T10:52:57.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:57.146+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.147667+0000) 2019-10-13T10:52:57.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:52:57.150+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.151052+0000) 2019-10-13T10:52:57.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:52:57.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197266 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:52:58.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:52:58.571+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:58.572390+0000) 2019-10-13T10:52:58.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:52:58.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:58.908091+0000) 2019-10-13T10:53:01.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:01.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.522545+0000) 2019-10-13T10:53:02.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:02.155+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.157130+0000) 2019-10-13T10:53:02.159 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:02.158+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.158755+0000) 2019-10-13T10:53:02.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:02.156+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.157561+0000) 2019-10-13T10:53:02.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:02.155+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.157379+0000) 2019-10-13T10:53:02.160 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:02.156+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.158248+0000) 2019-10-13T10:53:02.160 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:02.156+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.158307+0000) 2019-10-13T10:53:02.160 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:02.157+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.158476+0000) 2019-10-13T10:53:02.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:02.159+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:02.160698+0000) 2019-10-13T10:53:02.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:02.163+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:03.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:03.010+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:03.011653+0000) 2019-10-13T10:53:03.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:03.013+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:03.013942+0000) 2019-10-13T10:53:04.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:04.930+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:04.931125+0000) 2019-10-13T10:53:05.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:05.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.664071+0000) 2019-10-13T10:53:05.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:05.663+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:05.664071+0000) 2019-10-13T10:53:07.168 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:07.165+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.166672+0000) 2019-10-13T10:53:07.169 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:07.165+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.167084+0000) 2019-10-13T10:53:07.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:07.166+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.167435+0000) 2019-10-13T10:53:07.169 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:07.166+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.167550+0000) 2019-10-13T10:53:07.170 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:07.166+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.167969+0000) 2019-10-13T10:53:07.170 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:07.166+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.167931+0000) 2019-10-13T10:53:07.171 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:07.166+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.168091+0000) 2019-10-13T10:53:07.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:07.168+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.170655+0000) 2019-10-13T10:53:07.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:07.173+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197330 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:08.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:08.571+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:08.572591+0000) 2019-10-13T10:53:08.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:08.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:08.908246+0000) 2019-10-13T10:53:11.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:11.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.522652+0000) 2019-10-13T10:53:12.178 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:12.176+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.177815+0000) 2019-10-13T10:53:12.179 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:12.175+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.177206+0000) 2019-10-13T10:53:12.179 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:12.175+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.177362+0000) 2019-10-13T10:53:12.179 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:12.176+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.177634+0000) 2019-10-13T10:53:12.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:12.176+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.178024+0000) 2019-10-13T10:53:12.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:12.176+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.178242+0000) 2019-10-13T10:53:12.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:12.178+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.180053+0000) 2019-10-13T10:53:12.180 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:12.177+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:12.178422+0000) 2019-10-13T10:53:12.186 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:12.184+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:13.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:13.011+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:13.011852+0000) 2019-10-13T10:53:13.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:13.013+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:13.014073+0000) 2019-10-13T10:53:14.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:14.932+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:14.933465+0000) 2019-10-13T10:53:15.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:15.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.664275+0000) 2019-10-13T10:53:15.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:15.663+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:15.664293+0000) 2019-10-13T10:53:16.088 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:53:16.088 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:16.092 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:53:16.092 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:17.188 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:17.185+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.186910+0000) 2019-10-13T10:53:17.188 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:17.185+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.187103+0000) 2019-10-13T10:53:17.189 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:17.186+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.188397+0000) 2019-10-13T10:53:17.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:17.186+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.188184+0000) 2019-10-13T10:53:17.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:17.186+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.188338+0000) 2019-10-13T10:53:17.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:17.186+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.187797+0000) 2019-10-13T10:53:17.190 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:17.187+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.188688+0000) 2019-10-13T10:53:17.191 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:17.189+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.191712+0000) 2019-10-13T10:53:17.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:17.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197392 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:18.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:18.572+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:18.572784+0000) 2019-10-13T10:53:18.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:18.908+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:18.908501+0000) 2019-10-13T10:53:21.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:21.522+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.522798+0000) 2019-10-13T10:53:22.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:22.197+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.197753+0000) 2019-10-13T10:53:22.199 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:22.197+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.197969+0000) 2019-10-13T10:53:22.199 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:22.197+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.198061+0000) 2019-10-13T10:53:22.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:22.198+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.199701+0000) 2019-10-13T10:53:22.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:22.198+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.198467+0000) 2019-10-13T10:53:22.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:22.198+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.198735+0000) 2019-10-13T10:53:22.201 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:22.198+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.199106+0000) 2019-10-13T10:53:22.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:22.199+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:22.200602+0000) 2019-10-13T10:53:22.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:22.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:23.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:23.011+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:23.012018+0000) 2019-10-13T10:53:23.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:23.013+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:23.014182+0000) 2019-10-13T10:53:24.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:24.935+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:24.935884+0000) 2019-10-13T10:53:25.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:25.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.664484+0000) 2019-10-13T10:53:25.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:25.664+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:25.664491+0000) 2019-10-13T10:53:27.208 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:27.207+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.207485+0000) 2019-10-13T10:53:27.209 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:27.207+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.208562+0000) 2019-10-13T10:53:27.210 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:27.207+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.208166+0000) 2019-10-13T10:53:27.210 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:27.207+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.208005+0000) 2019-10-13T10:53:27.210 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:27.208+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.208562+0000) 2019-10-13T10:53:27.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:27.208+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.208784+0000) 2019-10-13T10:53:27.210 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:27.208+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.208960+0000) 2019-10-13T10:53:27.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:27.210+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.211245+0000) 2019-10-13T10:53:27.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:27.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197456 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:28.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:28.572+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:28.572974+0000) 2019-10-13T10:53:28.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:28.908+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:28.908655+0000) 2019-10-13T10:53:31.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:31.522+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.522971+0000) 2019-10-13T10:53:32.218 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:32.217+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.218168+0000) 2019-10-13T10:53:32.219 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:32.217+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.217565+0000) 2019-10-13T10:53:32.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:32.217+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.217737+0000) 2019-10-13T10:53:32.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:32.217+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.218035+0000) 2019-10-13T10:53:32.220 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:32.217+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.218162+0000) 2019-10-13T10:53:32.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:32.218+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.218735+0000) 2019-10-13T10:53:32.220 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:32.218+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.218745+0000) 2019-10-13T10:53:32.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:32.221+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:32.221779+0000) 2019-10-13T10:53:32.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:32.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:33.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:33.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:33.012181+0000) 2019-10-13T10:53:33.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:33.013+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:33.014327+0000) 2019-10-13T10:53:34.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:34.937+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:34.938378+0000) 2019-10-13T10:53:35.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:35.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.664678+0000) 2019-10-13T10:53:35.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:35.664+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:35.664724+0000) 2019-10-13T10:53:37.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:37.227+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.227474+0000) 2019-10-13T10:53:37.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:37.227+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.227624+0000) 2019-10-13T10:53:37.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:37.227+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.228041+0000) 2019-10-13T10:53:37.230 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:37.227+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.228196+0000) 2019-10-13T10:53:37.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:37.227+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.228024+0000) 2019-10-13T10:53:37.230 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:37.227+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.228316+0000) 2019-10-13T10:53:37.230 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:37.227+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.228655+0000) 2019-10-13T10:53:37.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:37.231+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.232153+0000) 2019-10-13T10:53:37.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:37.233+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197520 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:38.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:38.572+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:38.573144+0000) 2019-10-13T10:53:38.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:38.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:38.908894+0000) 2019-10-13T10:53:41.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:41.522+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.523134+0000) 2019-10-13T10:53:42.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:42.233+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.235289+0000) 2019-10-13T10:53:42.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:42.234+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.235511+0000) 2019-10-13T10:53:42.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:42.234+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.236103+0000) 2019-10-13T10:53:42.237 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:42.234+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.236254+0000) 2019-10-13T10:53:42.237 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:42.235+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.236504+0000) 2019-10-13T10:53:42.238 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:42.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.237210+0000) 2019-10-13T10:53:42.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:42.235+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.237029+0000) 2019-10-13T10:53:42.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:42.239+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:42.240357+0000) 2019-10-13T10:53:42.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:42.242+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197551 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:43.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:43.011+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:43.012318+0000) 2019-10-13T10:53:43.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:43.014+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:43.014449+0000) 2019-10-13T10:53:44.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:44.940+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:44.940798+0000) 2019-10-13T10:53:45.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:45.663+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.664876+0000) 2019-10-13T10:53:45.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:45.663+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:45.664883+0000) 2019-10-13T10:53:46.263 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:53:46.263 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:46.267 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:53:46.267 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:47.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:47.244+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.245596+0000) 2019-10-13T10:53:47.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:47.246+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.247189+0000) 2019-10-13T10:53:47.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:47.244+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.246272+0000) 2019-10-13T10:53:47.248 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:47.245+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.246649+0000) 2019-10-13T10:53:47.248 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:47.245+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.246765+0000) 2019-10-13T10:53:47.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:47.245+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.246958+0000) 2019-10-13T10:53:47.249 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:47.245+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.247294+0000) 2019-10-13T10:53:47.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:47.248+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.249086+0000) 2019-10-13T10:53:47.255 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:47.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197582 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:48.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:48.572+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:48.573353+0000) 2019-10-13T10:53:48.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:48.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:48.909101+0000) 2019-10-13T10:53:51.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:51.521+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.523288+0000) 2019-10-13T10:53:52.257 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:52.254+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.256164+0000) 2019-10-13T10:53:52.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:52.254+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.256355+0000) 2019-10-13T10:53:52.258 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:52.255+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.256578+0000) 2019-10-13T10:53:52.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:52.257+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.257797+0000) 2019-10-13T10:53:52.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:52.255+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.257059+0000) 2019-10-13T10:53:52.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:52.255+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.257250+0000) 2019-10-13T10:53:52.259 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:52.256+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.257437+0000) 2019-10-13T10:53:52.260 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:52.259+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:52.259770+0000) 2019-10-13T10:53:52.265 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:52.263+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:53.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:53.011+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:53.012448+0000) 2019-10-13T10:53:53.014 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:53.014+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:53.014602+0000) 2019-10-13T10:53:54.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:54.943+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:54.943116+0000) 2019-10-13T10:53:55.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:55.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.665094+0000) 2019-10-13T10:53:55.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:55.664+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:55.665102+0000) 2019-10-13T10:53:57.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:57.264+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.266356+0000) 2019-10-13T10:53:57.268 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:57.265+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.266539+0000) 2019-10-13T10:53:57.268 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:53:57.265+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.266829+0000) 2019-10-13T10:53:57.268 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:53:57.265+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.266947+0000) 2019-10-13T10:53:57.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:53:57.266+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.268366+0000) 2019-10-13T10:53:57.269 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:53:57.265+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.267271+0000) 2019-10-13T10:53:57.269 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:53:57.266+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.267629+0000) 2019-10-13T10:53:57.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:53:57.267+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.269691+0000) 2019-10-13T10:53:57.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:53:57.273+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:53:58.574 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:53:58.572+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:58.573563+0000) 2019-10-13T10:53:58.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:53:58.907+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:58.909323+0000) 2019-10-13T10:54:01.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:01.522+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.523484+0000) 2019-10-13T10:54:02.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:02.276+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.278329+0000) 2019-10-13T10:54:02.283 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:02.279+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.281034+0000) 2019-10-13T10:54:02.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:02.275+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.276673+0000) 2019-10-13T10:54:02.283 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:02.275+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.276884+0000) 2019-10-13T10:54:02.283 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:02.275+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.276999+0000) 2019-10-13T10:54:02.283 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:02.275+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.277172+0000) 2019-10-13T10:54:02.284 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:02.276+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.277597+0000) 2019-10-13T10:54:02.284 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:02.276+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:02.277768+0000) 2019-10-13T10:54:02.285 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:02.283+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:03.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:03.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:03.012574+0000) 2019-10-13T10:54:03.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:03.013+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:03.014759+0000) 2019-10-13T10:54:04.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:04.944+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:04.945399+0000) 2019-10-13T10:54:05.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:05.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.665307+0000) 2019-10-13T10:54:05.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:05.664+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:05.665306+0000) 2019-10-13T10:54:07.296 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:07.285+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.287580+0000) 2019-10-13T10:54:07.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:07.287+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.289209+0000) 2019-10-13T10:54:07.297 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:07.285+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.286315+0000) 2019-10-13T10:54:07.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:07.286+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.286487+0000) 2019-10-13T10:54:07.297 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:07.286+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.287133+0000) 2019-10-13T10:54:07.297 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:07.286+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.287064+0000) 2019-10-13T10:54:07.298 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:07.287+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.287487+0000) 2019-10-13T10:54:07.298 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:07.287+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.287473+0000) 2019-10-13T10:54:07.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:07.294+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:08.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:08.573+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:08.573740+0000) 2019-10-13T10:54:08.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:08.909+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:08.909484+0000) 2019-10-13T10:54:11.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:11.523+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.523647+0000) 2019-10-13T10:54:12.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:12.296+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.296918+0000) 2019-10-13T10:54:12.299 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:12.297+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.297905+0000) 2019-10-13T10:54:12.299 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:12.296+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.297257+0000) 2019-10-13T10:54:12.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:12.297+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.297442+0000) 2019-10-13T10:54:12.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:12.297+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.297612+0000) 2019-10-13T10:54:12.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:12.297+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.297782+0000) 2019-10-13T10:54:12.300 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:12.297+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.297964+0000) 2019-10-13T10:54:12.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:12.300+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:12.301092+0000) 2019-10-13T10:54:12.305 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:12.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:13.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:13.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:13.012714+0000) 2019-10-13T10:54:13.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:13.014+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:13.014918+0000) 2019-10-13T10:54:14.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:14.947+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:14.947841+0000) 2019-10-13T10:54:15.666 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:15.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.665508+0000) 2019-10-13T10:54:15.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:15.665+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:15.665505+0000) 2019-10-13T10:54:16.415 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:54:16.415 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:16.424 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:54:16.424 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:17.308 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:17.306+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.306529+0000) 2019-10-13T10:54:17.308 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:17.306+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.306756+0000) 2019-10-13T10:54:17.309 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:17.307+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.308157+0000) 2019-10-13T10:54:17.309 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:17.306+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.307311+0000) 2019-10-13T10:54:17.309 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:17.307+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.307690+0000) 2019-10-13T10:54:17.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:17.307+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.307923+0000) 2019-10-13T10:54:17.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:17.307+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.308128+0000) 2019-10-13T10:54:17.310 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:17.309+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.309829+0000) 2019-10-13T10:54:17.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:17.314+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:18.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:18.573+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:18.573938+0000) 2019-10-13T10:54:18.910 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:18.909+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:18.909629+0000) 2019-10-13T10:54:21.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:21.523+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.523840+0000) 2019-10-13T10:54:22.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:22.316+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.316448+0000) 2019-10-13T10:54:22.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:22.316+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.316797+0000) 2019-10-13T10:54:22.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:22.316+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.316936+0000) 2019-10-13T10:54:22.320 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:22.316+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.317230+0000) 2019-10-13T10:54:22.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:22.317+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.317461+0000) 2019-10-13T10:54:22.320 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:22.317+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.317855+0000) 2019-10-13T10:54:22.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:22.316+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.317116+0000) 2019-10-13T10:54:22.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:22.319+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:22.320616+0000) 2019-10-13T10:54:22.326 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:22.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:23.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:23.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:23.012913+0000) 2019-10-13T10:54:23.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:23.014+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:23.015076+0000) 2019-10-13T10:54:24.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:24.949+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:24.950135+0000) 2019-10-13T10:54:25.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:25.664+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.665742+0000) 2019-10-13T10:54:25.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:25.664+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:25.665716+0000) 2019-10-13T10:54:27.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:27.325+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.327384+0000) 2019-10-13T10:54:27.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:27.326+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.327618+0000) 2019-10-13T10:54:27.330 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:27.326+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.327726+0000) 2019-10-13T10:54:27.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:27.326+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.327964+0000) 2019-10-13T10:54:27.330 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:27.326+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.327985+0000) 2019-10-13T10:54:27.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:27.326+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.328217+0000) 2019-10-13T10:54:27.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:27.326+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.327664+0000) 2019-10-13T10:54:27.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:27.329+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.329927+0000) 2019-10-13T10:54:27.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:27.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:28.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:28.573+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:28.574135+0000) 2019-10-13T10:54:28.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:28.908+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:28.909856+0000) 2019-10-13T10:54:31.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:31.522+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.523997+0000) 2019-10-13T10:54:32.338 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:32.337+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.337996+0000) 2019-10-13T10:54:32.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:32.336+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.337581+0000) 2019-10-13T10:54:32.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:32.336+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.337968+0000) 2019-10-13T10:54:32.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:32.336+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.338085+0000) 2019-10-13T10:54:32.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:32.337+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.338435+0000) 2019-10-13T10:54:32.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:32.336+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.338266+0000) 2019-10-13T10:54:32.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:32.339+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.340633+0000) 2019-10-13T10:54:32.351 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:32.337+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:32.338875+0000) 2019-10-13T10:54:32.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:32.343+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:33.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:33.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:33.013032+0000) 2019-10-13T10:54:33.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:33.014+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:33.015266+0000) 2019-10-13T10:54:34.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:34.952+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:34.952556+0000) 2019-10-13T10:54:35.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:35.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.665919+0000) 2019-10-13T10:54:35.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:35.665+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:35.665918+0000) 2019-10-13T10:54:37.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:37.345+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.346870+0000) 2019-10-13T10:54:37.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:37.345+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.347000+0000) 2019-10-13T10:54:37.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:37.346+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.347406+0000) 2019-10-13T10:54:37.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:37.346+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.347626+0000) 2019-10-13T10:54:37.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:37.346+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.347674+0000) 2019-10-13T10:54:37.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:37.346+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.347950+0000) 2019-10-13T10:54:37.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:37.347+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.348107+0000) 2019-10-13T10:54:37.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:37.350+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.351487+0000) 2019-10-13T10:54:37.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:37.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:38.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:38.573+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:38.574350+0000) 2019-10-13T10:54:38.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:38.909+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:38.909958+0000) 2019-10-13T10:54:41.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:41.523+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.524104+0000) 2019-10-13T10:54:42.356 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:42.354+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.355617+0000) 2019-10-13T10:54:42.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:42.356+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.357418+0000) 2019-10-13T10:54:42.358 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:42.354+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.356170+0000) 2019-10-13T10:54:42.358 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:42.355+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.356436+0000) 2019-10-13T10:54:42.358 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:42.355+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.356908+0000) 2019-10-13T10:54:42.359 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:42.355+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.356774+0000) 2019-10-13T10:54:42.359 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:42.355+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.357371+0000) 2019-10-13T10:54:42.360 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:42.358+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:42.358765+0000) 2019-10-13T10:54:42.370 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:42.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:43.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:43.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:43.013174+0000) 2019-10-13T10:54:43.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:43.014+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:43.015429+0000) 2019-10-13T10:54:44.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:44.954+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:44.954931+0000) 2019-10-13T10:54:45.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:45.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.666103+0000) 2019-10-13T10:54:45.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:45.665+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:45.666109+0000) 2019-10-13T10:54:46.564 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:54:46.564 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:46.569 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:54:46.569 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:47.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:47.364+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.366121+0000) 2019-10-13T10:54:47.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:47.365+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.367240+0000) 2019-10-13T10:54:47.368 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:47.364+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.366228+0000) 2019-10-13T10:54:47.368 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:47.365+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.366572+0000) 2019-10-13T10:54:47.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:47.365+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.366936+0000) 2019-10-13T10:54:47.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:47.365+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.367273+0000) 2019-10-13T10:54:47.369 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:47.365+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.367067+0000) 2019-10-13T10:54:47.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:47.369+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.370884+0000) 2019-10-13T10:54:47.375 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:47.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197948 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:48.575 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:48.574+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:48.574587+0000) 2019-10-13T10:54:48.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:48.909+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:48.910053+0000) 2019-10-13T10:54:51.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:51.523+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.524322+0000) 2019-10-13T10:54:52.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:52.375+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.375669+0000) 2019-10-13T10:54:52.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:52.375+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.375906+0000) 2019-10-13T10:54:52.378 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:52.375+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.377282+0000) 2019-10-13T10:54:52.378 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:52.376+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.376856+0000) 2019-10-13T10:54:52.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:52.376+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.377117+0000) 2019-10-13T10:54:52.378 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:52.376+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.377211+0000) 2019-10-13T10:54:52.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:52.376+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.377281+0000) 2019-10-13T10:54:52.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:52.377+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:52.379192+0000) 2019-10-13T10:54:52.385 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:52.384+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 197979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:53.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:53.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:53.013311+0000) 2019-10-13T10:54:53.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:53.014+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:53.015589+0000) 2019-10-13T10:54:54.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:54.956+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:54.957277+0000) 2019-10-13T10:54:55.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:55.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.666254+0000) 2019-10-13T10:54:55.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:55.665+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:55.666224+0000) 2019-10-13T10:54:57.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:54:57.385+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.385988+0000) 2019-10-13T10:54:57.387 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:54:57.385+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.386352+0000) 2019-10-13T10:54:57.388 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:54:57.385+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.387426+0000) 2019-10-13T10:54:57.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:57.386+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.386887+0000) 2019-10-13T10:54:57.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:54:57.386+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.386966+0000) 2019-10-13T10:54:57.389 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:54:57.386+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.387094+0000) 2019-10-13T10:54:57.389 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:57.386+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.387162+0000) 2019-10-13T10:54:57.390 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:54:57.388+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.390413+0000) 2019-10-13T10:54:57.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:54:57.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:54:58.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:54:58.574+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:58.574778+0000) 2019-10-13T10:54:58.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:54:58.909+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:58.910221+0000) 2019-10-13T10:55:01.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:01.524+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.524526+0000) 2019-10-13T10:55:02.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:02.393+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.393612+0000) 2019-10-13T10:55:02.396 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:02.393+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.394723+0000) 2019-10-13T10:55:02.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:02.393+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.394128+0000) 2019-10-13T10:55:02.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:02.393+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.394266+0000) 2019-10-13T10:55:02.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:02.393+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.394277+0000) 2019-10-13T10:55:02.397 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:02.394+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.394633+0000) 2019-10-13T10:55:02.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:02.394+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.394852+0000) 2019-10-13T10:55:02.398 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:02.396+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:02.397460+0000) 2019-10-13T10:55:02.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:02.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:03.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:03.013+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:03.013452+0000) 2019-10-13T10:55:03.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:03.015+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:03.015727+0000) 2019-10-13T10:55:04.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:04.958+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:04.959597+0000) 2019-10-13T10:55:05.667 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:05.666+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.666386+0000) 2019-10-13T10:55:05.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:05.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:05.666427+0000) 2019-10-13T10:55:07.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:07.402+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.402880+0000) 2019-10-13T10:55:07.404 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:07.402+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.403067+0000) 2019-10-13T10:55:07.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:07.403+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.404433+0000) 2019-10-13T10:55:07.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:07.403+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.403417+0000) 2019-10-13T10:55:07.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:07.403+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.403626+0000) 2019-10-13T10:55:07.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:07.403+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.403734+0000) 2019-10-13T10:55:07.406 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:07.403+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.403942+0000) 2019-10-13T10:55:07.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:07.404+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.405669+0000) 2019-10-13T10:55:07.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:07.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:08.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:08.574+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:08.574972+0000) 2019-10-13T10:55:08.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:08.910+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:08.910402+0000) 2019-10-13T10:55:11.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:11.523+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.524738+0000) 2019-10-13T10:55:12.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:12.412+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.413128+0000) 2019-10-13T10:55:12.414 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:12.411+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.412554+0000) 2019-10-13T10:55:12.414 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:12.411+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.412828+0000) 2019-10-13T10:55:12.415 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:12.411+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.412999+0000) 2019-10-13T10:55:12.415 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:12.411+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.413346+0000) 2019-10-13T10:55:12.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:12.412+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.413723+0000) 2019-10-13T10:55:12.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:12.412+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.413851+0000) 2019-10-13T10:55:12.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:12.415+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:12.416476+0000) 2019-10-13T10:55:12.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:12.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198101 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:13.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:13.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:13.013638+0000) 2019-10-13T10:55:13.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:13.015+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:13.015933+0000) 2019-10-13T10:55:14.962 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:14.961+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:14.961963+0000) 2019-10-13T10:55:15.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:15.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.666588+0000) 2019-10-13T10:55:15.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:15.665+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:15.666581+0000) 2019-10-13T10:55:16.647 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:55:16.648 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:16.652 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:55:16.652 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:17.423 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:17.420+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.422247+0000) 2019-10-13T10:55:17.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:17.421+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.422536+0000) 2019-10-13T10:55:17.424 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:17.422+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.423209+0000) 2019-10-13T10:55:17.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:17.421+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.423231+0000) 2019-10-13T10:55:17.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:17.422+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.423588+0000) 2019-10-13T10:55:17.425 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:17.422+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.424203+0000) 2019-10-13T10:55:17.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:17.423+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.424434+0000) 2019-10-13T10:55:17.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:17.426+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.426813+0000) 2019-10-13T10:55:17.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:17.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198132 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:18.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:18.574+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:18.575157+0000) 2019-10-13T10:55:18.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:18.909+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:18.910502+0000) 2019-10-13T10:55:21.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:21.524+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.524931+0000) 2019-10-13T10:55:22.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:22.431+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.432423+0000) 2019-10-13T10:55:22.434 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:22.432+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.433397+0000) 2019-10-13T10:55:22.434 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:22.431+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.432755+0000) 2019-10-13T10:55:22.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:22.431+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.432947+0000) 2019-10-13T10:55:22.435 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:22.431+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.433153+0000) 2019-10-13T10:55:22.435 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:22.431+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.433032+0000) 2019-10-13T10:55:22.435 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:22.432+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.433489+0000) 2019-10-13T10:55:22.437 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:22.436+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:22.436897+0000) 2019-10-13T10:55:22.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:22.438+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:23.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:23.012+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:23.013826+0000) 2019-10-13T10:55:23.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:23.015+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:23.016031+0000) 2019-10-13T10:55:24.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:24.963+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:24.964373+0000) 2019-10-13T10:55:25.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:25.665+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.666729+0000) 2019-10-13T10:55:25.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:25.665+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:25.666808+0000) 2019-10-13T10:55:27.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:27.440+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.441756+0000) 2019-10-13T10:55:27.443 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:27.440+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.441903+0000) 2019-10-13T10:55:27.443 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:27.440+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.442145+0000) 2019-10-13T10:55:27.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:27.440+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.442280+0000) 2019-10-13T10:55:27.444 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:27.442+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.443210+0000) 2019-10-13T10:55:27.444 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:27.441+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.442600+0000) 2019-10-13T10:55:27.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:27.441+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.443014+0000) 2019-10-13T10:55:27.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:27.445+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.445833+0000) 2019-10-13T10:55:27.451 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:27.448+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:28.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:28.574+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:28.575331+0000) 2019-10-13T10:55:28.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:28.910+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:28.910665+0000) 2019-10-13T10:55:31.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:31.524+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.525120+0000) 2019-10-13T10:55:32.452 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:32.449+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.451359+0000) 2019-10-13T10:55:32.453 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:32.450+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.451606+0000) 2019-10-13T10:55:32.453 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:32.450+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.451762+0000) 2019-10-13T10:55:32.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:32.450+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.451772+0000) 2019-10-13T10:55:32.454 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:32.452+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.453007+0000) 2019-10-13T10:55:32.454 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:32.450+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.452277+0000) 2019-10-13T10:55:32.454 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:32.451+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.452404+0000) 2019-10-13T10:55:32.455 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:32.453+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:32.454365+0000) 2019-10-13T10:55:32.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:32.456+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:33.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:33.013+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:33.013992+0000) 2019-10-13T10:55:33.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:33.016+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:33.016199+0000) 2019-10-13T10:55:34.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:34.966+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:34.966829+0000) 2019-10-13T10:55:35.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:35.666+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.666922+0000) 2019-10-13T10:55:35.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:35.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:35.667045+0000) 2019-10-13T10:55:37.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:37.458+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.459853+0000) 2019-10-13T10:55:37.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:37.458+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.459970+0000) 2019-10-13T10:55:37.461 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:37.458+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.460121+0000) 2019-10-13T10:55:37.462 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:37.459+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.460566+0000) 2019-10-13T10:55:37.462 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:37.459+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.461064+0000) 2019-10-13T10:55:37.462 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:37.459+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.461122+0000) 2019-10-13T10:55:37.463 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:37.461+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.462064+0000) 2019-10-13T10:55:37.464 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:37.463+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.464445+0000) 2019-10-13T10:55:37.469 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:37.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:38.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:38.575+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:38.575526+0000) 2019-10-13T10:55:38.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:38.910+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:38.910855+0000) 2019-10-13T10:55:41.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:41.524+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.525287+0000) 2019-10-13T10:55:42.472 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:42.469+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.470183+0000) 2019-10-13T10:55:42.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:42.469+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.470357+0000) 2019-10-13T10:55:42.473 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:42.470+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.470483+0000) 2019-10-13T10:55:42.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:42.470+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.470731+0000) 2019-10-13T10:55:42.473 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:42.470+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.470938+0000) 2019-10-13T10:55:42.473 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:42.470+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.471277+0000) 2019-10-13T10:55:42.474 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:42.470+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.471916+0000) 2019-10-13T10:55:42.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:42.472+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:42.474484+0000) 2019-10-13T10:55:42.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:42.478+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:43.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:43.013+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:43.014117+0000) 2019-10-13T10:55:43.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:43.015+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:43.016358+0000) 2019-10-13T10:55:44.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:44.968+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:44.969286+0000) 2019-10-13T10:55:45.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:45.666+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.667115+0000) 2019-10-13T10:55:45.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:45.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:45.667242+0000) 2019-10-13T10:55:46.819 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:55:46.820 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:46.824 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:55:46.824 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:47.481 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:47.479+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.480992+0000) 2019-10-13T10:55:47.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:47.480+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.480391+0000) 2019-10-13T10:55:47.483 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:47.480+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.480610+0000) 2019-10-13T10:55:47.484 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:47.480+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.480895+0000) 2019-10-13T10:55:47.484 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:47.480+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.481037+0000) 2019-10-13T10:55:47.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:47.482+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.484587+0000) 2019-10-13T10:55:47.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:47.481+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.481673+0000) 2019-10-13T10:55:47.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:47.481+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.481989+0000) 2019-10-13T10:55:47.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:47.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:48.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:48.575+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:48.575701+0000) 2019-10-13T10:55:48.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:48.910+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:48.910987+0000) 2019-10-13T10:55:51.526 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:51.525+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.525472+0000) 2019-10-13T10:55:52.491 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:52.489+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.490056+0000) 2019-10-13T10:55:52.492 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:52.490+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.491170+0000) 2019-10-13T10:55:52.492 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:52.490+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.491334+0000) 2019-10-13T10:55:52.493 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:52.491+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.492656+0000) 2019-10-13T10:55:52.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:52.491+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.491598+0000) 2019-10-13T10:55:52.493 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:52.491+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.491685+0000) 2019-10-13T10:55:52.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:52.491+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.492078+0000) 2019-10-13T10:55:52.495 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:52.494+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:52.495086+0000) 2019-10-13T10:55:52.500 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:52.498+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:53.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:53.014+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:53.014264+0000) 2019-10-13T10:55:53.016 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:53.015+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:53.016516+0000) 2019-10-13T10:55:54.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:54.971+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:54.971751+0000) 2019-10-13T10:55:55.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:55.667+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.667315+0000) 2019-10-13T10:55:55.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:55.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:55.667439+0000) 2019-10-13T10:55:57.501 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:55:57.499+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.500747+0000) 2019-10-13T10:55:57.503 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:55:57.501+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.502124+0000) 2019-10-13T10:55:57.503 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:57.499+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.501051+0000) 2019-10-13T10:55:57.503 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:57.500+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.501467+0000) 2019-10-13T10:55:57.503 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:55:57.500+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.501838+0000) 2019-10-13T10:55:57.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:55:57.500+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.501931+0000) 2019-10-13T10:55:57.504 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:55:57.500+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.502171+0000) 2019-10-13T10:55:57.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:55:57.503+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.504452+0000) 2019-10-13T10:55:57.510 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:55:57.507+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:55:58.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:55:58.574+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:58.575895+0000) 2019-10-13T10:55:58.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:55:58.910+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:58.911153+0000) 2019-10-13T10:56:01.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:01.524+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.525686+0000) 2019-10-13T10:56:02.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:02.509+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.511324+0000) 2019-10-13T10:56:02.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:02.510+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.511879+0000) 2019-10-13T10:56:02.513 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:02.510+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.512075+0000) 2019-10-13T10:56:02.514 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:02.510+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.512205+0000) 2019-10-13T10:56:02.514 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:02.511+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.512534+0000) 2019-10-13T10:56:02.514 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:02.511+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.512765+0000) 2019-10-13T10:56:02.515 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:02.512+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.513404+0000) 2019-10-13T10:56:02.515 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:02.513+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:02.514500+0000) 2019-10-13T10:56:02.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:02.517+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:03.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:03.013+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:03.014427+0000) 2019-10-13T10:56:03.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:03.016+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:03.016680+0000) 2019-10-13T10:56:04.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:04.973+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:04.974083+0000) 2019-10-13T10:56:05.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:05.666+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.667509+0000) 2019-10-13T10:56:05.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:05.666+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:05.667627+0000) 2019-10-13T10:56:07.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:07.519+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.521353+0000) 2019-10-13T10:56:07.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:07.520+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.521567+0000) 2019-10-13T10:56:07.523 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:07.520+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.521877+0000) 2019-10-13T10:56:07.524 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:07.522+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.523094+0000) 2019-10-13T10:56:07.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:07.520+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.522140+0000) 2019-10-13T10:56:07.524 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:07.521+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.522630+0000) 2019-10-13T10:56:07.524 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:07.521+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.522958+0000) 2019-10-13T10:56:07.525 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:07.524+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.525402+0000) 2019-10-13T10:56:07.529 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:07.527+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:08.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:08.575+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:08.576094+0000) 2019-10-13T10:56:08.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:08.910+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:08.911293+0000) 2019-10-13T10:56:11.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:11.525+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.525881+0000) 2019-10-13T10:56:12.531 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:12.528+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.530320+0000) 2019-10-13T10:56:12.532 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:12.531+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.531910+0000) 2019-10-13T10:56:12.532 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:12.529+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.530906+0000) 2019-10-13T10:56:12.532 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:12.529+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.531370+0000) 2019-10-13T10:56:12.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:12.530+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.531595+0000) 2019-10-13T10:56:12.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:12.530+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.531673+0000) 2019-10-13T10:56:12.533 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:12.530+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.531819+0000) 2019-10-13T10:56:12.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:12.532+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:12.533712+0000) 2019-10-13T10:56:12.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:12.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:13.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:13.013+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:13.014601+0000) 2019-10-13T10:56:13.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:13.016+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:13.016832+0000) 2019-10-13T10:56:14.976 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:14.975+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:14.976287+0000) 2019-10-13T10:56:15.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:15.667+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.667722+0000) 2019-10-13T10:56:15.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:15.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:15.667815+0000) 2019-10-13T10:56:16.966 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:56:16.966 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:16.971 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:56:16.971 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:17.542 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:17.541+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.541839+0000) 2019-10-13T10:56:17.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:17.539+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.541279+0000) 2019-10-13T10:56:17.543 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:17.540+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.541491+0000) 2019-10-13T10:56:17.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:17.540+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.541905+0000) 2019-10-13T10:56:17.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:17.540+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.542089+0000) 2019-10-13T10:56:17.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:17.540+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.542118+0000) 2019-10-13T10:56:17.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:17.541+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.542526+0000) 2019-10-13T10:56:17.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:17.544+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.545478+0000) 2019-10-13T10:56:17.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:17.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198498 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:18.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:18.575+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:18.576285+0000) 2019-10-13T10:56:18.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:18.910+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:18.911442+0000) 2019-10-13T10:56:21.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:21.525+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.526039+0000) 2019-10-13T10:56:22.549 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:22.547+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.548591+0000) 2019-10-13T10:56:22.550 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:22.549+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.549784+0000) 2019-10-13T10:56:22.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:22.547+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.549275+0000) 2019-10-13T10:56:22.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:22.548+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.549640+0000) 2019-10-13T10:56:22.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:22.548+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.549772+0000) 2019-10-13T10:56:22.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:22.548+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.549700+0000) 2019-10-13T10:56:22.552 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:22.548+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.549952+0000) 2019-10-13T10:56:22.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:22.551+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:22.552263+0000) 2019-10-13T10:56:22.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:22.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198529 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:23.015 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:23.014+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:23.014728+0000) 2019-10-13T10:56:23.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:23.016+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:23.016986+0000) 2019-10-13T10:56:24.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:24.977+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:24.978538+0000) 2019-10-13T10:56:25.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:25.667+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.667913+0000) 2019-10-13T10:56:25.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:25.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:25.668026+0000) 2019-10-13T10:56:27.559 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:27.558+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.558605+0000) 2019-10-13T10:56:27.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:27.558+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.558914+0000) 2019-10-13T10:56:27.561 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:27.559+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.559915+0000) 2019-10-13T10:56:27.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:27.559+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.559479+0000) 2019-10-13T10:56:27.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:27.559+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.559790+0000) 2019-10-13T10:56:27.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:27.559+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.559807+0000) 2019-10-13T10:56:27.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:27.559+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.559992+0000) 2019-10-13T10:56:27.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:27.561+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.562060+0000) 2019-10-13T10:56:27.566 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:27.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:28.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:28.576+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:28.576473+0000) 2019-10-13T10:56:28.912 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:28.911+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:28.911679+0000) 2019-10-13T10:56:31.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:31.525+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.526231+0000) 2019-10-13T10:56:32.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:32.566+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.567235+0000) 2019-10-13T10:56:32.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:32.567+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.567547+0000) 2019-10-13T10:56:32.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:32.567+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.567795+0000) 2019-10-13T10:56:32.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:32.567+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.567908+0000) 2019-10-13T10:56:32.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:32.566+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.568177+0000) 2019-10-13T10:56:32.572 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:32.567+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.568048+0000) 2019-10-13T10:56:32.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:32.567+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.568179+0000) 2019-10-13T10:56:32.573 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:32.567+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:32.569612+0000) 2019-10-13T10:56:32.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:32.575+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:33.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:33.014+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:33.014870+0000) 2019-10-13T10:56:33.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:33.015+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:33.017113+0000) 2019-10-13T10:56:34.981 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:34.980+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:34.980947+0000) 2019-10-13T10:56:35.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:35.667+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.668113+0000) 2019-10-13T10:56:35.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:35.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:35.668238+0000) 2019-10-13T10:56:37.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:37.577+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.577490+0000) 2019-10-13T10:56:37.579 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:37.576+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.578480+0000) 2019-10-13T10:56:37.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:37.577+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.578140+0000) 2019-10-13T10:56:37.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:37.577+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.578365+0000) 2019-10-13T10:56:37.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:37.578+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.578577+0000) 2019-10-13T10:56:37.580 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:37.578+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.578723+0000) 2019-10-13T10:56:37.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:37.578+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.578796+0000) 2019-10-13T10:56:37.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:37.578+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.580699+0000) 2019-10-13T10:56:37.587 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:37.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:38.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:38.576+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:38.576652+0000) 2019-10-13T10:56:38.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:38.911+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:38.911836+0000) 2019-10-13T10:56:41.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:41.526+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.526400+0000) 2019-10-13T10:56:42.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:42.587+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.587638+0000) 2019-10-13T10:56:42.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:42.587+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.587986+0000) 2019-10-13T10:56:42.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:42.587+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.588253+0000) 2019-10-13T10:56:42.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:42.587+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.588383+0000) 2019-10-13T10:56:42.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:42.588+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.588616+0000) 2019-10-13T10:56:42.590 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:42.587+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.589626+0000) 2019-10-13T10:56:42.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:42.588+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.589351+0000) 2019-10-13T10:56:42.592 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:42.591+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:42.591910+0000) 2019-10-13T10:56:42.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:42.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:43.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:43.013+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:43.015016+0000) 2019-10-13T10:56:43.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:43.016+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:43.017305+0000) 2019-10-13T10:56:44.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:44.982+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:44.983280+0000) 2019-10-13T10:56:45.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:45.667+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.668284+0000) 2019-10-13T10:56:45.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:45.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:45.668374+0000) 2019-10-13T10:56:47.336 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:56:47.337 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:47.341 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:56:47.342 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:47.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:47.596+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.597733+0000) 2019-10-13T10:56:47.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:47.596+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.598354+0000) 2019-10-13T10:56:47.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:47.596+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.598176+0000) 2019-10-13T10:56:47.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:47.597+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.598654+0000) 2019-10-13T10:56:47.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:47.597+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.598937+0000) 2019-10-13T10:56:47.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:47.597+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.599029+0000) 2019-10-13T10:56:47.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:47.599+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.600219+0000) 2019-10-13T10:56:47.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:47.602+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.603228+0000) 2019-10-13T10:56:47.611 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:47.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:48.578 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:48.575+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:48.576842+0000) 2019-10-13T10:56:48.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:48.911+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:48.912018+0000) 2019-10-13T10:56:51.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:51.525+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.526539+0000) 2019-10-13T10:56:52.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:52.608+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.608875+0000) 2019-10-13T10:56:52.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:52.606+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.607921+0000) 2019-10-13T10:56:52.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:52.606+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.608229+0000) 2019-10-13T10:56:52.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:52.606+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.608348+0000) 2019-10-13T10:56:52.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:52.607+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.608542+0000) 2019-10-13T10:56:52.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:52.607+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.609317+0000) 2019-10-13T10:56:52.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:52.608+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.609487+0000) 2019-10-13T10:56:52.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:52.611+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:52.612078+0000) 2019-10-13T10:56:52.617 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:52.615+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198711 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:53.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:53.014+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:53.015221+0000) 2019-10-13T10:56:53.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:53.016+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:53.017469+0000) 2019-10-13T10:56:54.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:54.985+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:54.985694+0000) 2019-10-13T10:56:55.669 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:55.667+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.668466+0000) 2019-10-13T10:56:55.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:55.667+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:55.668496+0000) 2019-10-13T10:56:57.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:57.616+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.618258+0000) 2019-10-13T10:56:57.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:56:57.617+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.618598+0000) 2019-10-13T10:56:57.620 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:56:57.619+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.619793+0000) 2019-10-13T10:56:57.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:56:57.617+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.619048+0000) 2019-10-13T10:56:57.621 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:56:57.617+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.619259+0000) 2019-10-13T10:56:57.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:56:57.618+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.619470+0000) 2019-10-13T10:56:57.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:57.618+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.619631+0000) 2019-10-13T10:56:57.622 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:56:57.621+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.621894+0000) 2019-10-13T10:56:57.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:56:57.624+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:56:58.578 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:56:58.576+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:58.577016+0000) 2019-10-13T10:56:58.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:56:58.911+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:58.912200+0000) 2019-10-13T10:57:01.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:01.526+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.526725+0000) 2019-10-13T10:57:02.645 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:02.628+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.628794+0000) 2019-10-13T10:57:02.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:02.626+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.628215+0000) 2019-10-13T10:57:02.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:02.627+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.628504+0000) 2019-10-13T10:57:02.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:02.627+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.628655+0000) 2019-10-13T10:57:02.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:02.627+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.628851+0000) 2019-10-13T10:57:02.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:02.627+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.628967+0000) 2019-10-13T10:57:02.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:02.627+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.629159+0000) 2019-10-13T10:57:02.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:02.635+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:02.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:02.631+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:02.632071+0000) 2019-10-13T10:57:03.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:03.014+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:03.015424+0000) 2019-10-13T10:57:03.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:03.017+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:03.017649+0000) 2019-10-13T10:57:04.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:04.987+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:04.988114+0000) 2019-10-13T10:57:05.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:05.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.668689+0000) 2019-10-13T10:57:05.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:05.668+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:05.668682+0000) 2019-10-13T10:57:07.639 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:07.636+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.638174+0000) 2019-10-13T10:57:07.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:07.636+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.638308+0000) 2019-10-13T10:57:07.640 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:07.637+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.638546+0000) 2019-10-13T10:57:07.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:07.638+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.639578+0000) 2019-10-13T10:57:07.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:07.637+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.639005+0000) 2019-10-13T10:57:07.642 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:07.637+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.639193+0000) 2019-10-13T10:57:07.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:07.638+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.639494+0000) 2019-10-13T10:57:07.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:07.641+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.641858+0000) 2019-10-13T10:57:07.647 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:07.645+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:08.578 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:08.576+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:08.577184+0000) 2019-10-13T10:57:08.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:08.911+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:08.912385+0000) 2019-10-13T10:57:11.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:11.526+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.526963+0000) 2019-10-13T10:57:12.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:12.647+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.647943+0000) 2019-10-13T10:57:12.649 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:12.647+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.648077+0000) 2019-10-13T10:57:12.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:12.647+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.648322+0000) 2019-10-13T10:57:12.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:12.648+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.648527+0000) 2019-10-13T10:57:12.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:12.648+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.649023+0000) 2019-10-13T10:57:12.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:12.648+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.648711+0000) 2019-10-13T10:57:12.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:12.648+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.648946+0000) 2019-10-13T10:57:12.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:12.651+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:12.651790+0000) 2019-10-13T10:57:12.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:12.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:13.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:13.015+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:13.015559+0000) 2019-10-13T10:57:13.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:13.017+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:13.017808+0000) 2019-10-13T10:57:14.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:14.990+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:14.990382+0000) 2019-10-13T10:57:15.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:15.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:15.668888+0000) 2019-10-13T10:57:15.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:15.668+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:15.668885+0000) 2019-10-13T10:57:17.447 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:57:17.447 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:17.451 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:57:17.452 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:17.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:17.656+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.656977+0000) 2019-10-13T10:57:17.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:17.656+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.657114+0000) 2019-10-13T10:57:17.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:17.657+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.657514+0000) 2019-10-13T10:57:17.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:17.657+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.657701+0000) 2019-10-13T10:57:17.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:17.657+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.657783+0000) 2019-10-13T10:57:17.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:17.657+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.657885+0000) 2019-10-13T10:57:17.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:17.658+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.658883+0000) 2019-10-13T10:57:17.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:17.660+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.661262+0000) 2019-10-13T10:57:17.664 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:17.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:18.578 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:18.577+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:18.577333+0000) 2019-10-13T10:57:18.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:18.912+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:18.912618+0000) 2019-10-13T10:57:21.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:21.526+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.527130+0000) 2019-10-13T10:57:22.666 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:22.664+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.665770+0000) 2019-10-13T10:57:22.666 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:22.664+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.665312+0000) 2019-10-13T10:57:22.667 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:22.665+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.665484+0000) 2019-10-13T10:57:22.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:22.665+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.665865+0000) 2019-10-13T10:57:22.667 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:22.665+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.665812+0000) 2019-10-13T10:57:22.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:22.665+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.666091+0000) 2019-10-13T10:57:22.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:22.665+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.666325+0000) 2019-10-13T10:57:22.669 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:22.667+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:22.668860+0000) 2019-10-13T10:57:22.674 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:22.673+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:23.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:23.015+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:23.015687+0000) 2019-10-13T10:57:23.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:23.016+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:23.017931+0000) 2019-10-13T10:57:24.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:24.991+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:24.992611+0000) 2019-10-13T10:57:25.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:25.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:25.669083+0000) 2019-10-13T10:57:25.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:25.668+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:25.669085+0000) 2019-10-13T10:57:27.677 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:27.675+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.675715+0000) 2019-10-13T10:57:27.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:27.675+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.676045+0000) 2019-10-13T10:57:27.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:27.675+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.677501+0000) 2019-10-13T10:57:27.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:27.675+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.676349+0000) 2019-10-13T10:57:27.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:27.676+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.676613+0000) 2019-10-13T10:57:27.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:27.676+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.677086+0000) 2019-10-13T10:57:27.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:27.676+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.677226+0000) 2019-10-13T10:57:27.680 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:27.678+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.679815+0000) 2019-10-13T10:57:27.685 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:27.683+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:28.578 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:28.577+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:28.577480+0000) 2019-10-13T10:57:28.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:28.911+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:28.912775+0000) 2019-10-13T10:57:31.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:31.526+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.527320+0000) 2019-10-13T10:57:32.687 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:32.684+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.686242+0000) 2019-10-13T10:57:32.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:32.685+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.686480+0000) 2019-10-13T10:57:32.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:32.685+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.687419+0000) 2019-10-13T10:57:32.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:32.685+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.686804+0000) 2019-10-13T10:57:32.689 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:32.685+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.686997+0000) 2019-10-13T10:57:32.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:32.686+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.687413+0000) 2019-10-13T10:57:32.689 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:32.686+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.687492+0000) 2019-10-13T10:57:32.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:32.688+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:32.690215+0000) 2019-10-13T10:57:32.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:32.692+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198955 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:33.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:33.014+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:33.015813+0000) 2019-10-13T10:57:33.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:33.017+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:33.018064+0000) 2019-10-13T10:57:34.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:34.994+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:34.995115+0000) 2019-10-13T10:57:35.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:35.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:35.669278+0000) 2019-10-13T10:57:35.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:35.668+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:35.669279+0000) 2019-10-13T10:57:37.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:37.694+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.695939+0000) 2019-10-13T10:57:37.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:37.696+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.697082+0000) 2019-10-13T10:57:37.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:37.695+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.696431+0000) 2019-10-13T10:57:37.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:37.695+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.696785+0000) 2019-10-13T10:57:37.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:37.695+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.696999+0000) 2019-10-13T10:57:37.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:37.695+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.697097+0000) 2019-10-13T10:57:37.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:37.695+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.697363+0000) 2019-10-13T10:57:37.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:37.698+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.698996+0000) 2019-10-13T10:57:37.705 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:37.702+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 198986 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:38.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:38.576+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:38.577650+0000) 2019-10-13T10:57:38.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:38.912+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:38.912941+0000) 2019-10-13T10:57:41.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:41.526+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.527533+0000) 2019-10-13T10:57:42.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:42.704+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.705852+0000) 2019-10-13T10:57:42.707 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:42.706+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.706757+0000) 2019-10-13T10:57:42.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:42.704+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.706340+0000) 2019-10-13T10:57:42.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:42.705+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.706629+0000) 2019-10-13T10:57:42.708 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:42.705+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.707214+0000) 2019-10-13T10:57:42.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:42.705+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.707367+0000) 2019-10-13T10:57:42.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:42.706+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.707590+0000) 2019-10-13T10:57:42.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:42.709+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:42.710525+0000) 2019-10-13T10:57:42.715 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:42.713+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199017 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:43.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:43.014+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:43.016000+0000) 2019-10-13T10:57:43.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:43.017+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:43.018217+0000) 2019-10-13T10:57:44.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:44.996+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:44.997416+0000) 2019-10-13T10:57:45.670 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:45.668+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:45.669507+0000) 2019-10-13T10:57:45.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:45.668+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:45.669515+0000) 2019-10-13T10:57:47.629 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:57:47.629 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:47.633 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:57:47.633 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:47.716 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:47.714+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.715919+0000) 2019-10-13T10:57:47.718 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:47.716+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.716974+0000) 2019-10-13T10:57:47.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:47.714+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.716367+0000) 2019-10-13T10:57:47.718 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:47.715+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.716481+0000) 2019-10-13T10:57:47.719 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:47.715+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.716605+0000) 2019-10-13T10:57:47.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:47.715+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.716712+0000) 2019-10-13T10:57:47.719 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:47.715+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.717015+0000) 2019-10-13T10:57:47.720 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:47.719+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.719873+0000) 2019-10-13T10:57:47.724 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:47.721+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:48.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:48.577+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:48.577820+0000) 2019-10-13T10:57:48.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:48.912+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:48.913130+0000) 2019-10-13T10:57:51.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:51.527+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.527719+0000) 2019-10-13T10:57:52.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:52.725+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.725813+0000) 2019-10-13T10:57:52.727 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:52.723+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.725144+0000) 2019-10-13T10:57:52.727 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:52.723+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.725287+0000) 2019-10-13T10:57:52.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:52.724+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.725703+0000) 2019-10-13T10:57:52.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:52.724+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.726018+0000) 2019-10-13T10:57:52.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:52.724+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.726134+0000) 2019-10-13T10:57:52.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:52.724+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.726299+0000) 2019-10-13T10:57:52.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:52.727+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:52.728429+0000) 2019-10-13T10:57:52.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:52.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:53.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:53.015+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:53.016120+0000) 2019-10-13T10:57:53.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:53.017+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:53.018378+0000) 2019-10-13T10:57:55.000 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:54.999+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:54.999765+0000) 2019-10-13T10:57:55.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:55.669+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:55.669734+0000) 2019-10-13T10:57:55.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:55.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:55.669733+0000) 2019-10-13T10:57:57.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:57:57.734+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.735144+0000) 2019-10-13T10:57:57.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:57:57.734+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.735376+0000) 2019-10-13T10:57:57.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:57:57.735+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.735642+0000) 2019-10-13T10:57:57.737 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:57.735+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.735717+0000) 2019-10-13T10:57:57.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:57.735+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.735953+0000) 2019-10-13T10:57:57.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:57:57.735+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.736101+0000) 2019-10-13T10:57:57.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:57:57.735+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.736693+0000) 2019-10-13T10:57:57.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:57:57.737+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.737858+0000) 2019-10-13T10:57:57.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:57:57.741+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199108 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:57:58.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:57:58.577+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:58.578023+0000) 2019-10-13T10:57:58.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:57:58.912+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:58.913335+0000) 2019-10-13T10:58:01.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:01.527+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.527950+0000) 2019-10-13T10:58:02.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:02.743+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.744102+0000) 2019-10-13T10:58:02.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:02.743+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.744349+0000) 2019-10-13T10:58:02.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:02.744+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.744481+0000) 2019-10-13T10:58:02.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:02.744+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.744717+0000) 2019-10-13T10:58:02.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:02.744+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.745104+0000) 2019-10-13T10:58:02.746 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:02.744+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.745378+0000) 2019-10-13T10:58:02.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:02.745+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.746317+0000) 2019-10-13T10:58:02.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:02.747+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:02.748100+0000) 2019-10-13T10:58:02.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:02.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199139 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:03.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:03.015+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:03.016207+0000) 2019-10-13T10:58:03.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:03.017+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:03.018555+0000) 2019-10-13T10:58:05.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:05.001+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:05.002106+0000) 2019-10-13T10:58:05.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:05.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:05.669948+0000) 2019-10-13T10:58:05.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:05.669+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:05.669948+0000) 2019-10-13T10:58:07.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:07.753+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.753772+0000) 2019-10-13T10:58:07.755 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:07.753+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.753879+0000) 2019-10-13T10:58:07.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:07.754+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.755267+0000) 2019-10-13T10:58:07.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:07.753+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.754200+0000) 2019-10-13T10:58:07.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:07.754+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.754709+0000) 2019-10-13T10:58:07.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:07.754+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.755010+0000) 2019-10-13T10:58:07.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:07.754+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.754882+0000) 2019-10-13T10:58:07.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:07.756+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.756941+0000) 2019-10-13T10:58:07.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:07.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:08.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:08.578+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:08.578193+0000) 2019-10-13T10:58:08.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:08.913+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:08.913451+0000) 2019-10-13T10:58:11.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:11.528+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.528146+0000) 2019-10-13T10:58:12.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:12.763+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.764248+0000) 2019-10-13T10:58:12.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:12.764+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.764469+0000) 2019-10-13T10:58:12.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:12.764+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.764633+0000) 2019-10-13T10:58:12.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:12.764+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.765808+0000) 2019-10-13T10:58:12.767 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:12.764+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.764928+0000) 2019-10-13T10:58:12.767 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:12.764+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.765294+0000) 2019-10-13T10:58:12.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:12.765+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.765442+0000) 2019-10-13T10:58:12.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:12.765+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:12.767263+0000) 2019-10-13T10:58:12.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:12.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:13.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:13.015+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:13.016329+0000) 2019-10-13T10:58:13.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:13.017+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:13.018713+0000) 2019-10-13T10:58:15.005 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:15.003+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:15.004612+0000) 2019-10-13T10:58:15.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:15.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:15.670143+0000) 2019-10-13T10:58:15.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:15.669+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:15.670143+0000) 2019-10-13T10:58:17.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:17.773+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.774615+0000) 2019-10-13T10:58:17.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:17.773+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.774821+0000) 2019-10-13T10:58:17.776 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:17.773+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.775034+0000) 2019-10-13T10:58:17.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:17.773+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.775191+0000) 2019-10-13T10:58:17.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:17.773+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.775352+0000) 2019-10-13T10:58:17.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:17.774+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.775458+0000) 2019-10-13T10:58:17.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:17.773+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.775676+0000) 2019-10-13T10:58:17.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:17.776+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.778284+0000) 2019-10-13T10:58:17.782 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:17.779+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199230 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:17.783 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:58:17.783 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:17.787 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:58:17.788 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:18.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:18.577+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:18.578343+0000) 2019-10-13T10:58:18.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:18.912+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:18.913627+0000) 2019-10-13T10:58:21.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:21.527+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.528335+0000) 2019-10-13T10:58:22.784 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:22.781+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.783182+0000) 2019-10-13T10:58:22.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:22.781+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.783381+0000) 2019-10-13T10:58:22.785 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:22.782+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.783707+0000) 2019-10-13T10:58:22.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:22.783+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.784965+0000) 2019-10-13T10:58:22.787 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:22.782+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.784189+0000) 2019-10-13T10:58:22.787 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:22.782+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.784373+0000) 2019-10-13T10:58:22.787 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:22.783+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.784832+0000) 2019-10-13T10:58:22.788 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:22.785+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:22.786853+0000) 2019-10-13T10:58:22.792 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:22.790+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199261 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:23.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:23.015+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:23.016467+0000) 2019-10-13T10:58:23.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:23.018+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:23.018862+0000) 2019-10-13T10:58:25.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:25.006+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:25.006982+0000) 2019-10-13T10:58:25.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:25.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:25.670339+0000) 2019-10-13T10:58:25.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:25.669+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:25.670339+0000) 2019-10-13T10:58:27.794 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:27.793+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.793857+0000) 2019-10-13T10:58:27.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:27.792+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.793438+0000) 2019-10-13T10:58:27.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:27.792+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.793612+0000) 2019-10-13T10:58:27.795 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:27.792+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.793971+0000) 2019-10-13T10:58:27.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:27.792+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.794014+0000) 2019-10-13T10:58:27.796 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:27.792+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.794144+0000) 2019-10-13T10:58:27.796 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:27.792+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.793837+0000) 2019-10-13T10:58:27.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:27.796+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.797114+0000) 2019-10-13T10:58:27.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:27.800+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199290 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:28.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:28.577+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:28.578496+0000) 2019-10-13T10:58:28.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:28.913+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:28.913818+0000) 2019-10-13T10:58:31.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:31.527+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.528515+0000) 2019-10-13T10:58:32.804 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:32.803+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.804258+0000) 2019-10-13T10:58:32.805 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:32.802+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.803787+0000) 2019-10-13T10:58:32.806 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:32.802+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.804158+0000) 2019-10-13T10:58:32.806 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:32.803+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.804427+0000) 2019-10-13T10:58:32.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:32.803+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.804547+0000) 2019-10-13T10:58:32.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:32.803+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.804708+0000) 2019-10-13T10:58:32.807 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:32.803+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.805060+0000) 2019-10-13T10:58:32.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:32.807+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:32.808173+0000) 2019-10-13T10:58:32.813 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:32.810+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199321 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:33.017 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:33.015+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:33.016579+0000) 2019-10-13T10:58:33.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:33.018+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:33.019025+0000) 2019-10-13T10:58:35.009 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:35.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:35.009364+0000) 2019-10-13T10:58:35.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:35.669+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:35.670528+0000) 2019-10-13T10:58:35.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:35.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:35.670587+0000) 2019-10-13T10:58:37.815 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:37.812+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.814023+0000) 2019-10-13T10:58:37.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:37.812+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.814220+0000) 2019-10-13T10:58:37.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:37.813+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.814405+0000) 2019-10-13T10:58:37.816 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:37.814+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.815655+0000) 2019-10-13T10:58:37.817 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:37.813+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.815122+0000) 2019-10-13T10:58:37.817 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:37.814+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.815427+0000) 2019-10-13T10:58:37.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:37.814+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.815811+0000) 2019-10-13T10:58:37.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:37.817+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.818635+0000) 2019-10-13T10:58:37.823 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:37.821+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199352 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:38.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:38.577+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:38.578622+0000) 2019-10-13T10:58:38.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:38.913+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:38.914022+0000) 2019-10-13T10:58:41.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:41.528+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.528726+0000) 2019-10-13T10:58:42.825 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:42.824+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.824560+0000) 2019-10-13T10:58:42.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:42.824+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.824800+0000) 2019-10-13T10:58:42.827 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:42.825+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.825834+0000) 2019-10-13T10:58:42.827 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:42.824+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.825128+0000) 2019-10-13T10:58:42.827 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:42.825+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.825534+0000) 2019-10-13T10:58:42.827 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:42.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.825500+0000) 2019-10-13T10:58:42.827 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:42.825+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.825749+0000) 2019-10-13T10:58:42.828 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:42.826+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:42.827685+0000) 2019-10-13T10:58:42.833 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:42.832+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199382 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:43.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:43.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:43.016755+0000) 2019-10-13T10:58:43.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:43.018+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:43.019169+0000) 2019-10-13T10:58:45.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:45.011+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:45.011720+0000) 2019-10-13T10:58:45.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:45.670+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:45.670744+0000) 2019-10-13T10:58:45.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:45.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:45.670786+0000) 2019-10-13T10:58:47.835 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:47.834+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.834930+0000) 2019-10-13T10:58:47.836 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:47.834+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.834411+0000) 2019-10-13T10:58:47.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:47.834+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.834545+0000) 2019-10-13T10:58:47.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:47.834+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.834682+0000) 2019-10-13T10:58:47.836 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:47.834+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.835016+0000) 2019-10-13T10:58:47.837 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:47.834+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.835095+0000) 2019-10-13T10:58:47.837 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:47.835+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.835854+0000) 2019-10-13T10:58:47.838 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:47.837+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.838106+0000) 2019-10-13T10:58:47.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:47.842+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199411 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:47.879 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:58:47.879 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:47.883 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:58:47.883 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:48.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:48.578+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:48.578756+0000) 2019-10-13T10:58:48.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:48.913+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:48.914159+0000) 2019-10-13T10:58:51.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:51.528+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.528941+0000) 2019-10-13T10:58:52.845 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:52.844+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.844997+0000) 2019-10-13T10:58:52.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:52.844+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.844403+0000) 2019-10-13T10:58:52.846 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:52.844+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.844549+0000) 2019-10-13T10:58:52.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:52.844+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.844701+0000) 2019-10-13T10:58:52.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:52.844+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.844976+0000) 2019-10-13T10:58:52.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:52.844+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.845053+0000) 2019-10-13T10:58:52.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:52.844+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.845316+0000) 2019-10-13T10:58:52.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:52.846+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:52.847713+0000) 2019-10-13T10:58:52.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:52.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199441 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:53.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:53.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:53.016938+0000) 2019-10-13T10:58:53.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:53.018+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:53.019285+0000) 2019-10-13T10:58:55.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:55.014+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:55.014233+0000) 2019-10-13T10:58:55.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:55.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:55.670944+0000) 2019-10-13T10:58:55.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:55.670+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:55.670944+0000) 2019-10-13T10:58:57.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:57.853+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.853936+0000) 2019-10-13T10:58:57.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:58:57.853+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.854178+0000) 2019-10-13T10:58:57.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:58:57.854+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.854394+0000) 2019-10-13T10:58:57.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:58:57.854+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.854482+0000) 2019-10-13T10:58:57.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:58:57.853+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.854297+0000) 2019-10-13T10:58:57.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:57.854+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.854821+0000) 2019-10-13T10:58:57.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:58:57.854+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.855540+0000) 2019-10-13T10:58:57.857 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:58:57.856+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.857520+0000) 2019-10-13T10:58:57.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:58:57.859+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199472 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:58:58.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:58:58.578+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:58.578906+0000) 2019-10-13T10:58:58.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:58:58.913+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:58.914348+0000) 2019-10-13T10:59:01.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:01.528+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.529144+0000) 2019-10-13T10:59:02.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:02.860+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.862087+0000) 2019-10-13T10:59:02.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:02.860+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.861784+0000) 2019-10-13T10:59:02.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:02.860+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.861436+0000) 2019-10-13T10:59:02.863 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:02.860+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.862358+0000) 2019-10-13T10:59:02.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:02.861+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.862610+0000) 2019-10-13T10:59:02.864 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:02.861+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.862877+0000) 2019-10-13T10:59:02.864 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:02.861+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.863083+0000) 2019-10-13T10:59:02.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:02.863+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:02.865353+0000) 2019-10-13T10:59:02.871 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:02.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199502 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:03.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:03.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:03.017110+0000) 2019-10-13T10:59:03.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:03.018+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:03.019394+0000) 2019-10-13T10:59:05.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:05.015+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:05.016578+0000) 2019-10-13T10:59:05.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:05.669+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:05.671130+0000) 2019-10-13T10:59:05.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:05.669+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:05.671152+0000) 2019-10-13T10:59:07.873 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:07.870+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.872035+0000) 2019-10-13T10:59:07.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:07.870+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.872309+0000) 2019-10-13T10:59:07.874 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:07.871+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.872548+0000) 2019-10-13T10:59:07.874 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:07.871+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.872576+0000) 2019-10-13T10:59:07.875 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:07.871+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.872966+0000) 2019-10-13T10:59:07.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:07.873+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.874814+0000) 2019-10-13T10:59:07.875 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:07.871+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.872903+0000) 2019-10-13T10:59:07.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:07.871+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.873175+0000) 2019-10-13T10:59:07.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:07.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:08.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:08.578+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:08.579095+0000) 2019-10-13T10:59:08.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:08.913+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:08.914551+0000) 2019-10-13T10:59:11.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:11.527+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.529363+0000) 2019-10-13T10:59:12.882 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:12.880+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.882368+0000) 2019-10-13T10:59:12.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:12.880+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.882108+0000) 2019-10-13T10:59:12.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:12.880+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.882295+0000) 2019-10-13T10:59:12.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:12.881+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.882623+0000) 2019-10-13T10:59:12.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:12.881+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.882815+0000) 2019-10-13T10:59:12.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:12.881+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.882961+0000) 2019-10-13T10:59:12.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:12.881+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.883143+0000) 2019-10-13T10:59:12.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:12.883+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:12.884947+0000) 2019-10-13T10:59:12.891 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:12.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199561 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:13.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:13.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:13.017295+0000) 2019-10-13T10:59:13.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:13.018+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:13.019500+0000) 2019-10-13T10:59:15.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:15.018+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:15.018899+0000) 2019-10-13T10:59:15.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:15.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:15.671311+0000) 2019-10-13T10:59:15.672 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:15.670+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:15.671312+0000) 2019-10-13T10:59:17.893 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:17.892+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.892753+0000) 2019-10-13T10:59:17.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:17.891+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.892403+0000) 2019-10-13T10:59:17.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:17.891+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.892804+0000) 2019-10-13T10:59:17.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:17.891+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.893063+0000) 2019-10-13T10:59:17.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:17.891+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.893194+0000) 2019-10-13T10:59:17.895 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:17.891+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.893206+0000) 2019-10-13T10:59:17.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:17.892+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.893440+0000) 2019-10-13T10:59:17.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:17.894+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.895495+0000) 2019-10-13T10:59:17.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:17.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199592 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:18.041 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:59:18.041 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:18.045 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:59:18.046 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:18.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:18.578+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:18.579248+0000) 2019-10-13T10:59:18.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:18.914+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:18.914743+0000) 2019-10-13T10:59:21.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:21.528+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.529552+0000) 2019-10-13T10:59:22.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:22.900+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.902187+0000) 2019-10-13T10:59:22.904 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:22.902+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.903620+0000) 2019-10-13T10:59:22.904 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:22.901+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.902558+0000) 2019-10-13T10:59:22.904 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:22.901+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.902663+0000) 2019-10-13T10:59:22.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:22.901+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.902957+0000) 2019-10-13T10:59:22.905 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:22.901+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.903225+0000) 2019-10-13T10:59:22.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:22.904+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.905175+0000) 2019-10-13T10:59:22.905 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:22.902+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:22.903541+0000) 2019-10-13T10:59:22.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:22.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199622 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:23.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:23.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:23.017435+0000) 2019-10-13T10:59:23.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:23.019+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:23.019657+0000) 2019-10-13T10:59:25.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:25.020+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:25.020940+0000) 2019-10-13T10:59:25.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:25.670+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:25.671491+0000) 2019-10-13T10:59:25.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:25.670+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:25.671491+0000) 2019-10-13T10:59:27.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:27.910+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.912301+0000) 2019-10-13T10:59:27.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:27.911+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.912674+0000) 2019-10-13T10:59:27.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:27.912+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.913258+0000) 2019-10-13T10:59:27.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:27.911+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.912878+0000) 2019-10-13T10:59:27.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:27.911+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.912961+0000) 2019-10-13T10:59:27.915 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:27.911+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.913309+0000) 2019-10-13T10:59:27.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:27.912+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.913479+0000) 2019-10-13T10:59:27.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:27.916+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.917235+0000) 2019-10-13T10:59:27.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:27.919+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199652 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:28.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:28.578+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:28.579365+0000) 2019-10-13T10:59:28.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:28.914+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:28.914896+0000) 2019-10-13T10:59:31.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:31.529+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.529716+0000) 2019-10-13T10:59:32.924 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:32.922+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.922765+0000) 2019-10-13T10:59:32.924 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:32.922+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.922967+0000) 2019-10-13T10:59:32.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:32.922+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.923007+0000) 2019-10-13T10:59:32.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:32.923+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.924018+0000) 2019-10-13T10:59:32.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:32.924+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.925116+0000) 2019-10-13T10:59:32.925 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:32.924+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.924577+0000) 2019-10-13T10:59:32.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:32.924+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.924625+0000) 2019-10-13T10:59:32.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:32.926+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:32.927125+0000) 2019-10-13T10:59:32.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:32.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199681 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:33.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:33.017+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:33.017560+0000) 2019-10-13T10:59:33.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:33.019+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:33.019788+0000) 2019-10-13T10:59:35.023 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:35.023+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:35.023432+0000) 2019-10-13T10:59:35.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:35.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:35.671675+0000) 2019-10-13T10:59:35.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:35.671+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:35.671676+0000) 2019-10-13T10:59:37.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:37.933+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.933875+0000) 2019-10-13T10:59:37.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:37.933+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.933505+0000) 2019-10-13T10:59:37.935 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:37.933+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.933678+0000) 2019-10-13T10:59:37.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:37.933+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.934013+0000) 2019-10-13T10:59:37.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:37.933+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.934119+0000) 2019-10-13T10:59:37.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:37.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.934362+0000) 2019-10-13T10:59:37.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:37.934+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.934761+0000) 2019-10-13T10:59:37.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:37.936+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.937489+0000) 2019-10-13T10:59:37.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:37.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:38.580 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:38.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:38.579470+0000) 2019-10-13T10:59:38.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:38.914+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:38.915061+0000) 2019-10-13T10:59:41.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:41.529+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.529917+0000) 2019-10-13T10:59:42.944 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:42.942+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.943230+0000) 2019-10-13T10:59:42.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:42.943+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.944205+0000) 2019-10-13T10:59:42.945 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:42.943+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.943739+0000) 2019-10-13T10:59:42.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:42.943+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.943978+0000) 2019-10-13T10:59:42.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:42.944+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.944541+0000) 2019-10-13T10:59:42.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:42.944+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.944707+0000) 2019-10-13T10:59:42.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:42.944+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.945348+0000) 2019-10-13T10:59:42.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:42.946+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:42.946981+0000) 2019-10-13T10:59:42.953 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:42.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:43.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:43.017+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:43.017700+0000) 2019-10-13T10:59:43.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:43.019+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:43.019914+0000) 2019-10-13T10:59:45.026 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:45.025+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:45.025750+0000) 2019-10-13T10:59:45.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:45.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:45.671878+0000) 2019-10-13T10:59:45.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:45.671+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:45.671915+0000) 2019-10-13T10:59:47.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:47.954+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.954812+0000) 2019-10-13T10:59:47.956 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:47.953+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.954440+0000) 2019-10-13T10:59:47.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:47.953+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.954546+0000) 2019-10-13T10:59:47.956 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:47.953+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.954715+0000) 2019-10-13T10:59:47.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:47.953+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.954982+0000) 2019-10-13T10:59:47.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:47.953+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.955117+0000) 2019-10-13T10:59:47.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:47.954+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.955693+0000) 2019-10-13T10:59:47.958 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:47.957+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.958672+0000) 2019-10-13T10:59:47.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:47.961+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199772 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:48.204 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T10:59:48.204 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:48.209 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T10:59:48.209 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:48.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:48.578+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:48.579634+0000) 2019-10-13T10:59:48.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:48.914+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:48.915202+0000) 2019-10-13T10:59:51.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:51.529+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.530052+0000) 2019-10-13T10:59:52.965 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:52.964+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.965019+0000) 2019-10-13T10:59:52.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:52.963+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.964563+0000) 2019-10-13T10:59:52.966 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:52.963+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.964844+0000) 2019-10-13T10:59:52.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:52.963+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.964968+0000) 2019-10-13T10:59:52.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:52.963+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.965150+0000) 2019-10-13T10:59:52.967 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:52.964+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.965443+0000) 2019-10-13T10:59:52.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:52.966+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.967380+0000) 2019-10-13T10:59:52.967 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:52.964+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:52.965666+0000) 2019-10-13T10:59:52.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:52.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199801 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:53.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:53.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:53.017880+0000) 2019-10-13T10:59:53.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:53.020+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:53.020069+0000) 2019-10-13T10:59:55.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:55.027+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:55.028049+0000) 2019-10-13T10:59:55.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:55.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:55.672063+0000) 2019-10-13T10:59:55.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:55.671+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:55.672084+0000) 2019-10-13T10:59:57.975 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T10:59:57.973+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.975310+0000) 2019-10-13T10:59:57.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T10:59:57.973+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.974945+0000) 2019-10-13T10:59:57.976 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:57.973+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.975034+0000) 2019-10-13T10:59:57.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:57.973+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.975232+0000) 2019-10-13T10:59:57.977 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T10:59:57.973+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.975314+0000) 2019-10-13T10:59:57.977 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T10:59:57.974+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.975711+0000) 2019-10-13T10:59:57.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T10:59:57.974+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.975882+0000) 2019-10-13T10:59:57.979 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T10:59:57.977+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.978928+0000) 2019-10-13T10:59:57.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T10:59:57.982+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T10:59:58.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T10:59:58.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:58.579833+0000) 2019-10-13T10:59:58.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T10:59:58.914+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:58.915365+0000) 2019-10-13T11:00:01.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:01.529+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.530243+0000) 2019-10-13T11:00:02.986 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:02.983+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.985256+0000) 2019-10-13T11:00:02.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:02.984+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.985469+0000) 2019-10-13T11:00:02.987 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:02.984+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.985774+0000) 2019-10-13T11:00:02.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:02.984+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.985852+0000) 2019-10-13T11:00:02.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:02.984+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.985987+0000) 2019-10-13T11:00:02.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:02.985+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.987214+0000) 2019-10-13T11:00:02.988 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:02.984+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.986287+0000) 2019-10-13T11:00:02.989 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:02.987+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:02.989324+0000) 2019-10-13T11:00:02.995 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:02.992+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199862 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:03.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:03.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:03.017988+0000) 2019-10-13T11:00:03.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:03.019+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:03.020149+0000) 2019-10-13T11:00:05.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:05.029+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:05.030356+0000) 2019-10-13T11:00:05.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:05.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:05.672279+0000) 2019-10-13T11:00:05.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:05.671+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:05.672277+0000) 2019-10-13T11:00:07.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:07.994+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.996188+0000) 2019-10-13T11:00:08.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:07.994+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.996305+0000) 2019-10-13T11:00:08.154 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:07.995+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.996584+0000) 2019-10-13T11:00:08.155 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:07.996+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.997525+0000) 2019-10-13T11:00:08.156 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:07.998+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.999310+0000) 2019-10-13T11:00:08.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:07.995+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.997091+0000) 2019-10-13T11:00:08.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:07.995+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.997343+0000) 2019-10-13T11:00:08.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:07.996+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.997515+0000) 2019-10-13T11:00:08.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:08.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:08.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:08.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:08.579974+0000) 2019-10-13T11:00:08.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:08.914+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:08.915477+0000) 2019-10-13T11:00:11.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:11.529+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.530459+0000) 2019-10-13T11:00:13.007 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:13.006+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.006853+0000) 2019-10-13T11:00:13.008 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:13.004+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.005936+0000) 2019-10-13T11:00:13.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:13.005+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.006401+0000) 2019-10-13T11:00:13.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:13.004+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.006205+0000) 2019-10-13T11:00:13.008 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:13.005+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.006796+0000) 2019-10-13T11:00:13.008 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:13.005+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.006980+0000) 2019-10-13T11:00:13.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:13.006+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.007504+0000) 2019-10-13T11:00:13.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:13.009+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.009918+0000) 2019-10-13T11:00:13.015 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:13.012+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199921 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:13.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:13.016+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.018080+0000) 2019-10-13T11:00:13.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:13.019+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:13.020269+0000) 2019-10-13T11:00:15.033 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:15.031+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:15.032617+0000) 2019-10-13T11:00:15.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:15.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:15.672481+0000) 2019-10-13T11:00:15.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:15.672+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:15.672480+0000) 2019-10-13T11:00:18.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:18.015+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.016235+0000) 2019-10-13T11:00:18.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:18.015+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.016261+0000) 2019-10-13T11:00:18.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:18.016+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.016445+0000) 2019-10-13T11:00:18.018 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:18.016+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.016631+0000) 2019-10-13T11:00:18.018 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:18.016+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.016683+0000) 2019-10-13T11:00:18.019 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:18.016+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.017564+0000) 2019-10-13T11:00:18.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:18.018+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.019025+0000) 2019-10-13T11:00:18.019 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:18.016+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.017219+0000) 2019-10-13T11:00:18.025 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:18.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199953 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:18.290 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:00:18.290 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:18.295 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:00:18.295 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:18.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:18.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.580105+0000) 2019-10-13T11:00:18.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:18.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.915635+0000) 2019-10-13T11:00:21.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:21.530+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.530599+0000) 2019-10-13T11:00:23.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:23.017+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.018223+0000) 2019-10-13T11:00:23.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:23.019+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.020448+0000) 2019-10-13T11:00:23.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:23.025+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.025706+0000) 2019-10-13T11:00:23.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:23.026+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.026750+0000) 2019-10-13T11:00:23.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:23.025+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.026141+0000) 2019-10-13T11:00:23.027 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:23.025+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.026348+0000) 2019-10-13T11:00:23.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:23.026+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.026608+0000) 2019-10-13T11:00:23.028 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:23.026+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.026687+0000) 2019-10-13T11:00:23.028 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:23.026+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.027124+0000) 2019-10-13T11:00:23.030 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:23.029+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:23.030485+0000) 2019-10-13T11:00:23.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:23.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 199982 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:25.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:25.034+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:25.035034+0000) 2019-10-13T11:00:25.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:25.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:25.672710+0000) 2019-10-13T11:00:25.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:25.672+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:25.672710+0000) 2019-10-13T11:00:28.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:28.035+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.035995+0000) 2019-10-13T11:00:28.037 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:28.035+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.036170+0000) 2019-10-13T11:00:28.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:28.035+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.036351+0000) 2019-10-13T11:00:28.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:28.036+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.037704+0000) 2019-10-13T11:00:28.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:28.036+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.036688+0000) 2019-10-13T11:00:28.039 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:28.036+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.037090+0000) 2019-10-13T11:00:28.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:28.036+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.037300+0000) 2019-10-13T11:00:28.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:28.039+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.040004+0000) 2019-10-13T11:00:28.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:28.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200012 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:28.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:28.580+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.580274+0000) 2019-10-13T11:00:28.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:28.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.915806+0000) 2019-10-13T11:00:31.531 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:31.530+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.530802+0000) 2019-10-13T11:00:33.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:33.017+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.018421+0000) 2019-10-13T11:00:33.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:33.019+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.020679+0000) 2019-10-13T11:00:33.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:33.045+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.046507+0000) 2019-10-13T11:00:33.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:33.044+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.046051+0000) 2019-10-13T11:00:33.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:33.045+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.046611+0000) 2019-10-13T11:00:33.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:33.045+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.046504+0000) 2019-10-13T11:00:33.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:33.045+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.046862+0000) 2019-10-13T11:00:33.048 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:33.045+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.046902+0000) 2019-10-13T11:00:33.048 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:33.045+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.047218+0000) 2019-10-13T11:00:33.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:33.048+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:33.048746+0000) 2019-10-13T11:00:33.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:33.052+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:35.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:35.037+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:35.037428+0000) 2019-10-13T11:00:35.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:35.671+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:35.672899+0000) 2019-10-13T11:00:35.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:35.671+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:35.672899+0000) 2019-10-13T11:00:38.056 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:38.054+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.055427+0000) 2019-10-13T11:00:38.057 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:38.054+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.055764+0000) 2019-10-13T11:00:38.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:38.055+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.056375+0000) 2019-10-13T11:00:38.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:38.054+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.056243+0000) 2019-10-13T11:00:38.058 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:38.055+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.056617+0000) 2019-10-13T11:00:38.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:38.055+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.056682+0000) 2019-10-13T11:00:38.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:38.055+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.056820+0000) 2019-10-13T11:00:38.060 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:38.059+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.059838+0000) 2019-10-13T11:00:38.065 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:38.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:38.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:38.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.580455+0000) 2019-10-13T11:00:38.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:38.914+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.915989+0000) 2019-10-13T11:00:41.532 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:41.530+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.531011+0000) 2019-10-13T11:00:43.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:43.017+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.018630+0000) 2019-10-13T11:00:43.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:43.020+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.020816+0000) 2019-10-13T11:00:43.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:43.065+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.066400+0000) 2019-10-13T11:00:43.067 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:43.064+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.065827+0000) 2019-10-13T11:00:43.067 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:43.064+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.066157+0000) 2019-10-13T11:00:43.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:43.065+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.066489+0000) 2019-10-13T11:00:43.068 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:43.065+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.066747+0000) 2019-10-13T11:00:43.068 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:43.065+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.067246+0000) 2019-10-13T11:00:43.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:43.065+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.066861+0000) 2019-10-13T11:00:43.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:43.069+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:43.069903+0000) 2019-10-13T11:00:43.075 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:43.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200102 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:45.040 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:45.039+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:45.039820+0000) 2019-10-13T11:00:45.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:45.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:45.673076+0000) 2019-10-13T11:00:45.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:45.672+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:45.673077+0000) 2019-10-13T11:00:48.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:48.074+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.076487+0000) 2019-10-13T11:00:48.077 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:48.074+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.076053+0000) 2019-10-13T11:00:48.077 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:48.074+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.076274+0000) 2019-10-13T11:00:48.077 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:48.074+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.076331+0000) 2019-10-13T11:00:48.078 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:48.075+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.076894+0000) 2019-10-13T11:00:48.078 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:48.075+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.077319+0000) 2019-10-13T11:00:48.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:48.076+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.077464+0000) 2019-10-13T11:00:48.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:48.076+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.078557+0000) 2019-10-13T11:00:48.085 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:48.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200134 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:48.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:48.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.580626+0000) 2019-10-13T11:00:48.588 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:00:48.588 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:48.592 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:00:48.593 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:48.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:48.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.916121+0000) 2019-10-13T11:00:51.532 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:51.530+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.531207+0000) 2019-10-13T11:00:53.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:53.018+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.018816+0000) 2019-10-13T11:00:53.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:53.020+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.020979+0000) 2019-10-13T11:00:53.087 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:53.084+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.085943+0000) 2019-10-13T11:00:53.087 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:53.084+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.086238+0000) 2019-10-13T11:00:53.087 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:53.084+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.086099+0000) 2019-10-13T11:00:53.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:53.085+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.087425+0000) 2019-10-13T11:00:53.088 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:53.085+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.086832+0000) 2019-10-13T11:00:53.088 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:53.085+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.087104+0000) 2019-10-13T11:00:53.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:53.086+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.087422+0000) 2019-10-13T11:00:53.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:53.088+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:53.089825+0000) 2019-10-13T11:00:53.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:53.093+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200161 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:55.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:55.041+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:55.042231+0000) 2019-10-13T11:00:55.674 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:55.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:55.673284+0000) 2019-10-13T11:00:55.674 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:55.672+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:55.673283+0000) 2019-10-13T11:00:58.097 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:58.094+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.095997+0000) 2019-10-13T11:00:58.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:00:58.094+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.096187+0000) 2019-10-13T11:00:58.098 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:58.094+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.096358+0000) 2019-10-13T11:00:58.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:00:58.095+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.097401+0000) 2019-10-13T11:00:58.099 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:00:58.095+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.097016+0000) 2019-10-13T11:00:58.099 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:00:58.095+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.097068+0000) 2019-10-13T11:00:58.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:00:58.095+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.097255+0000) 2019-10-13T11:00:58.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:00:58.097+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.099328+0000) 2019-10-13T11:00:58.106 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:00:58.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:00:58.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:00:58.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.580820+0000) 2019-10-13T11:00:58.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:00:58.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.916285+0000) 2019-10-13T11:01:01.532 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:01.531+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.531423+0000) 2019-10-13T11:01:03.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:03.018+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.019056+0000) 2019-10-13T11:01:03.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:03.020+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.021179+0000) 2019-10-13T11:01:03.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:03.106+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.107338+0000) 2019-10-13T11:01:03.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:03.106+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.106778+0000) 2019-10-13T11:01:03.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:03.106+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.107061+0000) 2019-10-13T11:01:03.108 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:03.107+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.107415+0000) 2019-10-13T11:01:03.109 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:03.107+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.107514+0000) 2019-10-13T11:01:03.109 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:03.107+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.107790+0000) 2019-10-13T11:01:03.109 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:03.106+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.107285+0000) 2019-10-13T11:01:03.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:03.109+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:03.110104+0000) 2019-10-13T11:01:03.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:03.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200222 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:05.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:05.043+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:05.044630+0000) 2019-10-13T11:01:05.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:05.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:05.673478+0000) 2019-10-13T11:01:05.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:05.673+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:05.673477+0000) 2019-10-13T11:01:08.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:08.116+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.117349+0000) 2019-10-13T11:01:08.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:08.116+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.116804+0000) 2019-10-13T11:01:08.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:08.116+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.116957+0000) 2019-10-13T11:01:08.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:08.117+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.117404+0000) 2019-10-13T11:01:08.119 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:08.117+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.117800+0000) 2019-10-13T11:01:08.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:08.117+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.117895+0000) 2019-10-13T11:01:08.119 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:08.117+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.118095+0000) 2019-10-13T11:01:08.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:08.120+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.120763+0000) 2019-10-13T11:01:08.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:08.124+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:08.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:08.580+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.580986+0000) 2019-10-13T11:01:08.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:08.916+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.916448+0000) 2019-10-13T11:01:11.532 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:11.531+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.531605+0000) 2019-10-13T11:01:13.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:13.018+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.019195+0000) 2019-10-13T11:01:13.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:13.020+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.021362+0000) 2019-10-13T11:01:13.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:13.126+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.126965+0000) 2019-10-13T11:01:13.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:13.126+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.127188+0000) 2019-10-13T11:01:13.129 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:13.126+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.127313+0000) 2019-10-13T11:01:13.129 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:13.127+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.127484+0000) 2019-10-13T11:01:13.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:13.127+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.127651+0000) 2019-10-13T11:01:13.129 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:13.127+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.127852+0000) 2019-10-13T11:01:13.130 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:13.127+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.127864+0000) 2019-10-13T11:01:13.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:13.130+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:13.131022+0000) 2019-10-13T11:01:13.134 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:13.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:15.047 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:15.046+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:15.046934+0000) 2019-10-13T11:01:15.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:15.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:15.673707+0000) 2019-10-13T11:01:15.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:15.673+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:15.673705+0000) 2019-10-13T11:01:18.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:18.134+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.135224+0000) 2019-10-13T11:01:18.137 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:18.135+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.135610+0000) 2019-10-13T11:01:18.138 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:18.135+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.136689+0000) 2019-10-13T11:01:18.138 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:18.135+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.136039+0000) 2019-10-13T11:01:18.138 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:18.135+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.136152+0000) 2019-10-13T11:01:18.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:18.136+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.136497+0000) 2019-10-13T11:01:18.138 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:18.136+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.136667+0000) 2019-10-13T11:01:18.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:18.137+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.138579+0000) 2019-10-13T11:01:18.145 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:18.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:18.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:18.580+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.581179+0000) 2019-10-13T11:01:18.693 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:01:18.693 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:18.697 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:01:18.697 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:18.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:18.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.916584+0000) 2019-10-13T11:01:21.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:21.530+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.531782+0000) 2019-10-13T11:01:23.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:23.018+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.019359+0000) 2019-10-13T11:01:23.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:23.021+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.021568+0000) 2019-10-13T11:01:23.147 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:23.144+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.145803+0000) 2019-10-13T11:01:23.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:23.144+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.145998+0000) 2019-10-13T11:01:23.148 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:23.144+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.146248+0000) 2019-10-13T11:01:23.148 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:23.144+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.146078+0000) 2019-10-13T11:01:23.148 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:23.145+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.146529+0000) 2019-10-13T11:01:23.149 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:23.147+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.147981+0000) 2019-10-13T11:01:23.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:23.148+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.148752+0000) 2019-10-13T11:01:23.149 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:23.145+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:23.147054+0000) 2019-10-13T11:01:23.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:23.152+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200342 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:25.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:25.049+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:25.049399+0000) 2019-10-13T11:01:25.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:25.672+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:25.673887+0000) 2019-10-13T11:01:25.675 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:25.672+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:25.673881+0000) 2019-10-13T11:01:28.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:28.153+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.155295+0000) 2019-10-13T11:01:28.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:28.154+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.155499+0000) 2019-10-13T11:01:28.157 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:28.154+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.155633+0000) 2019-10-13T11:01:28.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:28.155+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.156633+0000) 2019-10-13T11:01:28.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:28.154+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.156223+0000) 2019-10-13T11:01:28.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:28.155+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.156432+0000) 2019-10-13T11:01:28.158 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:28.155+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.156589+0000) 2019-10-13T11:01:28.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:28.158+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.158864+0000) 2019-10-13T11:01:28.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:28.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:28.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:28.579+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.581306+0000) 2019-10-13T11:01:28.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:28.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.916689+0000) 2019-10-13T11:01:31.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:31.531+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.531948+0000) 2019-10-13T11:01:33.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:33.018+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.019544+0000) 2019-10-13T11:01:33.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:33.021+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.021768+0000) 2019-10-13T11:01:33.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:33.162+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.164211+0000) 2019-10-13T11:01:33.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:33.162+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.164381+0000) 2019-10-13T11:01:33.166 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:33.164+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.165328+0000) 2019-10-13T11:01:33.166 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:33.163+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.164747+0000) 2019-10-13T11:01:33.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:33.163+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.164868+0000) 2019-10-13T11:01:33.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:33.163+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.164979+0000) 2019-10-13T11:01:33.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:33.163+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.165141+0000) 2019-10-13T11:01:33.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:33.166+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:33.166824+0000) 2019-10-13T11:01:33.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:33.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200401 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:35.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:35.051+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:35.052624+0000) 2019-10-13T11:01:35.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:35.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:35.674063+0000) 2019-10-13T11:01:35.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:35.673+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:35.674062+0000) 2019-10-13T11:01:38.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:38.173+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.174513+0000) 2019-10-13T11:01:38.176 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:38.173+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.174681+0000) 2019-10-13T11:01:38.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:38.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.175858+0000) 2019-10-13T11:01:38.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:38.173+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.175212+0000) 2019-10-13T11:01:38.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:38.173+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.175322+0000) 2019-10-13T11:01:38.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:38.174+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.175530+0000) 2019-10-13T11:01:38.177 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:38.174+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.176046+0000) 2019-10-13T11:01:38.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:38.176+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.177744+0000) 2019-10-13T11:01:38.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:38.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:38.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:38.580+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.581432+0000) 2019-10-13T11:01:38.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:38.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.916847+0000) 2019-10-13T11:01:41.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:41.531+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.532143+0000) 2019-10-13T11:01:43.020 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:43.019+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.019729+0000) 2019-10-13T11:01:43.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:43.020+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.021931+0000) 2019-10-13T11:01:43.186 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:43.183+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.184796+0000) 2019-10-13T11:01:43.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:43.183+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.184976+0000) 2019-10-13T11:01:43.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:43.183+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.185370+0000) 2019-10-13T11:01:43.187 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:43.184+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.186711+0000) 2019-10-13T11:01:43.188 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:43.184+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.185523+0000) 2019-10-13T11:01:43.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:43.184+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.185727+0000) 2019-10-13T11:01:43.188 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:43.184+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.186083+0000) 2019-10-13T11:01:43.188 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:43.186+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:43.188590+0000) 2019-10-13T11:01:43.194 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:43.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:45.055 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:45.054+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:45.054920+0000) 2019-10-13T11:01:45.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:45.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:45.674258+0000) 2019-10-13T11:01:45.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:45.673+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:45.674257+0000) 2019-10-13T11:01:48.196 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:48.194+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.194915+0000) 2019-10-13T11:01:48.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:48.194+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.195051+0000) 2019-10-13T11:01:48.197 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:48.194+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.196476+0000) 2019-10-13T11:01:48.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:48.195+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.195495+0000) 2019-10-13T11:01:48.198 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:48.195+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.195537+0000) 2019-10-13T11:01:48.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:48.195+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.196019+0000) 2019-10-13T11:01:48.198 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:48.195+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.196336+0000) 2019-10-13T11:01:48.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:48.197+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.199420+0000) 2019-10-13T11:01:48.204 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:48.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200493 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:48.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:48.581+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.581522+0000) 2019-10-13T11:01:48.855 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:01:48.856 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:48.860 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:01:48.860 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:48.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:48.916+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.916964+0000) 2019-10-13T11:01:51.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:51.532+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.532385+0000) 2019-10-13T11:01:53.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:53.019+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.019892+0000) 2019-10-13T11:01:53.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:53.021+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.022069+0000) 2019-10-13T11:01:53.206 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:53.204+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.205150+0000) 2019-10-13T11:01:53.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:53.205+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.205406+0000) 2019-10-13T11:01:53.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:53.205+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.205591+0000) 2019-10-13T11:01:53.208 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:53.205+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.206497+0000) 2019-10-13T11:01:53.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:53.205+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.206362+0000) 2019-10-13T11:01:53.208 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:53.206+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.206471+0000) 2019-10-13T11:01:53.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:53.206+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.206668+0000) 2019-10-13T11:01:53.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:53.208+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:53.209221+0000) 2019-10-13T11:01:53.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:53.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200519 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:55.057 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:55.056+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:55.057164+0000) 2019-10-13T11:01:55.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:55.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:55.674460+0000) 2019-10-13T11:01:55.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:55.674+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:55.674459+0000) 2019-10-13T11:01:58.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:01:58.215+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.216331+0000) 2019-10-13T11:01:58.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:01:58.215+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.215415+0000) 2019-10-13T11:01:58.218 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:01:58.215+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.215736+0000) 2019-10-13T11:01:58.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:58.215+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.215890+0000) 2019-10-13T11:01:58.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:58.215+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.216229+0000) 2019-10-13T11:01:58.219 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:01:58.216+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.216822+0000) 2019-10-13T11:01:58.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:01:58.216+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.216996+0000) 2019-10-13T11:01:58.219 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:01:58.218+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.219610+0000) 2019-10-13T11:01:58.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:01:58.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:01:58.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:01:58.581+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.581652+0000) 2019-10-13T11:01:58.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:01:58.916+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.917125+0000) 2019-10-13T11:02:01.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:01.532+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.532565+0000) 2019-10-13T11:02:03.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:03.019+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.020041+0000) 2019-10-13T11:02:03.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:03.021+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.022258+0000) 2019-10-13T11:02:03.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:03.225+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.226173+0000) 2019-10-13T11:02:03.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:03.225+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.226225+0000) 2019-10-13T11:02:03.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:03.226+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.226476+0000) 2019-10-13T11:02:03.229 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:03.227+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.227774+0000) 2019-10-13T11:02:03.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:03.226+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.227138+0000) 2019-10-13T11:02:03.229 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:03.226+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.227254+0000) 2019-10-13T11:02:03.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:03.227+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.227793+0000) 2019-10-13T11:02:03.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:03.228+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:03.229656+0000) 2019-10-13T11:02:03.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:03.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:05.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:05.058+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:05.059545+0000) 2019-10-13T11:02:05.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:05.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:05.674645+0000) 2019-10-13T11:02:05.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:05.673+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:05.674645+0000) 2019-10-13T11:02:08.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:08.235+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.236720+0000) 2019-10-13T11:02:08.238 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:08.235+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.236759+0000) 2019-10-13T11:02:08.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:08.235+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.237048+0000) 2019-10-13T11:02:08.239 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:08.237+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.238075+0000) 2019-10-13T11:02:08.239 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:08.236+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.237701+0000) 2019-10-13T11:02:08.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:08.236+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.237847+0000) 2019-10-13T11:02:08.240 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:08.236+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.238062+0000) 2019-10-13T11:02:08.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:08.239+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.240100+0000) 2019-10-13T11:02:08.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:08.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:08.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:08.580+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.581820+0000) 2019-10-13T11:02:08.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:08.915+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.917258+0000) 2019-10-13T11:02:11.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:11.531+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.532750+0000) 2019-10-13T11:02:13.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:13.018+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.020168+0000) 2019-10-13T11:02:13.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:13.021+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.022439+0000) 2019-10-13T11:02:13.247 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:13.244+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.246239+0000) 2019-10-13T11:02:13.248 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:13.244+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.246325+0000) 2019-10-13T11:02:13.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:13.245+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.246506+0000) 2019-10-13T11:02:13.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:13.245+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.246786+0000) 2019-10-13T11:02:13.249 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:13.246+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.247305+0000) 2019-10-13T11:02:13.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:13.245+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.247202+0000) 2019-10-13T11:02:13.249 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:13.246+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.247491+0000) 2019-10-13T11:02:13.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:13.249+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:13.250127+0000) 2019-10-13T11:02:13.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:13.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:15.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:15.061+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:15.061861+0000) 2019-10-13T11:02:15.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:15.673+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:15.674833+0000) 2019-10-13T11:02:15.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:15.673+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:15.674834+0000) 2019-10-13T11:02:18.256 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:18.253+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.255112+0000) 2019-10-13T11:02:18.256 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:18.254+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.255456+0000) 2019-10-13T11:02:18.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:18.254+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.255617+0000) 2019-10-13T11:02:18.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:18.256+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.256990+0000) 2019-10-13T11:02:18.258 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:18.255+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.256427+0000) 2019-10-13T11:02:18.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:18.255+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.256625+0000) 2019-10-13T11:02:18.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:18.255+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.256700+0000) 2019-10-13T11:02:18.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:18.257+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.258471+0000) 2019-10-13T11:02:18.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:18.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:18.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:18.580+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.581944+0000) 2019-10-13T11:02:18.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:18.916+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.917393+0000) 2019-10-13T11:02:19.009 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:02:19.010 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:19.014 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:02:19.014 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:21.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:21.532+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.532946+0000) 2019-10-13T11:02:23.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:23.019+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.020332+0000) 2019-10-13T11:02:23.022 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:23.021+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.022656+0000) 2019-10-13T11:02:23.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:23.265+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.266234+0000) 2019-10-13T11:02:23.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:23.263+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.265377+0000) 2019-10-13T11:02:23.268 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:23.264+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.265803+0000) 2019-10-13T11:02:23.268 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:23.264+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.265668+0000) 2019-10-13T11:02:23.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:23.264+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.266234+0000) 2019-10-13T11:02:23.268 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:23.264+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.266341+0000) 2019-10-13T11:02:23.269 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:23.265+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.266534+0000) 2019-10-13T11:02:23.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:23.268+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:23.268988+0000) 2019-10-13T11:02:23.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:23.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:25.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:25.064+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:25.064315+0000) 2019-10-13T11:02:25.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:25.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:25.674982+0000) 2019-10-13T11:02:25.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:25.674+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:25.674982+0000) 2019-10-13T11:02:28.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:28.273+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.275136+0000) 2019-10-13T11:02:28.277 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:28.273+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.275370+0000) 2019-10-13T11:02:28.277 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:28.274+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.275550+0000) 2019-10-13T11:02:28.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:28.275+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.277552+0000) 2019-10-13T11:02:28.278 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:28.274+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.275887+0000) 2019-10-13T11:02:28.279 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:28.275+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.276405+0000) 2019-10-13T11:02:28.279 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:28.275+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.276409+0000) 2019-10-13T11:02:28.279 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:28.277+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.279235+0000) 2019-10-13T11:02:28.285 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:28.283+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200731 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:28.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:28.581+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.582107+0000) 2019-10-13T11:02:28.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:28.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.917561+0000) 2019-10-13T11:02:31.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:31.532+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.533151+0000) 2019-10-13T11:02:33.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:33.020+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.020510+0000) 2019-10-13T11:02:33.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:33.021+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.022820+0000) 2019-10-13T11:02:33.288 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:33.286+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.287158+0000) 2019-10-13T11:02:33.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:33.287+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.287839+0000) 2019-10-13T11:02:33.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:33.287+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.288004+0000) 2019-10-13T11:02:33.289 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:33.287+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.288176+0000) 2019-10-13T11:02:33.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:33.287+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.288371+0000) 2019-10-13T11:02:33.290 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:33.287+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.289391+0000) 2019-10-13T11:02:33.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:33.288+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.288818+0000) 2019-10-13T11:02:33.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:33.289+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:33.291096+0000) 2019-10-13T11:02:33.296 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:33.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:35.067 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:35.064+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:35.066634+0000) 2019-10-13T11:02:35.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:35.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:35.675166+0000) 2019-10-13T11:02:35.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:35.674+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:35.675166+0000) 2019-10-13T11:02:38.298 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:38.296+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.298171+0000) 2019-10-13T11:02:38.299 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:38.297+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.297672+0000) 2019-10-13T11:02:38.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:38.297+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.297910+0000) 2019-10-13T11:02:38.299 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:38.297+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.297980+0000) 2019-10-13T11:02:38.300 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:38.298+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.298404+0000) 2019-10-13T11:02:38.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:38.298+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.298478+0000) 2019-10-13T11:02:38.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:38.298+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.298666+0000) 2019-10-13T11:02:38.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:38.298+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.300632+0000) 2019-10-13T11:02:38.307 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:38.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200793 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:38.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:38.581+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.582301+0000) 2019-10-13T11:02:38.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:38.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.917690+0000) 2019-10-13T11:02:41.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:41.533+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.533344+0000) 2019-10-13T11:02:43.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:43.020+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.020698+0000) 2019-10-13T11:02:43.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:43.022+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.023005+0000) 2019-10-13T11:02:43.317 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:43.308+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.309161+0000) 2019-10-13T11:02:43.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:43.310+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.311620+0000) 2019-10-13T11:02:43.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:43.307+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.308232+0000) 2019-10-13T11:02:43.318 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:43.308+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.308716+0000) 2019-10-13T11:02:43.318 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:43.308+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.309026+0000) 2019-10-13T11:02:43.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:43.308+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.309171+0000) 2019-10-13T11:02:43.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:43.308+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.309303+0000) 2019-10-13T11:02:43.319 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:43.309+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:43.309664+0000) 2019-10-13T11:02:43.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:43.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:45.069 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:45.068+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:45.068976+0000) 2019-10-13T11:02:45.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:45.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:45.675361+0000) 2019-10-13T11:02:45.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:45.675+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:45.675367+0000) 2019-10-13T11:02:48.319 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:48.317+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.318172+0000) 2019-10-13T11:02:48.319 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:48.317+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.318370+0000) 2019-10-13T11:02:48.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:48.318+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.318539+0000) 2019-10-13T11:02:48.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:48.319+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.319929+0000) 2019-10-13T11:02:48.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:48.318+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.318989+0000) 2019-10-13T11:02:48.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:48.319+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.319645+0000) 2019-10-13T11:02:48.321 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:48.319+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.319852+0000) 2019-10-13T11:02:48.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:48.321+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.322347+0000) 2019-10-13T11:02:48.328 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:48.326+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:48.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:48.582+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.582484+0000) 2019-10-13T11:02:48.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:48.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.917831+0000) 2019-10-13T11:02:49.164 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:02:49.164 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:49.168 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:02:49.168 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:51.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:51.532+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.533593+0000) 2019-10-13T11:02:53.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:53.019+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.020910+0000) 2019-10-13T11:02:53.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:53.022+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.023209+0000) 2019-10-13T11:02:53.329 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:53.327+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.328664+0000) 2019-10-13T11:02:53.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:53.327+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.328936+0000) 2019-10-13T11:02:53.331 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:53.329+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.329861+0000) 2019-10-13T11:02:53.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:53.327+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.329320+0000) 2019-10-13T11:02:53.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:53.328+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.329499+0000) 2019-10-13T11:02:53.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:53.328+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.329860+0000) 2019-10-13T11:02:53.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:53.328+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.330330+0000) 2019-10-13T11:02:53.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:53.331+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:53.332688+0000) 2019-10-13T11:02:53.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:53.335+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200883 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:55.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:55.070+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:55.071565+0000) 2019-10-13T11:02:55.676 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:55.674+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:55.675558+0000) 2019-10-13T11:02:55.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:55.674+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:55.675578+0000) 2019-10-13T11:02:58.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:02:58.337+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.338609+0000) 2019-10-13T11:02:58.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:02:58.337+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.338693+0000) 2019-10-13T11:02:58.340 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:58.337+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.338855+0000) 2019-10-13T11:02:58.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:02:58.337+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.338986+0000) 2019-10-13T11:02:58.341 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:58.337+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.339113+0000) 2019-10-13T11:02:58.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:02:58.337+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.339203+0000) 2019-10-13T11:02:58.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:02:58.339+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.340063+0000) 2019-10-13T11:02:58.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:02:58.342+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.343433+0000) 2019-10-13T11:02:58.346 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:02:58.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200915 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:02:58.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:02:58.581+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.582652+0000) 2019-10-13T11:02:58.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:02:58.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.917943+0000) 2019-10-13T11:03:01.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:01.532+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.533787+0000) 2019-10-13T11:03:03.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:03.020+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.021097+0000) 2019-10-13T11:03:03.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:03.023+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.023396+0000) 2019-10-13T11:03:03.349 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:03.346+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.348266+0000) 2019-10-13T11:03:03.349 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:03.347+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.348499+0000) 2019-10-13T11:03:03.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:03.347+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.348723+0000) 2019-10-13T11:03:03.350 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:03.347+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.348789+0000) 2019-10-13T11:03:03.350 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:03.347+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.349026+0000) 2019-10-13T11:03:03.351 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:03.347+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.349137+0000) 2019-10-13T11:03:03.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:03.349+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.350426+0000) 2019-10-13T11:03:03.352 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:03.351+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:03.352495+0000) 2019-10-13T11:03:03.356 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:03.354+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:05.074 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:05.073+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:05.074057+0000) 2019-10-13T11:03:05.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:05.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:05.675742+0000) 2019-10-13T11:03:05.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:05.675+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:05.675741+0000) 2019-10-13T11:03:08.358 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:08.355+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.357340+0000) 2019-10-13T11:03:08.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:08.356+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.357424+0000) 2019-10-13T11:03:08.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:08.356+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.357711+0000) 2019-10-13T11:03:08.359 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:08.356+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.358142+0000) 2019-10-13T11:03:08.360 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:08.357+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.358484+0000) 2019-10-13T11:03:08.360 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:08.357+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.358923+0000) 2019-10-13T11:03:08.360 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:08.359+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.359902+0000) 2019-10-13T11:03:08.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:08.360+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.361276+0000) 2019-10-13T11:03:08.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:08.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 200975 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:08.584 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:08.582+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.582825+0000) 2019-10-13T11:03:08.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:08.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.918114+0000) 2019-10-13T11:03:11.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:11.532+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.534024+0000) 2019-10-13T11:03:13.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:13.020+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.021276+0000) 2019-10-13T11:03:13.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:13.023+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.023551+0000) 2019-10-13T11:03:13.367 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:13.364+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.365675+0000) 2019-10-13T11:03:13.367 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:13.364+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.365839+0000) 2019-10-13T11:03:13.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:13.364+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.366098+0000) 2019-10-13T11:03:13.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:13.366+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.367204+0000) 2019-10-13T11:03:13.368 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:13.364+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.366310+0000) 2019-10-13T11:03:13.368 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:13.364+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.366205+0000) 2019-10-13T11:03:13.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:13.365+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.366593+0000) 2019-10-13T11:03:13.369 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:13.368+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:13.368943+0000) 2019-10-13T11:03:13.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:13.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201005 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:15.076 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:15.076+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:15.076370+0000) 2019-10-13T11:03:15.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:15.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:15.675945+0000) 2019-10-13T11:03:15.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:15.675+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:15.675942+0000) 2019-10-13T11:03:18.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:18.372+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.374032+0000) 2019-10-13T11:03:18.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:18.372+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.374226+0000) 2019-10-13T11:03:18.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:18.373+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.374415+0000) 2019-10-13T11:03:18.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:18.373+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.374721+0000) 2019-10-13T11:03:18.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:18.373+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.374931+0000) 2019-10-13T11:03:18.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:18.373+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.374947+0000) 2019-10-13T11:03:18.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:18.374+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.375420+0000) 2019-10-13T11:03:18.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:18.377+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.377908+0000) 2019-10-13T11:03:18.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:18.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201037 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:18.584 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:18.582+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.583013+0000) 2019-10-13T11:03:18.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:18.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.918255+0000) 2019-10-13T11:03:19.318 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:03:19.319 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:19.324 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:03:19.324 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:21.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:21.533+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.534212+0000) 2019-10-13T11:03:23.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:23.021+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.021430+0000) 2019-10-13T11:03:23.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:23.022+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.023756+0000) 2019-10-13T11:03:23.385 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:23.383+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.383946+0000) 2019-10-13T11:03:23.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:23.383+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.384102+0000) 2019-10-13T11:03:23.385 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:23.384+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.384475+0000) 2019-10-13T11:03:23.386 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:23.383+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.385009+0000) 2019-10-13T11:03:23.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:23.384+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.384692+0000) 2019-10-13T11:03:23.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:23.384+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.384777+0000) 2019-10-13T11:03:23.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:23.384+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.385078+0000) 2019-10-13T11:03:23.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:23.386+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:23.388004+0000) 2019-10-13T11:03:23.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:23.391+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:25.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:25.078+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:25.079350+0000) 2019-10-13T11:03:25.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:25.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:25.676132+0000) 2019-10-13T11:03:25.677 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:25.675+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:25.676132+0000) 2019-10-13T11:03:28.395 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:28.393+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.394827+0000) 2019-10-13T11:03:28.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:28.394+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.394541+0000) 2019-10-13T11:03:28.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:28.394+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.394799+0000) 2019-10-13T11:03:28.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:28.394+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.394972+0000) 2019-10-13T11:03:28.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:28.394+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.394886+0000) 2019-10-13T11:03:28.397 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:28.394+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.395144+0000) 2019-10-13T11:03:28.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:28.395+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.395425+0000) 2019-10-13T11:03:28.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:28.395+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.397076+0000) 2019-10-13T11:03:28.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:28.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:28.584 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:28.582+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.583117+0000) 2019-10-13T11:03:28.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:28.918+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.918391+0000) 2019-10-13T11:03:31.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:31.534+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.534415+0000) 2019-10-13T11:03:33.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:33.021+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.021630+0000) 2019-10-13T11:03:33.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:33.023+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.023961+0000) 2019-10-13T11:03:33.404 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:33.402+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.402885+0000) 2019-10-13T11:03:33.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:33.403+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.403791+0000) 2019-10-13T11:03:33.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:33.403+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.403403+0000) 2019-10-13T11:03:33.405 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:33.402+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.403327+0000) 2019-10-13T11:03:33.405 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:33.403+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.403569+0000) 2019-10-13T11:03:33.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:33.403+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.403758+0000) 2019-10-13T11:03:33.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:33.403+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.404302+0000) 2019-10-13T11:03:33.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:33.406+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:33.406981+0000) 2019-10-13T11:03:33.412 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:33.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:35.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:35.080+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:35.081710+0000) 2019-10-13T11:03:35.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:35.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:35.676301+0000) 2019-10-13T11:03:35.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:35.676+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:35.676301+0000) 2019-10-13T11:03:38.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:38.412+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.413528+0000) 2019-10-13T11:03:38.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:38.411+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.412993+0000) 2019-10-13T11:03:38.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:38.411+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.413284+0000) 2019-10-13T11:03:38.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:38.411+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.413188+0000) 2019-10-13T11:03:38.415 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:38.412+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.413695+0000) 2019-10-13T11:03:38.415 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:38.412+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.413973+0000) 2019-10-13T11:03:38.416 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:38.412+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.414051+0000) 2019-10-13T11:03:38.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:38.416+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.417366+0000) 2019-10-13T11:03:38.422 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:38.419+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201159 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:38.584 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:38.582+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.583315+0000) 2019-10-13T11:03:38.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:38.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.918535+0000) 2019-10-13T11:03:41.535 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:41.533+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.534623+0000) 2019-10-13T11:03:43.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:43.020+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.021786+0000) 2019-10-13T11:03:43.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:43.023+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.024117+0000) 2019-10-13T11:03:43.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:43.422+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.423545+0000) 2019-10-13T11:03:43.424 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:43.421+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.423074+0000) 2019-10-13T11:03:43.424 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:43.421+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.423166+0000) 2019-10-13T11:03:43.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:43.422+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.423714+0000) 2019-10-13T11:03:43.425 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:43.422+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.423970+0000) 2019-10-13T11:03:43.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:43.422+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.424151+0000) 2019-10-13T11:03:43.426 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:43.422+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.424197+0000) 2019-10-13T11:03:43.426 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:43.425+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:43.426289+0000) 2019-10-13T11:03:43.433 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:43.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:45.095 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:45.083+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:45.084057+0000) 2019-10-13T11:03:45.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:45.675+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:45.676473+0000) 2019-10-13T11:03:45.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:45.675+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:45.676481+0000) 2019-10-13T11:03:48.435 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:48.432+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.434166+0000) 2019-10-13T11:03:48.436 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:48.432+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.434342+0000) 2019-10-13T11:03:48.436 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:48.433+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.434692+0000) 2019-10-13T11:03:48.436 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:48.433+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.434776+0000) 2019-10-13T11:03:48.437 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:48.434+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.435664+0000) 2019-10-13T11:03:48.437 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:48.433+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.435133+0000) 2019-10-13T11:03:48.437 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:48.434+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.435613+0000) 2019-10-13T11:03:48.438 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:48.437+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.438282+0000) 2019-10-13T11:03:48.443 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:48.441+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:48.584 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:48.582+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.583513+0000) 2019-10-13T11:03:48.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:48.917+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.918704+0000) 2019-10-13T11:03:49.406 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:03:49.406 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:49.410 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:03:49.411 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:51.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:51.534+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.534835+0000) 2019-10-13T11:03:53.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:53.021+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.021963+0000) 2019-10-13T11:03:53.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:53.023+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.024272+0000) 2019-10-13T11:03:53.446 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:53.443+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.444464+0000) 2019-10-13T11:03:53.446 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:53.443+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.444551+0000) 2019-10-13T11:03:53.446 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:53.443+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.444834+0000) 2019-10-13T11:03:53.446 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:53.443+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.445172+0000) 2019-10-13T11:03:53.447 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:53.445+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.446418+0000) 2019-10-13T11:03:53.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:53.446+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.447131+0000) 2019-10-13T11:03:53.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:53.443+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.445376+0000) 2019-10-13T11:03:53.448 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:53.444+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:53.445500+0000) 2019-10-13T11:03:53.453 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:53.451+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:55.086 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:55.086+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:55.086462+0000) 2019-10-13T11:03:55.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:55.676+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:55.676663+0000) 2019-10-13T11:03:55.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:55.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:55.676663+0000) 2019-10-13T11:03:58.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:03:58.452+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.454137+0000) 2019-10-13T11:03:58.456 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:58.453+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.454508+0000) 2019-10-13T11:03:58.456 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:58.453+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.454728+0000) 2019-10-13T11:03:58.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:03:58.453+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.454781+0000) 2019-10-13T11:03:58.456 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:03:58.453+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.454914+0000) 2019-10-13T11:03:58.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:03:58.455+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.455927+0000) 2019-10-13T11:03:58.457 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:03:58.454+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.455547+0000) 2019-10-13T11:03:58.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:03:58.457+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.457920+0000) 2019-10-13T11:03:58.465 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:03:58.461+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201281 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:03:58.584 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:03:58.583+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.583656+0000) 2019-10-13T11:03:58.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:03:58.918+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.918892+0000) 2019-10-13T11:04:01.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:01.534+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.535033+0000) 2019-10-13T11:04:03.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:03.021+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.022180+0000) 2019-10-13T11:04:03.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:03.024+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.024480+0000) 2019-10-13T11:04:03.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:03.463+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.464717+0000) 2019-10-13T11:04:03.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:03.463+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.464935+0000) 2019-10-13T11:04:03.466 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:03.463+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.465036+0000) 2019-10-13T11:04:03.467 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:03.465+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.465869+0000) 2019-10-13T11:04:03.467 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:03.464+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.465481+0000) 2019-10-13T11:04:03.467 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:03.463+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.465343+0000) 2019-10-13T11:04:03.468 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:03.464+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.466022+0000) 2019-10-13T11:04:03.468 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:03.467+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:03.468440+0000) 2019-10-13T11:04:03.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:03.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:05.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:05.088+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:05.088804+0000) 2019-10-13T11:04:05.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:05.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:05.676851+0000) 2019-10-13T11:04:05.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:05.676+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:05.676849+0000) 2019-10-13T11:04:08.476 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:08.474+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.474691+0000) 2019-10-13T11:04:08.477 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:08.474+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.474909+0000) 2019-10-13T11:04:08.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:08.474+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.475297+0000) 2019-10-13T11:04:08.477 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:08.474+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.475260+0000) 2019-10-13T11:04:08.477 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:08.475+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.475490+0000) 2019-10-13T11:04:08.478 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:08.475+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.476480+0000) 2019-10-13T11:04:08.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:08.475+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.476061+0000) 2019-10-13T11:04:08.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:08.477+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.478658+0000) 2019-10-13T11:04:08.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:08.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201341 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:08.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:08.583+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.583812+0000) 2019-10-13T11:04:08.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:08.918+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.918999+0000) 2019-10-13T11:04:11.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:11.534+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.535234+0000) 2019-10-13T11:04:13.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:13.021+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.022369+0000) 2019-10-13T11:04:13.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:13.022+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.024702+0000) 2019-10-13T11:04:13.486 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:13.484+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.485171+0000) 2019-10-13T11:04:13.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:13.484+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.486035+0000) 2019-10-13T11:04:13.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:13.485+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.485737+0000) 2019-10-13T11:04:13.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:13.485+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.485864+0000) 2019-10-13T11:04:13.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:13.485+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.486129+0000) 2019-10-13T11:04:13.488 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:13.486+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.486474+0000) 2019-10-13T11:04:13.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:13.486+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.486662+0000) 2019-10-13T11:04:13.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:13.486+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:13.487865+0000) 2019-10-13T11:04:13.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:13.494+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:15.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:15.089+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:15.090911+0000) 2019-10-13T11:04:15.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:15.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:15.677067+0000) 2019-10-13T11:04:15.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:15.676+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:15.677065+0000) 2019-10-13T11:04:18.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:18.496+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.496420+0000) 2019-10-13T11:04:18.498 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:18.496+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.496605+0000) 2019-10-13T11:04:18.498 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:18.496+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.496795+0000) 2019-10-13T11:04:18.498 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:18.496+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.496995+0000) 2019-10-13T11:04:18.499 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:18.496+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.498264+0000) 2019-10-13T11:04:18.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:18.496+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.497337+0000) 2019-10-13T11:04:18.499 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:18.497+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.497545+0000) 2019-10-13T11:04:18.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:18.498+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.500675+0000) 2019-10-13T11:04:18.508 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:18.504+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201403 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:18.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:18.583+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.584037+0000) 2019-10-13T11:04:18.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:18.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.919158+0000) 2019-10-13T11:04:19.536 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:04:19.536 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:19.541 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:04:19.541 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:21.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:21.535+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.535441+0000) 2019-10-13T11:04:23.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:23.021+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.022548+0000) 2019-10-13T11:04:23.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:23.024+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.024940+0000) 2019-10-13T11:04:23.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:23.505+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.506719+0000) 2019-10-13T11:04:23.508 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:23.505+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.506893+0000) 2019-10-13T11:04:23.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:23.505+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.507004+0000) 2019-10-13T11:04:23.508 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:23.505+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.507211+0000) 2019-10-13T11:04:23.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:23.506+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.508629+0000) 2019-10-13T11:04:23.509 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:23.506+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.507756+0000) 2019-10-13T11:04:23.510 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:23.506+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.508122+0000) 2019-10-13T11:04:23.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:23.508+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:23.510539+0000) 2019-10-13T11:04:23.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:23.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:25.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:25.092+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:25.093241+0000) 2019-10-13T11:04:25.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:25.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:25.677252+0000) 2019-10-13T11:04:25.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:25.676+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:25.677279+0000) 2019-10-13T11:04:28.521 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:28.517+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.519104+0000) 2019-10-13T11:04:28.521 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:28.517+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.519285+0000) 2019-10-13T11:04:28.521 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:28.518+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.519445+0000) 2019-10-13T11:04:28.521 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:28.518+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.519609+0000) 2019-10-13T11:04:28.522 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:28.518+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.519811+0000) 2019-10-13T11:04:28.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:28.518+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.519970+0000) 2019-10-13T11:04:28.522 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:28.520+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.520993+0000) 2019-10-13T11:04:28.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:28.522+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.523245+0000) 2019-10-13T11:04:28.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:28.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:28.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:28.583+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.584199+0000) 2019-10-13T11:04:28.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:28.918+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.919306+0000) 2019-10-13T11:04:31.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:31.534+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.535637+0000) 2019-10-13T11:04:33.023 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:33.021+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.022737+0000) 2019-10-13T11:04:33.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:33.024+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.025142+0000) 2019-10-13T11:04:33.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:33.527+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.529309+0000) 2019-10-13T11:04:33.530 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:33.528+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.529442+0000) 2019-10-13T11:04:33.531 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:33.529+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.530319+0000) 2019-10-13T11:04:33.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:33.528+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.529798+0000) 2019-10-13T11:04:33.532 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:33.528+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.530019+0000) 2019-10-13T11:04:33.532 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:33.528+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.530263+0000) 2019-10-13T11:04:33.532 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:33.529+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.530560+0000) 2019-10-13T11:04:33.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:33.531+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:33.531937+0000) 2019-10-13T11:04:33.538 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:33.536+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201493 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:35.096 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:35.095+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:35.096203+0000) 2019-10-13T11:04:35.678 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:35.676+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:35.677473+0000) 2019-10-13T11:04:35.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:35.676+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:35.677478+0000) 2019-10-13T11:04:38.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:38.538+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.539699+0000) 2019-10-13T11:04:38.541 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:38.538+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.539934+0000) 2019-10-13T11:04:38.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:38.538+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.540379+0000) 2019-10-13T11:04:38.542 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:38.539+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.540750+0000) 2019-10-13T11:04:38.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:38.539+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.541149+0000) 2019-10-13T11:04:38.543 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:38.541+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.541745+0000) 2019-10-13T11:04:38.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:38.539+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.541375+0000) 2019-10-13T11:04:38.543 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:38.543+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.543768+0000) 2019-10-13T11:04:38.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:38.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:38.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:38.582+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.584366+0000) 2019-10-13T11:04:38.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:38.918+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.919467+0000) 2019-10-13T11:04:41.537 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:41.535+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.535812+0000) 2019-10-13T11:04:43.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:43.022+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.022944+0000) 2019-10-13T11:04:43.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:43.024+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.025326+0000) 2019-10-13T11:04:43.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:43.547+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.549246+0000) 2019-10-13T11:04:43.551 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:43.548+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.550362+0000) 2019-10-13T11:04:43.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:43.549+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.550531+0000) 2019-10-13T11:04:43.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:43.550+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.551696+0000) 2019-10-13T11:04:43.552 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:43.549+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.550727+0000) 2019-10-13T11:04:43.553 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:43.549+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.551289+0000) 2019-10-13T11:04:43.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:43.550+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.551614+0000) 2019-10-13T11:04:43.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:43.553+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:43.554718+0000) 2019-10-13T11:04:43.559 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:43.556+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:45.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:45.098+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:45.098457+0000) 2019-10-13T11:04:45.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:45.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:45.677687+0000) 2019-10-13T11:04:45.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:45.677+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:45.677689+0000) 2019-10-13T11:04:48.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:48.558+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.559817+0000) 2019-10-13T11:04:48.562 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:48.560+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.561362+0000) 2019-10-13T11:04:48.562 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:48.558+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.560291+0000) 2019-10-13T11:04:48.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:48.559+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.560454+0000) 2019-10-13T11:04:48.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:48.559+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.560681+0000) 2019-10-13T11:04:48.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:48.559+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.560895+0000) 2019-10-13T11:04:48.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:48.559+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.560925+0000) 2019-10-13T11:04:48.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:48.562+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.562814+0000) 2019-10-13T11:04:48.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:48.566+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:48.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:48.584+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.584526+0000) 2019-10-13T11:04:48.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:48.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.919579+0000) 2019-10-13T11:04:49.632 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:04:49.633 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:49.637 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:04:49.637 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:51.537 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:51.535+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.535987+0000) 2019-10-13T11:04:53.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:53.022+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.023128+0000) 2019-10-13T11:04:53.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:53.025+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.025484+0000) 2019-10-13T11:04:53.570 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:53.569+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.569439+0000) 2019-10-13T11:04:53.570 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:53.569+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.569786+0000) 2019-10-13T11:04:53.571 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:53.570+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.570989+0000) 2019-10-13T11:04:53.572 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:53.569+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.569992+0000) 2019-10-13T11:04:53.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:53.569+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.570208+0000) 2019-10-13T11:04:53.572 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:53.570+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.570519+0000) 2019-10-13T11:04:53.572 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:53.570+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.570661+0000) 2019-10-13T11:04:53.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:53.572+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:53.573269+0000) 2019-10-13T11:04:53.578 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:53.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:55.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:55.100+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:55.100763+0000) 2019-10-13T11:04:55.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:55.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:55.677839+0000) 2019-10-13T11:04:55.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:55.677+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:55.677854+0000) 2019-10-13T11:04:58.581 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:58.579+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.579903+0000) 2019-10-13T11:04:58.581 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:58.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.580334+0000) 2019-10-13T11:04:58.582 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:04:58.580+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.580919+0000) 2019-10-13T11:04:58.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:04:58.580+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.580674+0000) 2019-10-13T11:04:58.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:04:58.580+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.580961+0000) 2019-10-13T11:04:58.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:04:58.580+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.581167+0000) 2019-10-13T11:04:58.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:04:58.581+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.581479+0000) 2019-10-13T11:04:58.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:04:58.582+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.583395+0000) 2019-10-13T11:04:58.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:04:58.584+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.584627+0000) 2019-10-13T11:04:58.588 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:04:58.587+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:04:58.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:04:58.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.919722+0000) 2019-10-13T11:05:01.537 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:01.535+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.536178+0000) 2019-10-13T11:05:03.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:03.022+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.023316+0000) 2019-10-13T11:05:03.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:03.024+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.025639+0000) 2019-10-13T11:05:03.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:03.589+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.589681+0000) 2019-10-13T11:05:03.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:03.589+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.590009+0000) 2019-10-13T11:05:03.591 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:03.589+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.590123+0000) 2019-10-13T11:05:03.592 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:03.588+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.590618+0000) 2019-10-13T11:05:03.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:03.590+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.590508+0000) 2019-10-13T11:05:03.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:03.590+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.590956+0000) 2019-10-13T11:05:03.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:03.590+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.591156+0000) 2019-10-13T11:05:03.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:03.591+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:03.593204+0000) 2019-10-13T11:05:03.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:03.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:05.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:05.102+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:05.103037+0000) 2019-10-13T11:05:05.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:05.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:05.678025+0000) 2019-10-13T11:05:05.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:05.677+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:05.678025+0000) 2019-10-13T11:05:08.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:08.584+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.584797+0000) 2019-10-13T11:05:08.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:08.599+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.600922+0000) 2019-10-13T11:05:08.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:08.600+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.600525+0000) 2019-10-13T11:05:08.602 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:08.600+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.600637+0000) 2019-10-13T11:05:08.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:08.600+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.600777+0000) 2019-10-13T11:05:08.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:08.600+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.600916+0000) 2019-10-13T11:05:08.603 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:08.600+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.601106+0000) 2019-10-13T11:05:08.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:08.601+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.602836+0000) 2019-10-13T11:05:08.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:08.601+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.601672+0000) 2019-10-13T11:05:08.609 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:08.608+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:08.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:08.918+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.919895+0000) 2019-10-13T11:05:11.537 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:11.535+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.536373+0000) 2019-10-13T11:05:13.024 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:13.022+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.023425+0000) 2019-10-13T11:05:13.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:13.025+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.025815+0000) 2019-10-13T11:05:13.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:13.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.610851+0000) 2019-10-13T11:05:13.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:13.609+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.610430+0000) 2019-10-13T11:05:13.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:13.609+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.610489+0000) 2019-10-13T11:05:13.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:13.609+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.610966+0000) 2019-10-13T11:05:13.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:13.609+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.611221+0000) 2019-10-13T11:05:13.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:13.609+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.611248+0000) 2019-10-13T11:05:13.613 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:13.609+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.611379+0000) 2019-10-13T11:05:13.614 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:13.612+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:13.614571+0000) 2019-10-13T11:05:13.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:13.617+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:15.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:15.104+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:15.105599+0000) 2019-10-13T11:05:15.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:15.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:15.678223+0000) 2019-10-13T11:05:15.679 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:15.677+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:15.678225+0000) 2019-10-13T11:05:18.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:18.584+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.584953+0000) 2019-10-13T11:05:18.621 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:18.620+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.621254+0000) 2019-10-13T11:05:18.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:18.619+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.621171+0000) 2019-10-13T11:05:18.622 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:18.620+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.621414+0000) 2019-10-13T11:05:18.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:18.620+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.621767+0000) 2019-10-13T11:05:18.623 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:18.620+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.622034+0000) 2019-10-13T11:05:18.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:18.620+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.622191+0000) 2019-10-13T11:05:18.623 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:18.620+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.622358+0000) 2019-10-13T11:05:18.624 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:18.622+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.623707+0000) 2019-10-13T11:05:18.630 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:18.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:18.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:18.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.920001+0000) 2019-10-13T11:05:19.719 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:05:19.719 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:19.723 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:05:19.724 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:21.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:21.535+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.536586+0000) 2019-10-13T11:05:23.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:23.022+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.023590+0000) 2019-10-13T11:05:23.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:23.025+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.025992+0000) 2019-10-13T11:05:23.631 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:23.629+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.630803+0000) 2019-10-13T11:05:23.632 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:23.629+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.630997+0000) 2019-10-13T11:05:23.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:23.631+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.631789+0000) 2019-10-13T11:05:23.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:23.629+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.631179+0000) 2019-10-13T11:05:23.633 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:23.629+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.631384+0000) 2019-10-13T11:05:23.633 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:23.630+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.631952+0000) 2019-10-13T11:05:23.633 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:23.630+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.632192+0000) 2019-10-13T11:05:23.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:23.633+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:23.634355+0000) 2019-10-13T11:05:23.641 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:23.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:25.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:25.107+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:25.108179+0000) 2019-10-13T11:05:25.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:25.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:25.678430+0000) 2019-10-13T11:05:25.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:25.677+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:25.678430+0000) 2019-10-13T11:05:28.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:28.584+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.585182+0000) 2019-10-13T11:05:28.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:28.640+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.641546+0000) 2019-10-13T11:05:28.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:28.639+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.641063+0000) 2019-10-13T11:05:28.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:28.640+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.641545+0000) 2019-10-13T11:05:28.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:28.640+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.641793+0000) 2019-10-13T11:05:28.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:28.640+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.641924+0000) 2019-10-13T11:05:28.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:28.640+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.642210+0000) 2019-10-13T11:05:28.644 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:28.640+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.642086+0000) 2019-10-13T11:05:28.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:28.643+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.643966+0000) 2019-10-13T11:05:28.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:28.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:28.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:28.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.920080+0000) 2019-10-13T11:05:31.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:31.536+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.536789+0000) 2019-10-13T11:05:33.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:33.023+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.023774+0000) 2019-10-13T11:05:33.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:33.025+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.026177+0000) 2019-10-13T11:05:33.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:33.649+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.651248+0000) 2019-10-13T11:05:33.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:33.651+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.652653+0000) 2019-10-13T11:05:33.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:33.650+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.652089+0000) 2019-10-13T11:05:33.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:33.650+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.652315+0000) 2019-10-13T11:05:33.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:33.651+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.652497+0000) 2019-10-13T11:05:33.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:33.651+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.652763+0000) 2019-10-13T11:05:33.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:33.651+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.652964+0000) 2019-10-13T11:05:33.656 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:33.655+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:33.655950+0000) 2019-10-13T11:05:33.660 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:33.658+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:35.111 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:35.110+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:35.110668+0000) 2019-10-13T11:05:35.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:35.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:35.678622+0000) 2019-10-13T11:05:35.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:35.678+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:35.678619+0000) 2019-10-13T11:05:38.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:38.584+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.585356+0000) 2019-10-13T11:05:38.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:38.661+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.662361+0000) 2019-10-13T11:05:38.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:38.661+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.661609+0000) 2019-10-13T11:05:38.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:38.661+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.661811+0000) 2019-10-13T11:05:38.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:38.661+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.661950+0000) 2019-10-13T11:05:38.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:38.662+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.662446+0000) 2019-10-13T11:05:38.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:38.662+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.662531+0000) 2019-10-13T11:05:38.664 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:38.662+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.662988+0000) 2019-10-13T11:05:38.665 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:38.664+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.665031+0000) 2019-10-13T11:05:38.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:38.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201891 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:38.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:38.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.920223+0000) 2019-10-13T11:05:41.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:41.536+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.536986+0000) 2019-10-13T11:05:43.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:43.023+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.023964+0000) 2019-10-13T11:05:43.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:43.025+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.026359+0000) 2019-10-13T11:05:43.673 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:43.671+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.672244+0000) 2019-10-13T11:05:43.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:43.672+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.672448+0000) 2019-10-13T11:05:43.674 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:43.672+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.673386+0000) 2019-10-13T11:05:43.674 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:43.672+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.672890+0000) 2019-10-13T11:05:43.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:43.672+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.673094+0000) 2019-10-13T11:05:43.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:43.672+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.673278+0000) 2019-10-13T11:05:43.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:43.673+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.673492+0000) 2019-10-13T11:05:43.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:43.674+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:43.675572+0000) 2019-10-13T11:05:43.680 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:43.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:45.113 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:45.112+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:45.113013+0000) 2019-10-13T11:05:45.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:45.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:45.678806+0000) 2019-10-13T11:05:45.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:45.678+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:45.678805+0000) 2019-10-13T11:05:48.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:48.585+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.585543+0000) 2019-10-13T11:05:48.683 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:48.680+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.681332+0000) 2019-10-13T11:05:48.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:48.681+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.681464+0000) 2019-10-13T11:05:48.683 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:48.681+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.681924+0000) 2019-10-13T11:05:48.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:48.681+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.682119+0000) 2019-10-13T11:05:48.685 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:48.681+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.682378+0000) 2019-10-13T11:05:48.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:48.683+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.684577+0000) 2019-10-13T11:05:48.685 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:48.682+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.682605+0000) 2019-10-13T11:05:48.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:48.682+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.682750+0000) 2019-10-13T11:05:48.691 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:48.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:48.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:48.920+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.920419+0000) 2019-10-13T11:05:49.827 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:05:49.828 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:49.832 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:05:49.832 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:51.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:51.536+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.537176+0000) 2019-10-13T11:05:53.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:53.024+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.024150+0000) 2019-10-13T11:05:53.026 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:53.025+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.026601+0000) 2019-10-13T11:05:53.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:53.691+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.691697+0000) 2019-10-13T11:05:53.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:53.691+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.691759+0000) 2019-10-13T11:05:53.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:53.691+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.692071+0000) 2019-10-13T11:05:53.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:53.690+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.692614+0000) 2019-10-13T11:05:53.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:53.692+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.692493+0000) 2019-10-13T11:05:53.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:53.692+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.692918+0000) 2019-10-13T11:05:53.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:53.691+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.692291+0000) 2019-10-13T11:05:53.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:53.694+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:53.696098+0000) 2019-10-13T11:05:53.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:53.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 201981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:55.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:55.113+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:55.115443+0000) 2019-10-13T11:05:55.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:55.677+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:55.679021+0000) 2019-10-13T11:05:55.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:55.677+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:55.679023+0000) 2019-10-13T11:05:58.586 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:58.584+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.585743+0000) 2019-10-13T11:05:58.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:05:58.700+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.702283+0000) 2019-10-13T11:05:58.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:05:58.701+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.702464+0000) 2019-10-13T11:05:58.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:05:58.701+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.702586+0000) 2019-10-13T11:05:58.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:05:58.701+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.702866+0000) 2019-10-13T11:05:58.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:05:58.701+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.703162+0000) 2019-10-13T11:05:58.705 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:58.701+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.703376+0000) 2019-10-13T11:05:58.705 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:05:58.702+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.704417+0000) 2019-10-13T11:05:58.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:05:58.703+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.704985+0000) 2019-10-13T11:05:58.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:05:58.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:05:58.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:05:58.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.920593+0000) 2019-10-13T11:06:01.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:01.536+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.537321+0000) 2019-10-13T11:06:03.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:03.023+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.024315+0000) 2019-10-13T11:06:03.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:03.025+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.026809+0000) 2019-10-13T11:06:03.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:03.710+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.711580+0000) 2019-10-13T11:06:03.713 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:03.710+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.711687+0000) 2019-10-13T11:06:03.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:03.710+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.711973+0000) 2019-10-13T11:06:03.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:03.711+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.712803+0000) 2019-10-13T11:06:03.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:03.710+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.712312+0000) 2019-10-13T11:06:03.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:03.711+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.712486+0000) 2019-10-13T11:06:03.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:03.711+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.712837+0000) 2019-10-13T11:06:03.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:03.713+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:03.715506+0000) 2019-10-13T11:06:03.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:03.717+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:05.118 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:05.116+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:05.117780+0000) 2019-10-13T11:06:05.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:05.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:05.679206+0000) 2019-10-13T11:06:05.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:05.678+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:05.679208+0000) 2019-10-13T11:06:08.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:08.585+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.585901+0000) 2019-10-13T11:06:08.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:08.720+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.721111+0000) 2019-10-13T11:06:08.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:08.719+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.720523+0000) 2019-10-13T11:06:08.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:08.719+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.720749+0000) 2019-10-13T11:06:08.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:08.719+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.720927+0000) 2019-10-13T11:06:08.723 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:08.719+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.721259+0000) 2019-10-13T11:06:08.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:08.720+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.721525+0000) 2019-10-13T11:06:08.723 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:08.720+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.721552+0000) 2019-10-13T11:06:08.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:08.723+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.723801+0000) 2019-10-13T11:06:08.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:08.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202073 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:08.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:08.919+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.920702+0000) 2019-10-13T11:06:11.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:11.536+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.537507+0000) 2019-10-13T11:06:13.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:13.023+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.024446+0000) 2019-10-13T11:06:13.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:13.026+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.026991+0000) 2019-10-13T11:06:13.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:13.728+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.730351+0000) 2019-10-13T11:06:13.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:13.729+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.730617+0000) 2019-10-13T11:06:13.732 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:13.729+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.730762+0000) 2019-10-13T11:06:13.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:13.729+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.730866+0000) 2019-10-13T11:06:13.733 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:13.729+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.731020+0000) 2019-10-13T11:06:13.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:13.731+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.732257+0000) 2019-10-13T11:06:13.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:13.730+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.731637+0000) 2019-10-13T11:06:13.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:13.734+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:13.734761+0000) 2019-10-13T11:06:13.740 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:13.737+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:15.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:15.119+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:15.120274+0000) 2019-10-13T11:06:15.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:15.678+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:15.679428+0000) 2019-10-13T11:06:15.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:15.678+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:15.679428+0000) 2019-10-13T11:06:18.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:18.585+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.586072+0000) 2019-10-13T11:06:18.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:18.739+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.740872+0000) 2019-10-13T11:06:18.742 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:18.739+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.741010+0000) 2019-10-13T11:06:18.742 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:18.739+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.741255+0000) 2019-10-13T11:06:18.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:18.741+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.742223+0000) 2019-10-13T11:06:18.743 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:18.740+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.741864+0000) 2019-10-13T11:06:18.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:18.740+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.742198+0000) 2019-10-13T11:06:18.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:18.740+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.742265+0000) 2019-10-13T11:06:18.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:18.743+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.744199+0000) 2019-10-13T11:06:18.750 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:18.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202135 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:18.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:18.920+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.920861+0000) 2019-10-13T11:06:19.941 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:06:19.941 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:19.945 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:06:19.945 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:21.538 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:21.537+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.537733+0000) 2019-10-13T11:06:23.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:23.024+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.024616+0000) 2019-10-13T11:06:23.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:23.026+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.027187+0000) 2019-10-13T11:06:23.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:23.750+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.751133+0000) 2019-10-13T11:06:23.752 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:23.750+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.751300+0000) 2019-10-13T11:06:23.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:23.751+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.751507+0000) 2019-10-13T11:06:23.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:23.751+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.751776+0000) 2019-10-13T11:06:23.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:23.751+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.752411+0000) 2019-10-13T11:06:23.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:23.752+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.752457+0000) 2019-10-13T11:06:23.754 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:23.752+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.752593+0000) 2019-10-13T11:06:23.755 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:23.754+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:23.755601+0000) 2019-10-13T11:06:23.759 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:23.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:25.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:25.122+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:25.122615+0000) 2019-10-13T11:06:25.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:25.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:25.679613+0000) 2019-10-13T11:06:25.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:25.679+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:25.679613+0000) 2019-10-13T11:06:28.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:28.585+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.586279+0000) 2019-10-13T11:06:28.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:28.759+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.760336+0000) 2019-10-13T11:06:28.761 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:28.760+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.760493+0000) 2019-10-13T11:06:28.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:28.760+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.761361+0000) 2019-10-13T11:06:28.763 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:28.760+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.761193+0000) 2019-10-13T11:06:28.763 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:28.761+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.761541+0000) 2019-10-13T11:06:28.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:28.761+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.761674+0000) 2019-10-13T11:06:28.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:28.761+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.762040+0000) 2019-10-13T11:06:28.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:28.764+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.764996+0000) 2019-10-13T11:06:28.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:28.768+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:28.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:28.920+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.921035+0000) 2019-10-13T11:06:31.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:31.537+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:31.537932+0000) 2019-10-13T11:06:33.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:33.024+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.024765+0000) 2019-10-13T11:06:33.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:33.027+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.027390+0000) 2019-10-13T11:06:33.771 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:33.770+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.770617+0000) 2019-10-13T11:06:33.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:33.770+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.770818+0000) 2019-10-13T11:06:33.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:33.770+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.771004+0000) 2019-10-13T11:06:33.773 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:33.771+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.772165+0000) 2019-10-13T11:06:33.773 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:33.771+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.771688+0000) 2019-10-13T11:06:33.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:33.771+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.771829+0000) 2019-10-13T11:06:33.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:33.771+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.772244+0000) 2019-10-13T11:06:33.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:33.773+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:33.774453+0000) 2019-10-13T11:06:33.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:33.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202225 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:35.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:35.124+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:35.124920+0000) 2019-10-13T11:06:35.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:35.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:35.679798+0000) 2019-10-13T11:06:35.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:35.679+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:35.679801+0000) 2019-10-13T11:06:38.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:38.586+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.586458+0000) 2019-10-13T11:06:38.780 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:38.779+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.780422+0000) 2019-10-13T11:06:38.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:38.779+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.779864+0000) 2019-10-13T11:06:38.782 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:38.779+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.780092+0000) 2019-10-13T11:06:38.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:38.780+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.780475+0000) 2019-10-13T11:06:38.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:38.780+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.780534+0000) 2019-10-13T11:06:38.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:38.780+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.780754+0000) 2019-10-13T11:06:38.783 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:38.780+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.780883+0000) 2019-10-13T11:06:38.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:38.782+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.783213+0000) 2019-10-13T11:06:38.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:38.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202257 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:38.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:38.921+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.921199+0000) 2019-10-13T11:06:41.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:41.537+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:41.538132+0000) 2019-10-13T11:06:43.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:43.023+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.024908+0000) 2019-10-13T11:06:43.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:43.027+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.027594+0000) 2019-10-13T11:06:43.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:43.787+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.789281+0000) 2019-10-13T11:06:43.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:43.789+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.790754+0000) 2019-10-13T11:06:43.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:43.788+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.789850+0000) 2019-10-13T11:06:43.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:43.788+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.790178+0000) 2019-10-13T11:06:43.792 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:43.789+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.790510+0000) 2019-10-13T11:06:43.792 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:43.789+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.790692+0000) 2019-10-13T11:06:43.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:43.789+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.790901+0000) 2019-10-13T11:06:43.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:43.790+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:43.792689+0000) 2019-10-13T11:06:43.798 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:43.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:45.127 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:45.126+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:45.127277+0000) 2019-10-13T11:06:45.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:45.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:45.679984+0000) 2019-10-13T11:06:45.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:45.679+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:45.679985+0000) 2019-10-13T11:06:48.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:48.585+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.586654+0000) 2019-10-13T11:06:48.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:48.798+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.800145+0000) 2019-10-13T11:06:48.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:48.798+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.799606+0000) 2019-10-13T11:06:48.801 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:48.798+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.799772+0000) 2019-10-13T11:06:48.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:48.798+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.800015+0000) 2019-10-13T11:06:48.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:48.798+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.800059+0000) 2019-10-13T11:06:48.802 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:48.798+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.800295+0000) 2019-10-13T11:06:48.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:48.799+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.800965+0000) 2019-10-13T11:06:48.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:48.801+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.803614+0000) 2019-10-13T11:06:48.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:48.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202317 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:48.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:48.920+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.921370+0000) 2019-10-13T11:06:50.064 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:06:50.064 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:50.068 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:06:50.068 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:51.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:51.537+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:51.538316+0000) 2019-10-13T11:06:53.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:53.024+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.025083+0000) 2019-10-13T11:06:53.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:53.026+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.027780+0000) 2019-10-13T11:06:53.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:53.808+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.810183+0000) 2019-10-13T11:06:53.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:53.808+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.810079+0000) 2019-10-13T11:06:53.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:53.809+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.810445+0000) 2019-10-13T11:06:53.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:53.809+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.810544+0000) 2019-10-13T11:06:53.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:53.809+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.810687+0000) 2019-10-13T11:06:53.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:53.809+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.810815+0000) 2019-10-13T11:06:53.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:53.809+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.811024+0000) 2019-10-13T11:06:53.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:53.811+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:53.812816+0000) 2019-10-13T11:06:53.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:53.816+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:55.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:55.128+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:55.129690+0000) 2019-10-13T11:06:55.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:55.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:55.680164+0000) 2019-10-13T11:06:55.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:55.679+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:55.680166+0000) 2019-10-13T11:06:58.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:58.586+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.586880+0000) 2019-10-13T11:06:58.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:06:58.818+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.819641+0000) 2019-10-13T11:06:58.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:58.818+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.819737+0000) 2019-10-13T11:06:58.821 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:06:58.818+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.819960+0000) 2019-10-13T11:06:58.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:06:58.819+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.820570+0000) 2019-10-13T11:06:58.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:06:58.819+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.820443+0000) 2019-10-13T11:06:58.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:06:58.819+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.820590+0000) 2019-10-13T11:06:58.823 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:06:58.819+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.820706+0000) 2019-10-13T11:06:58.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:06:58.823+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.823733+0000) 2019-10-13T11:06:58.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:06:58.825+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202379 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:06:58.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:06:58.920+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.921545+0000) 2019-10-13T11:07:01.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:01.537+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:01.538517+0000) 2019-10-13T11:07:03.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:03.024+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.025278+0000) 2019-10-13T11:07:03.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:03.027+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.027925+0000) 2019-10-13T11:07:03.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:03.826+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.828155+0000) 2019-10-13T11:07:03.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:03.826+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.828352+0000) 2019-10-13T11:07:03.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:03.827+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.828702+0000) 2019-10-13T11:07:03.831 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:03.829+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.830001+0000) 2019-10-13T11:07:03.831 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:03.827+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.828967+0000) 2019-10-13T11:07:03.831 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:03.827+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.828980+0000) 2019-10-13T11:07:03.831 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:03.828+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.829542+0000) 2019-10-13T11:07:03.832 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:03.830+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:03.831652+0000) 2019-10-13T11:07:03.837 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:03.835+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:05.132 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:05.131+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:05.132194+0000) 2019-10-13T11:07:05.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:05.679+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:05.680376+0000) 2019-10-13T11:07:05.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:05.679+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:05.680381+0000) 2019-10-13T11:07:08.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:08.586+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.587078+0000) 2019-10-13T11:07:08.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:08.838+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.839019+0000) 2019-10-13T11:07:08.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:08.838+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.839253+0000) 2019-10-13T11:07:08.841 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:08.839+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.839451+0000) 2019-10-13T11:07:08.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:08.839+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.840341+0000) 2019-10-13T11:07:08.842 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:08.839+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.839878+0000) 2019-10-13T11:07:08.842 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:08.839+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.840053+0000) 2019-10-13T11:07:08.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:08.839+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.840227+0000) 2019-10-13T11:07:08.843 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:08.842+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.843200+0000) 2019-10-13T11:07:08.847 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:08.846+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202440 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:08.922 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:08.921+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.921708+0000) 2019-10-13T11:07:11.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:11.538+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:11.538751+0000) 2019-10-13T11:07:13.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:13.025+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.025468+0000) 2019-10-13T11:07:13.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:13.027+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.028120+0000) 2019-10-13T11:07:13.849 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:13.847+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.848292+0000) 2019-10-13T11:07:13.849 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:13.848+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.849251+0000) 2019-10-13T11:07:13.850 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:13.848+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.849216+0000) 2019-10-13T11:07:13.850 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:13.848+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.849369+0000) 2019-10-13T11:07:13.851 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:13.849+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.849519+0000) 2019-10-13T11:07:13.851 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:13.849+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.849634+0000) 2019-10-13T11:07:13.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:13.849+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.849817+0000) 2019-10-13T11:07:13.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:13.852+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:13.853256+0000) 2019-10-13T11:07:13.857 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:13.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202470 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:15.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:15.134+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:15.135156+0000) 2019-10-13T11:07:15.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:15.680+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:15.680573+0000) 2019-10-13T11:07:15.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:15.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:15.680608+0000) 2019-10-13T11:07:18.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:18.587+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.587267+0000) 2019-10-13T11:07:18.859 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:18.858+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.858390+0000) 2019-10-13T11:07:18.860 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:18.858+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.858573+0000) 2019-10-13T11:07:18.860 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:18.858+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.858820+0000) 2019-10-13T11:07:18.861 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:18.859+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.860545+0000) 2019-10-13T11:07:18.861 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:18.858+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.859362+0000) 2019-10-13T11:07:18.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:18.859+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.859729+0000) 2019-10-13T11:07:18.861 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:18.859+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.859933+0000) 2019-10-13T11:07:18.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:18.861+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.862023+0000) 2019-10-13T11:07:18.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:18.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202503 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:18.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:18.921+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.921859+0000) 2019-10-13T11:07:20.261 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:07:20.262 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:20.266 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:07:20.266 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:21.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:21.538+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:21.538920+0000) 2019-10-13T11:07:23.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:23.025+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.025596+0000) 2019-10-13T11:07:23.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:23.027+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.028342+0000) 2019-10-13T11:07:23.869 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:23.868+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.868649+0000) 2019-10-13T11:07:23.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:23.868+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.868849+0000) 2019-10-13T11:07:23.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:23.869+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.869855+0000) 2019-10-13T11:07:23.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:23.869+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.869448+0000) 2019-10-13T11:07:23.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:23.869+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.869629+0000) 2019-10-13T11:07:23.871 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:23.869+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.869844+0000) 2019-10-13T11:07:23.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:23.869+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.870146+0000) 2019-10-13T11:07:23.872 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:23.871+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:23.872120+0000) 2019-10-13T11:07:23.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:23.877+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202531 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:25.137 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:25.137+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:25.137401+0000) 2019-10-13T11:07:25.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:25.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:25.680744+0000) 2019-10-13T11:07:25.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:25.680+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:25.680738+0000) 2019-10-13T11:07:28.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:28.585+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.587380+0000) 2019-10-13T11:07:28.879 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:28.877+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.878478+0000) 2019-10-13T11:07:28.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:28.877+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.878760+0000) 2019-10-13T11:07:28.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:28.879+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.879954+0000) 2019-10-13T11:07:28.881 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:28.877+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.879366+0000) 2019-10-13T11:07:28.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:28.878+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.879473+0000) 2019-10-13T11:07:28.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:28.878+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.879445+0000) 2019-10-13T11:07:28.881 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:28.878+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.879831+0000) 2019-10-13T11:07:28.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:28.881+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.881987+0000) 2019-10-13T11:07:28.886 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:28.883+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202564 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:28.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:28.920+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.922009+0000) 2019-10-13T11:07:31.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:31.538+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:31.539117+0000) 2019-10-13T11:07:33.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:33.024+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.025778+0000) 2019-10-13T11:07:33.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:33.028+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.028517+0000) 2019-10-13T11:07:33.887 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:33.886+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.887472+0000) 2019-10-13T11:07:33.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:33.885+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.887167+0000) 2019-10-13T11:07:33.889 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:33.885+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.887362+0000) 2019-10-13T11:07:33.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:33.886+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.887473+0000) 2019-10-13T11:07:33.889 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:33.886+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.887439+0000) 2019-10-13T11:07:33.889 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:33.886+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.887794+0000) 2019-10-13T11:07:33.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:33.886+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.888166+0000) 2019-10-13T11:07:33.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:33.889+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:33.890713+0000) 2019-10-13T11:07:33.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:33.893+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202594 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:35.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:35.138+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:35.139903+0000) 2019-10-13T11:07:35.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:35.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:35.680900+0000) 2019-10-13T11:07:35.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:35.680+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:35.680901+0000) 2019-10-13T11:07:38.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:38.586+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.587591+0000) 2019-10-13T11:07:38.898 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:38.895+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.897088+0000) 2019-10-13T11:07:38.898 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:38.895+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.897299+0000) 2019-10-13T11:07:38.898 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:38.896+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.897506+0000) 2019-10-13T11:07:38.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:38.896+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.898628+0000) 2019-10-13T11:07:38.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:38.896+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.898003+0000) 2019-10-13T11:07:38.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:38.896+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.898358+0000) 2019-10-13T11:07:38.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:38.897+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.898847+0000) 2019-10-13T11:07:38.901 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:38.899+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.901054+0000) 2019-10-13T11:07:38.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:38.904+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:38.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:38.921+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.922141+0000) 2019-10-13T11:07:41.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:41.538+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:41.539284+0000) 2019-10-13T11:07:43.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:43.025+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.025988+0000) 2019-10-13T11:07:43.029 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:43.027+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.028727+0000) 2019-10-13T11:07:43.907 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:43.905+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.906803+0000) 2019-10-13T11:07:43.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:43.905+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.906917+0000) 2019-10-13T11:07:43.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:43.906+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.908428+0000) 2019-10-13T11:07:43.909 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:43.905+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.907311+0000) 2019-10-13T11:07:43.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:43.906+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.907427+0000) 2019-10-13T11:07:43.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:43.906+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.907927+0000) 2019-10-13T11:07:43.909 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:43.906+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.908162+0000) 2019-10-13T11:07:43.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:43.908+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:43.910657+0000) 2019-10-13T11:07:43.914 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:43.911+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202655 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:45.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:45.140+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:45.142262+0000) 2019-10-13T11:07:45.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:45.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:45.681090+0000) 2019-10-13T11:07:45.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:45.680+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:45.681090+0000) 2019-10-13T11:07:48.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:48.587+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.587753+0000) 2019-10-13T11:07:48.916 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:48.913+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.914953+0000) 2019-10-13T11:07:48.916 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:48.913+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.915177+0000) 2019-10-13T11:07:48.917 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:48.915+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.915989+0000) 2019-10-13T11:07:48.917 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:48.914+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.915473+0000) 2019-10-13T11:07:48.917 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:48.914+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.916051+0000) 2019-10-13T11:07:48.918 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:48.914+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.916179+0000) 2019-10-13T11:07:48.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:48.914+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.916307+0000) 2019-10-13T11:07:48.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:48.918+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.918817+0000) 2019-10-13T11:07:48.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:48.920+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.922275+0000) 2019-10-13T11:07:48.924 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:48.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202688 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:50.501 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:07:50.501 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:50.505 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:07:50.505 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:51.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:51.538+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:51.539495+0000) 2019-10-13T11:07:53.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:53.025+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.026169+0000) 2019-10-13T11:07:53.029 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:53.028+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.028938+0000) 2019-10-13T11:07:53.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:53.924+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.924851+0000) 2019-10-13T11:07:53.926 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:53.924+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.925208+0000) 2019-10-13T11:07:53.927 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:53.925+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.926111+0000) 2019-10-13T11:07:53.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:53.925+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.925554+0000) 2019-10-13T11:07:53.927 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:53.925+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.926080+0000) 2019-10-13T11:07:53.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:53.925+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.926249+0000) 2019-10-13T11:07:53.928 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:53.926+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.926633+0000) 2019-10-13T11:07:53.929 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:53.928+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:53.929285+0000) 2019-10-13T11:07:53.934 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:53.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202718 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:07:55.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:55.144+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:55.144691+0000) 2019-10-13T11:07:55.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:55.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:55.681289+0000) 2019-10-13T11:07:55.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:55.680+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:55.681289+0000) 2019-10-13T11:07:58.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:58.587+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.587941+0000) 2019-10-13T11:07:58.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:58.922+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.922455+0000) 2019-10-13T11:07:58.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:07:58.934+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.935297+0000) 2019-10-13T11:07:58.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:07:58.935+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.936160+0000) 2019-10-13T11:07:58.937 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:07:58.935+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.935554+0000) 2019-10-13T11:07:58.937 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:07:58.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.935980+0000) 2019-10-13T11:07:58.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:07:58.935+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.936333+0000) 2019-10-13T11:07:58.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:07:58.936+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.936616+0000) 2019-10-13T11:07:58.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:07:58.936+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.936864+0000) 2019-10-13T11:07:58.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:07:58.937+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.938667+0000) 2019-10-13T11:07:58.944 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:07:58.943+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202751 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:01.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:01.539+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:01.539674+0000) 2019-10-13T11:08:03.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:03.026+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.026374+0000) 2019-10-13T11:08:03.029 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:03.028+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.029091+0000) 2019-10-13T11:08:03.946 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:03.945+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.945787+0000) 2019-10-13T11:08:03.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:03.945+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.946073+0000) 2019-10-13T11:08:03.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:03.945+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.946525+0000) 2019-10-13T11:08:03.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:03.946+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.946833+0000) 2019-10-13T11:08:03.948 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:03.946+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.947088+0000) 2019-10-13T11:08:03.948 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:03.946+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.947366+0000) 2019-10-13T11:08:03.949 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:03.946+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.947234+0000) 2019-10-13T11:08:03.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:03.948+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:03.949370+0000) 2019-10-13T11:08:03.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:03.953+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202779 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:05.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:05.146+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:05.147062+0000) 2019-10-13T11:08:05.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:05.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:05.681502+0000) 2019-10-13T11:08:05.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:05.681+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:05.681501+0000) 2019-10-13T11:08:08.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:08.587+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.588079+0000) 2019-10-13T11:08:08.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:08.922+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.922617+0000) 2019-10-13T11:08:08.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:08.954+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.955283+0000) 2019-10-13T11:08:08.957 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:08.955+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.955818+0000) 2019-10-13T11:08:08.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:08.955+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.955800+0000) 2019-10-13T11:08:08.957 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:08.955+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.956076+0000) 2019-10-13T11:08:08.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:08.956+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.957453+0000) 2019-10-13T11:08:08.958 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:08.956+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.956412+0000) 2019-10-13T11:08:08.958 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:08.956+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.956940+0000) 2019-10-13T11:08:08.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:08.959+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.960374+0000) 2019-10-13T11:08:08.964 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:08.963+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:11.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:11.539+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:11.539843+0000) 2019-10-13T11:08:13.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:13.025+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.026572+0000) 2019-10-13T11:08:13.029 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:13.029+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.029304+0000) 2019-10-13T11:08:13.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:13.963+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.965194+0000) 2019-10-13T11:08:13.967 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:13.963+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.965385+0000) 2019-10-13T11:08:13.968 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:13.966+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.966914+0000) 2019-10-13T11:08:13.968 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:13.964+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.965751+0000) 2019-10-13T11:08:13.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:13.964+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.965876+0000) 2019-10-13T11:08:13.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:13.964+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.966083+0000) 2019-10-13T11:08:13.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:13.965+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.966436+0000) 2019-10-13T11:08:13.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:13.967+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:13.968410+0000) 2019-10-13T11:08:13.975 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:13.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202842 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:15.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:15.149+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:15.149391+0000) 2019-10-13T11:08:15.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:15.680+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:15.681685+0000) 2019-10-13T11:08:15.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:15.680+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:15.681640+0000) 2019-10-13T11:08:18.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:18.587+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.588231+0000) 2019-10-13T11:08:18.923 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:18.921+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.922773+0000) 2019-10-13T11:08:18.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:18.974+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.975641+0000) 2019-10-13T11:08:18.977 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:18.976+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.976910+0000) 2019-10-13T11:08:18.978 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:18.974+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.975957+0000) 2019-10-13T11:08:18.978 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:18.975+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.976517+0000) 2019-10-13T11:08:18.978 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:18.975+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.976506+0000) 2019-10-13T11:08:18.978 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:18.975+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.976637+0000) 2019-10-13T11:08:18.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:18.975+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.976879+0000) 2019-10-13T11:08:18.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:18.979+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.980039+0000) 2019-10-13T11:08:18.985 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:18.983+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:20.648 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:08:20.648 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:20.656 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:08:20.656 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:21.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:21.539+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:21.540018+0000) 2019-10-13T11:08:23.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:23.025+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.026761+0000) 2019-10-13T11:08:23.029 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:23.029+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.029497+0000) 2019-10-13T11:08:23.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:23.984+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.986161+0000) 2019-10-13T11:08:23.988 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:23.986+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.987434+0000) 2019-10-13T11:08:23.988 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:23.985+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.986524+0000) 2019-10-13T11:08:23.988 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:23.985+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.986892+0000) 2019-10-13T11:08:23.988 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:23.985+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.987054+0000) 2019-10-13T11:08:23.989 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:23.985+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.987189+0000) 2019-10-13T11:08:23.989 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:23.986+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.987388+0000) 2019-10-13T11:08:23.989 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:23.988+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:23.989138+0000) 2019-10-13T11:08:23.995 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:23.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:25.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:25.151+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:25.151866+0000) 2019-10-13T11:08:25.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:25.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:25.681855+0000) 2019-10-13T11:08:25.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:25.681+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:25.681856+0000) 2019-10-13T11:08:28.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:28.587+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.588416+0000) 2019-10-13T11:08:28.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:28.922+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.922927+0000) 2019-10-13T11:08:28.997 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:28.994+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.996288+0000) 2019-10-13T11:08:28.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:28.995+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.997193+0000) 2019-10-13T11:08:28.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:28.995+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.996736+0000) 2019-10-13T11:08:28.999 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:28.995+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.996936+0000) 2019-10-13T11:08:28.999 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:28.995+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.997147+0000) 2019-10-13T11:08:28.999 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:28.995+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.997219+0000) 2019-10-13T11:08:28.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:28.996+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.997486+0000) 2019-10-13T11:08:29.000 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:28.999+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:29.000846+0000) 2019-10-13T11:08:29.008 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:29.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202936 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:31.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:31.539+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:31.540212+0000) 2019-10-13T11:08:33.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:33.026+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:33.026927+0000) 2019-10-13T11:08:33.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:33.028+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:33.029724+0000) 2019-10-13T11:08:34.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:34.005+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.006403+0000) 2019-10-13T11:08:34.017 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:34.005+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.006715+0000) 2019-10-13T11:08:34.018 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:34.005+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.006877+0000) 2019-10-13T11:08:34.018 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:34.005+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.007093+0000) 2019-10-13T11:08:34.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:34.006+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.007983+0000) 2019-10-13T11:08:34.019 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:34.006+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.007458+0000) 2019-10-13T11:08:34.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:34.008+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.010529+0000) 2019-10-13T11:08:34.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:34.006+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.007701+0000) 2019-10-13T11:08:34.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:34.011+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202966 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:35.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:35.153+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:35.154230+0000) 2019-10-13T11:08:35.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:35.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:35.682048+0000) 2019-10-13T11:08:35.683 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:35.681+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:35.682051+0000) 2019-10-13T11:08:38.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:38.588+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:38.588602+0000) 2019-10-13T11:08:38.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:38.922+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:38.923071+0000) 2019-10-13T11:08:39.016 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:39.013+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.014844+0000) 2019-10-13T11:08:39.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:39.013+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.015241+0000) 2019-10-13T11:08:39.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:39.014+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.016057+0000) 2019-10-13T11:08:39.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:39.014+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.015605+0000) 2019-10-13T11:08:39.017 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:39.014+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.015642+0000) 2019-10-13T11:08:39.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:39.014+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.015792+0000) 2019-10-13T11:08:39.018 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:39.014+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.016360+0000) 2019-10-13T11:08:39.019 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:39.017+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:39.018986+0000) 2019-10-13T11:08:39.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:39.022+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 202999 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:41.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:41.540+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:41.540426+0000) 2019-10-13T11:08:43.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:43.026+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:43.027116+0000) 2019-10-13T11:08:43.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:43.029+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:43.029912+0000) 2019-10-13T11:08:44.026 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:44.024+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.025277+0000) 2019-10-13T11:08:44.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:44.025+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.025422+0000) 2019-10-13T11:08:44.027 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:44.025+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.025712+0000) 2019-10-13T11:08:44.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:44.025+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.025730+0000) 2019-10-13T11:08:44.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:44.026+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.026808+0000) 2019-10-13T11:08:44.028 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:44.026+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.026592+0000) 2019-10-13T11:08:44.028 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:44.026+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.026713+0000) 2019-10-13T11:08:44.029 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:44.028+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.029232+0000) 2019-10-13T11:08:44.035 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:44.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203027 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:45.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:45.155+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:45.156567+0000) 2019-10-13T11:08:45.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:45.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:45.682226+0000) 2019-10-13T11:08:45.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:45.681+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:45.682267+0000) 2019-10-13T11:08:48.589 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:48.588+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:48.588764+0000) 2019-10-13T11:08:48.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:48.923+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:48.923211+0000) 2019-10-13T11:08:49.036 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:49.035+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.035469+0000) 2019-10-13T11:08:49.037 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:49.035+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.035694+0000) 2019-10-13T11:08:49.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:49.036+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.036989+0000) 2019-10-13T11:08:49.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:49.035+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.036084+0000) 2019-10-13T11:08:49.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:49.036+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.036498+0000) 2019-10-13T11:08:49.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:49.036+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.036525+0000) 2019-10-13T11:08:49.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:49.036+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.036956+0000) 2019-10-13T11:08:49.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:49.038+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:49.039683+0000) 2019-10-13T11:08:49.043 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:49.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203060 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:50.752 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:08:50.753 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:50.757 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:08:50.757 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:51.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:51.540+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:51.540595+0000) 2019-10-13T11:08:53.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:53.027+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:53.027300+0000) 2019-10-13T11:08:53.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:53.029+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:53.030096+0000) 2019-10-13T11:08:54.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:54.044+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.045467+0000) 2019-10-13T11:08:54.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:54.044+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.044560+0000) 2019-10-13T11:08:54.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:54.044+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.044877+0000) 2019-10-13T11:08:54.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:54.044+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.045202+0000) 2019-10-13T11:08:54.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:54.044+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.045263+0000) 2019-10-13T11:08:54.047 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:54.044+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.045377+0000) 2019-10-13T11:08:54.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:54.045+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.045739+0000) 2019-10-13T11:08:54.048 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:54.046+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.047210+0000) 2019-10-13T11:08:54.054 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:54.052+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203090 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:08:55.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:55.158+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:55.158928+0000) 2019-10-13T11:08:55.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:55.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:55.682431+0000) 2019-10-13T11:08:55.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:55.682+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:55.682431+0000) 2019-10-13T11:08:58.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:58.587+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:58.588943+0000) 2019-10-13T11:08:58.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:58.922+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:58.923383+0000) 2019-10-13T11:08:59.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:08:59.053+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.054444+0000) 2019-10-13T11:08:59.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:08:59.054+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.055399+0000) 2019-10-13T11:08:59.056 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:08:59.053+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.055066+0000) 2019-10-13T11:08:59.057 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:08:59.054+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.055469+0000) 2019-10-13T11:08:59.057 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:08:59.054+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.055637+0000) 2019-10-13T11:08:59.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:08:59.054+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.055847+0000) 2019-10-13T11:08:59.057 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:08:59.054+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.055971+0000) 2019-10-13T11:08:59.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:08:59.056+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:59.057706+0000) 2019-10-13T11:08:59.064 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:08:59.062+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:01.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:01.539+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:01.540827+0000) 2019-10-13T11:09:03.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:03.026+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:03.027483+0000) 2019-10-13T11:09:03.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:03.029+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:03.030270+0000) 2019-10-13T11:09:04.066 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:04.063+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.065321+0000) 2019-10-13T11:09:04.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:04.064+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.065514+0000) 2019-10-13T11:09:04.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:04.065+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.066584+0000) 2019-10-13T11:09:04.067 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:04.064+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.065625+0000) 2019-10-13T11:09:04.067 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:04.064+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.066050+0000) 2019-10-13T11:09:04.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:04.065+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.066604+0000) 2019-10-13T11:09:04.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:04.065+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.066873+0000) 2019-10-13T11:09:04.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:04.069+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.070074+0000) 2019-10-13T11:09:04.074 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:04.072+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203151 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:05.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:05.161+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:05.161355+0000) 2019-10-13T11:09:05.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:05.681+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:05.682551+0000) 2019-10-13T11:09:05.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:05.681+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:05.682597+0000) 2019-10-13T11:09:08.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:08.588+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:08.589136+0000) 2019-10-13T11:09:08.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:08.922+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:08.923572+0000) 2019-10-13T11:09:09.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:09.073+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.075245+0000) 2019-10-13T11:09:09.076 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:09.074+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.075579+0000) 2019-10-13T11:09:09.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:09.075+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.076558+0000) 2019-10-13T11:09:09.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:09.074+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.075938+0000) 2019-10-13T11:09:09.078 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:09.074+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.076067+0000) 2019-10-13T11:09:09.078 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:09.074+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.076341+0000) 2019-10-13T11:09:09.078 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:09.075+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.076970+0000) 2019-10-13T11:09:09.079 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:09.078+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:09.079659+0000) 2019-10-13T11:09:09.085 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:09.082+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:11.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:11.540+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:11.541006+0000) 2019-10-13T11:09:13.028 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:13.026+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:13.027668+0000) 2019-10-13T11:09:13.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:13.030+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:13.030433+0000) 2019-10-13T11:09:14.087 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:14.084+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.086217+0000) 2019-10-13T11:09:14.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:14.086+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.087238+0000) 2019-10-13T11:09:14.088 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:14.085+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.086728+0000) 2019-10-13T11:09:14.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:14.085+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.086954+0000) 2019-10-13T11:09:14.089 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:14.085+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.087189+0000) 2019-10-13T11:09:14.089 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:14.085+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.087254+0000) 2019-10-13T11:09:14.089 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:14.086+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.087620+0000) 2019-10-13T11:09:14.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:14.088+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.088889+0000) 2019-10-13T11:09:14.095 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:14.093+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203215 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:15.164 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:15.162+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:15.163687+0000) 2019-10-13T11:09:15.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:15.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:15.682729+0000) 2019-10-13T11:09:15.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:15.682+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:15.682748+0000) 2019-10-13T11:09:18.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:18.588+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:18.589284+0000) 2019-10-13T11:09:18.924 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:18.923+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:18.923773+0000) 2019-10-13T11:09:19.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:19.094+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.096178+0000) 2019-10-13T11:09:19.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:19.094+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.096293+0000) 2019-10-13T11:09:19.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:19.095+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.097595+0000) 2019-10-13T11:09:19.098 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:19.095+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.096668+0000) 2019-10-13T11:09:19.099 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:19.095+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.097051+0000) 2019-10-13T11:09:19.099 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:19.095+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.097178+0000) 2019-10-13T11:09:19.099 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:19.095+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.097229+0000) 2019-10-13T11:09:19.099 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:19.097+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:19.099349+0000) 2019-10-13T11:09:19.105 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:19.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:20.918 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:09:20.918 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:20.922 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:09:20.922 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:21.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:21.540+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:21.541218+0000) 2019-10-13T11:09:23.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:23.027+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:23.027841+0000) 2019-10-13T11:09:23.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:23.028+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:23.030603+0000) 2019-10-13T11:09:24.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:24.104+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.106115+0000) 2019-10-13T11:09:24.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:24.104+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.106314+0000) 2019-10-13T11:09:24.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:24.105+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.107512+0000) 2019-10-13T11:09:24.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:24.105+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.106579+0000) 2019-10-13T11:09:24.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:24.105+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.107120+0000) 2019-10-13T11:09:24.109 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:24.105+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.107382+0000) 2019-10-13T11:09:24.109 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:24.106+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.107557+0000) 2019-10-13T11:09:24.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:24.108+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.110168+0000) 2019-10-13T11:09:24.116 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:24.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:25.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:25.165+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:25.166101+0000) 2019-10-13T11:09:25.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:25.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:25.682910+0000) 2019-10-13T11:09:25.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:25.682+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:25.682964+0000) 2019-10-13T11:09:28.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:28.589+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:28.589481+0000) 2019-10-13T11:09:28.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:28.923+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:28.923878+0000) 2019-10-13T11:09:29.117 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:29.115+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.116391+0000) 2019-10-13T11:09:29.118 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:29.116+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.116572+0000) 2019-10-13T11:09:29.118 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:29.116+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.116645+0000) 2019-10-13T11:09:29.119 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:29.116+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.118064+0000) 2019-10-13T11:09:29.119 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:29.117+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.117445+0000) 2019-10-13T11:09:29.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:29.117+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.117721+0000) 2019-10-13T11:09:29.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:29.117+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.117895+0000) 2019-10-13T11:09:29.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:29.118+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:29.120114+0000) 2019-10-13T11:09:29.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:29.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203308 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:31.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:31.541+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:31.541393+0000) 2019-10-13T11:09:33.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:33.027+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:33.027990+0000) 2019-10-13T11:09:33.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:33.030+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:33.030832+0000) 2019-10-13T11:09:34.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:34.123+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.124038+0000) 2019-10-13T11:09:34.125 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:34.123+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.124316+0000) 2019-10-13T11:09:34.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:34.124+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.125265+0000) 2019-10-13T11:09:34.126 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:34.123+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.124188+0000) 2019-10-13T11:09:34.126 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:34.124+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.124835+0000) 2019-10-13T11:09:34.127 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:34.124+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.125060+0000) 2019-10-13T11:09:34.127 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:34.124+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.125156+0000) 2019-10-13T11:09:34.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:34.127+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.128314+0000) 2019-10-13T11:09:34.132 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:34.130+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203339 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:35.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:35.167+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:35.168577+0000) 2019-10-13T11:09:35.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:35.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:35.683068+0000) 2019-10-13T11:09:35.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:35.682+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:35.683090+0000) 2019-10-13T11:09:38.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:38.589+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.589668+0000) 2019-10-13T11:09:38.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:38.923+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.924058+0000) 2019-10-13T11:09:39.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:39.132+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.132631+0000) 2019-10-13T11:09:39.134 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:39.132+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.132897+0000) 2019-10-13T11:09:39.134 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:39.132+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.133179+0000) 2019-10-13T11:09:39.135 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:39.133+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.134507+0000) 2019-10-13T11:09:39.135 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:39.133+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.133415+0000) 2019-10-13T11:09:39.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:39.133+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.133503+0000) 2019-10-13T11:09:39.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:39.133+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.134226+0000) 2019-10-13T11:09:39.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:39.135+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:39.136554+0000) 2019-10-13T11:09:39.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:39.140+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203371 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:41.542 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:41.541+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:41.541568+0000) 2019-10-13T11:09:43.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:43.028+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.028154+0000) 2019-10-13T11:09:43.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:43.030+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.031053+0000) 2019-10-13T11:09:44.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:44.141+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.143118+0000) 2019-10-13T11:09:44.144 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:44.141+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.143018+0000) 2019-10-13T11:09:44.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:44.141+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.143240+0000) 2019-10-13T11:09:44.145 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:44.142+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.143599+0000) 2019-10-13T11:09:44.145 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:44.143+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.144495+0000) 2019-10-13T11:09:44.146 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:44.142+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.143931+0000) 2019-10-13T11:09:44.146 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:44.143+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.144392+0000) 2019-10-13T11:09:44.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:44.146+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.146939+0000) 2019-10-13T11:09:44.152 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:44.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:45.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:45.170+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:45.170520+0000) 2019-10-13T11:09:45.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:45.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:45.683208+0000) 2019-10-13T11:09:45.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:45.682+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:45.683225+0000) 2019-10-13T11:09:48.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:48.588+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.589804+0000) 2019-10-13T11:09:48.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:48.923+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.924227+0000) 2019-10-13T11:09:49.155 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:49.151+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.153098+0000) 2019-10-13T11:09:49.155 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:49.151+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.153292+0000) 2019-10-13T11:09:49.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:49.152+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.153785+0000) 2019-10-13T11:09:49.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:49.152+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.153916+0000) 2019-10-13T11:09:49.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:49.152+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.154153+0000) 2019-10-13T11:09:49.157 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:49.154+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.155070+0000) 2019-10-13T11:09:49.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:49.153+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.154578+0000) 2019-10-13T11:09:49.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:49.156+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:49.157422+0000) 2019-10-13T11:09:49.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:49.160+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203432 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:51.039 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:09:51.039 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:51.044 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:09:51.044 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:51.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:51.540+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:51.541753+0000) 2019-10-13T11:09:53.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:53.027+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.028305+0000) 2019-10-13T11:09:53.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:53.030+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.031257+0000) 2019-10-13T11:09:54.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:54.163+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.164005+0000) 2019-10-13T11:09:54.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:54.162+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.163465+0000) 2019-10-13T11:09:54.165 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:54.162+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.163646+0000) 2019-10-13T11:09:54.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:54.162+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.164106+0000) 2019-10-13T11:09:54.166 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:54.162+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.164296+0000) 2019-10-13T11:09:54.166 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:54.163+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.164560+0000) 2019-10-13T11:09:54.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:54.163+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.164716+0000) 2019-10-13T11:09:54.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:54.166+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.167474+0000) 2019-10-13T11:09:54.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:54.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:09:55.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:55.172+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:55.172845+0000) 2019-10-13T11:09:55.684 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:55.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:55.683427+0000) 2019-10-13T11:09:55.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:55.682+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:55.683413+0000) 2019-10-13T11:09:58.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:58.589+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.589981+0000) 2019-10-13T11:09:58.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:58.923+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.924393+0000) 2019-10-13T11:09:59.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:09:59.172+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.173843+0000) 2019-10-13T11:09:59.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:09:59.172+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.174010+0000) 2019-10-13T11:09:59.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:09:59.174+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.174858+0000) 2019-10-13T11:09:59.176 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:09:59.173+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.174674+0000) 2019-10-13T11:09:59.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:09:59.173+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.175016+0000) 2019-10-13T11:09:59.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:09:59.173+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.174860+0000) 2019-10-13T11:09:59.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:09:59.173+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.174984+0000) 2019-10-13T11:09:59.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:09:59.177+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:59.177880+0000) 2019-10-13T11:09:59.183 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:09:59.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:01.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:01.541+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:01.541955+0000) 2019-10-13T11:10:03.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:03.027+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.028472+0000) 2019-10-13T11:10:03.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:03.031+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.031446+0000) 2019-10-13T11:10:04.184 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:04.183+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.184401+0000) 2019-10-13T11:10:04.185 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:04.182+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.184157+0000) 2019-10-13T11:10:04.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:04.183+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.184405+0000) 2019-10-13T11:10:04.185 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:04.183+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.184458+0000) 2019-10-13T11:10:04.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:04.183+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.184767+0000) 2019-10-13T11:10:04.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:04.183+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.184943+0000) 2019-10-13T11:10:04.186 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:04.183+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.184944+0000) 2019-10-13T11:10:04.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:04.185+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.186663+0000) 2019-10-13T11:10:04.193 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:04.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:05.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:05.175+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:05.175247+0000) 2019-10-13T11:10:05.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:05.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:05.683613+0000) 2019-10-13T11:10:05.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:05.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:05.683629+0000) 2019-10-13T11:10:08.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:08.589+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.590171+0000) 2019-10-13T11:10:08.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:08.924+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.924545+0000) 2019-10-13T11:10:09.195 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:09.192+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.194167+0000) 2019-10-13T11:10:09.195 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:09.192+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.194350+0000) 2019-10-13T11:10:09.196 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:09.193+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.194633+0000) 2019-10-13T11:10:09.196 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:09.193+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.195551+0000) 2019-10-13T11:10:09.197 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:09.193+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.194976+0000) 2019-10-13T11:10:09.197 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:09.193+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.195092+0000) 2019-10-13T11:10:09.197 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:09.193+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.195339+0000) 2019-10-13T11:10:09.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:09.196+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:09.198125+0000) 2019-10-13T11:10:09.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:09.201+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:11.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:11.541+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:11.542162+0000) 2019-10-13T11:10:13.029 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:13.028+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.028665+0000) 2019-10-13T11:10:13.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:13.030+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.031642+0000) 2019-10-13T11:10:14.206 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:14.204+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.204576+0000) 2019-10-13T11:10:14.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:14.204+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.204700+0000) 2019-10-13T11:10:14.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:14.204+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.205106+0000) 2019-10-13T11:10:14.206 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:14.204+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.205277+0000) 2019-10-13T11:10:14.207 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:14.203+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.205630+0000) 2019-10-13T11:10:14.207 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:14.205+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.207083+0000) 2019-10-13T11:10:14.208 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:14.205+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.205423+0000) 2019-10-13T11:10:14.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:14.205+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.205622+0000) 2019-10-13T11:10:14.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:14.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:15.177 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:15.176+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:15.177484+0000) 2019-10-13T11:10:15.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:15.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:15.683775+0000) 2019-10-13T11:10:15.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:15.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:15.683781+0000) 2019-10-13T11:10:18.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:18.590+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.590363+0000) 2019-10-13T11:10:18.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:18.924+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.924735+0000) 2019-10-13T11:10:19.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:19.213+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.213833+0000) 2019-10-13T11:10:19.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:19.213+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.214094+0000) 2019-10-13T11:10:19.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:19.213+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.214922+0000) 2019-10-13T11:10:19.216 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:19.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.214333+0000) 2019-10-13T11:10:19.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:19.214+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.214813+0000) 2019-10-13T11:10:19.216 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:19.214+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.214912+0000) 2019-10-13T11:10:19.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:19.214+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.215231+0000) 2019-10-13T11:10:19.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:19.216+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:19.218568+0000) 2019-10-13T11:10:19.223 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:19.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203622 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:21.120 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:10:21.120 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:21.125 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:10:21.125 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:21.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:21.542+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:21.542335+0000) 2019-10-13T11:10:23.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:23.028+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.028889+0000) 2019-10-13T11:10:23.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:23.031+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.031824+0000) 2019-10-13T11:10:24.225 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:24.223+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.223932+0000) 2019-10-13T11:10:24.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:24.224+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.224512+0000) 2019-10-13T11:10:24.226 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:24.224+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.224808+0000) 2019-10-13T11:10:24.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:24.224+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.224836+0000) 2019-10-13T11:10:24.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:24.225+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.225746+0000) 2019-10-13T11:10:24.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:24.224+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.225193+0000) 2019-10-13T11:10:24.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:24.224+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.224628+0000) 2019-10-13T11:10:24.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:24.226+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.227144+0000) 2019-10-13T11:10:24.233 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:24.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:25.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:25.179+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:25.179756+0000) 2019-10-13T11:10:25.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:25.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:25.683928+0000) 2019-10-13T11:10:25.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:25.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:25.683929+0000) 2019-10-13T11:10:28.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:28.590+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.590592+0000) 2019-10-13T11:10:28.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:28.924+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.924941+0000) 2019-10-13T11:10:29.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:29.232+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.234032+0000) 2019-10-13T11:10:29.236 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:29.234+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.235441+0000) 2019-10-13T11:10:29.236 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:29.232+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.234149+0000) 2019-10-13T11:10:29.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:29.232+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.234323+0000) 2019-10-13T11:10:29.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:29.233+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.234464+0000) 2019-10-13T11:10:29.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:29.233+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.234659+0000) 2019-10-13T11:10:29.237 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:29.233+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.235174+0000) 2019-10-13T11:10:29.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:29.237+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:29.238014+0000) 2019-10-13T11:10:29.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:29.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:31.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:31.541+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:31.542518+0000) 2019-10-13T11:10:33.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:33.028+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.029077+0000) 2019-10-13T11:10:33.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:33.031+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.032000+0000) 2019-10-13T11:10:34.245 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:34.242+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.244238+0000) 2019-10-13T11:10:34.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:34.243+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.244486+0000) 2019-10-13T11:10:34.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:34.243+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.244743+0000) 2019-10-13T11:10:34.246 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:34.243+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.244783+0000) 2019-10-13T11:10:34.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:34.245+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.246247+0000) 2019-10-13T11:10:34.247 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:34.243+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.245138+0000) 2019-10-13T11:10:34.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:34.244+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.245530+0000) 2019-10-13T11:10:34.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:34.247+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.247821+0000) 2019-10-13T11:10:34.253 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:34.251+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203717 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:35.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:35.181+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:35.182007+0000) 2019-10-13T11:10:35.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:35.682+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:35.684055+0000) 2019-10-13T11:10:35.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:35.682+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:35.684067+0000) 2019-10-13T11:10:38.591 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:38.589+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.590806+0000) 2019-10-13T11:10:38.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:38.924+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.925127+0000) 2019-10-13T11:10:39.255 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:39.252+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.253810+0000) 2019-10-13T11:10:39.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:39.252+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.254150+0000) 2019-10-13T11:10:39.256 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:39.252+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.254323+0000) 2019-10-13T11:10:39.256 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:39.253+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.254573+0000) 2019-10-13T11:10:39.256 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:39.253+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.254661+0000) 2019-10-13T11:10:39.256 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:39.253+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.254836+0000) 2019-10-13T11:10:39.257 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:39.255+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.255914+0000) 2019-10-13T11:10:39.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:39.256+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:39.257521+0000) 2019-10-13T11:10:39.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:39.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203748 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:41.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:41.541+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:41.542710+0000) 2019-10-13T11:10:43.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:43.028+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.029264+0000) 2019-10-13T11:10:43.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:43.031+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.032198+0000) 2019-10-13T11:10:44.264 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:44.261+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.263156+0000) 2019-10-13T11:10:44.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:44.261+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.263352+0000) 2019-10-13T11:10:44.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:44.264+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.264823+0000) 2019-10-13T11:10:44.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:44.262+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.263903+0000) 2019-10-13T11:10:44.265 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:44.262+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.264249+0000) 2019-10-13T11:10:44.266 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:44.263+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.264487+0000) 2019-10-13T11:10:44.266 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:44.263+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.264684+0000) 2019-10-13T11:10:44.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:44.266+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.267414+0000) 2019-10-13T11:10:44.272 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:44.270+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203779 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:45.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:45.183+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:45.184200+0000) 2019-10-13T11:10:45.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:45.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:45.684186+0000) 2019-10-13T11:10:45.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:45.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:45.684210+0000) 2019-10-13T11:10:48.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:48.590+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.590992+0000) 2019-10-13T11:10:48.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:48.924+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.925287+0000) 2019-10-13T11:10:49.274 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:49.272+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.273539+0000) 2019-10-13T11:10:49.275 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:49.274+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.274830+0000) 2019-10-13T11:10:49.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:49.272+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.273958+0000) 2019-10-13T11:10:49.276 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:49.272+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.274312+0000) 2019-10-13T11:10:49.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:49.273+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.274426+0000) 2019-10-13T11:10:49.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:49.273+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.274501+0000) 2019-10-13T11:10:49.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:49.273+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.274983+0000) 2019-10-13T11:10:49.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:49.276+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:49.277538+0000) 2019-10-13T11:10:49.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:49.280+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203812 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:51.276 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:10:51.276 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:51.280 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:10:51.280 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:51.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:51.542+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:51.542878+0000) 2019-10-13T11:10:53.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:53.028+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.029370+0000) 2019-10-13T11:10:53.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:53.032+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.032384+0000) 2019-10-13T11:10:54.284 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:54.283+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.284060+0000) 2019-10-13T11:10:54.285 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:54.282+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.283499+0000) 2019-10-13T11:10:54.285 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:54.282+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.283668+0000) 2019-10-13T11:10:54.285 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:54.282+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.283835+0000) 2019-10-13T11:10:54.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:54.283+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.284547+0000) 2019-10-13T11:10:54.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:54.283+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.284533+0000) 2019-10-13T11:10:54.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:54.283+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.284839+0000) 2019-10-13T11:10:54.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:54.286+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.287010+0000) 2019-10-13T11:10:54.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:54.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203843 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:10:55.186 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:55.186+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:55.186515+0000) 2019-10-13T11:10:55.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:55.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:55.684348+0000) 2019-10-13T11:10:55.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:55.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:55.684433+0000) 2019-10-13T11:10:58.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:58.590+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.591174+0000) 2019-10-13T11:10:58.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:58.925+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.925471+0000) 2019-10-13T11:10:59.294 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:10:59.293+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.293432+0000) 2019-10-13T11:10:59.295 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:10:59.293+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.294625+0000) 2019-10-13T11:10:59.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:10:59.293+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.294042+0000) 2019-10-13T11:10:59.296 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:10:59.293+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.294154+0000) 2019-10-13T11:10:59.296 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:10:59.293+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.294287+0000) 2019-10-13T11:10:59.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:10:59.294+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.294489+0000) 2019-10-13T11:10:59.296 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:10:59.294+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.294630+0000) 2019-10-13T11:10:59.297 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:10:59.295+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:59.296571+0000) 2019-10-13T11:10:59.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:10:59.301+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203874 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:01.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:01.542+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:01.543100+0000) 2019-10-13T11:11:03.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:03.029+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.029542+0000) 2019-10-13T11:11:03.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:03.031+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.032578+0000) 2019-10-13T11:11:04.305 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:04.303+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.303887+0000) 2019-10-13T11:11:04.306 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:04.303+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.304782+0000) 2019-10-13T11:11:04.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:04.303+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.304334+0000) 2019-10-13T11:11:04.306 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:04.304+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.304481+0000) 2019-10-13T11:11:04.306 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:04.304+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.304856+0000) 2019-10-13T11:11:04.306 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:04.304+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.305067+0000) 2019-10-13T11:11:04.307 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:04.304+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.304574+0000) 2019-10-13T11:11:04.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:04.305+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.307587+0000) 2019-10-13T11:11:04.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:04.312+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:05.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:05.187+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:05.188773+0000) 2019-10-13T11:11:05.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:05.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:05.684529+0000) 2019-10-13T11:11:05.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:05.684+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:05.684537+0000) 2019-10-13T11:11:08.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:08.591+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.591312+0000) 2019-10-13T11:11:08.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:08.925+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.925654+0000) 2019-10-13T11:11:09.315 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:09.313+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.315055+0000) 2019-10-13T11:11:09.316 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:09.314+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.314674+0000) 2019-10-13T11:11:09.317 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:09.314+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.314947+0000) 2019-10-13T11:11:09.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:09.314+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.314949+0000) 2019-10-13T11:11:09.317 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:09.314+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.315174+0000) 2019-10-13T11:11:09.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:09.314+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.315256+0000) 2019-10-13T11:11:09.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:09.314+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.315370+0000) 2019-10-13T11:11:09.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:09.316+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:09.318450+0000) 2019-10-13T11:11:09.323 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:09.322+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203938 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:11.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:11.543+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:11.543278+0000) 2019-10-13T11:11:13.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:13.029+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.029748+0000) 2019-10-13T11:11:13.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:13.031+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.032765+0000) 2019-10-13T11:11:14.325 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:14.323+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.324683+0000) 2019-10-13T11:11:14.325 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:14.323+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.324248+0000) 2019-10-13T11:11:14.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:14.324+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.324575+0000) 2019-10-13T11:11:14.326 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:14.324+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.324677+0000) 2019-10-13T11:11:14.326 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:14.324+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.324739+0000) 2019-10-13T11:11:14.327 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:14.324+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.325324+0000) 2019-10-13T11:11:14.327 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:14.325+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.325505+0000) 2019-10-13T11:11:14.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:14.327+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.327729+0000) 2019-10-13T11:11:14.334 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:14.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 203969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:15.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:15.191+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:15.191861+0000) 2019-10-13T11:11:15.685 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:15.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:15.684699+0000) 2019-10-13T11:11:15.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:15.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:15.684668+0000) 2019-10-13T11:11:18.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:18.590+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.591490+0000) 2019-10-13T11:11:18.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:18.924+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.925840+0000) 2019-10-13T11:11:19.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:19.334+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.335486+0000) 2019-10-13T11:11:19.337 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:19.334+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.335652+0000) 2019-10-13T11:11:19.337 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:19.334+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.335867+0000) 2019-10-13T11:11:19.338 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:19.336+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.337163+0000) 2019-10-13T11:11:19.338 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:19.334+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.336229+0000) 2019-10-13T11:11:19.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:19.335+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.336599+0000) 2019-10-13T11:11:19.338 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:19.335+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.336758+0000) 2019-10-13T11:11:19.339 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:19.338+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:19.338762+0000) 2019-10-13T11:11:19.344 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:19.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:21.440 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:11:21.440 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:21.445 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:11:21.445 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:21.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:21.542+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:21.543447+0000) 2019-10-13T11:11:23.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:23.028+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.029873+0000) 2019-10-13T11:11:23.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:23.032+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.032957+0000) 2019-10-13T11:11:24.347 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:24.344+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.346129+0000) 2019-10-13T11:11:24.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:24.345+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.346505+0000) 2019-10-13T11:11:24.348 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:24.345+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.346643+0000) 2019-10-13T11:11:24.348 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:24.345+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.347018+0000) 2019-10-13T11:11:24.348 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:24.345+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.347345+0000) 2019-10-13T11:11:24.349 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:24.347+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.348219+0000) 2019-10-13T11:11:24.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:24.346+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.347708+0000) 2019-10-13T11:11:24.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:24.349+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.350604+0000) 2019-10-13T11:11:24.354 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:24.352+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204030 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:25.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:25.193+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:25.194069+0000) 2019-10-13T11:11:25.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:25.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:25.684869+0000) 2019-10-13T11:11:25.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:25.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:25.684862+0000) 2019-10-13T11:11:28.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:28.590+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.591637+0000) 2019-10-13T11:11:28.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:28.925+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.926030+0000) 2019-10-13T11:11:29.357 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:29.353+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.355379+0000) 2019-10-13T11:11:29.357 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:29.354+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.355557+0000) 2019-10-13T11:11:29.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:29.354+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.355727+0000) 2019-10-13T11:11:29.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:29.354+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.356044+0000) 2019-10-13T11:11:29.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:29.356+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.356889+0000) 2019-10-13T11:11:29.358 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:29.354+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.356278+0000) 2019-10-13T11:11:29.358 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:29.355+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.356897+0000) 2019-10-13T11:11:29.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:29.359+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:29.359860+0000) 2019-10-13T11:11:29.365 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:29.362+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204063 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:31.544 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:31.542+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:31.543632+0000) 2019-10-13T11:11:33.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:33.029+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.030034+0000) 2019-10-13T11:11:33.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:33.032+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.033169+0000) 2019-10-13T11:11:34.367 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:34.364+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.366088+0000) 2019-10-13T11:11:34.367 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:34.364+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.366247+0000) 2019-10-13T11:11:34.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:34.366+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.367440+0000) 2019-10-13T11:11:34.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:34.365+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.366859+0000) 2019-10-13T11:11:34.369 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:34.365+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.366962+0000) 2019-10-13T11:11:34.369 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:34.365+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.367088+0000) 2019-10-13T11:11:34.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:34.365+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.367304+0000) 2019-10-13T11:11:34.370 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:34.368+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.369347+0000) 2019-10-13T11:11:34.375 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:34.373+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204094 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:35.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:35.196+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:35.196378+0000) 2019-10-13T11:11:35.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:35.683+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:35.685059+0000) 2019-10-13T11:11:35.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:35.683+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:35.685051+0000) 2019-10-13T11:11:38.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:38.591+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.591824+0000) 2019-10-13T11:11:38.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:38.925+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.926218+0000) 2019-10-13T11:11:39.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:39.376+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.377148+0000) 2019-10-13T11:11:39.378 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:39.375+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.376528+0000) 2019-10-13T11:11:39.378 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:39.375+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.376757+0000) 2019-10-13T11:11:39.378 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:39.375+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.376856+0000) 2019-10-13T11:11:39.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:39.375+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.376990+0000) 2019-10-13T11:11:39.379 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:39.375+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.377297+0000) 2019-10-13T11:11:39.379 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:39.376+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.378068+0000) 2019-10-13T11:11:39.380 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:39.380+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:39.380805+0000) 2019-10-13T11:11:39.386 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:39.383+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:41.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:41.543+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:41.543800+0000) 2019-10-13T11:11:43.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:43.029+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.030242+0000) 2019-10-13T11:11:43.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:43.033+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.033353+0000) 2019-10-13T11:11:44.388 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:44.386+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.386921+0000) 2019-10-13T11:11:44.388 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:44.386+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.387099+0000) 2019-10-13T11:11:44.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:44.386+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.387314+0000) 2019-10-13T11:11:44.389 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:44.387+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.388159+0000) 2019-10-13T11:11:44.389 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:44.387+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.387675+0000) 2019-10-13T11:11:44.389 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:44.387+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.388032+0000) 2019-10-13T11:11:44.390 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:44.387+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.388144+0000) 2019-10-13T11:11:44.390 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:44.389+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.390608+0000) 2019-10-13T11:11:44.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:44.394+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:45.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:45.198+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:45.198769+0000) 2019-10-13T11:11:45.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:45.684+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:45.685238+0000) 2019-10-13T11:11:45.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:45.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:45.685250+0000) 2019-10-13T11:11:48.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:48.591+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.592011+0000) 2019-10-13T11:11:48.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:48.926+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.926358+0000) 2019-10-13T11:11:49.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:49.396+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.396973+0000) 2019-10-13T11:11:49.398 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:49.396+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.397133+0000) 2019-10-13T11:11:49.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:49.397+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.398127+0000) 2019-10-13T11:11:49.399 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:49.397+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.397706+0000) 2019-10-13T11:11:49.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:49.397+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.397909+0000) 2019-10-13T11:11:49.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:49.397+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.398190+0000) 2019-10-13T11:11:49.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:49.397+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.398368+0000) 2019-10-13T11:11:49.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:49.400+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:49.401477+0000) 2019-10-13T11:11:49.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:49.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204189 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:51.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:51.543+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:51.544010+0000) 2019-10-13T11:11:51.655 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:11:51.655 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:51.660 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:11:51.660 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:53.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:53.030+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.030428+0000) 2019-10-13T11:11:53.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:53.032+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.033554+0000) 2019-10-13T11:11:54.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:54.406+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.408587+0000) 2019-10-13T11:11:54.409 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:54.407+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.408316+0000) 2019-10-13T11:11:54.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:54.407+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.408382+0000) 2019-10-13T11:11:54.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:54.408+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.408606+0000) 2019-10-13T11:11:54.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:54.408+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.408680+0000) 2019-10-13T11:11:54.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:54.408+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.408804+0000) 2019-10-13T11:11:54.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:54.409+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.410801+0000) 2019-10-13T11:11:54.411 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:54.408+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.409168+0000) 2019-10-13T11:11:54.417 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:54.416+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204220 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:11:55.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:55.200+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:55.201171+0000) 2019-10-13T11:11:55.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:55.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:55.685410+0000) 2019-10-13T11:11:55.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:55.685+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:55.685408+0000) 2019-10-13T11:11:58.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:58.591+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.592163+0000) 2019-10-13T11:11:58.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:58.926+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.926503+0000) 2019-10-13T11:11:59.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:11:59.417+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.418034+0000) 2019-10-13T11:11:59.419 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:11:59.417+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.418188+0000) 2019-10-13T11:11:59.420 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:11:59.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.419257+0000) 2019-10-13T11:11:59.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:11:59.418+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.418603+0000) 2019-10-13T11:11:59.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:11:59.418+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.418956+0000) 2019-10-13T11:11:59.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:11:59.418+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.418991+0000) 2019-10-13T11:11:59.421 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:11:59.418+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.419198+0000) 2019-10-13T11:11:59.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:11:59.419+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:59.421172+0000) 2019-10-13T11:11:59.426 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:11:59.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204251 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:01.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:01.543+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:01.544186+0000) 2019-10-13T11:12:03.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:03.029+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.030609+0000) 2019-10-13T11:12:03.033 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:03.032+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.033765+0000) 2019-10-13T11:12:04.427 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:04.425+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.427374+0000) 2019-10-13T11:12:04.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:04.426+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.427605+0000) 2019-10-13T11:12:04.428 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:04.425+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.427236+0000) 2019-10-13T11:12:04.429 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:04.426+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.428033+0000) 2019-10-13T11:12:04.429 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:04.426+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.428190+0000) 2019-10-13T11:12:04.430 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:04.427+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.428506+0000) 2019-10-13T11:12:04.430 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:04.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.428677+0000) 2019-10-13T11:12:04.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:04.429+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.429992+0000) 2019-10-13T11:12:04.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:04.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:05.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:05.202+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:05.203715+0000) 2019-10-13T11:12:05.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:05.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:05.685602+0000) 2019-10-13T11:12:05.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:05.684+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:05.685603+0000) 2019-10-13T11:12:08.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:08.590+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.592298+0000) 2019-10-13T11:12:08.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:08.925+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.926598+0000) 2019-10-13T11:12:09.437 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:09.436+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.437458+0000) 2019-10-13T11:12:09.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:09.435+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.436954+0000) 2019-10-13T11:12:09.439 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:09.435+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.437361+0000) 2019-10-13T11:12:09.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:09.436+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.437475+0000) 2019-10-13T11:12:09.439 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:09.436+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.437518+0000) 2019-10-13T11:12:09.439 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:09.436+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.437709+0000) 2019-10-13T11:12:09.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:09.436+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.437970+0000) 2019-10-13T11:12:09.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:09.440+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:09.440949+0000) 2019-10-13T11:12:09.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:09.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:11.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:11.543+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:11.544326+0000) 2019-10-13T11:12:13.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:13.030+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.030799+0000) 2019-10-13T11:12:13.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:13.033+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.033942+0000) 2019-10-13T11:12:14.447 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:14.444+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.446176+0000) 2019-10-13T11:12:14.447 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:14.445+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.446498+0000) 2019-10-13T11:12:14.448 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:14.445+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.446662+0000) 2019-10-13T11:12:14.448 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:14.447+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.448319+0000) 2019-10-13T11:12:14.449 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:14.445+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.447063+0000) 2019-10-13T11:12:14.449 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:14.445+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.447126+0000) 2019-10-13T11:12:14.449 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:14.445+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.447203+0000) 2019-10-13T11:12:14.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:14.449+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.450202+0000) 2019-10-13T11:12:14.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:14.453+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:15.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:15.205+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:15.205973+0000) 2019-10-13T11:12:15.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:15.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:15.685816+0000) 2019-10-13T11:12:15.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:15.685+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:15.685816+0000) 2019-10-13T11:12:18.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:18.591+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.592463+0000) 2019-10-13T11:12:18.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:18.926+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.926731+0000) 2019-10-13T11:12:19.458 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:19.455+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.456767+0000) 2019-10-13T11:12:19.458 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:19.455+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.456899+0000) 2019-10-13T11:12:19.458 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:19.455+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.457333+0000) 2019-10-13T11:12:19.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:19.457+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.457832+0000) 2019-10-13T11:12:19.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:19.456+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.457461+0000) 2019-10-13T11:12:19.459 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:19.456+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.457588+0000) 2019-10-13T11:12:19.460 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:19.456+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.457679+0000) 2019-10-13T11:12:19.460 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:19.459+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:19.460090+0000) 2019-10-13T11:12:19.466 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:19.463+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204375 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:21.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:21.543+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:21.544503+0000) 2019-10-13T11:12:21.861 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:12:21.861 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:21.866 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:12:21.866 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:23.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:23.030+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.030985+0000) 2019-10-13T11:12:23.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:23.033+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.034103+0000) 2019-10-13T11:12:24.467 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:24.465+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.466548+0000) 2019-10-13T11:12:24.468 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:24.465+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.466651+0000) 2019-10-13T11:12:24.468 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:24.465+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.466935+0000) 2019-10-13T11:12:24.469 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:24.467+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.468227+0000) 2019-10-13T11:12:24.469 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:24.465+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.467134+0000) 2019-10-13T11:12:24.469 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:24.465+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.467380+0000) 2019-10-13T11:12:24.469 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:24.466+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.467731+0000) 2019-10-13T11:12:24.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:24.469+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.469931+0000) 2019-10-13T11:12:24.474 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:24.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:25.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:25.207+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:25.208270+0000) 2019-10-13T11:12:25.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:25.684+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:25.685951+0000) 2019-10-13T11:12:25.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:25.684+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:25.685968+0000) 2019-10-13T11:12:28.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:28.592+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.592610+0000) 2019-10-13T11:12:28.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:28.926+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.926901+0000) 2019-10-13T11:12:29.476 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:29.474+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.474818+0000) 2019-10-13T11:12:29.476 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:29.474+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.475245+0000) 2019-10-13T11:12:29.476 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:29.475+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.475745+0000) 2019-10-13T11:12:29.477 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:29.475+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.475953+0000) 2019-10-13T11:12:29.477 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:29.475+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.476069+0000) 2019-10-13T11:12:29.478 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:29.475+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.476275+0000) 2019-10-13T11:12:29.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:29.476+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.476792+0000) 2019-10-13T11:12:29.478 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:29.477+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:29.478248+0000) 2019-10-13T11:12:29.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:29.483+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:31.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:31.544+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:31.544699+0000) 2019-10-13T11:12:33.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:33.030+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.031164+0000) 2019-10-13T11:12:33.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:33.034+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.034279+0000) 2019-10-13T11:12:34.486 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:34.485+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.485922+0000) 2019-10-13T11:12:34.486 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:34.484+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.485309+0000) 2019-10-13T11:12:34.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:34.485+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.485578+0000) 2019-10-13T11:12:34.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:34.485+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.486263+0000) 2019-10-13T11:12:34.487 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:34.486+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.486473+0000) 2019-10-13T11:12:34.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:34.486+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.486678+0000) 2019-10-13T11:12:34.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:34.486+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.487066+0000) 2019-10-13T11:12:34.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:34.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.488930+0000) 2019-10-13T11:12:34.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:34.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204469 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:35.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:35.209+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:35.210552+0000) 2019-10-13T11:12:35.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:35.685+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:35.686108+0000) 2019-10-13T11:12:35.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:35.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:35.686124+0000) 2019-10-13T11:12:38.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:38.592+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.592782+0000) 2019-10-13T11:12:38.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:38.926+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.927040+0000) 2019-10-13T11:12:39.497 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:39.495+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.495495+0000) 2019-10-13T11:12:39.497 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:39.495+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.495687+0000) 2019-10-13T11:12:39.497 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:39.495+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.495977+0000) 2019-10-13T11:12:39.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:39.495+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.496078+0000) 2019-10-13T11:12:39.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:39.495+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.496238+0000) 2019-10-13T11:12:39.498 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:39.496+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.497207+0000) 2019-10-13T11:12:39.498 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:39.496+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.496541+0000) 2019-10-13T11:12:39.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:39.498+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:39.499188+0000) 2019-10-13T11:12:39.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:39.501+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204499 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:41.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:41.544+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:41.544896+0000) 2019-10-13T11:12:43.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:43.030+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.031338+0000) 2019-10-13T11:12:43.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:43.033+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.034464+0000) 2019-10-13T11:12:44.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:44.502+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.503243+0000) 2019-10-13T11:12:44.505 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:44.503+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.503543+0000) 2019-10-13T11:12:44.505 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:44.503+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.503685+0000) 2019-10-13T11:12:44.506 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:44.503+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.503884+0000) 2019-10-13T11:12:44.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:44.503+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.503940+0000) 2019-10-13T11:12:44.506 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:44.503+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.505174+0000) 2019-10-13T11:12:44.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:44.504+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.504541+0000) 2019-10-13T11:12:44.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:44.505+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.506977+0000) 2019-10-13T11:12:44.512 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:44.511+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:45.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:45.212+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:45.213109+0000) 2019-10-13T11:12:45.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:45.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:45.686353+0000) 2019-10-13T11:12:45.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:45.685+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:45.686355+0000) 2019-10-13T11:12:48.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:48.591+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.592967+0000) 2019-10-13T11:12:48.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:48.925+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.927153+0000) 2019-10-13T11:12:49.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:49.512+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.514471+0000) 2019-10-13T11:12:49.516 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:49.512+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.513899+0000) 2019-10-13T11:12:49.516 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:49.512+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.514186+0000) 2019-10-13T11:12:49.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:49.512+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.514349+0000) 2019-10-13T11:12:49.516 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:49.513+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.514592+0000) 2019-10-13T11:12:49.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:49.513+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.514815+0000) 2019-10-13T11:12:49.517 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:49.513+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.514981+0000) 2019-10-13T11:12:49.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:49.514+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:49.516446+0000) 2019-10-13T11:12:49.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:49.520+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204562 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:51.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:51.544+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:51.545048+0000) 2019-10-13T11:12:51.985 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:12:51.985 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:51.990 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:12:51.990 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:53.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:53.030+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.031505+0000) 2019-10-13T11:12:53.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:53.033+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.034674+0000) 2019-10-13T11:12:54.524 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:54.521+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.523244+0000) 2019-10-13T11:12:54.524 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:54.522+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.523555+0000) 2019-10-13T11:12:54.525 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:54.523+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.524780+0000) 2019-10-13T11:12:54.526 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:54.522+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.523879+0000) 2019-10-13T11:12:54.526 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:54.522+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.523779+0000) 2019-10-13T11:12:54.526 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:54.522+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.524364+0000) 2019-10-13T11:12:54.526 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:54.523+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.524672+0000) 2019-10-13T11:12:54.527 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:54.525+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.526998+0000) 2019-10-13T11:12:54.532 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:54.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204593 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:12:55.216 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:55.214+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:55.215519+0000) 2019-10-13T11:12:55.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:55.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:55.686560+0000) 2019-10-13T11:12:55.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:55.685+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:55.686576+0000) 2019-10-13T11:12:58.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:58.592+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.593127+0000) 2019-10-13T11:12:58.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:58.925+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.927338+0000) 2019-10-13T11:12:59.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:12:59.531+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.533257+0000) 2019-10-13T11:12:59.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:12:59.532+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.533513+0000) 2019-10-13T11:12:59.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:12:59.534+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.534928+0000) 2019-10-13T11:12:59.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:12:59.532+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.533786+0000) 2019-10-13T11:12:59.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:12:59.532+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.533991+0000) 2019-10-13T11:12:59.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:12:59.533+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.534671+0000) 2019-10-13T11:12:59.536 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:12:59.533+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.534882+0000) 2019-10-13T11:12:59.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:12:59.535+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:59.536601+0000) 2019-10-13T11:12:59.542 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:12:59.539+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204623 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:01.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:01.544+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:01.545223+0000) 2019-10-13T11:13:03.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:03.031+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.031676+0000) 2019-10-13T11:13:03.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:03.034+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.034854+0000) 2019-10-13T11:13:04.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:04.541+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.543043+0000) 2019-10-13T11:13:04.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:04.543+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.544057+0000) 2019-10-13T11:13:04.545 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:04.542+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.543495+0000) 2019-10-13T11:13:04.545 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:04.542+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.543660+0000) 2019-10-13T11:13:04.545 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:04.542+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.543997+0000) 2019-10-13T11:13:04.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:04.542+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.544209+0000) 2019-10-13T11:13:04.546 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:04.543+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.544584+0000) 2019-10-13T11:13:04.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:04.545+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.546699+0000) 2019-10-13T11:13:04.553 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:04.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204654 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:05.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:05.217+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:05.217857+0000) 2019-10-13T11:13:05.687 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:05.685+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:05.686711+0000) 2019-10-13T11:13:05.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:05.685+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:05.686780+0000) 2019-10-13T11:13:08.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:08.591+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.593306+0000) 2019-10-13T11:13:08.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:08.926+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.927510+0000) 2019-10-13T11:13:09.554 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:09.552+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.553773+0000) 2019-10-13T11:13:09.555 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:09.553+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.554661+0000) 2019-10-13T11:13:09.556 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:09.552+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.554297+0000) 2019-10-13T11:13:09.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:09.553+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.554404+0000) 2019-10-13T11:13:09.556 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:09.553+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.554650+0000) 2019-10-13T11:13:09.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:09.553+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.554896+0000) 2019-10-13T11:13:09.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:09.553+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.555254+0000) 2019-10-13T11:13:09.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:09.556+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:09.557696+0000) 2019-10-13T11:13:09.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:09.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204686 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:11.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:11.544+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:11.545391+0000) 2019-10-13T11:13:13.032 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:13.031+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.031827+0000) 2019-10-13T11:13:13.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:13.034+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.035035+0000) 2019-10-13T11:13:14.564 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:14.563+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.563662+0000) 2019-10-13T11:13:14.565 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:14.564+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.564812+0000) 2019-10-13T11:13:14.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:14.563+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.564265+0000) 2019-10-13T11:13:14.566 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:14.564+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.564560+0000) 2019-10-13T11:13:14.566 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:14.564+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.564760+0000) 2019-10-13T11:13:14.566 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:14.564+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.565088+0000) 2019-10-13T11:13:14.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:14.564+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.565196+0000) 2019-10-13T11:13:14.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:14.565+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.566661+0000) 2019-10-13T11:13:14.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:14.572+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204717 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:15.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:15.219+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:15.220156+0000) 2019-10-13T11:13:15.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:15.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:15.686894+0000) 2019-10-13T11:13:15.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:15.686+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:15.686931+0000) 2019-10-13T11:13:18.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:18.593+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.593491+0000) 2019-10-13T11:13:18.928 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:18.927+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.927719+0000) 2019-10-13T11:13:19.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:19.574+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.574692+0000) 2019-10-13T11:13:19.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:19.574+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.574893+0000) 2019-10-13T11:13:19.576 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:19.574+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.574979+0000) 2019-10-13T11:13:19.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:19.574+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.575250+0000) 2019-10-13T11:13:19.577 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:19.575+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.575890+0000) 2019-10-13T11:13:19.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:19.575+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.575403+0000) 2019-10-13T11:13:19.577 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:19.575+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.575945+0000) 2019-10-13T11:13:19.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:19.578+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:19.578935+0000) 2019-10-13T11:13:19.583 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:19.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:21.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:21.545+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:21.545561+0000) 2019-10-13T11:13:22.139 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:13:22.139 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:22.143 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:13:22.143 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:23.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:23.031+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.031998+0000) 2019-10-13T11:13:23.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:23.034+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.035240+0000) 2019-10-13T11:13:24.586 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:24.584+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.585299+0000) 2019-10-13T11:13:24.586 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:24.585+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.585475+0000) 2019-10-13T11:13:24.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:24.585+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.585674+0000) 2019-10-13T11:13:24.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:24.585+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.585928+0000) 2019-10-13T11:13:24.587 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:24.585+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.586264+0000) 2019-10-13T11:13:24.588 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:24.586+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.586657+0000) 2019-10-13T11:13:24.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:24.586+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.587476+0000) 2019-10-13T11:13:24.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:24.588+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.589078+0000) 2019-10-13T11:13:24.594 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:24.592+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204778 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:25.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:25.221+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:25.222547+0000) 2019-10-13T11:13:25.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:25.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:25.687075+0000) 2019-10-13T11:13:25.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:25.686+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:25.687121+0000) 2019-10-13T11:13:28.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:28.593+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.593655+0000) 2019-10-13T11:13:28.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:28.927+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.927848+0000) 2019-10-13T11:13:29.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:29.594+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.594606+0000) 2019-10-13T11:13:29.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:29.594+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.594863+0000) 2019-10-13T11:13:29.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:29.595+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.596153+0000) 2019-10-13T11:13:29.597 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:29.594+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.595331+0000) 2019-10-13T11:13:29.597 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:29.595+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.595486+0000) 2019-10-13T11:13:29.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:29.595+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.595630+0000) 2019-10-13T11:13:29.597 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:29.595+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.595754+0000) 2019-10-13T11:13:29.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:29.598+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:29.599145+0000) 2019-10-13T11:13:29.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:29.601+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:31.546 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:31.545+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:31.545740+0000) 2019-10-13T11:13:33.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:33.031+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.032182+0000) 2019-10-13T11:13:33.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:33.035+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.035417+0000) 2019-10-13T11:13:34.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:34.602+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.603857+0000) 2019-10-13T11:13:34.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:34.602+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.603689+0000) 2019-10-13T11:13:34.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:34.602+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.604053+0000) 2019-10-13T11:13:34.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:34.602+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.603929+0000) 2019-10-13T11:13:34.605 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:34.602+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.604205+0000) 2019-10-13T11:13:34.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:34.603+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.604551+0000) 2019-10-13T11:13:34.606 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:34.603+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.604549+0000) 2019-10-13T11:13:34.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:34.604+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.606605+0000) 2019-10-13T11:13:34.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:34.610+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:35.226 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:35.224+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:35.225681+0000) 2019-10-13T11:13:35.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:35.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:35.687264+0000) 2019-10-13T11:13:35.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:35.686+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:35.687281+0000) 2019-10-13T11:13:38.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:38.592+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.593847+0000) 2019-10-13T11:13:38.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:38.927+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.927985+0000) 2019-10-13T11:13:39.615 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:39.612+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.614008+0000) 2019-10-13T11:13:39.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:39.612+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.614295+0000) 2019-10-13T11:13:39.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:39.613+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.614899+0000) 2019-10-13T11:13:39.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:39.613+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.614600+0000) 2019-10-13T11:13:39.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:39.613+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.614516+0000) 2019-10-13T11:13:39.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:39.613+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.614816+0000) 2019-10-13T11:13:39.617 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:39.613+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.615050+0000) 2019-10-13T11:13:39.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:39.615+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:39.617628+0000) 2019-10-13T11:13:39.623 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:39.620+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204871 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:41.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:41.545+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:41.545897+0000) 2019-10-13T11:13:43.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:43.031+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.032324+0000) 2019-10-13T11:13:43.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:43.034+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.035596+0000) 2019-10-13T11:13:44.625 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:44.622+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.623657+0000) 2019-10-13T11:13:44.625 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:44.622+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.623876+0000) 2019-10-13T11:13:44.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:44.623+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.625079+0000) 2019-10-13T11:13:44.626 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:44.623+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.624523+0000) 2019-10-13T11:13:44.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:44.623+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.624716+0000) 2019-10-13T11:13:44.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:44.623+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.624817+0000) 2019-10-13T11:13:44.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:44.624+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.625539+0000) 2019-10-13T11:13:44.628 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:44.626+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.628263+0000) 2019-10-13T11:13:44.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:44.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204902 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:45.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:45.227+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:45.227838+0000) 2019-10-13T11:13:45.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:45.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:45.687464+0000) 2019-10-13T11:13:45.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:45.686+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:45.687466+0000) 2019-10-13T11:13:48.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:48.593+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.594038+0000) 2019-10-13T11:13:48.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:48.927+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.928162+0000) 2019-10-13T11:13:49.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:49.633+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.634472+0000) 2019-10-13T11:13:49.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:49.633+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.634897+0000) 2019-10-13T11:13:49.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:49.633+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.635078+0000) 2019-10-13T11:13:49.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:49.633+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.635302+0000) 2019-10-13T11:13:49.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:49.634+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.635491+0000) 2019-10-13T11:13:49.637 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:49.634+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.635717+0000) 2019-10-13T11:13:49.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:49.635+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.636586+0000) 2019-10-13T11:13:49.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:49.636+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:49.637692+0000) 2019-10-13T11:13:49.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:49.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:51.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:51.545+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:51.546064+0000) 2019-10-13T11:13:52.345 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:13:52.346 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:52.350 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:13:52.350 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:53.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:53.031+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.032506+0000) 2019-10-13T11:13:53.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:53.035+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.035800+0000) 2019-10-13T11:13:54.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:54.643+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.644633+0000) 2019-10-13T11:13:54.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:54.643+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.644856+0000) 2019-10-13T11:13:54.646 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:54.643+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.645231+0000) 2019-10-13T11:13:54.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:54.643+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.645379+0000) 2019-10-13T11:13:54.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:54.644+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.645493+0000) 2019-10-13T11:13:54.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:54.644+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.645741+0000) 2019-10-13T11:13:54.647 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:54.646+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.646830+0000) 2019-10-13T11:13:54.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:54.647+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.648553+0000) 2019-10-13T11:13:54.653 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:54.650+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:13:55.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:55.229+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:55.230105+0000) 2019-10-13T11:13:55.688 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:55.687+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:55.687606+0000) 2019-10-13T11:13:55.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:55.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:55.687640+0000) 2019-10-13T11:13:58.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:58.593+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.594250+0000) 2019-10-13T11:13:58.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:58.927+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.928331+0000) 2019-10-13T11:13:59.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:13:59.653+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.653601+0000) 2019-10-13T11:13:59.656 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:13:59.653+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.653901+0000) 2019-10-13T11:13:59.656 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:13:59.653+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.654338+0000) 2019-10-13T11:13:59.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:13:59.653+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.654382+0000) 2019-10-13T11:13:59.656 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:13:59.654+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.654664+0000) 2019-10-13T11:13:59.656 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:13:59.654+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.654718+0000) 2019-10-13T11:13:59.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:13:59.655+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.655788+0000) 2019-10-13T11:13:59.657 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:13:59.657+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:59.657830+0000) 2019-10-13T11:13:59.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:13:59.660+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 204995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:01.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:01.545+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:01.546271+0000) 2019-10-13T11:14:03.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:03.032+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.032711+0000) 2019-10-13T11:14:03.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:03.035+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.036008+0000) 2019-10-13T11:14:04.664 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:04.662+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.662864+0000) 2019-10-13T11:14:04.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:04.662+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.663238+0000) 2019-10-13T11:14:04.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:04.662+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.663136+0000) 2019-10-13T11:14:04.665 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:04.663+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.663446+0000) 2019-10-13T11:14:04.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:04.662+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.663325+0000) 2019-10-13T11:14:04.665 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:04.663+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.663539+0000) 2019-10-13T11:14:04.666 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:04.664+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.664829+0000) 2019-10-13T11:14:04.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:04.666+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.666782+0000) 2019-10-13T11:14:04.671 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:04.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205026 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:05.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:05.232+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:05.232457+0000) 2019-10-13T11:14:05.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:05.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:05.687791+0000) 2019-10-13T11:14:05.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:05.687+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:05.687792+0000) 2019-10-13T11:14:08.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:08.594+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.594433+0000) 2019-10-13T11:14:08.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:08.928+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.928478+0000) 2019-10-13T11:14:09.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:09.672+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.672806+0000) 2019-10-13T11:14:09.675 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:09.672+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.673155+0000) 2019-10-13T11:14:09.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:09.672+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.673038+0000) 2019-10-13T11:14:09.675 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:09.672+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.673293+0000) 2019-10-13T11:14:09.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:09.673+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.673408+0000) 2019-10-13T11:14:09.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:09.673+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.673578+0000) 2019-10-13T11:14:09.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:09.673+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.674614+0000) 2019-10-13T11:14:09.677 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:09.676+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:09.676808+0000) 2019-10-13T11:14:09.682 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:09.681+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205058 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:11.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:11.546+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:11.546450+0000) 2019-10-13T11:14:13.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:13.032+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.032880+0000) 2019-10-13T11:14:13.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:13.035+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.036210+0000) 2019-10-13T11:14:14.684 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:14.683+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.683825+0000) 2019-10-13T11:14:14.685 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:14.682+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.683236+0000) 2019-10-13T11:14:14.685 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:14.683+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.683690+0000) 2019-10-13T11:14:14.685 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:14.683+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.683742+0000) 2019-10-13T11:14:14.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:14.683+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.683510+0000) 2019-10-13T11:14:14.686 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:14.684+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.684448+0000) 2019-10-13T11:14:14.686 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:14.684+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.684664+0000) 2019-10-13T11:14:14.687 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:14.686+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.686934+0000) 2019-10-13T11:14:14.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:14.691+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205089 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:15.235 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:15.234+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:15.234837+0000) 2019-10-13T11:14:15.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:15.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:15.687979+0000) 2019-10-13T11:14:15.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:15.687+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:15.687990+0000) 2019-10-13T11:14:18.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:18.593+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.594551+0000) 2019-10-13T11:14:18.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:18.927+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.928658+0000) 2019-10-13T11:14:19.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:19.692+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.693496+0000) 2019-10-13T11:14:19.695 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:19.692+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.693707+0000) 2019-10-13T11:14:19.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:19.694+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.695187+0000) 2019-10-13T11:14:19.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:19.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.694004+0000) 2019-10-13T11:14:19.696 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:19.692+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.694070+0000) 2019-10-13T11:14:19.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:19.693+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.694560+0000) 2019-10-13T11:14:19.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:19.693+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.694691+0000) 2019-10-13T11:14:19.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:19.695+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:19.696061+0000) 2019-10-13T11:14:19.702 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:19.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205119 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:21.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:21.545+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:21.546643+0000) 2019-10-13T11:14:22.481 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:14:22.481 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:22.485 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:14:22.485 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:23.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:23.032+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.033065+0000) 2019-10-13T11:14:23.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:23.035+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.036375+0000) 2019-10-13T11:14:24.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:24.702+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.703511+0000) 2019-10-13T11:14:24.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:24.702+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.703750+0000) 2019-10-13T11:14:24.705 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:24.702+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.703878+0000) 2019-10-13T11:14:24.706 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:24.703+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.705122+0000) 2019-10-13T11:14:24.706 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:24.702+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.704248+0000) 2019-10-13T11:14:24.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:24.703+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.704596+0000) 2019-10-13T11:14:24.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:24.703+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.704767+0000) 2019-10-13T11:14:24.708 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:24.706+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.707975+0000) 2019-10-13T11:14:24.713 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:24.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:25.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:25.236+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:25.237163+0000) 2019-10-13T11:14:25.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:25.686+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:25.688213+0000) 2019-10-13T11:14:25.689 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:25.686+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:25.688210+0000) 2019-10-13T11:14:28.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:28.593+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.594698+0000) 2019-10-13T11:14:28.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:28.928+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.928847+0000) 2019-10-13T11:14:29.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:29.712+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.713726+0000) 2019-10-13T11:14:29.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:29.712+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.713882+0000) 2019-10-13T11:14:29.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:29.712+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.714098+0000) 2019-10-13T11:14:29.716 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:29.713+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.715068+0000) 2019-10-13T11:14:29.716 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:29.713+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.714515+0000) 2019-10-13T11:14:29.716 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:29.713+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.714839+0000) 2019-10-13T11:14:29.716 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:29.713+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.714977+0000) 2019-10-13T11:14:29.718 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:29.716+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:29.717855+0000) 2019-10-13T11:14:29.723 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:29.720+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205182 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:31.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:31.546+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:31.546819+0000) 2019-10-13T11:14:33.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:33.032+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.033252+0000) 2019-10-13T11:14:33.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:33.035+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.036550+0000) 2019-10-13T11:14:34.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:34.721+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.723188+0000) 2019-10-13T11:14:34.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:34.722+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.723571+0000) 2019-10-13T11:14:34.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:34.722+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.723647+0000) 2019-10-13T11:14:34.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:34.723+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.724763+0000) 2019-10-13T11:14:34.726 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:34.722+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.724193+0000) 2019-10-13T11:14:34.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:34.722+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.724334+0000) 2019-10-13T11:14:34.726 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:34.722+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.723817+0000) 2019-10-13T11:14:34.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:34.725+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.726790+0000) 2019-10-13T11:14:34.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:34.729+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205213 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:35.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:35.238+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:35.239401+0000) 2019-10-13T11:14:35.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:35.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:35.688414+0000) 2019-10-13T11:14:35.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:35.687+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:35.688386+0000) 2019-10-13T11:14:38.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:38.594+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.594892+0000) 2019-10-13T11:14:38.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:38.928+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.929039+0000) 2019-10-13T11:14:39.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:39.730+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.732242+0000) 2019-10-13T11:14:39.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:39.730+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.732372+0000) 2019-10-13T11:14:39.734 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:39.731+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.732591+0000) 2019-10-13T11:14:39.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:39.733+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.734329+0000) 2019-10-13T11:14:39.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:39.731+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.732999+0000) 2019-10-13T11:14:39.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:39.731+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.733238+0000) 2019-10-13T11:14:39.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:39.732+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.733831+0000) 2019-10-13T11:14:39.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:39.734+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:39.735414+0000) 2019-10-13T11:14:39.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:39.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205243 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:41.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:41.546+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:41.547031+0000) 2019-10-13T11:14:43.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:43.032+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.033462+0000) 2019-10-13T11:14:43.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:43.036+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.036726+0000) 2019-10-13T11:14:44.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:44.741+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.742696+0000) 2019-10-13T11:14:44.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:44.741+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.743067+0000) 2019-10-13T11:14:44.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:44.741+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.743204+0000) 2019-10-13T11:14:44.745 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:44.743+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.744631+0000) 2019-10-13T11:14:44.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:44.742+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.743601+0000) 2019-10-13T11:14:44.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:44.742+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.743772+0000) 2019-10-13T11:14:44.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:44.742+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.744241+0000) 2019-10-13T11:14:44.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:44.745+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.746431+0000) 2019-10-13T11:14:44.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:44.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205274 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:45.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:45.240+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:45.241505+0000) 2019-10-13T11:14:45.689 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:45.688+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:45.688539+0000) 2019-10-13T11:14:45.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:45.688+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:45.688546+0000) 2019-10-13T11:14:48.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:48.594+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.595073+0000) 2019-10-13T11:14:48.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:48.928+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.929242+0000) 2019-10-13T11:14:49.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:49.751+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.752276+0000) 2019-10-13T11:14:49.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:49.752+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.753384+0000) 2019-10-13T11:14:49.754 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:49.752+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.753040+0000) 2019-10-13T11:14:49.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:49.752+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.753247+0000) 2019-10-13T11:14:49.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:49.753+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.753577+0000) 2019-10-13T11:14:49.755 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:49.754+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.755371+0000) 2019-10-13T11:14:49.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:49.753+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.753754+0000) 2019-10-13T11:14:49.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:49.753+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:49.753930+0000) 2019-10-13T11:14:49.762 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:49.760+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:51.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:51.546+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:51.547140+0000) 2019-10-13T11:14:52.660 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:14:52.660 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:52.665 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:14:52.665 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:53.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:53.033+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.033624+0000) 2019-10-13T11:14:53.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:53.036+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.036930+0000) 2019-10-13T11:14:54.764 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:54.762+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.762979+0000) 2019-10-13T11:14:54.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:54.763+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.763431+0000) 2019-10-13T11:14:54.765 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:54.763+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.763649+0000) 2019-10-13T11:14:54.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:54.763+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.764290+0000) 2019-10-13T11:14:54.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:54.764+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.764435+0000) 2019-10-13T11:14:54.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:54.764+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.764925+0000) 2019-10-13T11:14:54.766 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:54.764+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.764755+0000) 2019-10-13T11:14:54.767 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:54.766+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.767275+0000) 2019-10-13T11:14:54.773 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:54.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:14:55.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:55.243+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:55.243821+0000) 2019-10-13T11:14:55.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:55.688+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:55.688654+0000) 2019-10-13T11:14:55.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:55.688+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:55.688694+0000) 2019-10-13T11:14:58.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:58.595+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.595260+0000) 2019-10-13T11:14:58.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:58.929+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.929478+0000) 2019-10-13T11:14:59.775 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:14:59.773+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.774173+0000) 2019-10-13T11:14:59.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:14:59.773+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.774351+0000) 2019-10-13T11:14:59.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:14:59.774+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.775279+0000) 2019-10-13T11:14:59.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:14:59.774+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.774692+0000) 2019-10-13T11:14:59.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:14:59.774+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.774954+0000) 2019-10-13T11:14:59.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:14:59.774+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.774829+0000) 2019-10-13T11:14:59.777 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:14:59.774+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.775284+0000) 2019-10-13T11:14:59.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:14:59.776+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:59.776923+0000) 2019-10-13T11:14:59.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:14:59.781+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:01.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:01.546+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:01.547295+0000) 2019-10-13T11:15:03.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:03.033+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.033806+0000) 2019-10-13T11:15:03.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:03.036+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.037156+0000) 2019-10-13T11:15:04.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:04.782+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.784145+0000) 2019-10-13T11:15:04.786 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:04.783+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.784543+0000) 2019-10-13T11:15:04.786 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:04.783+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.784637+0000) 2019-10-13T11:15:04.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:04.783+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.784788+0000) 2019-10-13T11:15:04.786 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:04.783+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.785171+0000) 2019-10-13T11:15:04.787 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:04.785+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.786075+0000) 2019-10-13T11:15:04.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:04.786+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.787260+0000) 2019-10-13T11:15:04.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:04.784+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.785562+0000) 2019-10-13T11:15:04.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:04.791+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:05.247 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:05.246+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:05.247030+0000) 2019-10-13T11:15:05.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:05.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:05.688870+0000) 2019-10-13T11:15:05.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:05.687+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:05.688890+0000) 2019-10-13T11:15:08.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:08.594+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.595433+0000) 2019-10-13T11:15:08.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:08.928+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.929640+0000) 2019-10-13T11:15:09.795 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:09.792+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.794150+0000) 2019-10-13T11:15:09.795 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:09.792+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.794310+0000) 2019-10-13T11:15:09.796 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:09.794+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.795461+0000) 2019-10-13T11:15:09.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:09.793+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.795359+0000) 2019-10-13T11:15:09.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:09.795+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.796458+0000) 2019-10-13T11:15:09.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:09.794+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.795882+0000) 2019-10-13T11:15:09.798 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:09.794+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.796048+0000) 2019-10-13T11:15:09.799 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:09.797+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:09.798678+0000) 2019-10-13T11:15:09.803 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:09.801+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:11.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:11.546+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:11.547471+0000) 2019-10-13T11:15:13.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:13.032+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.034010+0000) 2019-10-13T11:15:13.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:13.037+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.037317+0000) 2019-10-13T11:15:14.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:14.803+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.804482+0000) 2019-10-13T11:15:14.806 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:14.803+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.804595+0000) 2019-10-13T11:15:14.806 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:14.803+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.804996+0000) 2019-10-13T11:15:14.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:14.804+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.805393+0000) 2019-10-13T11:15:14.807 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:14.804+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.805465+0000) 2019-10-13T11:15:14.807 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:14.805+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.805868+0000) 2019-10-13T11:15:14.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:14.804+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.805939+0000) 2019-10-13T11:15:14.808 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:14.807+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.808562+0000) 2019-10-13T11:15:14.814 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:14.811+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:15.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:15.248+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:15.249009+0000) 2019-10-13T11:15:15.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:15.687+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:15.689034+0000) 2019-10-13T11:15:15.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:15.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:15.689033+0000) 2019-10-13T11:15:18.596 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:18.594+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.595608+0000) 2019-10-13T11:15:18.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:18.929+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.929841+0000) 2019-10-13T11:15:19.815 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:19.812+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.814216+0000) 2019-10-13T11:15:19.815 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:19.812+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.814380+0000) 2019-10-13T11:15:19.816 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:19.813+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.814599+0000) 2019-10-13T11:15:19.817 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:19.814+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.815856+0000) 2019-10-13T11:15:19.817 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:19.813+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.814822+0000) 2019-10-13T11:15:19.817 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:19.813+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.814952+0000) 2019-10-13T11:15:19.817 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:19.814+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.815501+0000) 2019-10-13T11:15:19.818 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:19.816+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:19.818261+0000) 2019-10-13T11:15:19.822 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:19.819+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:21.548 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:21.546+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:21.547655+0000) 2019-10-13T11:15:22.789 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:15:22.790 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:22.794 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:15:22.794 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:23.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:23.033+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.034260+0000) 2019-10-13T11:15:23.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:23.036+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.037504+0000) 2019-10-13T11:15:24.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:24.821+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.823227+0000) 2019-10-13T11:15:24.825 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:24.822+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.823443+0000) 2019-10-13T11:15:24.825 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:24.822+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.823514+0000) 2019-10-13T11:15:24.825 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:24.822+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.823801+0000) 2019-10-13T11:15:24.826 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:24.822+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.824468+0000) 2019-10-13T11:15:24.826 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:24.822+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.824020+0000) 2019-10-13T11:15:24.826 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:24.822+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.824372+0000) 2019-10-13T11:15:24.827 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:24.825+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.827018+0000) 2019-10-13T11:15:24.833 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:24.830+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:25.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:25.251+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:25.252139+0000) 2019-10-13T11:15:25.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:25.687+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:25.689145+0000) 2019-10-13T11:15:25.690 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:25.687+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:25.689148+0000) 2019-10-13T11:15:28.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:28.595+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.595803+0000) 2019-10-13T11:15:28.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:28.929+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.929933+0000) 2019-10-13T11:15:29.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:29.832+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.833694+0000) 2019-10-13T11:15:29.835 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:29.832+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.834065+0000) 2019-10-13T11:15:29.836 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:29.832+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.834335+0000) 2019-10-13T11:15:29.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:29.833+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.834430+0000) 2019-10-13T11:15:29.836 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:29.833+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.834941+0000) 2019-10-13T11:15:29.837 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:29.834+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.835311+0000) 2019-10-13T11:15:29.837 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:29.833+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.835147+0000) 2019-10-13T11:15:29.837 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:29.836+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:29.837661+0000) 2019-10-13T11:15:29.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:29.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:31.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:31.547+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:31.547866+0000) 2019-10-13T11:15:33.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:33.034+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.034460+0000) 2019-10-13T11:15:33.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:33.036+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.037699+0000) 2019-10-13T11:15:34.845 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:34.843+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.844137+0000) 2019-10-13T11:15:34.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:34.844+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.845062+0000) 2019-10-13T11:15:34.846 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:34.844+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.844802+0000) 2019-10-13T11:15:34.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:34.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.844868+0000) 2019-10-13T11:15:34.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:34.844+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.845179+0000) 2019-10-13T11:15:34.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:34.844+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.845346+0000) 2019-10-13T11:15:34.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:34.845+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.845455+0000) 2019-10-13T11:15:34.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:34.846+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.847464+0000) 2019-10-13T11:15:34.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:34.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205585 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:35.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:35.253+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:35.254231+0000) 2019-10-13T11:15:35.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:35.688+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:35.689262+0000) 2019-10-13T11:15:35.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:35.688+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:35.689273+0000) 2019-10-13T11:15:38.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:38.595+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.595974+0000) 2019-10-13T11:15:38.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:38.929+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.930049+0000) 2019-10-13T11:15:39.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:39.853+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.853946+0000) 2019-10-13T11:15:39.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:39.853+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.854131+0000) 2019-10-13T11:15:39.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:39.854+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.854520+0000) 2019-10-13T11:15:39.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:39.854+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.854849+0000) 2019-10-13T11:15:39.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:39.854+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.854912+0000) 2019-10-13T11:15:39.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:39.854+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.855045+0000) 2019-10-13T11:15:39.857 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:39.855+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.855975+0000) 2019-10-13T11:15:39.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:39.857+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:39.858401+0000) 2019-10-13T11:15:39.862 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:39.860+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:41.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:41.547+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:41.548038+0000) 2019-10-13T11:15:43.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:43.034+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.034668+0000) 2019-10-13T11:15:43.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:43.037+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.037849+0000) 2019-10-13T11:15:44.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:44.862+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.862404+0000) 2019-10-13T11:15:44.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:44.862+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.862500+0000) 2019-10-13T11:15:44.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:44.862+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.862711+0000) 2019-10-13T11:15:44.864 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:44.862+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.862870+0000) 2019-10-13T11:15:44.865 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:44.862+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.863482+0000) 2019-10-13T11:15:44.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:44.862+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.863114+0000) 2019-10-13T11:15:44.865 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:44.863+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.863506+0000) 2019-10-13T11:15:44.866 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:44.865+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.865898+0000) 2019-10-13T11:15:44.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:44.868+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:45.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:45.255+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:45.256545+0000) 2019-10-13T11:15:45.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:45.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:45.689432+0000) 2019-10-13T11:15:45.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:45.689+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:45.689429+0000) 2019-10-13T11:15:48.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:48.596+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.596162+0000) 2019-10-13T11:15:48.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:48.930+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.930224+0000) 2019-10-13T11:15:49.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:49.869+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.870411+0000) 2019-10-13T11:15:49.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:49.869+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.870671+0000) 2019-10-13T11:15:49.872 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:49.869+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.870747+0000) 2019-10-13T11:15:49.873 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:49.870+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.871725+0000) 2019-10-13T11:15:49.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:49.869+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.871194+0000) 2019-10-13T11:15:49.873 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:49.870+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.871440+0000) 2019-10-13T11:15:49.873 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:49.870+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.871581+0000) 2019-10-13T11:15:49.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:49.874+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:49.874905+0000) 2019-10-13T11:15:49.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:49.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:51.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:51.547+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:51.548219+0000) 2019-10-13T11:15:52.944 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:15:52.944 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:52.949 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:15:52.949 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:53.035 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:53.033+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.034822+0000) 2019-10-13T11:15:53.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:53.037+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.038028+0000) 2019-10-13T11:15:54.882 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:54.880+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.881637+0000) 2019-10-13T11:15:54.883 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:54.880+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.881843+0000) 2019-10-13T11:15:54.883 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:54.880+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.882146+0000) 2019-10-13T11:15:54.884 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:54.880+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.882282+0000) 2019-10-13T11:15:54.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:54.881+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.882502+0000) 2019-10-13T11:15:54.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:54.881+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.882467+0000) 2019-10-13T11:15:54.885 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:54.882+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.883550+0000) 2019-10-13T11:15:54.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:54.883+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.884419+0000) 2019-10-13T11:15:54.890 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:54.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205709 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:15:55.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:55.258+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:55.258803+0000) 2019-10-13T11:15:55.690 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:55.688+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:55.689614+0000) 2019-10-13T11:15:55.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:55.688+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:55.689605+0000) 2019-10-13T11:15:58.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:58.595+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.596418+0000) 2019-10-13T11:15:58.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:58.929+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.930427+0000) 2019-10-13T11:15:59.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:15:59.891+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.892391+0000) 2019-10-13T11:15:59.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:15:59.890+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.891866+0000) 2019-10-13T11:15:59.893 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:15:59.890+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.892109+0000) 2019-10-13T11:15:59.894 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:15:59.890+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.892261+0000) 2019-10-13T11:15:59.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:15:59.891+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.892621+0000) 2019-10-13T11:15:59.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:15:59.891+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.892713+0000) 2019-10-13T11:15:59.894 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:15:59.891+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.892935+0000) 2019-10-13T11:15:59.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:15:59.894+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:59.894989+0000) 2019-10-13T11:15:59.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:15:59.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205739 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:01.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:01.547+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:01.548411+0000) 2019-10-13T11:16:03.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:03.033+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.035040+0000) 2019-10-13T11:16:03.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:03.038+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.038222+0000) 2019-10-13T11:16:04.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:04.901+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.902260+0000) 2019-10-13T11:16:04.903 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:04.900+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.901808+0000) 2019-10-13T11:16:04.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:04.900+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.901966+0000) 2019-10-13T11:16:04.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:04.900+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.902091+0000) 2019-10-13T11:16:04.904 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:04.901+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.902502+0000) 2019-10-13T11:16:04.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:04.901+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.902942+0000) 2019-10-13T11:16:04.904 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:04.901+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.903085+0000) 2019-10-13T11:16:04.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:04.904+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.905641+0000) 2019-10-13T11:16:04.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:04.908+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205770 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:05.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:05.261+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:05.261088+0000) 2019-10-13T11:16:05.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:05.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:05.689761+0000) 2019-10-13T11:16:05.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:05.689+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:05.689761+0000) 2019-10-13T11:16:08.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:08.595+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.596602+0000) 2019-10-13T11:16:08.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:08.929+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.930612+0000) 2019-10-13T11:16:09.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:09.910+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.912681+0000) 2019-10-13T11:16:09.914 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:09.910+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.912116+0000) 2019-10-13T11:16:09.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:09.910+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.912335+0000) 2019-10-13T11:16:09.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:09.911+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.912711+0000) 2019-10-13T11:16:09.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:09.911+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.912878+0000) 2019-10-13T11:16:09.914 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:09.911+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.913048+0000) 2019-10-13T11:16:09.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:09.911+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.913267+0000) 2019-10-13T11:16:09.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:09.913+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:09.915278+0000) 2019-10-13T11:16:09.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:09.918+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:11.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:11.548+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:11.548604+0000) 2019-10-13T11:16:13.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:13.034+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.035224+0000) 2019-10-13T11:16:13.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:13.037+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.038424+0000) 2019-10-13T11:16:14.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:14.920+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.922621+0000) 2019-10-13T11:16:14.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:14.920+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.922253+0000) 2019-10-13T11:16:14.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:14.921+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.922442+0000) 2019-10-13T11:16:14.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:14.921+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.922479+0000) 2019-10-13T11:16:14.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:14.921+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.922645+0000) 2019-10-13T11:16:14.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:14.921+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.922905+0000) 2019-10-13T11:16:14.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:14.921+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.923079+0000) 2019-10-13T11:16:14.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:14.923+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.925105+0000) 2019-10-13T11:16:14.931 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:14.928+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:15.263 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:15.262+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:15.263531+0000) 2019-10-13T11:16:15.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:15.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:15.689913+0000) 2019-10-13T11:16:15.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:15.689+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:15.689868+0000) 2019-10-13T11:16:18.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:18.596+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.596772+0000) 2019-10-13T11:16:18.932 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:18.930+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.930823+0000) 2019-10-13T11:16:19.933 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:19.931+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.931957+0000) 2019-10-13T11:16:19.933 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:19.931+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.932322+0000) 2019-10-13T11:16:19.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:19.931+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.932877+0000) 2019-10-13T11:16:19.934 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:19.932+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.932756+0000) 2019-10-13T11:16:19.934 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:19.932+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.932919+0000) 2019-10-13T11:16:19.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:19.932+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.932652+0000) 2019-10-13T11:16:19.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:19.932+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.933368+0000) 2019-10-13T11:16:19.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:19.935+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:19.935967+0000) 2019-10-13T11:16:19.942 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:19.940+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:21.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:21.548+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:21.548788+0000) 2019-10-13T11:16:23.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:23.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.035420+0000) 2019-10-13T11:16:23.038 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:23.037+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.038581+0000) 2019-10-13T11:16:23.098 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:16:23.098 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:23.107 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:16:23.107 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:24.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:24.942+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.942765+0000) 2019-10-13T11:16:24.944 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:24.941+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.942165+0000) 2019-10-13T11:16:24.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:24.942+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.942482+0000) 2019-10-13T11:16:24.945 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:24.942+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.942666+0000) 2019-10-13T11:16:24.945 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:24.942+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.942788+0000) 2019-10-13T11:16:24.946 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:24.942+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.943025+0000) 2019-10-13T11:16:24.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:24.942+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.943229+0000) 2019-10-13T11:16:24.946 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:24.945+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.945842+0000) 2019-10-13T11:16:24.950 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:24.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205894 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:25.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:25.265+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:25.265999+0000) 2019-10-13T11:16:25.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:25.689+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:25.690027+0000) 2019-10-13T11:16:25.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:25.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:25.690048+0000) 2019-10-13T11:16:28.597 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:28.596+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.596911+0000) 2019-10-13T11:16:28.932 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:28.930+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.930991+0000) 2019-10-13T11:16:29.952 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:29.950+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.951088+0000) 2019-10-13T11:16:29.952 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:29.950+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.951204+0000) 2019-10-13T11:16:29.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:29.951+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.951465+0000) 2019-10-13T11:16:29.953 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:29.951+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.952146+0000) 2019-10-13T11:16:29.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:29.951+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.951921+0000) 2019-10-13T11:16:29.954 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:29.951+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.952020+0000) 2019-10-13T11:16:29.954 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:29.951+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.952138+0000) 2019-10-13T11:16:29.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:29.953+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:29.954469+0000) 2019-10-13T11:16:29.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:29.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:31.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:31.548+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:31.549003+0000) 2019-10-13T11:16:33.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:33.038+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.038769+0000) 2019-10-13T11:16:33.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:33.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.035593+0000) 2019-10-13T11:16:34.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:34.959+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.961038+0000) 2019-10-13T11:16:34.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:34.959+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.961234+0000) 2019-10-13T11:16:34.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:34.959+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.961340+0000) 2019-10-13T11:16:34.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:34.962+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.962741+0000) 2019-10-13T11:16:34.963 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:34.960+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.961742+0000) 2019-10-13T11:16:34.964 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:34.960+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.961969+0000) 2019-10-13T11:16:34.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:34.960+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.962217+0000) 2019-10-13T11:16:34.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:34.964+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.965051+0000) 2019-10-13T11:16:34.969 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:34.967+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205955 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:35.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:35.267+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:35.267992+0000) 2019-10-13T11:16:35.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:35.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:35.690210+0000) 2019-10-13T11:16:35.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:35.689+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:35.690208+0000) 2019-10-13T11:16:38.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:38.596+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.597067+0000) 2019-10-13T11:16:38.932 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:38.930+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.931175+0000) 2019-10-13T11:16:39.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:39.968+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.970103+0000) 2019-10-13T11:16:39.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:39.968+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.970272+0000) 2019-10-13T11:16:39.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:39.969+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.970467+0000) 2019-10-13T11:16:39.972 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:39.971+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.971980+0000) 2019-10-13T11:16:39.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:39.969+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.970770+0000) 2019-10-13T11:16:39.973 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:39.969+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.971166+0000) 2019-10-13T11:16:39.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:39.970+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.971489+0000) 2019-10-13T11:16:39.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:39.972+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:39.973593+0000) 2019-10-13T11:16:39.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:39.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 205983 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:41.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:41.548+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:41.549226+0000) 2019-10-13T11:16:43.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:43.034+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.035758+0000) 2019-10-13T11:16:43.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:43.038+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.038954+0000) 2019-10-13T11:16:44.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:44.980+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.981141+0000) 2019-10-13T11:16:44.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:44.979+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.980590+0000) 2019-10-13T11:16:44.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:44.979+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.981141+0000) 2019-10-13T11:16:44.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:44.979+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.981291+0000) 2019-10-13T11:16:44.983 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:44.980+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.981538+0000) 2019-10-13T11:16:44.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:44.980+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.981999+0000) 2019-10-13T11:16:44.984 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:44.980+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.982051+0000) 2019-10-13T11:16:44.985 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:44.982+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.983542+0000) 2019-10-13T11:16:44.990 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:44.987+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206014 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:45.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:45.270+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:45.270348+0000) 2019-10-13T11:16:45.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:45.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:45.690351+0000) 2019-10-13T11:16:45.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:45.689+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:45.690351+0000) 2019-10-13T11:16:48.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:48.596+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.597198+0000) 2019-10-13T11:16:48.932 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:48.930+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.931348+0000) 2019-10-13T11:16:49.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:49.991+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.991879+0000) 2019-10-13T11:16:49.993 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:49.989+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.991335+0000) 2019-10-13T11:16:49.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:49.990+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.991534+0000) 2019-10-13T11:16:49.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:49.990+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.991711+0000) 2019-10-13T11:16:49.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:49.990+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.991875+0000) 2019-10-13T11:16:49.994 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:49.990+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.992003+0000) 2019-10-13T11:16:49.994 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:49.990+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.992160+0000) 2019-10-13T11:16:49.994 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:49.993+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:49.993909+0000) 2019-10-13T11:16:50.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:49.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:51.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:51.548+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:51.549431+0000) 2019-10-13T11:16:53.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:53.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.035898+0000) 2019-10-13T11:16:53.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:53.038+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.039083+0000) 2019-10-13T11:16:53.253 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:16:53.253 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:53.257 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:16:53.257 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:55.002 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:54.999+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.001027+0000) 2019-10-13T11:16:55.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:16:55.001+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.002140+0000) 2019-10-13T11:16:55.003 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:16:55.000+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.001688+0000) 2019-10-13T11:16:55.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:55.000+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.001808+0000) 2019-10-13T11:16:55.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:55.000+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.001991+0000) 2019-10-13T11:16:55.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:55.000+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.002201+0000) 2019-10-13T11:16:55.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:16:55.001+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.002413+0000) 2019-10-13T11:16:55.004 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:55.003+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.004172+0000) 2019-10-13T11:16:55.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:16:55.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206075 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:16:55.273 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:16:55.272+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.272758+0000) 2019-10-13T11:16:55.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:16:55.689+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.690517+0000) 2019-10-13T11:16:55.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:16:55.689+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:55.690517+0000) 2019-10-13T11:16:58.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:16:58.596+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.597357+0000) 2019-10-13T11:16:58.932 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:16:58.931+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.931524+0000) 2019-10-13T11:17:00.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:00.009+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.011334+0000) 2019-10-13T11:17:00.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:00.010+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.012206+0000) 2019-10-13T11:17:00.013 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:00.010+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.011852+0000) 2019-10-13T11:17:00.013 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:00.010+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.012023+0000) 2019-10-13T11:17:00.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:00.013+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.015592+0000) 2019-10-13T11:17:00.016 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:00.010+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.012278+0000) 2019-10-13T11:17:00.016 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:00.011+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.012415+0000) 2019-10-13T11:17:00.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:00.011+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:00.013012+0000) 2019-10-13T11:17:00.021 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:00.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206103 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:01.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:01.549+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:01.549625+0000) 2019-10-13T11:17:03.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:03.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.036013+0000) 2019-10-13T11:17:03.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:03.038+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.039239+0000) 2019-10-13T11:17:05.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:05.022+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.022528+0000) 2019-10-13T11:17:05.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:05.022+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.022934+0000) 2019-10-13T11:17:05.024 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:05.022+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.022842+0000) 2019-10-13T11:17:05.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:05.021+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.023495+0000) 2019-10-13T11:17:05.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:05.022+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.023243+0000) 2019-10-13T11:17:05.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:05.022+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.023152+0000) 2019-10-13T11:17:05.025 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:05.023+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.023457+0000) 2019-10-13T11:17:05.026 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:05.024+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.026698+0000) 2019-10-13T11:17:05.031 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:05.029+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206134 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:05.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:05.273+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.274863+0000) 2019-10-13T11:17:05.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:05.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.690687+0000) 2019-10-13T11:17:05.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:05.690+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:05.690687+0000) 2019-10-13T11:17:08.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:08.597+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.597518+0000) 2019-10-13T11:17:08.932 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:08.931+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.931732+0000) 2019-10-13T11:17:10.033 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:10.031+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.031980+0000) 2019-10-13T11:17:10.033 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:10.031+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.032363+0000) 2019-10-13T11:17:10.034 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:10.031+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.033258+0000) 2019-10-13T11:17:10.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:10.032+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.032756+0000) 2019-10-13T11:17:10.034 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:10.032+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.032945+0000) 2019-10-13T11:17:10.035 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:10.032+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.033029+0000) 2019-10-13T11:17:10.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:10.032+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.033170+0000) 2019-10-13T11:17:10.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:10.033+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:10.035504+0000) 2019-10-13T11:17:10.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:10.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206164 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:11.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:11.549+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:11.549809+0000) 2019-10-13T11:17:13.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:13.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.036206+0000) 2019-10-13T11:17:13.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:13.038+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.039422+0000) 2019-10-13T11:17:15.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:15.039+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.040206+0000) 2019-10-13T11:17:15.041 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:15.040+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.040643+0000) 2019-10-13T11:17:15.042 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:15.040+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.040806+0000) 2019-10-13T11:17:15.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:15.041+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.041973+0000) 2019-10-13T11:17:15.043 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:15.040+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.041098+0000) 2019-10-13T11:17:15.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:15.040+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.040675+0000) 2019-10-13T11:17:15.043 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:15.041+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.041389+0000) 2019-10-13T11:17:15.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:15.043+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.044445+0000) 2019-10-13T11:17:15.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:15.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206195 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:15.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:15.276+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.276868+0000) 2019-10-13T11:17:15.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:15.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.690867+0000) 2019-10-13T11:17:15.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:15.690+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:15.690866+0000) 2019-10-13T11:17:18.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:18.597+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.597674+0000) 2019-10-13T11:17:18.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:18.931+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.931898+0000) 2019-10-13T11:17:20.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:20.049+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.050308+0000) 2019-10-13T11:17:20.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:20.050+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.050669+0000) 2019-10-13T11:17:20.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:20.050+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.051281+0000) 2019-10-13T11:17:20.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:20.050+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.050918+0000) 2019-10-13T11:17:20.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:20.051+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.051425+0000) 2019-10-13T11:17:20.053 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:20.051+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.051561+0000) 2019-10-13T11:17:20.053 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:20.051+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.051960+0000) 2019-10-13T11:17:20.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:20.053+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:20.054684+0000) 2019-10-13T11:17:20.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:20.058+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:21.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:21.548+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:21.549934+0000) 2019-10-13T11:17:23.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:23.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.036369+0000) 2019-10-13T11:17:23.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:23.039+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.039612+0000) 2019-10-13T11:17:23.399 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:17:23.399 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:23.406 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:17:23.406 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:25.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:25.059+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.061134+0000) 2019-10-13T11:17:25.062 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:25.060+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.061568+0000) 2019-10-13T11:17:25.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:25.061+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.062653+0000) 2019-10-13T11:17:25.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:25.060+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.062064+0000) 2019-10-13T11:17:25.064 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:25.060+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.062233+0000) 2019-10-13T11:17:25.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:25.061+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.062474+0000) 2019-10-13T11:17:25.064 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:25.061+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.062899+0000) 2019-10-13T11:17:25.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:25.063+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.064636+0000) 2019-10-13T11:17:25.070 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:25.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:25.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:25.279+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.279816+0000) 2019-10-13T11:17:25.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:25.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.691057+0000) 2019-10-13T11:17:25.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:25.690+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:25.691055+0000) 2019-10-13T11:17:28.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:28.596+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.597850+0000) 2019-10-13T11:17:28.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:28.930+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.932041+0000) 2019-10-13T11:17:30.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:30.070+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.071726+0000) 2019-10-13T11:17:30.078 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:30.070+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.071897+0000) 2019-10-13T11:17:30.079 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:30.070+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.072072+0000) 2019-10-13T11:17:30.079 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:30.070+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.072257+0000) 2019-10-13T11:17:30.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:30.070+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.072349+0000) 2019-10-13T11:17:30.080 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:30.071+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.072547+0000) 2019-10-13T11:17:30.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:30.071+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.071843+0000) 2019-10-13T11:17:30.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:30.073+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:30.074053+0000) 2019-10-13T11:17:30.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:30.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206284 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:31.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:31.549+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:31.550118+0000) 2019-10-13T11:17:33.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:33.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.036532+0000) 2019-10-13T11:17:33.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:33.039+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.039811+0000) 2019-10-13T11:17:35.082 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:35.079+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.081331+0000) 2019-10-13T11:17:35.083 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:35.080+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.081440+0000) 2019-10-13T11:17:35.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:35.080+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.081790+0000) 2019-10-13T11:17:35.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:35.081+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.082436+0000) 2019-10-13T11:17:35.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:35.080+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.082217+0000) 2019-10-13T11:17:35.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:35.080+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.082375+0000) 2019-10-13T11:17:35.085 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:35.081+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.082677+0000) 2019-10-13T11:17:35.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:35.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206315 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:35.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:35.085+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.086590+0000) 2019-10-13T11:17:35.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:35.281+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.282184+0000) 2019-10-13T11:17:35.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:35.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.691249+0000) 2019-10-13T11:17:35.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:35.690+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:35.691248+0000) 2019-10-13T11:17:38.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:38.597+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.598011+0000) 2019-10-13T11:17:38.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:38.931+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.932188+0000) 2019-10-13T11:17:40.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:40.091+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.092381+0000) 2019-10-13T11:17:40.093 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:40.090+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.092161+0000) 2019-10-13T11:17:40.094 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:40.090+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.092206+0000) 2019-10-13T11:17:40.094 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:40.090+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.092250+0000) 2019-10-13T11:17:40.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:40.091+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.092417+0000) 2019-10-13T11:17:40.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:40.091+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.092701+0000) 2019-10-13T11:17:40.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:40.091+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.092870+0000) 2019-10-13T11:17:40.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:40.095+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:40.096558+0000) 2019-10-13T11:17:40.101 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:40.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:41.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:41.549+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:41.550305+0000) 2019-10-13T11:17:43.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:43.036+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.036694+0000) 2019-10-13T11:17:43.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:43.039+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.039943+0000) 2019-10-13T11:17:45.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:45.100+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.101650+0000) 2019-10-13T11:17:45.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:45.100+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.102194+0000) 2019-10-13T11:17:45.103 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:45.101+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.102448+0000) 2019-10-13T11:17:45.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:45.102+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.103410+0000) 2019-10-13T11:17:45.104 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:45.101+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.103072+0000) 2019-10-13T11:17:45.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:45.101+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.103216+0000) 2019-10-13T11:17:45.105 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:45.101+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.103363+0000) 2019-10-13T11:17:45.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:45.104+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.104896+0000) 2019-10-13T11:17:45.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:45.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:45.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:45.284+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.284754+0000) 2019-10-13T11:17:45.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:45.690+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.691431+0000) 2019-10-13T11:17:45.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:45.690+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:45.691436+0000) 2019-10-13T11:17:48.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:48.597+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.598100+0000) 2019-10-13T11:17:48.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:48.931+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.932305+0000) 2019-10-13T11:17:50.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:50.111+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.112059+0000) 2019-10-13T11:17:50.113 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:50.111+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.112318+0000) 2019-10-13T11:17:50.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:50.112+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.112461+0000) 2019-10-13T11:17:50.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:50.112+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.112548+0000) 2019-10-13T11:17:50.114 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:50.112+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.112675+0000) 2019-10-13T11:17:50.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:50.111+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.112972+0000) 2019-10-13T11:17:50.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:50.112+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.112972+0000) 2019-10-13T11:17:50.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:50.114+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:50.115770+0000) 2019-10-13T11:17:50.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:50.118+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206404 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:51.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:51.550+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:51.550505+0000) 2019-10-13T11:17:53.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:53.036+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.036826+0000) 2019-10-13T11:17:53.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:53.039+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.040130+0000) 2019-10-13T11:17:53.595 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:17:53.596 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:53.600 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:17:53.600 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:55.122 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:55.120+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.120672+0000) 2019-10-13T11:17:55.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:55.120+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.120851+0000) 2019-10-13T11:17:55.123 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:17:55.120+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.121839+0000) 2019-10-13T11:17:55.123 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:17:55.120+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.121314+0000) 2019-10-13T11:17:55.123 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:55.121+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.121407+0000) 2019-10-13T11:17:55.123 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:17:55.121+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.121617+0000) 2019-10-13T11:17:55.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:55.121+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.121786+0000) 2019-10-13T11:17:55.125 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:55.123+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.125164+0000) 2019-10-13T11:17:55.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:17:55.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206435 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:17:55.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:17:55.286+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.286968+0000) 2019-10-13T11:17:55.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:17:55.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.691609+0000) 2019-10-13T11:17:55.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:17:55.691+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:55.691635+0000) 2019-10-13T11:17:58.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:17:58.598+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.598218+0000) 2019-10-13T11:17:58.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:17:58.932+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.932451+0000) 2019-10-13T11:18:00.132 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:00.130+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.131065+0000) 2019-10-13T11:18:00.132 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:00.131+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.131512+0000) 2019-10-13T11:18:00.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:00.130+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.132108+0000) 2019-10-13T11:18:00.133 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:00.131+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.131648+0000) 2019-10-13T11:18:00.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:00.131+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.131845+0000) 2019-10-13T11:18:00.134 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:00.131+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.131971+0000) 2019-10-13T11:18:00.134 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:00.132+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.132694+0000) 2019-10-13T11:18:00.136 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:00.134+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:00.135855+0000) 2019-10-13T11:18:00.141 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:00.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:01.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:01.550+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:01.550605+0000) 2019-10-13T11:18:03.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:03.036+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.036992+0000) 2019-10-13T11:18:03.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:03.039+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:03.040346+0000) 2019-10-13T11:18:05.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:05.141+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.141509+0000) 2019-10-13T11:18:05.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:05.141+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.141801+0000) 2019-10-13T11:18:05.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:05.141+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.142132+0000) 2019-10-13T11:18:05.143 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:05.141+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.142371+0000) 2019-10-13T11:18:05.144 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:05.141+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.142383+0000) 2019-10-13T11:18:05.144 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:05.142+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.142915+0000) 2019-10-13T11:18:05.145 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:05.142+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.143094+0000) 2019-10-13T11:18:05.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:05.143+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.144467+0000) 2019-10-13T11:18:05.151 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:05.149+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:05.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:05.288+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.289047+0000) 2019-10-13T11:18:05.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:05.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.691788+0000) 2019-10-13T11:18:05.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:05.691+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:05.691809+0000) 2019-10-13T11:18:08.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:08.597+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.598418+0000) 2019-10-13T11:18:08.933 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:08.931+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.932620+0000) 2019-10-13T11:18:10.153 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:10.151+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.152571+0000) 2019-10-13T11:18:10.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:10.150+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.152049+0000) 2019-10-13T11:18:10.155 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:10.150+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.152300+0000) 2019-10-13T11:18:10.155 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:10.151+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.152402+0000) 2019-10-13T11:18:10.155 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:10.151+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.152770+0000) 2019-10-13T11:18:10.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:10.151+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.152998+0000) 2019-10-13T11:18:10.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:10.151+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.153163+0000) 2019-10-13T11:18:10.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:10.155+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:10.155827+0000) 2019-10-13T11:18:10.162 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:10.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206524 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:11.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:11.549+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:11.550768+0000) 2019-10-13T11:18:13.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:13.036+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.037200+0000) 2019-10-13T11:18:13.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:13.039+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:13.040529+0000) 2019-10-13T11:18:15.164 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:15.161+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.162999+0000) 2019-10-13T11:18:15.164 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:15.161+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.163135+0000) 2019-10-13T11:18:15.165 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:15.164+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.164906+0000) 2019-10-13T11:18:15.165 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:15.162+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.163855+0000) 2019-10-13T11:18:15.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:15.162+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.163679+0000) 2019-10-13T11:18:15.166 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:15.162+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.164213+0000) 2019-10-13T11:18:15.166 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:15.162+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.164327+0000) 2019-10-13T11:18:15.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:15.165+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.166391+0000) 2019-10-13T11:18:15.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:15.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206555 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:15.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:15.290+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.291071+0000) 2019-10-13T11:18:15.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:15.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.691960+0000) 2019-10-13T11:18:15.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:15.691+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:15.691959+0000) 2019-10-13T11:18:18.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:18.597+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.598593+0000) 2019-10-13T11:18:18.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:18.932+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.932825+0000) 2019-10-13T11:18:20.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:20.172+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.173455+0000) 2019-10-13T11:18:20.317 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:20.172+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.173572+0000) 2019-10-13T11:18:20.318 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:20.172+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.173845+0000) 2019-10-13T11:18:20.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:20.172+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.173849+0000) 2019-10-13T11:18:20.319 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:20.174+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.175431+0000) 2019-10-13T11:18:20.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:20.177+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.177777+0000) 2019-10-13T11:18:20.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:20.172+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.173993+0000) 2019-10-13T11:18:20.320 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:20.172+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:20.174306+0000) 2019-10-13T11:18:20.320 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:20.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:21.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:21.549+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:21.550897+0000) 2019-10-13T11:18:23.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:23.035+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.037384+0000) 2019-10-13T11:18:23.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:23.039+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:23.040723+0000) 2019-10-13T11:18:23.766 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:18:23.766 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:23.770 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:18:23.770 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:25.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:25.181+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.182901+0000) 2019-10-13T11:18:25.184 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:25.181+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.183169+0000) 2019-10-13T11:18:25.185 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:25.182+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.183759+0000) 2019-10-13T11:18:25.185 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:25.182+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.183786+0000) 2019-10-13T11:18:25.185 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:25.182+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.184065+0000) 2019-10-13T11:18:25.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:25.182+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.184271+0000) 2019-10-13T11:18:25.186 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:25.184+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.184983+0000) 2019-10-13T11:18:25.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:25.186+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.187301+0000) 2019-10-13T11:18:25.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:25.190+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206614 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:25.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:25.293+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.293427+0000) 2019-10-13T11:18:25.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:25.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.692147+0000) 2019-10-13T11:18:25.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:25.691+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:25.692144+0000) 2019-10-13T11:18:28.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:28.598+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.598763+0000) 2019-10-13T11:18:28.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:28.931+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.933029+0000) 2019-10-13T11:18:30.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:30.193+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.194063+0000) 2019-10-13T11:18:30.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:30.192+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.193500+0000) 2019-10-13T11:18:30.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:30.192+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.193702+0000) 2019-10-13T11:18:30.195 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:30.192+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.194160+0000) 2019-10-13T11:18:30.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:30.192+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.194196+0000) 2019-10-13T11:18:30.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:30.193+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.194417+0000) 2019-10-13T11:18:30.196 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:30.193+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.194866+0000) 2019-10-13T11:18:30.198 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:30.197+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:30.197882+0000) 2019-10-13T11:18:30.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:30.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:31.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:31.550+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:31.551067+0000) 2019-10-13T11:18:33.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:33.037+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.037565+0000) 2019-10-13T11:18:33.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:33.040+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:33.040911+0000) 2019-10-13T11:18:35.205 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:35.203+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.203936+0000) 2019-10-13T11:18:35.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:35.204+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.205033+0000) 2019-10-13T11:18:35.206 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:35.204+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.204607+0000) 2019-10-13T11:18:35.206 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:35.204+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.204912+0000) 2019-10-13T11:18:35.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:35.204+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.204996+0000) 2019-10-13T11:18:35.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:35.204+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.205297+0000) 2019-10-13T11:18:35.207 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:35.205+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.205447+0000) 2019-10-13T11:18:35.207 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:35.206+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.206889+0000) 2019-10-13T11:18:35.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:35.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:35.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:35.295+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.295496+0000) 2019-10-13T11:18:35.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:35.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.692321+0000) 2019-10-13T11:18:35.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:35.691+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:35.692321+0000) 2019-10-13T11:18:38.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:38.598+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.598952+0000) 2019-10-13T11:18:38.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:38.932+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.933210+0000) 2019-10-13T11:18:40.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:40.214+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.214429+0000) 2019-10-13T11:18:40.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:40.214+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.214665+0000) 2019-10-13T11:18:40.216 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:40.214+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.215018+0000) 2019-10-13T11:18:40.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:40.213+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.215579+0000) 2019-10-13T11:18:40.217 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:40.215+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.215656+0000) 2019-10-13T11:18:40.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:40.215+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.215529+0000) 2019-10-13T11:18:40.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:40.214+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.215155+0000) 2019-10-13T11:18:40.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:40.216+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:40.218600+0000) 2019-10-13T11:18:40.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:40.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206703 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:41.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:41.550+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:41.551264+0000) 2019-10-13T11:18:43.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:43.037+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.037709+0000) 2019-10-13T11:18:43.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:43.040+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:43.041094+0000) 2019-10-13T11:18:45.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:45.224+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.224886+0000) 2019-10-13T11:18:45.227 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:45.224+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.225136+0000) 2019-10-13T11:18:45.227 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:45.225+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.225527+0000) 2019-10-13T11:18:45.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:45.224+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.225365+0000) 2019-10-13T11:18:45.227 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:45.225+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.225681+0000) 2019-10-13T11:18:45.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:45.225+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.225794+0000) 2019-10-13T11:18:45.228 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:45.225+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.226865+0000) 2019-10-13T11:18:45.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:45.227+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.229147+0000) 2019-10-13T11:18:45.234 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:45.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206734 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:45.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:45.296+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.297592+0000) 2019-10-13T11:18:45.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:45.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.692453+0000) 2019-10-13T11:18:45.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:45.692+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:45.692483+0000) 2019-10-13T11:18:48.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:48.599+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.599176+0000) 2019-10-13T11:18:48.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:48.932+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.933359+0000) 2019-10-13T11:18:50.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:50.234+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.234878+0000) 2019-10-13T11:18:50.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:50.234+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.235038+0000) 2019-10-13T11:18:50.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:50.234+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.235226+0000) 2019-10-13T11:18:50.237 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:50.234+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.235389+0000) 2019-10-13T11:18:50.237 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:50.234+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.236649+0000) 2019-10-13T11:18:50.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:50.235+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.235656+0000) 2019-10-13T11:18:50.238 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:50.235+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.236193+0000) 2019-10-13T11:18:50.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:50.237+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:50.238956+0000) 2019-10-13T11:18:50.245 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:50.243+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:51.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:51.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:51.551436+0000) 2019-10-13T11:18:53.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:53.036+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.037879+0000) 2019-10-13T11:18:53.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:53.040+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:53.041273+0000) 2019-10-13T11:18:53.852 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:18:53.852 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:53.857 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:18:53.857 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:55.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:18:55.243+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.245389+0000) 2019-10-13T11:18:55.247 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:18:55.244+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.245514+0000) 2019-10-13T11:18:55.247 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:55.244+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.245616+0000) 2019-10-13T11:18:55.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:55.244+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.245708+0000) 2019-10-13T11:18:55.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:55.244+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.245759+0000) 2019-10-13T11:18:55.248 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:55.244+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.246044+0000) 2019-10-13T11:18:55.248 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:18:55.246+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.247263+0000) 2019-10-13T11:18:55.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:55.248+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.249310+0000) 2019-10-13T11:18:55.254 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:18:55.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:18:55.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:18:55.298+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.299716+0000) 2019-10-13T11:18:55.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:18:55.691+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.692647+0000) 2019-10-13T11:18:55.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:18:55.691+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:55.692648+0000) 2019-10-13T11:18:58.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:18:58.598+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.599364+0000) 2019-10-13T11:18:58.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:18:58.932+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.933459+0000) 2019-10-13T11:19:00.256 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:00.254+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.255427+0000) 2019-10-13T11:19:00.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:00.254+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.255941+0000) 2019-10-13T11:19:00.257 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:00.254+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.256231+0000) 2019-10-13T11:19:00.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:00.256+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.257595+0000) 2019-10-13T11:19:00.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:00.255+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.256541+0000) 2019-10-13T11:19:00.258 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:00.255+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.256670+0000) 2019-10-13T11:19:00.258 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:00.255+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.256974+0000) 2019-10-13T11:19:00.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:00.257+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:00.258396+0000) 2019-10-13T11:19:00.264 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:00.262+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:01.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:01.550+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:01.551634+0000) 2019-10-13T11:19:03.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:03.037+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.038063+0000) 2019-10-13T11:19:03.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:03.040+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:03.041424+0000) 2019-10-13T11:19:05.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:05.264+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.265526+0000) 2019-10-13T11:19:05.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:05.264+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.265723+0000) 2019-10-13T11:19:05.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:05.264+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.265952+0000) 2019-10-13T11:19:05.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:05.264+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.266217+0000) 2019-10-13T11:19:05.268 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:05.266+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.267134+0000) 2019-10-13T11:19:05.268 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:05.265+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.266593+0000) 2019-10-13T11:19:05.269 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:05.265+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.266821+0000) 2019-10-13T11:19:05.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:05.269+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.269743+0000) 2019-10-13T11:19:05.275 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:05.272+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206854 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:05.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:05.301+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.301767+0000) 2019-10-13T11:19:05.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:05.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.692837+0000) 2019-10-13T11:19:05.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:05.692+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:05.692837+0000) 2019-10-13T11:19:08.600 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:08.598+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.599607+0000) 2019-10-13T11:19:08.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:08.932+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.933617+0000) 2019-10-13T11:19:10.277 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:10.274+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.275794+0000) 2019-10-13T11:19:10.277 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:10.274+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.276303+0000) 2019-10-13T11:19:10.277 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:10.275+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.276477+0000) 2019-10-13T11:19:10.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:10.276+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.277398+0000) 2019-10-13T11:19:10.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:10.275+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.276760+0000) 2019-10-13T11:19:10.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:10.275+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.276961+0000) 2019-10-13T11:19:10.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:10.275+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.277126+0000) 2019-10-13T11:19:10.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:10.279+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:10.280045+0000) 2019-10-13T11:19:10.285 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:10.283+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206885 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:11.552 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:11.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:11.551803+0000) 2019-10-13T11:19:13.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:13.037+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.038269+0000) 2019-10-13T11:19:13.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:13.040+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:13.041595+0000) 2019-10-13T11:19:15.287 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:15.284+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.286328+0000) 2019-10-13T11:19:15.287 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:15.285+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.286442+0000) 2019-10-13T11:19:15.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:15.285+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.286765+0000) 2019-10-13T11:19:15.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:15.287+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.288260+0000) 2019-10-13T11:19:15.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:15.285+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.287116+0000) 2019-10-13T11:19:15.289 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:15.286+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.287811+0000) 2019-10-13T11:19:15.289 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:15.286+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.287902+0000) 2019-10-13T11:19:15.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:15.288+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.289546+0000) 2019-10-13T11:19:15.295 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:15.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206916 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:15.304 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:15.303+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.303769+0000) 2019-10-13T11:19:15.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:15.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.693007+0000) 2019-10-13T11:19:15.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:15.692+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:15.693007+0000) 2019-10-13T11:19:18.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:18.599+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.599797+0000) 2019-10-13T11:19:18.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:18.933+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.933803+0000) 2019-10-13T11:19:20.297 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:20.296+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.296624+0000) 2019-10-13T11:19:20.298 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:20.296+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.297059+0000) 2019-10-13T11:19:20.299 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:20.297+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.297942+0000) 2019-10-13T11:19:20.299 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:20.296+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.297369+0000) 2019-10-13T11:19:20.299 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:20.297+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.297599+0000) 2019-10-13T11:19:20.299 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:20.297+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.297906+0000) 2019-10-13T11:19:20.300 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:20.297+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.298144+0000) 2019-10-13T11:19:20.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:20.299+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:20.299737+0000) 2019-10-13T11:19:20.306 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:20.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:21.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:21.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:21.551961+0000) 2019-10-13T11:19:23.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:23.038+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.038428+0000) 2019-10-13T11:19:23.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:23.041+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:23.041768+0000) 2019-10-13T11:19:23.992 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:19:23.992 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:23.996 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:19:23.997 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:25.306 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:25.305+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.305898+0000) 2019-10-13T11:19:25.308 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:25.306+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.307271+0000) 2019-10-13T11:19:25.309 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:25.307+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.307465+0000) 2019-10-13T11:19:25.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:25.307+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.307564+0000) 2019-10-13T11:19:25.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:25.307+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.307876+0000) 2019-10-13T11:19:25.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:25.308+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.308787+0000) 2019-10-13T11:19:25.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:25.307+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.308295+0000) 2019-10-13T11:19:25.310 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:25.308+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.308561+0000) 2019-10-13T11:19:25.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:25.310+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.311203+0000) 2019-10-13T11:19:25.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:25.315+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 206977 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:25.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:25.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.693195+0000) 2019-10-13T11:19:25.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:25.692+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:25.693194+0000) 2019-10-13T11:19:28.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:28.599+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.599924+0000) 2019-10-13T11:19:28.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:28.933+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.933932+0000) 2019-10-13T11:19:30.318 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:30.316+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.317385+0000) 2019-10-13T11:19:30.319 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:30.317+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.317933+0000) 2019-10-13T11:19:30.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:30.317+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.318037+0000) 2019-10-13T11:19:30.320 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:30.316+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.318415+0000) 2019-10-13T11:19:30.320 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:30.318+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.318469+0000) 2019-10-13T11:19:30.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:30.318+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.318841+0000) 2019-10-13T11:19:30.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:30.318+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.319032+0000) 2019-10-13T11:19:30.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:30.319+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:30.320982+0000) 2019-10-13T11:19:30.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:30.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:31.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:31.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:31.552156+0000) 2019-10-13T11:19:33.039 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:33.038+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.038608+0000) 2019-10-13T11:19:33.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:33.040+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:33.041952+0000) 2019-10-13T11:19:35.308 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:35.307+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.308427+0000) 2019-10-13T11:19:35.329 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:35.327+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.327805+0000) 2019-10-13T11:19:35.329 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:35.327+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.327833+0000) 2019-10-13T11:19:35.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:35.327+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.328021+0000) 2019-10-13T11:19:35.329 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:35.328+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.328406+0000) 2019-10-13T11:19:35.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:35.327+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.329006+0000) 2019-10-13T11:19:35.330 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:35.328+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.328671+0000) 2019-10-13T11:19:35.331 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:35.328+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.328873+0000) 2019-10-13T11:19:35.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:35.329+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.330781+0000) 2019-10-13T11:19:35.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:35.334+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207036 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:35.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:35.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.693334+0000) 2019-10-13T11:19:35.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:35.693+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:35.693334+0000) 2019-10-13T11:19:38.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:38.599+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.600068+0000) 2019-10-13T11:19:38.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:38.932+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.934105+0000) 2019-10-13T11:19:40.338 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:40.335+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.337025+0000) 2019-10-13T11:19:40.338 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:40.335+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.337265+0000) 2019-10-13T11:19:40.339 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:40.336+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.338522+0000) 2019-10-13T11:19:40.339 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:40.336+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.337649+0000) 2019-10-13T11:19:40.339 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:40.336+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.338055+0000) 2019-10-13T11:19:40.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:40.336+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.338218+0000) 2019-10-13T11:19:40.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:40.337+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.338550+0000) 2019-10-13T11:19:40.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:40.338+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:40.340277+0000) 2019-10-13T11:19:40.347 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:40.344+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207066 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:41.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:41.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:41.552358+0000) 2019-10-13T11:19:43.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:43.037+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.038793+0000) 2019-10-13T11:19:43.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:43.041+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:43.042134+0000) 2019-10-13T11:19:45.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:45.310+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.310737+0000) 2019-10-13T11:19:45.349 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:45.346+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.348006+0000) 2019-10-13T11:19:45.349 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:45.346+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.348154+0000) 2019-10-13T11:19:45.349 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:45.346+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.348296+0000) 2019-10-13T11:19:45.350 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:45.348+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.349544+0000) 2019-10-13T11:19:45.350 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:45.347+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.349046+0000) 2019-10-13T11:19:45.351 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:45.347+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.349106+0000) 2019-10-13T11:19:45.351 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:45.347+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.348573+0000) 2019-10-13T11:19:45.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:45.350+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.351406+0000) 2019-10-13T11:19:45.358 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:45.355+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207097 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:45.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:45.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.693471+0000) 2019-10-13T11:19:45.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:45.692+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:45.693472+0000) 2019-10-13T11:19:48.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:48.598+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.600264+0000) 2019-10-13T11:19:48.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:48.933+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.934291+0000) 2019-10-13T11:19:50.360 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:50.357+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.359154+0000) 2019-10-13T11:19:50.360 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:50.357+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.359387+0000) 2019-10-13T11:19:50.360 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:50.358+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.359557+0000) 2019-10-13T11:19:50.361 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:50.359+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.360457+0000) 2019-10-13T11:19:50.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:50.358+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.359795+0000) 2019-10-13T11:19:50.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:50.358+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.359927+0000) 2019-10-13T11:19:50.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:50.358+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.360093+0000) 2019-10-13T11:19:50.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:50.362+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:50.363004+0000) 2019-10-13T11:19:50.368 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:50.365+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207127 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:51.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:51.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:51.552568+0000) 2019-10-13T11:19:53.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:53.038+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.038943+0000) 2019-10-13T11:19:53.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:53.041+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:53.042321+0000) 2019-10-13T11:19:54.137 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:19:54.137 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:54.142 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:19:54.142 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:55.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:55.312+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.313094+0000) 2019-10-13T11:19:55.370 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:55.367+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.369182+0000) 2019-10-13T11:19:55.371 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:19:55.367+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.369312+0000) 2019-10-13T11:19:55.371 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:19:55.368+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.369641+0000) 2019-10-13T11:19:55.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:55.368+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.369785+0000) 2019-10-13T11:19:55.372 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:19:55.369+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.370526+0000) 2019-10-13T11:19:55.372 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:19:55.370+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.371624+0000) 2019-10-13T11:19:55.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:55.368+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.370245+0000) 2019-10-13T11:19:55.373 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:55.369+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.370512+0000) 2019-10-13T11:19:55.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:19:55.376+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:19:55.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:19:55.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.693629+0000) 2019-10-13T11:19:55.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:19:55.692+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:55.693628+0000) 2019-10-13T11:19:58.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:19:58.599+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.600456+0000) 2019-10-13T11:19:58.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:19:58.933+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.934485+0000) 2019-10-13T11:20:00.380 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:00.377+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.379056+0000) 2019-10-13T11:20:00.380 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:00.378+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.379502+0000) 2019-10-13T11:20:00.381 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:00.377+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.379283+0000) 2019-10-13T11:20:00.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:00.378+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.379800+0000) 2019-10-13T11:20:00.381 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:00.378+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.380107+0000) 2019-10-13T11:20:00.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:00.379+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.380702+0000) 2019-10-13T11:20:00.382 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:00.379+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.380500+0000) 2019-10-13T11:20:00.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:00.381+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:00.382318+0000) 2019-10-13T11:20:00.389 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:00.386+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207186 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:01.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:01.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:01.552760+0000) 2019-10-13T11:20:03.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:03.038+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.039119+0000) 2019-10-13T11:20:03.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:03.042+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:03.042468+0000) 2019-10-13T11:20:05.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:05.315+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.315466+0000) 2019-10-13T11:20:05.391 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:05.389+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.390094+0000) 2019-10-13T11:20:05.391 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:05.389+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.390231+0000) 2019-10-13T11:20:05.391 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:05.390+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.390500+0000) 2019-10-13T11:20:05.392 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:05.390+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.391706+0000) 2019-10-13T11:20:05.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:05.390+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.390698+0000) 2019-10-13T11:20:05.393 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:05.390+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.390832+0000) 2019-10-13T11:20:05.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:05.390+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.391158+0000) 2019-10-13T11:20:05.393 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:05.391+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.392697+0000) 2019-10-13T11:20:05.399 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:05.397+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207217 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:05.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:05.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.693796+0000) 2019-10-13T11:20:05.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:05.693+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:05.693794+0000) 2019-10-13T11:20:08.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:08.600+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.600642+0000) 2019-10-13T11:20:08.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:08.934+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.934656+0000) 2019-10-13T11:20:10.401 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:10.400+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.400812+0000) 2019-10-13T11:20:10.402 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:10.399+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.400346+0000) 2019-10-13T11:20:10.402 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:10.400+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.400701+0000) 2019-10-13T11:20:10.402 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:10.400+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.400861+0000) 2019-10-13T11:20:10.402 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:10.400+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.401022+0000) 2019-10-13T11:20:10.402 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:10.400+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.401354+0000) 2019-10-13T11:20:10.403 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:10.401+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.401496+0000) 2019-10-13T11:20:10.403 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:10.403+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:10.403867+0000) 2019-10-13T11:20:10.410 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:10.408+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:11.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:11.552+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:11.552965+0000) 2019-10-13T11:20:13.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:13.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.039271+0000) 2019-10-13T11:20:13.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:13.042+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:13.042625+0000) 2019-10-13T11:20:15.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:15.317+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.317787+0000) 2019-10-13T11:20:15.411 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:15.410+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.410635+0000) 2019-10-13T11:20:15.412 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:15.410+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.410803+0000) 2019-10-13T11:20:15.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:15.411+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.412241+0000) 2019-10-13T11:20:15.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:15.410+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.411314+0000) 2019-10-13T11:20:15.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:15.411+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.411392+0000) 2019-10-13T11:20:15.413 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:15.411+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.411838+0000) 2019-10-13T11:20:15.413 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:15.411+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.411984+0000) 2019-10-13T11:20:15.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:15.413+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.413926+0000) 2019-10-13T11:20:15.420 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:15.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:15.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:15.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.693974+0000) 2019-10-13T11:20:15.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:15.693+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:15.693995+0000) 2019-10-13T11:20:18.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:18.600+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.600821+0000) 2019-10-13T11:20:18.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:18.934+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.934805+0000) 2019-10-13T11:20:20.421 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:20.419+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.420207+0000) 2019-10-13T11:20:20.421 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:20.419+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.420357+0000) 2019-10-13T11:20:20.422 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:20.420+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.420679+0000) 2019-10-13T11:20:20.422 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:20.420+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.421085+0000) 2019-10-13T11:20:20.422 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:20.421+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.421647+0000) 2019-10-13T11:20:20.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:20.421+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.421875+0000) 2019-10-13T11:20:20.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:20.421+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.422836+0000) 2019-10-13T11:20:20.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:20.423+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:20.425590+0000) 2019-10-13T11:20:20.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:20.426+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:21.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:21.553+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:21.553144+0000) 2019-10-13T11:20:23.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:23.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.039462+0000) 2019-10-13T11:20:23.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:23.041+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:23.042822+0000) 2019-10-13T11:20:24.275 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:20:24.275 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:24.280 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:20:24.280 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:25.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:25.319+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.320074+0000) 2019-10-13T11:20:25.429 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:25.427+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.429580+0000) 2019-10-13T11:20:25.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:25.427+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.428867+0000) 2019-10-13T11:20:25.431 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:25.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.429381+0000) 2019-10-13T11:20:25.431 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:25.427+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.429019+0000) 2019-10-13T11:20:25.431 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:25.428+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.429528+0000) 2019-10-13T11:20:25.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:25.428+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.429694+0000) 2019-10-13T11:20:25.432 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:25.428+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.429855+0000) 2019-10-13T11:20:25.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:25.429+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.431145+0000) 2019-10-13T11:20:25.438 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:25.436+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:25.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:25.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.694145+0000) 2019-10-13T11:20:25.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:25.693+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:25.694143+0000) 2019-10-13T11:20:28.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:28.600+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.601013+0000) 2019-10-13T11:20:28.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:28.934+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.934956+0000) 2019-10-13T11:20:30.440 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:30.438+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.440371+0000) 2019-10-13T11:20:30.441 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:30.438+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.439907+0000) 2019-10-13T11:20:30.442 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:30.438+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.440012+0000) 2019-10-13T11:20:30.442 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:30.438+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.440150+0000) 2019-10-13T11:20:30.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:30.439+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.440710+0000) 2019-10-13T11:20:30.442 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:30.439+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.440574+0000) 2019-10-13T11:20:30.442 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:30.439+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.440847+0000) 2019-10-13T11:20:30.443 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:30.441+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:30.443050+0000) 2019-10-13T11:20:30.449 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:30.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:31.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:31.551+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:31.553292+0000) 2019-10-13T11:20:33.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:33.038+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.039637+0000) 2019-10-13T11:20:33.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:33.042+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:33.043033+0000) 2019-10-13T11:20:35.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:35.321+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.322523+0000) 2019-10-13T11:20:35.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:35.448+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.449850+0000) 2019-10-13T11:20:35.452 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:35.448+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.449995+0000) 2019-10-13T11:20:35.452 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:35.448+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.450139+0000) 2019-10-13T11:20:35.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:35.448+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.450313+0000) 2019-10-13T11:20:35.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:35.448+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.450339+0000) 2019-10-13T11:20:35.453 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:35.449+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.450955+0000) 2019-10-13T11:20:35.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:35.450+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.451287+0000) 2019-10-13T11:20:35.453 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:35.452+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.453605+0000) 2019-10-13T11:20:35.458 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:35.455+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:35.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:35.692+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.694338+0000) 2019-10-13T11:20:35.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:35.692+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:35.694337+0000) 2019-10-13T11:20:38.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:38.600+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.601208+0000) 2019-10-13T11:20:38.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:38.934+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.935117+0000) 2019-10-13T11:20:40.460 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:40.458+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.459445+0000) 2019-10-13T11:20:40.460 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:40.458+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.459673+0000) 2019-10-13T11:20:40.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:40.458+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.460072+0000) 2019-10-13T11:20:40.461 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:40.458+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.460089+0000) 2019-10-13T11:20:40.462 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:40.458+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.460375+0000) 2019-10-13T11:20:40.462 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:40.459+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.460815+0000) 2019-10-13T11:20:40.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:40.460+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.461229+0000) 2019-10-13T11:20:40.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:40.462+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:40.463155+0000) 2019-10-13T11:20:40.469 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:40.466+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207426 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:41.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:41.552+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:41.553479+0000) 2019-10-13T11:20:43.040 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:43.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.039801+0000) 2019-10-13T11:20:43.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:43.042+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:43.043208+0000) 2019-10-13T11:20:45.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:45.324+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.324835+0000) 2019-10-13T11:20:45.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:45.468+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.470275+0000) 2019-10-13T11:20:45.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:45.468+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.470370+0000) 2019-10-13T11:20:45.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:45.469+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.470717+0000) 2019-10-13T11:20:45.472 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:45.469+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.470787+0000) 2019-10-13T11:20:45.473 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:45.471+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.472044+0000) 2019-10-13T11:20:45.473 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:45.471+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.472634+0000) 2019-10-13T11:20:45.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:45.469+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.471051+0000) 2019-10-13T11:20:45.473 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:45.469+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.471133+0000) 2019-10-13T11:20:45.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:45.477+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:45.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:45.693+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.694458+0000) 2019-10-13T11:20:45.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:45.693+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:45.694461+0000) 2019-10-13T11:20:48.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:48.600+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.601385+0000) 2019-10-13T11:20:48.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:48.934+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.935272+0000) 2019-10-13T11:20:50.481 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:50.478+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.480341+0000) 2019-10-13T11:20:50.482 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:50.478+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.480273+0000) 2019-10-13T11:20:50.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:50.479+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.480830+0000) 2019-10-13T11:20:50.482 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:50.479+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.480958+0000) 2019-10-13T11:20:50.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:50.481+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.481808+0000) 2019-10-13T11:20:50.483 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:50.479+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.481281+0000) 2019-10-13T11:20:50.483 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:50.480+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.481409+0000) 2019-10-13T11:20:50.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:50.482+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:50.483366+0000) 2019-10-13T11:20:50.489 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:50.487+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:51.554 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:51.553+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:51.553676+0000) 2019-10-13T11:20:53.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:53.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.039966+0000) 2019-10-13T11:20:53.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:53.042+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:53.043386+0000) 2019-10-13T11:20:54.447 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:20:54.447 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:54.452 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:20:54.452 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:55.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:55.326+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.327242+0000) 2019-10-13T11:20:55.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:55.489+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.489881+0000) 2019-10-13T11:20:55.491 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:55.489+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.490045+0000) 2019-10-13T11:20:55.491 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:55.489+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.490172+0000) 2019-10-13T11:20:55.492 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:20:55.490+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.490431+0000) 2019-10-13T11:20:55.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:20:55.490+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.491524+0000) 2019-10-13T11:20:55.493 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:20:55.490+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.490741+0000) 2019-10-13T11:20:55.493 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:55.490+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.490936+0000) 2019-10-13T11:20:55.493 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:20:55.492+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.493102+0000) 2019-10-13T11:20:55.497 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:20:55.496+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:20:55.695 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:20:55.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.694624+0000) 2019-10-13T11:20:55.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:20:55.694+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:55.694624+0000) 2019-10-13T11:20:58.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:20:58.601+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.601614+0000) 2019-10-13T11:20:58.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:20:58.935+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.935446+0000) 2019-10-13T11:21:00.499 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:00.497+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.498152+0000) 2019-10-13T11:21:00.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:00.497+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.498321+0000) 2019-10-13T11:21:00.500 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:00.498+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.499559+0000) 2019-10-13T11:21:00.500 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:00.498+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.498949+0000) 2019-10-13T11:21:00.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:00.498+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.499112+0000) 2019-10-13T11:21:00.501 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:00.498+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.499257+0000) 2019-10-13T11:21:00.501 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:00.499+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.499534+0000) 2019-10-13T11:21:00.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:00.500+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:00.501636+0000) 2019-10-13T11:21:00.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:00.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:01.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:01.553+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:01.553863+0000) 2019-10-13T11:21:03.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:03.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.040149+0000) 2019-10-13T11:21:03.043 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:03.043+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:03.043569+0000) 2019-10-13T11:21:05.330 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:05.329+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.329670+0000) 2019-10-13T11:21:05.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:05.507+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.508682+0000) 2019-10-13T11:21:05.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:05.507+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.508145+0000) 2019-10-13T11:21:05.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:05.507+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.508266+0000) 2019-10-13T11:21:05.510 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:05.508+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.508455+0000) 2019-10-13T11:21:05.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:05.508+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.508628+0000) 2019-10-13T11:21:05.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:05.508+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.509206+0000) 2019-10-13T11:21:05.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:05.508+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.509353+0000) 2019-10-13T11:21:05.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:05.510+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.511580+0000) 2019-10-13T11:21:05.517 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:05.516+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:05.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:05.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.694797+0000) 2019-10-13T11:21:05.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:05.694+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:05.694796+0000) 2019-10-13T11:21:08.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:08.601+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.601833+0000) 2019-10-13T11:21:08.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:08.935+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.935638+0000) 2019-10-13T11:21:10.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:10.516+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.518152+0000) 2019-10-13T11:21:10.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:10.516+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.518359+0000) 2019-10-13T11:21:10.520 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:10.517+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.518639+0000) 2019-10-13T11:21:10.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:10.519+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.519955+0000) 2019-10-13T11:21:10.521 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:10.517+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.518824+0000) 2019-10-13T11:21:10.521 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:10.517+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.518841+0000) 2019-10-13T11:21:10.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:10.518+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.519631+0000) 2019-10-13T11:21:10.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:10.521+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:10.522391+0000) 2019-10-13T11:21:10.528 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:10.525+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:11.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:11.552+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:11.554004+0000) 2019-10-13T11:21:13.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:13.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.040339+0000) 2019-10-13T11:21:13.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:13.042+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:13.043764+0000) 2019-10-13T11:21:15.332 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:15.331+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.332106+0000) 2019-10-13T11:21:15.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:15.527+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.528818+0000) 2019-10-13T11:21:15.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:15.527+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.528841+0000) 2019-10-13T11:21:15.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:15.527+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.529132+0000) 2019-10-13T11:21:15.531 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:15.527+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.529293+0000) 2019-10-13T11:21:15.531 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:15.528+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.529450+0000) 2019-10-13T11:21:15.531 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:15.527+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.529676+0000) 2019-10-13T11:21:15.532 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:15.528+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.529823+0000) 2019-10-13T11:21:15.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:15.530+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.532239+0000) 2019-10-13T11:21:15.538 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:15.535+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207636 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:15.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:15.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.694981+0000) 2019-10-13T11:21:15.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:15.694+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:15.694975+0000) 2019-10-13T11:21:18.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:18.601+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.602005+0000) 2019-10-13T11:21:18.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:18.935+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.935829+0000) 2019-10-13T11:21:20.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:20.537+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.538598+0000) 2019-10-13T11:21:20.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:20.537+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.539420+0000) 2019-10-13T11:21:20.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:20.537+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.539050+0000) 2019-10-13T11:21:20.541 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:20.537+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.539361+0000) 2019-10-13T11:21:20.541 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:20.538+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.539619+0000) 2019-10-13T11:21:20.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:20.538+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.539779+0000) 2019-10-13T11:21:20.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:20.538+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.539830+0000) 2019-10-13T11:21:20.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:20.540+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:20.542264+0000) 2019-10-13T11:21:20.546 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:20.543+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:21.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:21.553+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:21.554215+0000) 2019-10-13T11:21:23.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:23.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.040556+0000) 2019-10-13T11:21:23.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:23.043+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:23.043949+0000) 2019-10-13T11:21:24.575 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:21:24.575 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:24.579 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:21:24.579 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:25.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:25.333+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.334513+0000) 2019-10-13T11:21:25.548 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:25.545+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.546856+0000) 2019-10-13T11:21:25.548 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:25.545+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.547055+0000) 2019-10-13T11:21:25.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:25.545+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.547326+0000) 2019-10-13T11:21:25.549 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:25.545+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.547352+0000) 2019-10-13T11:21:25.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:25.546+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.547590+0000) 2019-10-13T11:21:25.549 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:25.548+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.548947+0000) 2019-10-13T11:21:25.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:25.546+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.547887+0000) 2019-10-13T11:21:25.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:25.550+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.551251+0000) 2019-10-13T11:21:25.555 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:25.553+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207696 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:25.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:25.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.695146+0000) 2019-10-13T11:21:25.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:25.694+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:25.695149+0000) 2019-10-13T11:21:28.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:28.601+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.602204+0000) 2019-10-13T11:21:28.937 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:28.935+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.935980+0000) 2019-10-13T11:21:30.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:30.555+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.556853+0000) 2019-10-13T11:21:30.558 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:30.555+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.557048+0000) 2019-10-13T11:21:30.559 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:30.557+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.558223+0000) 2019-10-13T11:21:30.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:30.556+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.557451+0000) 2019-10-13T11:21:30.559 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:30.556+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.557713+0000) 2019-10-13T11:21:30.559 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:30.556+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.557875+0000) 2019-10-13T11:21:30.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:30.556+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.558027+0000) 2019-10-13T11:21:30.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:30.560+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:30.560793+0000) 2019-10-13T11:21:30.566 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:30.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207726 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:31.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:31.553+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:31.554391+0000) 2019-10-13T11:21:33.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:33.039+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.040736+0000) 2019-10-13T11:21:33.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:33.043+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:33.044145+0000) 2019-10-13T11:21:35.337 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:35.336+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.336985+0000) 2019-10-13T11:21:35.567 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:35.566+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.567583+0000) 2019-10-13T11:21:35.568 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:35.565+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.567247+0000) 2019-10-13T11:21:35.568 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:35.566+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.567478+0000) 2019-10-13T11:21:35.569 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:35.566+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.567881+0000) 2019-10-13T11:21:35.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:35.566+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.568245+0000) 2019-10-13T11:21:35.569 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:35.567+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.568493+0000) 2019-10-13T11:21:35.570 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:35.567+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.568705+0000) 2019-10-13T11:21:35.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:35.569+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.570102+0000) 2019-10-13T11:21:35.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:35.574+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207756 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:35.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:35.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.695346+0000) 2019-10-13T11:21:35.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:35.694+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:35.695344+0000) 2019-10-13T11:21:38.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:38.601+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.602374+0000) 2019-10-13T11:21:38.937 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:38.935+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.936170+0000) 2019-10-13T11:21:40.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:40.577+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.577599+0000) 2019-10-13T11:21:40.579 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:40.577+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.577809+0000) 2019-10-13T11:21:40.580 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:40.578+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.578874+0000) 2019-10-13T11:21:40.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:40.578+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.578421+0000) 2019-10-13T11:21:40.580 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:40.578+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.578528+0000) 2019-10-13T11:21:40.580 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:40.578+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.578709+0000) 2019-10-13T11:21:40.580 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:40.578+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.579060+0000) 2019-10-13T11:21:40.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:40.579+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:40.580310+0000) 2019-10-13T11:21:40.587 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:40.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207786 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:41.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:41.554+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:41.554581+0000) 2019-10-13T11:21:43.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:43.040+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.040952+0000) 2019-10-13T11:21:43.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:43.043+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:43.044327+0000) 2019-10-13T11:21:45.339 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:45.339+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.339398+0000) 2019-10-13T11:21:45.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:45.587+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.588464+0000) 2019-10-13T11:21:45.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:45.587+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.588037+0000) 2019-10-13T11:21:45.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:45.587+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.588146+0000) 2019-10-13T11:21:45.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:45.588+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.588632+0000) 2019-10-13T11:21:45.590 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:45.588+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.588779+0000) 2019-10-13T11:21:45.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:45.588+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.589102+0000) 2019-10-13T11:21:45.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:45.588+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.589056+0000) 2019-10-13T11:21:45.591 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:45.590+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.591647+0000) 2019-10-13T11:21:45.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:45.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:45.696 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:45.695+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.695482+0000) 2019-10-13T11:21:45.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:45.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:45.695484+0000) 2019-10-13T11:21:48.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:48.602+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.602606+0000) 2019-10-13T11:21:48.937 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:48.936+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.936352+0000) 2019-10-13T11:21:50.599 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:50.598+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.598895+0000) 2019-10-13T11:21:50.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:50.597+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.598368+0000) 2019-10-13T11:21:50.600 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:50.598+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.598786+0000) 2019-10-13T11:21:50.600 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:50.598+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.598913+0000) 2019-10-13T11:21:50.601 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:50.598+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.599045+0000) 2019-10-13T11:21:50.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:50.598+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.599241+0000) 2019-10-13T11:21:50.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:50.599+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.599478+0000) 2019-10-13T11:21:50.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:50.601+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:50.602231+0000) 2019-10-13T11:21:50.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:50.606+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207848 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:51.555 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:51.554+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:51.554759+0000) 2019-10-13T11:21:53.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:53.041+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.041161+0000) 2019-10-13T11:21:53.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:53.043+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:53.044527+0000) 2019-10-13T11:21:54.730 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:21:54.730 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:54.735 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:21:54.735 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:55.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:55.341+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.341710+0000) 2019-10-13T11:21:55.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:55.607+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.608723+0000) 2019-10-13T11:21:55.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:55.607+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.608934+0000) 2019-10-13T11:21:55.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:55.607+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.609031+0000) 2019-10-13T11:21:55.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:21:55.609+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.609928+0000) 2019-10-13T11:21:55.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:55.608+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.609884+0000) 2019-10-13T11:21:55.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:21:55.608+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.610089+0000) 2019-10-13T11:21:55.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:21:55.609+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.610471+0000) 2019-10-13T11:21:55.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:21:55.611+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.612188+0000) 2019-10-13T11:21:55.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:21:55.616+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207878 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:21:55.696 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:21:55.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.695644+0000) 2019-10-13T11:21:55.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:21:55.694+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:55.695645+0000) 2019-10-13T11:21:58.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:21:58.601+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.602811+0000) 2019-10-13T11:21:58.937 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:21:58.935+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.936552+0000) 2019-10-13T11:22:00.620 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:00.618+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.619587+0000) 2019-10-13T11:22:00.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:00.618+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.619710+0000) 2019-10-13T11:22:00.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:00.618+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.620116+0000) 2019-10-13T11:22:00.621 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:00.618+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.620259+0000) 2019-10-13T11:22:00.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:00.619+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.620420+0000) 2019-10-13T11:22:00.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:00.619+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.620864+0000) 2019-10-13T11:22:00.622 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:00.620+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.621318+0000) 2019-10-13T11:22:00.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:00.621+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:00.622649+0000) 2019-10-13T11:22:00.629 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:00.627+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207908 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:01.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:01.554+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:01.554930+0000) 2019-10-13T11:22:03.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:03.040+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.041345+0000) 2019-10-13T11:22:03.044 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:03.043+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:03.044689+0000) 2019-10-13T11:22:05.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:05.342+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.344041+0000) 2019-10-13T11:22:05.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:05.628+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.630064+0000) 2019-10-13T11:22:05.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:05.629+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.630577+0000) 2019-10-13T11:22:05.632 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:05.629+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.630810+0000) 2019-10-13T11:22:05.633 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:05.629+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.630818+0000) 2019-10-13T11:22:05.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:05.629+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.631023+0000) 2019-10-13T11:22:05.634 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:05.629+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.631122+0000) 2019-10-13T11:22:05.634 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:05.630+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.631449+0000) 2019-10-13T11:22:05.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:05.631+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.632878+0000) 2019-10-13T11:22:05.639 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:05.637+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:05.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:05.694+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.695790+0000) 2019-10-13T11:22:05.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:05.694+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:05.695801+0000) 2019-10-13T11:22:08.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:08.601+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.602980+0000) 2019-10-13T11:22:08.937 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:08.936+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.936715+0000) 2019-10-13T11:22:10.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:10.638+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.640368+0000) 2019-10-13T11:22:10.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:10.639+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.641672+0000) 2019-10-13T11:22:10.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:10.639+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.641205+0000) 2019-10-13T11:22:10.643 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:10.639+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.641371+0000) 2019-10-13T11:22:10.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:10.640+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.641537+0000) 2019-10-13T11:22:10.643 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:10.640+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.641705+0000) 2019-10-13T11:22:10.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:10.640+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.641926+0000) 2019-10-13T11:22:10.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:10.641+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:10.643497+0000) 2019-10-13T11:22:10.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:10.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 207970 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:11.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:11.554+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:11.555079+0000) 2019-10-13T11:22:13.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:13.040+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.041542+0000) 2019-10-13T11:22:13.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:13.044+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:13.044865+0000) 2019-10-13T11:22:15.346 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:15.345+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.346361+0000) 2019-10-13T11:22:15.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:15.648+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.650238+0000) 2019-10-13T11:22:15.651 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:15.649+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.650410+0000) 2019-10-13T11:22:15.652 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:15.650+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.651658+0000) 2019-10-13T11:22:15.652 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:15.649+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.650689+0000) 2019-10-13T11:22:15.653 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:15.649+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.650839+0000) 2019-10-13T11:22:15.653 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:15.649+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.651187+0000) 2019-10-13T11:22:15.653 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:15.650+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.651433+0000) 2019-10-13T11:22:15.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:15.652+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.653171+0000) 2019-10-13T11:22:15.657 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:15.655+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208000 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:15.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:15.695+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.695982+0000) 2019-10-13T11:22:15.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:15.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:15.695983+0000) 2019-10-13T11:22:18.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:18.602+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.603137+0000) 2019-10-13T11:22:18.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:18.936+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.936868+0000) 2019-10-13T11:22:20.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:20.657+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.658738+0000) 2019-10-13T11:22:20.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:20.657+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.658940+0000) 2019-10-13T11:22:20.660 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:20.657+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.659322+0000) 2019-10-13T11:22:20.661 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:20.658+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.659627+0000) 2019-10-13T11:22:20.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:20.658+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.659747+0000) 2019-10-13T11:22:20.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:20.658+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.660098+0000) 2019-10-13T11:22:20.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:20.660+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.660979+0000) 2019-10-13T11:22:20.662 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:20.661+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:20.661964+0000) 2019-10-13T11:22:20.668 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:20.666+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208030 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:21.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:21.554+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:21.555245+0000) 2019-10-13T11:22:23.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:23.041+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.041691+0000) 2019-10-13T11:22:23.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:23.044+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:23.045047+0000) 2019-10-13T11:22:24.867 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:22:24.868 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:24.872 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:22:24.872 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:25.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:25.348+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.348798+0000) 2019-10-13T11:22:25.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:25.669+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.669689+0000) 2019-10-13T11:22:25.671 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:25.669+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.669912+0000) 2019-10-13T11:22:25.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:25.669+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.670039+0000) 2019-10-13T11:22:25.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:25.669+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.670354+0000) 2019-10-13T11:22:25.672 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:25.670+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.670507+0000) 2019-10-13T11:22:25.673 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:25.671+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.671902+0000) 2019-10-13T11:22:25.673 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:25.670+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.670730+0000) 2019-10-13T11:22:25.674 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:25.673+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.673918+0000) 2019-10-13T11:22:25.679 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:25.678+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208062 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:25.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:25.695+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.696144+0000) 2019-10-13T11:22:25.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:25.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:25.696144+0000) 2019-10-13T11:22:28.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:28.602+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.603328+0000) 2019-10-13T11:22:28.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:28.936+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.937073+0000) 2019-10-13T11:22:30.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:30.679+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.680310+0000) 2019-10-13T11:22:30.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:30.680+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.680602+0000) 2019-10-13T11:22:30.682 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:30.680+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.681027+0000) 2019-10-13T11:22:30.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:30.680+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.681161+0000) 2019-10-13T11:22:30.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:30.681+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.681515+0000) 2019-10-13T11:22:30.683 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:30.681+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.681617+0000) 2019-10-13T11:22:30.683 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:30.681+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.682451+0000) 2019-10-13T11:22:30.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:30.682+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:30.683570+0000) 2019-10-13T11:22:30.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:30.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208092 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:31.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:31.555+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:31.555375+0000) 2019-10-13T11:22:33.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:33.041+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.041857+0000) 2019-10-13T11:22:33.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:33.044+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:33.045228+0000) 2019-10-13T11:22:35.351 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:35.350+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.351341+0000) 2019-10-13T11:22:35.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:35.691+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.691495+0000) 2019-10-13T11:22:35.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:35.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.691686+0000) 2019-10-13T11:22:35.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:35.691+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.692061+0000) 2019-10-13T11:22:35.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:35.691+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.692468+0000) 2019-10-13T11:22:35.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:35.691+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.692349+0000) 2019-10-13T11:22:35.694 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:35.692+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.692534+0000) 2019-10-13T11:22:35.694 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:35.692+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.692831+0000) 2019-10-13T11:22:35.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:35.694+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.694930+0000) 2019-10-13T11:22:35.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:35.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.696285+0000) 2019-10-13T11:22:35.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:35.695+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:35.696299+0000) 2019-10-13T11:22:35.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:35.700+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208122 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:38.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:38.603+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.603518+0000) 2019-10-13T11:22:38.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:38.937+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.937258+0000) 2019-10-13T11:22:40.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:40.700+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.702103+0000) 2019-10-13T11:22:40.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:40.700+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.702284+0000) 2019-10-13T11:22:40.704 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:40.703+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.703795+0000) 2019-10-13T11:22:40.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:40.701+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.702665+0000) 2019-10-13T11:22:40.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:40.701+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.702843+0000) 2019-10-13T11:22:40.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:40.701+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.703014+0000) 2019-10-13T11:22:40.705 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:40.702+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.703416+0000) 2019-10-13T11:22:40.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:40.704+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:40.704924+0000) 2019-10-13T11:22:40.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:40.709+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208152 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:41.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:41.554+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:41.555518+0000) 2019-10-13T11:22:43.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:43.040+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.041982+0000) 2019-10-13T11:22:43.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:43.045+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:43.045409+0000) 2019-10-13T11:22:45.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:45.353+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.353771+0000) 2019-10-13T11:22:45.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:45.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.696389+0000) 2019-10-13T11:22:45.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:45.695+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.696443+0000) 2019-10-13T11:22:45.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:45.711+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.712594+0000) 2019-10-13T11:22:45.714 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:45.712+0000 7f65be8ea700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.713483+0000) 2019-10-13T11:22:45.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:45.711+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.712891+0000) 2019-10-13T11:22:45.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:45.711+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.713166+0000) 2019-10-13T11:22:45.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:45.711+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.713339+0000) 2019-10-13T11:22:45.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:45.712+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.713537+0000) 2019-10-13T11:22:45.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:45.712+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.713664+0000) 2019-10-13T11:22:45.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:45.714+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:45.715723+0000) 2019-10-13T11:22:45.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:45.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:48.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:48.602+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.603646+0000) 2019-10-13T11:22:48.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:48.936+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.937374+0000) 2019-10-13T11:22:50.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:50.720+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.722198+0000) 2019-10-13T11:22:50.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:50.720+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.722353+0000) 2019-10-13T11:22:50.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:50.721+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.722451+0000) 2019-10-13T11:22:50.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:50.721+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.722805+0000) 2019-10-13T11:22:50.725 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:50.721+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.723290+0000) 2019-10-13T11:22:50.725 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:50.722+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.723528+0000) 2019-10-13T11:22:50.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:50.723+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.724051+0000) 2019-10-13T11:22:50.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:50.725+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:50.726288+0000) 2019-10-13T11:22:50.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:50.729+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208214 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:51.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:51.554+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:51.555690+0000) 2019-10-13T11:22:53.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:53.040+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.042170+0000) 2019-10-13T11:22:53.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:53.044+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:53.045637+0000) 2019-10-13T11:22:54.964 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:22:54.964 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:54.968 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:22:54.969 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:55.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:55.355+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.356232+0000) 2019-10-13T11:22:55.697 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:55.695+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.696574+0000) 2019-10-13T11:22:55.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:55.695+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.696590+0000) 2019-10-13T11:22:55.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:55.731+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.732913+0000) 2019-10-13T11:22:55.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:22:55.731+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.733178+0000) 2019-10-13T11:22:55.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:22:55.732+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.733439+0000) 2019-10-13T11:22:55.735 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:55.732+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.733430+0000) 2019-10-13T11:22:55.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:22:55.732+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.733583+0000) 2019-10-13T11:22:55.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:22:55.732+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.734674+0000) 2019-10-13T11:22:55.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:22:55.732+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.734038+0000) 2019-10-13T11:22:55.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:22:55.734+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:55.736720+0000) 2019-10-13T11:22:55.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:22:55.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208244 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:22:58.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:22:58.603+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.603852+0000) 2019-10-13T11:22:58.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:22:58.936+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.937521+0000) 2019-10-13T11:23:00.743 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:00.740+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.742080+0000) 2019-10-13T11:23:00.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:00.740+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.742251+0000) 2019-10-13T11:23:00.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:00.741+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.742493+0000) 2019-10-13T11:23:00.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:00.741+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.742720+0000) 2019-10-13T11:23:00.744 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:00.741+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.743042+0000) 2019-10-13T11:23:00.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:00.741+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.742999+0000) 2019-10-13T11:23:00.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:00.741+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.743224+0000) 2019-10-13T11:23:00.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:00.744+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:00.745972+0000) 2019-10-13T11:23:00.751 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:00.749+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208274 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:01.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:01.555+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:01.555858+0000) 2019-10-13T11:23:03.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:03.041+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.042321+0000) 2019-10-13T11:23:03.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:03.044+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:03.045809+0000) 2019-10-13T11:23:05.358 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:05.357+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.358628+0000) 2019-10-13T11:23:05.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:05.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.696770+0000) 2019-10-13T11:23:05.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:05.696+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.696776+0000) 2019-10-13T11:23:05.753 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:05.750+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.752284+0000) 2019-10-13T11:23:05.754 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:05.751+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.752692+0000) 2019-10-13T11:23:05.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:05.751+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.752835+0000) 2019-10-13T11:23:05.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:05.751+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.752956+0000) 2019-10-13T11:23:05.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:05.753+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.753755+0000) 2019-10-13T11:23:05.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:05.754+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.755272+0000) 2019-10-13T11:23:05.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:05.752+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.753414+0000) 2019-10-13T11:23:05.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:05.752+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:05.753629+0000) 2019-10-13T11:23:05.760 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:05.757+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:08.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:08.603+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.604024+0000) 2019-10-13T11:23:08.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:08.937+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.937699+0000) 2019-10-13T11:23:10.762 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:10.761+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.761803+0000) 2019-10-13T11:23:10.762 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:10.760+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.761290+0000) 2019-10-13T11:23:10.763 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:10.761+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.761521+0000) 2019-10-13T11:23:10.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:10.761+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.761836+0000) 2019-10-13T11:23:10.763 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:10.761+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.762081+0000) 2019-10-13T11:23:10.763 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:10.762+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.762498+0000) 2019-10-13T11:23:10.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:10.762+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.762747+0000) 2019-10-13T11:23:10.764 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:10.763+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:10.764379+0000) 2019-10-13T11:23:10.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:10.769+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:11.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:11.555+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:11.556001+0000) 2019-10-13T11:23:13.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:13.042+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.042501+0000) 2019-10-13T11:23:13.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:13.045+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:13.045995+0000) 2019-10-13T11:23:15.361 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:15.360+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.361037+0000) 2019-10-13T11:23:15.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:15.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.696984+0000) 2019-10-13T11:23:15.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:15.696+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.696975+0000) 2019-10-13T11:23:15.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:15.771+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.772044+0000) 2019-10-13T11:23:15.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:15.771+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.772164+0000) 2019-10-13T11:23:15.774 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:15.771+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.772362+0000) 2019-10-13T11:23:15.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:15.772+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.772504+0000) 2019-10-13T11:23:15.774 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:15.772+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.772694+0000) 2019-10-13T11:23:15.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:15.772+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.772842+0000) 2019-10-13T11:23:15.775 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:15.772+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.772936+0000) 2019-10-13T11:23:15.775 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:15.774+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:15.775713+0000) 2019-10-13T11:23:15.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:15.777+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208366 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:18.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:18.603+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.604212+0000) 2019-10-13T11:23:18.938 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:18.937+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.937891+0000) 2019-10-13T11:23:20.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:20.779+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.779587+0000) 2019-10-13T11:23:20.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:20.779+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.779700+0000) 2019-10-13T11:23:20.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:20.779+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.779918+0000) 2019-10-13T11:23:20.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:20.779+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.780219+0000) 2019-10-13T11:23:20.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:20.780+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.780405+0000) 2019-10-13T11:23:20.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:20.779+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.780688+0000) 2019-10-13T11:23:20.782 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:20.780+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.780542+0000) 2019-10-13T11:23:20.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:20.782+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:20.783084+0000) 2019-10-13T11:23:20.789 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:20.787+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:21.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:21.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:21.556142+0000) 2019-10-13T11:23:23.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:23.042+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.042683+0000) 2019-10-13T11:23:23.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:23.045+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:23.046200+0000) 2019-10-13T11:23:25.117 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:23:25.118 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:25.122 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:23:25.122 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:25.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:25.362+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.363568+0000) 2019-10-13T11:23:25.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:25.697+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.697127+0000) 2019-10-13T11:23:25.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:25.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.697122+0000) 2019-10-13T11:23:25.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:25.789+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.789538+0000) 2019-10-13T11:23:25.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:25.789+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.790019+0000) 2019-10-13T11:23:25.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:25.789+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.790114+0000) 2019-10-13T11:23:25.792 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:25.789+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.790340+0000) 2019-10-13T11:23:25.792 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:25.790+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.790456+0000) 2019-10-13T11:23:25.793 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:25.790+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.791284+0000) 2019-10-13T11:23:25.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:25.791+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.792386+0000) 2019-10-13T11:23:25.793 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:25.790+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:25.790734+0000) 2019-10-13T11:23:25.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:25.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:28.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:28.603+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.604414+0000) 2019-10-13T11:23:28.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:28.937+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.938064+0000) 2019-10-13T11:23:30.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:30.796+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.798100+0000) 2019-10-13T11:23:30.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:30.796+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.798282+0000) 2019-10-13T11:23:30.800 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:30.799+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.799804+0000) 2019-10-13T11:23:30.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:30.797+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.798634+0000) 2019-10-13T11:23:30.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:30.797+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.798760+0000) 2019-10-13T11:23:30.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:30.797+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.798977+0000) 2019-10-13T11:23:30.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:30.798+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.799454+0000) 2019-10-13T11:23:30.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:30.801+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:30.802335+0000) 2019-10-13T11:23:30.806 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:30.804+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208458 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:31.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:31.555+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:31.556280+0000) 2019-10-13T11:23:33.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:33.041+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.042859+0000) 2019-10-13T11:23:33.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:33.046+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:33.046382+0000) 2019-10-13T11:23:35.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:35.365+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.365780+0000) 2019-10-13T11:23:35.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:35.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.697251+0000) 2019-10-13T11:23:35.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:35.696+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.697237+0000) 2019-10-13T11:23:35.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:35.805+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.807277+0000) 2019-10-13T11:23:35.808 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:35.806+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.807507+0000) 2019-10-13T11:23:35.809 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:35.808+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.808763+0000) 2019-10-13T11:23:35.809 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:35.806+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.807795+0000) 2019-10-13T11:23:35.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:35.807+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.808486+0000) 2019-10-13T11:23:35.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:35.807+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.808860+0000) 2019-10-13T11:23:35.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:35.807+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.809086+0000) 2019-10-13T11:23:35.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:35.810+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:35.811057+0000) 2019-10-13T11:23:35.815 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:35.813+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208487 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:38.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:38.603+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.604545+0000) 2019-10-13T11:23:38.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:38.936+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.938272+0000) 2019-10-13T11:23:40.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:40.815+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.816692+0000) 2019-10-13T11:23:40.818 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:40.815+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.817026+0000) 2019-10-13T11:23:40.818 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:40.815+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.817191+0000) 2019-10-13T11:23:40.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:40.817+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.818078+0000) 2019-10-13T11:23:40.819 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:40.816+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.817562+0000) 2019-10-13T11:23:40.819 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:40.816+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.817798+0000) 2019-10-13T11:23:40.819 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:40.816+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.817869+0000) 2019-10-13T11:23:40.820 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:40.819+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:40.820102+0000) 2019-10-13T11:23:40.825 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:40.823+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:41.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:41.555+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:41.556454+0000) 2019-10-13T11:23:43.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:43.042+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.043040+0000) 2019-10-13T11:23:43.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:43.045+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:43.046547+0000) 2019-10-13T11:23:45.368 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:45.367+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.368182+0000) 2019-10-13T11:23:45.698 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:45.696+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.697426+0000) 2019-10-13T11:23:45.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:45.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.697446+0000) 2019-10-13T11:23:45.828 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:45.825+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.827571+0000) 2019-10-13T11:23:45.829 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:45.825+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.827336+0000) 2019-10-13T11:23:45.829 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:45.826+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.827666+0000) 2019-10-13T11:23:45.830 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:45.826+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.827817+0000) 2019-10-13T11:23:45.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:45.826+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.828110+0000) 2019-10-13T11:23:45.830 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:45.826+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.828184+0000) 2019-10-13T11:23:45.830 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:45.827+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.828500+0000) 2019-10-13T11:23:45.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:45.827+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:45.829540+0000) 2019-10-13T11:23:45.835 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:45.833+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208548 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:48.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:48.604+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.604734+0000) 2019-10-13T11:23:48.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:48.937+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.938434+0000) 2019-10-13T11:23:50.838 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:50.836+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.837728+0000) 2019-10-13T11:23:50.839 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:50.835+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.837255+0000) 2019-10-13T11:23:50.839 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:50.836+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.837716+0000) 2019-10-13T11:23:50.839 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:50.836+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.837802+0000) 2019-10-13T11:23:50.839 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:50.836+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.837917+0000) 2019-10-13T11:23:50.839 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:50.836+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.838134+0000) 2019-10-13T11:23:50.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:50.837+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.838692+0000) 2019-10-13T11:23:50.841 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:50.839+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:50.841196+0000) 2019-10-13T11:23:50.847 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:50.844+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:51.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:51.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:51.556566+0000) 2019-10-13T11:23:53.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:53.042+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.043248+0000) 2019-10-13T11:23:53.046 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:53.045+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:53.046701+0000) 2019-10-13T11:23:55.288 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:23:55.289 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:55.294 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:23:55.294 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:55.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:55.369+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.370640+0000) 2019-10-13T11:23:55.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:55.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.697648+0000) 2019-10-13T11:23:55.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:55.697+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.697643+0000) 2019-10-13T11:23:55.849 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:55.847+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.848054+0000) 2019-10-13T11:23:55.849 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:23:55.847+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.848232+0000) 2019-10-13T11:23:55.850 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:23:55.848+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.848515+0000) 2019-10-13T11:23:55.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:23:55.848+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.848755+0000) 2019-10-13T11:23:55.850 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:23:55.848+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.848955+0000) 2019-10-13T11:23:55.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:55.848+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.849078+0000) 2019-10-13T11:23:55.851 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:23:55.848+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.848909+0000) 2019-10-13T11:23:55.851 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:23:55.850+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:55.851571+0000) 2019-10-13T11:23:55.857 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:23:55.855+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208607 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:23:58.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:23:58.604+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.604902+0000) 2019-10-13T11:23:58.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:23:58.938+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.938589+0000) 2019-10-13T11:24:00.858 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:00.857+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.857747+0000) 2019-10-13T11:24:00.859 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:00.857+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.858228+0000) 2019-10-13T11:24:00.859 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:00.857+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.858275+0000) 2019-10-13T11:24:00.860 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:00.858+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.859645+0000) 2019-10-13T11:24:00.860 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:00.858+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.858869+0000) 2019-10-13T11:24:00.860 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:00.858+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.858693+0000) 2019-10-13T11:24:00.861 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:00.858+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.859256+0000) 2019-10-13T11:24:00.861 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:00.860+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:00.861385+0000) 2019-10-13T11:24:00.867 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:00.866+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:01.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:01.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:01.556733+0000) 2019-10-13T11:24:03.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:03.043+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.043450+0000) 2019-10-13T11:24:03.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:03.046+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:03.046885+0000) 2019-10-13T11:24:05.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:05.386+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.387008+0000) 2019-10-13T11:24:05.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:05.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.697841+0000) 2019-10-13T11:24:05.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:05.697+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.697839+0000) 2019-10-13T11:24:05.869 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:05.868+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.868407+0000) 2019-10-13T11:24:05.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:05.867+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.868152+0000) 2019-10-13T11:24:05.870 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:05.868+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.868571+0000) 2019-10-13T11:24:05.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:05.868+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.868818+0000) 2019-10-13T11:24:05.871 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:05.869+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.869892+0000) 2019-10-13T11:24:05.871 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:05.869+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.869694+0000) 2019-10-13T11:24:05.871 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:05.869+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.869913+0000) 2019-10-13T11:24:05.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:05.870+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:05.871725+0000) 2019-10-13T11:24:05.881 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:05.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208668 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:08.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:08.604+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.605090+0000) 2019-10-13T11:24:08.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:08.938+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.938720+0000) 2019-10-13T11:24:10.880 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:10.879+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.879769+0000) 2019-10-13T11:24:10.881 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:10.878+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.879219+0000) 2019-10-13T11:24:10.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:10.879+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.879449+0000) 2019-10-13T11:24:10.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:10.879+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.879509+0000) 2019-10-13T11:24:10.881 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:10.879+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.879665+0000) 2019-10-13T11:24:10.881 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:10.879+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.879930+0000) 2019-10-13T11:24:10.882 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:10.879+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.880266+0000) 2019-10-13T11:24:10.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:10.881+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:10.882510+0000) 2019-10-13T11:24:10.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:10.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208698 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:11.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:11.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:11.556902+0000) 2019-10-13T11:24:13.044 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:13.042+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.043601+0000) 2019-10-13T11:24:13.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:13.046+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:13.047088+0000) 2019-10-13T11:24:15.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:15.388+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.389304+0000) 2019-10-13T11:24:15.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:15.696+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.697998+0000) 2019-10-13T11:24:15.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:15.696+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.698030+0000) 2019-10-13T11:24:15.890 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:15.887+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.889129+0000) 2019-10-13T11:24:15.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:15.889+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.890052+0000) 2019-10-13T11:24:15.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:15.888+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.889743+0000) 2019-10-13T11:24:15.891 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:15.888+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.889779+0000) 2019-10-13T11:24:15.892 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:15.888+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.890058+0000) 2019-10-13T11:24:15.892 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:15.888+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.890171+0000) 2019-10-13T11:24:15.892 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:15.889+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.890666+0000) 2019-10-13T11:24:15.893 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:15.892+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:15.893288+0000) 2019-10-13T11:24:15.898 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:15.896+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:18.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:18.604+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.605245+0000) 2019-10-13T11:24:18.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:18.937+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.938908+0000) 2019-10-13T11:24:20.900 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:20.898+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.899417+0000) 2019-10-13T11:24:20.900 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:20.898+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.899758+0000) 2019-10-13T11:24:20.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:20.899+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.900231+0000) 2019-10-13T11:24:20.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:20.898+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.899871+0000) 2019-10-13T11:24:20.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:20.898+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.900182+0000) 2019-10-13T11:24:20.902 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:20.898+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.900307+0000) 2019-10-13T11:24:20.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:20.899+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.900459+0000) 2019-10-13T11:24:20.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:20.902+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:20.902827+0000) 2019-10-13T11:24:20.908 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:20.906+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:21.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:21.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:21.557066+0000) 2019-10-13T11:24:23.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:23.042+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.043778+0000) 2019-10-13T11:24:23.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:23.047+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:23.047291+0000) 2019-10-13T11:24:25.392 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:25.391+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.391551+0000) 2019-10-13T11:24:25.451 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:24:25.451 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:25.456 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:24:25.456 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:25.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:25.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.698194+0000) 2019-10-13T11:24:25.699 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:25.697+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.698194+0000) 2019-10-13T11:24:25.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:25.907+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.908950+0000) 2019-10-13T11:24:25.910 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:25.907+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.909183+0000) 2019-10-13T11:24:25.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:25.907+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.909359+0000) 2019-10-13T11:24:25.911 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:25.910+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.910863+0000) 2019-10-13T11:24:25.911 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:25.908+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.909592+0000) 2019-10-13T11:24:25.911 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:25.908+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.909816+0000) 2019-10-13T11:24:25.912 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:25.908+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.909955+0000) 2019-10-13T11:24:25.913 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:25.912+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:25.913718+0000) 2019-10-13T11:24:25.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:25.913+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208788 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:28.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:28.604+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.605445+0000) 2019-10-13T11:24:28.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:28.938+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.939090+0000) 2019-10-13T11:24:30.918 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:30.915+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.917209+0000) 2019-10-13T11:24:30.918 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:30.915+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.917309+0000) 2019-10-13T11:24:30.919 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:30.916+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.917535+0000) 2019-10-13T11:24:30.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:30.918+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.919298+0000) 2019-10-13T11:24:30.920 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:30.916+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.917885+0000) 2019-10-13T11:24:30.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:30.916+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.918064+0000) 2019-10-13T11:24:30.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:30.916+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.918355+0000) 2019-10-13T11:24:30.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:30.919+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:30.919794+0000) 2019-10-13T11:24:30.930 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:30.924+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208818 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:31.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:31.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:31.557266+0000) 2019-10-13T11:24:33.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:33.043+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:33.043958+0000) 2019-10-13T11:24:33.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:33.046+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:33.047470+0000) 2019-10-13T11:24:35.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:35.392+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.393937+0000) 2019-10-13T11:24:35.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:35.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.698433+0000) 2019-10-13T11:24:35.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:35.697+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.698442+0000) 2019-10-13T11:24:35.929 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:35.926+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.927909+0000) 2019-10-13T11:24:35.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:35.926+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.928121+0000) 2019-10-13T11:24:35.929 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:35.926+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.928141+0000) 2019-10-13T11:24:35.930 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:35.926+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.928362+0000) 2019-10-13T11:24:35.930 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:35.927+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.928476+0000) 2019-10-13T11:24:35.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:35.927+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.928616+0000) 2019-10-13T11:24:35.930 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:35.928+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.929776+0000) 2019-10-13T11:24:35.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:35.929+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:35.931683+0000) 2019-10-13T11:24:35.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:35.935+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:38.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:38.605+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:38.605644+0000) 2019-10-13T11:24:38.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:38.938+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:38.939233+0000) 2019-10-13T11:24:40.939 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:40.937+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.938352+0000) 2019-10-13T11:24:40.940 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:40.937+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.938155+0000) 2019-10-13T11:24:40.940 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:40.938+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.938535+0000) 2019-10-13T11:24:40.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:40.938+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.938640+0000) 2019-10-13T11:24:40.940 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:40.938+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.939018+0000) 2019-10-13T11:24:40.940 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:40.938+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.939113+0000) 2019-10-13T11:24:40.941 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:40.937+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.939448+0000) 2019-10-13T11:24:40.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:40.940+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:40.942590+0000) 2019-10-13T11:24:40.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:40.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208877 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:41.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:41.557+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:41.557461+0000) 2019-10-13T11:24:43.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:43.043+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:43.044143+0000) 2019-10-13T11:24:43.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:43.046+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:43.047662+0000) 2019-10-13T11:24:45.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:45.395+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.396294+0000) 2019-10-13T11:24:45.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:45.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.698627+0000) 2019-10-13T11:24:45.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:45.698+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.698660+0000) 2019-10-13T11:24:45.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:45.948+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.948497+0000) 2019-10-13T11:24:45.950 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:45.948+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.948618+0000) 2019-10-13T11:24:45.950 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:45.948+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.948816+0000) 2019-10-13T11:24:45.950 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:45.948+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.949023+0000) 2019-10-13T11:24:45.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:45.948+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.949179+0000) 2019-10-13T11:24:45.951 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:45.948+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.949977+0000) 2019-10-13T11:24:45.951 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:45.949+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.951062+0000) 2019-10-13T11:24:45.951 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:45.949+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:45.949496+0000) 2019-10-13T11:24:45.957 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:45.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208910 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:48.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:48.605+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:48.605832+0000) 2019-10-13T11:24:48.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:48.939+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:48.939426+0000) 2019-10-13T11:24:50.960 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:50.958+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.959116+0000) 2019-10-13T11:24:50.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:50.958+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.959023+0000) 2019-10-13T11:24:50.962 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:50.958+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.959245+0000) 2019-10-13T11:24:50.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:50.958+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.959373+0000) 2019-10-13T11:24:50.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:50.959+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.959577+0000) 2019-10-13T11:24:50.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:50.959+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.959686+0000) 2019-10-13T11:24:50.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:50.959+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.959849+0000) 2019-10-13T11:24:50.963 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:50.961+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:50.962342+0000) 2019-10-13T11:24:50.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:50.967+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208940 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:51.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:51.557+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:51.557632+0000) 2019-10-13T11:24:53.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:53.044+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:53.044319+0000) 2019-10-13T11:24:53.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:53.047+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:53.047846+0000) 2019-10-13T11:24:55.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:55.398+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.398797+0000) 2019-10-13T11:24:55.606 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:24:55.606 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:55.610 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:24:55.610 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:55.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:55.698+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.698801+0000) 2019-10-13T11:24:55.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:55.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.698824+0000) 2019-10-13T11:24:55.970 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:24:55.969+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.969751+0000) 2019-10-13T11:24:55.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:24:55.969+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.969549+0000) 2019-10-13T11:24:55.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:55.969+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.969682+0000) 2019-10-13T11:24:55.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:24:55.969+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.970032+0000) 2019-10-13T11:24:55.971 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:24:55.969+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.970249+0000) 2019-10-13T11:24:55.972 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:24:55.970+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.970427+0000) 2019-10-13T11:24:55.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:24:55.971+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.972145+0000) 2019-10-13T11:24:55.972 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:55.970+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:55.970457+0000) 2019-10-13T11:24:55.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:24:55.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 208971 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:24:58.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:24:58.605+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:58.606004+0000) 2019-10-13T11:24:58.940 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:24:58.938+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:58.939620+0000) 2019-10-13T11:25:00.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:00.978+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.979669+0000) 2019-10-13T11:25:00.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:00.978+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.979840+0000) 2019-10-13T11:25:00.981 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:00.980+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.981475+0000) 2019-10-13T11:25:00.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:00.978+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.980114+0000) 2019-10-13T11:25:00.982 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:00.979+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.980495+0000) 2019-10-13T11:25:00.982 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:00.979+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.980983+0000) 2019-10-13T11:25:00.982 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:00.980+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.981626+0000) 2019-10-13T11:25:00.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:00.983+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:00.983938+0000) 2019-10-13T11:25:00.989 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:00.987+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:01.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:01.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:01.557835+0000) 2019-10-13T11:25:03.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:03.043+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:03.044469+0000) 2019-10-13T11:25:03.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:03.047+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:03.048051+0000) 2019-10-13T11:25:05.401 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:05.400+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.401168+0000) 2019-10-13T11:25:05.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:05.697+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.698990+0000) 2019-10-13T11:25:05.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:05.697+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.698985+0000) 2019-10-13T11:25:05.991 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:05.990+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.990847+0000) 2019-10-13T11:25:05.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:05.989+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.990596+0000) 2019-10-13T11:25:05.992 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:05.989+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.990660+0000) 2019-10-13T11:25:05.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:05.989+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.990852+0000) 2019-10-13T11:25:05.992 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:05.989+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.991031+0000) 2019-10-13T11:25:05.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:05.989+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.991103+0000) 2019-10-13T11:25:05.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:05.989+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.991294+0000) 2019-10-13T11:25:05.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:05.992+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:05.993120+0000) 2019-10-13T11:25:05.999 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:05.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:08.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:08.605+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:08.606181+0000) 2019-10-13T11:25:08.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:08.939+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:08.939810+0000) 2019-10-13T11:25:11.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:11.000+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.001126+0000) 2019-10-13T11:25:11.002 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:10.999+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.000662+0000) 2019-10-13T11:25:11.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:10.999+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.001285+0000) 2019-10-13T11:25:11.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:11.000+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.001415+0000) 2019-10-13T11:25:11.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:11.000+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.001644+0000) 2019-10-13T11:25:11.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:11.000+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.001863+0000) 2019-10-13T11:25:11.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:11.000+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.002009+0000) 2019-10-13T11:25:11.004 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:11.003+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.004723+0000) 2019-10-13T11:25:11.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:11.007+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:11.558 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:11.556+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.557946+0000) 2019-10-13T11:25:13.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:13.043+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:13.044664+0000) 2019-10-13T11:25:13.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:13.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:13.048256+0000) 2019-10-13T11:25:15.403 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:15.403+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:15.403406+0000) 2019-10-13T11:25:15.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:15.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:15.699190+0000) 2019-10-13T11:25:15.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:15.698+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:15.699189+0000) 2019-10-13T11:25:16.011 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:16.009+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.010487+0000) 2019-10-13T11:25:16.012 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:16.009+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.010580+0000) 2019-10-13T11:25:16.012 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:16.009+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.010748+0000) 2019-10-13T11:25:16.013 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:16.011+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.012177+0000) 2019-10-13T11:25:16.013 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:16.009+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.011310+0000) 2019-10-13T11:25:16.013 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:16.010+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.011799+0000) 2019-10-13T11:25:16.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:16.010+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.012118+0000) 2019-10-13T11:25:16.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:16.013+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:16.014605+0000) 2019-10-13T11:25:16.019 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:16.017+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209095 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:18.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:18.605+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:18.606386+0000) 2019-10-13T11:25:18.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:18.939+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:18.939971+0000) 2019-10-13T11:25:21.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:21.019+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.020474+0000) 2019-10-13T11:25:21.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:21.019+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.020779+0000) 2019-10-13T11:25:21.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:21.019+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.020978+0000) 2019-10-13T11:25:21.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:21.021+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.022704+0000) 2019-10-13T11:25:21.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:21.020+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.021452+0000) 2019-10-13T11:25:21.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:21.020+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.021587+0000) 2019-10-13T11:25:21.025 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:21.020+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.021818+0000) 2019-10-13T11:25:21.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:21.023+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.024176+0000) 2019-10-13T11:25:21.030 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:21.027+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:21.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:21.557+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.558076+0000) 2019-10-13T11:25:23.045 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:23.044+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:23.044843+0000) 2019-10-13T11:25:23.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:23.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:23.048439+0000) 2019-10-13T11:25:25.406 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:25.404+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:25.405778+0000) 2019-10-13T11:25:25.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:25.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:25.699387+0000) 2019-10-13T11:25:25.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:25.698+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:25.699367+0000) 2019-10-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:25:25.777 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:25:25.781 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:26.032 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:26.030+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.030592+0000) 2019-10-13T11:25:26.033 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:26.030+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.030767+0000) 2019-10-13T11:25:26.033 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:26.030+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.031193+0000) 2019-10-13T11:25:26.033 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:26.031+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.031469+0000) 2019-10-13T11:25:26.034 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:26.031+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.031585+0000) 2019-10-13T11:25:26.034 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:26.031+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.031713+0000) 2019-10-13T11:25:26.035 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:26.030+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.032513+0000) 2019-10-13T11:25:26.035 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:26.031+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:26.033556+0000) 2019-10-13T11:25:26.039 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:26.037+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:28.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:28.606+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:28.606598+0000) 2019-10-13T11:25:28.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:28.939+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:28.940149+0000) 2019-10-13T11:25:31.041 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:31.039+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.040317+0000) 2019-10-13T11:25:31.041 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:31.040+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.040446+0000) 2019-10-13T11:25:31.042 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:31.040+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.040838+0000) 2019-10-13T11:25:31.042 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:31.040+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.041779+0000) 2019-10-13T11:25:31.043 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:31.040+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.040926+0000) 2019-10-13T11:25:31.043 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:31.040+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.041323+0000) 2019-10-13T11:25:31.043 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:31.041+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.041523+0000) 2019-10-13T11:25:31.043 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:31.041+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.043246+0000) 2019-10-13T11:25:31.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:31.048+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209188 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:31.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:31.557+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.558230+0000) 2019-10-13T11:25:33.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:33.044+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:33.045010+0000) 2019-10-13T11:25:33.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:33.047+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:33.048642+0000) 2019-10-13T11:25:35.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:35.407+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:35.408229+0000) 2019-10-13T11:25:35.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:35.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:35.699561+0000) 2019-10-13T11:25:35.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:35.699+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:35.699559+0000) 2019-10-13T11:25:36.051 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:36.049+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.050232+0000) 2019-10-13T11:25:36.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:36.050+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.050891+0000) 2019-10-13T11:25:36.052 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:36.050+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.051067+0000) 2019-10-13T11:25:36.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:36.050+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.051302+0000) 2019-10-13T11:25:36.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:36.051+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.051407+0000) 2019-10-13T11:25:36.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:36.050+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.051818+0000) 2019-10-13T11:25:36.053 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:36.051+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.051558+0000) 2019-10-13T11:25:36.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:36.052+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:36.053468+0000) 2019-10-13T11:25:36.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:36.059+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209219 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:38.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:38.606+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:38.606816+0000) 2019-10-13T11:25:38.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:38.940+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:38.940343+0000) 2019-10-13T11:25:41.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:41.060+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.061001+0000) 2019-10-13T11:25:41.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:41.061+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.061816+0000) 2019-10-13T11:25:41.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:41.061+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.061628+0000) 2019-10-13T11:25:41.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:41.061+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.061773+0000) 2019-10-13T11:25:41.064 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:41.061+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.062091+0000) 2019-10-13T11:25:41.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:41.061+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.062381+0000) 2019-10-13T11:25:41.064 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:41.062+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.062494+0000) 2019-10-13T11:25:41.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:41.063+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.064592+0000) 2019-10-13T11:25:41.070 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:41.069+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209249 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:41.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:41.558+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.558421+0000) 2019-10-13T11:25:43.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:43.045+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:43.045170+0000) 2019-10-13T11:25:43.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:43.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:43.048802+0000) 2019-10-13T11:25:45.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:45.410+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:45.410673+0000) 2019-10-13T11:25:45.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:45.698+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:45.699735+0000) 2019-10-13T11:25:45.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:45.698+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:45.699725+0000) 2019-10-13T11:25:46.072 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:46.070+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.071557+0000) 2019-10-13T11:25:46.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:46.070+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.071673+0000) 2019-10-13T11:25:46.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:46.071+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.072666+0000) 2019-10-13T11:25:46.073 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:46.070+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.072105+0000) 2019-10-13T11:25:46.074 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:46.070+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.072186+0000) 2019-10-13T11:25:46.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:46.070+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.072353+0000) 2019-10-13T11:25:46.075 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:46.071+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.072704+0000) 2019-10-13T11:25:46.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:46.074+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:46.075644+0000) 2019-10-13T11:25:46.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:46.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209282 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:48.607 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:48.605+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:48.606985+0000) 2019-10-13T11:25:48.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:48.939+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:48.940505+0000) 2019-10-13T11:25:51.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:51.079+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.080707+0000) 2019-10-13T11:25:51.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:51.079+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.080844+0000) 2019-10-13T11:25:51.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:51.079+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.081159+0000) 2019-10-13T11:25:51.082 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:51.079+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.080981+0000) 2019-10-13T11:25:51.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:51.082+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.082787+0000) 2019-10-13T11:25:51.083 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:51.080+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.081446+0000) 2019-10-13T11:25:51.083 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:51.080+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.081484+0000) 2019-10-13T11:25:51.084 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:51.083+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.084456+0000) 2019-10-13T11:25:51.090 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:51.087+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209312 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:51.559 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:51.557+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.558561+0000) 2019-10-13T11:25:53.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:53.044+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:53.045350+0000) 2019-10-13T11:25:53.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:53.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:53.048969+0000) 2019-10-13T11:25:55.413 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:55.412+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:55.413072+0000) 2019-10-13T11:25:55.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:55.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:55.699900+0000) 2019-10-13T11:25:55.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:55.699+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:55.699900+0000) 2019-10-13T11:25:55.898 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:25:55.898 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:55.902 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:25:55.902 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:56.093 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:25:56.089+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.091376+0000) 2019-10-13T11:25:56.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:25:56.090+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.091499+0000) 2019-10-13T11:25:56.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:25:56.090+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.091785+0000) 2019-10-13T11:25:56.095 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:25:56.090+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.092019+0000) 2019-10-13T11:25:56.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:56.090+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.092171+0000) 2019-10-13T11:25:56.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:25:56.091+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.092524+0000) 2019-10-13T11:25:56.097 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:25:56.093+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.093736+0000) 2019-10-13T11:25:56.097 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:56.091+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:56.092392+0000) 2019-10-13T11:25:56.100 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:25:56.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209343 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:25:58.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:25:58.606+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:58.607162+0000) 2019-10-13T11:25:58.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:25:58.939+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:58.940663+0000) 2019-10-13T11:26:01.102 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:01.099+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.101139+0000) 2019-10-13T11:26:01.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:01.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.101416+0000) 2019-10-13T11:26:01.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:01.101+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.102259+0000) 2019-10-13T11:26:01.103 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:01.100+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.101860+0000) 2019-10-13T11:26:01.104 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:01.100+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.102183+0000) 2019-10-13T11:26:01.104 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:01.100+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.102229+0000) 2019-10-13T11:26:01.104 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:01.101+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.102589+0000) 2019-10-13T11:26:01.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:01.104+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.105351+0000) 2019-10-13T11:26:01.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:01.109+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209373 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:01.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:01.558+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.558748+0000) 2019-10-13T11:26:03.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:03.044+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:03.045558+0000) 2019-10-13T11:26:03.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:03.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:03.049153+0000) 2019-10-13T11:26:05.415 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:05.415+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.415478+0000) 2019-10-13T11:26:05.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:05.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.700116+0000) 2019-10-13T11:26:05.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:05.699+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.700115+0000) 2019-10-13T11:26:06.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:06.112+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.112827+0000) 2019-10-13T11:26:06.113 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:06.111+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.112557+0000) 2019-10-13T11:26:06.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:06.111+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.112845+0000) 2019-10-13T11:26:06.114 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:06.111+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.113120+0000) 2019-10-13T11:26:06.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:06.111+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.113388+0000) 2019-10-13T11:26:06.115 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:06.112+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.113808+0000) 2019-10-13T11:26:06.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:06.112+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.114168+0000) 2019-10-13T11:26:06.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:06.115+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:06.116195+0000) 2019-10-13T11:26:06.123 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:06.120+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:08.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:08.606+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:08.607370+0000) 2019-10-13T11:26:08.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:08.940+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:08.940807+0000) 2019-10-13T11:26:11.124 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:11.123+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.124161+0000) 2019-10-13T11:26:11.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:11.123+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.123685+0000) 2019-10-13T11:26:11.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:11.123+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.123899+0000) 2019-10-13T11:26:11.126 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:11.123+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.124054+0000) 2019-10-13T11:26:11.126 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:11.123+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.124378+0000) 2019-10-13T11:26:11.126 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:11.124+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.124463+0000) 2019-10-13T11:26:11.126 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:11.124+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.124782+0000) 2019-10-13T11:26:11.127 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:11.126+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.127460+0000) 2019-10-13T11:26:11.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:11.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:11.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:11.558+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.558935+0000) 2019-10-13T11:26:13.046 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:13.045+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:13.045742+0000) 2019-10-13T11:26:13.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:13.049+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:13.049327+0000) 2019-10-13T11:26:15.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:15.416+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.417811+0000) 2019-10-13T11:26:15.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:15.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.700332+0000) 2019-10-13T11:26:15.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:15.699+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.700325+0000) 2019-10-13T11:26:16.135 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:16.132+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.134590+0000) 2019-10-13T11:26:16.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:16.133+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.134277+0000) 2019-10-13T11:26:16.136 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:16.133+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.134154+0000) 2019-10-13T11:26:16.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:16.134+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.134427+0000) 2019-10-13T11:26:16.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:16.134+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.134576+0000) 2019-10-13T11:26:16.136 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:16.134+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.134982+0000) 2019-10-13T11:26:16.137 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:16.134+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.135139+0000) 2019-10-13T11:26:16.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:16.136+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:16.137965+0000) 2019-10-13T11:26:16.143 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:16.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:18.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:18.607+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:18.607556+0000) 2019-10-13T11:26:18.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:18.940+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:18.940984+0000) 2019-10-13T11:26:21.145 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:21.143+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.144133+0000) 2019-10-13T11:26:21.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:21.143+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.144375+0000) 2019-10-13T11:26:21.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:21.144+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.144659+0000) 2019-10-13T11:26:21.146 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:21.143+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.145705+0000) 2019-10-13T11:26:21.146 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:21.144+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.144848+0000) 2019-10-13T11:26:21.147 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:21.144+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.144977+0000) 2019-10-13T11:26:21.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:21.144+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.145284+0000) 2019-10-13T11:26:21.148 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:21.146+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.148673+0000) 2019-10-13T11:26:21.154 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:21.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209497 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:21.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:21.558+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.559100+0000) 2019-10-13T11:26:23.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:23.045+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:23.045900+0000) 2019-10-13T11:26:23.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:23.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:23.049512+0000) 2019-10-13T11:26:25.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:25.419+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.420107+0000) 2019-10-13T11:26:25.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:25.700+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.700511+0000) 2019-10-13T11:26:25.701 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:25.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.700567+0000) 2019-10-13T11:26:26.044 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:26:26.044 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:26.049 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:26:26.049 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:26.156 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:26.154+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.155254+0000) 2019-10-13T11:26:26.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:26.155+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.155826+0000) 2019-10-13T11:26:26.157 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:26.155+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.155908+0000) 2019-10-13T11:26:26.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:26.155+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.156161+0000) 2019-10-13T11:26:26.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:26.154+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.156644+0000) 2019-10-13T11:26:26.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:26.156+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.156456+0000) 2019-10-13T11:26:26.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:26.156+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.156527+0000) 2019-10-13T11:26:26.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:26.158+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:26.159319+0000) 2019-10-13T11:26:26.163 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:26.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209530 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:28.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:28.607+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:28.607698+0000) 2019-10-13T11:26:28.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:28.941+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:28.941162+0000) 2019-10-13T11:26:31.165 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:31.163+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.164461+0000) 2019-10-13T11:26:31.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:31.163+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.164511+0000) 2019-10-13T11:26:31.166 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:31.163+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.164738+0000) 2019-10-13T11:26:31.166 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:31.164+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.165628+0000) 2019-10-13T11:26:31.167 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:31.163+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.165124+0000) 2019-10-13T11:26:31.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:31.164+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.165514+0000) 2019-10-13T11:26:31.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:31.164+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.165893+0000) 2019-10-13T11:26:31.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:31.166+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.167569+0000) 2019-10-13T11:26:31.174 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:31.172+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:31.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:31.558+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.559287+0000) 2019-10-13T11:26:33.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:33.045+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:33.046068+0000) 2019-10-13T11:26:33.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:33.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:33.049690+0000) 2019-10-13T11:26:35.423 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:35.421+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.422704+0000) 2019-10-13T11:26:35.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:35.699+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.700696+0000) 2019-10-13T11:26:35.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:35.699+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.700717+0000) 2019-10-13T11:26:36.177 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:36.173+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.175383+0000) 2019-10-13T11:26:36.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:36.174+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.175602+0000) 2019-10-13T11:26:36.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:36.174+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.175684+0000) 2019-10-13T11:26:36.177 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:36.174+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.175793+0000) 2019-10-13T11:26:36.177 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:36.174+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.175970+0000) 2019-10-13T11:26:36.178 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:36.174+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.176326+0000) 2019-10-13T11:26:36.178 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:36.176+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.176733+0000) 2019-10-13T11:26:36.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:36.177+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:36.177767+0000) 2019-10-13T11:26:36.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:36.182+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209591 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:38.608 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:38.606+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:38.607866+0000) 2019-10-13T11:26:38.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:38.939+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:38.941355+0000) 2019-10-13T11:26:41.186 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:41.185+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.186138+0000) 2019-10-13T11:26:41.187 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:41.184+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.185909+0000) 2019-10-13T11:26:41.187 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:41.184+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.186025+0000) 2019-10-13T11:26:41.187 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:41.185+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.186415+0000) 2019-10-13T11:26:41.187 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:41.184+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.186299+0000) 2019-10-13T11:26:41.188 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:41.185+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.187069+0000) 2019-10-13T11:26:41.188 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:41.185+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.187356+0000) 2019-10-13T11:26:41.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:41.188+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.189696+0000) 2019-10-13T11:26:41.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:41.193+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:41.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:41.558+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.559465+0000) 2019-10-13T11:26:43.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:43.045+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:43.046263+0000) 2019-10-13T11:26:43.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:43.049+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:43.049907+0000) 2019-10-13T11:26:45.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:45.424+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.425053+0000) 2019-10-13T11:26:45.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:45.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.700927+0000) 2019-10-13T11:26:45.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:45.700+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.700926+0000) 2019-10-13T11:26:46.197 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:46.194+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.196351+0000) 2019-10-13T11:26:46.198 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:46.196+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.197111+0000) 2019-10-13T11:26:46.199 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:46.195+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.197034+0000) 2019-10-13T11:26:46.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:46.195+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.197284+0000) 2019-10-13T11:26:46.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:46.195+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.197358+0000) 2019-10-13T11:26:46.199 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:46.196+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.197546+0000) 2019-10-13T11:26:46.199 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:46.196+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.197711+0000) 2019-10-13T11:26:46.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:46.198+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:46.199621+0000) 2019-10-13T11:26:46.205 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:46.203+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209654 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:48.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:48.607+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:48.608062+0000) 2019-10-13T11:26:48.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:48.940+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:48.941529+0000) 2019-10-13T11:26:51.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:51.205+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.207246+0000) 2019-10-13T11:26:51.209 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:51.205+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.207355+0000) 2019-10-13T11:26:51.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:51.206+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.207539+0000) 2019-10-13T11:26:51.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:51.206+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.207936+0000) 2019-10-13T11:26:51.210 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:51.206+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.208134+0000) 2019-10-13T11:26:51.211 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:51.207+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.208395+0000) 2019-10-13T11:26:51.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:51.207+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.208452+0000) 2019-10-13T11:26:51.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:51.210+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.211561+0000) 2019-10-13T11:26:51.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:51.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:51.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:51.559+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.559667+0000) 2019-10-13T11:26:53.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:53.045+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:53.046450+0000) 2019-10-13T11:26:53.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:53.049+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:53.050074+0000) 2019-10-13T11:26:55.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:55.427+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.427333+0000) 2019-10-13T11:26:55.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:55.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.701125+0000) 2019-10-13T11:26:55.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:55.700+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.701126+0000) 2019-10-13T11:26:56.147 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:26:56.147 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:56.151 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:26:56.151 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:56.218 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:56.215+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.216892+0000) 2019-10-13T11:26:56.218 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:26:56.215+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.217135+0000) 2019-10-13T11:26:56.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:26:56.216+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.217465+0000) 2019-10-13T11:26:56.219 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:26:56.217+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.218168+0000) 2019-10-13T11:26:56.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:56.216+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.217792+0000) 2019-10-13T11:26:56.220 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:26:56.216+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.217906+0000) 2019-10-13T11:26:56.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:26:56.216+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.218314+0000) 2019-10-13T11:26:56.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:26:56.219+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:56.220570+0000) 2019-10-13T11:26:56.225 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:26:56.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:26:58.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:26:58.607+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:58.608249+0000) 2019-10-13T11:26:58.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:26:58.941+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:58.941680+0000) 2019-10-13T11:27:01.227 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:01.225+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.226357+0000) 2019-10-13T11:27:01.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:01.226+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.226799+0000) 2019-10-13T11:27:01.228 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:01.227+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.227926+0000) 2019-10-13T11:27:01.228 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:01.226+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.227124+0000) 2019-10-13T11:27:01.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:01.227+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.227618+0000) 2019-10-13T11:27:01.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:01.227+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.227871+0000) 2019-10-13T11:27:01.229 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:01.227+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.228213+0000) 2019-10-13T11:27:01.230 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:01.229+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.230234+0000) 2019-10-13T11:27:01.236 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:01.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209745 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:01.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:01.559+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.559814+0000) 2019-10-13T11:27:03.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:03.046+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:03.046616+0000) 2019-10-13T11:27:03.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:03.050+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:03.050255+0000) 2019-10-13T11:27:05.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:05.428+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.429763+0000) 2019-10-13T11:27:05.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:05.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.701343+0000) 2019-10-13T11:27:05.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:05.700+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.701347+0000) 2019-10-13T11:27:06.238 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:06.235+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.237614+0000) 2019-10-13T11:27:06.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:06.236+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.237318+0000) 2019-10-13T11:27:06.239 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:06.237+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.237549+0000) 2019-10-13T11:27:06.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:06.237+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.237994+0000) 2019-10-13T11:27:06.239 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:06.237+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.238259+0000) 2019-10-13T11:27:06.240 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:06.238+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.238417+0000) 2019-10-13T11:27:06.240 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:06.238+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.238583+0000) 2019-10-13T11:27:06.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:06.239+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:06.241154+0000) 2019-10-13T11:27:06.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:06.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209778 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:08.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:08.608+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:08.608455+0000) 2019-10-13T11:27:08.942 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:08.941+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:08.941912+0000) 2019-10-13T11:27:11.248 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:11.247+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.247611+0000) 2019-10-13T11:27:11.249 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:11.247+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.247869+0000) 2019-10-13T11:27:11.250 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:11.246+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.248327+0000) 2019-10-13T11:27:11.250 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:11.247+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.248308+0000) 2019-10-13T11:27:11.250 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:11.248+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.248833+0000) 2019-10-13T11:27:11.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:11.248+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.249013+0000) 2019-10-13T11:27:11.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:11.248+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.249156+0000) 2019-10-13T11:27:11.251 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:11.249+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.251550+0000) 2019-10-13T11:27:11.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:11.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209808 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:11.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:11.559+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.560020+0000) 2019-10-13T11:27:13.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:13.046+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:13.046793+0000) 2019-10-13T11:27:13.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:13.048+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:13.050450+0000) 2019-10-13T11:27:15.432 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:15.431+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.432123+0000) 2019-10-13T11:27:15.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:15.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.701566+0000) 2019-10-13T11:27:15.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:15.700+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.701566+0000) 2019-10-13T11:27:16.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:16.256+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.257754+0000) 2019-10-13T11:27:16.259 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:16.256+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.258144+0000) 2019-10-13T11:27:16.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:16.257+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.258437+0000) 2019-10-13T11:27:16.259 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:16.257+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.258587+0000) 2019-10-13T11:27:16.260 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:16.257+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.259552+0000) 2019-10-13T11:27:16.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:16.257+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.259111+0000) 2019-10-13T11:27:16.261 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:16.257+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.259263+0000) 2019-10-13T11:27:16.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:16.259+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:16.261382+0000) 2019-10-13T11:27:16.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:16.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:18.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:18.607+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:18.608646+0000) 2019-10-13T11:27:18.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:18.941+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:18.942075+0000) 2019-10-13T11:27:21.269 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:21.266+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.268069+0000) 2019-10-13T11:27:21.269 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:21.267+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.268390+0000) 2019-10-13T11:27:21.270 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:21.268+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.269409+0000) 2019-10-13T11:27:21.270 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:21.267+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.268637+0000) 2019-10-13T11:27:21.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:21.267+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.269020+0000) 2019-10-13T11:27:21.271 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:21.267+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.269177+0000) 2019-10-13T11:27:21.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:21.267+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.269183+0000) 2019-10-13T11:27:21.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:21.270+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.270928+0000) 2019-10-13T11:27:21.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:21.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:21.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:21.559+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.560193+0000) 2019-10-13T11:27:23.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:23.046+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:23.046980+0000) 2019-10-13T11:27:23.050 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:23.049+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:23.050614+0000) 2019-10-13T11:27:25.435 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:25.433+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.434535+0000) 2019-10-13T11:27:25.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:25.700+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.701746+0000) 2019-10-13T11:27:25.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:25.700+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.701746+0000) 2019-10-13T11:27:26.279 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:26.276+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.277523+0000) 2019-10-13T11:27:26.279 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:26.276+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.278090+0000) 2019-10-13T11:27:26.280 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:26.276+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.277974+0000) 2019-10-13T11:27:26.280 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:26.278+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.278769+0000) 2019-10-13T11:27:26.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:26.277+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.278547+0000) 2019-10-13T11:27:26.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:26.277+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.278879+0000) 2019-10-13T11:27:26.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:26.277+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.279074+0000) 2019-10-13T11:27:26.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:26.279+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:26.280702+0000) 2019-10-13T11:27:26.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:26.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209902 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:26.303 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:27:26.303 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:26.307 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:27:26.307 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:28.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:28.607+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:28.608783+0000) 2019-10-13T11:27:28.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:28.941+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:28.942233+0000) 2019-10-13T11:27:31.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:31.287+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.288502+0000) 2019-10-13T11:27:31.290 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:31.287+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.288755+0000) 2019-10-13T11:27:31.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:31.287+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.289032+0000) 2019-10-13T11:27:31.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:31.287+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.288948+0000) 2019-10-13T11:27:31.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:31.288+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.289262+0000) 2019-10-13T11:27:31.291 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:31.287+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.289308+0000) 2019-10-13T11:27:31.291 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:31.288+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.289706+0000) 2019-10-13T11:27:31.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:31.291+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.291917+0000) 2019-10-13T11:27:31.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:31.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209932 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:31.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:31.559+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.560323+0000) 2019-10-13T11:27:33.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:33.046+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:33.047167+0000) 2019-10-13T11:27:33.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:33.050+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:33.050817+0000) 2019-10-13T11:27:35.437 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:35.436+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.437005+0000) 2019-10-13T11:27:35.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:35.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.701950+0000) 2019-10-13T11:27:35.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:35.701+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.701950+0000) 2019-10-13T11:27:36.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:36.297+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.298835+0000) 2019-10-13T11:27:36.300 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:36.297+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.299172+0000) 2019-10-13T11:27:36.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:36.299+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.300267+0000) 2019-10-13T11:27:36.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:36.297+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.299374+0000) 2019-10-13T11:27:36.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:36.298+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.299553+0000) 2019-10-13T11:27:36.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:36.298+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.299853+0000) 2019-10-13T11:27:36.302 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:36.298+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.300344+0000) 2019-10-13T11:27:36.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:36.300+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:36.301703+0000) 2019-10-13T11:27:36.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:36.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:38.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:38.608+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:38.608877+0000) 2019-10-13T11:27:38.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:38.941+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:38.942416+0000) 2019-10-13T11:27:41.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:41.308+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.309865+0000) 2019-10-13T11:27:41.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:41.308+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.310057+0000) 2019-10-13T11:27:41.311 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:41.308+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.310186+0000) 2019-10-13T11:27:41.312 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:41.310+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.310980+0000) 2019-10-13T11:27:41.312 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:41.309+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.310512+0000) 2019-10-13T11:27:41.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:41.309+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.310757+0000) 2019-10-13T11:27:41.313 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:41.309+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.310942+0000) 2019-10-13T11:27:41.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:41.311+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.312289+0000) 2019-10-13T11:27:41.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:41.317+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 209995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:41.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:41.560+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.560489+0000) 2019-10-13T11:27:43.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:43.046+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:43.047319+0000) 2019-10-13T11:27:43.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:43.050+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:43.050978+0000) 2019-10-13T11:27:45.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:45.439+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.439366+0000) 2019-10-13T11:27:45.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:45.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.702143+0000) 2019-10-13T11:27:45.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:45.701+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.702139+0000) 2019-10-13T11:27:46.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:46.320+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.320394+0000) 2019-10-13T11:27:46.322 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:46.320+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.321559+0000) 2019-10-13T11:27:46.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:46.320+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.320772+0000) 2019-10-13T11:27:46.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:46.320+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.321222+0000) 2019-10-13T11:27:46.323 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:46.320+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.321353+0000) 2019-10-13T11:27:46.323 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:46.321+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.321537+0000) 2019-10-13T11:27:46.323 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:46.321+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.321852+0000) 2019-10-13T11:27:46.324 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:46.323+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:46.323858+0000) 2019-10-13T11:27:46.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:46.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210026 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:48.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:48.608+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:48.609038+0000) 2019-10-13T11:27:48.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:48.942+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:48.942591+0000) 2019-10-13T11:27:51.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:51.330+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.330581+0000) 2019-10-13T11:27:51.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:51.330+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.330673+0000) 2019-10-13T11:27:51.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:51.330+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.331157+0000) 2019-10-13T11:27:51.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:51.330+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.331360+0000) 2019-10-13T11:27:51.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:51.331+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.332198+0000) 2019-10-13T11:27:51.333 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:51.331+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.332295+0000) 2019-10-13T11:27:51.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:51.332+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.332500+0000) 2019-10-13T11:27:51.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:51.333+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.334559+0000) 2019-10-13T11:27:51.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:51.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210056 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:51.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:51.560+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.560622+0000) 2019-10-13T11:27:53.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:53.047+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:53.047505+0000) 2019-10-13T11:27:53.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:53.051+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:53.051168+0000) 2019-10-13T11:27:55.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:55.440+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.441491+0000) 2019-10-13T11:27:55.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:55.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.702306+0000) 2019-10-13T11:27:55.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:55.702+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.702311+0000) 2019-10-13T11:27:56.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:27:56.341+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.341719+0000) 2019-10-13T11:27:56.343 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:56.341+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.342201+0000) 2019-10-13T11:27:56.343 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:27:56.342+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.342406+0000) 2019-10-13T11:27:56.344 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:27:56.341+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.343575+0000) 2019-10-13T11:27:56.344 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:56.342+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.342785+0000) 2019-10-13T11:27:56.344 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:27:56.342+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.343311+0000) 2019-10-13T11:27:56.344 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:27:56.342+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.343203+0000) 2019-10-13T11:27:56.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:27:56.343+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:56.345310+0000) 2019-10-13T11:27:56.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:27:56.349+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210087 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:27:56.465 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:27:56.465 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:56.469 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:27:56.470 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:58.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:27:58.609+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:58.609263+0000) 2019-10-13T11:27:58.943 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:27:58.942+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:58.942817+0000) 2019-10-13T11:28:01.352 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:01.350+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.351682+0000) 2019-10-13T11:28:01.353 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:01.350+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.351931+0000) 2019-10-13T11:28:01.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:01.350+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.352334+0000) 2019-10-13T11:28:01.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:01.351+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.352710+0000) 2019-10-13T11:28:01.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:01.351+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.352714+0000) 2019-10-13T11:28:01.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:01.352+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.353999+0000) 2019-10-13T11:28:01.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:01.351+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.353251+0000) 2019-10-13T11:28:01.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:01.354+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.355757+0000) 2019-10-13T11:28:01.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:01.359+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210119 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:01.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:01.559+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.560771+0000) 2019-10-13T11:28:03.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:03.046+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:03.047695+0000) 2019-10-13T11:28:03.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:03.049+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:03.051390+0000) 2019-10-13T11:28:05.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:05.442+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.443770+0000) 2019-10-13T11:28:05.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:05.701+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.702505+0000) 2019-10-13T11:28:05.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:05.701+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.702505+0000) 2019-10-13T11:28:06.363 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:06.360+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.362319+0000) 2019-10-13T11:28:06.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:06.360+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.362278+0000) 2019-10-13T11:28:06.364 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:06.361+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.362715+0000) 2019-10-13T11:28:06.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:06.361+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.362859+0000) 2019-10-13T11:28:06.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:06.361+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.362999+0000) 2019-10-13T11:28:06.365 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:06.362+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.364182+0000) 2019-10-13T11:28:06.365 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:06.361+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.363155+0000) 2019-10-13T11:28:06.366 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:06.364+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:06.366244+0000) 2019-10-13T11:28:06.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:06.369+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210150 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:08.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:08.608+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:08.609453+0000) 2019-10-13T11:28:08.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:08.942+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:08.942947+0000) 2019-10-13T11:28:11.373 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:11.371+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.372637+0000) 2019-10-13T11:28:11.374 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:11.371+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.373000+0000) 2019-10-13T11:28:11.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:11.372+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.373415+0000) 2019-10-13T11:28:11.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:11.372+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.373662+0000) 2019-10-13T11:28:11.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:11.372+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.373569+0000) 2019-10-13T11:28:11.375 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:11.372+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.373792+0000) 2019-10-13T11:28:11.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:11.372+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.374042+0000) 2019-10-13T11:28:11.376 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:11.375+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.375899+0000) 2019-10-13T11:28:11.382 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:11.380+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210180 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:11.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:11.560+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.560962+0000) 2019-10-13T11:28:13.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:13.047+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:13.047883+0000) 2019-10-13T11:28:13.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:13.050+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:13.051562+0000) 2019-10-13T11:28:15.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:15.446+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.447067+0000) 2019-10-13T11:28:15.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:15.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.702716+0000) 2019-10-13T11:28:15.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:15.702+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.702715+0000) 2019-10-13T11:28:16.384 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:16.382+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.383454+0000) 2019-10-13T11:28:16.385 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:16.383+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.384517+0000) 2019-10-13T11:28:16.385 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:16.382+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.383887+0000) 2019-10-13T11:28:16.385 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:16.382+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.384105+0000) 2019-10-13T11:28:16.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:16.382+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.384286+0000) 2019-10-13T11:28:16.386 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:16.383+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.384425+0000) 2019-10-13T11:28:16.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:16.383+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.384926+0000) 2019-10-13T11:28:16.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:16.386+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:16.387397+0000) 2019-10-13T11:28:16.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:16.390+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:18.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:18.608+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:18.609641+0000) 2019-10-13T11:28:18.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:18.942+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:18.943127+0000) 2019-10-13T11:28:21.395 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:21.394+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.394760+0000) 2019-10-13T11:28:21.396 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:21.393+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.394532+0000) 2019-10-13T11:28:21.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:21.393+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.394604+0000) 2019-10-13T11:28:21.396 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:21.393+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.394830+0000) 2019-10-13T11:28:21.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:21.393+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.395109+0000) 2019-10-13T11:28:21.397 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:21.394+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.395512+0000) 2019-10-13T11:28:21.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:21.394+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.395781+0000) 2019-10-13T11:28:21.397 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:21.396+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.397044+0000) 2019-10-13T11:28:21.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:21.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210243 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:21.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:21.560+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.561111+0000) 2019-10-13T11:28:23.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:23.047+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:23.048055+0000) 2019-10-13T11:28:23.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:23.051+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:23.051760+0000) 2019-10-13T11:28:25.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:25.448+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.449461+0000) 2019-10-13T11:28:25.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:25.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.702891+0000) 2019-10-13T11:28:25.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:25.702+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.702892+0000) 2019-10-13T11:28:26.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:26.402+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.404144+0000) 2019-10-13T11:28:26.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:26.402+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.404336+0000) 2019-10-13T11:28:26.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:26.403+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.404767+0000) 2019-10-13T11:28:26.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:26.403+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.404721+0000) 2019-10-13T11:28:26.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:26.403+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.404866+0000) 2019-10-13T11:28:26.407 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:26.403+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.405267+0000) 2019-10-13T11:28:26.407 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:26.405+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.406162+0000) 2019-10-13T11:28:26.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:26.407+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:26.407867+0000) 2019-10-13T11:28:26.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:26.409+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:26.627 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:28:26.628 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:26.631 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:28:26.631 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:28.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:28.609+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:28.609853+0000) 2019-10-13T11:28:28.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:28.942+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:28.943310+0000) 2019-10-13T11:28:31.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:31.412+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.412483+0000) 2019-10-13T11:28:31.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:31.412+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.412772+0000) 2019-10-13T11:28:31.414 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:31.412+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.413155+0000) 2019-10-13T11:28:31.415 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:31.413+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.413937+0000) 2019-10-13T11:28:31.415 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:31.413+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.413428+0000) 2019-10-13T11:28:31.415 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:31.413+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.413611+0000) 2019-10-13T11:28:31.416 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:31.413+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.413736+0000) 2019-10-13T11:28:31.416 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:31.414+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.415585+0000) 2019-10-13T11:28:31.421 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:31.420+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:31.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:31.560+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.561262+0000) 2019-10-13T11:28:33.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:33.047+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:33.048227+0000) 2019-10-13T11:28:33.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:33.051+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:33.051922+0000) 2019-10-13T11:28:35.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:35.451+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.451759+0000) 2019-10-13T11:28:35.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:35.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.703064+0000) 2019-10-13T11:28:35.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:35.702+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.703064+0000) 2019-10-13T11:28:36.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:36.422+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.423166+0000) 2019-10-13T11:28:36.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:36.422+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.422937+0000) 2019-10-13T11:28:36.425 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:36.422+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.423077+0000) 2019-10-13T11:28:36.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:36.422+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.423370+0000) 2019-10-13T11:28:36.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:36.423+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.423733+0000) 2019-10-13T11:28:36.426 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:36.423+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.424188+0000) 2019-10-13T11:28:36.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:36.424+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.424433+0000) 2019-10-13T11:28:36.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:36.425+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:36.426288+0000) 2019-10-13T11:28:36.432 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:36.430+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:38.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:38.609+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:38.610036+0000) 2019-10-13T11:28:38.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:38.943+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:38.943477+0000) 2019-10-13T11:28:41.434 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:41.432+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.433058+0000) 2019-10-13T11:28:41.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:41.432+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.433239+0000) 2019-10-13T11:28:41.435 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:41.432+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.433344+0000) 2019-10-13T11:28:41.435 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:41.433+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.433527+0000) 2019-10-13T11:28:41.435 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:41.433+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.433756+0000) 2019-10-13T11:28:41.435 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:41.433+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.433705+0000) 2019-10-13T11:28:41.436 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:41.433+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.434543+0000) 2019-10-13T11:28:41.437 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:41.436+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.437559+0000) 2019-10-13T11:28:41.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:41.440+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:41.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:41.561+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.561433+0000) 2019-10-13T11:28:43.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:43.048+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:43.048414+0000) 2019-10-13T11:28:43.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:43.052+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:43.052097+0000) 2019-10-13T11:28:45.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:45.452+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.453993+0000) 2019-10-13T11:28:45.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:45.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.703250+0000) 2019-10-13T11:28:45.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:45.703+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.703270+0000) 2019-10-13T11:28:46.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:46.441+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.442368+0000) 2019-10-13T11:28:46.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:46.442+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.442802+0000) 2019-10-13T11:28:46.444 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:46.442+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.442964+0000) 2019-10-13T11:28:46.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:46.442+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.444286+0000) 2019-10-13T11:28:46.445 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:46.443+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.443449+0000) 2019-10-13T11:28:46.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:46.442+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.443321+0000) 2019-10-13T11:28:46.445 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:46.443+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.443833+0000) 2019-10-13T11:28:46.446 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:46.443+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:46.445194+0000) 2019-10-13T11:28:46.452 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:46.450+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:48.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:48.608+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:48.610244+0000) 2019-10-13T11:28:48.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:48.942+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:48.943626+0000) 2019-10-13T11:28:51.455 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:51.452+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.454296+0000) 2019-10-13T11:28:51.456 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:51.452+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.454373+0000) 2019-10-13T11:28:51.456 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:51.453+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.454584+0000) 2019-10-13T11:28:51.456 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:51.453+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.454899+0000) 2019-10-13T11:28:51.456 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:51.453+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.455065+0000) 2019-10-13T11:28:51.457 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:51.453+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.455166+0000) 2019-10-13T11:28:51.457 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:51.453+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.455319+0000) 2019-10-13T11:28:51.458 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:51.456+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.458057+0000) 2019-10-13T11:28:51.463 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:51.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210428 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:51.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:51.560+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.561567+0000) 2019-10-13T11:28:53.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:53.047+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:53.048596+0000) 2019-10-13T11:28:53.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:53.051+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:53.052279+0000) 2019-10-13T11:28:55.457 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:55.455+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.456405+0000) 2019-10-13T11:28:55.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:55.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.703433+0000) 2019-10-13T11:28:55.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:55.702+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.703434+0000) 2019-10-13T11:28:56.464 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:28:56.462+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.464477+0000) 2019-10-13T11:28:56.465 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:56.462+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.464086+0000) 2019-10-13T11:28:56.466 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:28:56.462+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.464261+0000) 2019-10-13T11:28:56.466 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:28:56.462+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.464343+0000) 2019-10-13T11:28:56.466 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:28:56.463+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.464561+0000) 2019-10-13T11:28:56.466 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:28:56.463+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.465036+0000) 2019-10-13T11:28:56.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:28:56.464+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.466321+0000) 2019-10-13T11:28:56.467 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:56.463+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:56.465263+0000) 2019-10-13T11:28:56.474 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:28:56.472+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210462 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:28:56.765 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:28:56.765 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:56.770 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:28:56.770 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:58.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:28:58.609+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:58.610433+0000) 2019-10-13T11:28:58.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:28:58.942+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:58.943783+0000) 2019-10-13T11:29:01.476 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:01.475+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.476328+0000) 2019-10-13T11:29:01.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:01.474+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.475921+0000) 2019-10-13T11:29:01.477 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:01.474+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.476008+0000) 2019-10-13T11:29:01.477 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:01.474+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.476297+0000) 2019-10-13T11:29:01.478 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:01.475+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.476408+0000) 2019-10-13T11:29:01.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:01.475+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.476881+0000) 2019-10-13T11:29:01.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:01.475+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.477148+0000) 2019-10-13T11:29:01.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:01.479+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.480114+0000) 2019-10-13T11:29:01.485 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:01.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:01.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:01.561+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.561731+0000) 2019-10-13T11:29:03.049 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:03.048+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:03.048777+0000) 2019-10-13T11:29:03.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:03.051+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:03.052467+0000) 2019-10-13T11:29:05.459 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:05.458+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.458763+0000) 2019-10-13T11:29:05.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:05.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.703650+0000) 2019-10-13T11:29:05.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:05.703+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.703653+0000) 2019-10-13T11:29:06.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:06.484+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.486163+0000) 2019-10-13T11:29:06.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:06.485+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.486496+0000) 2019-10-13T11:29:06.488 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:06.486+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.486898+0000) 2019-10-13T11:29:06.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:06.485+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.486772+0000) 2019-10-13T11:29:06.488 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:06.485+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.487135+0000) 2019-10-13T11:29:06.489 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:06.485+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.487244+0000) 2019-10-13T11:29:06.489 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:06.485+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.487333+0000) 2019-10-13T11:29:06.489 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:06.488+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:06.489200+0000) 2019-10-13T11:29:06.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:06.493+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:08.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:08.610+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:08.610646+0000) 2019-10-13T11:29:08.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:08.943+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:08.944016+0000) 2019-10-13T11:29:11.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:11.495+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.496468+0000) 2019-10-13T11:29:11.498 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:11.497+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.497756+0000) 2019-10-13T11:29:11.498 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:11.495+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.496781+0000) 2019-10-13T11:29:11.498 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:11.495+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.497229+0000) 2019-10-13T11:29:11.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:11.496+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.497554+0000) 2019-10-13T11:29:11.499 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:11.496+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.497533+0000) 2019-10-13T11:29:11.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:11.496+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.497962+0000) 2019-10-13T11:29:11.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:11.499+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.499821+0000) 2019-10-13T11:29:11.506 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:11.503+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:11.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:11.561+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.561894+0000) 2019-10-13T11:29:13.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:13.048+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:13.048966+0000) 2019-10-13T11:29:13.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:13.052+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:13.052672+0000) 2019-10-13T11:29:15.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:15.460+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.461011+0000) 2019-10-13T11:29:15.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:15.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.703866+0000) 2019-10-13T11:29:15.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:15.703+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.703871+0000) 2019-10-13T11:29:16.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:16.506+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.506994+0000) 2019-10-13T11:29:16.509 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:16.507+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.508204+0000) 2019-10-13T11:29:16.509 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:16.507+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.507573+0000) 2019-10-13T11:29:16.509 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:16.507+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.507802+0000) 2019-10-13T11:29:16.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:16.507+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.507990+0000) 2019-10-13T11:29:16.510 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:16.507+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.508162+0000) 2019-10-13T11:29:16.510 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:16.507+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.508330+0000) 2019-10-13T11:29:16.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:16.509+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:16.509911+0000) 2019-10-13T11:29:16.516 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:16.515+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210586 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:18.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:18.610+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:18.610841+0000) 2019-10-13T11:29:18.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:18.943+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:18.944209+0000) 2019-10-13T11:29:21.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:21.517+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.517633+0000) 2019-10-13T11:29:21.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:21.517+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.517822+0000) 2019-10-13T11:29:21.519 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:21.517+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.518111+0000) 2019-10-13T11:29:21.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:21.517+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.518015+0000) 2019-10-13T11:29:21.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:21.518+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.518945+0000) 2019-10-13T11:29:21.520 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:21.518+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.518692+0000) 2019-10-13T11:29:21.521 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:21.518+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.519080+0000) 2019-10-13T11:29:21.521 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:21.520+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.521289+0000) 2019-10-13T11:29:21.527 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:21.526+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210615 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:21.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:21.561+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.562046+0000) 2019-10-13T11:29:23.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:23.048+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:23.049152+0000) 2019-10-13T11:29:23.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:23.052+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:23.052873+0000) 2019-10-13T11:29:25.463 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:25.463+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.463312+0000) 2019-10-13T11:29:25.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:25.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.704085+0000) 2019-10-13T11:29:25.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:25.703+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.704084+0000) 2019-10-13T11:29:26.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:26.528+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.528658+0000) 2019-10-13T11:29:26.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:26.528+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.528935+0000) 2019-10-13T11:29:26.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:26.528+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.529214+0000) 2019-10-13T11:29:26.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:26.529+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.529462+0000) 2019-10-13T11:29:26.531 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:26.529+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.529539+0000) 2019-10-13T11:29:26.531 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:26.529+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.530443+0000) 2019-10-13T11:29:26.531 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:26.529+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.530158+0000) 2019-10-13T11:29:26.532 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:26.531+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:26.532660+0000) 2019-10-13T11:29:26.538 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:26.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210649 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:26.920 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:29:26.920 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:26.924 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:29:26.924 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:28.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:28.610+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:28.611046+0000) 2019-10-13T11:29:28.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:28.944+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:28.944356+0000) 2019-10-13T11:29:31.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:31.538+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.539244+0000) 2019-10-13T11:29:31.540 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:31.538+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.539387+0000) 2019-10-13T11:29:31.541 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:31.539+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.539742+0000) 2019-10-13T11:29:31.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:31.539+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.540146+0000) 2019-10-13T11:29:31.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:31.539+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.540267+0000) 2019-10-13T11:29:31.542 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:31.540+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.540469+0000) 2019-10-13T11:29:31.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:31.540+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.540675+0000) 2019-10-13T11:29:31.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:31.541+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.542543+0000) 2019-10-13T11:29:31.549 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:31.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:31.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:31.562+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.562182+0000) 2019-10-13T11:29:33.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:33.047+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:33.049356+0000) 2019-10-13T11:29:33.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:33.052+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:33.053023+0000) 2019-10-13T11:29:35.466 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:35.465+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.465594+0000) 2019-10-13T11:29:35.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:35.702+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.704305+0000) 2019-10-13T11:29:35.705 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:35.702+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.704303+0000) 2019-10-13T11:29:36.551 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:36.549+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.550410+0000) 2019-10-13T11:29:36.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:36.548+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.550313+0000) 2019-10-13T11:29:36.552 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:36.549+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.550604+0000) 2019-10-13T11:29:36.552 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:36.549+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.551315+0000) 2019-10-13T11:29:36.553 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:36.549+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.551248+0000) 2019-10-13T11:29:36.553 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:36.549+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.551188+0000) 2019-10-13T11:29:36.553 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:36.550+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.551607+0000) 2019-10-13T11:29:36.554 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:36.552+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:36.554457+0000) 2019-10-13T11:29:36.560 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:36.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210710 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:38.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:38.610+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:38.611216+0000) 2019-10-13T11:29:38.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:38.943+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:38.944539+0000) 2019-10-13T11:29:41.562 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:41.559+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.561507+0000) 2019-10-13T11:29:41.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:41.559+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.561244+0000) 2019-10-13T11:29:41.563 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:41.560+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.561583+0000) 2019-10-13T11:29:41.563 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:41.560+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.561740+0000) 2019-10-13T11:29:41.563 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:41.560+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.561987+0000) 2019-10-13T11:29:41.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:41.560+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.562388+0000) 2019-10-13T11:29:41.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:41.561+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.562440+0000) 2019-10-13T11:29:41.564 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:41.561+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.562394+0000) 2019-10-13T11:29:41.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:41.562+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.563794+0000) 2019-10-13T11:29:41.571 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:41.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210739 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:43.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:43.048+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:43.049530+0000) 2019-10-13T11:29:43.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:43.052+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:43.053190+0000) 2019-10-13T11:29:45.468 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:45.467+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.467878+0000) 2019-10-13T11:29:45.705 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:45.703+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.704500+0000) 2019-10-13T11:29:45.706 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:45.703+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.704501+0000) 2019-10-13T11:29:46.573 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:46.571+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.572775+0000) 2019-10-13T11:29:46.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:46.571+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.572439+0000) 2019-10-13T11:29:46.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:46.571+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.572594+0000) 2019-10-13T11:29:46.574 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:46.571+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.573213+0000) 2019-10-13T11:29:46.574 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:46.571+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.573124+0000) 2019-10-13T11:29:46.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:46.572+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.573485+0000) 2019-10-13T11:29:46.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:46.572+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.573660+0000) 2019-10-13T11:29:46.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:46.573+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:46.575551+0000) 2019-10-13T11:29:46.581 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:46.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:48.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:48.610+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:48.611394+0000) 2019-10-13T11:29:48.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:48.944+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:48.944756+0000) 2019-10-13T11:29:51.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:51.562+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.562635+0000) 2019-10-13T11:29:51.583 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:51.581+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.582680+0000) 2019-10-13T11:29:51.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:51.582+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.583592+0000) 2019-10-13T11:29:51.584 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:51.581+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.583263+0000) 2019-10-13T11:29:51.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:51.582+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.583783+0000) 2019-10-13T11:29:51.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:51.582+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.583537+0000) 2019-10-13T11:29:51.585 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:51.582+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.583949+0000) 2019-10-13T11:29:51.585 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:51.582+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.584138+0000) 2019-10-13T11:29:51.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:51.585+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.586063+0000) 2019-10-13T11:29:51.592 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:51.590+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:53.050 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:53.049+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:53.049681+0000) 2019-10-13T11:29:53.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:53.052+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:53.053347+0000) 2019-10-13T11:29:55.470 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:55.469+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.470282+0000) 2019-10-13T11:29:55.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:55.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.704696+0000) 2019-10-13T11:29:55.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:55.704+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.704695+0000) 2019-10-13T11:29:56.595 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:29:56.593+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.594304+0000) 2019-10-13T11:29:56.595 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:56.592+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.593601+0000) 2019-10-13T11:29:56.595 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:29:56.592+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.593421+0000) 2019-10-13T11:29:56.595 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:29:56.592+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.593867+0000) 2019-10-13T11:29:56.596 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:56.593+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.594596+0000) 2019-10-13T11:29:56.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:29:56.593+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.594950+0000) 2019-10-13T11:29:56.597 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:29:56.593+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.595079+0000) 2019-10-13T11:29:56.597 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:29:56.596+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:56.597124+0000) 2019-10-13T11:29:56.603 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:29:56.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210834 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:29:57.057 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:29:57.057 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:57.061 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:29:57.061 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:58.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:29:58.611+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:58.611617+0000) 2019-10-13T11:29:58.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:29:58.944+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:58.944900+0000) 2019-10-13T11:30:01.563 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:01.562+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.562812+0000) 2019-10-13T11:30:01.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:01.603+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.603999+0000) 2019-10-13T11:30:01.606 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:01.604+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.604749+0000) 2019-10-13T11:30:01.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:01.603+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.604373+0000) 2019-10-13T11:30:01.606 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:01.604+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.604685+0000) 2019-10-13T11:30:01.606 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:01.604+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.604992+0000) 2019-10-13T11:30:01.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:01.604+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.605323+0000) 2019-10-13T11:30:01.607 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:01.604+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.604835+0000) 2019-10-13T11:30:01.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:01.605+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.606348+0000) 2019-10-13T11:30:01.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:01.612+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:03.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:03.049+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:03.049829+0000) 2019-10-13T11:30:03.053 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:03.053+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:03.053478+0000) 2019-10-13T11:30:05.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:05.472+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.472470+0000) 2019-10-13T11:30:05.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:05.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.704891+0000) 2019-10-13T11:30:05.706 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:05.704+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.704890+0000) 2019-10-13T11:30:06.615 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:06.614+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.614418+0000) 2019-10-13T11:30:06.616 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:06.614+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.615465+0000) 2019-10-13T11:30:06.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:06.614+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.614817+0000) 2019-10-13T11:30:06.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:06.614+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.615119+0000) 2019-10-13T11:30:06.617 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:06.615+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.615466+0000) 2019-10-13T11:30:06.617 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:06.615+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.615577+0000) 2019-10-13T11:30:06.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:06.615+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.615937+0000) 2019-10-13T11:30:06.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:06.617+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:06.618473+0000) 2019-10-13T11:30:06.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:06.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:08.612 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:08.611+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:08.611793+0000) 2019-10-13T11:30:08.945 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:08.944+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:08.945013+0000) 2019-10-13T11:30:11.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:11.562+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.563014+0000) 2019-10-13T11:30:11.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:11.625+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.625459+0000) 2019-10-13T11:30:11.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:11.625+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.625742+0000) 2019-10-13T11:30:11.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:11.625+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.625857+0000) 2019-10-13T11:30:11.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:11.625+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.626170+0000) 2019-10-13T11:30:11.628 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:11.625+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.626159+0000) 2019-10-13T11:30:11.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:11.626+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.626428+0000) 2019-10-13T11:30:11.628 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:11.626+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.626671+0000) 2019-10-13T11:30:11.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:11.627+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.628610+0000) 2019-10-13T11:30:11.635 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:11.633+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210924 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:13.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:13.049+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:13.050044+0000) 2019-10-13T11:30:13.054 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:13.053+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:13.053670+0000) 2019-10-13T11:30:15.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:15.474+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.474777+0000) 2019-10-13T11:30:15.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:15.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.705114+0000) 2019-10-13T11:30:15.706 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:15.704+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.705111+0000) 2019-10-13T11:30:16.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:16.635+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.636208+0000) 2019-10-13T11:30:16.638 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:16.636+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.636557+0000) 2019-10-13T11:30:16.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:16.636+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.636440+0000) 2019-10-13T11:30:16.638 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:16.636+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.636730+0000) 2019-10-13T11:30:16.639 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:16.636+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.637659+0000) 2019-10-13T11:30:16.639 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:16.636+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.637289+0000) 2019-10-13T11:30:16.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:16.637+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.637406+0000) 2019-10-13T11:30:16.640 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:16.639+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:16.639753+0000) 2019-10-13T11:30:16.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:16.644+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:18.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:18.610+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:18.611962+0000) 2019-10-13T11:30:18.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:18.944+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:18.945173+0000) 2019-10-13T11:30:21.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:21.562+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.563212+0000) 2019-10-13T11:30:21.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:21.644+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.646387+0000) 2019-10-13T11:30:21.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:21.645+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.646680+0000) 2019-10-13T11:30:21.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:21.645+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.646856+0000) 2019-10-13T11:30:21.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:21.645+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.647049+0000) 2019-10-13T11:30:21.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:21.645+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.647158+0000) 2019-10-13T11:30:21.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:21.647+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.647810+0000) 2019-10-13T11:30:21.649 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:21.648+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.649208+0000) 2019-10-13T11:30:21.649 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:21.646+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.647737+0000) 2019-10-13T11:30:21.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:21.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 210987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:23.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:23.049+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:23.050235+0000) 2019-10-13T11:30:23.054 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:23.053+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:23.053854+0000) 2019-10-13T11:30:25.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:25.477+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.477102+0000) 2019-10-13T11:30:25.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:25.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.705283+0000) 2019-10-13T11:30:25.706 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:25.704+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.705284+0000) 2019-10-13T11:30:26.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:26.655+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.656994+0000) 2019-10-13T11:30:26.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:26.655+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.657196+0000) 2019-10-13T11:30:26.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:26.655+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.657303+0000) 2019-10-13T11:30:26.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:26.656+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.658473+0000) 2019-10-13T11:30:26.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:26.656+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.657705+0000) 2019-10-13T11:30:26.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:26.656+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.658146+0000) 2019-10-13T11:30:26.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:26.657+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.658546+0000) 2019-10-13T11:30:26.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:26.658+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:26.660588+0000) 2019-10-13T11:30:26.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:26.664+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:27.236 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:30:27.237 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:27.242 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:30:27.242 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:28.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:28.610+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:28.612154+0000) 2019-10-13T11:30:28.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:28.944+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:28.945380+0000) 2019-10-13T11:30:31.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:31.562+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.563349+0000) 2019-10-13T11:30:31.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:31.666+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.667414+0000) 2019-10-13T11:30:31.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:31.666+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.667587+0000) 2019-10-13T11:30:31.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:31.667+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.668984+0000) 2019-10-13T11:30:31.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:31.666+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.668214+0000) 2019-10-13T11:30:31.670 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:31.667+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.668661+0000) 2019-10-13T11:30:31.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:31.667+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.668948+0000) 2019-10-13T11:30:31.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:31.667+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.669112+0000) 2019-10-13T11:30:31.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:31.670+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.672132+0000) 2019-10-13T11:30:31.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:31.675+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:33.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:33.049+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:33.050381+0000) 2019-10-13T11:30:33.054 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:33.053+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:33.054034+0000) 2019-10-13T11:30:35.479 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:35.478+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.479449+0000) 2019-10-13T11:30:35.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:35.704+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.705445+0000) 2019-10-13T11:30:35.706 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:35.704+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.705446+0000) 2019-10-13T11:30:36.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:36.677+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.678728+0000) 2019-10-13T11:30:36.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:36.678+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.679990+0000) 2019-10-13T11:30:36.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:36.677+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.679000+0000) 2019-10-13T11:30:36.681 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:36.677+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.679107+0000) 2019-10-13T11:30:36.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:36.677+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.679323+0000) 2019-10-13T11:30:36.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:36.678+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.680089+0000) 2019-10-13T11:30:36.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:36.678+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.680122+0000) 2019-10-13T11:30:36.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:36.680+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:36.682500+0000) 2019-10-13T11:30:36.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:36.686+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211082 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:38.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:38.611+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:38.612325+0000) 2019-10-13T11:30:38.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:38.945+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:38.945587+0000) 2019-10-13T11:30:41.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:41.562+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.563524+0000) 2019-10-13T11:30:41.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:41.687+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.689220+0000) 2019-10-13T11:30:41.690 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:41.687+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.689348+0000) 2019-10-13T11:30:41.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:41.688+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.689463+0000) 2019-10-13T11:30:41.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:41.688+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.689607+0000) 2019-10-13T11:30:41.691 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:41.688+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.689853+0000) 2019-10-13T11:30:41.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:41.689+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.690660+0000) 2019-10-13T11:30:41.692 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:41.688+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.690135+0000) 2019-10-13T11:30:41.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:41.691+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.692437+0000) 2019-10-13T11:30:41.696 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:41.694+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:43.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:43.050+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:43.050536+0000) 2019-10-13T11:30:43.054 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:43.053+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:43.054211+0000) 2019-10-13T11:30:45.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:45.481+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.481832+0000) 2019-10-13T11:30:45.706 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:45.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.705626+0000) 2019-10-13T11:30:45.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:45.705+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.705627+0000) 2019-10-13T11:30:46.698 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:46.697+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.697688+0000) 2019-10-13T11:30:46.699 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:46.697+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.697973+0000) 2019-10-13T11:30:46.699 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:46.698+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.698815+0000) 2019-10-13T11:30:46.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:46.698+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.698508+0000) 2019-10-13T11:30:46.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:46.698+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.698431+0000) 2019-10-13T11:30:46.700 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:46.698+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.698745+0000) 2019-10-13T11:30:46.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:46.698+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.698995+0000) 2019-10-13T11:30:46.701 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:46.699+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:46.699998+0000) 2019-10-13T11:30:46.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:46.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211145 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:48.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:48.612+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:48.612510+0000) 2019-10-13T11:30:48.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:48.945+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:48.945767+0000) 2019-10-13T11:30:51.564 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:51.563+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.563709+0000) 2019-10-13T11:30:51.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:51.707+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.708626+0000) 2019-10-13T11:30:51.709 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:51.707+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.708312+0000) 2019-10-13T11:30:51.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:51.708+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.708461+0000) 2019-10-13T11:30:51.710 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:51.708+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.708925+0000) 2019-10-13T11:30:51.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:51.708+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.709164+0000) 2019-10-13T11:30:51.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:51.708+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.709271+0000) 2019-10-13T11:30:51.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:51.709+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.710224+0000) 2019-10-13T11:30:51.711 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:51.709+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.709635+0000) 2019-10-13T11:30:51.717 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:51.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:53.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:53.050+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:53.050745+0000) 2019-10-13T11:30:53.054 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:53.053+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:53.054360+0000) 2019-10-13T11:30:55.485 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:55.484+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.484890+0000) 2019-10-13T11:30:55.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:55.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.705819+0000) 2019-10-13T11:30:55.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:55.705+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.705837+0000) 2019-10-13T11:30:56.719 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:30:56.718+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.719220+0000) 2019-10-13T11:30:56.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:30:56.718+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.719155+0000) 2019-10-13T11:30:56.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:30:56.718+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.719344+0000) 2019-10-13T11:30:56.720 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:56.719+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.719633+0000) 2019-10-13T11:30:56.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:30:56.719+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.719869+0000) 2019-10-13T11:30:56.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:30:56.719+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.720034+0000) 2019-10-13T11:30:56.721 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:56.719+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.720168+0000) 2019-10-13T11:30:56.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:30:56.721+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:56.721776+0000) 2019-10-13T11:30:56.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:30:56.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:30:57.341 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:30:57.342 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:57.346 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:30:57.346 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:58.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:30:58.612+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:58.612694+0000) 2019-10-13T11:30:58.946 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:30:58.945+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:58.945865+0000) 2019-10-13T11:31:01.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:01.563+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.563919+0000) 2019-10-13T11:31:01.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:01.729+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.729587+0000) 2019-10-13T11:31:01.731 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:01.729+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.730243+0000) 2019-10-13T11:31:01.731 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:01.729+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.730186+0000) 2019-10-13T11:31:01.732 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:01.730+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.730504+0000) 2019-10-13T11:31:01.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:01.730+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.730713+0000) 2019-10-13T11:31:01.732 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:01.730+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.730928+0000) 2019-10-13T11:31:01.732 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:01.731+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.731515+0000) 2019-10-13T11:31:01.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:01.733+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.733962+0000) 2019-10-13T11:31:01.739 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:01.738+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:03.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:03.050+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:03.050957+0000) 2019-10-13T11:31:03.054 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:03.054+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:03.054556+0000) 2019-10-13T11:31:05.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:05.487+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.487971+0000) 2019-10-13T11:31:05.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:05.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.706019+0000) 2019-10-13T11:31:05.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:05.705+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.706026+0000) 2019-10-13T11:31:06.742 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:06.739+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.740940+0000) 2019-10-13T11:31:06.742 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:06.739+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.741225+0000) 2019-10-13T11:31:06.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:06.739+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.741318+0000) 2019-10-13T11:31:06.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:06.740+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.741512+0000) 2019-10-13T11:31:06.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:06.741+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.742495+0000) 2019-10-13T11:31:06.743 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:06.742+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.743313+0000) 2019-10-13T11:31:06.744 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:06.740+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.741771+0000) 2019-10-13T11:31:06.744 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:06.740+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:06.742035+0000) 2019-10-13T11:31:06.750 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:06.747+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211269 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:08.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:08.611+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:08.612939+0000) 2019-10-13T11:31:08.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:08.945+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:08.946031+0000) 2019-10-13T11:31:11.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:11.563+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.564122+0000) 2019-10-13T11:31:11.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:11.749+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.750720+0000) 2019-10-13T11:31:11.752 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:11.749+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.751026+0000) 2019-10-13T11:31:11.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:11.749+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.750922+0000) 2019-10-13T11:31:11.753 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:11.749+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.751218+0000) 2019-10-13T11:31:11.753 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:11.750+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.751815+0000) 2019-10-13T11:31:11.753 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:11.750+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.751875+0000) 2019-10-13T11:31:11.754 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:11.752+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.753289+0000) 2019-10-13T11:31:11.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:11.753+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.754603+0000) 2019-10-13T11:31:11.760 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:11.758+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211296 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:13.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:13.050+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:13.051126+0000) 2019-10-13T11:31:13.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:13.054+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:13.054780+0000) 2019-10-13T11:31:15.490 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:15.490+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.490343+0000) 2019-10-13T11:31:15.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:15.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.706239+0000) 2019-10-13T11:31:15.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:15.705+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.706240+0000) 2019-10-13T11:31:16.763 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:16.760+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.761691+0000) 2019-10-13T11:31:16.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:16.760+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.761866+0000) 2019-10-13T11:31:16.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:16.760+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.762180+0000) 2019-10-13T11:31:16.764 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:16.760+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.762191+0000) 2019-10-13T11:31:16.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:16.760+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.762299+0000) 2019-10-13T11:31:16.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:16.761+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.762474+0000) 2019-10-13T11:31:16.765 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:16.760+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.762472+0000) 2019-10-13T11:31:16.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:16.763+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:16.765005+0000) 2019-10-13T11:31:16.770 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:16.767+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211330 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:18.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:18.611+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:18.613147+0000) 2019-10-13T11:31:18.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:18.945+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:18.946184+0000) 2019-10-13T11:31:21.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:21.563+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.564306+0000) 2019-10-13T11:31:21.771 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:21.769+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.771339+0000) 2019-10-13T11:31:21.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:21.769+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.770931+0000) 2019-10-13T11:31:21.773 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:21.769+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.771270+0000) 2019-10-13T11:31:21.773 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:21.770+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.771568+0000) 2019-10-13T11:31:21.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:21.770+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.771878+0000) 2019-10-13T11:31:21.773 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:21.770+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.771953+0000) 2019-10-13T11:31:21.774 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:21.770+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.772302+0000) 2019-10-13T11:31:21.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:21.771+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.773697+0000) 2019-10-13T11:31:21.780 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:21.778+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211359 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:23.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:23.050+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:23.051266+0000) 2019-10-13T11:31:23.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:23.054+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:23.054930+0000) 2019-10-13T11:31:25.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:25.492+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.493519+0000) 2019-10-13T11:31:25.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:25.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.706483+0000) 2019-10-13T11:31:25.707 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:25.705+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.706484+0000) 2019-10-13T11:31:26.783 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:26.780+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.781791+0000) 2019-10-13T11:31:26.783 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:26.780+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.781640+0000) 2019-10-13T11:31:26.784 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:26.780+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.782049+0000) 2019-10-13T11:31:26.784 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:26.780+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.782117+0000) 2019-10-13T11:31:26.784 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:26.781+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.782507+0000) 2019-10-13T11:31:26.784 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:26.781+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.782784+0000) 2019-10-13T11:31:26.785 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:26.781+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.783198+0000) 2019-10-13T11:31:26.785 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:26.783+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:26.784929+0000) 2019-10-13T11:31:26.791 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:26.788+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:27.512 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:31:27.512 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:27.517 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:31:27.517 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:28.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:28.612+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:28.613326+0000) 2019-10-13T11:31:28.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:28.945+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:28.946325+0000) 2019-10-13T11:31:31.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:31.564+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.564495+0000) 2019-10-13T11:31:31.793 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:31.791+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.791749+0000) 2019-10-13T11:31:31.793 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:31.791+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.792024+0000) 2019-10-13T11:31:31.793 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:31.791+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.792014+0000) 2019-10-13T11:31:31.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:31.791+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.792246+0000) 2019-10-13T11:31:31.794 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:31.792+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.792404+0000) 2019-10-13T11:31:31.795 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:31.792+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.793203+0000) 2019-10-13T11:31:31.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:31.793+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.794598+0000) 2019-10-13T11:31:31.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:31.792+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.792922+0000) 2019-10-13T11:31:31.801 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:31.799+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211420 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:33.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:33.051+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:33.051439+0000) 2019-10-13T11:31:33.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:33.054+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:33.055131+0000) 2019-10-13T11:31:35.497 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:35.495+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.496578+0000) 2019-10-13T11:31:35.707 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:35.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.706676+0000) 2019-10-13T11:31:35.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:35.706+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.706703+0000) 2019-10-13T11:31:36.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:36.801+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.801654+0000) 2019-10-13T11:31:36.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:36.801+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.801740+0000) 2019-10-13T11:31:36.804 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:36.801+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.802106+0000) 2019-10-13T11:31:36.804 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:36.801+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.802264+0000) 2019-10-13T11:31:36.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:36.802+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.802473+0000) 2019-10-13T11:31:36.805 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:36.802+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.802822+0000) 2019-10-13T11:31:36.805 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:36.802+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.803223+0000) 2019-10-13T11:31:36.806 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:36.803+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:36.804262+0000) 2019-10-13T11:31:36.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:36.808+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:38.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:38.613+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:38.613457+0000) 2019-10-13T11:31:38.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:38.946+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:38.946491+0000) 2019-10-13T11:31:41.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:41.564+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.564708+0000) 2019-10-13T11:31:41.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:41.809+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.810048+0000) 2019-10-13T11:31:41.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:41.810+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.810449+0000) 2019-10-13T11:31:41.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:41.810+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.810661+0000) 2019-10-13T11:31:41.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:41.810+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.811338+0000) 2019-10-13T11:31:41.813 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:41.810+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.811032+0000) 2019-10-13T11:31:41.813 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:41.811+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.811645+0000) 2019-10-13T11:31:41.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:41.811+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.811965+0000) 2019-10-13T11:31:41.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:41.812+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.813280+0000) 2019-10-13T11:31:41.820 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:41.819+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:43.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:43.051+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:43.051647+0000) 2019-10-13T11:31:43.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:43.054+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:43.055340+0000) 2019-10-13T11:31:45.499 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:45.498+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.498920+0000) 2019-10-13T11:31:45.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:45.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.706899+0000) 2019-10-13T11:31:45.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:45.706+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.706920+0000) 2019-10-13T11:31:46.822 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:46.820+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.821144+0000) 2019-10-13T11:31:46.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:46.820+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.821257+0000) 2019-10-13T11:31:46.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:46.821+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.821487+0000) 2019-10-13T11:31:46.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:46.821+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.822154+0000) 2019-10-13T11:31:46.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:46.821+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.821908+0000) 2019-10-13T11:31:46.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:46.821+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.822302+0000) 2019-10-13T11:31:46.824 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:46.822+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.822495+0000) 2019-10-13T11:31:46.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:46.823+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:46.824458+0000) 2019-10-13T11:31:46.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:46.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:48.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:48.613+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:48.613607+0000) 2019-10-13T11:31:48.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:48.946+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:48.946686+0000) 2019-10-13T11:31:51.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:51.563+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.564887+0000) 2019-10-13T11:31:51.832 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:51.829+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.830866+0000) 2019-10-13T11:31:51.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:51.829+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.831153+0000) 2019-10-13T11:31:51.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:51.830+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.831450+0000) 2019-10-13T11:31:51.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:51.831+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.832117+0000) 2019-10-13T11:31:51.833 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:51.830+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.831686+0000) 2019-10-13T11:31:51.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:51.830+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.831919+0000) 2019-10-13T11:31:51.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:51.830+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.832248+0000) 2019-10-13T11:31:51.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:51.834+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.835198+0000) 2019-10-13T11:31:51.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:51.837+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211545 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:53.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:53.050+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:53.051829+0000) 2019-10-13T11:31:53.055 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:53.054+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:53.055541+0000) 2019-10-13T11:31:55.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:55.501+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.501332+0000) 2019-10-13T11:31:55.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:55.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.707107+0000) 2019-10-13T11:31:55.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:55.705+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.707117+0000) 2019-10-13T11:31:56.842 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:56.839+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.840791+0000) 2019-10-13T11:31:56.842 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:56.839+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.841028+0000) 2019-10-13T11:31:56.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:31:56.839+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.841329+0000) 2019-10-13T11:31:56.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:31:56.841+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.842500+0000) 2019-10-13T11:31:56.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:31:56.840+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.841650+0000) 2019-10-13T11:31:56.843 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:31:56.840+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.841786+0000) 2019-10-13T11:31:56.843 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:31:56.840+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.842342+0000) 2019-10-13T11:31:56.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:31:56.843+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:56.844532+0000) 2019-10-13T11:31:56.850 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:31:56.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211579 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:31:57.685 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:31:57.685 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:57.689 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:31:57.689 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:58.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:31:58.612+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:58.613825+0000) 2019-10-13T11:31:58.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:31:58.946+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:58.946878+0000) 2019-10-13T11:32:01.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:01.564+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.565047+0000) 2019-10-13T11:32:01.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:01.849+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.851106+0000) 2019-10-13T11:32:01.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:01.849+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.851302+0000) 2019-10-13T11:32:01.853 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:01.850+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.851677+0000) 2019-10-13T11:32:01.853 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:01.850+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.852111+0000) 2019-10-13T11:32:01.853 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:01.850+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.852332+0000) 2019-10-13T11:32:01.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:01.850+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.851499+0000) 2019-10-13T11:32:01.854 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:01.852+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.852951+0000) 2019-10-13T11:32:01.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:01.853+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.854239+0000) 2019-10-13T11:32:01.858 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:01.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:03.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:03.051+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:03.052030+0000) 2019-10-13T11:32:03.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:03.055+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:03.055746+0000) 2019-10-13T11:32:05.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:05.503+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.503751+0000) 2019-10-13T11:32:05.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:05.705+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.707294+0000) 2019-10-13T11:32:05.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:05.705+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.707294+0000) 2019-10-13T11:32:06.861 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:06.858+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.860132+0000) 2019-10-13T11:32:06.861 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:06.858+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.860352+0000) 2019-10-13T11:32:06.862 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:06.860+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.861029+0000) 2019-10-13T11:32:06.862 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:06.859+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.860834+0000) 2019-10-13T11:32:06.863 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:06.859+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.861077+0000) 2019-10-13T11:32:06.863 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:06.859+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.861104+0000) 2019-10-13T11:32:06.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:06.860+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.861601+0000) 2019-10-13T11:32:06.870 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:06.867+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:06.870 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:06.863+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:06.863982+0000) 2019-10-13T11:32:08.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:08.613+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:08.614010+0000) 2019-10-13T11:32:08.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:08.945+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:08.947007+0000) 2019-10-13T11:32:11.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:11.564+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.565209+0000) 2019-10-13T11:32:11.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:11.869+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.871393+0000) 2019-10-13T11:32:11.873 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:11.869+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.871227+0000) 2019-10-13T11:32:11.874 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:11.870+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.871538+0000) 2019-10-13T11:32:11.874 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:11.870+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.871691+0000) 2019-10-13T11:32:11.874 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:11.870+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.872178+0000) 2019-10-13T11:32:11.875 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:11.871+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.872466+0000) 2019-10-13T11:32:11.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:11.871+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.872748+0000) 2019-10-13T11:32:11.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:11.872+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.874669+0000) 2019-10-13T11:32:11.880 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:11.878+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:13.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:13.051+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:13.052178+0000) 2019-10-13T11:32:13.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:13.054+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:13.055906+0000) 2019-10-13T11:32:15.507 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:15.505+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.506662+0000) 2019-10-13T11:32:15.708 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:15.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.707484+0000) 2019-10-13T11:32:15.708 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:15.707+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.707486+0000) 2019-10-13T11:32:16.882 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:16.880+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.881560+0000) 2019-10-13T11:32:16.883 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:16.880+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.881886+0000) 2019-10-13T11:32:16.883 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:16.880+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.882542+0000) 2019-10-13T11:32:16.884 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:16.881+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.882482+0000) 2019-10-13T11:32:16.884 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:16.881+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.882681+0000) 2019-10-13T11:32:16.884 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:16.881+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.882905+0000) 2019-10-13T11:32:16.884 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:16.881+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.883157+0000) 2019-10-13T11:32:16.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:16.883+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:16.885417+0000) 2019-10-13T11:32:16.892 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:16.888+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211705 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:18.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:18.613+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:18.614199+0000) 2019-10-13T11:32:18.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:18.946+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:18.947176+0000) 2019-10-13T11:32:21.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:21.565+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.565392+0000) 2019-10-13T11:32:21.892 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:21.891+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.892210+0000) 2019-10-13T11:32:21.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:21.891+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.892139+0000) 2019-10-13T11:32:21.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:21.891+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.892281+0000) 2019-10-13T11:32:21.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:21.892+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.892545+0000) 2019-10-13T11:32:21.894 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:21.892+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.892666+0000) 2019-10-13T11:32:21.894 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:21.892+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.892905+0000) 2019-10-13T11:32:21.894 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:21.892+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.893176+0000) 2019-10-13T11:32:21.896 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:21.895+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.896027+0000) 2019-10-13T11:32:21.900 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:21.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:23.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:23.052+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:23.052368+0000) 2019-10-13T11:32:23.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:23.055+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:23.056098+0000) 2019-10-13T11:32:25.510 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:25.509+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.509725+0000) 2019-10-13T11:32:25.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:25.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.707675+0000) 2019-10-13T11:32:25.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:25.707+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.707674+0000) 2019-10-13T11:32:26.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:26.901+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.902632+0000) 2019-10-13T11:32:26.904 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:26.901+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.901765+0000) 2019-10-13T11:32:26.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:26.901+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.902274+0000) 2019-10-13T11:32:26.904 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:26.902+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.902498+0000) 2019-10-13T11:32:26.904 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:26.902+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.902676+0000) 2019-10-13T11:32:26.905 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:26.902+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.903096+0000) 2019-10-13T11:32:26.905 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:26.902+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.903031+0000) 2019-10-13T11:32:26.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:26.904+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:26.904983+0000) 2019-10-13T11:32:26.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:26.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211769 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:27.845 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:32:27.846 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:27.851 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:32:27.851 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:28.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:28.614+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:28.614393+0000) 2019-10-13T11:32:28.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:28.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:28.947333+0000) 2019-10-13T11:32:31.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:31.565+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.565574+0000) 2019-10-13T11:32:31.913 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:31.912+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.912802+0000) 2019-10-13T11:32:31.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:31.912+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.912492+0000) 2019-10-13T11:32:31.914 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:31.912+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.912714+0000) 2019-10-13T11:32:31.914 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:31.912+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.913058+0000) 2019-10-13T11:32:31.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:31.912+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.913227+0000) 2019-10-13T11:32:31.915 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:31.913+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.913677+0000) 2019-10-13T11:32:31.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:31.913+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.913825+0000) 2019-10-13T11:32:31.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:31.914+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.915460+0000) 2019-10-13T11:32:31.921 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:31.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:33.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:33.052+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:33.052569+0000) 2019-10-13T11:32:33.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:33.055+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:33.056256+0000) 2019-10-13T11:32:35.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:35.511+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.512062+0000) 2019-10-13T11:32:35.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:35.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.707861+0000) 2019-10-13T11:32:35.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:35.706+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.707882+0000) 2019-10-13T11:32:36.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:36.922+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.923233+0000) 2019-10-13T11:32:36.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:36.921+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.922952+0000) 2019-10-13T11:32:36.925 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:36.921+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.923127+0000) 2019-10-13T11:32:36.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:36.921+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.923258+0000) 2019-10-13T11:32:36.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:36.922+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.923472+0000) 2019-10-13T11:32:36.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:36.922+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.923665+0000) 2019-10-13T11:32:36.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:36.922+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.923958+0000) 2019-10-13T11:32:36.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:36.924+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:36.925091+0000) 2019-10-13T11:32:36.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:36.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211831 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:38.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:38.613+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:38.614547+0000) 2019-10-13T11:32:38.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:38.946+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:38.947520+0000) 2019-10-13T11:32:41.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:41.564+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.565705+0000) 2019-10-13T11:32:41.934 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:41.931+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.933306+0000) 2019-10-13T11:32:41.934 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:41.932+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.933484+0000) 2019-10-13T11:32:41.934 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:41.932+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.933624+0000) 2019-10-13T11:32:41.935 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:41.933+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.934477+0000) 2019-10-13T11:32:41.935 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:41.932+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.934022+0000) 2019-10-13T11:32:41.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:41.932+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.934147+0000) 2019-10-13T11:32:41.936 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:41.932+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.934362+0000) 2019-10-13T11:32:41.936 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:41.935+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.936486+0000) 2019-10-13T11:32:41.940 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:41.938+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:43.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:43.051+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:43.052745+0000) 2019-10-13T11:32:43.056 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:43.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:43.056402+0000) 2019-10-13T11:32:45.515 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:45.513+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.514490+0000) 2019-10-13T11:32:45.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:45.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.708045+0000) 2019-10-13T11:32:45.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:45.707+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.708067+0000) 2019-10-13T11:32:46.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:46.940+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.941769+0000) 2019-10-13T11:32:46.943 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:46.940+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.942061+0000) 2019-10-13T11:32:46.944 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:46.942+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.943039+0000) 2019-10-13T11:32:46.944 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:46.940+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.942248+0000) 2019-10-13T11:32:46.944 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:46.941+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.942751+0000) 2019-10-13T11:32:46.944 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:46.941+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.943090+0000) 2019-10-13T11:32:46.944 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:46.942+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.943447+0000) 2019-10-13T11:32:46.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:46.944+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:46.945540+0000) 2019-10-13T11:32:46.951 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:46.949+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:48.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:48.613+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:48.614728+0000) 2019-10-13T11:32:48.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:48.946+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:48.947664+0000) 2019-10-13T11:32:51.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:51.565+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.565878+0000) 2019-10-13T11:32:51.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:51.950+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.952198+0000) 2019-10-13T11:32:51.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:51.951+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.952414+0000) 2019-10-13T11:32:51.954 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:51.951+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.952743+0000) 2019-10-13T11:32:51.954 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:51.951+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.952828+0000) 2019-10-13T11:32:51.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:51.951+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.953109+0000) 2019-10-13T11:32:51.955 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:51.953+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.953979+0000) 2019-10-13T11:32:51.955 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:51.952+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.954021+0000) 2019-10-13T11:32:51.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:51.955+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.956334+0000) 2019-10-13T11:32:51.962 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:51.959+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:53.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:53.052+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:53.052921+0000) 2019-10-13T11:32:53.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:53.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:53.056560+0000) 2019-10-13T11:32:55.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:55.516+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.516923+0000) 2019-10-13T11:32:55.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:55.706+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.708232+0000) 2019-10-13T11:32:55.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:55.706+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.708225+0000) 2019-10-13T11:32:56.964 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:56.961+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.963188+0000) 2019-10-13T11:32:56.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:32:56.961+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.963389+0000) 2019-10-13T11:32:56.965 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:32:56.962+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.963925+0000) 2019-10-13T11:32:56.965 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:32:56.962+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.964193+0000) 2019-10-13T11:32:56.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:56.962+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.964028+0000) 2019-10-13T11:32:56.966 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:32:56.963+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.964587+0000) 2019-10-13T11:32:56.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:32:56.964+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.964928+0000) 2019-10-13T11:32:56.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:32:56.966+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:56.967406+0000) 2019-10-13T11:32:56.973 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:32:56.971+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211957 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:32:57.933 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:32:57.933 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:57.938 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:32:57.938 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:58.615 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:32:58.614+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:58.614904+0000) 2019-10-13T11:32:58.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:32:58.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:58.947781+0000) 2019-10-13T11:33:01.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:01.565+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.566085+0000) 2019-10-13T11:33:01.975 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:01.972+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.974076+0000) 2019-10-13T11:33:01.976 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:01.973+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.974917+0000) 2019-10-13T11:33:01.976 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:01.973+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.974826+0000) 2019-10-13T11:33:01.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:01.973+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.975033+0000) 2019-10-13T11:33:01.976 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:01.973+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.975247+0000) 2019-10-13T11:33:01.977 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:01.974+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.975707+0000) 2019-10-13T11:33:01.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:01.974+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.975881+0000) 2019-10-13T11:33:01.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:01.976+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.977947+0000) 2019-10-13T11:33:01.984 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:01.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 211987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:03.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:03.052+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:03.053102+0000) 2019-10-13T11:33:03.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:03.055+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:03.056709+0000) 2019-10-13T11:33:05.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:05.518+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.519341+0000) 2019-10-13T11:33:05.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:05.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.708427+0000) 2019-10-13T11:33:05.709 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:05.708+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.708421+0000) 2019-10-13T11:33:06.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:06.983+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.985474+0000) 2019-10-13T11:33:06.987 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:06.984+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.985150+0000) 2019-10-13T11:33:06.987 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:06.985+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.985460+0000) 2019-10-13T11:33:06.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:06.985+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.985712+0000) 2019-10-13T11:33:06.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:06.985+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.985779+0000) 2019-10-13T11:33:06.987 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:06.985+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.985987+0000) 2019-10-13T11:33:06.987 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:06.985+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.986273+0000) 2019-10-13T11:33:06.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:06.986+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:06.988543+0000) 2019-10-13T11:33:06.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:06.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:08.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:08.614+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:08.615138+0000) 2019-10-13T11:33:08.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:08.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:08.947941+0000) 2019-10-13T11:33:11.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:11.566+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.566294+0000) 2019-10-13T11:33:11.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:11.995+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.996054+0000) 2019-10-13T11:33:11.997 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:11.995+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.996069+0000) 2019-10-13T11:33:11.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:11.995+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.996136+0000) 2019-10-13T11:33:11.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:11.996+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.996470+0000) 2019-10-13T11:33:11.998 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:11.996+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.996623+0000) 2019-10-13T11:33:11.998 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:11.996+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.996793+0000) 2019-10-13T11:33:11.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:11.997+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.998385+0000) 2019-10-13T11:33:11.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:11.996+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.997019+0000) 2019-10-13T11:33:12.005 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:12.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:13.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:13.053+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:13.053285+0000) 2019-10-13T11:33:13.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:13.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:13.056864+0000) 2019-10-13T11:33:15.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:15.520+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.521612+0000) 2019-10-13T11:33:15.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:15.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.708616+0000) 2019-10-13T11:33:15.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:15.708+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.708617+0000) 2019-10-13T11:33:17.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:17.006+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.006633+0000) 2019-10-13T11:33:17.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:17.006+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.006776+0000) 2019-10-13T11:33:17.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:17.006+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.007707+0000) 2019-10-13T11:33:17.009 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:17.007+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.007492+0000) 2019-10-13T11:33:17.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:17.007+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.007677+0000) 2019-10-13T11:33:17.009 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:17.007+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.007811+0000) 2019-10-13T11:33:17.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:17.008+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.008579+0000) 2019-10-13T11:33:17.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:17.011+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.011797+0000) 2019-10-13T11:33:17.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:17.015+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:18.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:18.615+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:18.615311+0000) 2019-10-13T11:33:18.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:18.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:18.948078+0000) 2019-10-13T11:33:21.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:21.565+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:21.566494+0000) 2019-10-13T11:33:22.019 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:22.016+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.017879+0000) 2019-10-13T11:33:22.019 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:22.016+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.018011+0000) 2019-10-13T11:33:22.019 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:22.016+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.018257+0000) 2019-10-13T11:33:22.019 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:22.016+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.018354+0000) 2019-10-13T11:33:22.020 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:22.018+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.019186+0000) 2019-10-13T11:33:22.020 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:22.017+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.018992+0000) 2019-10-13T11:33:22.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:22.017+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.019232+0000) 2019-10-13T11:33:22.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:22.020+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.021419+0000) 2019-10-13T11:33:22.027 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:22.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:23.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:23.052+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:23.053447+0000) 2019-10-13T11:33:23.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:23.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:23.057033+0000) 2019-10-13T11:33:25.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:25.523+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.523929+0000) 2019-10-13T11:33:25.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:25.707+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.708825+0000) 2019-10-13T11:33:25.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:25.707+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.708827+0000) 2019-10-13T11:33:27.029 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:27.028+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.029272+0000) 2019-10-13T11:33:27.030 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:27.027+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.029332+0000) 2019-10-13T11:33:27.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:27.028+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.029526+0000) 2019-10-13T11:33:27.030 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:27.028+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.029718+0000) 2019-10-13T11:33:27.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:27.028+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.030018+0000) 2019-10-13T11:33:27.031 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:27.028+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.029671+0000) 2019-10-13T11:33:27.031 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:27.028+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.030280+0000) 2019-10-13T11:33:27.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:27.031+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.031788+0000) 2019-10-13T11:33:27.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:27.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:28.065 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:33:28.066 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:28.070 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:33:28.070 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:28.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:28.614+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:28.615502+0000) 2019-10-13T11:33:28.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:28.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:28.948232+0000) 2019-10-13T11:33:31.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:31.565+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:31.566639+0000) 2019-10-13T11:33:32.040 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:32.037+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.039296+0000) 2019-10-13T11:33:32.040 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:32.038+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.039433+0000) 2019-10-13T11:33:32.041 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:32.038+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.039551+0000) 2019-10-13T11:33:32.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:32.039+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.040349+0000) 2019-10-13T11:33:32.042 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:32.038+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.040024+0000) 2019-10-13T11:33:32.042 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:32.039+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.040432+0000) 2019-10-13T11:33:32.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:32.039+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.040590+0000) 2019-10-13T11:33:32.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:32.041+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.042371+0000) 2019-10-13T11:33:32.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:32.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212175 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:33.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:33.052+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:33.053600+0000) 2019-10-13T11:33:33.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:33.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:33.057217+0000) 2019-10-13T11:33:35.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:35.525+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.526210+0000) 2019-10-13T11:33:35.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:35.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.709039+0000) 2019-10-13T11:33:35.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:35.708+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.709036+0000) 2019-10-13T11:33:37.051 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:37.049+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.050453+0000) 2019-10-13T11:33:37.051 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:37.049+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.050673+0000) 2019-10-13T11:33:37.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:37.049+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.050876+0000) 2019-10-13T11:33:37.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:37.051+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.051751+0000) 2019-10-13T11:33:37.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:37.049+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.051331+0000) 2019-10-13T11:33:37.053 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:37.050+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.051664+0000) 2019-10-13T11:33:37.053 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:37.050+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.051556+0000) 2019-10-13T11:33:37.055 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:37.053+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.054567+0000) 2019-10-13T11:33:37.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:37.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212209 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:38.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:38.615+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:38.615667+0000) 2019-10-13T11:33:38.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:38.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:38.948437+0000) 2019-10-13T11:33:41.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:41.566+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:41.566829+0000) 2019-10-13T11:33:42.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:42.059+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.061152+0000) 2019-10-13T11:33:42.062 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:42.059+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.061307+0000) 2019-10-13T11:33:42.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:42.060+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.061484+0000) 2019-10-13T11:33:42.063 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:42.060+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.061601+0000) 2019-10-13T11:33:42.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:42.062+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.062924+0000) 2019-10-13T11:33:42.064 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:42.060+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.062043+0000) 2019-10-13T11:33:42.064 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:42.060+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.062157+0000) 2019-10-13T11:33:42.064 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:42.063+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.064349+0000) 2019-10-13T11:33:42.070 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:42.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:43.054 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:43.053+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:43.053769+0000) 2019-10-13T11:33:43.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:43.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:43.057393+0000) 2019-10-13T11:33:45.528 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:45.528+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.528452+0000) 2019-10-13T11:33:45.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:45.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.709202+0000) 2019-10-13T11:33:45.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:45.708+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.709219+0000) 2019-10-13T11:33:47.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:47.070+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.071600+0000) 2019-10-13T11:33:47.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:47.070+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.071835+0000) 2019-10-13T11:33:47.073 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:47.070+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.072057+0000) 2019-10-13T11:33:47.073 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:47.070+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.072126+0000) 2019-10-13T11:33:47.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:47.070+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.072262+0000) 2019-10-13T11:33:47.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:47.071+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.072424+0000) 2019-10-13T11:33:47.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:47.072+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.073003+0000) 2019-10-13T11:33:47.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:47.074+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.075539+0000) 2019-10-13T11:33:47.081 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:47.078+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:48.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:48.615+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:48.615842+0000) 2019-10-13T11:33:48.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:48.948+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:48.948595+0000) 2019-10-13T11:33:51.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:51.566+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:51.566974+0000) 2019-10-13T11:33:52.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:52.081+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.081948+0000) 2019-10-13T11:33:52.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:52.080+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.082581+0000) 2019-10-13T11:33:52.084 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:52.082+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.082405+0000) 2019-10-13T11:33:52.084 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:52.081+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.082333+0000) 2019-10-13T11:33:52.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:52.082+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.082617+0000) 2019-10-13T11:33:52.084 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:52.082+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.082996+0000) 2019-10-13T11:33:52.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:52.082+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.083139+0000) 2019-10-13T11:33:52.085 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:52.083+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.085246+0000) 2019-10-13T11:33:52.091 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:52.090+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212301 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:53.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:53.053+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:53.053970+0000) 2019-10-13T11:33:53.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:53.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:53.057551+0000) 2019-10-13T11:33:55.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:55.529+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.530776+0000) 2019-10-13T11:33:55.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:55.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.709387+0000) 2019-10-13T11:33:55.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:55.709+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.709393+0000) 2019-10-13T11:33:57.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:33:57.091+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.093416+0000) 2019-10-13T11:33:57.094 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:33:57.092+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.092827+0000) 2019-10-13T11:33:57.095 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:57.092+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.093166+0000) 2019-10-13T11:33:57.095 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:33:57.092+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.093327+0000) 2019-10-13T11:33:57.095 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:57.093+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.093731+0000) 2019-10-13T11:33:57.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:33:57.093+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.093978+0000) 2019-10-13T11:33:57.095 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:33:57.093+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.093913+0000) 2019-10-13T11:33:57.096 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:33:57.094+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.095859+0000) 2019-10-13T11:33:57.102 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:33:57.101+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:33:58.217 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:33:58.217 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:58.221 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:33:58.222 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:58.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:33:58.615+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:58.616062+0000) 2019-10-13T11:33:58.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:33:58.948+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:58.948763+0000) 2019-10-13T11:34:01.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:01.566+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:01.567117+0000) 2019-10-13T11:34:02.104 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:02.102+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.103331+0000) 2019-10-13T11:34:02.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:02.103+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.104508+0000) 2019-10-13T11:34:02.105 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:02.103+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.103852+0000) 2019-10-13T11:34:02.105 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:02.103+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.104185+0000) 2019-10-13T11:34:02.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:02.103+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.104355+0000) 2019-10-13T11:34:02.106 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:02.104+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.104483+0000) 2019-10-13T11:34:02.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:02.105+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.106328+0000) 2019-10-13T11:34:02.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:02.104+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.104864+0000) 2019-10-13T11:34:02.113 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:02.112+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:03.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:03.054+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:03.054131+0000) 2019-10-13T11:34:03.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:03.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:03.057712+0000) 2019-10-13T11:34:05.533 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:05.532+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.533113+0000) 2019-10-13T11:34:05.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:05.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.709630+0000) 2019-10-13T11:34:05.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:05.709+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.709630+0000) 2019-10-13T11:34:07.115 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:07.113+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.114587+0000) 2019-10-13T11:34:07.116 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:07.113+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.114821+0000) 2019-10-13T11:34:07.116 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:07.113+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.114934+0000) 2019-10-13T11:34:07.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:07.115+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.116284+0000) 2019-10-13T11:34:07.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:07.113+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.115162+0000) 2019-10-13T11:34:07.117 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:07.113+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.115264+0000) 2019-10-13T11:34:07.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:07.114+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.115632+0000) 2019-10-13T11:34:07.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:07.118+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.118734+0000) 2019-10-13T11:34:07.124 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:07.122+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:08.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:08.614+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:08.616286+0000) 2019-10-13T11:34:08.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:08.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:08.948935+0000) 2019-10-13T11:34:11.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:11.566+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.567283+0000) 2019-10-13T11:34:12.126 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:12.124+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.125401+0000) 2019-10-13T11:34:12.127 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:12.124+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.125715+0000) 2019-10-13T11:34:12.127 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:12.124+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.125863+0000) 2019-10-13T11:34:12.127 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:12.124+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.125999+0000) 2019-10-13T11:34:12.128 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:12.126+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.126750+0000) 2019-10-13T11:34:12.128 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:12.125+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.126974+0000) 2019-10-13T11:34:12.128 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:12.125+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.127063+0000) 2019-10-13T11:34:12.129 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:12.128+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.129406+0000) 2019-10-13T11:34:12.135 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:12.133+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212427 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:13.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:13.053+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:13.054267+0000) 2019-10-13T11:34:13.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:13.057+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:13.057891+0000) 2019-10-13T11:34:15.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:15.534+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.535498+0000) 2019-10-13T11:34:15.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:15.708+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.709811+0000) 2019-10-13T11:34:15.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:15.708+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:15.709812+0000) 2019-10-13T11:34:17.137 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:17.135+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.136818+0000) 2019-10-13T11:34:17.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:17.135+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.137022+0000) 2019-10-13T11:34:17.139 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:17.136+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.137658+0000) 2019-10-13T11:34:17.139 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:17.135+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.137267+0000) 2019-10-13T11:34:17.139 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:17.136+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.137434+0000) 2019-10-13T11:34:17.139 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:17.136+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.137717+0000) 2019-10-13T11:34:17.139 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:17.136+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.137749+0000) 2019-10-13T11:34:17.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:17.138+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.139094+0000) 2019-10-13T11:34:17.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:17.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:18.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:18.615+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:18.616476+0000) 2019-10-13T11:34:18.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:18.947+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:18.949119+0000) 2019-10-13T11:34:21.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:21.566+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.567510+0000) 2019-10-13T11:34:22.148 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:22.145+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.147317+0000) 2019-10-13T11:34:22.148 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:22.146+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.147520+0000) 2019-10-13T11:34:22.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:22.146+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.147541+0000) 2019-10-13T11:34:22.149 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:22.148+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.148788+0000) 2019-10-13T11:34:22.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:22.147+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.148424+0000) 2019-10-13T11:34:22.150 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:22.147+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.148778+0000) 2019-10-13T11:34:22.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:22.147+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.149104+0000) 2019-10-13T11:34:22.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:22.150+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.151162+0000) 2019-10-13T11:34:22.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:22.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:23.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:23.053+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:23.054411+0000) 2019-10-13T11:34:23.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:23.057+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:23.058067+0000) 2019-10-13T11:34:25.538 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:25.537+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.538046+0000) 2019-10-13T11:34:25.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:25.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.709974+0000) 2019-10-13T11:34:25.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:25.709+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:25.710028+0000) 2019-10-13T11:34:27.159 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:27.157+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.158677+0000) 2019-10-13T11:34:27.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:27.158+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.159435+0000) 2019-10-13T11:34:27.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:27.157+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.159134+0000) 2019-10-13T11:34:27.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:27.157+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.159352+0000) 2019-10-13T11:34:27.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:27.158+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.159499+0000) 2019-10-13T11:34:27.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:27.158+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.159726+0000) 2019-10-13T11:34:27.161 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:27.158+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.159835+0000) 2019-10-13T11:34:27.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:27.161+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.162649+0000) 2019-10-13T11:34:27.168 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:27.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212525 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:28.511 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:34:28.511 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:28.516 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:34:28.516 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:28.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:28.616+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:28.616609+0000) 2019-10-13T11:34:28.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:28.948+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:28.949336+0000) 2019-10-13T11:34:31.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:31.567+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.567686+0000) 2019-10-13T11:34:32.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:32.167+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.169202+0000) 2019-10-13T11:34:32.170 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:32.168+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.169559+0000) 2019-10-13T11:34:32.171 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:32.169+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.170270+0000) 2019-10-13T11:34:32.171 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:32.168+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.169941+0000) 2019-10-13T11:34:32.172 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:32.168+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.170082+0000) 2019-10-13T11:34:32.172 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:32.168+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.170294+0000) 2019-10-13T11:34:32.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:32.169+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.170522+0000) 2019-10-13T11:34:32.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:32.172+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.173003+0000) 2019-10-13T11:34:32.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:32.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:33.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:33.054+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:33.054548+0000) 2019-10-13T11:34:33.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:33.058+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:33.058218+0000) 2019-10-13T11:34:35.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:35.540+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.540536+0000) 2019-10-13T11:34:35.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:35.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.710191+0000) 2019-10-13T11:34:35.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:35.709+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:35.710200+0000) 2019-10-13T11:34:37.180 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:37.179+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.179714+0000) 2019-10-13T11:34:37.181 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:37.179+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.180523+0000) 2019-10-13T11:34:37.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:37.179+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.180254+0000) 2019-10-13T11:34:37.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:37.180+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.180545+0000) 2019-10-13T11:34:37.182 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:37.180+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.180705+0000) 2019-10-13T11:34:37.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:37.180+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.180865+0000) 2019-10-13T11:34:37.182 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:37.180+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.180994+0000) 2019-10-13T11:34:37.184 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:37.183+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.183913+0000) 2019-10-13T11:34:37.188 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:37.187+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:38.617 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:38.616+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:38.616804+0000) 2019-10-13T11:34:38.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:38.949+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:38.949656+0000) 2019-10-13T11:34:41.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:41.567+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.567874+0000) 2019-10-13T11:34:42.191 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:42.188+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.189314+0000) 2019-10-13T11:34:42.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:42.189+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.189511+0000) 2019-10-13T11:34:42.192 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:42.189+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.189790+0000) 2019-10-13T11:34:42.192 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:42.189+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.189730+0000) 2019-10-13T11:34:42.192 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:42.189+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.190319+0000) 2019-10-13T11:34:42.192 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:42.190+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.190484+0000) 2019-10-13T11:34:42.193 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:42.188+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.190721+0000) 2019-10-13T11:34:42.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:42.190+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.192487+0000) 2019-10-13T11:34:42.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:42.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:43.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:43.054+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:43.054746+0000) 2019-10-13T11:34:43.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:43.056+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:43.058368+0000) 2019-10-13T11:34:45.543 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:45.541+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.542863+0000) 2019-10-13T11:34:45.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:45.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.710430+0000) 2019-10-13T11:34:45.711 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:45.710+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:45.710436+0000) 2019-10-13T11:34:47.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:47.197+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.198198+0000) 2019-10-13T11:34:47.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:47.197+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.198334+0000) 2019-10-13T11:34:47.200 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:47.198+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.198582+0000) 2019-10-13T11:34:47.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:47.198+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.198836+0000) 2019-10-13T11:34:47.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:47.197+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.199213+0000) 2019-10-13T11:34:47.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:47.198+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.199073+0000) 2019-10-13T11:34:47.201 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:47.198+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.199137+0000) 2019-10-13T11:34:47.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:47.201+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.203188+0000) 2019-10-13T11:34:47.208 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:47.206+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:48.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:48.616+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:48.617038+0000) 2019-10-13T11:34:48.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:48.949+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:48.949772+0000) 2019-10-13T11:34:51.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:51.568+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.568065+0000) 2019-10-13T11:34:52.210 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:52.207+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.209457+0000) 2019-10-13T11:34:52.211 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:52.208+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.209130+0000) 2019-10-13T11:34:52.211 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:52.208+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.209354+0000) 2019-10-13T11:34:52.211 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:52.208+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.209282+0000) 2019-10-13T11:34:52.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:52.209+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.209504+0000) 2019-10-13T11:34:52.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:52.209+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.209557+0000) 2019-10-13T11:34:52.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:52.209+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.209851+0000) 2019-10-13T11:34:52.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:52.211+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.212259+0000) 2019-10-13T11:34:52.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:52.214+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212679 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:53.055 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:53.053+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:53.054878+0000) 2019-10-13T11:34:53.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:53.057+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:53.058548+0000) 2019-10-13T11:34:55.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:55.544+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.545149+0000) 2019-10-13T11:34:55.711 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:55.709+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.710622+0000) 2019-10-13T11:34:55.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:55.709+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:55.710648+0000) 2019-10-13T11:34:57.218 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:57.215+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.216952+0000) 2019-10-13T11:34:57.219 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:34:57.215+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.217045+0000) 2019-10-13T11:34:57.219 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:57.216+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.217433+0000) 2019-10-13T11:34:57.219 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:34:57.216+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.217688+0000) 2019-10-13T11:34:57.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:34:57.216+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.217795+0000) 2019-10-13T11:34:57.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:34:57.217+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.218135+0000) 2019-10-13T11:34:57.221 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:34:57.216+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.218273+0000) 2019-10-13T11:34:57.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:34:57.220+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.221612+0000) 2019-10-13T11:34:57.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:34:57.221+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212713 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:34:58.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:34:58.616+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:58.617163+0000) 2019-10-13T11:34:58.742 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:34:58.743 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:58.747 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:34:58.747 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:58.950 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:34:58.949+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:58.949988+0000) 2019-10-13T11:35:01.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:01.567+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.568196+0000) 2019-10-13T11:35:02.226 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:02.223+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.225208+0000) 2019-10-13T11:35:02.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:02.223+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.225370+0000) 2019-10-13T11:35:02.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:02.224+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.225575+0000) 2019-10-13T11:35:02.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:02.224+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.225840+0000) 2019-10-13T11:35:02.228 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:02.226+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.227086+0000) 2019-10-13T11:35:02.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:02.224+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.226121+0000) 2019-10-13T11:35:02.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:02.225+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.226481+0000) 2019-10-13T11:35:02.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:02.227+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.228502+0000) 2019-10-13T11:35:02.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:02.232+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212743 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:03.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:03.054+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:03.055045+0000) 2019-10-13T11:35:03.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:03.058+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:03.058729+0000) 2019-10-13T11:35:05.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:05.546+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.547514+0000) 2019-10-13T11:35:05.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:05.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.710817+0000) 2019-10-13T11:35:05.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:05.710+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:05.710847+0000) 2019-10-13T11:35:07.237 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:07.234+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.235730+0000) 2019-10-13T11:35:07.237 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:07.234+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.235808+0000) 2019-10-13T11:35:07.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:07.234+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.235847+0000) 2019-10-13T11:35:07.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:07.234+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.236120+0000) 2019-10-13T11:35:07.238 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:07.234+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.236249+0000) 2019-10-13T11:35:07.238 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:07.235+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.236496+0000) 2019-10-13T11:35:07.239 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:07.235+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.236913+0000) 2019-10-13T11:35:07.239 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:07.238+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.239306+0000) 2019-10-13T11:35:07.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:07.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212777 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:08.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:08.616+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:08.617341+0000) 2019-10-13T11:35:08.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:08.949+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:08.950166+0000) 2019-10-13T11:35:11.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:11.567+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.568386+0000) 2019-10-13T11:35:12.245 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:12.242+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.244179+0000) 2019-10-13T11:35:12.246 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:12.243+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.244458+0000) 2019-10-13T11:35:12.246 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:12.243+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.244683+0000) 2019-10-13T11:35:12.246 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:12.243+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.244938+0000) 2019-10-13T11:35:12.246 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:12.243+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.244964+0000) 2019-10-13T11:35:12.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:12.244+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.245298+0000) 2019-10-13T11:35:12.247 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:12.243+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.245248+0000) 2019-10-13T11:35:12.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:12.247+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.247945+0000) 2019-10-13T11:35:12.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:12.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212805 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:13.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:13.054+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:13.055201+0000) 2019-10-13T11:35:13.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:13.058+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:13.058895+0000) 2019-10-13T11:35:15.550 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:15.549+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.549985+0000) 2019-10-13T11:35:15.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:15.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.711023+0000) 2019-10-13T11:35:15.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:15.710+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:15.711045+0000) 2019-10-13T11:35:17.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:17.250+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.252300+0000) 2019-10-13T11:35:17.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:17.251+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.252570+0000) 2019-10-13T11:35:17.254 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:17.251+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.252740+0000) 2019-10-13T11:35:17.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:17.251+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.252984+0000) 2019-10-13T11:35:17.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:17.251+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.253104+0000) 2019-10-13T11:35:17.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:17.253+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.254245+0000) 2019-10-13T11:35:17.255 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:17.251+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.253271+0000) 2019-10-13T11:35:17.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:17.255+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.256446+0000) 2019-10-13T11:35:17.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:17.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:18.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:18.617+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:18.617546+0000) 2019-10-13T11:35:18.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:18.949+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:18.950357+0000) 2019-10-13T11:35:21.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:21.568+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.568557+0000) 2019-10-13T11:35:22.264 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:22.262+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.263046+0000) 2019-10-13T11:35:22.265 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:22.263+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.263490+0000) 2019-10-13T11:35:22.265 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:22.263+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.263743+0000) 2019-10-13T11:35:22.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:22.263+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.264447+0000) 2019-10-13T11:35:22.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:22.263+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.264151+0000) 2019-10-13T11:35:22.267 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:22.263+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.264277+0000) 2019-10-13T11:35:22.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:22.264+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.264421+0000) 2019-10-13T11:35:22.268 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:22.265+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.266629+0000) 2019-10-13T11:35:22.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:22.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212869 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:23.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:23.055+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:23.055386+0000) 2019-10-13T11:35:23.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:23.058+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:23.059060+0000) 2019-10-13T11:35:25.553 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:25.552+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.552380+0000) 2019-10-13T11:35:25.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:25.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.711232+0000) 2019-10-13T11:35:25.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:25.710+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:25.711245+0000) 2019-10-13T11:35:27.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:27.273+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.274215+0000) 2019-10-13T11:35:27.275 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:27.273+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.273907+0000) 2019-10-13T11:35:27.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:27.273+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.274142+0000) 2019-10-13T11:35:27.276 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:27.274+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.274424+0000) 2019-10-13T11:35:27.276 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:27.274+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.274522+0000) 2019-10-13T11:35:27.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:27.274+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.274749+0000) 2019-10-13T11:35:27.276 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:27.274+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.275026+0000) 2019-10-13T11:35:27.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:27.276+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.277269+0000) 2019-10-13T11:35:27.283 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:27.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:28.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:28.617+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:28.617729+0000) 2019-10-13T11:35:28.931 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:35:28.931 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:28.935 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:35:28.935 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:28.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:28.950+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:28.950541+0000) 2019-10-13T11:35:31.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:31.568+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.568752+0000) 2019-10-13T11:35:32.284 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:32.283+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.283791+0000) 2019-10-13T11:35:32.286 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:32.282+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.284704+0000) 2019-10-13T11:35:32.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:32.283+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.283928+0000) 2019-10-13T11:35:32.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:32.283+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.284060+0000) 2019-10-13T11:35:32.287 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:32.283+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.284310+0000) 2019-10-13T11:35:32.287 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:32.284+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.284858+0000) 2019-10-13T11:35:32.287 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:32.284+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.284911+0000) 2019-10-13T11:35:32.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:32.285+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.287082+0000) 2019-10-13T11:35:32.292 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:32.290+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212931 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:33.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:33.055+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:33.055585+0000) 2019-10-13T11:35:33.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:33.058+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:33.059181+0000) 2019-10-13T11:35:35.555 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:35.553+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.554702+0000) 2019-10-13T11:35:35.712 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:35.711+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.711451+0000) 2019-10-13T11:35:35.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:35.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:35.711446+0000) 2019-10-13T11:35:37.293 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:37.292+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.292459+0000) 2019-10-13T11:35:37.294 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:37.291+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.293521+0000) 2019-10-13T11:35:37.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:37.292+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.293028+0000) 2019-10-13T11:35:37.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:37.292+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.293302+0000) 2019-10-13T11:35:37.295 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:37.292+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.293249+0000) 2019-10-13T11:35:37.295 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:37.293+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.293449+0000) 2019-10-13T11:35:37.295 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:37.293+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.293880+0000) 2019-10-13T11:35:37.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:37.293+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.295479+0000) 2019-10-13T11:35:37.300 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:37.298+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212965 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:38.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:38.616+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:38.617900+0000) 2019-10-13T11:35:38.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:38.949+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:38.950711+0000) 2019-10-13T11:35:41.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:41.567+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.568945+0000) 2019-10-13T11:35:42.302 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:42.300+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.301391+0000) 2019-10-13T11:35:42.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:42.300+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.301500+0000) 2019-10-13T11:35:42.303 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:42.300+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.301820+0000) 2019-10-13T11:35:42.303 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:42.300+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.302151+0000) 2019-10-13T11:35:42.304 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:42.300+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.302016+0000) 2019-10-13T11:35:42.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:42.300+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.302153+0000) 2019-10-13T11:35:42.304 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:42.301+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.302457+0000) 2019-10-13T11:35:42.305 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:42.304+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.305414+0000) 2019-10-13T11:35:42.310 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:42.308+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 212995 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:43.056 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:43.054+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:43.055721+0000) 2019-10-13T11:35:43.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:43.058+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:43.059343+0000) 2019-10-13T11:35:45.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:45.556+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.556971+0000) 2019-10-13T11:35:45.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:45.710+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.711605+0000) 2019-10-13T11:35:45.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:45.710+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:45.711604+0000) 2019-10-13T11:35:47.312 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:47.311+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.312242+0000) 2019-10-13T11:35:47.313 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:47.310+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.311647+0000) 2019-10-13T11:35:47.314 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:47.310+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.312036+0000) 2019-10-13T11:35:47.314 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:47.310+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.312113+0000) 2019-10-13T11:35:47.314 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:47.310+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.312255+0000) 2019-10-13T11:35:47.314 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:47.311+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.312446+0000) 2019-10-13T11:35:47.314 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:47.311+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.312988+0000) 2019-10-13T11:35:47.315 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:47.314+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.315381+0000) 2019-10-13T11:35:47.321 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:47.319+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213027 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:48.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:48.617+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:48.618100+0000) 2019-10-13T11:35:48.951 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:48.950+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:48.950892+0000) 2019-10-13T11:35:51.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:51.568+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.569133+0000) 2019-10-13T11:35:52.324 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:52.321+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.323088+0000) 2019-10-13T11:35:52.324 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:52.321+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.323232+0000) 2019-10-13T11:35:52.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:52.322+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.323405+0000) 2019-10-13T11:35:52.325 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:52.322+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.323613+0000) 2019-10-13T11:35:52.325 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:52.323+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.324611+0000) 2019-10-13T11:35:52.326 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:52.322+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.323829+0000) 2019-10-13T11:35:52.326 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:52.323+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.324421+0000) 2019-10-13T11:35:52.327 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:52.326+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.326957+0000) 2019-10-13T11:35:52.332 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:52.330+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213055 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:53.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:53.058+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:53.059550+0000) 2019-10-13T11:35:53.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:53.055+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:53.055885+0000) 2019-10-13T11:35:55.560 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:55.558+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.559247+0000) 2019-10-13T11:35:55.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:55.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.711759+0000) 2019-10-13T11:35:55.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:55.711+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:55.711760+0000) 2019-10-13T11:35:57.334 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:57.332+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.333605+0000) 2019-10-13T11:35:57.335 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:35:57.332+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.333932+0000) 2019-10-13T11:35:57.335 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:35:57.334+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.335063+0000) 2019-10-13T11:35:57.336 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:35:57.332+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.334171+0000) 2019-10-13T11:35:57.336 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:35:57.333+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.334532+0000) 2019-10-13T11:35:57.336 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:57.333+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.334710+0000) 2019-10-13T11:35:57.336 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:35:57.333+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.334909+0000) 2019-10-13T11:35:57.337 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:35:57.335+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.336612+0000) 2019-10-13T11:35:57.343 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:35:57.341+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213089 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:35:58.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:35:58.617+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:58.618306+0000) 2019-10-13T11:35:58.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:35:58.950+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:58.951082+0000) 2019-10-13T11:35:59.070 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:35:59.070 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:59.075 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:35:59.075 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:01.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:01.568+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.569331+0000) 2019-10-13T11:36:02.345 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:02.343+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.344605+0000) 2019-10-13T11:36:02.345 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:02.343+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.344391+0000) 2019-10-13T11:36:02.346 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:02.343+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.344629+0000) 2019-10-13T11:36:02.346 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:02.343+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.344757+0000) 2019-10-13T11:36:02.346 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:02.343+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.345055+0000) 2019-10-13T11:36:02.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:02.344+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.345402+0000) 2019-10-13T11:36:02.347 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:02.344+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.345545+0000) 2019-10-13T11:36:02.347 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:02.346+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.347469+0000) 2019-10-13T11:36:02.353 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:02.351+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213117 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:03.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:03.055+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:03.056044+0000) 2019-10-13T11:36:03.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:03.059+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:03.059722+0000) 2019-10-13T11:36:05.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:05.561+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.561638+0000) 2019-10-13T11:36:05.712 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:05.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.711922+0000) 2019-10-13T11:36:05.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:05.711+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:05.711920+0000) 2019-10-13T11:36:07.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:07.353+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.354130+0000) 2019-10-13T11:36:07.355 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:07.354+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.354558+0000) 2019-10-13T11:36:07.356 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:07.354+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.354924+0000) 2019-10-13T11:36:07.356 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:07.354+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.355014+0000) 2019-10-13T11:36:07.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:07.354+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.354977+0000) 2019-10-13T11:36:07.357 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:07.354+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.355175+0000) 2019-10-13T11:36:07.357 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:07.354+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.355373+0000) 2019-10-13T11:36:07.357 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:07.356+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.356829+0000) 2019-10-13T11:36:07.362 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:07.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:08.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:08.618+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:08.618486+0000) 2019-10-13T11:36:08.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:08.950+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:08.951236+0000) 2019-10-13T11:36:11.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:11.569+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.569557+0000) 2019-10-13T11:36:12.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:12.363+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.363953+0000) 2019-10-13T11:36:12.365 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:12.363+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.364138+0000) 2019-10-13T11:36:12.365 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:12.363+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.364270+0000) 2019-10-13T11:36:12.366 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:12.364+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.364958+0000) 2019-10-13T11:36:12.366 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:12.364+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.364776+0000) 2019-10-13T11:36:12.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:12.364+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.365191+0000) 2019-10-13T11:36:12.367 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:12.364+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.364711+0000) 2019-10-13T11:36:12.369 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:12.367+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.368722+0000) 2019-10-13T11:36:12.374 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:12.372+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213177 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:13.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:13.055+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:13.056213+0000) 2019-10-13T11:36:13.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:13.059+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:13.059886+0000) 2019-10-13T11:36:15.564 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:15.563+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.564019+0000) 2019-10-13T11:36:15.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:15.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.712121+0000) 2019-10-13T11:36:15.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:15.711+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:15.712119+0000) 2019-10-13T11:36:17.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:17.374+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.374943+0000) 2019-10-13T11:36:17.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:17.374+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.374871+0000) 2019-10-13T11:36:17.376 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:17.374+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.374753+0000) 2019-10-13T11:36:17.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:17.374+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.375017+0000) 2019-10-13T11:36:17.377 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:17.374+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.375177+0000) 2019-10-13T11:36:17.377 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:17.374+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.375339+0000) 2019-10-13T11:36:17.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:17.375+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.375609+0000) 2019-10-13T11:36:17.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:17.376+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.376816+0000) 2019-10-13T11:36:17.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:17.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:18.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:18.618+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:18.618655+0000) 2019-10-13T11:36:18.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:18.951+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:18.951413+0000) 2019-10-13T11:36:21.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:21.569+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.569732+0000) 2019-10-13T11:36:22.385 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:22.383+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.385179+0000) 2019-10-13T11:36:22.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:22.384+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.384917+0000) 2019-10-13T11:36:22.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:22.384+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.385211+0000) 2019-10-13T11:36:22.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:22.384+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.385261+0000) 2019-10-13T11:36:22.387 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:22.385+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.385473+0000) 2019-10-13T11:36:22.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:22.385+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.385772+0000) 2019-10-13T11:36:22.388 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:22.385+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.386184+0000) 2019-10-13T11:36:22.388 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:22.386+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.388130+0000) 2019-10-13T11:36:22.394 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:22.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213239 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:23.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:23.056+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:23.056376+0000) 2019-10-13T11:36:23.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:23.059+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:23.060024+0000) 2019-10-13T11:36:25.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:25.564+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.566377+0000) 2019-10-13T11:36:25.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:25.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.712323+0000) 2019-10-13T11:36:25.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:25.711+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:25.712324+0000) 2019-10-13T11:36:27.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:27.393+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.394907+0000) 2019-10-13T11:36:27.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:27.393+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.395130+0000) 2019-10-13T11:36:27.397 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:27.394+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.395895+0000) 2019-10-13T11:36:27.397 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:27.394+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.395623+0000) 2019-10-13T11:36:27.397 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:27.394+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.395701+0000) 2019-10-13T11:36:27.397 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:27.394+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.396046+0000) 2019-10-13T11:36:27.398 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:27.394+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.396193+0000) 2019-10-13T11:36:27.398 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:27.396+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.398446+0000) 2019-10-13T11:36:27.403 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:27.401+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213271 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:28.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:28.617+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.618816+0000) 2019-10-13T11:36:28.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:28.950+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.951595+0000) 2019-10-13T11:36:29.205 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:36:29.205 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:29.209 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:36:29.209 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:31.570 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:31.569+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.569901+0000) 2019-10-13T11:36:32.405 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:32.403+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.404494+0000) 2019-10-13T11:36:32.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:32.403+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.404760+0000) 2019-10-13T11:36:32.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:32.403+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.404857+0000) 2019-10-13T11:36:32.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:32.403+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.404948+0000) 2019-10-13T11:36:32.407 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:32.403+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.405603+0000) 2019-10-13T11:36:32.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:32.403+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.405264+0000) 2019-10-13T11:36:32.407 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:32.404+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.405586+0000) 2019-10-13T11:36:32.407 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:32.405+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.407622+0000) 2019-10-13T11:36:32.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:32.412+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213299 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:33.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:33.055+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:33.056524+0000) 2019-10-13T11:36:33.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:33.059+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:33.060155+0000) 2019-10-13T11:36:35.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:35.568+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.568867+0000) 2019-10-13T11:36:35.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:35.711+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.712424+0000) 2019-10-13T11:36:35.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:35.711+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:35.712416+0000) 2019-10-13T11:36:37.416 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:37.414+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.415535+0000) 2019-10-13T11:36:37.417 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:37.413+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.414769+0000) 2019-10-13T11:36:37.417 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:37.413+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.415336+0000) 2019-10-13T11:36:37.417 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:37.414+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.415459+0000) 2019-10-13T11:36:37.417 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:37.414+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.415540+0000) 2019-10-13T11:36:37.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:37.414+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.415616+0000) 2019-10-13T11:36:37.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:37.414+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.415945+0000) 2019-10-13T11:36:37.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:37.417+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.418407+0000) 2019-10-13T11:36:37.423 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:37.420+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:38.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:38.617+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.618976+0000) 2019-10-13T11:36:38.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:38.951+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.951784+0000) 2019-10-13T11:36:41.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:41.569+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.570079+0000) 2019-10-13T11:36:42.425 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:42.422+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.423580+0000) 2019-10-13T11:36:42.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:42.422+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.424113+0000) 2019-10-13T11:36:42.425 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:42.422+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.424265+0000) 2019-10-13T11:36:42.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:42.422+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.424306+0000) 2019-10-13T11:36:42.426 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:42.423+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.424656+0000) 2019-10-13T11:36:42.426 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:42.423+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.424545+0000) 2019-10-13T11:36:42.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:42.423+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.424972+0000) 2019-10-13T11:36:42.427 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:42.426+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.427420+0000) 2019-10-13T11:36:42.431 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:42.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:43.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:43.056+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:43.056657+0000) 2019-10-13T11:36:43.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:43.059+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:43.060316+0000) 2019-10-13T11:36:45.571 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:45.570+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.571256+0000) 2019-10-13T11:36:45.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:45.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.712586+0000) 2019-10-13T11:36:45.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:45.712+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:45.712586+0000) 2019-10-13T11:36:47.433 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:47.430+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.432222+0000) 2019-10-13T11:36:47.433 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:47.431+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.432393+0000) 2019-10-13T11:36:47.434 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:47.431+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.432712+0000) 2019-10-13T11:36:47.434 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:47.432+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.433055+0000) 2019-10-13T11:36:47.435 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:47.431+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.432812+0000) 2019-10-13T11:36:47.435 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:47.431+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.433079+0000) 2019-10-13T11:36:47.435 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:47.431+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.433210+0000) 2019-10-13T11:36:47.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:47.435+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.436145+0000) 2019-10-13T11:36:47.440 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:47.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:48.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:48.618+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.619159+0000) 2019-10-13T11:36:48.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:48.951+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.951897+0000) 2019-10-13T11:36:51.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:51.569+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.570268+0000) 2019-10-13T11:36:52.442 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:52.440+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.441179+0000) 2019-10-13T11:36:52.442 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:52.441+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.441489+0000) 2019-10-13T11:36:52.443 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:52.441+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.441743+0000) 2019-10-13T11:36:52.443 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:52.441+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.441743+0000) 2019-10-13T11:36:52.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:52.441+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.441930+0000) 2019-10-13T11:36:52.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:52.441+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.442089+0000) 2019-10-13T11:36:52.444 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:52.441+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.442316+0000) 2019-10-13T11:36:52.445 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:52.444+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.445080+0000) 2019-10-13T11:36:52.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:52.449+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213421 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:53.057 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:53.056+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:53.056801+0000) 2019-10-13T11:36:53.060 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:53.060+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:53.060507+0000) 2019-10-13T11:36:55.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:55.573+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.573594+0000) 2019-10-13T11:36:55.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:55.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.712776+0000) 2019-10-13T11:36:55.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:55.712+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:55.712776+0000) 2019-10-13T11:36:57.452 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:57.450+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.451260+0000) 2019-10-13T11:36:57.453 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:36:57.451+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.452200+0000) 2019-10-13T11:36:57.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:36:57.451+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.452343+0000) 2019-10-13T11:36:57.454 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:36:57.452+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.453315+0000) 2019-10-13T11:36:57.454 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:36:57.452+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.452592+0000) 2019-10-13T11:36:57.454 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:57.452+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.452679+0000) 2019-10-13T11:36:57.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:36:57.452+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.453010+0000) 2019-10-13T11:36:57.456 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:36:57.455+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.455781+0000) 2019-10-13T11:36:57.460 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:36:57.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213455 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:36:58.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:36:58.619+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.619342+0000) 2019-10-13T11:36:58.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:36:58.951+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.952067+0000) 2019-10-13T11:36:59.359 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:36:59.359 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:59.364 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:36:59.364 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:01.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:01.570+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.570449+0000) 2019-10-13T11:37:02.462 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:02.461+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.461996+0000) 2019-10-13T11:37:02.463 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:02.461+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.461788+0000) 2019-10-13T11:37:02.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:02.461+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.461898+0000) 2019-10-13T11:37:02.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:02.461+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.462201+0000) 2019-10-13T11:37:02.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:02.461+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.462283+0000) 2019-10-13T11:37:02.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:02.462+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.462485+0000) 2019-10-13T11:37:02.464 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:02.462+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.463058+0000) 2019-10-13T11:37:02.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:02.464+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.465231+0000) 2019-10-13T11:37:02.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:02.470+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:03.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:03.056+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:03.056988+0000) 2019-10-13T11:37:03.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:03.060+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:03.060664+0000) 2019-10-13T11:37:05.576 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:05.575+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.576031+0000) 2019-10-13T11:37:05.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:05.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.712951+0000) 2019-10-13T11:37:05.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:05.712+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:05.712951+0000) 2019-10-13T11:37:07.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:07.472+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.472714+0000) 2019-10-13T11:37:07.474 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:07.472+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.472920+0000) 2019-10-13T11:37:07.474 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:07.472+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.473171+0000) 2019-10-13T11:37:07.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:07.472+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.473282+0000) 2019-10-13T11:37:07.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:07.473+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.473624+0000) 2019-10-13T11:37:07.476 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:07.473+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.473933+0000) 2019-10-13T11:37:07.476 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:07.474+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.475070+0000) 2019-10-13T11:37:07.476 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:07.473+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.474072+0000) 2019-10-13T11:37:07.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:07.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:08.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:08.619+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.619527+0000) 2019-10-13T11:37:08.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:08.952+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.952225+0000) 2019-10-13T11:37:11.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:11.569+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.570629+0000) 2019-10-13T11:37:12.484 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:12.481+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.483324+0000) 2019-10-13T11:37:12.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:12.482+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.483838+0000) 2019-10-13T11:37:12.485 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:12.482+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.483892+0000) 2019-10-13T11:37:12.486 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:12.482+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.484174+0000) 2019-10-13T11:37:12.486 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:12.483+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.484414+0000) 2019-10-13T11:37:12.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:12.483+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.484826+0000) 2019-10-13T11:37:12.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:12.483+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.484796+0000) 2019-10-13T11:37:12.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:12.484+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.486324+0000) 2019-10-13T11:37:12.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:12.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213543 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:13.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:13.056+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:13.057156+0000) 2019-10-13T11:37:13.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:13.059+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:13.060817+0000) 2019-10-13T11:37:15.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:15.576+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.578362+0000) 2019-10-13T11:37:15.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:15.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.713131+0000) 2019-10-13T11:37:15.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:15.712+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:15.713131+0000) 2019-10-13T11:37:17.495 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:17.493+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.495066+0000) 2019-10-13T11:37:17.496 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:17.493+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.494789+0000) 2019-10-13T11:37:17.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:17.493+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.495097+0000) 2019-10-13T11:37:17.496 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:17.493+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.495204+0000) 2019-10-13T11:37:17.497 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:17.494+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.495674+0000) 2019-10-13T11:37:17.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:17.494+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.495888+0000) 2019-10-13T11:37:17.497 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:17.494+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.496167+0000) 2019-10-13T11:37:17.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:17.496+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.498069+0000) 2019-10-13T11:37:17.504 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:17.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:18.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:18.618+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.619719+0000) 2019-10-13T11:37:18.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:18.951+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.952393+0000) 2019-10-13T11:37:21.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:21.570+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.570831+0000) 2019-10-13T11:37:22.506 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:22.504+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.505557+0000) 2019-10-13T11:37:22.507 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:22.504+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.506443+0000) 2019-10-13T11:37:22.507 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:22.504+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.506151+0000) 2019-10-13T11:37:22.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:22.504+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.506377+0000) 2019-10-13T11:37:22.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:22.505+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.506582+0000) 2019-10-13T11:37:22.508 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:22.505+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.506624+0000) 2019-10-13T11:37:22.508 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:22.505+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.506878+0000) 2019-10-13T11:37:22.509 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:22.507+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.509322+0000) 2019-10-13T11:37:22.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:22.513+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213605 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:23.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:23.056+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:23.057339+0000) 2019-10-13T11:37:23.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:23.060+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:23.060985+0000) 2019-10-13T11:37:25.581 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:25.579+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.580607+0000) 2019-10-13T11:37:25.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:25.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.713351+0000) 2019-10-13T11:37:25.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:25.712+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:25.713351+0000) 2019-10-13T11:37:27.517 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:27.515+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.516626+0000) 2019-10-13T11:37:27.518 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:27.515+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.516976+0000) 2019-10-13T11:37:27.518 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:27.515+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.517086+0000) 2019-10-13T11:37:27.519 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:27.517+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.518107+0000) 2019-10-13T11:37:27.519 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:27.515+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.517318+0000) 2019-10-13T11:37:27.519 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:27.516+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.517885+0000) 2019-10-13T11:37:27.519 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:27.516+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.518088+0000) 2019-10-13T11:37:27.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:27.518+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.519426+0000) 2019-10-13T11:37:27.526 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:27.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213637 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:28.620 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:28.619+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.619889+0000) 2019-10-13T11:37:28.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:28.951+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.952528+0000) 2019-10-13T11:37:29.745 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:37:29.746 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:29.750 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:37:29.750 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:31.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:31.570+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.570990+0000) 2019-10-13T11:37:32.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:32.526+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.527663+0000) 2019-10-13T11:37:32.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:32.526+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.528094+0000) 2019-10-13T11:37:32.529 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:32.527+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.528227+0000) 2019-10-13T11:37:32.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:32.527+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.528437+0000) 2019-10-13T11:37:32.530 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:32.527+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.528471+0000) 2019-10-13T11:37:32.530 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:32.527+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.528825+0000) 2019-10-13T11:37:32.530 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:32.527+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.529068+0000) 2019-10-13T11:37:32.531 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:32.529+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.530718+0000) 2019-10-13T11:37:32.537 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:32.535+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213666 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:33.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:33.056+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:33.057503+0000) 2019-10-13T11:37:33.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:33.060+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:33.061122+0000) 2019-10-13T11:37:35.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:35.582+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.582999+0000) 2019-10-13T11:37:35.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:35.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.713553+0000) 2019-10-13T11:37:35.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:35.713+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:35.713553+0000) 2019-10-13T11:37:37.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:37.538+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.538450+0000) 2019-10-13T11:37:37.541 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:37.538+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.539670+0000) 2019-10-13T11:37:37.541 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:37.538+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.539137+0000) 2019-10-13T11:37:37.541 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:37.538+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.539389+0000) 2019-10-13T11:37:37.541 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:37.539+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.539574+0000) 2019-10-13T11:37:37.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:37.539+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.539696+0000) 2019-10-13T11:37:37.542 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:37.539+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.540007+0000) 2019-10-13T11:37:37.542 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:37.540+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.540914+0000) 2019-10-13T11:37:37.548 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:37.546+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:38.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:38.619+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.620036+0000) 2019-10-13T11:37:38.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:38.952+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.952673+0000) 2019-10-13T11:37:41.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:41.570+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.571160+0000) 2019-10-13T11:37:42.550 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:42.548+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.548967+0000) 2019-10-13T11:37:42.550 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:42.548+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.549341+0000) 2019-10-13T11:37:42.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:42.549+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.549443+0000) 2019-10-13T11:37:42.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:42.549+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.550288+0000) 2019-10-13T11:37:42.551 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:42.549+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.549624+0000) 2019-10-13T11:37:42.552 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:42.549+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.550214+0000) 2019-10-13T11:37:42.552 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:42.549+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.550268+0000) 2019-10-13T11:37:42.552 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:42.551+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.552570+0000) 2019-10-13T11:37:42.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:42.557+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213728 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:43.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:43.057+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:43.057687+0000) 2019-10-13T11:37:43.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:43.060+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:43.061278+0000) 2019-10-13T11:37:45.586 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:45.584+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.585603+0000) 2019-10-13T11:37:45.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:45.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.713705+0000) 2019-10-13T11:37:45.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:45.713+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:45.713705+0000) 2019-10-13T11:37:47.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:47.559+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.559505+0000) 2019-10-13T11:37:47.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:47.559+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.559851+0000) 2019-10-13T11:37:47.561 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:47.559+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.560018+0000) 2019-10-13T11:37:47.562 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:47.560+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.561382+0000) 2019-10-13T11:37:47.562 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:47.559+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.560310+0000) 2019-10-13T11:37:47.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:47.560+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.560685+0000) 2019-10-13T11:37:47.562 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:47.560+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.560405+0000) 2019-10-13T11:37:47.563 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:47.562+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.563545+0000) 2019-10-13T11:37:47.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:47.568+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213759 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:48.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:48.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.620196+0000) 2019-10-13T11:37:48.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:48.952+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.952829+0000) 2019-10-13T11:37:51.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:51.571+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.571354+0000) 2019-10-13T11:37:52.571 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:52.570+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.570832+0000) 2019-10-13T11:37:52.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:52.570+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.570787+0000) 2019-10-13T11:37:52.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:52.570+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.571338+0000) 2019-10-13T11:37:52.573 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:52.571+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.571499+0000) 2019-10-13T11:37:52.573 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:52.571+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.571675+0000) 2019-10-13T11:37:52.573 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:52.571+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.571768+0000) 2019-10-13T11:37:52.574 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:52.571+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.572432+0000) 2019-10-13T11:37:52.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:52.573+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.573949+0000) 2019-10-13T11:37:52.580 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:52.579+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213790 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:53.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:53.057+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:53.057821+0000) 2019-10-13T11:37:53.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:53.061+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:53.061437+0000) 2019-10-13T11:37:55.588 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:55.587+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.587816+0000) 2019-10-13T11:37:55.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:55.712+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.713897+0000) 2019-10-13T11:37:55.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:55.712+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:55.713897+0000) 2019-10-13T11:37:57.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:57.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.581164+0000) 2019-10-13T11:37:57.582 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:37:57.580+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.581637+0000) 2019-10-13T11:37:57.583 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:37:57.581+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.582419+0000) 2019-10-13T11:37:57.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:37:57.580+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.581791+0000) 2019-10-13T11:37:57.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:57.580+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.582218+0000) 2019-10-13T11:37:57.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:37:57.580+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.582311+0000) 2019-10-13T11:37:57.584 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:37:57.581+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.582453+0000) 2019-10-13T11:37:57.584 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:37:57.583+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.584176+0000) 2019-10-13T11:37:57.591 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:37:57.589+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213821 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:37:58.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:37:58.619+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.620373+0000) 2019-10-13T11:37:58.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:37:58.952+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.953000+0000) 2019-10-13T11:37:59.826 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:37:59.827 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:59.831 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:37:59.832 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:01.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:01.570+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.571553+0000) 2019-10-13T11:38:02.593 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:02.591+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.592829+0000) 2019-10-13T11:38:02.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:02.591+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.592639+0000) 2019-10-13T11:38:02.594 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:02.591+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.592789+0000) 2019-10-13T11:38:02.594 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:02.591+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.593021+0000) 2019-10-13T11:38:02.594 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:02.592+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.593393+0000) 2019-10-13T11:38:02.595 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:02.591+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.593334+0000) 2019-10-13T11:38:02.595 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:02.592+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.593620+0000) 2019-10-13T11:38:02.596 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:02.594+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:02.596217+0000) 2019-10-13T11:38:02.602 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:02.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213852 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:03.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:03.057+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:03.057958+0000) 2019-10-13T11:38:03.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:03.060+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:03.061625+0000) 2019-10-13T11:38:05.590 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:05.589+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.590274+0000) 2019-10-13T11:38:05.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:05.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.714092+0000) 2019-10-13T11:38:05.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:05.713+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:05.714092+0000) 2019-10-13T11:38:07.604 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:07.602+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.603864+0000) 2019-10-13T11:38:07.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:07.602+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.603390+0000) 2019-10-13T11:38:07.605 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:07.602+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.604260+0000) 2019-10-13T11:38:07.606 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:07.602+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.603790+0000) 2019-10-13T11:38:07.606 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:07.602+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.603940+0000) 2019-10-13T11:38:07.606 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:07.603+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.604472+0000) 2019-10-13T11:38:07.606 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:07.603+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.604712+0000) 2019-10-13T11:38:07.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:07.604+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.606636+0000) 2019-10-13T11:38:07.613 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:07.610+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213881 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:08.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:08.619+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.620609+0000) 2019-10-13T11:38:08.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:08.952+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.953132+0000) 2019-10-13T11:38:11.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:11.571+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.571722+0000) 2019-10-13T11:38:12.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:12.612+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.614415+0000) 2019-10-13T11:38:12.615 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:12.612+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.614069+0000) 2019-10-13T11:38:12.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:12.613+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.614682+0000) 2019-10-13T11:38:12.616 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:12.613+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.614766+0000) 2019-10-13T11:38:12.616 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:12.613+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.614915+0000) 2019-10-13T11:38:12.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:12.613+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.615275+0000) 2019-10-13T11:38:12.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:12.614+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.615521+0000) 2019-10-13T11:38:12.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:12.615+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:12.616795+0000) 2019-10-13T11:38:12.624 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:12.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213912 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:13.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:13.057+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:13.058098+0000) 2019-10-13T11:38:13.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:13.060+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:13.061790+0000) 2019-10-13T11:38:15.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:15.592+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.592828+0000) 2019-10-13T11:38:15.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:15.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.714283+0000) 2019-10-13T11:38:15.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:15.713+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:15.714283+0000) 2019-10-13T11:38:17.625 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:17.624+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.625330+0000) 2019-10-13T11:38:17.626 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:17.623+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.625018+0000) 2019-10-13T11:38:17.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:17.623+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.625141+0000) 2019-10-13T11:38:17.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:17.623+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.625279+0000) 2019-10-13T11:38:17.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:17.624+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.625620+0000) 2019-10-13T11:38:17.627 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:17.624+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.625822+0000) 2019-10-13T11:38:17.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:17.624+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.626044+0000) 2019-10-13T11:38:17.629 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:17.627+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.628668+0000) 2019-10-13T11:38:17.634 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:17.632+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213943 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:18.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:18.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.620801+0000) 2019-10-13T11:38:18.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:18.952+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.953293+0000) 2019-10-13T11:38:21.572 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:21.571+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.571915+0000) 2019-10-13T11:38:22.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:22.634+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.635718+0000) 2019-10-13T11:38:22.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:22.634+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.636108+0000) 2019-10-13T11:38:22.637 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:22.634+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.636222+0000) 2019-10-13T11:38:22.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:22.635+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.636346+0000) 2019-10-13T11:38:22.638 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:22.635+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.636485+0000) 2019-10-13T11:38:22.638 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:22.635+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.636530+0000) 2019-10-13T11:38:22.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:22.635+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.637045+0000) 2019-10-13T11:38:22.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:22.637+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:22.638717+0000) 2019-10-13T11:38:22.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:22.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 213974 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:23.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:23.057+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:23.058251+0000) 2019-10-13T11:38:23.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:23.061+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:23.061948+0000) 2019-10-13T11:38:25.595 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:25.594+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.595181+0000) 2019-10-13T11:38:25.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:25.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.714466+0000) 2019-10-13T11:38:25.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:25.714+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:25.714466+0000) 2019-10-13T11:38:27.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:27.645+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.646284+0000) 2019-10-13T11:38:27.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:27.646+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.646513+0000) 2019-10-13T11:38:27.648 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:27.646+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.646570+0000) 2019-10-13T11:38:27.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:27.646+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.646778+0000) 2019-10-13T11:38:27.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:27.646+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.647319+0000) 2019-10-13T11:38:27.649 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:27.647+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.648725+0000) 2019-10-13T11:38:27.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:27.646+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.647191+0000) 2019-10-13T11:38:27.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:27.646+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.647360+0000) 2019-10-13T11:38:27.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:27.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214003 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:28.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:28.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.620946+0000) 2019-10-13T11:38:28.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:28.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.953465+0000) 2019-10-13T11:38:29.981 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:38:29.981 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:29.985 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:38:29.986 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:31.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:31.571+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.572103+0000) 2019-10-13T11:38:32.657 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:32.656+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.657220+0000) 2019-10-13T11:38:32.658 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:32.656+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.657199+0000) 2019-10-13T11:38:32.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:32.657+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.657408+0000) 2019-10-13T11:38:32.659 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:32.657+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.657735+0000) 2019-10-13T11:38:32.659 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:32.657+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.657937+0000) 2019-10-13T11:38:32.659 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:32.657+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.657920+0000) 2019-10-13T11:38:32.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:32.657+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.658137+0000) 2019-10-13T11:38:32.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:32.658+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:32.659318+0000) 2019-10-13T11:38:32.666 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:32.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:33.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:33.058+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:33.058424+0000) 2019-10-13T11:38:33.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:33.061+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:33.062132+0000) 2019-10-13T11:38:35.598 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:35.597+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.597467+0000) 2019-10-13T11:38:35.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:35.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.714680+0000) 2019-10-13T11:38:35.716 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:35.714+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:35.714680+0000) 2019-10-13T11:38:37.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:37.667+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.667736+0000) 2019-10-13T11:38:37.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:37.667+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.667892+0000) 2019-10-13T11:38:37.669 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:37.667+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.668123+0000) 2019-10-13T11:38:37.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:37.668+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.668529+0000) 2019-10-13T11:38:37.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:37.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.668762+0000) 2019-10-13T11:38:37.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:37.668+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.668766+0000) 2019-10-13T11:38:37.671 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:37.668+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.668969+0000) 2019-10-13T11:38:37.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:37.670+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.671183+0000) 2019-10-13T11:38:37.677 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:37.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214065 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:38.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:38.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.621107+0000) 2019-10-13T11:38:38.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:38.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.953651+0000) 2019-10-13T11:38:41.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:41.570+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.572275+0000) 2019-10-13T11:38:42.679 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:42.677+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.678453+0000) 2019-10-13T11:38:42.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:42.677+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.678724+0000) 2019-10-13T11:38:42.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:42.678+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.679022+0000) 2019-10-13T11:38:42.681 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:42.677+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.679093+0000) 2019-10-13T11:38:42.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:42.677+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.679273+0000) 2019-10-13T11:38:42.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:42.677+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.679132+0000) 2019-10-13T11:38:42.681 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:42.677+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.679381+0000) 2019-10-13T11:38:42.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:42.681+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:42.682324+0000) 2019-10-13T11:38:42.687 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:42.685+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214096 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:43.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:43.057+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:43.058591+0000) 2019-10-13T11:38:43.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:43.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:43.062286+0000) 2019-10-13T11:38:45.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:45.599+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.599918+0000) 2019-10-13T11:38:45.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:45.713+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.714903+0000) 2019-10-13T11:38:45.716 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:45.713+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:45.714903+0000) 2019-10-13T11:38:47.702 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:47.687+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.688575+0000) 2019-10-13T11:38:47.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:47.687+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.688731+0000) 2019-10-13T11:38:47.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:47.687+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.688977+0000) 2019-10-13T11:38:47.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:47.688+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.689345+0000) 2019-10-13T11:38:47.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:47.687+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.689151+0000) 2019-10-13T11:38:47.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:47.688+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.689408+0000) 2019-10-13T11:38:47.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:47.688+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.689509+0000) 2019-10-13T11:38:47.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:47.691+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.691963+0000) 2019-10-13T11:38:47.715 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:47.693+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214125 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:48.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:48.619+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.621258+0000) 2019-10-13T11:38:48.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:48.952+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.953807+0000) 2019-10-13T11:38:51.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:51.571+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.572508+0000) 2019-10-13T11:38:52.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:52.695+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.696786+0000) 2019-10-13T11:38:52.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:52.695+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.697341+0000) 2019-10-13T11:38:52.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:52.695+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.697161+0000) 2019-10-13T11:38:52.699 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:52.696+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.697524+0000) 2019-10-13T11:38:52.699 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:52.696+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.697693+0000) 2019-10-13T11:38:52.699 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:52.696+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.697832+0000) 2019-10-13T11:38:52.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:52.696+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.698138+0000) 2019-10-13T11:38:52.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:52.698+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:52.700635+0000) 2019-10-13T11:38:52.706 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:52.704+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214156 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:53.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:53.057+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:53.058727+0000) 2019-10-13T11:38:53.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:53.061+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:53.062445+0000) 2019-10-13T11:38:55.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:55.601+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.602140+0000) 2019-10-13T11:38:55.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:55.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.715118+0000) 2019-10-13T11:38:55.716 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:55.714+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:55.715118+0000) 2019-10-13T11:38:57.708 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:57.705+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.707335+0000) 2019-10-13T11:38:57.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:38:57.706+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.707777+0000) 2019-10-13T11:38:57.709 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:38:57.706+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.707991+0000) 2019-10-13T11:38:57.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:57.706+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.707909+0000) 2019-10-13T11:38:57.710 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:38:57.706+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.708084+0000) 2019-10-13T11:38:57.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:38:57.707+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.708609+0000) 2019-10-13T11:38:57.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:38:57.707+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.708510+0000) 2019-10-13T11:38:57.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:38:57.708+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.710725+0000) 2019-10-13T11:38:57.716 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:38:57.713+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:38:58.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:38:58.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.621431+0000) 2019-10-13T11:38:58.954 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:38:58.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.953967+0000) 2019-10-13T11:39:00.136 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:39:00.136 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:00.140 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:39:00.140 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:01.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:01.571+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.572614+0000) 2019-10-13T11:39:02.717 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:02.715+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.717089+0000) 2019-10-13T11:39:02.718 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:02.715+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.716963+0000) 2019-10-13T11:39:02.718 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:02.715+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.717156+0000) 2019-10-13T11:39:02.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:02.715+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.717287+0000) 2019-10-13T11:39:02.719 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:02.716+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.717564+0000) 2019-10-13T11:39:02.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:02.716+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.717800+0000) 2019-10-13T11:39:02.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:02.716+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.717911+0000) 2019-10-13T11:39:02.720 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:02.717+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:02.719649+0000) 2019-10-13T11:39:02.725 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:02.722+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214218 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:03.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:03.058+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:03.058859+0000) 2019-10-13T11:39:03.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:03.061+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:03.062574+0000) 2019-10-13T11:39:05.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:05.603+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.604500+0000) 2019-10-13T11:39:05.716 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:05.714+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.715356+0000) 2019-10-13T11:39:05.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:05.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:05.715356+0000) 2019-10-13T11:39:07.726 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:07.724+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.725706+0000) 2019-10-13T11:39:07.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:07.724+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.725882+0000) 2019-10-13T11:39:07.727 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:07.725+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.726516+0000) 2019-10-13T11:39:07.728 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:07.724+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.726254+0000) 2019-10-13T11:39:07.728 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:07.725+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.726562+0000) 2019-10-13T11:39:07.728 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:07.725+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.726677+0000) 2019-10-13T11:39:07.728 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:07.725+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.726849+0000) 2019-10-13T11:39:07.729 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:07.727+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.728468+0000) 2019-10-13T11:39:07.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:07.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:08.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:08.621+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.621604+0000) 2019-10-13T11:39:08.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:08.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.954168+0000) 2019-10-13T11:39:11.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:11.572+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.572773+0000) 2019-10-13T11:39:12.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:12.734+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.735511+0000) 2019-10-13T11:39:12.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:12.734+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.735276+0000) 2019-10-13T11:39:12.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:12.735+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.735500+0000) 2019-10-13T11:39:12.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:12.735+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.735631+0000) 2019-10-13T11:39:12.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:12.735+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.735781+0000) 2019-10-13T11:39:12.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:12.735+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.736115+0000) 2019-10-13T11:39:12.738 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:12.735+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.736151+0000) 2019-10-13T11:39:12.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:12.737+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:12.738319+0000) 2019-10-13T11:39:12.744 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:12.743+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:13.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:13.058+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:13.059004+0000) 2019-10-13T11:39:13.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:13.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:13.062710+0000) 2019-10-13T11:39:15.607 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:15.606+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.606728+0000) 2019-10-13T11:39:15.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:15.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.715626+0000) 2019-10-13T11:39:15.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:15.715+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:15.715626+0000) 2019-10-13T11:39:17.753 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:17.745+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.746126+0000) 2019-10-13T11:39:17.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:17.745+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.746315+0000) 2019-10-13T11:39:17.754 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:17.745+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.746220+0000) 2019-10-13T11:39:17.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:17.746+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.746470+0000) 2019-10-13T11:39:17.755 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:17.746+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.746740+0000) 2019-10-13T11:39:17.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:17.746+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.746948+0000) 2019-10-13T11:39:17.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:17.745+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.746302+0000) 2019-10-13T11:39:17.755 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:17.747+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.748219+0000) 2019-10-13T11:39:17.756 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:17.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214309 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:18.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:18.621+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.621774+0000) 2019-10-13T11:39:18.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:18.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.954343+0000) 2019-10-13T11:39:21.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:21.572+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.572977+0000) 2019-10-13T11:39:22.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:22.756+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.757059+0000) 2019-10-13T11:39:22.758 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:22.757+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.757413+0000) 2019-10-13T11:39:22.760 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:22.756+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.757645+0000) 2019-10-13T11:39:22.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:22.757+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.757793+0000) 2019-10-13T11:39:22.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:22.757+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.758092+0000) 2019-10-13T11:39:22.761 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:22.757+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.758194+0000) 2019-10-13T11:39:22.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:22.758+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.758625+0000) 2019-10-13T11:39:22.761 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:22.760+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:22.760969+0000) 2019-10-13T11:39:22.766 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:22.765+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214340 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:23.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:23.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:23.059197+0000) 2019-10-13T11:39:23.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:23.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:23.062879+0000) 2019-10-13T11:39:25.609 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:25.608+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.608774+0000) 2019-10-13T11:39:25.716 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:25.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.715841+0000) 2019-10-13T11:39:25.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:25.715+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:25.715841+0000) 2019-10-13T11:39:27.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:27.766+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.767479+0000) 2019-10-13T11:39:27.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:27.765+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.767251+0000) 2019-10-13T11:39:27.768 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:27.766+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.767554+0000) 2019-10-13T11:39:27.769 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:27.766+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.768146+0000) 2019-10-13T11:39:27.769 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:27.767+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.768408+0000) 2019-10-13T11:39:27.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:27.767+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.768573+0000) 2019-10-13T11:39:27.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:27.767+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.769190+0000) 2019-10-13T11:39:27.771 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:27.770+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.771470+0000) 2019-10-13T11:39:27.777 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:27.775+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:28.622 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:28.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.621969+0000) 2019-10-13T11:39:28.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:28.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.954486+0000) 2019-10-13T11:39:30.256 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:39:30.256 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:30.264 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:39:30.264 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:31.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:31.571+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.573133+0000) 2019-10-13T11:39:32.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:32.778+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.778835+0000) 2019-10-13T11:39:32.780 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:32.777+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.778632+0000) 2019-10-13T11:39:32.780 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:32.777+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.778952+0000) 2019-10-13T11:39:32.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:32.777+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.778972+0000) 2019-10-13T11:39:32.780 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:32.777+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.779355+0000) 2019-10-13T11:39:32.781 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:32.778+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.779631+0000) 2019-10-13T11:39:32.781 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:32.778+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.779770+0000) 2019-10-13T11:39:32.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:32.781+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:32.781807+0000) 2019-10-13T11:39:32.787 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:32.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214400 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:33.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:33.058+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:33.059345+0000) 2019-10-13T11:39:33.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:33.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:33.063040+0000) 2019-10-13T11:39:35.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:35.610+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.611126+0000) 2019-10-13T11:39:35.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:35.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.716028+0000) 2019-10-13T11:39:35.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:35.715+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:35.716028+0000) 2019-10-13T11:39:37.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:37.788+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.788996+0000) 2019-10-13T11:39:37.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:37.787+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.788896+0000) 2019-10-13T11:39:37.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:37.787+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.789024+0000) 2019-10-13T11:39:37.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:37.787+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.789133+0000) 2019-10-13T11:39:37.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:37.788+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.789413+0000) 2019-10-13T11:39:37.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:37.788+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.789483+0000) 2019-10-13T11:39:37.791 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:37.788+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.789739+0000) 2019-10-13T11:39:37.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:37.790+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.791329+0000) 2019-10-13T11:39:37.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:37.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:38.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:38.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.622133+0000) 2019-10-13T11:39:38.955 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:38.953+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.954663+0000) 2019-10-13T11:39:41.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:41.572+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.573293+0000) 2019-10-13T11:39:42.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:42.796+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.798715+0000) 2019-10-13T11:39:42.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:42.797+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.798448+0000) 2019-10-13T11:39:42.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:42.797+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.798655+0000) 2019-10-13T11:39:42.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:42.797+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.799046+0000) 2019-10-13T11:39:42.803 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:42.797+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.799011+0000) 2019-10-13T11:39:42.803 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:42.797+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.799290+0000) 2019-10-13T11:39:42.803 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:42.798+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.799509+0000) 2019-10-13T11:39:42.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:42.798+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:42.800681+0000) 2019-10-13T11:39:42.808 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:42.805+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214461 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:43.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:43.058+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:43.059492+0000) 2019-10-13T11:39:43.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:43.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:43.063195+0000) 2019-10-13T11:39:45.614 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:45.612+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.613578+0000) 2019-10-13T11:39:45.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:45.714+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.716277+0000) 2019-10-13T11:39:45.717 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:45.714+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:45.716277+0000) 2019-10-13T11:39:47.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:47.807+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.808953+0000) 2019-10-13T11:39:47.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:47.807+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.809117+0000) 2019-10-13T11:39:47.810 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:47.808+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.809618+0000) 2019-10-13T11:39:47.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:47.808+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.810430+0000) 2019-10-13T11:39:47.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:47.808+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.809915+0000) 2019-10-13T11:39:47.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:47.808+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.810347+0000) 2019-10-13T11:39:47.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:47.810+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.811427+0000) 2019-10-13T11:39:47.813 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:47.811+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.813444+0000) 2019-10-13T11:39:47.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:47.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214490 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:48.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:48.620+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.622273+0000) 2019-10-13T11:39:48.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:48.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.954890+0000) 2019-10-13T11:39:51.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:51.572+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.573493+0000) 2019-10-13T11:39:52.821 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:52.819+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.820857+0000) 2019-10-13T11:39:52.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:52.819+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.821233+0000) 2019-10-13T11:39:52.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:52.820+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.822010+0000) 2019-10-13T11:39:52.823 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:52.820+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.821404+0000) 2019-10-13T11:39:52.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:52.820+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.821878+0000) 2019-10-13T11:39:52.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:52.820+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.822068+0000) 2019-10-13T11:39:52.824 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:52.820+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.822252+0000) 2019-10-13T11:39:52.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:52.822+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:52.824261+0000) 2019-10-13T11:39:52.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:52.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214521 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:53.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:53.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:53.059641+0000) 2019-10-13T11:39:53.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:53.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:53.063363+0000) 2019-10-13T11:39:55.616 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:55.615+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.615942+0000) 2019-10-13T11:39:55.717 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:55.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.716508+0000) 2019-10-13T11:39:55.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:55.716+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:55.716507+0000) 2019-10-13T11:39:57.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:39:57.831+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.831937+0000) 2019-10-13T11:39:57.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:39:57.831+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.832078+0000) 2019-10-13T11:39:57.833 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:39:57.831+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.832242+0000) 2019-10-13T11:39:57.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:57.832+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.832415+0000) 2019-10-13T11:39:57.833 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:57.832+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.832534+0000) 2019-10-13T11:39:57.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:39:57.832+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.832779+0000) 2019-10-13T11:39:57.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:39:57.832+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.833020+0000) 2019-10-13T11:39:57.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:39:57.834+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.835457+0000) 2019-10-13T11:39:57.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:39:57.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214552 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:39:58.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:39:58.622+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.622427+0000) 2019-10-13T11:39:58.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:39:58.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.955069+0000) 2019-10-13T11:40:00.403 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:40:00.403 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:00.407 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:40:00.407 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:01.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:01.573+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.573687+0000) 2019-10-13T11:40:02.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:02.841+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.842309+0000) 2019-10-13T11:40:02.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:02.842+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.842531+0000) 2019-10-13T11:40:02.844 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:02.842+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.843498+0000) 2019-10-13T11:40:02.844 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:02.842+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.842625+0000) 2019-10-13T11:40:02.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:02.842+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.842917+0000) 2019-10-13T11:40:02.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:02.843+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.843489+0000) 2019-10-13T11:40:02.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:02.843+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.843705+0000) 2019-10-13T11:40:02.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:02.843+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:02.844669+0000) 2019-10-13T11:40:02.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:02.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214583 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:03.060 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:03.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:03.059790+0000) 2019-10-13T11:40:03.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:03.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:03.063543+0000) 2019-10-13T11:40:05.618 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:05.617+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.618228+0000) 2019-10-13T11:40:05.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:05.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.716690+0000) 2019-10-13T11:40:05.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:05.716+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:05.716690+0000) 2019-10-13T11:40:07.854 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:07.853+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.853420+0000) 2019-10-13T11:40:07.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:07.853+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.854000+0000) 2019-10-13T11:40:07.855 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:07.853+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.854055+0000) 2019-10-13T11:40:07.856 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:07.853+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.854242+0000) 2019-10-13T11:40:07.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:07.854+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.854439+0000) 2019-10-13T11:40:07.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:07.854+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.854688+0000) 2019-10-13T11:40:07.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:07.854+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.854731+0000) 2019-10-13T11:40:07.857 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:07.855+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.856311+0000) 2019-10-13T11:40:07.863 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:07.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214612 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:08.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:08.622+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.622573+0000) 2019-10-13T11:40:08.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:08.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.955185+0000) 2019-10-13T11:40:11.574 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:11.573+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.573855+0000) 2019-10-13T11:40:12.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:12.863+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.864140+0000) 2019-10-13T11:40:12.865 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:12.862+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.863932+0000) 2019-10-13T11:40:12.866 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:12.862+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.864028+0000) 2019-10-13T11:40:12.866 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:12.862+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.864230+0000) 2019-10-13T11:40:12.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:12.863+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.864458+0000) 2019-10-13T11:40:12.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:12.863+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.864719+0000) 2019-10-13T11:40:12.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:12.863+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.864766+0000) 2019-10-13T11:40:12.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:12.866+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:12.867553+0000) 2019-10-13T11:40:12.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:12.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214643 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:13.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:13.058+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:13.059924+0000) 2019-10-13T11:40:13.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:13.063+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:13.063701+0000) 2019-10-13T11:40:15.621 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:15.620+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.620544+0000) 2019-10-13T11:40:15.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:15.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.716906+0000) 2019-10-13T11:40:15.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:15.715+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:15.716906+0000) 2019-10-13T11:40:17.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:17.874+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.875457+0000) 2019-10-13T11:40:17.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:17.873+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.874907+0000) 2019-10-13T11:40:17.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:17.873+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.875330+0000) 2019-10-13T11:40:17.877 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:17.874+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.875625+0000) 2019-10-13T11:40:17.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:17.874+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.875767+0000) 2019-10-13T11:40:17.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:17.874+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.876045+0000) 2019-10-13T11:40:17.878 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:17.874+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.876366+0000) 2019-10-13T11:40:17.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:17.877+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.878021+0000) 2019-10-13T11:40:17.884 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:17.882+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214674 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:18.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:18.621+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.622724+0000) 2019-10-13T11:40:18.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:18.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.955350+0000) 2019-10-13T11:40:21.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:21.573+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.574052+0000) 2019-10-13T11:40:22.886 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:22.883+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.885152+0000) 2019-10-13T11:40:22.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:22.883+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.885390+0000) 2019-10-13T11:40:22.887 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:22.884+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.885481+0000) 2019-10-13T11:40:22.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:22.884+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.885596+0000) 2019-10-13T11:40:22.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:22.885+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.886122+0000) 2019-10-13T11:40:22.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:22.884+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.886232+0000) 2019-10-13T11:40:22.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:22.885+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.886522+0000) 2019-10-13T11:40:22.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:22.887+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:22.888498+0000) 2019-10-13T11:40:22.895 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:22.892+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:23.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:23.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:23.060060+0000) 2019-10-13T11:40:23.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:23.063+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:23.063816+0000) 2019-10-13T11:40:25.623 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:25.622+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.622867+0000) 2019-10-13T11:40:25.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:25.716+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.717072+0000) 2019-10-13T11:40:25.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:25.716+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:25.717072+0000) 2019-10-13T11:40:27.897 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:27.894+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.896236+0000) 2019-10-13T11:40:27.897 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:27.895+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.896447+0000) 2019-10-13T11:40:27.898 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:27.895+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.896729+0000) 2019-10-13T11:40:27.898 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:27.896+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.897006+0000) 2019-10-13T11:40:27.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:27.895+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.897017+0000) 2019-10-13T11:40:27.899 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:27.895+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.896972+0000) 2019-10-13T11:40:27.899 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:27.895+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.897297+0000) 2019-10-13T11:40:27.900 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:27.899+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.900221+0000) 2019-10-13T11:40:27.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:27.903+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:28.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:28.622+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.622865+0000) 2019-10-13T11:40:28.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:28.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.955526+0000) 2019-10-13T11:40:30.540 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:40:30.541 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:30.546 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:40:30.546 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:31.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:31.573+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.574250+0000) 2019-10-13T11:40:32.907 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:32.905+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.906772+0000) 2019-10-13T11:40:32.908 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:32.905+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.907276+0000) 2019-10-13T11:40:32.909 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:32.906+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.907447+0000) 2019-10-13T11:40:32.909 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:32.906+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.907745+0000) 2019-10-13T11:40:32.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:32.906+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.907783+0000) 2019-10-13T11:40:32.909 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:32.906+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.908061+0000) 2019-10-13T11:40:32.910 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:32.908+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.908766+0000) 2019-10-13T11:40:32.910 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:32.908+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:32.909483+0000) 2019-10-13T11:40:32.916 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:32.914+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:33.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:33.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:33.060213+0000) 2019-10-13T11:40:33.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:33.062+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:33.063951+0000) 2019-10-13T11:40:35.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:35.624+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.625262+0000) 2019-10-13T11:40:35.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:35.715+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.717283+0000) 2019-10-13T11:40:35.718 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:35.715+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:35.717283+0000) 2019-10-13T11:40:37.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:37.916+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.917922+0000) 2019-10-13T11:40:37.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:37.916+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.918125+0000) 2019-10-13T11:40:37.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:37.917+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.918473+0000) 2019-10-13T11:40:37.920 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:37.917+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.918489+0000) 2019-10-13T11:40:37.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:37.916+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.918382+0000) 2019-10-13T11:40:37.920 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:37.917+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.918810+0000) 2019-10-13T11:40:37.921 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:37.917+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.919437+0000) 2019-10-13T11:40:37.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:37.919+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.921434+0000) 2019-10-13T11:40:37.926 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:37.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214794 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:38.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:38.622+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.623010+0000) 2019-10-13T11:40:38.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:38.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.955716+0000) 2019-10-13T11:40:41.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:41.574+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.574419+0000) 2019-10-13T11:40:42.928 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:42.926+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.927197+0000) 2019-10-13T11:40:42.928 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:42.927+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.927554+0000) 2019-10-13T11:40:42.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:42.926+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.927754+0000) 2019-10-13T11:40:42.929 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:42.927+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.927757+0000) 2019-10-13T11:40:42.929 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:42.927+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.927966+0000) 2019-10-13T11:40:42.930 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:42.927+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.928289+0000) 2019-10-13T11:40:42.930 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:42.927+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.928095+0000) 2019-10-13T11:40:42.931 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:42.929+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:42.930813+0000) 2019-10-13T11:40:42.937 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:42.935+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214824 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:43.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:43.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:43.060361+0000) 2019-10-13T11:40:43.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:43.063+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:43.064113+0000) 2019-10-13T11:40:45.628 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:45.626+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.627728+0000) 2019-10-13T11:40:45.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:45.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.717508+0000) 2019-10-13T11:40:45.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:45.717+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:45.717508+0000) 2019-10-13T11:40:47.939 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:47.937+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.938119+0000) 2019-10-13T11:40:47.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:47.938+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.938427+0000) 2019-10-13T11:40:47.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:47.938+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.938501+0000) 2019-10-13T11:40:47.940 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:47.938+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.938927+0000) 2019-10-13T11:40:47.941 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:47.938+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.938897+0000) 2019-10-13T11:40:47.941 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:47.938+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.939113+0000) 2019-10-13T11:40:47.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:47.938+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.939387+0000) 2019-10-13T11:40:47.943 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:47.942+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.942807+0000) 2019-10-13T11:40:47.948 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:47.946+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:48.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:48.622+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.623200+0000) 2019-10-13T11:40:48.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:48.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.955844+0000) 2019-10-13T11:40:51.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:51.574+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.574590+0000) 2019-10-13T11:40:52.949 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:52.948+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.949324+0000) 2019-10-13T11:40:52.952 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:52.948+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.949294+0000) 2019-10-13T11:40:52.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:52.949+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.949587+0000) 2019-10-13T11:40:52.952 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:52.949+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.949823+0000) 2019-10-13T11:40:52.953 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:52.949+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.949975+0000) 2019-10-13T11:40:52.953 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:52.949+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.950091+0000) 2019-10-13T11:40:52.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:52.950+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.950452+0000) 2019-10-13T11:40:52.953 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:52.951+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:52.951928+0000) 2019-10-13T11:40:52.959 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:52.957+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214883 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:53.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:53.060+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:53.060479+0000) 2019-10-13T11:40:53.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:53.063+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:53.064267+0000) 2019-10-13T11:40:55.630 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:55.629+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.630171+0000) 2019-10-13T11:40:55.718 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:55.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.717718+0000) 2019-10-13T11:40:55.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:55.717+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:55.717718+0000) 2019-10-13T11:40:57.961 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:40:57.959+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.960029+0000) 2019-10-13T11:40:57.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:40:57.959+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.960585+0000) 2019-10-13T11:40:57.962 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:40:57.960+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.960519+0000) 2019-10-13T11:40:57.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:40:57.960+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.960714+0000) 2019-10-13T11:40:57.962 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:40:57.960+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.960983+0000) 2019-10-13T11:40:57.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:57.960+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.961166+0000) 2019-10-13T11:40:57.963 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:57.961+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.961392+0000) 2019-10-13T11:40:57.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:40:57.963+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.964171+0000) 2019-10-13T11:40:57.970 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:40:57.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214914 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:40:58.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:40:58.622+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.623344+0000) 2019-10-13T11:40:58.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:40:58.954+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.956018+0000) 2019-10-13T11:41:00.703 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:41:00.703 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:00.707 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:41:00.708 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:01.575 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:01.573+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.574787+0000) 2019-10-13T11:41:02.972 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:02.969+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.970884+0000) 2019-10-13T11:41:02.972 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:02.969+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.971288+0000) 2019-10-13T11:41:02.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:02.969+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.971346+0000) 2019-10-13T11:41:02.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:02.971+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.972318+0000) 2019-10-13T11:41:02.973 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:02.970+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.971687+0000) 2019-10-13T11:41:02.973 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:02.970+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.971853+0000) 2019-10-13T11:41:02.974 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:02.970+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.972018+0000) 2019-10-13T11:41:02.974 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:02.972+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:02.973250+0000) 2019-10-13T11:41:02.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:02.978+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214944 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:03.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:03.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:03.060672+0000) 2019-10-13T11:41:03.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:03.063+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:03.064390+0000) 2019-10-13T11:41:05.633 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:05.632+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.632491+0000) 2019-10-13T11:41:05.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:05.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.717907+0000) 2019-10-13T11:41:05.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:05.717+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:05.717907+0000) 2019-10-13T11:41:07.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:07.979+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.981287+0000) 2019-10-13T11:41:07.982 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:07.981+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.981888+0000) 2019-10-13T11:41:07.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:07.980+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.982172+0000) 2019-10-13T11:41:07.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:07.980+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.982298+0000) 2019-10-13T11:41:07.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:07.981+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.982439+0000) 2019-10-13T11:41:07.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:07.981+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.982655+0000) 2019-10-13T11:41:07.984 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:07.981+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.982965+0000) 2019-10-13T11:41:07.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:07.983+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.984114+0000) 2019-10-13T11:41:07.990 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:07.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 214973 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:08.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:08.622+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.623510+0000) 2019-10-13T11:41:08.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:08.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.956179+0000) 2019-10-13T11:41:11.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:11.574+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.574991+0000) 2019-10-13T11:41:12.993 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:12.990+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.991696+0000) 2019-10-13T11:41:12.993 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:12.990+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.991822+0000) 2019-10-13T11:41:12.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:12.990+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.991914+0000) 2019-10-13T11:41:12.993 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:12.990+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.992268+0000) 2019-10-13T11:41:12.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:12.990+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.992385+0000) 2019-10-13T11:41:12.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:12.992+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.993364+0000) 2019-10-13T11:41:12.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:12.991+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.992630+0000) 2019-10-13T11:41:12.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:12.994+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:12.995338+0000) 2019-10-13T11:41:13.000 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:12.998+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215003 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:13.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:13.059+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:13.060831+0000) 2019-10-13T11:41:13.064 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:13.064+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:13.064524+0000) 2019-10-13T11:41:15.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:15.634+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.634516+0000) 2019-10-13T11:41:15.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:15.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.718098+0000) 2019-10-13T11:41:15.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:15.717+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:15.718098+0000) 2019-10-13T11:41:18.002 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:18.001+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.002036+0000) 2019-10-13T11:41:18.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:18.000+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.001823+0000) 2019-10-13T11:41:18.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:18.000+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.002234+0000) 2019-10-13T11:41:18.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:18.001+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.002473+0000) 2019-10-13T11:41:18.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:18.001+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.002698+0000) 2019-10-13T11:41:18.004 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:18.001+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.003060+0000) 2019-10-13T11:41:18.004 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:18.001+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.003340+0000) 2019-10-13T11:41:18.006 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:18.005+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.006243+0000) 2019-10-13T11:41:18.011 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:18.009+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215034 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:18.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:18.623+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.623705+0000) 2019-10-13T11:41:18.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:18.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.956343+0000) 2019-10-13T11:41:21.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:21.574+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.575180+0000) 2019-10-13T11:41:23.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:23.011+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.012442+0000) 2019-10-13T11:41:23.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:23.011+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.012875+0000) 2019-10-13T11:41:23.014 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:23.011+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.013043+0000) 2019-10-13T11:41:23.014 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:23.011+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.013358+0000) 2019-10-13T11:41:23.015 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:23.012+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.013635+0000) 2019-10-13T11:41:23.015 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:23.012+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.013658+0000) 2019-10-13T11:41:23.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:23.012+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.014033+0000) 2019-10-13T11:41:23.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:23.014+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.015143+0000) 2019-10-13T11:41:23.022 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:23.020+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215064 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:23.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:23.060+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.061027+0000) 2019-10-13T11:41:23.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:23.063+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.064654+0000) 2019-10-13T11:41:25.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:25.635+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.636828+0000) 2019-10-13T11:41:25.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:25.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.718320+0000) 2019-10-13T11:41:25.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:25.717+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:25.718320+0000) 2019-10-13T11:41:28.024 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:28.022+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.023224+0000) 2019-10-13T11:41:28.024 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:28.022+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.023364+0000) 2019-10-13T11:41:28.024 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:28.023+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.023507+0000) 2019-10-13T11:41:28.025 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:28.022+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.024033+0000) 2019-10-13T11:41:28.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:28.023+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.024220+0000) 2019-10-13T11:41:28.026 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:28.024+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.024468+0000) 2019-10-13T11:41:28.026 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:28.024+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.024815+0000) 2019-10-13T11:41:28.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:28.025+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.026878+0000) 2019-10-13T11:41:28.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:28.032+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215093 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:28.624 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:28.623+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.623870+0000) 2019-10-13T11:41:28.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:28.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.956541+0000) 2019-10-13T11:41:30.865 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:41:30.865 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:30.870 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:41:30.870 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:31.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:31.575+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.575377+0000) 2019-10-13T11:41:33.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:33.033+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.034211+0000) 2019-10-13T11:41:33.035 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:33.034+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.034617+0000) 2019-10-13T11:41:33.036 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:33.033+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.034791+0000) 2019-10-13T11:41:33.036 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:33.034+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.034977+0000) 2019-10-13T11:41:33.037 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:33.034+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.035145+0000) 2019-10-13T11:41:33.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:33.034+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.035339+0000) 2019-10-13T11:41:33.037 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:33.035+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.035516+0000) 2019-10-13T11:41:33.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:33.035+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.037337+0000) 2019-10-13T11:41:33.044 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:33.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:33.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:33.060+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.061181+0000) 2019-10-13T11:41:33.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:33.063+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.064764+0000) 2019-10-13T11:41:35.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:35.638+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.639280+0000) 2019-10-13T11:41:35.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:35.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.718547+0000) 2019-10-13T11:41:35.719 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:35.718+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:35.718547+0000) 2019-10-13T11:41:38.045 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:38.044+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.045349+0000) 2019-10-13T11:41:38.047 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:38.045+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.045436+0000) 2019-10-13T11:41:38.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:38.045+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.045515+0000) 2019-10-13T11:41:38.047 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:38.045+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.045707+0000) 2019-10-13T11:41:38.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:38.045+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.045695+0000) 2019-10-13T11:41:38.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:38.045+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.045889+0000) 2019-10-13T11:41:38.048 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:38.045+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.046110+0000) 2019-10-13T11:41:38.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:38.048+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.048836+0000) 2019-10-13T11:41:38.055 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:38.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215154 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:38.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:38.623+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.624050+0000) 2019-10-13T11:41:38.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:38.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.956686+0000) 2019-10-13T11:41:41.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:41.575+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.575631+0000) 2019-10-13T11:41:43.057 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:43.055+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.056488+0000) 2019-10-13T11:41:43.058 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:43.056+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.056392+0000) 2019-10-13T11:41:43.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:43.056+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.056594+0000) 2019-10-13T11:41:43.059 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:43.056+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.056798+0000) 2019-10-13T11:41:43.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:43.056+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.056927+0000) 2019-10-13T11:41:43.060 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:43.057+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.057434+0000) 2019-10-13T11:41:43.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:43.057+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.057622+0000) 2019-10-13T11:41:43.060 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:43.058+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.059537+0000) 2019-10-13T11:41:43.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:43.060+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.061319+0000) 2019-10-13T11:41:43.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:43.064+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.064867+0000) 2019-10-13T11:41:43.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:43.065+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215184 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:45.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:45.641+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:45.641763+0000) 2019-10-13T11:41:45.719 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:45.717+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:45.718733+0000) 2019-10-13T11:41:45.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:45.717+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:45.718733+0000) 2019-10-13T11:41:48.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:48.066+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.067607+0000) 2019-10-13T11:41:48.069 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:48.066+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.067775+0000) 2019-10-13T11:41:48.069 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:48.066+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.068001+0000) 2019-10-13T11:41:48.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:48.066+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.068057+0000) 2019-10-13T11:41:48.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:48.068+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.069204+0000) 2019-10-13T11:41:48.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:48.067+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.068625+0000) 2019-10-13T11:41:48.070 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:48.067+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.068982+0000) 2019-10-13T11:41:48.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:48.069+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.070103+0000) 2019-10-13T11:41:48.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:48.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215213 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:48.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:48.623+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.624242+0000) 2019-10-13T11:41:48.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:48.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.956861+0000) 2019-10-13T11:41:51.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:51.574+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.575785+0000) 2019-10-13T11:41:53.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:53.060+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.061516+0000) 2019-10-13T11:41:53.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:53.064+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.065008+0000) 2019-10-13T11:41:53.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:53.077+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.078313+0000) 2019-10-13T11:41:53.079 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:53.076+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.078132+0000) 2019-10-13T11:41:53.079 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:53.076+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.078312+0000) 2019-10-13T11:41:53.079 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:53.076+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.078334+0000) 2019-10-13T11:41:53.080 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:53.077+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.078746+0000) 2019-10-13T11:41:53.080 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:53.077+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.078568+0000) 2019-10-13T11:41:53.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:53.077+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.079175+0000) 2019-10-13T11:41:53.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:53.080+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.081384+0000) 2019-10-13T11:41:53.086 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:53.084+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215243 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:55.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:55.643+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:55.644247+0000) 2019-10-13T11:41:55.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:55.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:55.718952+0000) 2019-10-13T11:41:55.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:55.718+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:55.718952+0000) 2019-10-13T11:41:58.088 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:41:58.085+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.087314+0000) 2019-10-13T11:41:58.089 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:58.086+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.087461+0000) 2019-10-13T11:41:58.089 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:58.086+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.087794+0000) 2019-10-13T11:41:58.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:41:58.087+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.087978+0000) 2019-10-13T11:41:58.090 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:41:58.086+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.088192+0000) 2019-10-13T11:41:58.090 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:41:58.087+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.088428+0000) 2019-10-13T11:41:58.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:41:58.087+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.088700+0000) 2019-10-13T11:41:58.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:41:58.089+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.090500+0000) 2019-10-13T11:41:58.096 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:41:58.094+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215274 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:41:58.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:41:58.623+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.624436+0000) 2019-10-13T11:41:58.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:41:58.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.956999+0000) 2019-10-13T11:42:01.036 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:42:01.036 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:01.040 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:42:01.040 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:01.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:01.575+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.575959+0000) 2019-10-13T11:42:03.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:03.061+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.061664+0000) 2019-10-13T11:42:03.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:03.064+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.065191+0000) 2019-10-13T11:42:03.098 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:03.096+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.097815+0000) 2019-10-13T11:42:03.099 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:03.096+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.098019+0000) 2019-10-13T11:42:03.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:03.096+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.098154+0000) 2019-10-13T11:42:03.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:03.097+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.098667+0000) 2019-10-13T11:42:03.100 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:03.097+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.098779+0000) 2019-10-13T11:42:03.100 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:03.097+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.098809+0000) 2019-10-13T11:42:03.100 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:03.097+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.098945+0000) 2019-10-13T11:42:03.101 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:03.100+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.100921+0000) 2019-10-13T11:42:03.107 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:03.105+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:05.647 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:05.646+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:05.646488+0000) 2019-10-13T11:42:05.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:05.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:05.719142+0000) 2019-10-13T11:42:05.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:05.718+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:05.719142+0000) 2019-10-13T11:42:08.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:08.108+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.109167+0000) 2019-10-13T11:42:08.110 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:08.106+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.108350+0000) 2019-10-13T11:42:08.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:08.107+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.108477+0000) 2019-10-13T11:42:08.111 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:08.107+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.108691+0000) 2019-10-13T11:42:08.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:08.107+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.109079+0000) 2019-10-13T11:42:08.111 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:08.107+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.109368+0000) 2019-10-13T11:42:08.112 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:08.108+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.109462+0000) 2019-10-13T11:42:08.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:08.110+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.111387+0000) 2019-10-13T11:42:08.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:08.115+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215333 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:08.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:08.623+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.624595+0000) 2019-10-13T11:42:08.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:08.955+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.957131+0000) 2019-10-13T11:42:11.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:11.575+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.576136+0000) 2019-10-13T11:42:13.062 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:13.061+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.061854+0000) 2019-10-13T11:42:13.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:13.064+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.065397+0000) 2019-10-13T11:42:13.119 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:13.118+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.118864+0000) 2019-10-13T11:42:13.120 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:13.118+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.119342+0000) 2019-10-13T11:42:13.121 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:13.119+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.119586+0000) 2019-10-13T11:42:13.121 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:13.119+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.119762+0000) 2019-10-13T11:42:13.121 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:13.119+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.119992+0000) 2019-10-13T11:42:13.121 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:13.119+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.120119+0000) 2019-10-13T11:42:13.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:13.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.120279+0000) 2019-10-13T11:42:13.122 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:13.121+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.121851+0000) 2019-10-13T11:42:13.128 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:13.127+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:15.649 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:15.647+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:15.648832+0000) 2019-10-13T11:42:15.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:15.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:15.719335+0000) 2019-10-13T11:42:15.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:15.719+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:15.719335+0000) 2019-10-13T11:42:18.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:18.129+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.129565+0000) 2019-10-13T11:42:18.131 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:18.129+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.130124+0000) 2019-10-13T11:42:18.131 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:18.129+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.130365+0000) 2019-10-13T11:42:18.131 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:18.130+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.130556+0000) 2019-10-13T11:42:18.132 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:18.129+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.131155+0000) 2019-10-13T11:42:18.132 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:18.130+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.130954+0000) 2019-10-13T11:42:18.132 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:18.130+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.131159+0000) 2019-10-13T11:42:18.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:18.132+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.134113+0000) 2019-10-13T11:42:18.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:18.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215394 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:18.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:18.624+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.624775+0000) 2019-10-13T11:42:18.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:18.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.957293+0000) 2019-10-13T11:42:21.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:21.576+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.576325+0000) 2019-10-13T11:42:23.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:23.061+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.062045+0000) 2019-10-13T11:42:23.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:23.064+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.065597+0000) 2019-10-13T11:42:23.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:23.139+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.141079+0000) 2019-10-13T11:42:23.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:23.140+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.141105+0000) 2019-10-13T11:42:23.142 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:23.140+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.141373+0000) 2019-10-13T11:42:23.142 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:23.141+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.141505+0000) 2019-10-13T11:42:23.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:23.141+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.141699+0000) 2019-10-13T11:42:23.143 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:23.141+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.141822+0000) 2019-10-13T11:42:23.143 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:23.141+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.142048+0000) 2019-10-13T11:42:23.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:23.142+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.144639+0000) 2019-10-13T11:42:23.150 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:23.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215424 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:25.651 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:25.650+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:25.651217+0000) 2019-10-13T11:42:25.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:25.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:25.719534+0000) 2019-10-13T11:42:25.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:25.719+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:25.719534+0000) 2019-10-13T11:42:28.152 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:28.150+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.151114+0000) 2019-10-13T11:42:28.152 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:28.150+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.151297+0000) 2019-10-13T11:42:28.152 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:28.151+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.151530+0000) 2019-10-13T11:42:28.153 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:28.151+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.151785+0000) 2019-10-13T11:42:28.153 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:28.151+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.151845+0000) 2019-10-13T11:42:28.154 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:28.151+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.152007+0000) 2019-10-13T11:42:28.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:28.152+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.152393+0000) 2019-10-13T11:42:28.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:28.153+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.154196+0000) 2019-10-13T11:42:28.160 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:28.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215453 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:28.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:28.624+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.624961+0000) 2019-10-13T11:42:28.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:28.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.957420+0000) 2019-10-13T11:42:31.225 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:42:31.226 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:31.230 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:42:31.230 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:31.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:31.575+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.576513+0000) 2019-10-13T11:42:33.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:33.061+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.062249+0000) 2019-10-13T11:42:33.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:33.065+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.065816+0000) 2019-10-13T11:42:33.163 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:33.160+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.162275+0000) 2019-10-13T11:42:33.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:33.162+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.163300+0000) 2019-10-13T11:42:33.164 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:33.161+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.162600+0000) 2019-10-13T11:42:33.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:33.161+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.162744+0000) 2019-10-13T11:42:33.164 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:33.161+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.163126+0000) 2019-10-13T11:42:33.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:33.161+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.163363+0000) 2019-10-13T11:42:33.165 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:33.162+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.163913+0000) 2019-10-13T11:42:33.167 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:33.165+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.166693+0000) 2019-10-13T11:42:33.171 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:33.169+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:35.654 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:35.652+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:35.653585+0000) 2019-10-13T11:42:35.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:35.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:35.719720+0000) 2019-10-13T11:42:35.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:35.718+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:35.719720+0000) 2019-10-13T11:42:38.173 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:38.171+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.172795+0000) 2019-10-13T11:42:38.174 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:38.172+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.173568+0000) 2019-10-13T11:42:38.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:38.171+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.173125+0000) 2019-10-13T11:42:38.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:38.172+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.173454+0000) 2019-10-13T11:42:38.175 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:38.172+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.173865+0000) 2019-10-13T11:42:38.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:38.172+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.174074+0000) 2019-10-13T11:42:38.176 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:38.172+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.174147+0000) 2019-10-13T11:42:38.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:38.175+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.175760+0000) 2019-10-13T11:42:38.182 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:38.180+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215514 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:38.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:38.624+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.625097+0000) 2019-10-13T11:42:38.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:38.956+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.957587+0000) 2019-10-13T11:42:41.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:41.575+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.576681+0000) 2019-10-13T11:42:43.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:43.061+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.062431+0000) 2019-10-13T11:42:43.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:43.065+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.065984+0000) 2019-10-13T11:42:43.185 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:43.182+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.183900+0000) 2019-10-13T11:42:43.185 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:43.182+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.183958+0000) 2019-10-13T11:42:43.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:43.182+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.184103+0000) 2019-10-13T11:42:43.186 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:43.182+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.184292+0000) 2019-10-13T11:42:43.186 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:43.183+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.184617+0000) 2019-10-13T11:42:43.187 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:43.184+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.184834+0000) 2019-10-13T11:42:43.187 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:43.183+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.185327+0000) 2019-10-13T11:42:43.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:43.186+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.187488+0000) 2019-10-13T11:42:43.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:43.191+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:45.656 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:45.655+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:45.656014+0000) 2019-10-13T11:42:45.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:45.718+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:45.719946+0000) 2019-10-13T11:42:45.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:45.718+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:45.719946+0000) 2019-10-13T11:42:48.196 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:48.193+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.194961+0000) 2019-10-13T11:42:48.196 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:48.193+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.195078+0000) 2019-10-13T11:42:48.196 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:48.193+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.195332+0000) 2019-10-13T11:42:48.197 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:48.194+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.195505+0000) 2019-10-13T11:42:48.197 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:48.195+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.195771+0000) 2019-10-13T11:42:48.198 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:48.194+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.195838+0000) 2019-10-13T11:42:48.198 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:48.194+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.196019+0000) 2019-10-13T11:42:48.199 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:48.198+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.198952+0000) 2019-10-13T11:42:48.202 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:48.200+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:48.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:48.624+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.625255+0000) 2019-10-13T11:42:48.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:48.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.957747+0000) 2019-10-13T11:42:51.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:51.575+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.576872+0000) 2019-10-13T11:42:53.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:53.061+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.062574+0000) 2019-10-13T11:42:53.066 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:53.065+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.066186+0000) 2019-10-13T11:42:53.204 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:53.201+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.203192+0000) 2019-10-13T11:42:53.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:53.202+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.203573+0000) 2019-10-13T11:42:53.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:53.203+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.204120+0000) 2019-10-13T11:42:53.205 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:53.202+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.203922+0000) 2019-10-13T11:42:53.206 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:53.202+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.204340+0000) 2019-10-13T11:42:53.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:53.203+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.204699+0000) 2019-10-13T11:42:53.206 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:53.203+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.204757+0000) 2019-10-13T11:42:53.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:53.205+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.206525+0000) 2019-10-13T11:42:53.216 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:53.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215603 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:55.658 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:55.657+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:55.658378+0000) 2019-10-13T11:42:55.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:55.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:55.720120+0000) 2019-10-13T11:42:55.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:55.719+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:55.720120+0000) 2019-10-13T11:42:58.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:42:58.213+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.214210+0000) 2019-10-13T11:42:58.215 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:42:58.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.214348+0000) 2019-10-13T11:42:58.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:42:58.214+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.214661+0000) 2019-10-13T11:42:58.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:42:58.214+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.215307+0000) 2019-10-13T11:42:58.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:58.214+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.215094+0000) 2019-10-13T11:42:58.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:58.214+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.215355+0000) 2019-10-13T11:42:58.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:42:58.215+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.215547+0000) 2019-10-13T11:42:58.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:42:58.217+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.218330+0000) 2019-10-13T11:42:58.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:42:58.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215634 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:42:58.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:42:58.625+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.625420+0000) 2019-10-13T11:42:58.958 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:42:58.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.957918+0000) 2019-10-13T11:43:01.423 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:43:01.423 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:01.428 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:43:01.428 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:01.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:01.576+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.577054+0000) 2019-10-13T11:43:03.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:03.062+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.062790+0000) 2019-10-13T11:43:03.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:03.066+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.066396+0000) 2019-10-13T11:43:03.226 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:03.224+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.225126+0000) 2019-10-13T11:43:03.226 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:03.225+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.225620+0000) 2019-10-13T11:43:03.227 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:03.225+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.225759+0000) 2019-10-13T11:43:03.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:03.225+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.225891+0000) 2019-10-13T11:43:03.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:03.226+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.226748+0000) 2019-10-13T11:43:03.228 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:03.226+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.226528+0000) 2019-10-13T11:43:03.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:03.226+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.226790+0000) 2019-10-13T11:43:03.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:03.227+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.227845+0000) 2019-10-13T11:43:03.235 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:03.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215663 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:05.661 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:05.659+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:05.660622+0000) 2019-10-13T11:43:05.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:05.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:05.720293+0000) 2019-10-13T11:43:05.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:05.719+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:05.720293+0000) 2019-10-13T11:43:08.237 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:08.235+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.236253+0000) 2019-10-13T11:43:08.237 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:08.236+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.236461+0000) 2019-10-13T11:43:08.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:08.236+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.236582+0000) 2019-10-13T11:43:08.238 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:08.236+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.236829+0000) 2019-10-13T11:43:08.238 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:08.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.238194+0000) 2019-10-13T11:43:08.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:08.236+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.236958+0000) 2019-10-13T11:43:08.239 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:08.237+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.237476+0000) 2019-10-13T11:43:08.240 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:08.238+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.240181+0000) 2019-10-13T11:43:08.246 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:08.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215692 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:08.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:08.625+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.625565+0000) 2019-10-13T11:43:08.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:08.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.958089+0000) 2019-10-13T11:43:11.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:11.577+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.577250+0000) 2019-10-13T11:43:13.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:13.062+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.062974+0000) 2019-10-13T11:43:13.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:13.065+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.066586+0000) 2019-10-13T11:43:13.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:13.246+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.247023+0000) 2019-10-13T11:43:13.249 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:13.246+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.247724+0000) 2019-10-13T11:43:13.249 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:13.247+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.247618+0000) 2019-10-13T11:43:13.249 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:13.247+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.247751+0000) 2019-10-13T11:43:13.249 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:13.247+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.248042+0000) 2019-10-13T11:43:13.249 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:13.247+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.248382+0000) 2019-10-13T11:43:13.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:13.247+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.249591+0000) 2019-10-13T11:43:13.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:13.248+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.248633+0000) 2019-10-13T11:43:13.257 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:13.255+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215721 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:15.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:15.661+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:15.662990+0000) 2019-10-13T11:43:15.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:15.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:15.720478+0000) 2019-10-13T11:43:15.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:15.719+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:15.720478+0000) 2019-10-13T11:43:18.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:18.256+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.258017+0000) 2019-10-13T11:43:18.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:18.256+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.257875+0000) 2019-10-13T11:43:18.259 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:18.257+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.258446+0000) 2019-10-13T11:43:18.259 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:18.257+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.258509+0000) 2019-10-13T11:43:18.260 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:18.257+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.258663+0000) 2019-10-13T11:43:18.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:18.257+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.258856+0000) 2019-10-13T11:43:18.260 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:18.257+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.259040+0000) 2019-10-13T11:43:18.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:18.259+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.260409+0000) 2019-10-13T11:43:18.267 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:18.265+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215752 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:18.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:18.624+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.625710+0000) 2019-10-13T11:43:18.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:18.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.958277+0000) 2019-10-13T11:43:21.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:21.576+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.577442+0000) 2019-10-13T11:43:23.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:23.062+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.063155+0000) 2019-10-13T11:43:23.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:23.066+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.066810+0000) 2019-10-13T11:43:23.270 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:23.267+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.268782+0000) 2019-10-13T11:43:23.270 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:23.267+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.269032+0000) 2019-10-13T11:43:23.270 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:23.267+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.268884+0000) 2019-10-13T11:43:23.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:23.267+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.269264+0000) 2019-10-13T11:43:23.271 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:23.268+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.269536+0000) 2019-10-13T11:43:23.271 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:23.268+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.269711+0000) 2019-10-13T11:43:23.272 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:23.268+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.270072+0000) 2019-10-13T11:43:23.272 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:23.271+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.272389+0000) 2019-10-13T11:43:23.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:23.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215781 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:25.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:25.664+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:25.665252+0000) 2019-10-13T11:43:25.721 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:25.719+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:25.720657+0000) 2019-10-13T11:43:25.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:25.719+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:25.720657+0000) 2019-10-13T11:43:28.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:28.278+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.279476+0000) 2019-10-13T11:43:28.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:28.278+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.279580+0000) 2019-10-13T11:43:28.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:28.278+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.279897+0000) 2019-10-13T11:43:28.282 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:28.279+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.280656+0000) 2019-10-13T11:43:28.282 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:28.279+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.280499+0000) 2019-10-13T11:43:28.282 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:28.279+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.280716+0000) 2019-10-13T11:43:28.283 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:28.279+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.280858+0000) 2019-10-13T11:43:28.283 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:28.281+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.282567+0000) 2019-10-13T11:43:28.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:28.286+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:28.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:28.625+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.625905+0000) 2019-10-13T11:43:28.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:28.957+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.958381+0000) 2019-10-13T11:43:31.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:31.577+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.577667+0000) 2019-10-13T11:43:31.591 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:43:31.591 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:31.595 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:43:31.595 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:33.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:33.062+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.063287+0000) 2019-10-13T11:43:33.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:33.066+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.066989+0000) 2019-10-13T11:43:33.290 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:33.287+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.289194+0000) 2019-10-13T11:43:33.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:33.288+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.289792+0000) 2019-10-13T11:43:33.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:33.289+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.290361+0000) 2019-10-13T11:43:33.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:33.288+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.290384+0000) 2019-10-13T11:43:33.292 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:33.288+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.290238+0000) 2019-10-13T11:43:33.292 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:33.289+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.290609+0000) 2019-10-13T11:43:33.292 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:33.289+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.290765+0000) 2019-10-13T11:43:33.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:33.291+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.292579+0000) 2019-10-13T11:43:33.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:33.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215839 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:35.668 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:35.667+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:35.667524+0000) 2019-10-13T11:43:35.721 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:35.720+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:35.720829+0000) 2019-10-13T11:43:35.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:35.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:35.720829+0000) 2019-10-13T11:43:38.301 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:38.297+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.298959+0000) 2019-10-13T11:43:38.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:38.297+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.299203+0000) 2019-10-13T11:43:38.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:38.297+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.299317+0000) 2019-10-13T11:43:38.301 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:38.298+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.300013+0000) 2019-10-13T11:43:38.302 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:38.298+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.299773+0000) 2019-10-13T11:43:38.302 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:38.298+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.300181+0000) 2019-10-13T11:43:38.302 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:38.299+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.299985+0000) 2019-10-13T11:43:38.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:38.301+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.302312+0000) 2019-10-13T11:43:38.309 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:38.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215870 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:38.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:38.625+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.626068+0000) 2019-10-13T11:43:38.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:38.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.958541+0000) 2019-10-13T11:43:41.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:41.577+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.577862+0000) 2019-10-13T11:43:43.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:43.063+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.063446+0000) 2019-10-13T11:43:43.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:43.066+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.067165+0000) 2019-10-13T11:43:43.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:43.308+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.309287+0000) 2019-10-13T11:43:43.310 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:43.308+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.309330+0000) 2019-10-13T11:43:43.310 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:43.309+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.309451+0000) 2019-10-13T11:43:43.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:43.309+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.309901+0000) 2019-10-13T11:43:43.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:43.309+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.310269+0000) 2019-10-13T11:43:43.312 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:43.310+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.311096+0000) 2019-10-13T11:43:43.312 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:43.310+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.310918+0000) 2019-10-13T11:43:43.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:43.311+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.312510+0000) 2019-10-13T11:43:43.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:43.317+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215899 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:45.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:45.669+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:45.669960+0000) 2019-10-13T11:43:45.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:45.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:45.721015+0000) 2019-10-13T11:43:45.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:45.720+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:45.721015+0000) 2019-10-13T11:43:48.320 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:48.318+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.319173+0000) 2019-10-13T11:43:48.320 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:48.318+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.319381+0000) 2019-10-13T11:43:48.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:48.319+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.319876+0000) 2019-10-13T11:43:48.321 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:48.319+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.320029+0000) 2019-10-13T11:43:48.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:48.319+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.320098+0000) 2019-10-13T11:43:48.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:48.319+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.319946+0000) 2019-10-13T11:43:48.322 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:48.320+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.320429+0000) 2019-10-13T11:43:48.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:48.322+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.323052+0000) 2019-10-13T11:43:48.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:48.328+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215928 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:48.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:48.625+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.626255+0000) 2019-10-13T11:43:48.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:48.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.958704+0000) 2019-10-13T11:43:51.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:51.577+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.578102+0000) 2019-10-13T11:43:53.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:53.063+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.063642+0000) 2019-10-13T11:43:53.067 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:53.067+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.067335+0000) 2019-10-13T11:43:53.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:53.329+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.329697+0000) 2019-10-13T11:43:53.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:53.329+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.330014+0000) 2019-10-13T11:43:53.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:53.329+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.330072+0000) 2019-10-13T11:43:53.332 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:53.329+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.330150+0000) 2019-10-13T11:43:53.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:53.330+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.330458+0000) 2019-10-13T11:43:53.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:53.330+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.330594+0000) 2019-10-13T11:43:53.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:53.330+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.331107+0000) 2019-10-13T11:43:53.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:53.332+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.333343+0000) 2019-10-13T11:43:53.337 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:53.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:55.673 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:55.671+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:55.672274+0000) 2019-10-13T11:43:55.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:55.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:55.721205+0000) 2019-10-13T11:43:55.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:55.721+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:55.721205+0000) 2019-10-13T11:43:58.339 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:58.338+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.338692+0000) 2019-10-13T11:43:58.340 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:43:58.337+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.339597+0000) 2019-10-13T11:43:58.340 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:58.338+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.339040+0000) 2019-10-13T11:43:58.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:43:58.338+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.338975+0000) 2019-10-13T11:43:58.341 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:43:58.338+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.339063+0000) 2019-10-13T11:43:58.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:43:58.338+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.339366+0000) 2019-10-13T11:43:58.341 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:43:58.339+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.339929+0000) 2019-10-13T11:43:58.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:43:58.340+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.341978+0000) 2019-10-13T11:43:58.348 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:43:58.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 215990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:43:58.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:43:58.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.626425+0000) 2019-10-13T11:43:58.959 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:43:58.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.958905+0000) 2019-10-13T11:44:01.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:01.576+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.578296+0000) 2019-10-13T11:44:01.760 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:44:01.760 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:01.765 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:44:01.765 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:03.064 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:03.062+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.063824+0000) 2019-10-13T11:44:03.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:03.065+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.067525+0000) 2019-10-13T11:44:03.350 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:03.347+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.349378+0000) 2019-10-13T11:44:03.350 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:03.348+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.349629+0000) 2019-10-13T11:44:03.351 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:03.348+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.350639+0000) 2019-10-13T11:44:03.352 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:03.348+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.350013+0000) 2019-10-13T11:44:03.352 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:03.348+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.350153+0000) 2019-10-13T11:44:03.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:03.349+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.350541+0000) 2019-10-13T11:44:03.352 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:03.349+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.350753+0000) 2019-10-13T11:44:03.353 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:03.350+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.352437+0000) 2019-10-13T11:44:03.359 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:03.356+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216021 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:05.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:05.673+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:05.674740+0000) 2019-10-13T11:44:05.722 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:05.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:05.721431+0000) 2019-10-13T11:44:05.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:05.720+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:05.721431+0000) 2019-10-13T11:44:08.361 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:08.358+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.360040+0000) 2019-10-13T11:44:08.361 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:08.358+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.360248+0000) 2019-10-13T11:44:08.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:08.359+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.360693+0000) 2019-10-13T11:44:08.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:08.359+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.360827+0000) 2019-10-13T11:44:08.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:08.359+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.360767+0000) 2019-10-13T11:44:08.363 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:08.359+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.360943+0000) 2019-10-13T11:44:08.363 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:08.359+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.361303+0000) 2019-10-13T11:44:08.364 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:08.362+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.364045+0000) 2019-10-13T11:44:08.371 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:08.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:08.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:08.625+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.626592+0000) 2019-10-13T11:44:08.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:08.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.959021+0000) 2019-10-13T11:44:11.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:11.577+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.578476+0000) 2019-10-13T11:44:13.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:13.063+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.063997+0000) 2019-10-13T11:44:13.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:13.066+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.067710+0000) 2019-10-13T11:44:13.371 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:13.370+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.370975+0000) 2019-10-13T11:44:13.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:13.369+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.370728+0000) 2019-10-13T11:44:13.372 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:13.369+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.370858+0000) 2019-10-13T11:44:13.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:13.369+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.371148+0000) 2019-10-13T11:44:13.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:13.370+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.371412+0000) 2019-10-13T11:44:13.373 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:13.370+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.371824+0000) 2019-10-13T11:44:13.373 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:13.370+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.371948+0000) 2019-10-13T11:44:13.375 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:13.373+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.374638+0000) 2019-10-13T11:44:13.380 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:13.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216083 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:15.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:15.677+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:15.677796+0000) 2019-10-13T11:44:15.722 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:15.720+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:15.721641+0000) 2019-10-13T11:44:15.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:15.720+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:15.721642+0000) 2019-10-13T11:44:18.382 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:18.381+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.381756+0000) 2019-10-13T11:44:18.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:18.380+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.381699+0000) 2019-10-13T11:44:18.383 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:18.380+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.381760+0000) 2019-10-13T11:44:18.383 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:18.380+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.382080+0000) 2019-10-13T11:44:18.384 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:18.380+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.382381+0000) 2019-10-13T11:44:18.384 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:18.381+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.382659+0000) 2019-10-13T11:44:18.384 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:18.381+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.382764+0000) 2019-10-13T11:44:18.385 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:18.383+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.384688+0000) 2019-10-13T11:44:18.391 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:18.389+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216112 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:18.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:18.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.626760+0000) 2019-10-13T11:44:18.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:18.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.959180+0000) 2019-10-13T11:44:21.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:21.578+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.578667+0000) 2019-10-13T11:44:23.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:23.063+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.064206+0000) 2019-10-13T11:44:23.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:23.067+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.067908+0000) 2019-10-13T11:44:23.393 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:23.392+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.393325+0000) 2019-10-13T11:44:23.394 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:23.391+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.393118+0000) 2019-10-13T11:44:23.395 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:23.391+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.393222+0000) 2019-10-13T11:44:23.395 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:23.392+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.393513+0000) 2019-10-13T11:44:23.395 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:23.392+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.393762+0000) 2019-10-13T11:44:23.395 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:23.392+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.393801+0000) 2019-10-13T11:44:23.395 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:23.392+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.394137+0000) 2019-10-13T11:44:23.396 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:23.394+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.395348+0000) 2019-10-13T11:44:23.402 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:23.400+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:25.680 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:25.679+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:25.679882+0000) 2019-10-13T11:44:25.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:25.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:25.721861+0000) 2019-10-13T11:44:25.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:25.721+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:25.721861+0000) 2019-10-13T11:44:28.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:28.403+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.404460+0000) 2019-10-13T11:44:28.405 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:28.403+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.403924+0000) 2019-10-13T11:44:28.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:28.403+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.404285+0000) 2019-10-13T11:44:28.406 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:28.404+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.404486+0000) 2019-10-13T11:44:28.407 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:28.404+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.404629+0000) 2019-10-13T11:44:28.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:28.404+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.404971+0000) 2019-10-13T11:44:28.407 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:28.404+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.405002+0000) 2019-10-13T11:44:28.408 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:28.406+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.407612+0000) 2019-10-13T11:44:28.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:28.412+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216172 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:28.627 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:28.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.626922+0000) 2019-10-13T11:44:28.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:28.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.959346+0000) 2019-10-13T11:44:31.579 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:31.578+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.578849+0000) 2019-10-13T11:44:31.924 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:44:31.924 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:31.929 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:44:31.929 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:33.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:33.064+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.064394+0000) 2019-10-13T11:44:33.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:33.067+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.068096+0000) 2019-10-13T11:44:33.415 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:33.414+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.414845+0000) 2019-10-13T11:44:33.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:33.414+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.415163+0000) 2019-10-13T11:44:33.417 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:33.415+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.416134+0000) 2019-10-13T11:44:33.417 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:33.414+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.415302+0000) 2019-10-13T11:44:33.417 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:33.415+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.415464+0000) 2019-10-13T11:44:33.417 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:33.415+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.415612+0000) 2019-10-13T11:44:33.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:33.415+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.416289+0000) 2019-10-13T11:44:33.418 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:33.416+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.417498+0000) 2019-10-13T11:44:33.425 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:33.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:35.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:35.681+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:35.682162+0000) 2019-10-13T11:44:35.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:35.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:35.722015+0000) 2019-10-13T11:44:35.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:35.721+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:35.722016+0000) 2019-10-13T11:44:38.427 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:38.426+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.426797+0000) 2019-10-13T11:44:38.428 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:38.426+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.426646+0000) 2019-10-13T11:44:38.428 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:38.426+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.426759+0000) 2019-10-13T11:44:38.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:38.426+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.426886+0000) 2019-10-13T11:44:38.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:38.427+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.427419+0000) 2019-10-13T11:44:38.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:38.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.427673+0000) 2019-10-13T11:44:38.429 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:38.427+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.427972+0000) 2019-10-13T11:44:38.431 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:38.430+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.430792+0000) 2019-10-13T11:44:38.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:38.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216234 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:38.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:38.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.627120+0000) 2019-10-13T11:44:38.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:38.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.959458+0000) 2019-10-13T11:44:41.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:41.578+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.579026+0000) 2019-10-13T11:44:43.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:43.064+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.064570+0000) 2019-10-13T11:44:43.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:43.068+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.068264+0000) 2019-10-13T11:44:43.439 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:43.437+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.437994+0000) 2019-10-13T11:44:43.440 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:43.438+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.438888+0000) 2019-10-13T11:44:43.440 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:43.438+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.438655+0000) 2019-10-13T11:44:43.440 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:43.438+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.438891+0000) 2019-10-13T11:44:43.441 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:43.438+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.439143+0000) 2019-10-13T11:44:43.441 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:43.438+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.439303+0000) 2019-10-13T11:44:43.441 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:43.439+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.439544+0000) 2019-10-13T11:44:43.442 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:43.441+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.441965+0000) 2019-10-13T11:44:43.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:43.446+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216265 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:45.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:45.684+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:45.684674+0000) 2019-10-13T11:44:45.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:45.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:45.722171+0000) 2019-10-13T11:44:45.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:45.721+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:45.722171+0000) 2019-10-13T11:44:48.450 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:48.447+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.448595+0000) 2019-10-13T11:44:48.450 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:48.447+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.448900+0000) 2019-10-13T11:44:48.450 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:48.447+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.448945+0000) 2019-10-13T11:44:48.450 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:48.447+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.449277+0000) 2019-10-13T11:44:48.451 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:48.448+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.449432+0000) 2019-10-13T11:44:48.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:48.448+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.450005+0000) 2019-10-13T11:44:48.452 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:48.449+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.451026+0000) 2019-10-13T11:44:48.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:48.448+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.450007+0000) 2019-10-13T11:44:48.457 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:48.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216294 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:48.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:48.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.627306+0000) 2019-10-13T11:44:48.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:48.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.959547+0000) 2019-10-13T11:44:51.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:51.578+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.579224+0000) 2019-10-13T11:44:53.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:53.063+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.064774+0000) 2019-10-13T11:44:53.068 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:53.066+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.068414+0000) 2019-10-13T11:44:53.459 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:53.456+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.458630+0000) 2019-10-13T11:44:53.460 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:53.457+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.458438+0000) 2019-10-13T11:44:53.460 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:53.457+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.458684+0000) 2019-10-13T11:44:53.460 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:53.457+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.458744+0000) 2019-10-13T11:44:53.461 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:53.457+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.459012+0000) 2019-10-13T11:44:53.461 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:53.457+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.459120+0000) 2019-10-13T11:44:53.461 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:53.458+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.459631+0000) 2019-10-13T11:44:53.462 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:53.459+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.460971+0000) 2019-10-13T11:44:53.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:53.465+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216327 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:55.687 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:55.686+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:55.687114+0000) 2019-10-13T11:44:55.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:55.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:55.722359+0000) 2019-10-13T11:44:55.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:55.721+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:55.722359+0000) 2019-10-13T11:44:58.469 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:44:58.467+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.468793+0000) 2019-10-13T11:44:58.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:58.467+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.468637+0000) 2019-10-13T11:44:58.470 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:58.467+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.468758+0000) 2019-10-13T11:44:58.470 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:44:58.467+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.469046+0000) 2019-10-13T11:44:58.471 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:44:58.467+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.469231+0000) 2019-10-13T11:44:58.471 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:44:58.468+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.469563+0000) 2019-10-13T11:44:58.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:44:58.468+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.469683+0000) 2019-10-13T11:44:58.471 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:44:58.469+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.471042+0000) 2019-10-13T11:44:58.476 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:44:58.474+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216356 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:44:58.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:44:58.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.627441+0000) 2019-10-13T11:44:58.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:44:58.958+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.959693+0000) 2019-10-13T11:45:01.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:01.578+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.579384+0000) 2019-10-13T11:45:02.014 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:45:02.014 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:02.045 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:45:02.045 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:03.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:03.064+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.064976+0000) 2019-10-13T11:45:03.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:03.067+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.068586+0000) 2019-10-13T11:45:03.478 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:03.476+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.477547+0000) 2019-10-13T11:45:03.479 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:03.477+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.478271+0000) 2019-10-13T11:45:03.479 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:03.476+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.477823+0000) 2019-10-13T11:45:03.479 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:03.477+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.478425+0000) 2019-10-13T11:45:03.480 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:03.477+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.478514+0000) 2019-10-13T11:45:03.480 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:03.477+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.478632+0000) 2019-10-13T11:45:03.480 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:03.477+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.478905+0000) 2019-10-13T11:45:03.480 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:03.479+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.480011+0000) 2019-10-13T11:45:03.487 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:03.485+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:05.690 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:05.688+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:05.689581+0000) 2019-10-13T11:45:05.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:05.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:05.722527+0000) 2019-10-13T11:45:05.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:05.721+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:05.722527+0000) 2019-10-13T11:45:08.490 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:08.487+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.489123+0000) 2019-10-13T11:45:08.490 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:08.488+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.489483+0000) 2019-10-13T11:45:08.491 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:08.488+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.489638+0000) 2019-10-13T11:45:08.491 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:08.488+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.489677+0000) 2019-10-13T11:45:08.491 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:08.488+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.489862+0000) 2019-10-13T11:45:08.492 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:08.488+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.490303+0000) 2019-10-13T11:45:08.492 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:08.488+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.490379+0000) 2019-10-13T11:45:08.492 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:08.491+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.492588+0000) 2019-10-13T11:45:08.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:08.495+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216416 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:08.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:08.627+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.627588+0000) 2019-10-13T11:45:08.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:08.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.959839+0000) 2019-10-13T11:45:11.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:11.579+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.579509+0000) 2019-10-13T11:45:13.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:13.064+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.065099+0000) 2019-10-13T11:45:13.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:13.068+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.068720+0000) 2019-10-13T11:45:13.500 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:13.498+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.499151+0000) 2019-10-13T11:45:13.500 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:13.499+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.499534+0000) 2019-10-13T11:45:13.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:13.499+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.499801+0000) 2019-10-13T11:45:13.501 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:13.499+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.499991+0000) 2019-10-13T11:45:13.502 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:13.499+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.500567+0000) 2019-10-13T11:45:13.502 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:13.500+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.500414+0000) 2019-10-13T11:45:13.502 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:13.500+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.501015+0000) 2019-10-13T11:45:13.503 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:13.502+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.503218+0000) 2019-10-13T11:45:13.509 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:13.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216449 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:15.692 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:15.691+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:15.692105+0000) 2019-10-13T11:45:15.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:15.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:15.722682+0000) 2019-10-13T11:45:15.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:15.722+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:15.722682+0000) 2019-10-13T11:45:18.511 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:18.509+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.510636+0000) 2019-10-13T11:45:18.512 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:18.510+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.510592+0000) 2019-10-13T11:45:18.512 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:18.510+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.510708+0000) 2019-10-13T11:45:18.513 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:18.510+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.511053+0000) 2019-10-13T11:45:18.513 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:18.510+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.511327+0000) 2019-10-13T11:45:18.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:18.511+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.511489+0000) 2019-10-13T11:45:18.514 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:18.511+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.511713+0000) 2019-10-13T11:45:18.514 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:18.512+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.512862+0000) 2019-10-13T11:45:18.520 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:18.518+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216478 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:18.628 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:18.627+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.627754+0000) 2019-10-13T11:45:18.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:18.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.960020+0000) 2019-10-13T11:45:21.580 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:21.579+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.579693+0000) 2019-10-13T11:45:23.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:23.065+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.065265+0000) 2019-10-13T11:45:23.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:23.068+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.068890+0000) 2019-10-13T11:45:23.522 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:23.520+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.521483+0000) 2019-10-13T11:45:23.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:23.520+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.521225+0000) 2019-10-13T11:45:23.523 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:23.520+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.521339+0000) 2019-10-13T11:45:23.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:23.521+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.521755+0000) 2019-10-13T11:45:23.523 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:23.521+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.521936+0000) 2019-10-13T11:45:23.524 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:23.522+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.522525+0000) 2019-10-13T11:45:23.524 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:23.522+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.522570+0000) 2019-10-13T11:45:23.524 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:23.523+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.524316+0000) 2019-10-13T11:45:23.532 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:23.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:25.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:25.694+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:25.694388+0000) 2019-10-13T11:45:25.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:25.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:25.722819+0000) 2019-10-13T11:45:25.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:25.722+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:25.722819+0000) 2019-10-13T11:45:28.534 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:28.532+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.532776+0000) 2019-10-13T11:45:28.534 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:28.532+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.532930+0000) 2019-10-13T11:45:28.534 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:28.532+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.533228+0000) 2019-10-13T11:45:28.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:28.533+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.533442+0000) 2019-10-13T11:45:28.535 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:28.532+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.533665+0000) 2019-10-13T11:45:28.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:28.533+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.533668+0000) 2019-10-13T11:45:28.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:28.533+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.533829+0000) 2019-10-13T11:45:28.536 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:28.535+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.535962+0000) 2019-10-13T11:45:28.540 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:28.539+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216538 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:28.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:28.627+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.627953+0000) 2019-10-13T11:45:28.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:28.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.960149+0000) 2019-10-13T11:45:31.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:31.578+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.579889+0000) 2019-10-13T11:45:32.202 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:45:32.202 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:32.207 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:45:32.208 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:33.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:33.064+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.065446+0000) 2019-10-13T11:45:33.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:33.068+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.069048+0000) 2019-10-13T11:45:33.542 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:33.540+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.541413+0000) 2019-10-13T11:45:33.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:33.540+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.541812+0000) 2019-10-13T11:45:33.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:33.540+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.541582+0000) 2019-10-13T11:45:33.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:33.540+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.541950+0000) 2019-10-13T11:45:33.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:33.541+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.542552+0000) 2019-10-13T11:45:33.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:33.541+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.542637+0000) 2019-10-13T11:45:33.544 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:33.541+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.542569+0000) 2019-10-13T11:45:33.545 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:33.543+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.544443+0000) 2019-10-13T11:45:33.551 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:33.549+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216571 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:35.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:35.696+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:35.696705+0000) 2019-10-13T11:45:35.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:35.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:35.722977+0000) 2019-10-13T11:45:35.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:35.722+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:35.722977+0000) 2019-10-13T11:45:38.553 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:38.550+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.552123+0000) 2019-10-13T11:45:38.553 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:38.550+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.552185+0000) 2019-10-13T11:45:38.554 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:38.551+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.552863+0000) 2019-10-13T11:45:38.554 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:38.551+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.553069+0000) 2019-10-13T11:45:38.555 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:38.552+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.553429+0000) 2019-10-13T11:45:38.555 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:38.552+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.553925+0000) 2019-10-13T11:45:38.555 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:38.552+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.554130+0000) 2019-10-13T11:45:38.556 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:38.554+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.556395+0000) 2019-10-13T11:45:38.562 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:38.560+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216600 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:38.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:38.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.628138+0000) 2019-10-13T11:45:38.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:38.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.960306+0000) 2019-10-13T11:45:41.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:41.579+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.580088+0000) 2019-10-13T11:45:43.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:43.064+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.065640+0000) 2019-10-13T11:45:43.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:43.068+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.069214+0000) 2019-10-13T11:45:43.564 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:43.561+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.563526+0000) 2019-10-13T11:45:43.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:43.561+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.563355+0000) 2019-10-13T11:45:43.565 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:43.562+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.563492+0000) 2019-10-13T11:45:43.565 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:43.562+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.563794+0000) 2019-10-13T11:45:43.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:43.562+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.563944+0000) 2019-10-13T11:45:43.566 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:43.562+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.564184+0000) 2019-10-13T11:45:43.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:43.563+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.564494+0000) 2019-10-13T11:45:43.567 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:43.565+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.567066+0000) 2019-10-13T11:45:43.573 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:43.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216631 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:45.700 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:45.698+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:45.699808+0000) 2019-10-13T11:45:45.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:45.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:45.723131+0000) 2019-10-13T11:45:45.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:45.722+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:45.723131+0000) 2019-10-13T11:45:48.575 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:48.573+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.574570+0000) 2019-10-13T11:45:48.576 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:48.573+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.574783+0000) 2019-10-13T11:45:48.577 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:48.573+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.575444+0000) 2019-10-13T11:45:48.577 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:48.573+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.575363+0000) 2019-10-13T11:45:48.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:48.574+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.575492+0000) 2019-10-13T11:45:48.577 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:48.574+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.575616+0000) 2019-10-13T11:45:48.578 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:48.574+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.575954+0000) 2019-10-13T11:45:48.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:48.575+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.576823+0000) 2019-10-13T11:45:48.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:48.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216660 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:48.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:48.626+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.628310+0000) 2019-10-13T11:45:48.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:48.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.960411+0000) 2019-10-13T11:45:51.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:51.579+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.580253+0000) 2019-10-13T11:45:53.066 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:53.065+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.065878+0000) 2019-10-13T11:45:53.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:53.068+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.069420+0000) 2019-10-13T11:45:53.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:53.583+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.585389+0000) 2019-10-13T11:45:53.587 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:53.584+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.585638+0000) 2019-10-13T11:45:53.587 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:53.584+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.585811+0000) 2019-10-13T11:45:53.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:53.586+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.586762+0000) 2019-10-13T11:45:53.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:53.584+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.586221+0000) 2019-10-13T11:45:53.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:53.585+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.586522+0000) 2019-10-13T11:45:53.588 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:53.585+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.586697+0000) 2019-10-13T11:45:53.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:53.588+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.589302+0000) 2019-10-13T11:45:53.595 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:53.592+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216693 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:55.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:55.701+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:55.702081+0000) 2019-10-13T11:45:55.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:55.721+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:55.723327+0000) 2019-10-13T11:45:55.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:55.721+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:55.723327+0000) 2019-10-13T11:45:58.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:45:58.595+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.596421+0000) 2019-10-13T11:45:58.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:45:58.595+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.596359+0000) 2019-10-13T11:45:58.598 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:58.596+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.596734+0000) 2019-10-13T11:45:58.598 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:58.596+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.596824+0000) 2019-10-13T11:45:58.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:45:58.596+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.596871+0000) 2019-10-13T11:45:58.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:45:58.596+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.597096+0000) 2019-10-13T11:45:58.599 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:45:58.596+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.597239+0000) 2019-10-13T11:45:58.599 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:45:58.597+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.598295+0000) 2019-10-13T11:45:58.605 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:45:58.604+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216722 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:45:58.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:45:58.628+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.628475+0000) 2019-10-13T11:45:58.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:45:58.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.960568+0000) 2019-10-13T11:46:01.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:01.580+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.580433+0000) 2019-10-13T11:46:02.393 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:46:02.394 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:02.398 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:46:02.398 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:03.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:03.065+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.066084+0000) 2019-10-13T11:46:03.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:03.069+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.069560+0000) 2019-10-13T11:46:03.607 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:03.606+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.606574+0000) 2019-10-13T11:46:03.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:03.606+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.606887+0000) 2019-10-13T11:46:03.608 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:03.606+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.607043+0000) 2019-10-13T11:46:03.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:03.606+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.607000+0000) 2019-10-13T11:46:03.609 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:03.607+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.607527+0000) 2019-10-13T11:46:03.609 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:03.607+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.607695+0000) 2019-10-13T11:46:03.609 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:03.607+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.607842+0000) 2019-10-13T11:46:03.610 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:03.608+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.609537+0000) 2019-10-13T11:46:03.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:03.615+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216753 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:05.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:05.704+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:05.704506+0000) 2019-10-13T11:46:05.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:05.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:05.723574+0000) 2019-10-13T11:46:05.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:05.723+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:05.723574+0000) 2019-10-13T11:46:08.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:08.616+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.617214+0000) 2019-10-13T11:46:08.618 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:08.617+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.617557+0000) 2019-10-13T11:46:08.619 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:08.617+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.617779+0000) 2019-10-13T11:46:08.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:08.617+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.618040+0000) 2019-10-13T11:46:08.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:08.617+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.618032+0000) 2019-10-13T11:46:08.620 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:08.617+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.618312+0000) 2019-10-13T11:46:08.620 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:08.618+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.618423+0000) 2019-10-13T11:46:08.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:08.619+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.620099+0000) 2019-10-13T11:46:08.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:08.625+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216782 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:08.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:08.628+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.628592+0000) 2019-10-13T11:46:08.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:08.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.960703+0000) 2019-10-13T11:46:11.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:11.580+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.580636+0000) 2019-10-13T11:46:13.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:13.065+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.066263+0000) 2019-10-13T11:46:13.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:13.069+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.069725+0000) 2019-10-13T11:46:13.629 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:13.627+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.628499+0000) 2019-10-13T11:46:13.630 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:13.627+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.628010+0000) 2019-10-13T11:46:13.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:13.627+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.628360+0000) 2019-10-13T11:46:13.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:13.628+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.628557+0000) 2019-10-13T11:46:13.631 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:13.628+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.629071+0000) 2019-10-13T11:46:13.631 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:13.628+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.629287+0000) 2019-10-13T11:46:13.631 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:13.629+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.629714+0000) 2019-10-13T11:46:13.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:13.630+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.631454+0000) 2019-10-13T11:46:13.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:13.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216815 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:15.707 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:15.706+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:15.706802+0000) 2019-10-13T11:46:15.724 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:15.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:15.723760+0000) 2019-10-13T11:46:15.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:15.722+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:15.723760+0000) 2019-10-13T11:46:18.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:18.627+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.628747+0000) 2019-10-13T11:46:18.640 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:18.637+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.638842+0000) 2019-10-13T11:46:18.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:18.637+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.639018+0000) 2019-10-13T11:46:18.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:18.637+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.639137+0000) 2019-10-13T11:46:18.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:18.637+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.639375+0000) 2019-10-13T11:46:18.641 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:18.639+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.639878+0000) 2019-10-13T11:46:18.641 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:18.638+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.639971+0000) 2019-10-13T11:46:18.642 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:18.639+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.640895+0000) 2019-10-13T11:46:18.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:18.642+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.643313+0000) 2019-10-13T11:46:18.649 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:18.646+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216845 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:18.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:18.959+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.960864+0000) 2019-10-13T11:46:21.581 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:21.579+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.580849+0000) 2019-10-13T11:46:23.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:23.065+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.066465+0000) 2019-10-13T11:46:23.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:23.069+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.069905+0000) 2019-10-13T11:46:23.650 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:23.649+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.650130+0000) 2019-10-13T11:46:23.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:23.648+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.650174+0000) 2019-10-13T11:46:23.652 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:23.648+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.650265+0000) 2019-10-13T11:46:23.652 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:23.648+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.650307+0000) 2019-10-13T11:46:23.652 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:23.649+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.650642+0000) 2019-10-13T11:46:23.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:23.649+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.650393+0000) 2019-10-13T11:46:23.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:23.649+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.651162+0000) 2019-10-13T11:46:23.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:23.652+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.653172+0000) 2019-10-13T11:46:23.659 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:23.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216875 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:25.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:25.709+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:25.709163+0000) 2019-10-13T11:46:25.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:25.722+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:25.723973+0000) 2019-10-13T11:46:25.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:25.722+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:25.723972+0000) 2019-10-13T11:46:28.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:28.628+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.628933+0000) 2019-10-13T11:46:28.661 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:28.659+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.660432+0000) 2019-10-13T11:46:28.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:28.659+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.660535+0000) 2019-10-13T11:46:28.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:28.659+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.660612+0000) 2019-10-13T11:46:28.662 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:28.659+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.661154+0000) 2019-10-13T11:46:28.663 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:28.659+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.661036+0000) 2019-10-13T11:46:28.663 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:28.659+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.661066+0000) 2019-10-13T11:46:28.663 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:28.659+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.661206+0000) 2019-10-13T11:46:28.664 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:28.661+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.663709+0000) 2019-10-13T11:46:28.667 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:28.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216905 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:28.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:28.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.961037+0000) 2019-10-13T11:46:31.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:31.580+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.581058+0000) 2019-10-13T11:46:32.599 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:46:32.599 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:32.603 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:46:32.604 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:33.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:33.065+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.066611+0000) 2019-10-13T11:46:33.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:33.069+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.070075+0000) 2019-10-13T11:46:33.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:33.667+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.668836+0000) 2019-10-13T11:46:33.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:33.667+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.669105+0000) 2019-10-13T11:46:33.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:33.667+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.669210+0000) 2019-10-13T11:46:33.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:33.668+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.670245+0000) 2019-10-13T11:46:33.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:33.668+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.669474+0000) 2019-10-13T11:46:33.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:33.668+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.669704+0000) 2019-10-13T11:46:33.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:33.668+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.670096+0000) 2019-10-13T11:46:33.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:33.669+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.671413+0000) 2019-10-13T11:46:33.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:33.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216937 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:35.712 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:35.710+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:35.711602+0000) 2019-10-13T11:46:35.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:35.723+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:35.724222+0000) 2019-10-13T11:46:35.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:35.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:35.724222+0000) 2019-10-13T11:46:38.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:38.628+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.629100+0000) 2019-10-13T11:46:38.680 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:38.678+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.680058+0000) 2019-10-13T11:46:38.681 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:38.678+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.679929+0000) 2019-10-13T11:46:38.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:38.678+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.680125+0000) 2019-10-13T11:46:38.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:38.679+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.680391+0000) 2019-10-13T11:46:38.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:38.679+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.680539+0000) 2019-10-13T11:46:38.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:38.679+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.681048+0000) 2019-10-13T11:46:38.682 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:38.679+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.680783+0000) 2019-10-13T11:46:38.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:38.681+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.683451+0000) 2019-10-13T11:46:38.689 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:38.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216967 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:38.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:38.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.961182+0000) 2019-10-13T11:46:41.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:41.580+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.581278+0000) 2019-10-13T11:46:43.067 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:43.066+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.066803+0000) 2019-10-13T11:46:43.070 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:43.069+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.070313+0000) 2019-10-13T11:46:43.692 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:43.690+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.691144+0000) 2019-10-13T11:46:43.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:43.690+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.691715+0000) 2019-10-13T11:46:43.693 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:43.691+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.691770+0000) 2019-10-13T11:46:43.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:43.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.692114+0000) 2019-10-13T11:46:43.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:43.691+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.692214+0000) 2019-10-13T11:46:43.694 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:43.692+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.692456+0000) 2019-10-13T11:46:43.694 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:43.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.692562+0000) 2019-10-13T11:46:43.694 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:43.693+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.693956+0000) 2019-10-13T11:46:43.701 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:43.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 216997 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:45.714 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:45.713+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:45.713986+0000) 2019-10-13T11:46:45.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:45.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:45.724408+0000) 2019-10-13T11:46:45.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:45.724+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:45.724408+0000) 2019-10-13T11:46:48.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:48.629+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.629284+0000) 2019-10-13T11:46:48.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:48.701+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.701874+0000) 2019-10-13T11:46:48.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:48.701+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.702047+0000) 2019-10-13T11:46:48.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:48.702+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.703021+0000) 2019-10-13T11:46:48.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:48.701+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.702190+0000) 2019-10-13T11:46:48.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:48.702+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.702532+0000) 2019-10-13T11:46:48.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:48.702+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.703260+0000) 2019-10-13T11:46:48.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:48.703+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.703501+0000) 2019-10-13T11:46:48.705 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:48.704+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.705289+0000) 2019-10-13T11:46:48.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:48.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217028 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:48.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:48.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.961291+0000) 2019-10-13T11:46:51.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:51.581+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.581473+0000) 2019-10-13T11:46:53.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:53.066+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.066990+0000) 2019-10-13T11:46:53.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:53.069+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.070517+0000) 2019-10-13T11:46:53.713 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:53.712+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.712637+0000) 2019-10-13T11:46:53.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:53.712+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.712767+0000) 2019-10-13T11:46:53.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:53.712+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.712932+0000) 2019-10-13T11:46:53.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:53.712+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.713703+0000) 2019-10-13T11:46:53.715 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:53.712+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.713291+0000) 2019-10-13T11:46:53.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:53.713+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.713663+0000) 2019-10-13T11:46:53.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:53.713+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.713552+0000) 2019-10-13T11:46:53.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:53.715+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.716500+0000) 2019-10-13T11:46:53.722 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:53.721+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217059 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:55.717 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:55.715+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:55.716306+0000) 2019-10-13T11:46:55.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:55.724+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:55.724581+0000) 2019-10-13T11:46:55.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:55.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:55.724581+0000) 2019-10-13T11:46:58.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:58.629+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.629527+0000) 2019-10-13T11:46:58.724 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:46:58.722+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.723395+0000) 2019-10-13T11:46:58.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:46:58.722+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.723255+0000) 2019-10-13T11:46:58.725 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:46:58.723+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.723519+0000) 2019-10-13T11:46:58.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:46:58.723+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.723741+0000) 2019-10-13T11:46:58.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:46:58.723+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.723832+0000) 2019-10-13T11:46:58.726 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:46:58.723+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.724159+0000) 2019-10-13T11:46:58.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:58.724+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.724493+0000) 2019-10-13T11:46:58.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:46:58.724+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.725527+0000) 2019-10-13T11:46:58.735 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:46:58.732+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:46:58.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:46:58.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.961435+0000) 2019-10-13T11:47:01.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:01.580+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.581666+0000) 2019-10-13T11:47:02.755 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:47:02.755 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:02.759 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:47:02.759 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:03.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:03.066+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.067175+0000) 2019-10-13T11:47:03.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:03.070+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.070713+0000) 2019-10-13T11:47:03.735 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:03.732+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.734159+0000) 2019-10-13T11:47:03.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:03.732+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.734344+0000) 2019-10-13T11:47:03.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:03.733+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.734586+0000) 2019-10-13T11:47:03.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:03.733+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.734792+0000) 2019-10-13T11:47:03.737 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:03.734+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.735282+0000) 2019-10-13T11:47:03.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:03.733+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.735070+0000) 2019-10-13T11:47:03.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:03.733+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.735337+0000) 2019-10-13T11:47:03.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:03.736+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.737176+0000) 2019-10-13T11:47:03.742 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:03.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217119 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:05.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:05.718+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:05.718612+0000) 2019-10-13T11:47:05.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:05.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:05.724764+0000) 2019-10-13T11:47:05.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:05.723+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:05.724764+0000) 2019-10-13T11:47:08.630 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:08.628+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.629738+0000) 2019-10-13T11:47:08.744 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:08.742+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.743803+0000) 2019-10-13T11:47:08.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:08.742+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.743979+0000) 2019-10-13T11:47:08.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:08.743+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.744493+0000) 2019-10-13T11:47:08.746 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:08.743+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.744570+0000) 2019-10-13T11:47:08.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:08.743+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.744746+0000) 2019-10-13T11:47:08.746 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:08.743+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.744939+0000) 2019-10-13T11:47:08.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:08.744+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.745505+0000) 2019-10-13T11:47:08.747 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:08.746+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.747066+0000) 2019-10-13T11:47:08.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:08.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217151 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:08.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:08.960+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.961523+0000) 2019-10-13T11:47:11.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:11.581+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.581832+0000) 2019-10-13T11:47:13.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:13.066+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.067329+0000) 2019-10-13T11:47:13.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:13.070+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.070879+0000) 2019-10-13T11:47:13.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:13.753+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.754935+0000) 2019-10-13T11:47:13.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:13.754+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.755480+0000) 2019-10-13T11:47:13.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:13.754+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.755466+0000) 2019-10-13T11:47:13.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:13.754+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.755678+0000) 2019-10-13T11:47:13.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:13.754+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.755984+0000) 2019-10-13T11:47:13.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:13.754+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.756224+0000) 2019-10-13T11:47:13.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:13.754+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.756386+0000) 2019-10-13T11:47:13.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:13.757+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.758524+0000) 2019-10-13T11:47:13.764 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:13.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217181 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:15.721 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:15.720+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:15.720905+0000) 2019-10-13T11:47:15.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:15.723+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:15.724890+0000) 2019-10-13T11:47:15.726 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:15.723+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:15.724890+0000) 2019-10-13T11:47:18.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:18.629+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.629943+0000) 2019-10-13T11:47:18.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:18.764+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.765606+0000) 2019-10-13T11:47:18.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:18.764+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.766298+0000) 2019-10-13T11:47:18.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:18.764+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.766242+0000) 2019-10-13T11:47:18.768 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:18.765+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.766493+0000) 2019-10-13T11:47:18.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:18.765+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.766545+0000) 2019-10-13T11:47:18.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:18.765+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.767130+0000) 2019-10-13T11:47:18.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:18.765+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.767110+0000) 2019-10-13T11:47:18.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:18.766+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.768633+0000) 2019-10-13T11:47:18.775 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:18.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217213 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:18.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:18.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.961691+0000) 2019-10-13T11:47:21.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:21.581+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.581984+0000) 2019-10-13T11:47:23.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:23.066+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.067479+0000) 2019-10-13T11:47:23.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:23.070+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.071028+0000) 2019-10-13T11:47:23.777 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:23.775+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.776423+0000) 2019-10-13T11:47:23.777 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:23.775+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.777334+0000) 2019-10-13T11:47:23.778 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:23.775+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.777180+0000) 2019-10-13T11:47:23.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:23.775+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.777316+0000) 2019-10-13T11:47:23.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:23.776+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.777595+0000) 2019-10-13T11:47:23.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:23.776+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.777768+0000) 2019-10-13T11:47:23.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:23.776+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.777975+0000) 2019-10-13T11:47:23.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:23.777+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.779233+0000) 2019-10-13T11:47:23.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:23.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217241 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:25.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:25.722+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:25.723223+0000) 2019-10-13T11:47:25.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:25.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:25.725038+0000) 2019-10-13T11:47:25.726 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:25.724+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:25.725038+0000) 2019-10-13T11:47:28.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:28.629+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.630133+0000) 2019-10-13T11:47:28.788 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:28.787+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.787407+0000) 2019-10-13T11:47:28.789 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:28.787+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.787564+0000) 2019-10-13T11:47:28.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:28.787+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.787720+0000) 2019-10-13T11:47:28.790 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:28.786+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.788414+0000) 2019-10-13T11:47:28.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:28.788+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.788482+0000) 2019-10-13T11:47:28.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:28.788+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.788759+0000) 2019-10-13T11:47:28.790 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:28.788+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.788970+0000) 2019-10-13T11:47:28.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:28.788+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.790707+0000) 2019-10-13T11:47:28.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:28.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217273 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:28.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:28.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.961802+0000) 2019-10-13T11:47:31.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:31.581+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.582166+0000) 2019-10-13T11:47:32.908 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:47:32.909 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:32.913 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:47:32.913 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:33.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:33.067+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.067628+0000) 2019-10-13T11:47:33.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:33.070+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.071221+0000) 2019-10-13T11:47:33.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:33.797+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.798291+0000) 2019-10-13T11:47:33.799 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:33.798+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.798507+0000) 2019-10-13T11:47:33.799 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:33.798+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.798618+0000) 2019-10-13T11:47:33.800 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:33.798+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.798971+0000) 2019-10-13T11:47:33.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:33.798+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.799180+0000) 2019-10-13T11:47:33.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:33.799+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.799455+0000) 2019-10-13T11:47:33.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:33.799+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.800134+0000) 2019-10-13T11:47:33.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:33.802+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.802943+0000) 2019-10-13T11:47:33.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:33.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217303 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:35.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:35.724+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:35.725286+0000) 2019-10-13T11:47:35.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:35.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:35.725213+0000) 2019-10-13T11:47:35.726 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:35.724+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:35.725212+0000) 2019-10-13T11:47:38.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:38.630+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.630308+0000) 2019-10-13T11:47:38.810 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:38.808+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.808823+0000) 2019-10-13T11:47:38.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:38.808+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.808999+0000) 2019-10-13T11:47:38.810 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:38.808+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.809180+0000) 2019-10-13T11:47:38.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:38.809+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.809484+0000) 2019-10-13T11:47:38.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:38.809+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.810620+0000) 2019-10-13T11:47:38.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:38.809+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.809678+0000) 2019-10-13T11:47:38.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:38.809+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.810104+0000) 2019-10-13T11:47:38.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:38.811+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.811797+0000) 2019-10-13T11:47:38.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:38.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217335 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:38.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:38.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.961990+0000) 2019-10-13T11:47:41.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:41.581+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.582374+0000) 2019-10-13T11:47:43.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:43.067+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.067808+0000) 2019-10-13T11:47:43.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:43.070+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.071404+0000) 2019-10-13T11:47:43.820 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:43.818+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.819310+0000) 2019-10-13T11:47:43.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:43.818+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.819190+0000) 2019-10-13T11:47:43.821 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:43.819+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.819540+0000) 2019-10-13T11:47:43.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:43.819+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.820201+0000) 2019-10-13T11:47:43.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:43.820+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.820416+0000) 2019-10-13T11:47:43.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:43.820+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.820908+0000) 2019-10-13T11:47:43.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:43.820+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.821710+0000) 2019-10-13T11:47:43.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:43.822+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.823638+0000) 2019-10-13T11:47:43.829 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:43.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217363 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:45.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:45.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:45.725434+0000) 2019-10-13T11:47:45.726 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:45.724+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:45.725434+0000) 2019-10-13T11:47:45.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:45.727+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:45.727627+0000) 2019-10-13T11:47:48.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:48.629+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.630498+0000) 2019-10-13T11:47:48.832 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:48.829+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.830866+0000) 2019-10-13T11:47:48.832 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:48.829+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.830924+0000) 2019-10-13T11:47:48.832 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:48.829+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.831282+0000) 2019-10-13T11:47:48.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:48.829+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.831130+0000) 2019-10-13T11:47:48.833 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:48.831+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.832279+0000) 2019-10-13T11:47:48.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:48.830+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.831702+0000) 2019-10-13T11:47:48.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:48.830+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.831790+0000) 2019-10-13T11:47:48.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:48.833+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.833888+0000) 2019-10-13T11:47:48.840 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:48.837+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217395 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:48.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:48.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.962167+0000) 2019-10-13T11:47:51.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:51.581+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.582557+0000) 2019-10-13T11:47:53.068 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:53.067+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.067958+0000) 2019-10-13T11:47:53.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:53.071+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.071606+0000) 2019-10-13T11:47:53.841 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:53.839+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.840568+0000) 2019-10-13T11:47:53.842 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:53.839+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.840989+0000) 2019-10-13T11:47:53.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:53.839+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.841173+0000) 2019-10-13T11:47:53.842 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:53.839+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.841275+0000) 2019-10-13T11:47:53.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:53.840+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.841401+0000) 2019-10-13T11:47:53.843 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:53.840+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.841663+0000) 2019-10-13T11:47:53.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:53.841+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.841944+0000) 2019-10-13T11:47:53.844 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:53.842+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.843441+0000) 2019-10-13T11:47:53.848 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:53.846+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:55.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:55.724+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:55.725653+0000) 2019-10-13T11:47:55.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:55.724+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:55.725653+0000) 2019-10-13T11:47:55.730 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:55.729+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:55.730026+0000) 2019-10-13T11:47:58.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:58.629+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.630696+0000) 2019-10-13T11:47:58.849 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:47:58.847+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.848887+0000) 2019-10-13T11:47:58.850 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:47:58.847+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.849342+0000) 2019-10-13T11:47:58.851 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:47:58.848+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.849531+0000) 2019-10-13T11:47:58.851 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:58.848+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.849667+0000) 2019-10-13T11:47:58.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:47:58.848+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.849888+0000) 2019-10-13T11:47:58.851 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:47:58.848+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.850023+0000) 2019-10-13T11:47:58.852 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:47:58.850+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.850781+0000) 2019-10-13T11:47:58.856 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:47:58.852+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.852785+0000) 2019-10-13T11:47:58.858 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:47:58.856+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217457 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:47:58.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:47:58.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.962364+0000) 2019-10-13T11:48:01.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:01.582+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.582704+0000) 2019-10-13T11:48:03.001 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:48:03.001 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:03.005 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:48:03.005 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:03.069 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:03.066+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.068140+0000) 2019-10-13T11:48:03.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:03.071+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.071814+0000) 2019-10-13T11:48:03.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:03.858+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.859904+0000) 2019-10-13T11:48:03.861 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:03.858+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.860085+0000) 2019-10-13T11:48:03.862 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:03.859+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.860444+0000) 2019-10-13T11:48:03.862 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:03.859+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.860668+0000) 2019-10-13T11:48:03.862 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:03.859+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.860866+0000) 2019-10-13T11:48:03.863 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:03.860+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.860941+0000) 2019-10-13T11:48:03.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:03.859+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.861016+0000) 2019-10-13T11:48:03.863 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:03.862+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.862824+0000) 2019-10-13T11:48:03.869 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:03.867+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:05.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:05.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:05.725869+0000) 2019-10-13T11:48:05.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:05.725+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:05.725869+0000) 2019-10-13T11:48:05.732 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:05.732+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:05.732308+0000) 2019-10-13T11:48:08.631 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:08.630+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.630862+0000) 2019-10-13T11:48:08.870 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:08.868+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.869800+0000) 2019-10-13T11:48:08.871 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:08.868+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.870281+0000) 2019-10-13T11:48:08.871 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:08.869+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.870492+0000) 2019-10-13T11:48:08.872 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:08.868+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.870297+0000) 2019-10-13T11:48:08.872 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:08.869+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.870646+0000) 2019-10-13T11:48:08.872 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:08.869+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.870992+0000) 2019-10-13T11:48:08.872 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:08.869+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.871145+0000) 2019-10-13T11:48:08.873 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:08.871+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.873547+0000) 2019-10-13T11:48:08.878 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:08.876+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217517 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:08.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:08.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.962528+0000) 2019-10-13T11:48:11.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:11.582+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.582872+0000) 2019-10-13T11:48:13.069 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:13.067+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.068324+0000) 2019-10-13T11:48:13.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:13.071+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.072005+0000) 2019-10-13T11:48:13.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:13.878+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.878894+0000) 2019-10-13T11:48:13.880 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:13.878+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.879365+0000) 2019-10-13T11:48:13.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:13.879+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.879559+0000) 2019-10-13T11:48:13.881 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:13.878+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.879991+0000) 2019-10-13T11:48:13.881 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:13.879+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.879798+0000) 2019-10-13T11:48:13.881 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:13.879+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.880345+0000) 2019-10-13T11:48:13.881 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:13.879+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.880388+0000) 2019-10-13T11:48:13.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:13.880+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.882330+0000) 2019-10-13T11:48:13.888 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:13.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217547 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:15.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:15.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:15.726079+0000) 2019-10-13T11:48:15.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:15.725+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:15.726079+0000) 2019-10-13T11:48:15.735 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:15.733+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:15.734831+0000) 2019-10-13T11:48:18.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:18.630+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.631048+0000) 2019-10-13T11:48:18.899 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:18.889+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.889605+0000) 2019-10-13T11:48:18.900 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:18.888+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.890222+0000) 2019-10-13T11:48:18.900 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:18.889+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.890098+0000) 2019-10-13T11:48:18.900 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:18.889+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.890350+0000) 2019-10-13T11:48:18.900 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:18.890+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.890590+0000) 2019-10-13T11:48:18.901 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:18.890+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.890691+0000) 2019-10-13T11:48:18.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:18.890+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.890909+0000) 2019-10-13T11:48:18.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:18.892+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.894375+0000) 2019-10-13T11:48:18.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:18.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217580 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:18.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:18.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.962629+0000) 2019-10-13T11:48:21.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:21.582+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.583044+0000) 2019-10-13T11:48:23.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:23.068+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.068515+0000) 2019-10-13T11:48:23.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:23.071+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.072194+0000) 2019-10-13T11:48:23.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:23.900+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.900777+0000) 2019-10-13T11:48:23.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:23.900+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.901051+0000) 2019-10-13T11:48:23.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:23.900+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.901244+0000) 2019-10-13T11:48:23.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:23.901+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.902359+0000) 2019-10-13T11:48:23.903 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:23.901+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.901519+0000) 2019-10-13T11:48:23.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:23.901+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.901684+0000) 2019-10-13T11:48:23.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:23.901+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.902189+0000) 2019-10-13T11:48:23.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:23.903+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.903986+0000) 2019-10-13T11:48:23.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:23.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:25.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:25.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:25.726231+0000) 2019-10-13T11:48:25.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:25.726+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:25.726231+0000) 2019-10-13T11:48:25.737 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:25.736+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:25.737306+0000) 2019-10-13T11:48:28.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:28.631+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.631243+0000) 2019-10-13T11:48:28.912 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:28.911+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.911821+0000) 2019-10-13T11:48:28.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:28.911+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.912350+0000) 2019-10-13T11:48:28.914 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:28.912+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.912487+0000) 2019-10-13T11:48:28.914 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:28.912+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.912610+0000) 2019-10-13T11:48:28.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:28.912+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.912945+0000) 2019-10-13T11:48:28.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:28.912+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.913585+0000) 2019-10-13T11:48:28.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:28.913+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.913515+0000) 2019-10-13T11:48:28.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:28.913+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.914660+0000) 2019-10-13T11:48:28.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:28.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217641 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:28.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:28.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.962759+0000) 2019-10-13T11:48:31.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:31.582+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.583234+0000) 2019-10-13T11:48:33.069 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:33.067+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.068702+0000) 2019-10-13T11:48:33.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:33.071+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.072375+0000) 2019-10-13T11:48:33.175 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:48:33.175 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:33.179 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:48:33.180 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:33.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:33.921+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.923114+0000) 2019-10-13T11:48:33.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:33.921+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.923186+0000) 2019-10-13T11:48:33.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:33.923+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.924433+0000) 2019-10-13T11:48:33.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:33.922+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.923754+0000) 2019-10-13T11:48:33.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:33.922+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.923948+0000) 2019-10-13T11:48:33.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:33.922+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.924324+0000) 2019-10-13T11:48:33.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:33.923+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.924719+0000) 2019-10-13T11:48:33.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:33.925+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.926655+0000) 2019-10-13T11:48:33.932 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:33.930+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:35.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:35.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:35.726427+0000) 2019-10-13T11:48:35.727 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:35.725+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:35.726427+0000) 2019-10-13T11:48:35.740 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:35.739+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:35.739596+0000) 2019-10-13T11:48:38.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:38.630+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.631433+0000) 2019-10-13T11:48:38.934 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:38.933+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.934017+0000) 2019-10-13T11:48:38.935 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:38.932+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.934097+0000) 2019-10-13T11:48:38.935 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:38.932+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.934253+0000) 2019-10-13T11:48:38.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:38.932+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.934125+0000) 2019-10-13T11:48:38.936 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:38.933+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.934395+0000) 2019-10-13T11:48:38.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:38.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.934507+0000) 2019-10-13T11:48:38.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:38.933+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.934915+0000) 2019-10-13T11:48:38.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:38.936+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.937266+0000) 2019-10-13T11:48:38.943 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:38.941+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:38.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:38.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.962863+0000) 2019-10-13T11:48:41.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:41.582+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.583458+0000) 2019-10-13T11:48:43.069 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:43.068+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.068887+0000) 2019-10-13T11:48:43.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:43.072+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.072561+0000) 2019-10-13T11:48:43.945 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:43.944+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.945253+0000) 2019-10-13T11:48:43.946 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:43.943+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.945218+0000) 2019-10-13T11:48:43.947 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:43.943+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.945294+0000) 2019-10-13T11:48:43.947 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:43.944+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.945621+0000) 2019-10-13T11:48:43.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:43.944+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.945861+0000) 2019-10-13T11:48:43.947 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:43.944+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.945967+0000) 2019-10-13T11:48:43.948 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:43.946+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.947259+0000) 2019-10-13T11:48:43.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:43.945+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.946462+0000) 2019-10-13T11:48:43.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:43.952+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217732 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:45.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:45.725+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:45.726600+0000) 2019-10-13T11:48:45.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:45.725+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:45.726600+0000) 2019-10-13T11:48:45.742 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:45.741+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:45.741898+0000) 2019-10-13T11:48:48.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:48.630+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.631602+0000) 2019-10-13T11:48:48.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:48.954+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.955655+0000) 2019-10-13T11:48:48.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:48.954+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.955901+0000) 2019-10-13T11:48:48.957 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:48.954+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.955984+0000) 2019-10-13T11:48:48.957 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:48.954+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.956107+0000) 2019-10-13T11:48:48.958 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:48.954+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.956294+0000) 2019-10-13T11:48:48.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:48.956+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.956783+0000) 2019-10-13T11:48:48.959 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:48.957+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.957916+0000) 2019-10-13T11:48:48.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:48.955+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.956955+0000) 2019-10-13T11:48:48.963 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:48.961+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.963013+0000) 2019-10-13T11:48:48.966 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:48.963+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217765 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:51.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:51.583+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.583644+0000) 2019-10-13T11:48:53.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:53.068+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.069067+0000) 2019-10-13T11:48:53.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:53.072+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.072762+0000) 2019-10-13T11:48:53.968 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:53.965+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.966876+0000) 2019-10-13T11:48:53.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:53.965+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.967031+0000) 2019-10-13T11:48:53.968 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:53.965+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.967063+0000) 2019-10-13T11:48:53.969 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:53.965+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.967235+0000) 2019-10-13T11:48:53.969 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:53.966+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.967421+0000) 2019-10-13T11:48:53.969 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:53.966+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.967711+0000) 2019-10-13T11:48:53.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:53.966+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.967861+0000) 2019-10-13T11:48:53.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:53.969+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.970350+0000) 2019-10-13T11:48:53.976 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:53.974+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217795 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:48:55.727 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:55.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:55.726779+0000) 2019-10-13T11:48:55.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:55.726+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:55.726779+0000) 2019-10-13T11:48:55.744 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:55.743+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:55.744261+0000) 2019-10-13T11:48:58.632 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:58.631+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.631775+0000) 2019-10-13T11:48:58.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:58.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.963187+0000) 2019-10-13T11:48:58.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:48:58.977+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.977424+0000) 2019-10-13T11:48:58.979 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:48:58.977+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.977547+0000) 2019-10-13T11:48:58.979 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:48:58.977+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.977828+0000) 2019-10-13T11:48:58.979 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:48:58.977+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.977736+0000) 2019-10-13T11:48:58.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:48:58.977+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.978167+0000) 2019-10-13T11:48:58.980 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:48:58.978+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.978736+0000) 2019-10-13T11:48:58.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:48:58.978+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.979319+0000) 2019-10-13T11:48:58.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:48:58.980+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.981524+0000) 2019-10-13T11:48:58.987 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:48:58.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:01.584 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:01.583+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.583826+0000) 2019-10-13T11:49:03.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:03.068+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.069280+0000) 2019-10-13T11:49:03.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:03.071+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.072931+0000) 2019-10-13T11:49:03.321 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:49:03.321 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:03.325 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:49:03.325 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:03.989 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:03.988+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.988866+0000) 2019-10-13T11:49:03.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:03.988+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.989244+0000) 2019-10-13T11:49:03.990 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:03.988+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.989386+0000) 2019-10-13T11:49:03.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:03.989+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.989666+0000) 2019-10-13T11:49:03.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:03.989+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.989574+0000) 2019-10-13T11:49:03.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:03.989+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.989987+0000) 2019-10-13T11:49:03.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:03.988+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.990201+0000) 2019-10-13T11:49:03.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:03.989+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.991380+0000) 2019-10-13T11:49:03.998 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:03.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217856 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:05.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:05.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:05.727020+0000) 2019-10-13T11:49:05.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:05.726+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:05.727020+0000) 2019-10-13T11:49:05.747 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:05.745+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:05.746667+0000) 2019-10-13T11:49:08.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:08.631+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.631986+0000) 2019-10-13T11:49:08.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:08.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.963372+0000) 2019-10-13T11:49:09.000 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:08.998+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.999062+0000) 2019-10-13T11:49:09.000 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:08.998+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.999188+0000) 2019-10-13T11:49:09.000 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:08.998+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.999323+0000) 2019-10-13T11:49:09.000 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:08.999+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:09.000213+0000) 2019-10-13T11:49:09.001 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:08.998+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:09.000648+0000) 2019-10-13T11:49:09.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:09.000+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:09.000507+0000) 2019-10-13T11:49:09.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:09.000+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:09.000903+0000) 2019-10-13T11:49:09.003 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:09.001+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:09.003045+0000) 2019-10-13T11:49:09.009 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:09.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217889 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:11.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:11.583+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.584067+0000) 2019-10-13T11:49:13.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:13.069+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:13.069456+0000) 2019-10-13T11:49:13.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:13.072+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:13.073113+0000) 2019-10-13T11:49:14.011 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:14.010+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.010576+0000) 2019-10-13T11:49:14.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:14.010+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.011258+0000) 2019-10-13T11:49:14.012 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:14.010+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.011131+0000) 2019-10-13T11:49:14.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:14.011+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.011480+0000) 2019-10-13T11:49:14.013 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:14.011+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.011772+0000) 2019-10-13T11:49:14.013 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:14.011+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.011906+0000) 2019-10-13T11:49:14.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:14.012+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.013202+0000) 2019-10-13T11:49:14.014 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:14.011+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.012281+0000) 2019-10-13T11:49:14.020 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:14.018+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:15.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:15.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:15.727256+0000) 2019-10-13T11:49:15.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:15.726+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:15.727256+0000) 2019-10-13T11:49:15.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:15.748+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:15.749179+0000) 2019-10-13T11:49:18.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:18.631+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.632208+0000) 2019-10-13T11:49:18.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:18.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.963510+0000) 2019-10-13T11:49:19.021 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:19.019+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.020595+0000) 2019-10-13T11:49:19.022 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:19.019+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.020860+0000) 2019-10-13T11:49:19.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:19.019+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.020941+0000) 2019-10-13T11:49:19.023 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:19.020+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.021161+0000) 2019-10-13T11:49:19.023 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:19.019+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.021222+0000) 2019-10-13T11:49:19.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:19.019+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.021314+0000) 2019-10-13T11:49:19.023 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:19.020+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.021599+0000) 2019-10-13T11:49:19.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:19.023+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:19.024045+0000) 2019-10-13T11:49:19.028 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:19.026+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217952 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:21.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:21.583+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.584215+0000) 2019-10-13T11:49:23.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:23.068+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:23.069640+0000) 2019-10-13T11:49:23.073 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:23.072+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:23.073294+0000) 2019-10-13T11:49:24.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:24.027+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.029119+0000) 2019-10-13T11:49:24.031 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:24.029+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.029762+0000) 2019-10-13T11:49:24.031 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:24.028+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.029512+0000) 2019-10-13T11:49:24.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:24.028+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.029644+0000) 2019-10-13T11:49:24.031 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:24.028+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.029835+0000) 2019-10-13T11:49:24.031 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:24.028+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.029919+0000) 2019-10-13T11:49:24.032 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:24.028+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.030215+0000) 2019-10-13T11:49:24.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:24.031+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.032468+0000) 2019-10-13T11:49:24.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:24.036+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 217980 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:25.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:25.726+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:25.727471+0000) 2019-10-13T11:49:25.728 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:25.726+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:25.727471+0000) 2019-10-13T11:49:25.752 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:25.751+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:25.751749+0000) 2019-10-13T11:49:28.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:28.631+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.632356+0000) 2019-10-13T11:49:28.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:28.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.963632+0000) 2019-10-13T11:49:29.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:29.038+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.039610+0000) 2019-10-13T11:49:29.041 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:29.038+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.039426+0000) 2019-10-13T11:49:29.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:29.038+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.039797+0000) 2019-10-13T11:49:29.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:29.038+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.039931+0000) 2019-10-13T11:49:29.042 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:29.038+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.040083+0000) 2019-10-13T11:49:29.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:29.039+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.040430+0000) 2019-10-13T11:49:29.042 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:29.039+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.040485+0000) 2019-10-13T11:49:29.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:29.041+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:29.041985+0000) 2019-10-13T11:49:29.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:29.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218013 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:31.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:31.583+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.584346+0000) 2019-10-13T11:49:33.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:33.069+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:33.069817+0000) 2019-10-13T11:49:33.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:33.073+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:33.073477+0000) 2019-10-13T11:49:33.443 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:49:33.443 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:33.447 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:49:33.448 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:34.050 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:34.047+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.049282+0000) 2019-10-13T11:49:34.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:34.048+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.049580+0000) 2019-10-13T11:49:34.051 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:34.048+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.049548+0000) 2019-10-13T11:49:34.051 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:34.048+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.049897+0000) 2019-10-13T11:49:34.051 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:34.048+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.050061+0000) 2019-10-13T11:49:34.052 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:34.049+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.050297+0000) 2019-10-13T11:49:34.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:34.049+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.050507+0000) 2019-10-13T11:49:34.053 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:34.051+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.052687+0000) 2019-10-13T11:49:34.058 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:34.055+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218043 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:35.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:35.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:35.727670+0000) 2019-10-13T11:49:35.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:35.727+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:35.727670+0000) 2019-10-13T11:49:35.754 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:35.753+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:35.754019+0000) 2019-10-13T11:49:38.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:38.631+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.632517+0000) 2019-10-13T11:49:38.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:38.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.963746+0000) 2019-10-13T11:49:39.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:39.057+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.058814+0000) 2019-10-13T11:49:39.060 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:39.057+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.059094+0000) 2019-10-13T11:49:39.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:39.057+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.059384+0000) 2019-10-13T11:49:39.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:39.058+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.059559+0000) 2019-10-13T11:49:39.061 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:39.058+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.059741+0000) 2019-10-13T11:49:39.061 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:39.058+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.059857+0000) 2019-10-13T11:49:39.062 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:39.059+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.060633+0000) 2019-10-13T11:49:39.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:39.060+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:39.061182+0000) 2019-10-13T11:49:39.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:39.066+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218076 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:41.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:41.584+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.584536+0000) 2019-10-13T11:49:43.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:43.069+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:43.069996+0000) 2019-10-13T11:49:43.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:43.073+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:43.073657+0000) 2019-10-13T11:49:44.070 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:44.069+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.069492+0000) 2019-10-13T11:49:44.071 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:44.069+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.069653+0000) 2019-10-13T11:49:44.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:44.069+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.069795+0000) 2019-10-13T11:49:44.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:44.069+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.069932+0000) 2019-10-13T11:49:44.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:44.069+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.070607+0000) 2019-10-13T11:49:44.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:44.070+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.071652+0000) 2019-10-13T11:49:44.072 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:44.070+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.070659+0000) 2019-10-13T11:49:44.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:44.070+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.070786+0000) 2019-10-13T11:49:44.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:44.077+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:45.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:45.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:45.727853+0000) 2019-10-13T11:49:45.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:45.727+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:45.727853+0000) 2019-10-13T11:49:45.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:45.756+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:45.756486+0000) 2019-10-13T11:49:48.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:48.632+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.632673+0000) 2019-10-13T11:49:48.964 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:48.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.963917+0000) 2019-10-13T11:49:49.081 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:49.079+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.080238+0000) 2019-10-13T11:49:49.081 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:49.080+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.080572+0000) 2019-10-13T11:49:49.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:49.080+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.081498+0000) 2019-10-13T11:49:49.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:49.080+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.080744+0000) 2019-10-13T11:49:49.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:49.080+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.080948+0000) 2019-10-13T11:49:49.083 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:49.080+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.081092+0000) 2019-10-13T11:49:49.083 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:49.081+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.081482+0000) 2019-10-13T11:49:49.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:49.081+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:49.082404+0000) 2019-10-13T11:49:49.089 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:49.088+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218137 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:51.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:51.584+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.584741+0000) 2019-10-13T11:49:53.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:53.069+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.070231+0000) 2019-10-13T11:49:53.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:53.072+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.073855+0000) 2019-10-13T11:49:54.092 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:54.090+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.090701+0000) 2019-10-13T11:49:54.092 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:54.090+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.090839+0000) 2019-10-13T11:49:54.092 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:54.090+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.091161+0000) 2019-10-13T11:49:54.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:54.090+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.092064+0000) 2019-10-13T11:49:54.093 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:54.090+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.091340+0000) 2019-10-13T11:49:54.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:54.091+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.091602+0000) 2019-10-13T11:49:54.093 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:54.091+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.091868+0000) 2019-10-13T11:49:54.094 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:54.091+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.092950+0000) 2019-10-13T11:49:54.100 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:54.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:49:55.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:55.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:55.728080+0000) 2019-10-13T11:49:55.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:55.727+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:55.728080+0000) 2019-10-13T11:49:55.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:55.757+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:55.758789+0000) 2019-10-13T11:49:58.633 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:58.632+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.632893+0000) 2019-10-13T11:49:58.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:58.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.964050+0000) 2019-10-13T11:49:59.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:49:59.100+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.100835+0000) 2019-10-13T11:49:59.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:49:59.100+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.100981+0000) 2019-10-13T11:49:59.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:49:59.101+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.102742+0000) 2019-10-13T11:49:59.105 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:49:59.101+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.101432+0000) 2019-10-13T11:49:59.105 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:49:59.101+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.101736+0000) 2019-10-13T11:49:59.105 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:49:59.101+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.101918+0000) 2019-10-13T11:49:59.105 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:49:59.102+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.102615+0000) 2019-10-13T11:49:59.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:49:59.102+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:59.104055+0000) 2019-10-13T11:49:59.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:49:59.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:01.585 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:01.583+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.584902+0000) 2019-10-13T11:50:03.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:03.069+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.070422+0000) 2019-10-13T11:50:03.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:03.073+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.074082+0000) 2019-10-13T11:50:03.533 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:50:03.533 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:03.537 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:50:03.537 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:04.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:04.110+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.112193+0000) 2019-10-13T11:50:04.113 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:04.111+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.112470+0000) 2019-10-13T11:50:04.113 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:04.111+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.112628+0000) 2019-10-13T11:50:04.114 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:04.111+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.112892+0000) 2019-10-13T11:50:04.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:04.111+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.113288+0000) 2019-10-13T11:50:04.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:04.113+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.114498+0000) 2019-10-13T11:50:04.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:04.113+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.114447+0000) 2019-10-13T11:50:04.116 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:04.115+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.115815+0000) 2019-10-13T11:50:04.122 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:04.120+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218228 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:05.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:05.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:05.728279+0000) 2019-10-13T11:50:05.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:05.727+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:05.728279+0000) 2019-10-13T11:50:05.762 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:05.761+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:05.761972+0000) 2019-10-13T11:50:08.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:08.632+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.633089+0000) 2019-10-13T11:50:08.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:08.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.964177+0000) 2019-10-13T11:50:09.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:09.122+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.123437+0000) 2019-10-13T11:50:09.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:09.122+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.123635+0000) 2019-10-13T11:50:09.125 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:09.122+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.123848+0000) 2019-10-13T11:50:09.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:09.122+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.124060+0000) 2019-10-13T11:50:09.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:09.123+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.124681+0000) 2019-10-13T11:50:09.126 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:09.122+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.124213+0000) 2019-10-13T11:50:09.126 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:09.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.124512+0000) 2019-10-13T11:50:09.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:09.125+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:09.126565+0000) 2019-10-13T11:50:09.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:09.131+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218261 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:11.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:11.583+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:11.585115+0000) 2019-10-13T11:50:13.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:13.069+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.070612+0000) 2019-10-13T11:50:13.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:13.073+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.074302+0000) 2019-10-13T11:50:14.136 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:14.133+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.134742+0000) 2019-10-13T11:50:14.136 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:14.133+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.134571+0000) 2019-10-13T11:50:14.136 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:14.133+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.134818+0000) 2019-10-13T11:50:14.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:14.133+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.134930+0000) 2019-10-13T11:50:14.137 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:14.133+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.135372+0000) 2019-10-13T11:50:14.137 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:14.134+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.135411+0000) 2019-10-13T11:50:14.137 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:14.135+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.135976+0000) 2019-10-13T11:50:14.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:14.137+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.138304+0000) 2019-10-13T11:50:14.144 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:14.142+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218291 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:15.729 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:15.727+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:15.728462+0000) 2019-10-13T11:50:15.729 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:15.727+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:15.728461+0000) 2019-10-13T11:50:15.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:15.764+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:15.764995+0000) 2019-10-13T11:50:18.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:18.632+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.633243+0000) 2019-10-13T11:50:18.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:18.962+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.964273+0000) 2019-10-13T11:50:19.146 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:19.143+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.145318+0000) 2019-10-13T11:50:19.147 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:19.145+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.146239+0000) 2019-10-13T11:50:19.147 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:19.144+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.146006+0000) 2019-10-13T11:50:19.147 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:19.144+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.146196+0000) 2019-10-13T11:50:19.148 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:19.144+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.146255+0000) 2019-10-13T11:50:19.148 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:19.145+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.146457+0000) 2019-10-13T11:50:19.148 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:19.145+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.146702+0000) 2019-10-13T11:50:19.148 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:19.147+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:19.148112+0000) 2019-10-13T11:50:19.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:19.153+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218324 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:21.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:21.584+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:21.585309+0000) 2019-10-13T11:50:23.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:23.070+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.070775+0000) 2019-10-13T11:50:23.074 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:23.074+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.074460+0000) 2019-10-13T11:50:24.157 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:24.155+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.156578+0000) 2019-10-13T11:50:24.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:24.155+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.156744+0000) 2019-10-13T11:50:24.158 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:24.156+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.157278+0000) 2019-10-13T11:50:24.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:24.155+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.157380+0000) 2019-10-13T11:50:24.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:24.156+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.157578+0000) 2019-10-13T11:50:24.159 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:24.156+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.157663+0000) 2019-10-13T11:50:24.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:24.156+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.157799+0000) 2019-10-13T11:50:24.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:24.158+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.159721+0000) 2019-10-13T11:50:24.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:24.164+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218352 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:25.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:25.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:25.728683+0000) 2019-10-13T11:50:25.730 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:25.728+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:25.728683+0000) 2019-10-13T11:50:25.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:25.766+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:25.767315+0000) 2019-10-13T11:50:28.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:28.633+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.633436+0000) 2019-10-13T11:50:28.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:28.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.964440+0000) 2019-10-13T11:50:29.168 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:29.166+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.167305+0000) 2019-10-13T11:50:29.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:29.167+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.167473+0000) 2019-10-13T11:50:29.169 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:29.167+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.167864+0000) 2019-10-13T11:50:29.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:29.167+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.167842+0000) 2019-10-13T11:50:29.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:29.167+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.168179+0000) 2019-10-13T11:50:29.170 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:29.167+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.168359+0000) 2019-10-13T11:50:29.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:29.168+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.168601+0000) 2019-10-13T11:50:29.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:29.169+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:29.170501+0000) 2019-10-13T11:50:29.176 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:29.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218385 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:31.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:31.585+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:31.585498+0000) 2019-10-13T11:50:33.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:33.070+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.070944+0000) 2019-10-13T11:50:33.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:33.074+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.074658+0000) 2019-10-13T11:50:33.631 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:50:33.631 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:33.635 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:50:33.635 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:34.179 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:34.177+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.177526+0000) 2019-10-13T11:50:34.179 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:34.177+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.177601+0000) 2019-10-13T11:50:34.179 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:34.177+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.177966+0000) 2019-10-13T11:50:34.179 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:34.177+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.178188+0000) 2019-10-13T11:50:34.179 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:34.177+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.178320+0000) 2019-10-13T11:50:34.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:34.177+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.178144+0000) 2019-10-13T11:50:34.180 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:34.178+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.178772+0000) 2019-10-13T11:50:34.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:34.179+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.180447+0000) 2019-10-13T11:50:34.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:34.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218415 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:35.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:35.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:35.728898+0000) 2019-10-13T11:50:35.730 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:35.728+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:35.728897+0000) 2019-10-13T11:50:35.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:35.768+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:35.769575+0000) 2019-10-13T11:50:38.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:38.633+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.633603+0000) 2019-10-13T11:50:38.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:38.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.964586+0000) 2019-10-13T11:50:39.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:39.187+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.188061+0000) 2019-10-13T11:50:39.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:39.187+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.188629+0000) 2019-10-13T11:50:39.190 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:39.188+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.188670+0000) 2019-10-13T11:50:39.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:39.188+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.188782+0000) 2019-10-13T11:50:39.190 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:39.188+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.188985+0000) 2019-10-13T11:50:39.190 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:39.188+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.188914+0000) 2019-10-13T11:50:39.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:39.188+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.189197+0000) 2019-10-13T11:50:39.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:39.191+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:39.191833+0000) 2019-10-13T11:50:39.197 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:39.195+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218448 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:41.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:41.585+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:41.585691+0000) 2019-10-13T11:50:43.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:43.071+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.071129+0000) 2019-10-13T11:50:43.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:43.073+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.074841+0000) 2019-10-13T11:50:44.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:44.197+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.197925+0000) 2019-10-13T11:50:44.199 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:44.197+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.198104+0000) 2019-10-13T11:50:44.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:44.196+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.198590+0000) 2019-10-13T11:50:44.200 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:44.198+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.198566+0000) 2019-10-13T11:50:44.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:44.198+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.198796+0000) 2019-10-13T11:50:44.200 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:44.198+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.199011+0000) 2019-10-13T11:50:44.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:44.198+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.199277+0000) 2019-10-13T11:50:44.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:44.199+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.200773+0000) 2019-10-13T11:50:44.207 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:44.206+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218476 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:45.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:45.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:45.729110+0000) 2019-10-13T11:50:45.730 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:45.728+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:45.729110+0000) 2019-10-13T11:50:45.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:45.770+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:45.771980+0000) 2019-10-13T11:50:48.634 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:48.632+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.633767+0000) 2019-10-13T11:50:48.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:48.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.964708+0000) 2019-10-13T11:50:49.209 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:49.207+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.208796+0000) 2019-10-13T11:50:49.211 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:49.208+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.209808+0000) 2019-10-13T11:50:49.211 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:49.207+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.209037+0000) 2019-10-13T11:50:49.211 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:49.208+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.209790+0000) 2019-10-13T11:50:49.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:49.208+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.209953+0000) 2019-10-13T11:50:49.212 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:49.208+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.210016+0000) 2019-10-13T11:50:49.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:49.208+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.210217+0000) 2019-10-13T11:50:49.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:49.209+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:49.211718+0000) 2019-10-13T11:50:49.219 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:49.216+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218509 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:51.586 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:51.584+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:51.585850+0000) 2019-10-13T11:50:53.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:53.070+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.071294+0000) 2019-10-13T11:50:53.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:53.074+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.075015+0000) 2019-10-13T11:50:54.221 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:54.218+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.220129+0000) 2019-10-13T11:50:54.221 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:54.218+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.220228+0000) 2019-10-13T11:50:54.222 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:54.219+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.220422+0000) 2019-10-13T11:50:54.222 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:54.219+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.220803+0000) 2019-10-13T11:50:54.222 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:54.219+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.220875+0000) 2019-10-13T11:50:54.223 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:54.219+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.221039+0000) 2019-10-13T11:50:54.223 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:54.220+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.221610+0000) 2019-10-13T11:50:54.224 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:54.221+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.223632+0000) 2019-10-13T11:50:54.229 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:54.227+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218539 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:50:55.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:55.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:55.729340+0000) 2019-10-13T11:50:55.730 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:55.728+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:55.729340+0000) 2019-10-13T11:50:55.774 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:55.773+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:55.774327+0000) 2019-10-13T11:50:58.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:58.633+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.633940+0000) 2019-10-13T11:50:58.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:58.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.964882+0000) 2019-10-13T11:50:59.232 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:50:59.229+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.230828+0000) 2019-10-13T11:50:59.232 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:50:59.229+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.230955+0000) 2019-10-13T11:50:59.232 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:50:59.229+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.231164+0000) 2019-10-13T11:50:59.233 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:50:59.230+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.231580+0000) 2019-10-13T11:50:59.233 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:50:59.230+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.231639+0000) 2019-10-13T11:50:59.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:50:59.230+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.231823+0000) 2019-10-13T11:50:59.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:50:59.230+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.231982+0000) 2019-10-13T11:50:59.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:50:59.233+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:59.233827+0000) 2019-10-13T11:50:59.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:50:59.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:01.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:01.585+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:01.586057+0000) 2019-10-13T11:51:03.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:03.070+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.071466+0000) 2019-10-13T11:51:03.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:03.074+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.075247+0000) 2019-10-13T11:51:03.822 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:51:03.823 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:03.826 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:51:03.827 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:04.242 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:04.240+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.241482+0000) 2019-10-13T11:51:04.243 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:04.240+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.241615+0000) 2019-10-13T11:51:04.243 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:04.240+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.241888+0000) 2019-10-13T11:51:04.244 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:04.241+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.242373+0000) 2019-10-13T11:51:04.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:04.240+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.242221+0000) 2019-10-13T11:51:04.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:04.240+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.242213+0000) 2019-10-13T11:51:04.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:04.241+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.242564+0000) 2019-10-13T11:51:04.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:04.244+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.245619+0000) 2019-10-13T11:51:04.249 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:04.247+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218601 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:05.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:05.728+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:05.729586+0000) 2019-10-13T11:51:05.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:05.728+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:05.729586+0000) 2019-10-13T11:51:05.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:05.776+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:05.776726+0000) 2019-10-13T11:51:08.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:08.632+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.634181+0000) 2019-10-13T11:51:08.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:08.963+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.965072+0000) 2019-10-13T11:51:09.251 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:09.249+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.250718+0000) 2019-10-13T11:51:09.251 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:09.249+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.250837+0000) 2019-10-13T11:51:09.252 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:09.250+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.251632+0000) 2019-10-13T11:51:09.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:09.249+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.251153+0000) 2019-10-13T11:51:09.253 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:09.250+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.251532+0000) 2019-10-13T11:51:09.253 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:09.250+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.251752+0000) 2019-10-13T11:51:09.253 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:09.250+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.251994+0000) 2019-10-13T11:51:09.254 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:09.253+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:09.253893+0000) 2019-10-13T11:51:09.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:09.258+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:11.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:11.585+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:11.586255+0000) 2019-10-13T11:51:13.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:13.071+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.071624+0000) 2019-10-13T11:51:13.075 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:13.075+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.075452+0000) 2019-10-13T11:51:14.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:14.261+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.261847+0000) 2019-10-13T11:51:14.263 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:14.261+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.261797+0000) 2019-10-13T11:51:14.263 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:14.261+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.262003+0000) 2019-10-13T11:51:14.263 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:14.261+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.262309+0000) 2019-10-13T11:51:14.263 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:14.262+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.262413+0000) 2019-10-13T11:51:14.264 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:14.262+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.262727+0000) 2019-10-13T11:51:14.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:14.262+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.262689+0000) 2019-10-13T11:51:14.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:14.264+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.265330+0000) 2019-10-13T11:51:14.271 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:14.270+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:15.730 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:15.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:15.729775+0000) 2019-10-13T11:51:15.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:15.729+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:15.729775+0000) 2019-10-13T11:51:15.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:15.778+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:15.778871+0000) 2019-10-13T11:51:18.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:18.634+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.634370+0000) 2019-10-13T11:51:18.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:18.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.965258+0000) 2019-10-13T11:51:19.272 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:19.271+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.271933+0000) 2019-10-13T11:51:19.273 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:19.271+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.272353+0000) 2019-10-13T11:51:19.273 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:19.272+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.272641+0000) 2019-10-13T11:51:19.274 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:19.272+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.273140+0000) 2019-10-13T11:51:19.275 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:19.272+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.273017+0000) 2019-10-13T11:51:19.275 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:19.273+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.273429+0000) 2019-10-13T11:51:19.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:19.273+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.273529+0000) 2019-10-13T11:51:19.275 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:19.274+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:19.275511+0000) 2019-10-13T11:51:19.282 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:19.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218699 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:21.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:21.586+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:21.586406+0000) 2019-10-13T11:51:23.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:23.071+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.071788+0000) 2019-10-13T11:51:23.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:23.075+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.075611+0000) 2019-10-13T11:51:24.284 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:24.283+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.284054+0000) 2019-10-13T11:51:24.285 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:24.283+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.283924+0000) 2019-10-13T11:51:24.285 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:24.283+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.284112+0000) 2019-10-13T11:51:24.286 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:24.283+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.284376+0000) 2019-10-13T11:51:24.286 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:24.284+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.284591+0000) 2019-10-13T11:51:24.286 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:24.284+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.284668+0000) 2019-10-13T11:51:24.286 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:24.284+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.285024+0000) 2019-10-13T11:51:24.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:24.286+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.287213+0000) 2019-10-13T11:51:24.294 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:24.292+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218727 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:25.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:25.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:25.729999+0000) 2019-10-13T11:51:25.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:25.729+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:25.729999+0000) 2019-10-13T11:51:25.782 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:25.781+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:25.782095+0000) 2019-10-13T11:51:28.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:28.634+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.634566+0000) 2019-10-13T11:51:28.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:28.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.965387+0000) 2019-10-13T11:51:29.296 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:29.294+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.295548+0000) 2019-10-13T11:51:29.297 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:29.294+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.295364+0000) 2019-10-13T11:51:29.297 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:29.295+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.295556+0000) 2019-10-13T11:51:29.297 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:29.295+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.295783+0000) 2019-10-13T11:51:29.297 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:29.295+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.295805+0000) 2019-10-13T11:51:29.298 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:29.295+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.296354+0000) 2019-10-13T11:51:29.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:29.295+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.296320+0000) 2019-10-13T11:51:29.298 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:29.297+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:29.298385+0000) 2019-10-13T11:51:29.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:29.303+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218761 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:31.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:31.585+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:31.586589+0000) 2019-10-13T11:51:33.072 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:33.070+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.071965+0000) 2019-10-13T11:51:33.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:33.074+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.075793+0000) 2019-10-13T11:51:33.985 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:51:33.985 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:33.991 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:51:33.992 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:34.306 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:34.304+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.305782+0000) 2019-10-13T11:51:34.307 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:34.304+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.306039+0000) 2019-10-13T11:51:34.308 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:34.304+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.306568+0000) 2019-10-13T11:51:34.308 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:34.305+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.306480+0000) 2019-10-13T11:51:34.308 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:34.305+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.306671+0000) 2019-10-13T11:51:34.308 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:34.305+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.307067+0000) 2019-10-13T11:51:34.308 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:34.305+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.307015+0000) 2019-10-13T11:51:34.309 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:34.306+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.308554+0000) 2019-10-13T11:51:34.316 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:34.313+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218791 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:35.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:35.729+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:35.730185+0000) 2019-10-13T11:51:35.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:35.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:35.730185+0000) 2019-10-13T11:51:35.784 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:35.783+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:35.784303+0000) 2019-10-13T11:51:38.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:38.633+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.634736+0000) 2019-10-13T11:51:38.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:38.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.965549+0000) 2019-10-13T11:51:39.317 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:39.315+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.317142+0000) 2019-10-13T11:51:39.318 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:39.315+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.317036+0000) 2019-10-13T11:51:39.319 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:39.315+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.317343+0000) 2019-10-13T11:51:39.319 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:39.316+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.317716+0000) 2019-10-13T11:51:39.319 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:39.316+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.317830+0000) 2019-10-13T11:51:39.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:39.316+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.317996+0000) 2019-10-13T11:51:39.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:39.316+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.318366+0000) 2019-10-13T11:51:39.321 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:39.319+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:39.320733+0000) 2019-10-13T11:51:39.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:39.324+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218825 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:41.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:41.585+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:41.586758+0000) 2019-10-13T11:51:43.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:43.071+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.072080+0000) 2019-10-13T11:51:43.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:43.075+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.075978+0000) 2019-10-13T11:51:44.328 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:44.326+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.327961+0000) 2019-10-13T11:51:44.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:44.326+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.328420+0000) 2019-10-13T11:51:44.330 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:44.327+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.328462+0000) 2019-10-13T11:51:44.330 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:44.327+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.328579+0000) 2019-10-13T11:51:44.330 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:44.327+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.328889+0000) 2019-10-13T11:51:44.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:44.327+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.329165+0000) 2019-10-13T11:51:44.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:44.327+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.329223+0000) 2019-10-13T11:51:44.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:44.329+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.331274+0000) 2019-10-13T11:51:44.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:44.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218853 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:45.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:45.729+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:45.730349+0000) 2019-10-13T11:51:45.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:45.729+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:45.730349+0000) 2019-10-13T11:51:45.787 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:45.785+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:45.786699+0000) 2019-10-13T11:51:48.635 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:48.634+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.634880+0000) 2019-10-13T11:51:48.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:48.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.965756+0000) 2019-10-13T11:51:49.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:49.337+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.339092+0000) 2019-10-13T11:51:49.340 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:49.337+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.339319+0000) 2019-10-13T11:51:49.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:49.339+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.339786+0000) 2019-10-13T11:51:49.341 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:49.338+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.339724+0000) 2019-10-13T11:51:49.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:49.341+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.342449+0000) 2019-10-13T11:51:49.344 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:49.339+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.340436+0000) 2019-10-13T11:51:49.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:49.339+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.340761+0000) 2019-10-13T11:51:49.345 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:49.339+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:49.341010+0000) 2019-10-13T11:51:49.353 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:49.346+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:51.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:51.586+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:51.586919+0000) 2019-10-13T11:51:53.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:53.071+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.072268+0000) 2019-10-13T11:51:53.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:53.075+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.076165+0000) 2019-10-13T11:51:54.351 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:54.348+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.350245+0000) 2019-10-13T11:51:54.351 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:54.349+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.350605+0000) 2019-10-13T11:51:54.352 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:54.350+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.350753+0000) 2019-10-13T11:51:54.352 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:54.349+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.350792+0000) 2019-10-13T11:51:54.352 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:54.349+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.350872+0000) 2019-10-13T11:51:54.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:54.349+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.351060+0000) 2019-10-13T11:51:54.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:54.349+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.351294+0000) 2019-10-13T11:51:54.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:54.353+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.354143+0000) 2019-10-13T11:51:54.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:54.358+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218917 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:51:55.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:55.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:55.730559+0000) 2019-10-13T11:51:55.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:55.730+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:55.730558+0000) 2019-10-13T11:51:55.789 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:55.788+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:55.789169+0000) 2019-10-13T11:51:58.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:58.634+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.635052+0000) 2019-10-13T11:51:58.966 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:58.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.965939+0000) 2019-10-13T11:51:59.362 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:51:59.360+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.360949+0000) 2019-10-13T11:51:59.362 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:51:59.360+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.361084+0000) 2019-10-13T11:51:59.362 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:51:59.360+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.361146+0000) 2019-10-13T11:51:59.362 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:51:59.360+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.361299+0000) 2019-10-13T11:51:59.363 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:51:59.361+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.361856+0000) 2019-10-13T11:51:59.363 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:51:59.362+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.362949+0000) 2019-10-13T11:51:59.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:51:59.361+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.361892+0000) 2019-10-13T11:51:59.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:51:59.361+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:59.362096+0000) 2019-10-13T11:51:59.369 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:51:59.367+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218951 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:01.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:01.586+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:01.587118+0000) 2019-10-13T11:52:03.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:03.072+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.072476+0000) 2019-10-13T11:52:03.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:03.075+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.076368+0000) 2019-10-13T11:52:04.089 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:52:04.091 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:04.095 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:52:04.095 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:04.372 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:04.369+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.370367+0000) 2019-10-13T11:52:04.372 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:04.370+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.370489+0000) 2019-10-13T11:52:04.372 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:04.370+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.370756+0000) 2019-10-13T11:52:04.373 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:04.370+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.370945+0000) 2019-10-13T11:52:04.373 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:04.370+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.371090+0000) 2019-10-13T11:52:04.373 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:04.370+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.371663+0000) 2019-10-13T11:52:04.374 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:04.372+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.373349+0000) 2019-10-13T11:52:04.374 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:04.371+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.371587+0000) 2019-10-13T11:52:04.379 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:04.378+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 218979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:05.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:05.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:05.730728+0000) 2019-10-13T11:52:05.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:05.730+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:05.730728+0000) 2019-10-13T11:52:05.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:05.790+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:05.791577+0000) 2019-10-13T11:52:08.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:08.635+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.635263+0000) 2019-10-13T11:52:08.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:08.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.966055+0000) 2019-10-13T11:52:09.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:09.380+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.380612+0000) 2019-10-13T11:52:09.382 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:09.380+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.380934+0000) 2019-10-13T11:52:09.382 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:09.380+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.380816+0000) 2019-10-13T11:52:09.383 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:09.380+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.381117+0000) 2019-10-13T11:52:09.383 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:09.381+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.381412+0000) 2019-10-13T11:52:09.383 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:09.380+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.381299+0000) 2019-10-13T11:52:09.384 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:09.381+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.381867+0000) 2019-10-13T11:52:09.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:09.382+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:09.383403+0000) 2019-10-13T11:52:09.395 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:09.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219015 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:11.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:11.587+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:11.587345+0000) 2019-10-13T11:52:13.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:13.072+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.072688+0000) 2019-10-13T11:52:13.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:13.076+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.076540+0000) 2019-10-13T11:52:14.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:14.390+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.391359+0000) 2019-10-13T11:52:14.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:14.390+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.390804+0000) 2019-10-13T11:52:14.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:14.390+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.391220+0000) 2019-10-13T11:52:14.393 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:14.391+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.391570+0000) 2019-10-13T11:52:14.393 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:14.391+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.391879+0000) 2019-10-13T11:52:14.393 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:14.391+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.391665+0000) 2019-10-13T11:52:14.394 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:14.391+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.392315+0000) 2019-10-13T11:52:14.395 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:14.393+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.394604+0000) 2019-10-13T11:52:14.401 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:14.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219043 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:15.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:15.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:15.730912+0000) 2019-10-13T11:52:15.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:15.730+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:15.730912+0000) 2019-10-13T11:52:15.795 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:15.793+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:15.794623+0000) 2019-10-13T11:52:18.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:18.634+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.635484+0000) 2019-10-13T11:52:18.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:18.964+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.966203+0000) 2019-10-13T11:52:19.402 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:19.400+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.401536+0000) 2019-10-13T11:52:19.402 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:19.400+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.401845+0000) 2019-10-13T11:52:19.403 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:19.400+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.401971+0000) 2019-10-13T11:52:19.403 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:19.400+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.402092+0000) 2019-10-13T11:52:19.404 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:19.400+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.402372+0000) 2019-10-13T11:52:19.404 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:19.401+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.402600+0000) 2019-10-13T11:52:19.404 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:19.402+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.403013+0000) 2019-10-13T11:52:19.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:19.404+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:19.405546+0000) 2019-10-13T11:52:19.409 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:19.407+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219077 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:21.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:21.586+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:21.587555+0000) 2019-10-13T11:52:23.073 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:23.071+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.072840+0000) 2019-10-13T11:52:23.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:23.075+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.076684+0000) 2019-10-13T11:52:24.412 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:24.409+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.411168+0000) 2019-10-13T11:52:24.412 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:24.409+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.411312+0000) 2019-10-13T11:52:24.412 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:24.410+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.411600+0000) 2019-10-13T11:52:24.413 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:24.410+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.411756+0000) 2019-10-13T11:52:24.413 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:24.410+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.411782+0000) 2019-10-13T11:52:24.414 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:24.410+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.411859+0000) 2019-10-13T11:52:24.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:24.410+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.412263+0000) 2019-10-13T11:52:24.414 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:24.411+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.413306+0000) 2019-10-13T11:52:24.424 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:24.418+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219105 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:25.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:25.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:25.731122+0000) 2019-10-13T11:52:25.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:25.730+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:25.731122+0000) 2019-10-13T11:52:25.797 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:25.795+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:25.796959+0000) 2019-10-13T11:52:28.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:28.634+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.635661+0000) 2019-10-13T11:52:28.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:28.965+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.966389+0000) 2019-10-13T11:52:29.422 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:29.419+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.421313+0000) 2019-10-13T11:52:29.423 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:29.420+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.421769+0000) 2019-10-13T11:52:29.423 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:29.420+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.421696+0000) 2019-10-13T11:52:29.423 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:29.420+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.421938+0000) 2019-10-13T11:52:29.423 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:29.421+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.422523+0000) 2019-10-13T11:52:29.424 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:29.421+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.422749+0000) 2019-10-13T11:52:29.424 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:29.420+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.422220+0000) 2019-10-13T11:52:29.425 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:29.422+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:29.424444+0000) 2019-10-13T11:52:29.432 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:29.429+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:31.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:31.587+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:31.587715+0000) 2019-10-13T11:52:33.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:33.072+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.073019+0000) 2019-10-13T11:52:33.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:33.076+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.076888+0000) 2019-10-13T11:52:34.245 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:52:34.245 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:34.250 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:52:34.250 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:34.433 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:34.431+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.432768+0000) 2019-10-13T11:52:34.434 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:34.431+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.432987+0000) 2019-10-13T11:52:34.434 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:34.431+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.432885+0000) 2019-10-13T11:52:34.434 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:34.431+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.433169+0000) 2019-10-13T11:52:34.435 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:34.431+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.433661+0000) 2019-10-13T11:52:34.435 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:34.432+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.433574+0000) 2019-10-13T11:52:34.435 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:34.432+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.433891+0000) 2019-10-13T11:52:34.436 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:34.435+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.435997+0000) 2019-10-13T11:52:34.441 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:34.439+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219169 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:35.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:35.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:35.731349+0000) 2019-10-13T11:52:35.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:35.730+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:35.731349+0000) 2019-10-13T11:52:35.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:35.798+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:35.799347+0000) 2019-10-13T11:52:38.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:38.635+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.635826+0000) 2019-10-13T11:52:38.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:38.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.966569+0000) 2019-10-13T11:52:39.444 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:39.441+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.443186+0000) 2019-10-13T11:52:39.444 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:39.441+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.443360+0000) 2019-10-13T11:52:39.445 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:39.442+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.443641+0000) 2019-10-13T11:52:39.445 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:39.444+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.444985+0000) 2019-10-13T11:52:39.446 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:39.442+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.443890+0000) 2019-10-13T11:52:39.446 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:39.442+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.444175+0000) 2019-10-13T11:52:39.446 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:39.442+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.444334+0000) 2019-10-13T11:52:39.447 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:39.445+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:39.446464+0000) 2019-10-13T11:52:39.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:39.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219203 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:41.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:41.587+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:41.587905+0000) 2019-10-13T11:52:43.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:43.072+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.073162+0000) 2019-10-13T11:52:43.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:43.076+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.077034+0000) 2019-10-13T11:52:44.452 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:44.450+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.451169+0000) 2019-10-13T11:52:44.452 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:44.450+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.451691+0000) 2019-10-13T11:52:44.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:44.451+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.451600+0000) 2019-10-13T11:52:44.453 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:44.451+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.451821+0000) 2019-10-13T11:52:44.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:44.452+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.452535+0000) 2019-10-13T11:52:44.454 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:44.452+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.452678+0000) 2019-10-13T11:52:44.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:44.452+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.452897+0000) 2019-10-13T11:52:44.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:44.453+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.454412+0000) 2019-10-13T11:52:44.461 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:44.459+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:45.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:45.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:45.731517+0000) 2019-10-13T11:52:45.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:45.731+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:45.731544+0000) 2019-10-13T11:52:45.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:45.801+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:45.801772+0000) 2019-10-13T11:52:48.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:48.635+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.635958+0000) 2019-10-13T11:52:48.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:48.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.966729+0000) 2019-10-13T11:52:49.463 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:49.461+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.462242+0000) 2019-10-13T11:52:49.463 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:49.462+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.462609+0000) 2019-10-13T11:52:49.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:49.462+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.462729+0000) 2019-10-13T11:52:49.464 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:49.463+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.463771+0000) 2019-10-13T11:52:49.465 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:49.462+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.462949+0000) 2019-10-13T11:52:49.465 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:49.462+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.463288+0000) 2019-10-13T11:52:49.465 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:49.463+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.463602+0000) 2019-10-13T11:52:49.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:49.466+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:49.466958+0000) 2019-10-13T11:52:49.472 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:49.470+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:51.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:51.587+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:51.588093+0000) 2019-10-13T11:52:53.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:53.073+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.073326+0000) 2019-10-13T11:52:53.077 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:53.076+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.077200+0000) 2019-10-13T11:52:54.473 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:54.472+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.473079+0000) 2019-10-13T11:52:54.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:54.472+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.472977+0000) 2019-10-13T11:52:54.475 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:54.472+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.473103+0000) 2019-10-13T11:52:54.475 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:54.473+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.473464+0000) 2019-10-13T11:52:54.475 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:54.473+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.473673+0000) 2019-10-13T11:52:54.475 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:54.473+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.473768+0000) 2019-10-13T11:52:54.475 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:54.473+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.473966+0000) 2019-10-13T11:52:54.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:54.475+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.476660+0000) 2019-10-13T11:52:54.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:54.481+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219295 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:52:55.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:55.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:55.731683+0000) 2019-10-13T11:52:55.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:55.731+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:55.731689+0000) 2019-10-13T11:52:55.804 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:55.803+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:55.804189+0000) 2019-10-13T11:52:58.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:58.635+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.636076+0000) 2019-10-13T11:52:58.967 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:58.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.966882+0000) 2019-10-13T11:52:59.484 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:52:59.482+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.483455+0000) 2019-10-13T11:52:59.485 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:52:59.483+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.483448+0000) 2019-10-13T11:52:59.485 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:52:59.483+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.483542+0000) 2019-10-13T11:52:59.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:52:59.483+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.483789+0000) 2019-10-13T11:52:59.485 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:52:59.483+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.484060+0000) 2019-10-13T11:52:59.486 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:52:59.483+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.484185+0000) 2019-10-13T11:52:59.486 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:52:59.483+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.484368+0000) 2019-10-13T11:52:59.486 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:52:59.484+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:59.485696+0000) 2019-10-13T11:52:59.492 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:52:59.490+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:01.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:01.587+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:01.588280+0000) 2019-10-13T11:53:03.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:03.072+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.073509+0000) 2019-10-13T11:53:03.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:03.077+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.077379+0000) 2019-10-13T11:53:04.418 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:53:04.418 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:04.422 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:53:04.422 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:04.493 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:04.492+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.493006+0000) 2019-10-13T11:53:04.494 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:04.491+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.493069+0000) 2019-10-13T11:53:04.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:04.491+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.493212+0000) 2019-10-13T11:53:04.494 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:04.491+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.493298+0000) 2019-10-13T11:53:04.495 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:04.492+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.493649+0000) 2019-10-13T11:53:04.495 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:04.492+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.493605+0000) 2019-10-13T11:53:04.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:04.492+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.494086+0000) 2019-10-13T11:53:04.496 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:04.494+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.495139+0000) 2019-10-13T11:53:04.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:04.499+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:05.732 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:05.730+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:05.731822+0000) 2019-10-13T11:53:05.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:05.730+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:05.731856+0000) 2019-10-13T11:53:05.807 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:05.806+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:05.806607+0000) 2019-10-13T11:53:08.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:08.635+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.636249+0000) 2019-10-13T11:53:08.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:08.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.967056+0000) 2019-10-13T11:53:09.503 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:09.501+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.502748+0000) 2019-10-13T11:53:09.504 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:09.501+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.502930+0000) 2019-10-13T11:53:09.504 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:09.502+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.503411+0000) 2019-10-13T11:53:09.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:09.501+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.503244+0000) 2019-10-13T11:53:09.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:09.501+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.503334+0000) 2019-10-13T11:53:09.505 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:09.502+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.503968+0000) 2019-10-13T11:53:09.505 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:09.502+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.504053+0000) 2019-10-13T11:53:09.506 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:09.505+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:09.506183+0000) 2019-10-13T11:53:09.510 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:09.508+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219393 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:11.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:11.587+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:11.588463+0000) 2019-10-13T11:53:13.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:13.072+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.073699+0000) 2019-10-13T11:53:13.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:13.076+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.077567+0000) 2019-10-13T11:53:14.513 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:14.510+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.512223+0000) 2019-10-13T11:53:14.513 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:14.510+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.512107+0000) 2019-10-13T11:53:14.514 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:14.511+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.512475+0000) 2019-10-13T11:53:14.514 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:14.511+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.512676+0000) 2019-10-13T11:53:14.514 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:14.511+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.513199+0000) 2019-10-13T11:53:14.515 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:14.511+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.513011+0000) 2019-10-13T11:53:14.515 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:14.511+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.513184+0000) 2019-10-13T11:53:14.516 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:14.514+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.516313+0000) 2019-10-13T11:53:14.522 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:14.519+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219421 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:15.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:15.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:15.732029+0000) 2019-10-13T11:53:15.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:15.731+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:15.732029+0000) 2019-10-13T11:53:15.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:15.807+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:15.808958+0000) 2019-10-13T11:53:18.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:18.635+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.636460+0000) 2019-10-13T11:53:18.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:18.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.967257+0000) 2019-10-13T11:53:19.524 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:19.521+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.523041+0000) 2019-10-13T11:53:19.524 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:19.521+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.523357+0000) 2019-10-13T11:53:19.525 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:19.522+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.523586+0000) 2019-10-13T11:53:19.525 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:19.522+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.523799+0000) 2019-10-13T11:53:19.525 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:19.522+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.523987+0000) 2019-10-13T11:53:19.525 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:19.522+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.524257+0000) 2019-10-13T11:53:19.526 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:19.523+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.524736+0000) 2019-10-13T11:53:19.526 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:19.524+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:19.526264+0000) 2019-10-13T11:53:19.533 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:19.530+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219455 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:21.589 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:21.588+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:21.588681+0000) 2019-10-13T11:53:23.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:23.073+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.073891+0000) 2019-10-13T11:53:23.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:23.076+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.077789+0000) 2019-10-13T11:53:24.534 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:24.532+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.534158+0000) 2019-10-13T11:53:24.535 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:24.532+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.534132+0000) 2019-10-13T11:53:24.535 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:24.532+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.534276+0000) 2019-10-13T11:53:24.536 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:24.533+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.534535+0000) 2019-10-13T11:53:24.536 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:24.533+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.534680+0000) 2019-10-13T11:53:24.536 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:24.533+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.534907+0000) 2019-10-13T11:53:24.536 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:24.533+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.534963+0000) 2019-10-13T11:53:24.538 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:24.536+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.537558+0000) 2019-10-13T11:53:24.543 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:24.541+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:25.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:25.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:25.732227+0000) 2019-10-13T11:53:25.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:25.731+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:25.732228+0000) 2019-10-13T11:53:25.811 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:25.810+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:25.811393+0000) 2019-10-13T11:53:28.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:28.636+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.636635+0000) 2019-10-13T11:53:28.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:28.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.967438+0000) 2019-10-13T11:53:29.545 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:29.544+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.544864+0000) 2019-10-13T11:53:29.546 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:29.544+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.544732+0000) 2019-10-13T11:53:29.546 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:29.544+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.544909+0000) 2019-10-13T11:53:29.546 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:29.544+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.545127+0000) 2019-10-13T11:53:29.547 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:29.544+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.545320+0000) 2019-10-13T11:53:29.547 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:29.545+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.545498+0000) 2019-10-13T11:53:29.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:29.545+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.546037+0000) 2019-10-13T11:53:29.548 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:29.546+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:29.547404+0000) 2019-10-13T11:53:29.554 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:29.552+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219519 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:31.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:31.588+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:31.588872+0000) 2019-10-13T11:53:33.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:33.073+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.074074+0000) 2019-10-13T11:53:33.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:33.077+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.077959+0000) 2019-10-13T11:53:34.556 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:34.554+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.555700+0000) 2019-10-13T11:53:34.557 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:34.555+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.555587+0000) 2019-10-13T11:53:34.557 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:34.555+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.555946+0000) 2019-10-13T11:53:34.557 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:34.555+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.555866+0000) 2019-10-13T11:53:34.558 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:34.556+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.556430+0000) 2019-10-13T11:53:34.558 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:34.555+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.556164+0000) 2019-10-13T11:53:34.558 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:34.556+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.556895+0000) 2019-10-13T11:53:34.559 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:34.558+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.559345+0000) 2019-10-13T11:53:34.563 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:53:34.563 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:34.565 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:34.564+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219547 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:34.567 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:53:34.568 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:35.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:35.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:35.732430+0000) 2019-10-13T11:53:35.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:35.732+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:35.732433+0000) 2019-10-13T11:53:35.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:35.813+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:35.813768+0000) 2019-10-13T11:53:38.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:38.636+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.636853+0000) 2019-10-13T11:53:38.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:38.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.967617+0000) 2019-10-13T11:53:39.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:39.565+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.566203+0000) 2019-10-13T11:53:39.567 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:39.566+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.566567+0000) 2019-10-13T11:53:39.568 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:39.566+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.566606+0000) 2019-10-13T11:53:39.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:39.566+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.566769+0000) 2019-10-13T11:53:39.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:39.566+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.567178+0000) 2019-10-13T11:53:39.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:39.566+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.566958+0000) 2019-10-13T11:53:39.569 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:39.566+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.567084+0000) 2019-10-13T11:53:39.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:39.568+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:39.569397+0000) 2019-10-13T11:53:39.574 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:39.573+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219581 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:41.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:41.588+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:41.589059+0000) 2019-10-13T11:53:43.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:43.074+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.074262+0000) 2019-10-13T11:53:43.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:43.077+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.078171+0000) 2019-10-13T11:53:44.576 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:44.575+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.575694+0000) 2019-10-13T11:53:44.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:44.575+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.575876+0000) 2019-10-13T11:53:44.577 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:44.575+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.576351+0000) 2019-10-13T11:53:44.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:44.576+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.576569+0000) 2019-10-13T11:53:44.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:44.576+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.577358+0000) 2019-10-13T11:53:44.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:44.576+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.576754+0000) 2019-10-13T11:53:44.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:44.576+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.576906+0000) 2019-10-13T11:53:44.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:44.578+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.579362+0000) 2019-10-13T11:53:44.586 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:44.585+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:45.733 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:45.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:45.732603+0000) 2019-10-13T11:53:45.733 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:45.732+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:45.732636+0000) 2019-10-13T11:53:45.816 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:45.815+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:45.816186+0000) 2019-10-13T11:53:48.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:48.636+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.637040+0000) 2019-10-13T11:53:48.968 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:48.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.967816+0000) 2019-10-13T11:53:49.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:49.587+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.588291+0000) 2019-10-13T11:53:49.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:49.586+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.588190+0000) 2019-10-13T11:53:49.590 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:49.587+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.588402+0000) 2019-10-13T11:53:49.590 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:49.587+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.588453+0000) 2019-10-13T11:53:49.590 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:49.587+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.588839+0000) 2019-10-13T11:53:49.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:49.587+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.589059+0000) 2019-10-13T11:53:49.591 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:49.589+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.590141+0000) 2019-10-13T11:53:49.591 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:49.588+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:49.589466+0000) 2019-10-13T11:53:49.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:49.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219645 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:51.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:51.588+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:51.589282+0000) 2019-10-13T11:53:53.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:53.073+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.074459+0000) 2019-10-13T11:53:53.078 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:53.078+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.078339+0000) 2019-10-13T11:53:54.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:54.597+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.599108+0000) 2019-10-13T11:53:54.600 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:54.597+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.599326+0000) 2019-10-13T11:53:54.600 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:54.598+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.599640+0000) 2019-10-13T11:53:54.601 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:54.599+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.600566+0000) 2019-10-13T11:53:54.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:54.598+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.599789+0000) 2019-10-13T11:53:54.602 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:54.598+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.599947+0000) 2019-10-13T11:53:54.602 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:54.599+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.600634+0000) 2019-10-13T11:53:54.603 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:54.601+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.602537+0000) 2019-10-13T11:53:54.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:54.605+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219673 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:53:55.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:55.731+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:55.732785+0000) 2019-10-13T11:53:55.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:55.731+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:55.732810+0000) 2019-10-13T11:53:55.819 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:55.818+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:55.818377+0000) 2019-10-13T11:53:58.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:58.635+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.637245+0000) 2019-10-13T11:53:58.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:58.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.968007+0000) 2019-10-13T11:53:59.609 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:53:59.606+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.608331+0000) 2019-10-13T11:53:59.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:53:59.607+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.608517+0000) 2019-10-13T11:53:59.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:53:59.607+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.608646+0000) 2019-10-13T11:53:59.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:53:59.607+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.608750+0000) 2019-10-13T11:53:59.611 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:53:59.608+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.609400+0000) 2019-10-13T11:53:59.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:53:59.608+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.609521+0000) 2019-10-13T11:53:59.611 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:53:59.608+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.609647+0000) 2019-10-13T11:53:59.613 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:53:59.611+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:59.612585+0000) 2019-10-13T11:53:59.616 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:53:59.614+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:01.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:01.588+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:01.589449+0000) 2019-10-13T11:54:03.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:03.073+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.074661+0000) 2019-10-13T11:54:03.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:03.077+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.078546+0000) 2019-10-13T11:54:04.618 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:04.615+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.616892+0000) 2019-10-13T11:54:04.618 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:04.615+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.617044+0000) 2019-10-13T11:54:04.618 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:04.615+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.617239+0000) 2019-10-13T11:54:04.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:04.616+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.618431+0000) 2019-10-13T11:54:04.619 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:04.616+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.617930+0000) 2019-10-13T11:54:04.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:04.617+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.618457+0000) 2019-10-13T11:54:04.619 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:04.617+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.618512+0000) 2019-10-13T11:54:04.620 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:04.618+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.620114+0000) 2019-10-13T11:54:04.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:04.623+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:04.676 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:54:04.676 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:04.680 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:54:04.681 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:05.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:05.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:05.732977+0000) 2019-10-13T11:54:05.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:05.732+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:05.733000+0000) 2019-10-13T11:54:05.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:05.819+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:05.820669+0000) 2019-10-13T11:54:08.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:08.636+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.637410+0000) 2019-10-13T11:54:08.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:08.966+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.968126+0000) 2019-10-13T11:54:09.628 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:09.625+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.627484+0000) 2019-10-13T11:54:09.629 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:09.625+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.627376+0000) 2019-10-13T11:54:09.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:09.626+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.627601+0000) 2019-10-13T11:54:09.629 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:09.626+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.627807+0000) 2019-10-13T11:54:09.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:09.626+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.628023+0000) 2019-10-13T11:54:09.629 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:09.626+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.628030+0000) 2019-10-13T11:54:09.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:09.627+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.628409+0000) 2019-10-13T11:54:09.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:09.629+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:09.631415+0000) 2019-10-13T11:54:09.637 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:09.634+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:11.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:11.589+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:11.589624+0000) 2019-10-13T11:54:13.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:13.074+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.074846+0000) 2019-10-13T11:54:13.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:13.077+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.078711+0000) 2019-10-13T11:54:14.639 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:14.637+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.638165+0000) 2019-10-13T11:54:14.640 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:14.638+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.638918+0000) 2019-10-13T11:54:14.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:14.638+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.638752+0000) 2019-10-13T11:54:14.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:14.638+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.638845+0000) 2019-10-13T11:54:14.640 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:14.638+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.639043+0000) 2019-10-13T11:54:14.640 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:14.638+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.639133+0000) 2019-10-13T11:54:14.641 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:14.639+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.639412+0000) 2019-10-13T11:54:14.642 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:14.640+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.641801+0000) 2019-10-13T11:54:14.648 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:14.647+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:15.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:15.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:15.733200+0000) 2019-10-13T11:54:15.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:15.732+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:15.733200+0000) 2019-10-13T11:54:15.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:15.822+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:15.823182+0000) 2019-10-13T11:54:18.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:18.637+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.637583+0000) 2019-10-13T11:54:18.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:18.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.968273+0000) 2019-10-13T11:54:19.650 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:19.648+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.648885+0000) 2019-10-13T11:54:19.650 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:19.648+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.649111+0000) 2019-10-13T11:54:19.650 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:19.648+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.649235+0000) 2019-10-13T11:54:19.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:19.649+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.649498+0000) 2019-10-13T11:54:19.651 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:19.649+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.649903+0000) 2019-10-13T11:54:19.651 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:19.649+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.649994+0000) 2019-10-13T11:54:19.651 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:19.649+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.650320+0000) 2019-10-13T11:54:19.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:19.651+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:19.652613+0000) 2019-10-13T11:54:19.658 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:19.657+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:21.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:21.589+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:21.589805+0000) 2019-10-13T11:54:23.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:23.074+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.074975+0000) 2019-10-13T11:54:23.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:23.078+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.078832+0000) 2019-10-13T11:54:24.660 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:24.659+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.659801+0000) 2019-10-13T11:54:24.661 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:24.659+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.660111+0000) 2019-10-13T11:54:24.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:24.659+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.660273+0000) 2019-10-13T11:54:24.662 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:24.660+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.660788+0000) 2019-10-13T11:54:24.662 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:24.660+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.660467+0000) 2019-10-13T11:54:24.662 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:24.660+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.660867+0000) 2019-10-13T11:54:24.662 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:24.660+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.661044+0000) 2019-10-13T11:54:24.663 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:24.661+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.661982+0000) 2019-10-13T11:54:24.669 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:24.668+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:25.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:25.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:25.733423+0000) 2019-10-13T11:54:25.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:25.733+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:25.733425+0000) 2019-10-13T11:54:25.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:25.824+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:25.825526+0000) 2019-10-13T11:54:28.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:28.637+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.637781+0000) 2019-10-13T11:54:28.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:28.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.968464+0000) 2019-10-13T11:54:29.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:29.669+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.670305+0000) 2019-10-13T11:54:29.671 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:29.670+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.670762+0000) 2019-10-13T11:54:29.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:29.670+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.670821+0000) 2019-10-13T11:54:29.672 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:29.671+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.671391+0000) 2019-10-13T11:54:29.673 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:29.671+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.671437+0000) 2019-10-13T11:54:29.673 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:29.671+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.671639+0000) 2019-10-13T11:54:29.673 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:29.672+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.672702+0000) 2019-10-13T11:54:29.675 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:29.673+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:29.674675+0000) 2019-10-13T11:54:29.681 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:29.679+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219897 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:31.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:31.589+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:31.589964+0000) 2019-10-13T11:54:33.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:33.074+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.075145+0000) 2019-10-13T11:54:33.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:33.078+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.078961+0000) 2019-10-13T11:54:34.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:34.681+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.682261+0000) 2019-10-13T11:54:34.683 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:34.680+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.682162+0000) 2019-10-13T11:54:34.683 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:34.681+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.682405+0000) 2019-10-13T11:54:34.684 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:34.681+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.682536+0000) 2019-10-13T11:54:34.684 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:34.681+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.682828+0000) 2019-10-13T11:54:34.684 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:34.681+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.683291+0000) 2019-10-13T11:54:34.685 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:34.681+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.683287+0000) 2019-10-13T11:54:34.685 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:34.684+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.685023+0000) 2019-10-13T11:54:34.692 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:34.689+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219925 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:34.839 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:54:34.839 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:34.843 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:54:34.843 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:35.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:35.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:35.733533+0000) 2019-10-13T11:54:35.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:35.732+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:35.733569+0000) 2019-10-13T11:54:35.828 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:35.827+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:35.827724+0000) 2019-10-13T11:54:38.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:38.637+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.637937+0000) 2019-10-13T11:54:38.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:38.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.968656+0000) 2019-10-13T11:54:39.693 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:39.691+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.692486+0000) 2019-10-13T11:54:39.694 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:39.691+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.692667+0000) 2019-10-13T11:54:39.694 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:39.691+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.692819+0000) 2019-10-13T11:54:39.694 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:39.691+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.693182+0000) 2019-10-13T11:54:39.695 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:39.692+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.693171+0000) 2019-10-13T11:54:39.696 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:39.691+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.693344+0000) 2019-10-13T11:54:39.696 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:39.692+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.693530+0000) 2019-10-13T11:54:39.697 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:39.694+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:39.695562+0000) 2019-10-13T11:54:39.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:39.698+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:41.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:41.589+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:41.590156+0000) 2019-10-13T11:54:43.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:43.074+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.075322+0000) 2019-10-13T11:54:43.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:43.078+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.079132+0000) 2019-10-13T11:54:44.702 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:44.700+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.701457+0000) 2019-10-13T11:54:44.702 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:44.700+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.701761+0000) 2019-10-13T11:54:44.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:44.700+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.701829+0000) 2019-10-13T11:54:44.703 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:44.700+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.701958+0000) 2019-10-13T11:54:44.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:44.701+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.702317+0000) 2019-10-13T11:54:44.704 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:44.700+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.702254+0000) 2019-10-13T11:54:44.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:44.700+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.702348+0000) 2019-10-13T11:54:44.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:44.702+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.703653+0000) 2019-10-13T11:54:44.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:44.709+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 219987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:45.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:45.732+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:45.733654+0000) 2019-10-13T11:54:45.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:45.732+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:45.733717+0000) 2019-10-13T11:54:45.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:45.830+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:45.830766+0000) 2019-10-13T11:54:48.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:48.637+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.638119+0000) 2019-10-13T11:54:48.969 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:48.967+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.968787+0000) 2019-10-13T11:54:49.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:49.711+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.712546+0000) 2019-10-13T11:54:49.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:49.711+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.712860+0000) 2019-10-13T11:54:49.714 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:49.711+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.712907+0000) 2019-10-13T11:54:49.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:49.711+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.713137+0000) 2019-10-13T11:54:49.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:49.711+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.713214+0000) 2019-10-13T11:54:49.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:49.711+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.713359+0000) 2019-10-13T11:54:49.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:49.712+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.713607+0000) 2019-10-13T11:54:49.715 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:49.714+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:49.715521+0000) 2019-10-13T11:54:49.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:49.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220023 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:51.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:51.589+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:51.590342+0000) 2019-10-13T11:54:53.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:53.074+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.075529+0000) 2019-10-13T11:54:53.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:53.078+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.079305+0000) 2019-10-13T11:54:54.723 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:54.721+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.723105+0000) 2019-10-13T11:54:54.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:54.721+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.723074+0000) 2019-10-13T11:54:54.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:54.721+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.723255+0000) 2019-10-13T11:54:54.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:54.721+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.723389+0000) 2019-10-13T11:54:54.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:54.722+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.723565+0000) 2019-10-13T11:54:54.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:54.722+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.723958+0000) 2019-10-13T11:54:54.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:54.722+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.724300+0000) 2019-10-13T11:54:54.726 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:54.724+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.726475+0000) 2019-10-13T11:54:54.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:54.730+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220051 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:54:55.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:55.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:55.733829+0000) 2019-10-13T11:54:55.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:55.733+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:55.733878+0000) 2019-10-13T11:54:55.834 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:55.832+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:55.833761+0000) 2019-10-13T11:54:58.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:58.636+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.638305+0000) 2019-10-13T11:54:58.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:58.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.968939+0000) 2019-10-13T11:54:59.735 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:54:59.732+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.734393+0000) 2019-10-13T11:54:59.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:54:59.733+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.734260+0000) 2019-10-13T11:54:59.736 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:54:59.734+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.734485+0000) 2019-10-13T11:54:59.736 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:54:59.734+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.734550+0000) 2019-10-13T11:54:59.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:54:59.734+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.734745+0000) 2019-10-13T11:54:59.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:54:59.734+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.734924+0000) 2019-10-13T11:54:59.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:54:59.734+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.735357+0000) 2019-10-13T11:54:59.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:54:59.736+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:59.738238+0000) 2019-10-13T11:54:59.744 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:54:59.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:01.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:01.590+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:01.590469+0000) 2019-10-13T11:55:03.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:03.075+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.075713+0000) 2019-10-13T11:55:03.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:03.078+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.079496+0000) 2019-10-13T11:55:04.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:04.744+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.744663+0000) 2019-10-13T11:55:04.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:04.744+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.744795+0000) 2019-10-13T11:55:04.746 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:04.744+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.744932+0000) 2019-10-13T11:55:04.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:04.744+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.745175+0000) 2019-10-13T11:55:04.747 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:04.743+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.745624+0000) 2019-10-13T11:55:04.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:04.745+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.745457+0000) 2019-10-13T11:55:04.747 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:04.745+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.745644+0000) 2019-10-13T11:55:04.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:04.745+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.747495+0000) 2019-10-13T11:55:04.752 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:04.751+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:05.010 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:55:05.010 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:05.014 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:55:05.014 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:05.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:05.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:05.734011+0000) 2019-10-13T11:55:05.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:05.733+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:05.734053+0000) 2019-10-13T11:55:05.836 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:05.835+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:05.836042+0000) 2019-10-13T11:55:08.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:08.638+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.638491+0000) 2019-10-13T11:55:08.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:08.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.969126+0000) 2019-10-13T11:55:09.754 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:09.753+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.753413+0000) 2019-10-13T11:55:09.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:09.753+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.753579+0000) 2019-10-13T11:55:09.755 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:09.753+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.753642+0000) 2019-10-13T11:55:09.755 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:09.753+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.753852+0000) 2019-10-13T11:55:09.756 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:09.753+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.754329+0000) 2019-10-13T11:55:09.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:09.753+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.754158+0000) 2019-10-13T11:55:09.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:09.753+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.754284+0000) 2019-10-13T11:55:09.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:09.756+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:09.756779+0000) 2019-10-13T11:55:09.761 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:09.760+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220149 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:11.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:11.590+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:11.590645+0000) 2019-10-13T11:55:13.076 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:13.075+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.075904+0000) 2019-10-13T11:55:13.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:13.079+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.079664+0000) 2019-10-13T11:55:14.764 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:14.762+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.762806+0000) 2019-10-13T11:55:14.764 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:14.762+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.762983+0000) 2019-10-13T11:55:14.764 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:14.762+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.763058+0000) 2019-10-13T11:55:14.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:14.762+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.763216+0000) 2019-10-13T11:55:14.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:14.762+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.763318+0000) 2019-10-13T11:55:14.765 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:14.763+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.763509+0000) 2019-10-13T11:55:14.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:14.763+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.763921+0000) 2019-10-13T11:55:14.766 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:14.764+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.764914+0000) 2019-10-13T11:55:14.772 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:14.771+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:15.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:15.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:15.734199+0000) 2019-10-13T11:55:15.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:15.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:15.734213+0000) 2019-10-13T11:55:15.839 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:15.837+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:15.838438+0000) 2019-10-13T11:55:18.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:18.637+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.638610+0000) 2019-10-13T11:55:18.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:18.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.969317+0000) 2019-10-13T11:55:19.775 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:19.772+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.773956+0000) 2019-10-13T11:55:19.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:19.772+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.773854+0000) 2019-10-13T11:55:19.775 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:19.772+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.774116+0000) 2019-10-13T11:55:19.776 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:19.772+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.774322+0000) 2019-10-13T11:55:19.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:19.774+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.774870+0000) 2019-10-13T11:55:19.777 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:19.773+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.775057+0000) 2019-10-13T11:55:19.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:19.773+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.775346+0000) 2019-10-13T11:55:19.777 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:19.776+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:19.777159+0000) 2019-10-13T11:55:19.783 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:19.781+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220211 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:21.591 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:21.589+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:21.590849+0000) 2019-10-13T11:55:23.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:23.074+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.076111+0000) 2019-10-13T11:55:23.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:23.079+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.079874+0000) 2019-10-13T11:55:24.785 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:24.782+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.784099+0000) 2019-10-13T11:55:24.785 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:24.782+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.784332+0000) 2019-10-13T11:55:24.785 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:24.783+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.784601+0000) 2019-10-13T11:55:24.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:24.784+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.785129+0000) 2019-10-13T11:55:24.786 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:24.783+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.784777+0000) 2019-10-13T11:55:24.786 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:24.783+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.785065+0000) 2019-10-13T11:55:24.787 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:24.784+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.785503+0000) 2019-10-13T11:55:24.788 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:24.787+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.787799+0000) 2019-10-13T11:55:24.793 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:24.791+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220240 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:25.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:25.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:25.734389+0000) 2019-10-13T11:55:25.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:25.733+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:25.734390+0000) 2019-10-13T11:55:25.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:25.840+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:25.841488+0000) 2019-10-13T11:55:28.639 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:28.637+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.638782+0000) 2019-10-13T11:55:28.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:28.968+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.969474+0000) 2019-10-13T11:55:29.795 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:29.793+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.794517+0000) 2019-10-13T11:55:29.795 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:29.793+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.794716+0000) 2019-10-13T11:55:29.796 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:29.793+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.795166+0000) 2019-10-13T11:55:29.796 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:29.794+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.795550+0000) 2019-10-13T11:55:29.797 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:29.794+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.795621+0000) 2019-10-13T11:55:29.797 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:29.794+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.795990+0000) 2019-10-13T11:55:29.797 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:29.795+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.796092+0000) 2019-10-13T11:55:29.798 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:29.796+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:29.797522+0000) 2019-10-13T11:55:29.804 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:29.802+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:31.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:31.590+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:31.590978+0000) 2019-10-13T11:55:33.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:33.075+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.076273+0000) 2019-10-13T11:55:33.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:33.079+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.080067+0000) 2019-10-13T11:55:34.806 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:34.804+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.805624+0000) 2019-10-13T11:55:34.807 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:34.804+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.805751+0000) 2019-10-13T11:55:34.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:34.804+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.805866+0000) 2019-10-13T11:55:34.808 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:34.806+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.807144+0000) 2019-10-13T11:55:34.808 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:34.805+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.806454+0000) 2019-10-13T11:55:34.808 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:34.805+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.806715+0000) 2019-10-13T11:55:34.808 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:34.805+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.806588+0000) 2019-10-13T11:55:34.809 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:34.807+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.808536+0000) 2019-10-13T11:55:34.816 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:34.813+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:35.097 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:55:35.097 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:35.102 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:55:35.102 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:35.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:35.733+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:35.734566+0000) 2019-10-13T11:55:35.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:35.733+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:35.734567+0000) 2019-10-13T11:55:35.845 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:35.844+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:35.844556+0000) 2019-10-13T11:55:38.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:38.638+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.638961+0000) 2019-10-13T11:55:38.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:38.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.969661+0000) 2019-10-13T11:55:39.818 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:39.815+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.817020+0000) 2019-10-13T11:55:39.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:39.815+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.817162+0000) 2019-10-13T11:55:39.818 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:39.815+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.817307+0000) 2019-10-13T11:55:39.819 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:39.816+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.817477+0000) 2019-10-13T11:55:39.819 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:39.817+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.818314+0000) 2019-10-13T11:55:39.820 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:39.816+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.817668+0000) 2019-10-13T11:55:39.820 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:39.816+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.817882+0000) 2019-10-13T11:55:39.821 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:39.819+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:39.820506+0000) 2019-10-13T11:55:39.827 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:39.824+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220337 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:41.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:41.590+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:41.591152+0000) 2019-10-13T11:55:43.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:43.075+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.076428+0000) 2019-10-13T11:55:43.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:43.079+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.080237+0000) 2019-10-13T11:55:44.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:44.827+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.827948+0000) 2019-10-13T11:55:44.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:44.827+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.828178+0000) 2019-10-13T11:55:44.830 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:44.826+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.828526+0000) 2019-10-13T11:55:44.830 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:44.828+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.828724+0000) 2019-10-13T11:55:44.830 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:44.828+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.828979+0000) 2019-10-13T11:55:44.830 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:44.828+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.829102+0000) 2019-10-13T11:55:44.831 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:44.828+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.829319+0000) 2019-10-13T11:55:44.831 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:44.829+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.830824+0000) 2019-10-13T11:55:44.838 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:44.836+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220367 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:45.735 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:45.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:45.734746+0000) 2019-10-13T11:55:45.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:45.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:45.734735+0000) 2019-10-13T11:55:45.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:45.845+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:45.846910+0000) 2019-10-13T11:55:48.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:48.638+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.639113+0000) 2019-10-13T11:55:48.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:48.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.969832+0000) 2019-10-13T11:55:49.840 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:49.838+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.838701+0000) 2019-10-13T11:55:49.840 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:49.838+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.839102+0000) 2019-10-13T11:55:49.840 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:49.838+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.839237+0000) 2019-10-13T11:55:49.841 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:49.837+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.839673+0000) 2019-10-13T11:55:49.841 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:49.839+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.839821+0000) 2019-10-13T11:55:49.841 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:49.839+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.839778+0000) 2019-10-13T11:55:49.842 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:49.839+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.840053+0000) 2019-10-13T11:55:49.842 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:49.840+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:49.841981+0000) 2019-10-13T11:55:49.849 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:49.848+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220401 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:51.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:51.590+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:51.591353+0000) 2019-10-13T11:55:53.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:53.076+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.076593+0000) 2019-10-13T11:55:53.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:53.079+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.080386+0000) 2019-10-13T11:55:54.852 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:54.850+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.850440+0000) 2019-10-13T11:55:54.852 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:54.850+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.850629+0000) 2019-10-13T11:55:54.852 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:54.850+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.851076+0000) 2019-10-13T11:55:54.852 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:54.851+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.851447+0000) 2019-10-13T11:55:54.853 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:54.849+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.851575+0000) 2019-10-13T11:55:54.853 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:54.851+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.851767+0000) 2019-10-13T11:55:54.854 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:54.851+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.851960+0000) 2019-10-13T11:55:54.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:54.851+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.853204+0000) 2019-10-13T11:55:54.861 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:54.859+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220431 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:55:55.735 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:55.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:55.734890+0000) 2019-10-13T11:55:55.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:55.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:55.734892+0000) 2019-10-13T11:55:55.849 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:55.848+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:55.849269+0000) 2019-10-13T11:55:58.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:58.638+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.639302+0000) 2019-10-13T11:55:58.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:58.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.970024+0000) 2019-10-13T11:55:59.863 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:55:59.861+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.861998+0000) 2019-10-13T11:55:59.863 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:55:59.861+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.862312+0000) 2019-10-13T11:55:59.863 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:55:59.862+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.862533+0000) 2019-10-13T11:55:59.864 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:55:59.862+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.863453+0000) 2019-10-13T11:55:59.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:55:59.862+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.862789+0000) 2019-10-13T11:55:59.864 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:55:59.862+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.863156+0000) 2019-10-13T11:55:59.865 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:55:59.862+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.863383+0000) 2019-10-13T11:55:59.865 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:55:59.863+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:59.864577+0000) 2019-10-13T11:55:59.872 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:55:59.871+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220463 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:01.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:01.591+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:01.591447+0000) 2019-10-13T11:56:03.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:03.075+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.076703+0000) 2019-10-13T11:56:03.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:03.079+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.080588+0000) 2019-10-13T11:56:04.875 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:04.872+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.874199+0000) 2019-10-13T11:56:04.875 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:04.873+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.874520+0000) 2019-10-13T11:56:04.876 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:04.874+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.874856+0000) 2019-10-13T11:56:04.876 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:04.873+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.874718+0000) 2019-10-13T11:56:04.876 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:04.873+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.874927+0000) 2019-10-13T11:56:04.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:04.873+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.875260+0000) 2019-10-13T11:56:04.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:04.874+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.875432+0000) 2019-10-13T11:56:04.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:04.876+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.877493+0000) 2019-10-13T11:56:04.882 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:04.880+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:05.243 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:56:05.243 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:05.247 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:56:05.247 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:05.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:05.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:05.735042+0000) 2019-10-13T11:56:05.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:05.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:05.735042+0000) 2019-10-13T11:56:05.852 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:05.850+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:05.851563+0000) 2019-10-13T11:56:08.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:08.638+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.639474+0000) 2019-10-13T11:56:08.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:08.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.970229+0000) 2019-10-13T11:56:09.885 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:09.882+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.883501+0000) 2019-10-13T11:56:09.885 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:09.882+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.883647+0000) 2019-10-13T11:56:09.885 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:09.882+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.883915+0000) 2019-10-13T11:56:09.885 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:09.882+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.883966+0000) 2019-10-13T11:56:09.886 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:09.883+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.884501+0000) 2019-10-13T11:56:09.886 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:09.883+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.884647+0000) 2019-10-13T11:56:09.886 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:09.883+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.884570+0000) 2019-10-13T11:56:09.887 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:09.885+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:09.886359+0000) 2019-10-13T11:56:09.893 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:09.890+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220527 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:11.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:11.590+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:11.591625+0000) 2019-10-13T11:56:13.077 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:13.076+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.076853+0000) 2019-10-13T11:56:13.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:13.080+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.080749+0000) 2019-10-13T11:56:14.895 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:14.892+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.893818+0000) 2019-10-13T11:56:14.895 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:14.892+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.894040+0000) 2019-10-13T11:56:14.895 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:14.892+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.894206+0000) 2019-10-13T11:56:14.896 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:14.892+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.894259+0000) 2019-10-13T11:56:14.896 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:14.893+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.894452+0000) 2019-10-13T11:56:14.896 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:14.893+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.894562+0000) 2019-10-13T11:56:14.896 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:14.893+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.894649+0000) 2019-10-13T11:56:14.898 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:14.897+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.898413+0000) 2019-10-13T11:56:14.901 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:14.899+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:15.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:15.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:15.735200+0000) 2019-10-13T11:56:15.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:15.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:15.735200+0000) 2019-10-13T11:56:15.854 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:15.853+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:15.853761+0000) 2019-10-13T11:56:18.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:18.638+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.639620+0000) 2019-10-13T11:56:18.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:18.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.970434+0000) 2019-10-13T11:56:19.904 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:19.901+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.902960+0000) 2019-10-13T11:56:19.904 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:19.901+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.903308+0000) 2019-10-13T11:56:19.905 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:19.903+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.903813+0000) 2019-10-13T11:56:19.905 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:19.902+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.903519+0000) 2019-10-13T11:56:19.905 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:19.902+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.903680+0000) 2019-10-13T11:56:19.905 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:19.902+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.903855+0000) 2019-10-13T11:56:19.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:19.902+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.904382+0000) 2019-10-13T11:56:19.906 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:19.904+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:19.905511+0000) 2019-10-13T11:56:19.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:19.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:21.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:21.590+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:21.591847+0000) 2019-10-13T11:56:23.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:23.076+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.077035+0000) 2019-10-13T11:56:23.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:23.080+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.080909+0000) 2019-10-13T11:56:24.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:24.912+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.913467+0000) 2019-10-13T11:56:24.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:24.912+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.913517+0000) 2019-10-13T11:56:24.916 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:24.912+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.913748+0000) 2019-10-13T11:56:24.916 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:24.912+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.913944+0000) 2019-10-13T11:56:24.916 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:24.912+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.914199+0000) 2019-10-13T11:56:24.917 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:24.912+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.914355+0000) 2019-10-13T11:56:24.917 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:24.913+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.914708+0000) 2019-10-13T11:56:24.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:24.916+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.916940+0000) 2019-10-13T11:56:24.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:24.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:25.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:25.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:25.735375+0000) 2019-10-13T11:56:25.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:25.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:25.735369+0000) 2019-10-13T11:56:25.857 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:25.855+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:25.856159+0000) 2019-10-13T11:56:28.640 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:28.639+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.639780+0000) 2019-10-13T11:56:28.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:28.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.970606+0000) 2019-10-13T11:56:29.925 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:29.924+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.924799+0000) 2019-10-13T11:56:29.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:29.924+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.924695+0000) 2019-10-13T11:56:29.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:29.924+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.925256+0000) 2019-10-13T11:56:29.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:29.925+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.925402+0000) 2019-10-13T11:56:29.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:29.925+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.925538+0000) 2019-10-13T11:56:29.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:29.925+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.925661+0000) 2019-10-13T11:56:29.927 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:29.925+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.925831+0000) 2019-10-13T11:56:29.928 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:29.927+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:29.928103+0000) 2019-10-13T11:56:29.934 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:29.933+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220653 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:31.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:31.591+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:31.592045+0000) 2019-10-13T11:56:33.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:33.076+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.077171+0000) 2019-10-13T11:56:33.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:33.080+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.081059+0000) 2019-10-13T11:56:34.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:34.935+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.935506+0000) 2019-10-13T11:56:34.936 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:34.935+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.935690+0000) 2019-10-13T11:56:34.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:34.934+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.936672+0000) 2019-10-13T11:56:34.937 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:34.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.935982+0000) 2019-10-13T11:56:34.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:34.935+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.936200+0000) 2019-10-13T11:56:34.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:34.936+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.936520+0000) 2019-10-13T11:56:34.938 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:34.936+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.936797+0000) 2019-10-13T11:56:34.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:34.936+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.938539+0000) 2019-10-13T11:56:34.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:34.944+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220684 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:35.380 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:56:35.380 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:35.386 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:56:35.386 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:35.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:35.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:35.735496+0000) 2019-10-13T11:56:35.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:35.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:35.735577+0000) 2019-10-13T11:56:35.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:35.856+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:35.858605+0000) 2019-10-13T11:56:38.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:38.639+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.639946+0000) 2019-10-13T11:56:38.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:38.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.970799+0000) 2019-10-13T11:56:39.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:39.945+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.947025+0000) 2019-10-13T11:56:39.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:39.946+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.947070+0000) 2019-10-13T11:56:39.949 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:39.947+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.947438+0000) 2019-10-13T11:56:39.949 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:39.947+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.947562+0000) 2019-10-13T11:56:39.949 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:39.947+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.947814+0000) 2019-10-13T11:56:39.949 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:39.947+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.947930+0000) 2019-10-13T11:56:39.949 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:39.947+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.948149+0000) 2019-10-13T11:56:39.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:39.947+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:39.949618+0000) 2019-10-13T11:56:39.957 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:39.955+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220715 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:41.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:41.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:41.592271+0000) 2019-10-13T11:56:43.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:43.077+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.077335+0000) 2019-10-13T11:56:43.081 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:43.080+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.081248+0000) 2019-10-13T11:56:44.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:44.955+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.957499+0000) 2019-10-13T11:56:44.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:44.957+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.957783+0000) 2019-10-13T11:56:44.959 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:44.957+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.958353+0000) 2019-10-13T11:56:44.959 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:44.958+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.958479+0000) 2019-10-13T11:56:44.959 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:44.958+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.958868+0000) 2019-10-13T11:56:44.960 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:44.958+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.959200+0000) 2019-10-13T11:56:44.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:44.958+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.959274+0000) 2019-10-13T11:56:44.960 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:44.958+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.960203+0000) 2019-10-13T11:56:44.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:44.966+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220746 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:45.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:45.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:45.735692+0000) 2019-10-13T11:56:45.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:45.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:45.735727+0000) 2019-10-13T11:56:45.862 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:45.860+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:45.861627+0000) 2019-10-13T11:56:48.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:48.639+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.640116+0000) 2019-10-13T11:56:48.971 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:48.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.970991+0000) 2019-10-13T11:56:49.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:49.967+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.969192+0000) 2019-10-13T11:56:49.970 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:49.968+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.969472+0000) 2019-10-13T11:56:49.970 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:49.968+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.969507+0000) 2019-10-13T11:56:49.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:49.969+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.970589+0000) 2019-10-13T11:56:49.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:49.968+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.970015+0000) 2019-10-13T11:56:49.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:49.969+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.970608+0000) 2019-10-13T11:56:49.972 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:49.969+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.970842+0000) 2019-10-13T11:56:49.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:49.970+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:49.971311+0000) 2019-10-13T11:56:49.980 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:49.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220779 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:51.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:51.591+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:51.592459+0000) 2019-10-13T11:56:53.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:53.076+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.077548+0000) 2019-10-13T11:56:53.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:53.080+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.081433+0000) 2019-10-13T11:56:54.981 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:54.978+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.980052+0000) 2019-10-13T11:56:54.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:54.979+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.980508+0000) 2019-10-13T11:56:54.982 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:54.979+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.980697+0000) 2019-10-13T11:56:54.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:54.979+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.980803+0000) 2019-10-13T11:56:54.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:54.979+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.980914+0000) 2019-10-13T11:56:54.983 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:54.979+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.981349+0000) 2019-10-13T11:56:54.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:54.980+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.981358+0000) 2019-10-13T11:56:54.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:54.981+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.982672+0000) 2019-10-13T11:56:54.988 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:54.986+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220810 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:56:55.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:55.734+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:55.735834+0000) 2019-10-13T11:56:55.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:55.734+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:55.735883+0000) 2019-10-13T11:56:55.864 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:55.863+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:55.863961+0000) 2019-10-13T11:56:58.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:58.639+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.640305+0000) 2019-10-13T11:56:58.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:58.969+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.971182+0000) 2019-10-13T11:56:59.991 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:56:59.988+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.989510+0000) 2019-10-13T11:56:59.991 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:56:59.988+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.989652+0000) 2019-10-13T11:56:59.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:56:59.988+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.989765+0000) 2019-10-13T11:56:59.991 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:56:59.988+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.989930+0000) 2019-10-13T11:56:59.992 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:56:59.988+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.990270+0000) 2019-10-13T11:56:59.992 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:56:59.989+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.990562+0000) 2019-10-13T11:56:59.992 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:56:59.989+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.990366+0000) 2019-10-13T11:56:59.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:56:59.992+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:59.993138+0000) 2019-10-13T11:57:00.001 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:56:59.997+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220841 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:01.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:01.591+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:01.592585+0000) 2019-10-13T11:57:03.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:03.077+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.077706+0000) 2019-10-13T11:57:03.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:03.080+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.081570+0000) 2019-10-13T11:57:05.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:04.999+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.000868+0000) 2019-10-13T11:57:05.002 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:04.999+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.000690+0000) 2019-10-13T11:57:05.002 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:04.999+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.001215+0000) 2019-10-13T11:57:05.003 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:05.000+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.001579+0000) 2019-10-13T11:57:05.003 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:05.000+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.001805+0000) 2019-10-13T11:57:05.003 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:05.000+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.001876+0000) 2019-10-13T11:57:05.003 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:05.000+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.002097+0000) 2019-10-13T11:57:05.004 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:05.003+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.003970+0000) 2019-10-13T11:57:05.010 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:05.008+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220872 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:05.572 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:57:05.573 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:05.577 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:57:05.577 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:05.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:05.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.736012+0000) 2019-10-13T11:57:05.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:05.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.736019+0000) 2019-10-13T11:57:05.867 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:05.866+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:05.866410+0000) 2019-10-13T11:57:08.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:08.639+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.640488+0000) 2019-10-13T11:57:08.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:08.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.971355+0000) 2019-10-13T11:57:10.012 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:10.010+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.011701+0000) 2019-10-13T11:57:10.013 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:10.010+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.011762+0000) 2019-10-13T11:57:10.013 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:10.010+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.011866+0000) 2019-10-13T11:57:10.014 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:10.011+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.012634+0000) 2019-10-13T11:57:10.014 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:10.011+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.012426+0000) 2019-10-13T11:57:10.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:10.010+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.012173+0000) 2019-10-13T11:57:10.014 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:10.011+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.012805+0000) 2019-10-13T11:57:10.015 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:10.013+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:10.014594+0000) 2019-10-13T11:57:10.021 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:10.019+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220903 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:11.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:11.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:11.592738+0000) 2019-10-13T11:57:13.078 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:13.077+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.077874+0000) 2019-10-13T11:57:13.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:13.081+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.081706+0000) 2019-10-13T11:57:15.024 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:15.022+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.023382+0000) 2019-10-13T11:57:15.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:15.022+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.022933+0000) 2019-10-13T11:57:15.025 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:15.022+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.023389+0000) 2019-10-13T11:57:15.025 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:15.023+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.023526+0000) 2019-10-13T11:57:15.025 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:15.023+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.023645+0000) 2019-10-13T11:57:15.026 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:15.023+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.023938+0000) 2019-10-13T11:57:15.026 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:15.023+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.024225+0000) 2019-10-13T11:57:15.026 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:15.025+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.026397+0000) 2019-10-13T11:57:15.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:15.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220934 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:15.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:15.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.736176+0000) 2019-10-13T11:57:15.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:15.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.736185+0000) 2019-10-13T11:57:15.869 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:15.867+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:15.868543+0000) 2019-10-13T11:57:18.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:18.640+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.640640+0000) 2019-10-13T11:57:18.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:18.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.971509+0000) 2019-10-13T11:57:20.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:20.033+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.034223+0000) 2019-10-13T11:57:20.036 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:20.033+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.034120+0000) 2019-10-13T11:57:20.036 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:20.033+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.034375+0000) 2019-10-13T11:57:20.036 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:20.034+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.034620+0000) 2019-10-13T11:57:20.036 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:20.034+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.034816+0000) 2019-10-13T11:57:20.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:20.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.035412+0000) 2019-10-13T11:57:20.037 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:20.035+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.036362+0000) 2019-10-13T11:57:20.037 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:20.034+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:20.035270+0000) 2019-10-13T11:57:20.044 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:20.042+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220963 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:21.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:21.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:21.592874+0000) 2019-10-13T11:57:23.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:23.077+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.078051+0000) 2019-10-13T11:57:23.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:23.080+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.081896+0000) 2019-10-13T11:57:25.046 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:25.044+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.044829+0000) 2019-10-13T11:57:25.046 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:25.044+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.044888+0000) 2019-10-13T11:57:25.046 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:25.044+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.044948+0000) 2019-10-13T11:57:25.047 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:25.043+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.045372+0000) 2019-10-13T11:57:25.047 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:25.045+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.045483+0000) 2019-10-13T11:57:25.047 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:25.044+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.045316+0000) 2019-10-13T11:57:25.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:25.045+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.046041+0000) 2019-10-13T11:57:25.048 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:25.046+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.048214+0000) 2019-10-13T11:57:25.055 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:25.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 220994 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:25.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:25.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.736369+0000) 2019-10-13T11:57:25.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:25.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.736371+0000) 2019-10-13T11:57:25.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:25.869+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:25.870980+0000) 2019-10-13T11:57:28.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:28.640+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.640810+0000) 2019-10-13T11:57:28.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:28.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.971661+0000) 2019-10-13T11:57:30.057 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:30.055+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.056368+0000) 2019-10-13T11:57:30.057 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:30.056+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.056514+0000) 2019-10-13T11:57:30.057 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:30.056+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.056725+0000) 2019-10-13T11:57:30.058 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:30.055+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.057495+0000) 2019-10-13T11:57:30.058 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:30.056+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.056981+0000) 2019-10-13T11:57:30.059 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:30.056+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.057128+0000) 2019-10-13T11:57:30.059 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:30.056+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.057347+0000) 2019-10-13T11:57:30.059 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:30.056+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:30.058582+0000) 2019-10-13T11:57:30.066 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:30.065+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221025 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:31.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:31.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:31.593042+0000) 2019-10-13T11:57:33.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:33.077+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:33.078213+0000) 2019-10-13T11:57:33.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:33.081+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:33.082076+0000) 2019-10-13T11:57:35.068 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:35.066+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.067429+0000) 2019-10-13T11:57:35.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:35.066+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.067576+0000) 2019-10-13T11:57:35.069 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:35.066+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.067772+0000) 2019-10-13T11:57:35.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:35.066+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.068260+0000) 2019-10-13T11:57:35.070 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:35.066+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.068243+0000) 2019-10-13T11:57:35.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:35.066+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.068347+0000) 2019-10-13T11:57:35.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:35.067+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.068545+0000) 2019-10-13T11:57:35.071 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:35.069+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.070852+0000) 2019-10-13T11:57:35.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:35.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221056 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:35.679 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:57:35.680 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:35.684 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:57:35.684 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:35.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:35.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.736496+0000) 2019-10-13T11:57:35.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:35.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.736535+0000) 2019-10-13T11:57:35.874 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:35.872+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:35.873492+0000) 2019-10-13T11:57:38.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:38.640+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.640965+0000) 2019-10-13T11:57:38.972 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:38.970+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.971824+0000) 2019-10-13T11:57:40.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:40.076+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.078308+0000) 2019-10-13T11:57:40.079 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:40.077+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.078665+0000) 2019-10-13T11:57:40.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:40.078+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.079722+0000) 2019-10-13T11:57:40.080 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:40.077+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.078929+0000) 2019-10-13T11:57:40.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:40.077+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.079105+0000) 2019-10-13T11:57:40.081 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:40.077+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.079284+0000) 2019-10-13T11:57:40.081 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:40.078+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.079568+0000) 2019-10-13T11:57:40.081 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:40.080+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:40.080762+0000) 2019-10-13T11:57:40.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:40.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221085 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:41.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:41.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:41.593218+0000) 2019-10-13T11:57:43.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:43.077+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:43.078428+0000) 2019-10-13T11:57:43.082 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:43.081+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:43.082260+0000) 2019-10-13T11:57:45.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:45.087+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.089292+0000) 2019-10-13T11:57:45.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:45.088+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.089686+0000) 2019-10-13T11:57:45.091 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:45.088+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.089775+0000) 2019-10-13T11:57:45.091 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:45.088+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.090203+0000) 2019-10-13T11:57:45.092 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:45.089+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.090467+0000) 2019-10-13T11:57:45.092 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:45.089+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.090632+0000) 2019-10-13T11:57:45.092 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:45.089+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.090675+0000) 2019-10-13T11:57:45.092 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:45.091+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.091957+0000) 2019-10-13T11:57:45.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:45.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:45.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:45.735+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.736653+0000) 2019-10-13T11:57:45.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:45.735+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.736679+0000) 2019-10-13T11:57:45.876 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:45.875+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:45.875845+0000) 2019-10-13T11:57:48.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:48.639+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.641190+0000) 2019-10-13T11:57:48.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:48.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.972029+0000) 2019-10-13T11:57:50.101 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:50.099+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.100673+0000) 2019-10-13T11:57:50.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:50.099+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.100862+0000) 2019-10-13T11:57:50.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:50.099+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.101047+0000) 2019-10-13T11:57:50.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:50.101+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.101833+0000) 2019-10-13T11:57:50.103 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:50.100+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.101395+0000) 2019-10-13T11:57:50.103 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:50.100+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.101920+0000) 2019-10-13T11:57:50.103 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:50.100+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.102116+0000) 2019-10-13T11:57:50.104 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:50.103+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:50.103855+0000) 2019-10-13T11:57:50.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:50.108+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221147 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:51.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:51.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:51.593427+0000) 2019-10-13T11:57:53.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:53.078+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:53.078578+0000) 2019-10-13T11:57:53.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:53.082+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:53.082461+0000) 2019-10-13T11:57:55.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:57:55.111+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.111744+0000) 2019-10-13T11:57:55.114 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:55.110+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.111887+0000) 2019-10-13T11:57:55.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:55.111+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.112424+0000) 2019-10-13T11:57:55.114 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:57:55.111+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.112672+0000) 2019-10-13T11:57:55.114 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:57:55.111+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.112836+0000) 2019-10-13T11:57:55.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:55.111+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.113015+0000) 2019-10-13T11:57:55.115 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:55.111+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.113191+0000) 2019-10-13T11:57:55.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:55.113+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.113818+0000) 2019-10-13T11:57:55.122 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:57:55.119+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:57:55.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:57:55.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.736828+0000) 2019-10-13T11:57:55.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:57:55.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.736845+0000) 2019-10-13T11:57:55.882 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:57:55.877+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:55.878242+0000) 2019-10-13T11:57:58.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:57:58.640+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.641401+0000) 2019-10-13T11:57:58.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:57:58.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.972244+0000) 2019-10-13T11:58:00.123 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:00.122+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.123138+0000) 2019-10-13T11:58:00.124 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:00.122+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.123081+0000) 2019-10-13T11:58:00.125 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:00.122+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.123241+0000) 2019-10-13T11:58:00.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:00.122+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.123353+0000) 2019-10-13T11:58:00.125 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:00.123+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.123885+0000) 2019-10-13T11:58:00.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:00.123+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.123988+0000) 2019-10-13T11:58:00.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:00.123+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.124226+0000) 2019-10-13T11:58:00.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:00.125+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:00.126435+0000) 2019-10-13T11:58:00.133 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:00.132+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:01.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:01.593+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:01.593595+0000) 2019-10-13T11:58:03.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:03.078+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:03.078721+0000) 2019-10-13T11:58:03.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:03.082+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:03.082650+0000) 2019-10-13T11:58:05.136 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:05.134+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.135228+0000) 2019-10-13T11:58:05.136 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:05.135+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.135827+0000) 2019-10-13T11:58:05.137 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:05.135+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.136117+0000) 2019-10-13T11:58:05.137 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:05.135+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.136311+0000) 2019-10-13T11:58:05.138 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:05.135+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.136383+0000) 2019-10-13T11:58:05.138 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:05.135+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.136233+0000) 2019-10-13T11:58:05.138 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:05.136+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.136508+0000) 2019-10-13T11:58:05.139 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:05.138+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.139147+0000) 2019-10-13T11:58:05.144 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:05.143+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:05.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:05.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.737009+0000) 2019-10-13T11:58:05.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:05.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.737014+0000) 2019-10-13T11:58:05.839 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:58:05.839 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:05.844 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:58:05.844 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:05.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:05.879+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:05.880444+0000) 2019-10-13T11:58:08.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:08.641+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.641561+0000) 2019-10-13T11:58:08.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:08.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.972342+0000) 2019-10-13T11:58:10.147 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:10.145+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.145786+0000) 2019-10-13T11:58:10.147 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:10.145+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.145915+0000) 2019-10-13T11:58:10.147 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:10.145+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.146092+0000) 2019-10-13T11:58:10.147 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:10.145+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.146328+0000) 2019-10-13T11:58:10.148 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:10.145+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.146442+0000) 2019-10-13T11:58:10.148 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:10.146+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.146558+0000) 2019-10-13T11:58:10.148 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:10.146+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.147051+0000) 2019-10-13T11:58:10.149 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:10.148+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:10.149185+0000) 2019-10-13T11:58:10.156 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:10.154+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:11.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:11.593+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:11.593780+0000) 2019-10-13T11:58:13.079 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:13.078+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:13.078876+0000) 2019-10-13T11:58:13.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:13.081+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:13.082885+0000) 2019-10-13T11:58:15.157 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:15.155+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.156765+0000) 2019-10-13T11:58:15.158 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:15.156+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.156729+0000) 2019-10-13T11:58:15.158 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:15.156+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.157297+0000) 2019-10-13T11:58:15.158 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:15.157+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.157462+0000) 2019-10-13T11:58:15.158 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:15.157+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.157673+0000) 2019-10-13T11:58:15.159 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:15.157+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.157890+0000) 2019-10-13T11:58:15.159 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:15.157+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.158035+0000) 2019-10-13T11:58:15.160 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:15.157+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.159557+0000) 2019-10-13T11:58:15.166 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:15.165+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:15.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:15.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.737190+0000) 2019-10-13T11:58:15.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:15.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.737190+0000) 2019-10-13T11:58:15.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:15.881+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:15.882824+0000) 2019-10-13T11:58:18.642 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:18.640+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.641757+0000) 2019-10-13T11:58:18.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:18.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.972477+0000) 2019-10-13T11:58:20.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:20.166+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.167475+0000) 2019-10-13T11:58:20.169 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:20.166+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.167691+0000) 2019-10-13T11:58:20.169 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:20.166+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.167945+0000) 2019-10-13T11:58:20.170 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:20.166+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.168487+0000) 2019-10-13T11:58:20.170 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:20.167+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.168500+0000) 2019-10-13T11:58:20.170 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:20.167+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.168614+0000) 2019-10-13T11:58:20.170 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:20.167+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.168909+0000) 2019-10-13T11:58:20.171 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:20.168+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:20.170696+0000) 2019-10-13T11:58:20.178 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:20.175+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:21.594 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:21.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:21.593938+0000) 2019-10-13T11:58:23.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:23.077+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:23.079078+0000) 2019-10-13T11:58:23.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:23.081+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:23.083033+0000) 2019-10-13T11:58:25.179 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:25.177+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.178402+0000) 2019-10-13T11:58:25.179 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:25.177+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.178516+0000) 2019-10-13T11:58:25.180 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:25.177+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.179635+0000) 2019-10-13T11:58:25.181 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:25.177+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.179151+0000) 2019-10-13T11:58:25.181 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:25.177+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.179216+0000) 2019-10-13T11:58:25.181 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:25.178+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.179757+0000) 2019-10-13T11:58:25.181 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:25.178+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.179515+0000) 2019-10-13T11:58:25.181 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:25.179+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.181135+0000) 2019-10-13T11:58:25.187 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:25.185+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:25.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:25.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.737332+0000) 2019-10-13T11:58:25.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:25.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.737334+0000) 2019-10-13T11:58:25.885 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:25.884+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:25.885135+0000) 2019-10-13T11:58:28.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:28.641+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.641952+0000) 2019-10-13T11:58:28.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:28.971+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.972685+0000) 2019-10-13T11:58:30.189 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:30.187+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.188512+0000) 2019-10-13T11:58:30.190 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:30.188+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.189107+0000) 2019-10-13T11:58:30.190 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:30.187+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.189252+0000) 2019-10-13T11:58:30.191 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:30.187+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.189142+0000) 2019-10-13T11:58:30.191 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:30.188+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.189512+0000) 2019-10-13T11:58:30.191 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:30.188+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.189665+0000) 2019-10-13T11:58:30.191 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:30.188+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.189925+0000) 2019-10-13T11:58:30.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:30.190+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:30.191698+0000) 2019-10-13T11:58:30.198 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:30.196+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221387 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:31.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:31.592+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:31.594117+0000) 2019-10-13T11:58:33.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:33.078+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.079232+0000) 2019-10-13T11:58:33.083 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:33.082+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.083228+0000) 2019-10-13T11:58:35.200 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:35.199+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.199888+0000) 2019-10-13T11:58:35.201 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:35.198+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.199838+0000) 2019-10-13T11:58:35.201 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:35.198+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.200020+0000) 2019-10-13T11:58:35.201 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:35.198+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.200178+0000) 2019-10-13T11:58:35.202 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:35.199+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.200466+0000) 2019-10-13T11:58:35.202 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:35.199+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.200910+0000) 2019-10-13T11:58:35.202 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:35.199+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.200842+0000) 2019-10-13T11:58:35.204 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:35.202+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.203474+0000) 2019-10-13T11:58:35.211 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:35.207+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221418 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:35.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:35.736+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.737513+0000) 2019-10-13T11:58:35.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:35.736+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.737514+0000) 2019-10-13T11:58:35.888 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:35.886+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:35.887557+0000) 2019-10-13T11:58:35.986 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:58:35.986 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:35.990 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:58:35.990 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:38.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:38.641+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.642141+0000) 2019-10-13T11:58:38.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:38.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.972878+0000) 2019-10-13T11:58:40.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:40.209+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.210740+0000) 2019-10-13T11:58:40.212 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:40.209+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.211010+0000) 2019-10-13T11:58:40.212 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:40.209+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.211162+0000) 2019-10-13T11:58:40.213 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:40.211+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.212338+0000) 2019-10-13T11:58:40.213 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:40.210+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.211517+0000) 2019-10-13T11:58:40.213 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:40.210+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.211775+0000) 2019-10-13T11:58:40.214 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:40.210+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.211858+0000) 2019-10-13T11:58:40.215 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:40.213+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:40.214536+0000) 2019-10-13T11:58:40.221 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:40.218+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221446 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:41.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:41.593+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:41.594312+0000) 2019-10-13T11:58:43.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:43.078+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.079429+0000) 2019-10-13T11:58:43.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:43.082+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.083403+0000) 2019-10-13T11:58:45.222 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:45.220+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.221617+0000) 2019-10-13T11:58:45.223 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:45.220+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.222034+0000) 2019-10-13T11:58:45.223 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:45.220+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.222112+0000) 2019-10-13T11:58:45.223 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:45.220+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.222289+0000) 2019-10-13T11:58:45.224 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:45.221+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.222704+0000) 2019-10-13T11:58:45.224 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:45.221+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.222710+0000) 2019-10-13T11:58:45.224 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:45.221+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.222986+0000) 2019-10-13T11:58:45.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:45.224+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.225166+0000) 2019-10-13T11:58:45.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:45.229+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221477 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:45.738 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:45.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.737685+0000) 2019-10-13T11:58:45.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:45.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.737688+0000) 2019-10-13T11:58:45.890 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:45.889+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:45.889904+0000) 2019-10-13T11:58:48.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:48.641+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.642332+0000) 2019-10-13T11:58:48.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:48.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.973067+0000) 2019-10-13T11:58:50.234 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:50.232+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.232934+0000) 2019-10-13T11:58:50.234 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:50.233+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.233457+0000) 2019-10-13T11:58:50.235 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:50.233+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.234304+0000) 2019-10-13T11:58:50.235 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:50.233+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.233559+0000) 2019-10-13T11:58:50.235 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:50.233+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.233827+0000) 2019-10-13T11:58:50.236 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:50.233+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.234230+0000) 2019-10-13T11:58:50.236 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:50.234+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.234549+0000) 2019-10-13T11:58:50.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:50.235+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:50.236418+0000) 2019-10-13T11:58:50.243 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:50.242+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221507 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:51.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:51.594+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:51.594505+0000) 2019-10-13T11:58:53.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:53.079+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.079620+0000) 2019-10-13T11:58:53.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:53.083+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.083615+0000) 2019-10-13T11:58:55.246 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:55.245+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.245413+0000) 2019-10-13T11:58:55.247 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:58:55.245+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.245820+0000) 2019-10-13T11:58:55.247 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:55.245+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.245914+0000) 2019-10-13T11:58:55.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:58:55.245+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.246196+0000) 2019-10-13T11:58:55.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:55.245+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.246299+0000) 2019-10-13T11:58:55.248 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:58:55.246+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.246499+0000) 2019-10-13T11:58:55.248 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:55.246+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.246844+0000) 2019-10-13T11:58:55.249 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:55.248+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.249050+0000) 2019-10-13T11:58:55.256 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:58:55.253+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221538 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:58:55.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:58:55.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.737874+0000) 2019-10-13T11:58:55.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:58:55.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.737873+0000) 2019-10-13T11:58:55.893 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:58:55.892+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:55.892313+0000) 2019-10-13T11:58:58.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:58:58.642+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.642513+0000) 2019-10-13T11:58:58.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:58:58.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.973271+0000) 2019-10-13T11:59:00.256 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:00.254+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.255389+0000) 2019-10-13T11:59:00.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:00.255+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.255744+0000) 2019-10-13T11:59:00.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:00.255+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.256533+0000) 2019-10-13T11:59:00.258 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:00.255+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.256371+0000) 2019-10-13T11:59:00.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:00.255+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.256276+0000) 2019-10-13T11:59:00.259 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:00.256+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.256569+0000) 2019-10-13T11:59:00.259 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:00.256+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.256673+0000) 2019-10-13T11:59:00.260 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:00.257+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:00.258660+0000) 2019-10-13T11:59:00.265 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:00.264+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221566 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:01.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:01.594+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:01.594724+0000) 2019-10-13T11:59:03.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:03.079+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.079741+0000) 2019-10-13T11:59:03.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:03.082+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.083794+0000) 2019-10-13T11:59:05.267 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:05.265+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.266786+0000) 2019-10-13T11:59:05.268 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:05.265+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.266989+0000) 2019-10-13T11:59:05.268 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:05.265+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.267222+0000) 2019-10-13T11:59:05.268 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:05.266+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.267740+0000) 2019-10-13T11:59:05.269 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:05.266+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.267898+0000) 2019-10-13T11:59:05.269 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:05.266+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.268079+0000) 2019-10-13T11:59:05.269 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:05.266+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.267831+0000) 2019-10-13T11:59:05.270 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:05.267+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.269698+0000) 2019-10-13T11:59:05.276 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:05.274+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221597 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:05.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:05.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.738044+0000) 2019-10-13T11:59:05.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:05.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.738064+0000) 2019-10-13T11:59:05.895 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:05.893+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:05.894698+0000) 2019-10-13T11:59:06.148 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:59:06.148 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:06.152 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:59:06.152 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:08.643 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:08.641+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.642708+0000) 2019-10-13T11:59:08.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:08.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.973451+0000) 2019-10-13T11:59:10.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:10.275+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.277728+0000) 2019-10-13T11:59:10.279 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:10.276+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.277641+0000) 2019-10-13T11:59:10.279 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:10.276+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.277785+0000) 2019-10-13T11:59:10.280 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:10.276+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.277919+0000) 2019-10-13T11:59:10.280 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:10.276+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.278318+0000) 2019-10-13T11:59:10.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:10.277+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.278629+0000) 2019-10-13T11:59:10.280 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:10.277+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.278680+0000) 2019-10-13T11:59:10.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:10.279+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:10.281414+0000) 2019-10-13T11:59:10.287 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:10.285+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221627 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:11.595 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:11.594+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:11.594910+0000) 2019-10-13T11:59:13.080 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:13.079+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.079909+0000) 2019-10-13T11:59:13.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:13.082+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.083952+0000) 2019-10-13T11:59:15.290 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:15.287+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.288727+0000) 2019-10-13T11:59:15.290 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:15.287+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.288884+0000) 2019-10-13T11:59:15.290 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:15.287+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.289021+0000) 2019-10-13T11:59:15.290 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:15.287+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.289187+0000) 2019-10-13T11:59:15.291 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:15.287+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.289636+0000) 2019-10-13T11:59:15.291 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:15.288+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.290064+0000) 2019-10-13T11:59:15.291 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:15.289+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.290628+0000) 2019-10-13T11:59:15.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:15.290+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.292023+0000) 2019-10-13T11:59:15.298 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:15.296+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221658 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:15.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:15.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.738221+0000) 2019-10-13T11:59:15.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:15.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.738221+0000) 2019-10-13T11:59:15.897 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:15.895+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:15.896693+0000) 2019-10-13T11:59:18.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:18.642+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.642900+0000) 2019-10-13T11:59:18.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:18.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.973620+0000) 2019-10-13T11:59:20.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:20.297+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.299246+0000) 2019-10-13T11:59:20.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:20.298+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.299540+0000) 2019-10-13T11:59:20.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:20.298+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.299775+0000) 2019-10-13T11:59:20.301 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:20.299+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.300224+0000) 2019-10-13T11:59:20.302 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:20.298+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.299703+0000) 2019-10-13T11:59:20.302 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:20.298+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.300273+0000) 2019-10-13T11:59:20.302 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:20.298+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.300357+0000) 2019-10-13T11:59:20.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:20.301+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:20.302436+0000) 2019-10-13T11:59:20.307 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:20.305+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221686 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:21.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:21.593+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:21.595086+0000) 2019-10-13T11:59:23.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:23.078+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.080093+0000) 2019-10-13T11:59:23.084 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:23.083+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.084127+0000) 2019-10-13T11:59:25.309 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:25.307+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.308494+0000) 2019-10-13T11:59:25.309 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:25.307+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.308679+0000) 2019-10-13T11:59:25.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:25.308+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.309332+0000) 2019-10-13T11:59:25.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:25.307+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.308959+0000) 2019-10-13T11:59:25.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:25.307+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.309388+0000) 2019-10-13T11:59:25.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:25.308+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.309761+0000) 2019-10-13T11:59:25.311 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:25.308+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.310189+0000) 2019-10-13T11:59:25.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:25.310+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.311583+0000) 2019-10-13T11:59:25.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:25.316+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221717 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:25.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:25.737+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.738427+0000) 2019-10-13T11:59:25.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:25.737+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.738428+0000) 2019-10-13T11:59:25.899 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:25.898+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:25.898892+0000) 2019-10-13T11:59:28.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:28.642+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.643088+0000) 2019-10-13T11:59:28.974 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:28.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.973846+0000) 2019-10-13T11:59:30.320 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:30.318+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.319501+0000) 2019-10-13T11:59:30.321 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:30.318+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.319764+0000) 2019-10-13T11:59:30.321 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:30.318+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.319980+0000) 2019-10-13T11:59:30.322 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:30.319+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.320029+0000) 2019-10-13T11:59:30.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:30.319+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.320529+0000) 2019-10-13T11:59:30.322 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:30.319+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.320693+0000) 2019-10-13T11:59:30.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:30.319+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.321049+0000) 2019-10-13T11:59:30.323 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:30.321+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:30.322299+0000) 2019-10-13T11:59:30.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:30.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221747 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:31.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:31.594+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:31.595245+0000) 2019-10-13T11:59:33.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:33.079+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.080309+0000) 2019-10-13T11:59:33.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:33.083+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.084309+0000) 2019-10-13T11:59:35.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:35.330+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.330682+0000) 2019-10-13T11:59:35.332 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:35.330+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.330766+0000) 2019-10-13T11:59:35.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:35.330+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.330929+0000) 2019-10-13T11:59:35.332 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:35.330+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.331269+0000) 2019-10-13T11:59:35.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:35.331+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.331871+0000) 2019-10-13T11:59:35.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:35.331+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.332700+0000) 2019-10-13T11:59:35.333 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:35.331+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.331454+0000) 2019-10-13T11:59:35.334 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:35.331+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.331751+0000) 2019-10-13T11:59:35.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:35.339+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221778 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:35.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:35.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.738599+0000) 2019-10-13T11:59:35.739 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:35.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.738598+0000) 2019-10-13T11:59:35.902 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:35.901+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:35.901410+0000) 2019-10-13T11:59:36.302 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T11:59:36.303 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:36.307 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T11:59:36.308 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:38.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:38.642+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.643271+0000) 2019-10-13T11:59:38.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:38.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.974050+0000) 2019-10-13T11:59:40.342 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:40.340+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.341176+0000) 2019-10-13T11:59:40.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:40.340+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.341346+0000) 2019-10-13T11:59:40.343 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:40.341+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.341775+0000) 2019-10-13T11:59:40.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:40.341+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.342157+0000) 2019-10-13T11:59:40.343 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:40.341+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.342191+0000) 2019-10-13T11:59:40.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:40.342+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.342499+0000) 2019-10-13T11:59:40.344 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:40.342+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.342901+0000) 2019-10-13T11:59:40.344 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:40.343+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:40.344208+0000) 2019-10-13T11:59:40.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:40.350+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221806 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:41.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:41.595+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:41.595472+0000) 2019-10-13T11:59:43.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:43.080+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.080495+0000) 2019-10-13T11:59:43.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:43.084+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.084498+0000) 2019-10-13T11:59:45.353 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:45.352+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.352492+0000) 2019-10-13T11:59:45.354 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:45.352+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.352871+0000) 2019-10-13T11:59:45.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:45.352+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.353110+0000) 2019-10-13T11:59:45.355 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:45.353+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.353802+0000) 2019-10-13T11:59:45.355 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:45.353+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.353744+0000) 2019-10-13T11:59:45.355 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:45.353+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.353866+0000) 2019-10-13T11:59:45.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:45.353+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.354287+0000) 2019-10-13T11:59:45.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:45.354+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.355155+0000) 2019-10-13T11:59:45.362 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:45.361+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221837 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:45.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:45.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.738726+0000) 2019-10-13T11:59:45.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:45.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.738764+0000) 2019-10-13T11:59:45.905 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:45.904+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:45.904422+0000) 2019-10-13T11:59:48.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:48.642+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.643455+0000) 2019-10-13T11:59:48.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:48.972+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.974237+0000) 2019-10-13T11:59:50.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:50.362+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.363629+0000) 2019-10-13T11:59:50.365 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:50.362+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.363958+0000) 2019-10-13T11:59:50.366 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:50.364+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.364744+0000) 2019-10-13T11:59:50.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:50.362+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.364210+0000) 2019-10-13T11:59:50.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:50.362+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.364315+0000) 2019-10-13T11:59:50.366 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:50.363+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.364805+0000) 2019-10-13T11:59:50.367 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:50.363+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.364680+0000) 2019-10-13T11:59:50.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:50.365+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:50.366093+0000) 2019-10-13T11:59:50.373 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:50.371+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221867 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:51.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:51.594+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:51.595663+0000) 2019-10-13T11:59:53.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:53.079+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.080639+0000) 2019-10-13T11:59:53.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:53.083+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.084644+0000) 2019-10-13T11:59:55.375 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:55.373+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.374853+0000) 2019-10-13T11:59:55.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:55.373+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.374848+0000) 2019-10-13T11:59:55.377 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T11:59:55.373+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.375248+0000) 2019-10-13T11:59:55.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:55.374+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.375440+0000) 2019-10-13T11:59:55.377 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T11:59:55.374+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.375672+0000) 2019-10-13T11:59:55.377 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:55.374+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.375847+0000) 2019-10-13T11:59:55.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T11:59:55.374+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.376385+0000) 2019-10-13T11:59:55.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:55.376+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.378327+0000) 2019-10-13T11:59:55.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T11:59:55.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T11:59:55.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T11:59:55.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.738866+0000) 2019-10-13T11:59:55.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T11:59:55.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.738905+0000) 2019-10-13T11:59:55.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T11:59:55.905+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:55.906774+0000) 2019-10-13T11:59:58.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T11:59:58.642+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.643618+0000) 2019-10-13T11:59:58.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T11:59:58.973+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.974454+0000) 2019-10-13T12:00:00.386 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:00.383+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.385512+0000) 2019-10-13T12:00:00.387 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:00.384+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.385606+0000) 2019-10-13T12:00:00.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:00.384+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.386078+0000) 2019-10-13T12:00:00.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:00.384+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.386192+0000) 2019-10-13T12:00:00.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:00.385+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.386412+0000) 2019-10-13T12:00:00.388 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:00.385+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.386649+0000) 2019-10-13T12:00:00.388 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:00.385+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.386922+0000) 2019-10-13T12:00:00.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:00.386+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:00.388260+0000) 2019-10-13T12:00:00.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:00.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:01.596 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:01.595+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:01.595835+0000) 2019-10-13T12:00:03.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:03.080+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.080845+0000) 2019-10-13T12:00:03.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:03.083+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.084827+0000) 2019-10-13T12:00:05.405 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:05.395+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.396941+0000) 2019-10-13T12:00:05.406 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:05.395+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.397239+0000) 2019-10-13T12:00:05.406 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:05.395+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.397357+0000) 2019-10-13T12:00:05.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:05.396+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.397522+0000) 2019-10-13T12:00:05.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:05.396+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.397707+0000) 2019-10-13T12:00:05.408 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:05.396+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.397790+0000) 2019-10-13T12:00:05.408 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:05.396+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.397936+0000) 2019-10-13T12:00:05.409 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:05.397+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.399129+0000) 2019-10-13T12:00:05.414 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:05.404+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221959 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:05.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:05.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.739050+0000) 2019-10-13T12:00:05.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:05.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.739053+0000) 2019-10-13T12:00:05.909 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:05.908+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:05.909047+0000) 2019-10-13T12:00:06.434 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:00:06.434 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:06.439 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:00:06.439 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:08.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:08.643+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.643783+0000) 2019-10-13T12:00:08.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:08.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.974644+0000) 2019-10-13T12:00:10.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:10.406+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.407375+0000) 2019-10-13T12:00:10.408 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:10.405+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.407026+0000) 2019-10-13T12:00:10.409 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:10.405+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.407270+0000) 2019-10-13T12:00:10.409 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:10.406+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.407669+0000) 2019-10-13T12:00:10.409 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:10.406+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.407937+0000) 2019-10-13T12:00:10.410 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:10.407+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.408462+0000) 2019-10-13T12:00:10.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:10.407+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.408659+0000) 2019-10-13T12:00:10.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:10.409+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:10.410594+0000) 2019-10-13T12:00:10.417 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:10.414+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 221987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:11.597 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:11.595+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:11.596016+0000) 2019-10-13T12:00:13.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:13.080+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.080997+0000) 2019-10-13T12:00:13.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:13.084+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.084924+0000) 2019-10-13T12:00:15.418 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:15.417+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.418181+0000) 2019-10-13T12:00:15.419 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:15.416+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.418058+0000) 2019-10-13T12:00:15.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:15.416+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.418191+0000) 2019-10-13T12:00:15.420 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:15.417+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.418512+0000) 2019-10-13T12:00:15.420 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:15.417+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.418517+0000) 2019-10-13T12:00:15.420 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:15.417+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.418751+0000) 2019-10-13T12:00:15.420 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:15.417+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.419135+0000) 2019-10-13T12:00:15.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:15.420+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.421237+0000) 2019-10-13T12:00:15.426 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:15.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:15.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:15.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.739192+0000) 2019-10-13T12:00:15.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:15.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.739193+0000) 2019-10-13T12:00:15.912 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:15.911+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:15.911629+0000) 2019-10-13T12:00:18.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:18.643+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.643930+0000) 2019-10-13T12:00:18.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:18.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.974845+0000) 2019-10-13T12:00:20.428 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:20.426+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.427482+0000) 2019-10-13T12:00:20.429 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:20.426+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.427368+0000) 2019-10-13T12:00:20.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:20.427+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.427606+0000) 2019-10-13T12:00:20.429 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:20.427+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.427875+0000) 2019-10-13T12:00:20.429 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:20.427+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.427955+0000) 2019-10-13T12:00:20.430 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:20.427+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.427696+0000) 2019-10-13T12:00:20.430 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:20.427+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.428114+0000) 2019-10-13T12:00:20.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:20.428+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:20.429408+0000) 2019-10-13T12:00:20.437 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:20.435+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222046 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:21.597 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:21.595+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:21.596207+0000) 2019-10-13T12:00:23.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:23.080+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.081164+0000) 2019-10-13T12:00:23.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:23.084+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.085099+0000) 2019-10-13T12:00:25.439 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:25.437+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.438242+0000) 2019-10-13T12:00:25.440 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:25.437+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.438230+0000) 2019-10-13T12:00:25.440 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:25.438+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.438558+0000) 2019-10-13T12:00:25.440 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:25.438+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.438450+0000) 2019-10-13T12:00:25.440 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:25.438+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.439142+0000) 2019-10-13T12:00:25.441 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:25.439+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.439537+0000) 2019-10-13T12:00:25.441 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:25.439+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.439675+0000) 2019-10-13T12:00:25.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:25.440+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.441013+0000) 2019-10-13T12:00:25.448 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:25.447+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222079 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:25.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:25.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.739365+0000) 2019-10-13T12:00:25.740 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:25.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.739366+0000) 2019-10-13T12:00:25.917 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:25.914+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:25.914679+0000) 2019-10-13T12:00:28.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:28.643+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.644134+0000) 2019-10-13T12:00:28.975 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:28.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.975015+0000) 2019-10-13T12:00:30.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:30.448+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.449082+0000) 2019-10-13T12:00:30.450 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:30.448+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.449310+0000) 2019-10-13T12:00:30.451 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:30.448+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.449653+0000) 2019-10-13T12:00:30.451 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:30.449+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.449680+0000) 2019-10-13T12:00:30.451 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:30.449+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.449930+0000) 2019-10-13T12:00:30.452 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:30.449+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.450252+0000) 2019-10-13T12:00:30.452 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:30.450+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.450419+0000) 2019-10-13T12:00:30.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:30.451+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:30.452445+0000) 2019-10-13T12:00:30.459 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:30.458+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:31.597 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:31.596+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:31.596391+0000) 2019-10-13T12:00:33.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:33.080+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.081292+0000) 2019-10-13T12:00:33.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:33.085+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.085314+0000) 2019-10-13T12:00:35.462 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:35.459+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.461194+0000) 2019-10-13T12:00:35.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:35.460+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.461428+0000) 2019-10-13T12:00:35.463 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:35.460+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.461832+0000) 2019-10-13T12:00:35.463 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:35.460+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.461945+0000) 2019-10-13T12:00:35.464 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:35.462+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.462858+0000) 2019-10-13T12:00:35.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:35.460+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.462254+0000) 2019-10-13T12:00:35.464 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:35.461+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.462465+0000) 2019-10-13T12:00:35.465 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:35.464+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.464775+0000) 2019-10-13T12:00:35.471 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:35.468+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222138 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:35.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:35.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.739519+0000) 2019-10-13T12:00:35.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:35.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.739519+0000) 2019-10-13T12:00:35.918 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:35.917+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:35.917552+0000) 2019-10-13T12:00:36.728 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:00:36.728 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:36.732 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:00:36.732 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:38.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:38.643+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.644284+0000) 2019-10-13T12:00:38.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:38.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.975215+0000) 2019-10-13T12:00:40.473 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:40.470+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.472017+0000) 2019-10-13T12:00:40.473 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:40.470+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.472155+0000) 2019-10-13T12:00:40.474 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:40.471+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.472909+0000) 2019-10-13T12:00:40.474 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:40.471+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.472628+0000) 2019-10-13T12:00:40.474 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:40.471+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.473004+0000) 2019-10-13T12:00:40.474 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:40.472+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.473438+0000) 2019-10-13T12:00:40.475 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:40.472+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.473443+0000) 2019-10-13T12:00:40.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:40.474+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:40.475099+0000) 2019-10-13T12:00:40.482 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:40.480+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:41.597 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:41.595+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:41.596633+0000) 2019-10-13T12:00:43.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:43.080+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.081448+0000) 2019-10-13T12:00:43.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:43.083+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.085502+0000) 2019-10-13T12:00:45.485 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:45.481+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.483309+0000) 2019-10-13T12:00:45.485 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:45.482+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.483559+0000) 2019-10-13T12:00:45.485 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:45.482+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.483949+0000) 2019-10-13T12:00:45.485 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:45.482+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.484311+0000) 2019-10-13T12:00:45.486 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:45.482+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.484322+0000) 2019-10-13T12:00:45.486 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:45.483+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.484451+0000) 2019-10-13T12:00:45.486 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:45.483+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.484558+0000) 2019-10-13T12:00:45.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:45.484+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.486587+0000) 2019-10-13T12:00:45.493 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:45.491+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222199 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:45.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:45.738+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.739684+0000) 2019-10-13T12:00:45.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:45.738+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.739685+0000) 2019-10-13T12:00:45.920 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:45.918+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:45.919872+0000) 2019-10-13T12:00:48.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:48.643+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.644456+0000) 2019-10-13T12:00:48.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:48.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.975428+0000) 2019-10-13T12:00:50.495 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:50.492+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.494068+0000) 2019-10-13T12:00:50.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:50.492+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.494197+0000) 2019-10-13T12:00:50.496 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:50.492+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.494633+0000) 2019-10-13T12:00:50.496 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:50.493+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.494950+0000) 2019-10-13T12:00:50.496 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:50.493+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.495003+0000) 2019-10-13T12:00:50.497 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:50.493+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.495149+0000) 2019-10-13T12:00:50.497 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:50.493+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.495309+0000) 2019-10-13T12:00:50.498 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:50.496+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:50.497901+0000) 2019-10-13T12:00:50.504 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:50.502+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:51.597 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:51.596+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:51.596798+0000) 2019-10-13T12:00:53.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:53.081+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.081633+0000) 2019-10-13T12:00:53.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:53.084+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.085656+0000) 2019-10-13T12:00:55.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:55.503+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.505336+0000) 2019-10-13T12:00:55.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:00:55.504+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.505676+0000) 2019-10-13T12:00:55.507 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:55.504+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.505835+0000) 2019-10-13T12:00:55.507 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:55.504+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.506037+0000) 2019-10-13T12:00:55.507 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:55.504+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.506333+0000) 2019-10-13T12:00:55.508 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:00:55.505+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.507027+0000) 2019-10-13T12:00:55.508 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:00:55.505+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.506947+0000) 2019-10-13T12:00:55.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:55.506+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.507861+0000) 2019-10-13T12:00:55.515 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:00:55.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222258 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:00:55.740 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:00:55.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.739848+0000) 2019-10-13T12:00:55.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:00:55.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.739855+0000) 2019-10-13T12:00:55.923 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:00:55.921+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:55.922295+0000) 2019-10-13T12:00:58.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:00:58.644+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.644669+0000) 2019-10-13T12:00:58.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:00:58.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.975620+0000) 2019-10-13T12:01:00.517 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:00.514+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.515852+0000) 2019-10-13T12:01:00.517 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:00.514+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.516011+0000) 2019-10-13T12:01:00.517 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:00.514+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.516202+0000) 2019-10-13T12:01:00.517 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:00.514+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.516331+0000) 2019-10-13T12:01:00.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:00.516+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.517287+0000) 2019-10-13T12:01:00.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:00.515+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.516669+0000) 2019-10-13T12:01:00.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:00.515+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.516875+0000) 2019-10-13T12:01:00.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:00.518+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:00.518892+0000) 2019-10-13T12:01:00.527 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:00.523+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222286 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:01.597 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:01.596+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:01.596924+0000) 2019-10-13T12:01:03.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:03.081+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.081802+0000) 2019-10-13T12:01:03.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:03.085+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.085811+0000) 2019-10-13T12:01:05.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:05.526+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.526410+0000) 2019-10-13T12:01:05.528 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:05.526+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.526681+0000) 2019-10-13T12:01:05.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:05.526+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.526980+0000) 2019-10-13T12:01:05.529 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:05.526+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.527036+0000) 2019-10-13T12:01:05.529 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:05.526+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.527261+0000) 2019-10-13T12:01:05.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:05.527+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.527664+0000) 2019-10-13T12:01:05.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:05.527+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.527922+0000) 2019-10-13T12:01:05.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:05.529+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.530275+0000) 2019-10-13T12:01:05.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:05.535+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222319 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:05.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:05.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.740034+0000) 2019-10-13T12:01:05.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:05.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.740033+0000) 2019-10-13T12:01:05.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:05.924+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:05.924851+0000) 2019-10-13T12:01:06.874 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:01:06.874 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:06.879 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:01:06.879 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:08.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:08.644+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.644860+0000) 2019-10-13T12:01:08.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:08.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.975805+0000) 2019-10-13T12:01:10.539 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:10.536+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.537620+0000) 2019-10-13T12:01:10.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:10.536+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.537217+0000) 2019-10-13T12:01:10.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:10.537+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.537584+0000) 2019-10-13T12:01:10.539 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:10.537+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.537896+0000) 2019-10-13T12:01:10.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:10.537+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.538070+0000) 2019-10-13T12:01:10.540 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:10.537+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.538283+0000) 2019-10-13T12:01:10.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:10.538+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.538401+0000) 2019-10-13T12:01:10.540 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:10.539+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:10.540043+0000) 2019-10-13T12:01:10.545 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:10.544+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:11.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:11.596+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:11.597093+0000) 2019-10-13T12:01:13.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:13.081+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.081972+0000) 2019-10-13T12:01:13.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:13.085+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.085986+0000) 2019-10-13T12:01:15.547 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:15.546+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.546524+0000) 2019-10-13T12:01:15.548 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:15.546+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.546725+0000) 2019-10-13T12:01:15.548 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:15.546+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.547549+0000) 2019-10-13T12:01:15.549 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:15.547+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.547686+0000) 2019-10-13T12:01:15.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:15.547+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.547735+0000) 2019-10-13T12:01:15.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:15.547+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.548001+0000) 2019-10-13T12:01:15.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:15.548+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.548441+0000) 2019-10-13T12:01:15.550 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:15.549+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.550203+0000) 2019-10-13T12:01:15.557 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:15.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222378 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:15.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:15.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.740193+0000) 2019-10-13T12:01:15.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:15.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.740213+0000) 2019-10-13T12:01:15.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:15.926+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:15.926902+0000) 2019-10-13T12:01:18.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:18.645+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.645052+0000) 2019-10-13T12:01:18.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:18.974+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.975946+0000) 2019-10-13T12:01:20.558 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:20.557+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.557985+0000) 2019-10-13T12:01:20.559 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:20.556+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.557957+0000) 2019-10-13T12:01:20.560 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:20.556+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.558210+0000) 2019-10-13T12:01:20.560 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:20.556+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.558376+0000) 2019-10-13T12:01:20.560 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:20.557+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.558550+0000) 2019-10-13T12:01:20.560 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:20.557+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.558827+0000) 2019-10-13T12:01:20.560 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:20.557+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.559013+0000) 2019-10-13T12:01:20.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:20.560+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:20.561565+0000) 2019-10-13T12:01:20.568 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:20.565+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222406 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:21.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:21.596+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:21.597283+0000) 2019-10-13T12:01:23.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:23.081+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.082156+0000) 2019-10-13T12:01:23.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:23.085+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.086168+0000) 2019-10-13T12:01:25.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:25.568+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.568836+0000) 2019-10-13T12:01:25.570 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:25.567+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.569141+0000) 2019-10-13T12:01:25.570 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:25.567+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.569224+0000) 2019-10-13T12:01:25.570 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:25.567+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.569364+0000) 2019-10-13T12:01:25.571 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:25.568+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.569892+0000) 2019-10-13T12:01:25.571 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:25.568+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.569768+0000) 2019-10-13T12:01:25.571 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:25.568+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.570069+0000) 2019-10-13T12:01:25.573 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:25.572+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.572784+0000) 2019-10-13T12:01:25.579 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:25.576+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222439 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:25.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:25.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.740357+0000) 2019-10-13T12:01:25.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:25.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.740358+0000) 2019-10-13T12:01:25.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:25.928+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:25.929324+0000) 2019-10-13T12:01:28.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:28.644+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.645236+0000) 2019-10-13T12:01:28.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:28.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.976069+0000) 2019-10-13T12:01:30.581 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:30.578+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.580036+0000) 2019-10-13T12:01:30.581 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:30.578+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.580322+0000) 2019-10-13T12:01:30.582 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:30.580+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.581199+0000) 2019-10-13T12:01:30.582 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:30.579+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.580665+0000) 2019-10-13T12:01:30.582 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:30.579+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.580826+0000) 2019-10-13T12:01:30.582 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:30.579+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.581322+0000) 2019-10-13T12:01:30.583 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:30.580+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.581496+0000) 2019-10-13T12:01:30.583 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:30.582+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:30.582814+0000) 2019-10-13T12:01:30.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:30.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:31.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:31.596+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:31.597476+0000) 2019-10-13T12:01:33.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:33.081+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.082368+0000) 2019-10-13T12:01:33.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:33.086+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.086346+0000) 2019-10-13T12:01:35.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:35.589+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.591059+0000) 2019-10-13T12:01:35.593 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:35.589+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.591133+0000) 2019-10-13T12:01:35.593 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:35.589+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.591317+0000) 2019-10-13T12:01:35.593 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:35.590+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.591605+0000) 2019-10-13T12:01:35.593 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:35.590+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.591701+0000) 2019-10-13T12:01:35.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:35.590+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.591987+0000) 2019-10-13T12:01:35.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:35.590+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.592321+0000) 2019-10-13T12:01:35.594 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:35.592+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.593839+0000) 2019-10-13T12:01:35.599 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:35.597+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222498 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:35.741 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:35.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.740550+0000) 2019-10-13T12:01:35.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:35.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.740550+0000) 2019-10-13T12:01:35.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:35.930+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:35.932286+0000) 2019-10-13T12:01:37.012 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:01:37.012 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:37.017 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:01:37.017 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:38.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:38.644+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.645439+0000) 2019-10-13T12:01:38.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:38.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.976255+0000) 2019-10-13T12:01:40.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:40.598+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.599952+0000) 2019-10-13T12:01:40.601 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:40.598+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.600230+0000) 2019-10-13T12:01:40.601 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:40.599+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.600447+0000) 2019-10-13T12:01:40.602 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:40.598+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.600341+0000) 2019-10-13T12:01:40.603 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:40.599+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.600765+0000) 2019-10-13T12:01:40.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:40.599+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.601028+0000) 2019-10-13T12:01:40.603 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:40.599+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.600852+0000) 2019-10-13T12:01:40.604 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:40.601+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:40.602945+0000) 2019-10-13T12:01:40.609 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:40.607+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222526 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:41.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:41.597+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:41.597673+0000) 2019-10-13T12:01:43.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:43.082+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.082569+0000) 2019-10-13T12:01:43.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:43.085+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.086506+0000) 2019-10-13T12:01:45.611 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:45.609+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.610440+0000) 2019-10-13T12:01:45.611 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:45.609+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.610776+0000) 2019-10-13T12:01:45.612 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:45.609+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.610839+0000) 2019-10-13T12:01:45.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:45.609+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.611101+0000) 2019-10-13T12:01:45.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:45.609+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.611381+0000) 2019-10-13T12:01:45.613 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:45.610+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.611669+0000) 2019-10-13T12:01:45.613 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:45.610+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.611561+0000) 2019-10-13T12:01:45.615 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:45.612+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.614469+0000) 2019-10-13T12:01:45.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:45.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222559 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:45.741 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:45.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.740714+0000) 2019-10-13T12:01:45.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:45.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.740712+0000) 2019-10-13T12:01:45.935 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:45.933+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:45.934585+0000) 2019-10-13T12:01:48.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:48.645+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.645606+0000) 2019-10-13T12:01:48.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:48.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.976444+0000) 2019-10-13T12:01:50.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:50.621+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.621879+0000) 2019-10-13T12:01:50.623 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:50.621+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.622192+0000) 2019-10-13T12:01:50.624 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:50.621+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.622283+0000) 2019-10-13T12:01:50.624 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:50.622+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.622584+0000) 2019-10-13T12:01:50.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:50.622+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.622676+0000) 2019-10-13T12:01:50.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:50.622+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.622864+0000) 2019-10-13T12:01:50.625 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:50.622+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.623044+0000) 2019-10-13T12:01:50.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:50.623+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:50.624614+0000) 2019-10-13T12:01:50.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:50.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222587 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:51.598 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:51.597+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:51.597842+0000) 2019-10-13T12:01:53.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:53.082+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.082748+0000) 2019-10-13T12:01:53.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:53.085+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.086694+0000) 2019-10-13T12:01:55.634 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:01:55.633+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.634199+0000) 2019-10-13T12:01:55.635 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:55.633+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.634171+0000) 2019-10-13T12:01:55.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:01:55.634+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.634422+0000) 2019-10-13T12:01:55.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:01:55.634+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.634648+0000) 2019-10-13T12:01:55.636 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:55.634+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.634818+0000) 2019-10-13T12:01:55.637 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:55.634+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.635204+0000) 2019-10-13T12:01:55.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:55.634+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.635383+0000) 2019-10-13T12:01:55.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:55.636+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.637511+0000) 2019-10-13T12:01:55.644 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:01:55.643+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222618 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:01:55.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:01:55.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.740883+0000) 2019-10-13T12:01:55.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:01:55.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.740882+0000) 2019-10-13T12:01:55.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:01:55.936+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:55.936959+0000) 2019-10-13T12:01:58.646 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:01:58.645+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.645810+0000) 2019-10-13T12:01:58.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:01:58.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.976640+0000) 2019-10-13T12:02:00.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:00.644+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.645408+0000) 2019-10-13T12:02:00.647 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:00.645+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.645647+0000) 2019-10-13T12:02:00.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:00.645+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.645721+0000) 2019-10-13T12:02:00.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:00.645+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.645999+0000) 2019-10-13T12:02:00.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:00.646+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.646413+0000) 2019-10-13T12:02:00.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:00.646+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.646594+0000) 2019-10-13T12:02:00.648 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:00.646+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.647189+0000) 2019-10-13T12:02:00.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:00.647+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:00.648220+0000) 2019-10-13T12:02:00.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:00.654+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222646 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:01.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:01.597+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:01.598005+0000) 2019-10-13T12:02:03.083 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:03.082+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.082898+0000) 2019-10-13T12:02:03.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:03.086+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.086868+0000) 2019-10-13T12:02:05.657 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:05.654+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.655892+0000) 2019-10-13T12:02:05.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:05.654+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.656116+0000) 2019-10-13T12:02:05.657 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:05.654+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.656359+0000) 2019-10-13T12:02:05.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:05.656+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.657302+0000) 2019-10-13T12:02:05.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:05.655+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.656866+0000) 2019-10-13T12:02:05.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:05.655+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.657361+0000) 2019-10-13T12:02:05.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:05.656+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.657464+0000) 2019-10-13T12:02:05.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:05.657+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.658566+0000) 2019-10-13T12:02:05.665 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:05.663+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222679 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:05.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:05.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.741059+0000) 2019-10-13T12:02:05.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:05.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.741060+0000) 2019-10-13T12:02:05.940 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:05.938+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:05.939327+0000) 2019-10-13T12:02:07.167 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:02:07.167 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:07.171 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:02:07.172 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:08.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:08.645+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.646001+0000) 2019-10-13T12:02:08.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:08.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.976845+0000) 2019-10-13T12:02:10.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:10.666+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.666850+0000) 2019-10-13T12:02:10.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:10.665+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.666537+0000) 2019-10-13T12:02:10.668 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:10.665+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.666825+0000) 2019-10-13T12:02:10.668 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:10.665+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.666971+0000) 2019-10-13T12:02:10.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:10.665+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.667134+0000) 2019-10-13T12:02:10.669 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:10.666+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.667455+0000) 2019-10-13T12:02:10.669 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:10.666+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.667807+0000) 2019-10-13T12:02:10.670 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:10.669+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:10.669811+0000) 2019-10-13T12:02:10.676 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:10.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222707 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:11.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:11.597+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:11.598169+0000) 2019-10-13T12:02:13.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:13.082+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.083072+0000) 2019-10-13T12:02:13.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:13.086+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.087048+0000) 2019-10-13T12:02:15.678 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:15.676+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.677526+0000) 2019-10-13T12:02:15.679 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:15.677+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.678592+0000) 2019-10-13T12:02:15.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:15.677+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.678493+0000) 2019-10-13T12:02:15.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:15.677+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.678649+0000) 2019-10-13T12:02:15.680 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:15.677+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.678871+0000) 2019-10-13T12:02:15.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:15.677+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.679075+0000) 2019-10-13T12:02:15.680 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:15.677+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.679159+0000) 2019-10-13T12:02:15.681 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:15.679+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.680272+0000) 2019-10-13T12:02:15.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:15.686+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:15.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:15.739+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.741256+0000) 2019-10-13T12:02:15.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:15.739+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.741256+0000) 2019-10-13T12:02:15.942 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:15.941+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:15.941662+0000) 2019-10-13T12:02:18.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:18.645+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.646197+0000) 2019-10-13T12:02:18.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:18.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.977035+0000) 2019-10-13T12:02:20.690 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:20.688+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.689600+0000) 2019-10-13T12:02:20.691 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:20.688+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.690157+0000) 2019-10-13T12:02:20.691 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:20.689+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.690402+0000) 2019-10-13T12:02:20.691 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:20.689+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.690430+0000) 2019-10-13T12:02:20.692 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:20.690+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.690871+0000) 2019-10-13T12:02:20.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:20.689+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.690782+0000) 2019-10-13T12:02:20.692 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:20.689+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.691285+0000) 2019-10-13T12:02:20.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:20.692+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:20.693306+0000) 2019-10-13T12:02:20.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:20.697+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:21.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:21.597+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:21.598307+0000) 2019-10-13T12:02:23.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:23.082+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.083284+0000) 2019-10-13T12:02:23.087 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:23.087+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:23.087232+0000) 2019-10-13T12:02:25.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:25.699+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.701287+0000) 2019-10-13T12:02:25.703 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:25.699+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.701240+0000) 2019-10-13T12:02:25.703 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:25.700+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.701394+0000) 2019-10-13T12:02:25.703 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:25.700+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.701682+0000) 2019-10-13T12:02:25.703 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:25.700+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.701853+0000) 2019-10-13T12:02:25.703 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:25.700+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.701933+0000) 2019-10-13T12:02:25.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:25.700+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.701571+0000) 2019-10-13T12:02:25.704 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:25.701+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.703298+0000) 2019-10-13T12:02:25.711 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:25.708+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:25.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:25.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.741406+0000) 2019-10-13T12:02:25.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:25.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.741412+0000) 2019-10-13T12:02:25.944 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:25.943+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:25.944022+0000) 2019-10-13T12:02:28.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:28.645+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.646430+0000) 2019-10-13T12:02:28.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:28.975+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.977248+0000) 2019-10-13T12:02:30.713 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:30.710+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.712359+0000) 2019-10-13T12:02:30.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:30.710+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.712347+0000) 2019-10-13T12:02:30.714 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:30.711+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.712491+0000) 2019-10-13T12:02:30.714 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:30.711+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.712663+0000) 2019-10-13T12:02:30.714 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:30.711+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.712981+0000) 2019-10-13T12:02:30.714 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:30.711+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.713073+0000) 2019-10-13T12:02:30.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:30.712+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.713623+0000) 2019-10-13T12:02:30.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:30.713+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:30.715549+0000) 2019-10-13T12:02:30.721 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:30.719+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222827 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:31.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:31.598+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:31.598471+0000) 2019-10-13T12:02:33.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:33.083+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.083406+0000) 2019-10-13T12:02:33.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:33.086+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:33.087410+0000) 2019-10-13T12:02:35.724 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:35.722+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.722849+0000) 2019-10-13T12:02:35.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:35.722+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.722885+0000) 2019-10-13T12:02:35.724 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:35.722+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.723342+0000) 2019-10-13T12:02:35.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:35.723+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.723497+0000) 2019-10-13T12:02:35.725 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:35.723+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.723564+0000) 2019-10-13T12:02:35.725 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:35.723+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.723920+0000) 2019-10-13T12:02:35.726 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:35.723+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.724339+0000) 2019-10-13T12:02:35.727 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:35.724+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.726356+0000) 2019-10-13T12:02:35.732 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:35.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222858 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:35.742 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:35.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.741501+0000) 2019-10-13T12:02:35.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:35.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.741500+0000) 2019-10-13T12:02:35.947 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:35.945+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:35.946503+0000) 2019-10-13T12:02:37.340 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:02:37.340 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:37.345 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:02:37.345 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:38.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:38.646+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.646658+0000) 2019-10-13T12:02:38.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:38.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.977443+0000) 2019-10-13T12:02:40.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:40.733+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.733724+0000) 2019-10-13T12:02:40.735 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:40.733+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.734109+0000) 2019-10-13T12:02:40.735 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:40.733+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.734255+0000) 2019-10-13T12:02:40.736 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:40.733+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.734659+0000) 2019-10-13T12:02:40.736 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:40.734+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.734576+0000) 2019-10-13T12:02:40.736 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:40.734+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.734726+0000) 2019-10-13T12:02:40.737 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:40.734+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.734975+0000) 2019-10-13T12:02:40.737 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:40.735+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:40.736138+0000) 2019-10-13T12:02:40.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:40.742+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:41.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:41.598+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:41.598605+0000) 2019-10-13T12:02:43.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:43.083+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.083548+0000) 2019-10-13T12:02:43.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:43.086+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:43.087563+0000) 2019-10-13T12:02:45.742 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:45.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.741652+0000) 2019-10-13T12:02:45.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:45.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.741654+0000) 2019-10-13T12:02:45.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:45.743+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.744313+0000) 2019-10-13T12:02:45.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:45.744+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.744479+0000) 2019-10-13T12:02:45.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:45.744+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.744857+0000) 2019-10-13T12:02:45.746 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:45.744+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.745024+0000) 2019-10-13T12:02:45.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:45.744+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.745032+0000) 2019-10-13T12:02:45.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:45.744+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.745229+0000) 2019-10-13T12:02:45.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:45.745+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.745443+0000) 2019-10-13T12:02:45.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:45.747+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.748294+0000) 2019-10-13T12:02:45.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:45.752+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:45.950 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:45.948+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:45.949570+0000) 2019-10-13T12:02:48.647 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:48.646+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.646835+0000) 2019-10-13T12:02:48.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:48.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.977637+0000) 2019-10-13T12:02:50.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:50.753+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.754844+0000) 2019-10-13T12:02:50.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:50.753+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.754734+0000) 2019-10-13T12:02:50.756 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:50.753+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.755267+0000) 2019-10-13T12:02:50.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:50.754+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.755370+0000) 2019-10-13T12:02:50.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:50.754+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.755671+0000) 2019-10-13T12:02:50.757 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:50.754+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.755912+0000) 2019-10-13T12:02:50.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:50.754+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.756075+0000) 2019-10-13T12:02:50.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:50.757+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:50.757793+0000) 2019-10-13T12:02:50.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:50.762+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222947 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:51.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:51.597+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:51.598755+0000) 2019-10-13T12:02:53.084 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:53.082+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.083769+0000) 2019-10-13T12:02:53.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:53.087+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:53.087756+0000) 2019-10-13T12:02:55.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:55.740+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.741872+0000) 2019-10-13T12:02:55.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:55.740+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.741874+0000) 2019-10-13T12:02:55.766 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:02:55.765+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.766048+0000) 2019-10-13T12:02:55.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:02:55.764+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.766189+0000) 2019-10-13T12:02:55.767 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:02:55.765+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.766403+0000) 2019-10-13T12:02:55.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:02:55.765+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.766542+0000) 2019-10-13T12:02:55.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:55.765+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.766802+0000) 2019-10-13T12:02:55.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:02:55.765+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.767203+0000) 2019-10-13T12:02:55.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:55.766+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.767433+0000) 2019-10-13T12:02:55.769 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:55.768+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.769369+0000) 2019-10-13T12:02:55.776 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:02:55.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 222978 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:02:55.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:02:55.951+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:55.951938+0000) 2019-10-13T12:02:58.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:02:58.646+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.647008+0000) 2019-10-13T12:02:58.978 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:02:58.976+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.977811+0000) 2019-10-13T12:03:00.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:00.775+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.776551+0000) 2019-10-13T12:03:00.778 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:00.776+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.777640+0000) 2019-10-13T12:03:00.779 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:00.776+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.777757+0000) 2019-10-13T12:03:00.779 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:00.776+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.778108+0000) 2019-10-13T12:03:00.779 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:00.776+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.778247+0000) 2019-10-13T12:03:00.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:00.776+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.778259+0000) 2019-10-13T12:03:00.780 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:00.776+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.778361+0000) 2019-10-13T12:03:00.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:00.779+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:00.780011+0000) 2019-10-13T12:03:00.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:00.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223006 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:01.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:01.598+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:01.598923+0000) 2019-10-13T12:03:03.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:03.083+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.083963+0000) 2019-10-13T12:03:03.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:03.087+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:03.087907+0000) 2019-10-13T12:03:05.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:05.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.742089+0000) 2019-10-13T12:03:05.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:05.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.742087+0000) 2019-10-13T12:03:05.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:05.787+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.788893+0000) 2019-10-13T12:03:05.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:05.787+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.789035+0000) 2019-10-13T12:03:05.791 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:05.788+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.789532+0000) 2019-10-13T12:03:05.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:05.788+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.789625+0000) 2019-10-13T12:03:05.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:05.788+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.789566+0000) 2019-10-13T12:03:05.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:05.788+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.790011+0000) 2019-10-13T12:03:05.792 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:05.789+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.790421+0000) 2019-10-13T12:03:05.793 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:05.791+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.792667+0000) 2019-10-13T12:03:05.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:05.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223039 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:05.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:05.953+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:05.954092+0000) 2019-10-13T12:03:07.501 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:03:07.501 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:07.505 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:03:07.505 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:08.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:08.646+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.647159+0000) 2019-10-13T12:03:08.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:08.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.978020+0000) 2019-10-13T12:03:10.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:10.798+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.799656+0000) 2019-10-13T12:03:10.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:10.798+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.799881+0000) 2019-10-13T12:03:10.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:10.798+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.800031+0000) 2019-10-13T12:03:10.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:10.798+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.800172+0000) 2019-10-13T12:03:10.802 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:10.800+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.800852+0000) 2019-10-13T12:03:10.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:10.799+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.801036+0000) 2019-10-13T12:03:10.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:10.799+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.801233+0000) 2019-10-13T12:03:10.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:10.802+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:10.802887+0000) 2019-10-13T12:03:10.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:10.807+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223067 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:11.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:11.598+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:11.599072+0000) 2019-10-13T12:03:13.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:13.083+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.084125+0000) 2019-10-13T12:03:13.088 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:13.088+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:13.088127+0000) 2019-10-13T12:03:15.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:15.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.742299+0000) 2019-10-13T12:03:15.743 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:15.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.742280+0000) 2019-10-13T12:03:15.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:15.808+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.810229+0000) 2019-10-13T12:03:15.811 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:15.809+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.810522+0000) 2019-10-13T12:03:15.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:15.809+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.810722+0000) 2019-10-13T12:03:15.812 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:15.809+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.810963+0000) 2019-10-13T12:03:15.812 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:15.809+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.811364+0000) 2019-10-13T12:03:15.813 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:15.810+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.811555+0000) 2019-10-13T12:03:15.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:15.810+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.811700+0000) 2019-10-13T12:03:15.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:15.811+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.813716+0000) 2019-10-13T12:03:15.819 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:15.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223098 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:15.957 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:15.955+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:15.956397+0000) 2019-10-13T12:03:18.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:18.646+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.647331+0000) 2019-10-13T12:03:18.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:18.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.978200+0000) 2019-10-13T12:03:20.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:20.820+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.820625+0000) 2019-10-13T12:03:20.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:20.819+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.821013+0000) 2019-10-13T12:03:20.823 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:20.820+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.821319+0000) 2019-10-13T12:03:20.823 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:20.821+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.821685+0000) 2019-10-13T12:03:20.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:20.821+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.821757+0000) 2019-10-13T12:03:20.823 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:20.821+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.821873+0000) 2019-10-13T12:03:20.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:20.821+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.822138+0000) 2019-10-13T12:03:20.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:20.821+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:20.823193+0000) 2019-10-13T12:03:20.831 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:20.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223126 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:21.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:21.598+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:21.599238+0000) 2019-10-13T12:03:23.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:23.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.084298+0000) 2019-10-13T12:03:23.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:23.087+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:23.088287+0000) 2019-10-13T12:03:25.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:25.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.742501+0000) 2019-10-13T12:03:25.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:25.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.742495+0000) 2019-10-13T12:03:25.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:25.831+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.831764+0000) 2019-10-13T12:03:25.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:25.831+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.831972+0000) 2019-10-13T12:03:25.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:25.831+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.832294+0000) 2019-10-13T12:03:25.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:25.831+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.833041+0000) 2019-10-13T12:03:25.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:25.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.833006+0000) 2019-10-13T12:03:25.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:25.832+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.833177+0000) 2019-10-13T12:03:25.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:25.833+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.833619+0000) 2019-10-13T12:03:25.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:25.832+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.834727+0000) 2019-10-13T12:03:25.842 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:25.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223159 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:25.961 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:25.958+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:25.959260+0000) 2019-10-13T12:03:28.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:28.647+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.647488+0000) 2019-10-13T12:03:28.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:28.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.978355+0000) 2019-10-13T12:03:30.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:30.843+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.843446+0000) 2019-10-13T12:03:30.845 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:30.843+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.843492+0000) 2019-10-13T12:03:30.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:30.843+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.843825+0000) 2019-10-13T12:03:30.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:30.843+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.843993+0000) 2019-10-13T12:03:30.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:30.843+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.844354+0000) 2019-10-13T12:03:30.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:30.843+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.844285+0000) 2019-10-13T12:03:30.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:30.844+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.844786+0000) 2019-10-13T12:03:30.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:30.846+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:30.847517+0000) 2019-10-13T12:03:30.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:30.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:31.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:31.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:31.599400+0000) 2019-10-13T12:03:33.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:33.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.084505+0000) 2019-10-13T12:03:33.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:33.087+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:33.088445+0000) 2019-10-13T12:03:35.743 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:35.741+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.742648+0000) 2019-10-13T12:03:35.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:35.741+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.742682+0000) 2019-10-13T12:03:35.855 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:35.854+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.854873+0000) 2019-10-13T12:03:35.856 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:35.853+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.854784+0000) 2019-10-13T12:03:35.856 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:35.853+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.854930+0000) 2019-10-13T12:03:35.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:35.854+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.855476+0000) 2019-10-13T12:03:35.857 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:35.854+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.855506+0000) 2019-10-13T12:03:35.857 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:35.854+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.855677+0000) 2019-10-13T12:03:35.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:35.854+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.856154+0000) 2019-10-13T12:03:35.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:35.857+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.858185+0000) 2019-10-13T12:03:35.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:35.862+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223218 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:35.962 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:35.960+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:35.961333+0000) 2019-10-13T12:03:37.671 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:03:37.671 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:37.676 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:03:37.676 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:38.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:38.646+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.647657+0000) 2019-10-13T12:03:38.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:38.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.978530+0000) 2019-10-13T12:03:40.866 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:40.864+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.865636+0000) 2019-10-13T12:03:40.866 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:40.863+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.865369+0000) 2019-10-13T12:03:40.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:40.864+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.865556+0000) 2019-10-13T12:03:40.867 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:40.864+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.866082+0000) 2019-10-13T12:03:40.868 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:40.864+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.866213+0000) 2019-10-13T12:03:40.868 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:40.865+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.866409+0000) 2019-10-13T12:03:40.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:40.865+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.866492+0000) 2019-10-13T12:03:40.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:40.867+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:40.868139+0000) 2019-10-13T12:03:40.874 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:40.872+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:41.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:41.598+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:41.599509+0000) 2019-10-13T12:03:43.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:43.083+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.084674+0000) 2019-10-13T12:03:43.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:43.087+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:43.088620+0000) 2019-10-13T12:03:45.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:45.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.742842+0000) 2019-10-13T12:03:45.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:45.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.742862+0000) 2019-10-13T12:03:45.876 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:45.873+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.875363+0000) 2019-10-13T12:03:45.877 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:45.875+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.875833+0000) 2019-10-13T12:03:45.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:45.874+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.875750+0000) 2019-10-13T12:03:45.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:45.874+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.875900+0000) 2019-10-13T12:03:45.877 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:45.874+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.876034+0000) 2019-10-13T12:03:45.878 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:45.874+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.876358+0000) 2019-10-13T12:03:45.878 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:45.874+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.876349+0000) 2019-10-13T12:03:45.878 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:45.876+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.877580+0000) 2019-10-13T12:03:45.885 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:45.882+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223278 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:45.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:45.963+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:45.963413+0000) 2019-10-13T12:03:48.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:48.647+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.647850+0000) 2019-10-13T12:03:48.979 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:48.977+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.978707+0000) 2019-10-13T12:03:50.887 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:50.884+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.885951+0000) 2019-10-13T12:03:50.887 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:50.884+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.886245+0000) 2019-10-13T12:03:50.888 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:50.885+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.886319+0000) 2019-10-13T12:03:50.888 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:50.885+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.886573+0000) 2019-10-13T12:03:50.888 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:50.885+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.887053+0000) 2019-10-13T12:03:50.888 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:50.885+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.887254+0000) 2019-10-13T12:03:50.889 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:50.886+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.887511+0000) 2019-10-13T12:03:50.889 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:50.887+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:50.888668+0000) 2019-10-13T12:03:50.896 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:50.894+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223307 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:51.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:51.598+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:51.599626+0000) 2019-10-13T12:03:53.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:53.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.084849+0000) 2019-10-13T12:03:53.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:53.088+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:53.088792+0000) 2019-10-13T12:03:55.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:55.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.742989+0000) 2019-10-13T12:03:55.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:55.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.743014+0000) 2019-10-13T12:03:55.898 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:55.895+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.897090+0000) 2019-10-13T12:03:55.898 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:55.895+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.897217+0000) 2019-10-13T12:03:55.899 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:03:55.896+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.897688+0000) 2019-10-13T12:03:55.899 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:03:55.896+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.897625+0000) 2019-10-13T12:03:55.899 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:03:55.896+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.897678+0000) 2019-10-13T12:03:55.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:03:55.896+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.897899+0000) 2019-10-13T12:03:55.899 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:03:55.896+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.898324+0000) 2019-10-13T12:03:55.900 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:55.898+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.899664+0000) 2019-10-13T12:03:55.906 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:03:55.904+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:03:55.966 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:03:55.965+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:55.965634+0000) 2019-10-13T12:03:58.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:03:58.647+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.648068+0000) 2019-10-13T12:03:58.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:03:58.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.978891+0000) 2019-10-13T12:04:00.908 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:00.905+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.907059+0000) 2019-10-13T12:04:00.908 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:00.906+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.907444+0000) 2019-10-13T12:04:00.909 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:00.906+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.907693+0000) 2019-10-13T12:04:00.909 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:00.907+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.908020+0000) 2019-10-13T12:04:00.910 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:00.906+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.908188+0000) 2019-10-13T12:04:00.910 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:00.907+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.908490+0000) 2019-10-13T12:04:00.910 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:00.907+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.908509+0000) 2019-10-13T12:04:00.911 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:00.910+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:00.911050+0000) 2019-10-13T12:04:00.917 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:00.916+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223365 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:01.600 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:01.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:01.599768+0000) 2019-10-13T12:04:03.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:03.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.085003+0000) 2019-10-13T12:04:03.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:03.088+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:03.088958+0000) 2019-10-13T12:04:05.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:05.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.743174+0000) 2019-10-13T12:04:05.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:05.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.743137+0000) 2019-10-13T12:04:05.918 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:05.917+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.917866+0000) 2019-10-13T12:04:05.919 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:05.917+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.917964+0000) 2019-10-13T12:04:05.920 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:05.916+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.918413+0000) 2019-10-13T12:04:05.920 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:05.917+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.918376+0000) 2019-10-13T12:04:05.920 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:05.918+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.918653+0000) 2019-10-13T12:04:05.920 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:05.918+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.918959+0000) 2019-10-13T12:04:05.920 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:05.918+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.919158+0000) 2019-10-13T12:04:05.921 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:05.919+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.921119+0000) 2019-10-13T12:04:05.928 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:05.927+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:05.968 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:05.966+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:05.967852+0000) 2019-10-13T12:04:07.834 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:04:07.834 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:07.838 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:04:07.838 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:08.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:08.647+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.648229+0000) 2019-10-13T12:04:08.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:08.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.979077+0000) 2019-10-13T12:04:10.930 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:10.928+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.929220+0000) 2019-10-13T12:04:10.930 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:10.929+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.929543+0000) 2019-10-13T12:04:10.931 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:10.929+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.929665+0000) 2019-10-13T12:04:10.931 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:10.929+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.930915+0000) 2019-10-13T12:04:10.932 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:10.929+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.930001+0000) 2019-10-13T12:04:10.932 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:10.930+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.930428+0000) 2019-10-13T12:04:10.932 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:10.930+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.931062+0000) 2019-10-13T12:04:10.933 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:10.931+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:10.933368+0000) 2019-10-13T12:04:10.939 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:10.937+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223425 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:11.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:11.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:11.599939+0000) 2019-10-13T12:04:13.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:13.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.085157+0000) 2019-10-13T12:04:13.089 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:13.088+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:13.089131+0000) 2019-10-13T12:04:15.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:15.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.743337+0000) 2019-10-13T12:04:15.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:15.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.743330+0000) 2019-10-13T12:04:15.941 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:15.939+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.939881+0000) 2019-10-13T12:04:15.942 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:15.939+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.939999+0000) 2019-10-13T12:04:15.942 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:15.939+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.940253+0000) 2019-10-13T12:04:15.942 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:15.940+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.940495+0000) 2019-10-13T12:04:15.942 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:15.940+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.940533+0000) 2019-10-13T12:04:15.942 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:15.940+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.940866+0000) 2019-10-13T12:04:15.943 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:15.939+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.940877+0000) 2019-10-13T12:04:15.944 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:15.941+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.943451+0000) 2019-10-13T12:04:15.949 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:15.948+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:15.970 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:15.969+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:15.969944+0000) 2019-10-13T12:04:18.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:18.648+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.648419+0000) 2019-10-13T12:04:18.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:18.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.979313+0000) 2019-10-13T12:04:20.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:20.949+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.950642+0000) 2019-10-13T12:04:20.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:20.950+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.951046+0000) 2019-10-13T12:04:20.952 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:20.949+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.951030+0000) 2019-10-13T12:04:20.953 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:20.949+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.951344+0000) 2019-10-13T12:04:20.953 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:20.950+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.951563+0000) 2019-10-13T12:04:20.953 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:20.950+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.951710+0000) 2019-10-13T12:04:20.953 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:20.950+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.951975+0000) 2019-10-13T12:04:20.955 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:20.954+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:20.954756+0000) 2019-10-13T12:04:20.960 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:20.958+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:21.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:21.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:21.600104+0000) 2019-10-13T12:04:23.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:23.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.085326+0000) 2019-10-13T12:04:23.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:23.088+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:23.089335+0000) 2019-10-13T12:04:25.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:25.742+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.743548+0000) 2019-10-13T12:04:25.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:25.742+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.743546+0000) 2019-10-13T12:04:25.962 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:25.959+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.961307+0000) 2019-10-13T12:04:25.963 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:25.960+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.961663+0000) 2019-10-13T12:04:25.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:25.960+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.961664+0000) 2019-10-13T12:04:25.963 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:25.960+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.961789+0000) 2019-10-13T12:04:25.964 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:25.960+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.962280+0000) 2019-10-13T12:04:25.964 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:25.961+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.962508+0000) 2019-10-13T12:04:25.964 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:25.961+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.962872+0000) 2019-10-13T12:04:25.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:25.964+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.965210+0000) 2019-10-13T12:04:25.971 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:25.969+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:25.972 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:25.971+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:25.972036+0000) 2019-10-13T12:04:28.649 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:28.647+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.648599+0000) 2019-10-13T12:04:28.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:28.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.979515+0000) 2019-10-13T12:04:30.973 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:30.970+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.972311+0000) 2019-10-13T12:04:30.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:30.971+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.972615+0000) 2019-10-13T12:04:30.974 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:30.972+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.973358+0000) 2019-10-13T12:04:30.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:30.971+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.972834+0000) 2019-10-13T12:04:30.975 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:30.971+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.972970+0000) 2019-10-13T12:04:30.975 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:30.972+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.973460+0000) 2019-10-13T12:04:30.975 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:30.972+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.973758+0000) 2019-10-13T12:04:30.975 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:30.973+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:30.974556+0000) 2019-10-13T12:04:30.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:30.980+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223544 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:31.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:31.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:31.600229+0000) 2019-10-13T12:04:33.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:33.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.085474+0000) 2019-10-13T12:04:33.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:33.088+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:33.089507+0000) 2019-10-13T12:04:35.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:35.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.743726+0000) 2019-10-13T12:04:35.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:35.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.743726+0000) 2019-10-13T12:04:35.975 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:35.973+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.974154+0000) 2019-10-13T12:04:35.984 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:35.982+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.983385+0000) 2019-10-13T12:04:35.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:35.982+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.983436+0000) 2019-10-13T12:04:35.985 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:35.982+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.983583+0000) 2019-10-13T12:04:35.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:35.982+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.983910+0000) 2019-10-13T12:04:35.985 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:35.982+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.984031+0000) 2019-10-13T12:04:35.985 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:35.982+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.984113+0000) 2019-10-13T12:04:35.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:35.983+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.984657+0000) 2019-10-13T12:04:35.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:35.985+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:35.986021+0000) 2019-10-13T12:04:35.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:35.991+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223573 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:37.963 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:04:37.963 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:37.968 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:04:37.968 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:38.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:38.648+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.648790+0000) 2019-10-13T12:04:38.980 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:38.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.979645+0000) 2019-10-13T12:04:40.995 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:40.992+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.994301+0000) 2019-10-13T12:04:40.995 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:40.993+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.994617+0000) 2019-10-13T12:04:40.996 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:40.993+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.994687+0000) 2019-10-13T12:04:40.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:40.993+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.994840+0000) 2019-10-13T12:04:40.997 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:40.993+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.995032+0000) 2019-10-13T12:04:40.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:40.993+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.995268+0000) 2019-10-13T12:04:40.997 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:40.994+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.995911+0000) 2019-10-13T12:04:40.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:40.996+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:40.997440+0000) 2019-10-13T12:04:41.004 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:41.002+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223602 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:41.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:41.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:41.600381+0000) 2019-10-13T12:04:43.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:43.085+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.085611+0000) 2019-10-13T12:04:43.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:43.089+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:43.089690+0000) 2019-10-13T12:04:45.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:45.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:45.743932+0000) 2019-10-13T12:04:45.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:45.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:45.743933+0000) 2019-10-13T12:04:45.977 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:45.976+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:45.976712+0000) 2019-10-13T12:04:46.007 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:46.004+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.006261+0000) 2019-10-13T12:04:46.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:46.005+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.006448+0000) 2019-10-13T12:04:46.007 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:46.005+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.006551+0000) 2019-10-13T12:04:46.008 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:46.006+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.006924+0000) 2019-10-13T12:04:46.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:46.005+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.006997+0000) 2019-10-13T12:04:46.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:46.005+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.007099+0000) 2019-10-13T12:04:46.009 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:46.006+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.007676+0000) 2019-10-13T12:04:46.011 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:46.009+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:46.010618+0000) 2019-10-13T12:04:46.015 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:46.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223635 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:48.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:48.648+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.648979+0000) 2019-10-13T12:04:48.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:48.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.979866+0000) 2019-10-13T12:04:51.017 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:51.015+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.016006+0000) 2019-10-13T12:04:51.017 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:51.015+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.016129+0000) 2019-10-13T12:04:51.017 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:51.015+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.016254+0000) 2019-10-13T12:04:51.018 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:51.015+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.016330+0000) 2019-10-13T12:04:51.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:51.016+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.016585+0000) 2019-10-13T12:04:51.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:51.016+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.016888+0000) 2019-10-13T12:04:51.019 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:51.016+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.017127+0000) 2019-10-13T12:04:51.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:51.018+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.019221+0000) 2019-10-13T12:04:51.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:51.022+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223665 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:51.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:51.600+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.600530+0000) 2019-10-13T12:04:53.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:53.085+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.085758+0000) 2019-10-13T12:04:53.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:53.089+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:53.089913+0000) 2019-10-13T12:04:55.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:55.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:55.744118+0000) 2019-10-13T12:04:55.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:55.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:55.744121+0000) 2019-10-13T12:04:55.980 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:55.978+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:55.979198+0000) 2019-10-13T12:04:56.025 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:04:56.024+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.024659+0000) 2019-10-13T12:04:56.026 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:56.024+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.025016+0000) 2019-10-13T12:04:56.026 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:56.024+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.025108+0000) 2019-10-13T12:04:56.027 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:04:56.023+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.025586+0000) 2019-10-13T12:04:56.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:04:56.025+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.025646+0000) 2019-10-13T12:04:56.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:04:56.025+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.025903+0000) 2019-10-13T12:04:56.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:04:56.025+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.026151+0000) 2019-10-13T12:04:56.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:04:56.026+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:56.027904+0000) 2019-10-13T12:04:56.034 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:04:56.033+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223695 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:04:58.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:04:58.649+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.649180+0000) 2019-10-13T12:04:58.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:04:58.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.980054+0000) 2019-10-13T12:05:01.036 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:01.035+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.035806+0000) 2019-10-13T12:05:01.037 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:01.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.036125+0000) 2019-10-13T12:05:01.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:01.035+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.036335+0000) 2019-10-13T12:05:01.038 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:01.036+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.036679+0000) 2019-10-13T12:05:01.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:01.036+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.036889+0000) 2019-10-13T12:05:01.038 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:01.036+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.037120+0000) 2019-10-13T12:05:01.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:01.036+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.037287+0000) 2019-10-13T12:05:01.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:01.037+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.038779+0000) 2019-10-13T12:05:01.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:01.044+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223725 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:01.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:01.600+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.600678+0000) 2019-10-13T12:05:03.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:03.085+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.085932+0000) 2019-10-13T12:05:03.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:03.089+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:03.090061+0000) 2019-10-13T12:05:05.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:05.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.744325+0000) 2019-10-13T12:05:05.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:05.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.744346+0000) 2019-10-13T12:05:05.982 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:05.980+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.981597+0000) 2019-10-13T12:05:06.047 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:06.045+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.046619+0000) 2019-10-13T12:05:06.048 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:06.045+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.047139+0000) 2019-10-13T12:05:06.048 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:06.045+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.047305+0000) 2019-10-13T12:05:06.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:06.046+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.047421+0000) 2019-10-13T12:05:06.049 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:06.046+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.047661+0000) 2019-10-13T12:05:06.049 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:06.046+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.047892+0000) 2019-10-13T12:05:06.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:06.046+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.047761+0000) 2019-10-13T12:05:06.050 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:06.048+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:06.049853+0000) 2019-10-13T12:05:06.057 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:06.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223757 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:08.067 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:05:08.068 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:08.072 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:05:08.072 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:08.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:08.648+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.649370+0000) 2019-10-13T12:05:08.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:08.978+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.980264+0000) 2019-10-13T12:05:11.058 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:11.055+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.057314+0000) 2019-10-13T12:05:11.059 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:11.057+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.057812+0000) 2019-10-13T12:05:11.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:11.056+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.058188+0000) 2019-10-13T12:05:11.059 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:11.057+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.058470+0000) 2019-10-13T12:05:11.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:11.057+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.059032+0000) 2019-10-13T12:05:11.060 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:11.057+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.058962+0000) 2019-10-13T12:05:11.060 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:11.058+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.059417+0000) 2019-10-13T12:05:11.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:11.059+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.060567+0000) 2019-10-13T12:05:11.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:11.066+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223787 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:11.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:11.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.600851+0000) 2019-10-13T12:05:13.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:13.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.086090+0000) 2019-10-13T12:05:13.090 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:13.089+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:13.090213+0000) 2019-10-13T12:05:15.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:15.743+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.744509+0000) 2019-10-13T12:05:15.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:15.743+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.744511+0000) 2019-10-13T12:05:15.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:15.983+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.983921+0000) 2019-10-13T12:05:16.070 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:16.068+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.069459+0000) 2019-10-13T12:05:16.070 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:16.068+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.069604+0000) 2019-10-13T12:05:16.071 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:16.069+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.070138+0000) 2019-10-13T12:05:16.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:16.068+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.070059+0000) 2019-10-13T12:05:16.071 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:16.068+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.070184+0000) 2019-10-13T12:05:16.072 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:16.069+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.070692+0000) 2019-10-13T12:05:16.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:16.070+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.071157+0000) 2019-10-13T12:05:16.072 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:16.069+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:16.070904+0000) 2019-10-13T12:05:16.079 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:16.076+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223817 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:18.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:18.648+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.649562+0000) 2019-10-13T12:05:18.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:18.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.980411+0000) 2019-10-13T12:05:21.080 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:21.079+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.080049+0000) 2019-10-13T12:05:21.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:21.078+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.080027+0000) 2019-10-13T12:05:21.082 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:21.078+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.080074+0000) 2019-10-13T12:05:21.082 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:21.079+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.080427+0000) 2019-10-13T12:05:21.082 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:21.079+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.080724+0000) 2019-10-13T12:05:21.082 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:21.079+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.080836+0000) 2019-10-13T12:05:21.082 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:21.079+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.080591+0000) 2019-10-13T12:05:21.083 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:21.081+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.081866+0000) 2019-10-13T12:05:21.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:21.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223847 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:21.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:21.599+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.601033+0000) 2019-10-13T12:05:23.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:23.084+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.086235+0000) 2019-10-13T12:05:23.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:23.089+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:23.090385+0000) 2019-10-13T12:05:25.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:25.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.744714+0000) 2019-10-13T12:05:25.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:25.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.744710+0000) 2019-10-13T12:05:25.987 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:25.986+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.987067+0000) 2019-10-13T12:05:26.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:26.088+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.089649+0000) 2019-10-13T12:05:26.091 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:26.088+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.090031+0000) 2019-10-13T12:05:26.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:26.089+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.090004+0000) 2019-10-13T12:05:26.092 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:26.088+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.090229+0000) 2019-10-13T12:05:26.092 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:26.088+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.090254+0000) 2019-10-13T12:05:26.092 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:26.089+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.090618+0000) 2019-10-13T12:05:26.092 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:26.089+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.090836+0000) 2019-10-13T12:05:26.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:26.090+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:26.091419+0000) 2019-10-13T12:05:26.099 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:26.097+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223879 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:28.650 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:28.649+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.649759+0000) 2019-10-13T12:05:28.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:28.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.980554+0000) 2019-10-13T12:05:31.101 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:31.098+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.100030+0000) 2019-10-13T12:05:31.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:31.099+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.100478+0000) 2019-10-13T12:05:31.102 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:31.099+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.100591+0000) 2019-10-13T12:05:31.102 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:31.099+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.100947+0000) 2019-10-13T12:05:31.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:31.099+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.101092+0000) 2019-10-13T12:05:31.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:31.099+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.101224+0000) 2019-10-13T12:05:31.103 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:31.099+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.101368+0000) 2019-10-13T12:05:31.103 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:31.101+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.102591+0000) 2019-10-13T12:05:31.110 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:31.108+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223909 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:31.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:31.600+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.601189+0000) 2019-10-13T12:05:33.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:33.086+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.086425+0000) 2019-10-13T12:05:33.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:33.090+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:33.090576+0000) 2019-10-13T12:05:35.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:35.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.744918+0000) 2019-10-13T12:05:35.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:35.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.744928+0000) 2019-10-13T12:05:35.990 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:35.989+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.989393+0000) 2019-10-13T12:05:36.111 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:36.110+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.110600+0000) 2019-10-13T12:05:36.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:36.110+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.110886+0000) 2019-10-13T12:05:36.112 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:36.110+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.111010+0000) 2019-10-13T12:05:36.112 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:36.110+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.111104+0000) 2019-10-13T12:05:36.113 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:36.110+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.111391+0000) 2019-10-13T12:05:36.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:36.111+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.111736+0000) 2019-10-13T12:05:36.113 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:36.111+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.111856+0000) 2019-10-13T12:05:36.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:36.113+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:36.113922+0000) 2019-10-13T12:05:36.118 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:36.117+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223939 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:38.188 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:05:38.189 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:38.193 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:05:38.193 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:38.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:38.649+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.649958+0000) 2019-10-13T12:05:38.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:38.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.980712+0000) 2019-10-13T12:05:41.120 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:41.119+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.119403+0000) 2019-10-13T12:05:41.121 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:41.119+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.119557+0000) 2019-10-13T12:05:41.121 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:41.119+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.119727+0000) 2019-10-13T12:05:41.122 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:41.119+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.120143+0000) 2019-10-13T12:05:41.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:41.119+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.120077+0000) 2019-10-13T12:05:41.122 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:41.120+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.120793+0000) 2019-10-13T12:05:41.122 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:41.120+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.120836+0000) 2019-10-13T12:05:41.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:41.122+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.122803+0000) 2019-10-13T12:05:41.130 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:41.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 223969 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:41.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:41.600+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.601367+0000) 2019-10-13T12:05:43.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:43.086+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.086608+0000) 2019-10-13T12:05:43.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:43.090+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:43.090804+0000) 2019-10-13T12:05:45.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:45.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.745131+0000) 2019-10-13T12:05:45.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:45.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.745138+0000) 2019-10-13T12:05:45.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:45.990+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.991769+0000) 2019-10-13T12:05:46.131 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:46.130+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.130610+0000) 2019-10-13T12:05:46.132 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:46.130+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.130852+0000) 2019-10-13T12:05:46.132 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:46.130+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.131142+0000) 2019-10-13T12:05:46.133 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:46.129+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.131719+0000) 2019-10-13T12:05:46.133 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:46.130+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.131271+0000) 2019-10-13T12:05:46.133 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:46.131+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.131853+0000) 2019-10-13T12:05:46.133 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:46.130+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.131380+0000) 2019-10-13T12:05:46.134 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:46.131+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:46.133128+0000) 2019-10-13T12:05:46.139 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:46.138+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224001 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:48.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:48.650+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.650157+0000) 2019-10-13T12:05:48.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:48.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.980858+0000) 2019-10-13T12:05:51.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:51.138+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.140717+0000) 2019-10-13T12:05:51.142 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:51.139+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.140823+0000) 2019-10-13T12:05:51.142 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:51.139+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.140965+0000) 2019-10-13T12:05:51.142 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:51.139+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.141260+0000) 2019-10-13T12:05:51.142 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:51.140+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.141630+0000) 2019-10-13T12:05:51.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:51.140+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.141871+0000) 2019-10-13T12:05:51.143 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:51.140+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.142123+0000) 2019-10-13T12:05:51.144 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:51.141+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.143347+0000) 2019-10-13T12:05:51.150 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:51.148+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224031 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:51.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:51.600+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.601556+0000) 2019-10-13T12:05:53.087 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:53.085+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.086762+0000) 2019-10-13T12:05:53.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:53.090+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:53.091007+0000) 2019-10-13T12:05:55.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:55.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.745329+0000) 2019-10-13T12:05:55.746 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:55.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.745330+0000) 2019-10-13T12:05:55.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:55.993+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.994200+0000) 2019-10-13T12:05:56.152 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:05:56.149+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.151530+0000) 2019-10-13T12:05:56.153 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:05:56.150+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.151621+0000) 2019-10-13T12:05:56.153 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:05:56.150+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.151774+0000) 2019-10-13T12:05:56.153 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:05:56.150+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.152069+0000) 2019-10-13T12:05:56.153 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:56.150+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.152174+0000) 2019-10-13T12:05:56.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:56.151+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.152497+0000) 2019-10-13T12:05:56.154 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:05:56.151+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.152794+0000) 2019-10-13T12:05:56.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:05:56.152+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:56.154235+0000) 2019-10-13T12:05:56.161 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:05:56.159+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224061 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:05:58.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:05:58.649+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.650366+0000) 2019-10-13T12:05:58.981 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:05:58.979+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.980973+0000) 2019-10-13T12:06:01.163 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:01.160+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.162175+0000) 2019-10-13T12:06:01.164 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:01.162+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.162884+0000) 2019-10-13T12:06:01.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:01.161+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.162879+0000) 2019-10-13T12:06:01.164 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:01.161+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.163145+0000) 2019-10-13T12:06:01.165 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:01.161+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.163271+0000) 2019-10-13T12:06:01.165 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:01.162+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.163451+0000) 2019-10-13T12:06:01.165 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:01.162+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.163672+0000) 2019-10-13T12:06:01.165 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:01.163+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.164698+0000) 2019-10-13T12:06:01.172 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:01.170+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224091 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:01.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:01.600+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.601702+0000) 2019-10-13T12:06:03.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:03.085+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.086983+0000) 2019-10-13T12:06:03.091 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:03.090+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:03.091217+0000) 2019-10-13T12:06:05.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:05.744+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.745551+0000) 2019-10-13T12:06:05.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:05.744+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.745550+0000) 2019-10-13T12:06:05.997 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:05.995+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.996522+0000) 2019-10-13T12:06:06.174 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:06.171+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.173260+0000) 2019-10-13T12:06:06.174 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:06.172+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.173443+0000) 2019-10-13T12:06:06.175 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:06.172+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.173497+0000) 2019-10-13T12:06:06.175 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:06.172+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.173709+0000) 2019-10-13T12:06:06.175 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:06.172+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.173822+0000) 2019-10-13T12:06:06.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:06.173+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.174264+0000) 2019-10-13T12:06:06.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:06.172+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.174119+0000) 2019-10-13T12:06:06.176 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:06.174+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:06.175740+0000) 2019-10-13T12:06:06.181 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:06.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224123 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:08.351 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:06:08.352 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:08.356 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:06:08.357 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:08.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:08.649+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.650499+0000) 2019-10-13T12:06:08.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:08.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.981127+0000) 2019-10-13T12:06:11.183 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:11.180+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.181742+0000) 2019-10-13T12:06:11.183 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:11.180+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.182227+0000) 2019-10-13T12:06:11.184 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:11.181+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.182473+0000) 2019-10-13T12:06:11.184 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:11.181+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.182693+0000) 2019-10-13T12:06:11.184 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:11.181+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.183005+0000) 2019-10-13T12:06:11.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:11.181+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.182845+0000) 2019-10-13T12:06:11.185 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:11.181+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.183249+0000) 2019-10-13T12:06:11.186 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:11.184+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.185638+0000) 2019-10-13T12:06:11.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:11.190+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224153 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:11.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:11.601+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.601899+0000) 2019-10-13T12:06:13.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:13.085+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.087190+0000) 2019-10-13T12:06:13.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:13.090+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:13.091419+0000) 2019-10-13T12:06:15.746 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:15.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.745734+0000) 2019-10-13T12:06:15.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:15.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.745734+0000) 2019-10-13T12:06:15.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:15.998+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.999026+0000) 2019-10-13T12:06:16.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:16.191+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.192794+0000) 2019-10-13T12:06:16.194 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:16.191+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.193009+0000) 2019-10-13T12:06:16.194 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:16.191+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.193245+0000) 2019-10-13T12:06:16.194 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:16.191+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.193376+0000) 2019-10-13T12:06:16.195 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:16.192+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.193654+0000) 2019-10-13T12:06:16.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:16.192+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.193747+0000) 2019-10-13T12:06:16.195 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:16.192+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.193835+0000) 2019-10-13T12:06:16.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:16.195+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:16.195745+0000) 2019-10-13T12:06:16.200 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:16.199+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224183 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:18.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:18.650+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.650683+0000) 2019-10-13T12:06:18.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:18.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.981304+0000) 2019-10-13T12:06:21.203 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:21.201+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.201793+0000) 2019-10-13T12:06:21.203 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:21.201+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.201988+0000) 2019-10-13T12:06:21.203 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:21.201+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.202073+0000) 2019-10-13T12:06:21.204 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:21.202+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.203209+0000) 2019-10-13T12:06:21.204 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:21.202+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.202574+0000) 2019-10-13T12:06:21.204 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:21.202+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.202786+0000) 2019-10-13T12:06:21.204 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:21.203+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.203411+0000) 2019-10-13T12:06:21.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:21.204+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.205544+0000) 2019-10-13T12:06:21.212 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:21.210+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224213 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:21.602 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:21.601+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.602053+0000) 2019-10-13T12:06:23.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:23.086+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.087347+0000) 2019-10-13T12:06:23.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:23.090+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:23.091619+0000) 2019-10-13T12:06:25.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:25.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.745927+0000) 2019-10-13T12:06:25.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:25.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.745930+0000) 2019-10-13T12:06:26.002 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:26.000+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.001464+0000) 2019-10-13T12:06:26.214 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:26.212+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.212968+0000) 2019-10-13T12:06:26.214 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:26.212+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.213314+0000) 2019-10-13T12:06:26.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:26.213+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.213834+0000) 2019-10-13T12:06:26.215 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:26.213+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.213913+0000) 2019-10-13T12:06:26.215 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:26.213+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.214241+0000) 2019-10-13T12:06:26.216 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:26.214+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.214740+0000) 2019-10-13T12:06:26.216 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:26.214+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.214675+0000) 2019-10-13T12:06:26.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:26.215+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:26.216630+0000) 2019-10-13T12:06:26.223 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:26.222+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224245 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:28.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:28.650+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.650868+0000) 2019-10-13T12:06:28.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:28.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.981491+0000) 2019-10-13T12:06:31.225 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:31.224+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.224414+0000) 2019-10-13T12:06:31.226 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:31.224+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.224529+0000) 2019-10-13T12:06:31.226 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:31.224+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.224747+0000) 2019-10-13T12:06:31.226 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:31.224+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.224885+0000) 2019-10-13T12:06:31.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:31.224+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.224941+0000) 2019-10-13T12:06:31.227 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:31.224+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.225290+0000) 2019-10-13T12:06:31.227 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:31.225+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.225416+0000) 2019-10-13T12:06:31.228 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:31.227+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.227932+0000) 2019-10-13T12:06:31.232 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:31.231+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224275 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:31.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:31.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.602207+0000) 2019-10-13T12:06:33.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:33.087+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.087466+0000) 2019-10-13T12:06:33.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:33.091+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:33.091758+0000) 2019-10-13T12:06:35.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:35.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.746128+0000) 2019-10-13T12:06:35.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:35.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.746134+0000) 2019-10-13T12:06:36.004 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:36.002+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.003929+0000) 2019-10-13T12:06:36.235 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:36.232+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.233924+0000) 2019-10-13T12:06:36.235 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:36.232+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.234021+0000) 2019-10-13T12:06:36.235 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:36.232+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.234222+0000) 2019-10-13T12:06:36.235 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:36.232+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.234383+0000) 2019-10-13T12:06:36.236 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:36.233+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.234762+0000) 2019-10-13T12:06:36.236 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:36.233+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.234926+0000) 2019-10-13T12:06:36.237 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:36.233+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.235260+0000) 2019-10-13T12:06:36.237 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:36.235+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:36.236955+0000) 2019-10-13T12:06:36.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:36.241+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224306 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:38.539 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:06:38.540 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:38.544 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:06:38.544 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:38.651 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:38.650+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:38.651058+0000) 2019-10-13T12:06:38.982 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:38.980+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:38.981698+0000) 2019-10-13T12:06:41.246 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:41.243+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.245229+0000) 2019-10-13T12:06:41.247 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:41.243+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.245340+0000) 2019-10-13T12:06:41.247 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:41.244+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.245447+0000) 2019-10-13T12:06:41.247 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:41.244+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.245599+0000) 2019-10-13T12:06:41.247 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:41.244+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.245725+0000) 2019-10-13T12:06:41.247 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:41.244+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.245910+0000) 2019-10-13T12:06:41.248 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:41.244+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.246069+0000) 2019-10-13T12:06:41.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:41.245+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.247055+0000) 2019-10-13T12:06:41.255 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:41.252+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224336 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:41.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:41.601+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.602364+0000) 2019-10-13T12:06:43.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:43.086+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:43.087614+0000) 2019-10-13T12:06:43.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:43.090+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:43.091938+0000) 2019-10-13T12:06:45.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:45.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.746336+0000) 2019-10-13T12:06:45.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:45.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.746336+0000) 2019-10-13T12:06:46.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:46.005+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.006311+0000) 2019-10-13T12:06:46.256 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:46.254+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.255492+0000) 2019-10-13T12:06:46.257 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:46.254+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.255739+0000) 2019-10-13T12:06:46.257 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:46.254+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.255949+0000) 2019-10-13T12:06:46.258 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:46.254+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.255836+0000) 2019-10-13T12:06:46.258 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:46.254+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.256026+0000) 2019-10-13T12:06:46.258 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:46.254+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.256241+0000) 2019-10-13T12:06:46.258 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:46.254+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.256383+0000) 2019-10-13T12:06:46.259 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:46.256+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:46.258645+0000) 2019-10-13T12:06:46.263 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:46.260+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:48.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:48.650+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:48.651192+0000) 2019-10-13T12:06:48.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:48.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:48.981890+0000) 2019-10-13T12:06:51.265 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:51.262+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.264077+0000) 2019-10-13T12:06:51.265 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:51.263+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.264406+0000) 2019-10-13T12:06:51.266 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:51.264+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.265405+0000) 2019-10-13T12:06:51.266 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:51.263+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.264684+0000) 2019-10-13T12:06:51.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:51.263+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.264843+0000) 2019-10-13T12:06:51.267 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:51.263+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.265371+0000) 2019-10-13T12:06:51.267 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:51.264+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.265569+0000) 2019-10-13T12:06:51.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:51.266+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.267329+0000) 2019-10-13T12:06:51.274 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:51.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224399 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:51.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:51.601+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.602551+0000) 2019-10-13T12:06:53.088 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:53.087+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:53.087813+0000) 2019-10-13T12:06:53.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:53.091+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:53.092119+0000) 2019-10-13T12:06:55.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:55.745+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.746516+0000) 2019-10-13T12:06:55.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:55.745+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.746514+0000) 2019-10-13T12:06:56.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:56.008+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.008782+0000) 2019-10-13T12:06:56.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:56.274+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.275551+0000) 2019-10-13T12:06:56.277 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:56.274+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.275711+0000) 2019-10-13T12:06:56.278 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:06:56.275+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.276029+0000) 2019-10-13T12:06:56.279 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:06:56.274+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.276160+0000) 2019-10-13T12:06:56.279 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:06:56.274+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.276266+0000) 2019-10-13T12:06:56.279 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:06:56.274+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.276351+0000) 2019-10-13T12:06:56.280 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:06:56.275+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.276666+0000) 2019-10-13T12:06:56.281 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:06:56.277+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:56.277776+0000) 2019-10-13T12:06:56.286 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:06:56.283+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:06:58.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:06:58.650+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:58.651363+0000) 2019-10-13T12:06:58.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:06:58.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:58.982083+0000) 2019-10-13T12:07:01.288 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:01.285+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.286797+0000) 2019-10-13T12:07:01.288 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:01.285+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.286960+0000) 2019-10-13T12:07:01.288 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:01.285+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.287211+0000) 2019-10-13T12:07:01.289 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:01.286+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.287667+0000) 2019-10-13T12:07:01.289 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:01.286+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.287658+0000) 2019-10-13T12:07:01.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:01.286+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.288148+0000) 2019-10-13T12:07:01.290 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:01.287+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.288459+0000) 2019-10-13T12:07:01.290 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:01.289+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.290297+0000) 2019-10-13T12:07:01.297 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:01.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:01.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:01.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.602732+0000) 2019-10-13T12:07:03.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:03.087+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:03.088021+0000) 2019-10-13T12:07:03.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:03.091+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:03.092263+0000) 2019-10-13T12:07:05.747 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:05.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.746696+0000) 2019-10-13T12:07:05.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:05.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.746697+0000) 2019-10-13T12:07:06.012 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:06.010+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.011111+0000) 2019-10-13T12:07:06.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:06.297+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.297669+0000) 2019-10-13T12:07:06.299 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:06.297+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.298141+0000) 2019-10-13T12:07:06.299 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:06.297+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.298155+0000) 2019-10-13T12:07:06.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:06.297+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.298531+0000) 2019-10-13T12:07:06.300 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:06.298+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.298668+0000) 2019-10-13T12:07:06.300 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:06.298+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.298860+0000) 2019-10-13T12:07:06.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:06.298+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.298934+0000) 2019-10-13T12:07:06.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:06.299+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:06.300138+0000) 2019-10-13T12:07:06.307 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:06.306+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224493 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:08.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:08.651+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:08.651577+0000) 2019-10-13T12:07:08.719 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:07:08.719 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:08.724 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:07:08.724 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:08.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:08.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:08.982270+0000) 2019-10-13T12:07:11.309 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:11.307+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.308464+0000) 2019-10-13T12:07:11.309 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:11.308+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.308452+0000) 2019-10-13T12:07:11.310 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:11.308+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.308705+0000) 2019-10-13T12:07:11.310 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:11.308+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.308910+0000) 2019-10-13T12:07:11.310 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:11.308+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.309033+0000) 2019-10-13T12:07:11.311 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:11.309+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.309415+0000) 2019-10-13T12:07:11.311 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:11.309+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.309688+0000) 2019-10-13T12:07:11.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:11.311+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.312416+0000) 2019-10-13T12:07:11.318 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:11.317+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224523 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:11.603 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:11.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.602903+0000) 2019-10-13T12:07:13.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:13.088+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:13.088207+0000) 2019-10-13T12:07:13.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:13.091+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:13.092407+0000) 2019-10-13T12:07:15.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:15.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.746887+0000) 2019-10-13T12:07:15.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:15.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.746889+0000) 2019-10-13T12:07:16.014 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:16.013+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.013376+0000) 2019-10-13T12:07:16.320 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:16.318+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.319207+0000) 2019-10-13T12:07:16.321 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:16.319+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.319838+0000) 2019-10-13T12:07:16.321 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:16.319+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.319427+0000) 2019-10-13T12:07:16.321 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:16.319+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.319806+0000) 2019-10-13T12:07:16.321 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:16.319+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.320262+0000) 2019-10-13T12:07:16.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:16.319+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.320390+0000) 2019-10-13T12:07:16.322 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:16.320+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.320482+0000) 2019-10-13T12:07:16.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:16.320+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:16.321505+0000) 2019-10-13T12:07:16.329 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:16.327+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:18.652 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:18.651+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:18.651768+0000) 2019-10-13T12:07:18.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:18.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:18.982433+0000) 2019-10-13T12:07:21.331 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:21.328+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.330130+0000) 2019-10-13T12:07:21.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:21.328+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.330272+0000) 2019-10-13T12:07:21.332 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:21.329+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.330439+0000) 2019-10-13T12:07:21.332 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:21.329+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.330710+0000) 2019-10-13T12:07:21.332 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:21.330+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.330787+0000) 2019-10-13T12:07:21.333 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:21.329+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.331343+0000) 2019-10-13T12:07:21.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:21.330+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.331577+0000) 2019-10-13T12:07:21.334 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:21.332+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.333425+0000) 2019-10-13T12:07:21.340 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:21.338+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:21.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:21.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.603078+0000) 2019-10-13T12:07:23.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:23.087+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:23.088429+0000) 2019-10-13T12:07:23.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:23.092+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:23.092585+0000) 2019-10-13T12:07:25.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:25.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.747086+0000) 2019-10-13T12:07:25.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:25.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.747087+0000) 2019-10-13T12:07:26.016 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:26.014+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.015753+0000) 2019-10-13T12:07:26.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:26.340+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.341437+0000) 2019-10-13T12:07:26.342 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:26.340+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.341631+0000) 2019-10-13T12:07:26.343 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:26.340+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.342644+0000) 2019-10-13T12:07:26.343 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:26.340+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.341986+0000) 2019-10-13T12:07:26.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:26.340+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.342139+0000) 2019-10-13T12:07:26.344 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:26.340+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.342231+0000) 2019-10-13T12:07:26.344 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:26.341+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.342688+0000) 2019-10-13T12:07:26.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:26.341+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:26.343556+0000) 2019-10-13T12:07:26.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:26.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224617 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:28.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:28.651+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:28.651948+0000) 2019-10-13T12:07:28.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:28.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:28.982623+0000) 2019-10-13T12:07:31.352 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:31.350+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.352199+0000) 2019-10-13T12:07:31.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:31.350+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.352118+0000) 2019-10-13T12:07:31.354 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:31.350+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.352255+0000) 2019-10-13T12:07:31.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:31.351+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.352646+0000) 2019-10-13T12:07:31.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:31.351+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.352873+0000) 2019-10-13T12:07:31.354 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:31.351+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.352912+0000) 2019-10-13T12:07:31.355 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:31.352+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.353437+0000) 2019-10-13T12:07:31.356 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:31.353+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.355660+0000) 2019-10-13T12:07:31.362 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:31.360+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224647 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:31.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:31.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.603219+0000) 2019-10-13T12:07:33.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:33.087+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:33.088615+0000) 2019-10-13T12:07:33.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:33.091+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:33.092768+0000) 2019-10-13T12:07:35.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:35.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.747294+0000) 2019-10-13T12:07:35.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:35.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.747289+0000) 2019-10-13T12:07:36.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:36.017+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.018016+0000) 2019-10-13T12:07:36.365 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:36.362+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.363670+0000) 2019-10-13T12:07:36.365 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:36.362+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.363930+0000) 2019-10-13T12:07:36.365 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:36.362+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.364389+0000) 2019-10-13T12:07:36.366 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:36.362+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.364420+0000) 2019-10-13T12:07:36.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:36.363+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.364621+0000) 2019-10-13T12:07:36.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:36.363+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.364735+0000) 2019-10-13T12:07:36.367 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:36.363+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.365230+0000) 2019-10-13T12:07:36.367 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:36.364+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:36.366316+0000) 2019-10-13T12:07:36.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:36.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224678 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:38.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:38.651+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:38.652152+0000) 2019-10-13T12:07:38.881 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:07:38.881 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:38.885 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:07:38.886 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:38.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:38.981+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:38.982787+0000) 2019-10-13T12:07:41.374 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:41.372+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.373567+0000) 2019-10-13T12:07:41.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:41.373+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.374315+0000) 2019-10-13T12:07:41.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:41.372+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.374024+0000) 2019-10-13T12:07:41.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:41.373+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.374525+0000) 2019-10-13T12:07:41.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:41.373+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.374722+0000) 2019-10-13T12:07:41.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:41.373+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.374836+0000) 2019-10-13T12:07:41.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:41.373+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.375058+0000) 2019-10-13T12:07:41.377 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:41.375+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.376064+0000) 2019-10-13T12:07:41.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:41.382+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224708 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:41.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:41.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.603362+0000) 2019-10-13T12:07:43.089 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:43.087+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:43.088794+0000) 2019-10-13T12:07:43.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:43.092+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:43.092921+0000) 2019-10-13T12:07:45.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:45.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.747488+0000) 2019-10-13T12:07:45.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:45.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.747490+0000) 2019-10-13T12:07:46.021 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:46.019+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.020554+0000) 2019-10-13T12:07:46.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:46.384+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.385776+0000) 2019-10-13T12:07:46.393 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:46.384+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.385868+0000) 2019-10-13T12:07:46.394 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:46.384+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.385989+0000) 2019-10-13T12:07:46.394 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:46.384+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.386229+0000) 2019-10-13T12:07:46.394 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:46.384+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.386384+0000) 2019-10-13T12:07:46.394 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:46.385+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.386574+0000) 2019-10-13T12:07:46.394 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:46.384+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.385560+0000) 2019-10-13T12:07:46.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:46.387+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:46.388022+0000) 2019-10-13T12:07:46.396 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:46.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:48.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:48.651+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:48.652329+0000) 2019-10-13T12:07:48.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:48.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:48.982972+0000) 2019-10-13T12:07:51.397 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:51.395+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.396074+0000) 2019-10-13T12:07:51.397 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:51.395+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.396206+0000) 2019-10-13T12:07:51.399 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:51.395+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.396531+0000) 2019-10-13T12:07:51.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:51.396+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.397005+0000) 2019-10-13T12:07:51.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:51.396+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.397261+0000) 2019-10-13T12:07:51.400 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:51.397+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.397496+0000) 2019-10-13T12:07:51.400 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:51.397+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.397723+0000) 2019-10-13T12:07:51.400 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:51.398+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.399670+0000) 2019-10-13T12:07:51.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:51.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224771 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:51.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:51.603+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.603484+0000) 2019-10-13T12:07:53.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:53.088+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:53.088970+0000) 2019-10-13T12:07:53.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:53.092+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:53.093057+0000) 2019-10-13T12:07:55.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:55.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.747687+0000) 2019-10-13T12:07:55.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:55.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.747687+0000) 2019-10-13T12:07:56.024 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:56.023+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.023545+0000) 2019-10-13T12:07:56.407 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:07:56.405+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.405492+0000) 2019-10-13T12:07:56.407 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:56.405+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.405648+0000) 2019-10-13T12:07:56.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:07:56.405+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.405851+0000) 2019-10-13T12:07:56.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:07:56.405+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.406027+0000) 2019-10-13T12:07:56.407 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:56.405+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.406158+0000) 2019-10-13T12:07:56.408 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:07:56.405+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.406323+0000) 2019-10-13T12:07:56.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:07:56.406+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.406866+0000) 2019-10-13T12:07:56.409 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:07:56.408+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:56.408847+0000) 2019-10-13T12:07:56.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:07:56.414+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:07:58.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:07:58.652+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:58.652535+0000) 2019-10-13T12:07:58.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:07:58.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:58.983121+0000) 2019-10-13T12:08:01.418 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:01.416+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.416701+0000) 2019-10-13T12:08:01.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:01.416+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.416954+0000) 2019-10-13T12:08:01.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:01.416+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.416973+0000) 2019-10-13T12:08:01.418 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:01.416+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.417071+0000) 2019-10-13T12:08:01.419 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:01.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.418078+0000) 2019-10-13T12:08:01.419 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:01.417+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.418079+0000) 2019-10-13T12:08:01.419 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:01.417+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.418232+0000) 2019-10-13T12:08:01.420 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:01.419+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.420055+0000) 2019-10-13T12:08:01.425 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:01.423+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224834 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:01.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:01.603+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.603607+0000) 2019-10-13T12:08:03.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:03.088+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:03.089132+0000) 2019-10-13T12:08:03.093 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:03.092+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:03.093217+0000) 2019-10-13T12:08:05.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:05.746+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.747869+0000) 2019-10-13T12:08:05.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:05.746+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.747874+0000) 2019-10-13T12:08:06.027 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:06.026+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.026572+0000) 2019-10-13T12:08:06.426 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:06.423+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.425172+0000) 2019-10-13T12:08:06.426 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:06.424+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.425404+0000) 2019-10-13T12:08:06.427 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:06.423+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.425258+0000) 2019-10-13T12:08:06.427 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:06.424+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.425516+0000) 2019-10-13T12:08:06.428 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:06.425+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.425845+0000) 2019-10-13T12:08:06.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:06.425+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.426677+0000) 2019-10-13T12:08:06.428 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:06.425+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.426799+0000) 2019-10-13T12:08:06.429 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:06.428+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:06.428783+0000) 2019-10-13T12:08:06.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:06.431+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224865 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:08.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:08.651+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:08.652732+0000) 2019-10-13T12:08:08.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:08.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:08.983322+0000) 2019-10-13T12:08:09.019 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:08:09.019 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:09.023 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:08:09.023 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:11.436 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:11.433+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.435357+0000) 2019-10-13T12:08:11.437 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:11.434+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.435502+0000) 2019-10-13T12:08:11.437 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:11.434+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.435674+0000) 2019-10-13T12:08:11.438 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:11.435+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.436051+0000) 2019-10-13T12:08:11.438 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:11.434+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.436077+0000) 2019-10-13T12:08:11.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:11.434+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.436192+0000) 2019-10-13T12:08:11.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:11.435+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.436405+0000) 2019-10-13T12:08:11.439 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:11.437+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.438239+0000) 2019-10-13T12:08:11.445 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:11.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224895 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:11.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:11.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.603773+0000) 2019-10-13T12:08:13.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:13.088+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:13.089317+0000) 2019-10-13T12:08:13.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:13.093+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:13.093384+0000) 2019-10-13T12:08:15.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:15.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.748042+0000) 2019-10-13T12:08:15.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:15.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.748046+0000) 2019-10-13T12:08:16.029 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:16.028+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.029055+0000) 2019-10-13T12:08:16.447 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:16.444+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.446295+0000) 2019-10-13T12:08:16.447 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:16.445+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.446541+0000) 2019-10-13T12:08:16.448 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:16.445+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.446807+0000) 2019-10-13T12:08:16.448 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:16.445+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.447428+0000) 2019-10-13T12:08:16.449 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:16.445+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.446996+0000) 2019-10-13T12:08:16.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:16.445+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.447330+0000) 2019-10-13T12:08:16.449 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:16.446+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.447700+0000) 2019-10-13T12:08:16.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:16.447+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:16.449104+0000) 2019-10-13T12:08:16.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:16.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:18.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:18.651+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:18.652865+0000) 2019-10-13T12:08:18.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:18.982+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:18.983517+0000) 2019-10-13T12:08:21.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:21.456+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.457503+0000) 2019-10-13T12:08:21.459 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:21.456+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.457731+0000) 2019-10-13T12:08:21.459 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:21.456+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.457949+0000) 2019-10-13T12:08:21.460 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:21.456+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.458226+0000) 2019-10-13T12:08:21.460 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:21.456+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.458260+0000) 2019-10-13T12:08:21.460 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:21.456+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.458355+0000) 2019-10-13T12:08:21.460 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:21.456+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.458178+0000) 2019-10-13T12:08:21.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:21.459+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.461361+0000) 2019-10-13T12:08:21.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:21.465+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:21.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:21.602+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.603935+0000) 2019-10-13T12:08:23.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:23.088+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:23.089467+0000) 2019-10-13T12:08:23.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:23.092+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:23.093582+0000) 2019-10-13T12:08:25.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:25.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.748238+0000) 2019-10-13T12:08:25.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:25.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.748239+0000) 2019-10-13T12:08:26.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:26.030+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.031603+0000) 2019-10-13T12:08:26.470 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:26.467+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.469226+0000) 2019-10-13T12:08:26.470 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:26.467+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.469319+0000) 2019-10-13T12:08:26.471 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:26.468+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.469501+0000) 2019-10-13T12:08:26.471 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:26.468+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.469729+0000) 2019-10-13T12:08:26.471 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:26.468+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.469804+0000) 2019-10-13T12:08:26.472 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:26.468+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.469981+0000) 2019-10-13T12:08:26.472 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:26.468+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.469869+0000) 2019-10-13T12:08:26.475 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:26.473+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:26.475046+0000) 2019-10-13T12:08:26.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:26.477+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 224989 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:28.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:28.652+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:28.653044+0000) 2019-10-13T12:08:28.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:28.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:28.983697+0000) 2019-10-13T12:08:31.482 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:31.479+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.481107+0000) 2019-10-13T12:08:31.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:31.479+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.481342+0000) 2019-10-13T12:08:31.482 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:31.479+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.481350+0000) 2019-10-13T12:08:31.483 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:31.480+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.481637+0000) 2019-10-13T12:08:31.483 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:31.480+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.481724+0000) 2019-10-13T12:08:31.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:31.480+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.481733+0000) 2019-10-13T12:08:31.484 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:31.480+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.481917+0000) 2019-10-13T12:08:31.484 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:31.482+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.483070+0000) 2019-10-13T12:08:31.491 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:31.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225019 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:31.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:31.603+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.604105+0000) 2019-10-13T12:08:33.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:33.089+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:33.089645+0000) 2019-10-13T12:08:33.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:33.093+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:33.093794+0000) 2019-10-13T12:08:35.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:35.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.748439+0000) 2019-10-13T12:08:35.749 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:35.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.748440+0000) 2019-10-13T12:08:36.034 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:36.033+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.033986+0000) 2019-10-13T12:08:36.492 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:36.491+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.492043+0000) 2019-10-13T12:08:36.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:36.491+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.492010+0000) 2019-10-13T12:08:36.494 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:36.491+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.492195+0000) 2019-10-13T12:08:36.494 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:36.491+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.492338+0000) 2019-10-13T12:08:36.494 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:36.492+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.492487+0000) 2019-10-13T12:08:36.494 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:36.492+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.492763+0000) 2019-10-13T12:08:36.494 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:36.492+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.493025+0000) 2019-10-13T12:08:36.496 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:36.494+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:36.495687+0000) 2019-10-13T12:08:36.502 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:36.501+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:38.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:38.652+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:38.653211+0000) 2019-10-13T12:08:38.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:38.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:38.983864+0000) 2019-10-13T12:08:39.173 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:08:39.173 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:39.177 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:08:39.177 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:41.504 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:41.502+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.503697+0000) 2019-10-13T12:08:41.505 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:41.503+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.503624+0000) 2019-10-13T12:08:41.506 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:41.503+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.503817+0000) 2019-10-13T12:08:41.506 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:41.503+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.504056+0000) 2019-10-13T12:08:41.506 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:41.503+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.504319+0000) 2019-10-13T12:08:41.506 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:41.504+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.504453+0000) 2019-10-13T12:08:41.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:41.504+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.504649+0000) 2019-10-13T12:08:41.508 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:41.506+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.507144+0000) 2019-10-13T12:08:41.514 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:41.512+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225082 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:41.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:41.603+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.604272+0000) 2019-10-13T12:08:43.090 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:43.089+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:43.089835+0000) 2019-10-13T12:08:43.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:43.093+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:43.093979+0000) 2019-10-13T12:08:45.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:45.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.748639+0000) 2019-10-13T12:08:45.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:45.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.748641+0000) 2019-10-13T12:08:46.038 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:46.036+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.036429+0000) 2019-10-13T12:08:46.516 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:46.514+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.514852+0000) 2019-10-13T12:08:46.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:46.515+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.515407+0000) 2019-10-13T12:08:46.517 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:46.515+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.515597+0000) 2019-10-13T12:08:46.518 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:46.514+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.515215+0000) 2019-10-13T12:08:46.518 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:46.515+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.516021+0000) 2019-10-13T12:08:46.518 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:46.515+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.516307+0000) 2019-10-13T12:08:46.518 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:46.516+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.516641+0000) 2019-10-13T12:08:46.519 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:46.516+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:46.517510+0000) 2019-10-13T12:08:46.525 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:46.524+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225113 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:48.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:48.652+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:48.653356+0000) 2019-10-13T12:08:48.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:48.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:48.984048+0000) 2019-10-13T12:08:51.527 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:51.525+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.526518+0000) 2019-10-13T12:08:51.528 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:51.526+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.526942+0000) 2019-10-13T12:08:51.529 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:51.525+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.526908+0000) 2019-10-13T12:08:51.529 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:51.525+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.527167+0000) 2019-10-13T12:08:51.529 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:51.525+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.527384+0000) 2019-10-13T12:08:51.529 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:51.526+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.527662+0000) 2019-10-13T12:08:51.529 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:51.526+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.527990+0000) 2019-10-13T12:08:51.530 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:51.529+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.530066+0000) 2019-10-13T12:08:51.536 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:51.534+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225143 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:51.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:51.603+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.604416+0000) 2019-10-13T12:08:53.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:53.088+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:53.090039+0000) 2019-10-13T12:08:53.094 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:53.093+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:53.094159+0000) 2019-10-13T12:08:55.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:55.747+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.748838+0000) 2019-10-13T12:08:55.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:55.747+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.748838+0000) 2019-10-13T12:08:56.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:56.038+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.038664+0000) 2019-10-13T12:08:56.539 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:56.535+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.537388+0000) 2019-10-13T12:08:56.539 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:08:56.536+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.537727+0000) 2019-10-13T12:08:56.539 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:08:56.536+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.537530+0000) 2019-10-13T12:08:56.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:08:56.536+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.538256+0000) 2019-10-13T12:08:56.540 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:08:56.537+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.537959+0000) 2019-10-13T12:08:56.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:56.537+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.538570+0000) 2019-10-13T12:08:56.540 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:08:56.537+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.538818+0000) 2019-10-13T12:08:56.541 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:08:56.540+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:56.540924+0000) 2019-10-13T12:08:56.547 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:08:56.545+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225174 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:08:58.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:08:58.652+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:58.653515+0000) 2019-10-13T12:08:58.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:08:58.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:58.984212+0000) 2019-10-13T12:09:01.549 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:01.546+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.548201+0000) 2019-10-13T12:09:01.549 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:01.547+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.548430+0000) 2019-10-13T12:09:01.550 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:01.547+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.548713+0000) 2019-10-13T12:09:01.550 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:01.547+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.548874+0000) 2019-10-13T12:09:01.550 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:01.548+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.549533+0000) 2019-10-13T12:09:01.551 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:01.548+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.549564+0000) 2019-10-13T12:09:01.551 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:01.548+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.549814+0000) 2019-10-13T12:09:01.551 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:01.550+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.551320+0000) 2019-10-13T12:09:01.558 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:01.555+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225206 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:01.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:01.603+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.604582+0000) 2019-10-13T12:09:03.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:03.089+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:03.090181+0000) 2019-10-13T12:09:03.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:03.094+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:03.094341+0000) 2019-10-13T12:09:05.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:05.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.749043+0000) 2019-10-13T12:09:05.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:05.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.749038+0000) 2019-10-13T12:09:06.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:06.040+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.041105+0000) 2019-10-13T12:09:06.559 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:06.557+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.559108+0000) 2019-10-13T12:09:06.560 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:06.557+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.559135+0000) 2019-10-13T12:09:06.561 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:06.557+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.559278+0000) 2019-10-13T12:09:06.561 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:06.558+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.559627+0000) 2019-10-13T12:09:06.561 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:06.558+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.559721+0000) 2019-10-13T12:09:06.562 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:06.559+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.560464+0000) 2019-10-13T12:09:06.562 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:06.559+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.560874+0000) 2019-10-13T12:09:06.562 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:06.559+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:06.561342+0000) 2019-10-13T12:09:06.569 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:06.567+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:08.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:08.653+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:08.653621+0000) 2019-10-13T12:09:08.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:08.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:08.984357+0000) 2019-10-13T12:09:09.274 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:09:09.274 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:09.283 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:09:09.283 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:11.571 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:11.568+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.570069+0000) 2019-10-13T12:09:11.571 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:11.568+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.570238+0000) 2019-10-13T12:09:11.571 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:11.569+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.570420+0000) 2019-10-13T12:09:11.572 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:11.569+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.571425+0000) 2019-10-13T12:09:11.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:11.569+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.570892+0000) 2019-10-13T12:09:11.573 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:11.569+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.570993+0000) 2019-10-13T12:09:11.573 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:11.569+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.570683+0000) 2019-10-13T12:09:11.574 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:11.571+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.573716+0000) 2019-10-13T12:09:11.579 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:11.577+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225267 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:11.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:11.604+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.604720+0000) 2019-10-13T12:09:13.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:13.089+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:13.090351+0000) 2019-10-13T12:09:13.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:13.093+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:13.094535+0000) 2019-10-13T12:09:15.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:15.748+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.749246+0000) 2019-10-13T12:09:15.750 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:15.748+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.749252+0000) 2019-10-13T12:09:16.044 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:16.042+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.043455+0000) 2019-10-13T12:09:16.582 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:16.579+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.580605+0000) 2019-10-13T12:09:16.582 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:16.579+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.580847+0000) 2019-10-13T12:09:16.583 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:16.579+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.581013+0000) 2019-10-13T12:09:16.583 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:16.579+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.581387+0000) 2019-10-13T12:09:16.583 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:16.580+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.581575+0000) 2019-10-13T12:09:16.583 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:16.580+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.581706+0000) 2019-10-13T12:09:16.584 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:16.579+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.581386+0000) 2019-10-13T12:09:16.584 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:16.580+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:16.582611+0000) 2019-10-13T12:09:16.590 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:16.588+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:18.654 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:18.653+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:18.653787+0000) 2019-10-13T12:09:18.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:18.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:18.984527+0000) 2019-10-13T12:09:21.592 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:21.590+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.591170+0000) 2019-10-13T12:09:21.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:21.590+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.591301+0000) 2019-10-13T12:09:21.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:21.591+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.591587+0000) 2019-10-13T12:09:21.593 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:21.591+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.591635+0000) 2019-10-13T12:09:21.594 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:21.592+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.592745+0000) 2019-10-13T12:09:21.594 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:21.592+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.592685+0000) 2019-10-13T12:09:21.594 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:21.592+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.592835+0000) 2019-10-13T12:09:21.596 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:21.595+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.595780+0000) 2019-10-13T12:09:21.601 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:21.600+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225330 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:21.605 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:21.604+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.604812+0000) 2019-10-13T12:09:23.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:23.090+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:23.090471+0000) 2019-10-13T12:09:23.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:23.093+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:23.094701+0000) 2019-10-13T12:09:25.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:25.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.749444+0000) 2019-10-13T12:09:25.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:25.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.749444+0000) 2019-10-13T12:09:26.046 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:26.045+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.046007+0000) 2019-10-13T12:09:26.603 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:26.602+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.602788+0000) 2019-10-13T12:09:26.604 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:26.602+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.602733+0000) 2019-10-13T12:09:26.604 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:26.602+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.603052+0000) 2019-10-13T12:09:26.604 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:26.602+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.603098+0000) 2019-10-13T12:09:26.605 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:26.602+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.603273+0000) 2019-10-13T12:09:26.605 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:26.603+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.603672+0000) 2019-10-13T12:09:26.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:26.603+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.603859+0000) 2019-10-13T12:09:26.606 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:26.605+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:26.606279+0000) 2019-10-13T12:09:26.612 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:26.611+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225361 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:28.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:28.653+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:28.653961+0000) 2019-10-13T12:09:28.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:28.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:28.984715+0000) 2019-10-13T12:09:31.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:31.604+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.604958+0000) 2019-10-13T12:09:31.614 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:31.612+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.613365+0000) 2019-10-13T12:09:31.614 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:31.613+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.613710+0000) 2019-10-13T12:09:31.614 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:31.613+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.613889+0000) 2019-10-13T12:09:31.615 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:31.613+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.614288+0000) 2019-10-13T12:09:31.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:31.614+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.614487+0000) 2019-10-13T12:09:31.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:31.614+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.614515+0000) 2019-10-13T12:09:31.616 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:31.614+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.614794+0000) 2019-10-13T12:09:31.617 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:31.616+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.617323+0000) 2019-10-13T12:09:31.622 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:31.621+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225391 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:33.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:33.090+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:33.090629+0000) 2019-10-13T12:09:33.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:33.094+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:33.094879+0000) 2019-10-13T12:09:35.750 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:35.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.749655+0000) 2019-10-13T12:09:35.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:35.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.749655+0000) 2019-10-13T12:09:36.049 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:36.047+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.048353+0000) 2019-10-13T12:09:36.624 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:36.622+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.623630+0000) 2019-10-13T12:09:36.626 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:36.623+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.624533+0000) 2019-10-13T12:09:36.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:36.622+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.624221+0000) 2019-10-13T12:09:36.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:36.623+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.624535+0000) 2019-10-13T12:09:36.626 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:36.623+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.624787+0000) 2019-10-13T12:09:36.627 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:36.623+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.624957+0000) 2019-10-13T12:09:36.627 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:36.623+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.625336+0000) 2019-10-13T12:09:36.627 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:36.625+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:36.626204+0000) 2019-10-13T12:09:36.633 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:36.631+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:38.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:38.653+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:38.654117+0000) 2019-10-13T12:09:38.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:38.983+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:38.984903+0000) 2019-10-13T12:09:39.432 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:09:39.432 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:39.437 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:09:39.437 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:41.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:41.604+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.605118+0000) 2019-10-13T12:09:41.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:41.633+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.634625+0000) 2019-10-13T12:09:41.636 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:41.633+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.635109+0000) 2019-10-13T12:09:41.636 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:41.634+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.635520+0000) 2019-10-13T12:09:41.637 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:41.634+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.635716+0000) 2019-10-13T12:09:41.637 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:41.634+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.636029+0000) 2019-10-13T12:09:41.637 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:41.634+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.636194+0000) 2019-10-13T12:09:41.638 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:41.635+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.636091+0000) 2019-10-13T12:09:41.638 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:41.636+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.637620+0000) 2019-10-13T12:09:41.645 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:41.642+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225454 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:43.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:43.089+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:43.090826+0000) 2019-10-13T12:09:43.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:43.094+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:43.095065+0000) 2019-10-13T12:09:45.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:45.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.749822+0000) 2019-10-13T12:09:45.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:45.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.749822+0000) 2019-10-13T12:09:46.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:46.050+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.050834+0000) 2019-10-13T12:09:46.647 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:46.644+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.646138+0000) 2019-10-13T12:09:46.648 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:46.645+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.646627+0000) 2019-10-13T12:09:46.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:46.645+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.646705+0000) 2019-10-13T12:09:46.648 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:46.645+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.646449+0000) 2019-10-13T12:09:46.648 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:46.645+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.647255+0000) 2019-10-13T12:09:46.649 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:46.645+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.647379+0000) 2019-10-13T12:09:46.649 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:46.647+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.647778+0000) 2019-10-13T12:09:46.650 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:46.648+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:46.649461+0000) 2019-10-13T12:09:46.656 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:46.653+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225485 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:48.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:48.652+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:48.654278+0000) 2019-10-13T12:09:48.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:48.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:48.985113+0000) 2019-10-13T12:09:51.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:51.604+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.605331+0000) 2019-10-13T12:09:51.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:51.655+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.657129+0000) 2019-10-13T12:09:51.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:51.655+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.657276+0000) 2019-10-13T12:09:51.658 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:51.656+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.657626+0000) 2019-10-13T12:09:51.659 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:51.657+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.658391+0000) 2019-10-13T12:09:51.659 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:51.656+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.657937+0000) 2019-10-13T12:09:51.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:51.656+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.658053+0000) 2019-10-13T12:09:51.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:51.657+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.658772+0000) 2019-10-13T12:09:51.660 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:51.658+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.659368+0000) 2019-10-13T12:09:51.667 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:51.665+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225515 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:53.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:53.090+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:53.091040+0000) 2019-10-13T12:09:53.095 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:53.095+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:53.095233+0000) 2019-10-13T12:09:55.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:55.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.750020+0000) 2019-10-13T12:09:55.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:55.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.750015+0000) 2019-10-13T12:09:56.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:56.052+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.053119+0000) 2019-10-13T12:09:56.669 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:09:56.667+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.668724+0000) 2019-10-13T12:09:56.670 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:09:56.666+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.668379+0000) 2019-10-13T12:09:56.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:56.667+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.668989+0000) 2019-10-13T12:09:56.671 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:09:56.667+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.669365+0000) 2019-10-13T12:09:56.671 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:09:56.668+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.669517+0000) 2019-10-13T12:09:56.671 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:09:56.668+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.669689+0000) 2019-10-13T12:09:56.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:56.668+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.669984+0000) 2019-10-13T12:09:56.672 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:09:56.669+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:56.670980+0000) 2019-10-13T12:09:56.678 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:09:56.676+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:09:58.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:09:58.653+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:58.654468+0000) 2019-10-13T12:09:58.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:09:58.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:58.985343+0000) 2019-10-13T12:10:01.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:01.605+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.605536+0000) 2019-10-13T12:10:01.680 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:01.678+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.679800+0000) 2019-10-13T12:10:01.681 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:01.678+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.680186+0000) 2019-10-13T12:10:01.682 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:01.679+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.680894+0000) 2019-10-13T12:10:01.682 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:01.679+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.680392+0000) 2019-10-13T12:10:01.682 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:01.679+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.680639+0000) 2019-10-13T12:10:01.682 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:01.679+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.681005+0000) 2019-10-13T12:10:01.683 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:01.679+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.681111+0000) 2019-10-13T12:10:01.683 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:01.680+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.682326+0000) 2019-10-13T12:10:01.690 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:01.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:03.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:03.090+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:03.091211+0000) 2019-10-13T12:10:03.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:03.094+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:03.095370+0000) 2019-10-13T12:10:05.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:05.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.750220+0000) 2019-10-13T12:10:05.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:05.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.750219+0000) 2019-10-13T12:10:06.056 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:06.054+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.055469+0000) 2019-10-13T12:10:06.692 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:06.690+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.691078+0000) 2019-10-13T12:10:06.693 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:06.690+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.691291+0000) 2019-10-13T12:10:06.693 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:06.691+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.691512+0000) 2019-10-13T12:10:06.693 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:06.691+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.691571+0000) 2019-10-13T12:10:06.693 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:06.691+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.691744+0000) 2019-10-13T12:10:06.694 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:06.690+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.692517+0000) 2019-10-13T12:10:06.694 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:06.692+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.692432+0000) 2019-10-13T12:10:06.696 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:06.694+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:06.695774+0000) 2019-10-13T12:10:06.700 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:06.699+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225609 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:08.655 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:08.654+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:08.654716+0000) 2019-10-13T12:10:08.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:08.985+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:08.985530+0000) 2019-10-13T12:10:09.545 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:10:09.545 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:09.550 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:10:09.551 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:11.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:11.605+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.605741+0000) 2019-10-13T12:10:11.703 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:11.701+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.702002+0000) 2019-10-13T12:10:11.704 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:11.701+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.701993+0000) 2019-10-13T12:10:11.704 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:11.701+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.702364+0000) 2019-10-13T12:10:11.704 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:11.702+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.702523+0000) 2019-10-13T12:10:11.704 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:11.702+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.702644+0000) 2019-10-13T12:10:11.704 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:11.702+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.702800+0000) 2019-10-13T12:10:11.705 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:11.702+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.702992+0000) 2019-10-13T12:10:11.706 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:11.705+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.705838+0000) 2019-10-13T12:10:11.712 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:11.710+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225639 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:13.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:13.091+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:13.091418+0000) 2019-10-13T12:10:13.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:13.094+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:13.095570+0000) 2019-10-13T12:10:15.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:15.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.750404+0000) 2019-10-13T12:10:15.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:15.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.750389+0000) 2019-10-13T12:10:16.058 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:16.057+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.057771+0000) 2019-10-13T12:10:16.713 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:16.712+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.712621+0000) 2019-10-13T12:10:16.714 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:16.712+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.713024+0000) 2019-10-13T12:10:16.715 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:16.712+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.713006+0000) 2019-10-13T12:10:16.715 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:16.713+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.713693+0000) 2019-10-13T12:10:16.715 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:16.713+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.713916+0000) 2019-10-13T12:10:16.715 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:16.713+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.713987+0000) 2019-10-13T12:10:16.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:16.713+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.714133+0000) 2019-10-13T12:10:16.716 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:16.714+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:16.715500+0000) 2019-10-13T12:10:16.723 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:16.721+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225670 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:18.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:18.654+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:18.654925+0000) 2019-10-13T12:10:18.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:18.985+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:18.985699+0000) 2019-10-13T12:10:21.606 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:21.604+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.605930+0000) 2019-10-13T12:10:21.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:21.722+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.723900+0000) 2019-10-13T12:10:21.725 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:21.722+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.724117+0000) 2019-10-13T12:10:21.725 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:21.723+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.724707+0000) 2019-10-13T12:10:21.726 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:21.723+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.725055+0000) 2019-10-13T12:10:21.726 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:21.724+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.725448+0000) 2019-10-13T12:10:21.726 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:21.723+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.725277+0000) 2019-10-13T12:10:21.727 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:21.724+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.725732+0000) 2019-10-13T12:10:21.728 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:21.726+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.727445+0000) 2019-10-13T12:10:21.734 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:21.731+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225702 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:23.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:23.090+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:23.091583+0000) 2019-10-13T12:10:23.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:23.095+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:23.095793+0000) 2019-10-13T12:10:25.751 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:25.749+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.750590+0000) 2019-10-13T12:10:25.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:25.749+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.750591+0000) 2019-10-13T12:10:26.061 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:26.059+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.060179+0000) 2019-10-13T12:10:26.735 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:26.733+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.734821+0000) 2019-10-13T12:10:26.736 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:26.733+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.735031+0000) 2019-10-13T12:10:26.737 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:26.733+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.735317+0000) 2019-10-13T12:10:26.737 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:26.734+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.735740+0000) 2019-10-13T12:10:26.737 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:26.734+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.735896+0000) 2019-10-13T12:10:26.737 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:26.734+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.736141+0000) 2019-10-13T12:10:26.738 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:26.735+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.736485+0000) 2019-10-13T12:10:26.738 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:26.736+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:26.737582+0000) 2019-10-13T12:10:26.745 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:26.743+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225733 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:28.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:28.654+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:28.655102+0000) 2019-10-13T12:10:28.986 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:28.985+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:28.985886+0000) 2019-10-13T12:10:31.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:31.605+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.606098+0000) 2019-10-13T12:10:31.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:31.745+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.746422+0000) 2019-10-13T12:10:31.748 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:31.745+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.746504+0000) 2019-10-13T12:10:31.748 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:31.745+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.746744+0000) 2019-10-13T12:10:31.748 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:31.745+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.747062+0000) 2019-10-13T12:10:31.749 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:31.746+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.747463+0000) 2019-10-13T12:10:31.749 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:31.747+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.748586+0000) 2019-10-13T12:10:31.749 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:31.746+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.747434+0000) 2019-10-13T12:10:31.750 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:31.746+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.747613+0000) 2019-10-13T12:10:31.756 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:31.754+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225763 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:33.092 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:33.090+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:33.091733+0000) 2019-10-13T12:10:33.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:33.095+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:33.096019+0000) 2019-10-13T12:10:35.751 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:35.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.750770+0000) 2019-10-13T12:10:35.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:35.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.750765+0000) 2019-10-13T12:10:36.063 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:36.062+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.062503+0000) 2019-10-13T12:10:36.758 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:36.755+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.757346+0000) 2019-10-13T12:10:36.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:36.756+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.757493+0000) 2019-10-13T12:10:36.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:36.756+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.757784+0000) 2019-10-13T12:10:36.759 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:36.757+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.757813+0000) 2019-10-13T12:10:36.760 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:36.756+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.758102+0000) 2019-10-13T12:10:36.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:36.756+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.758372+0000) 2019-10-13T12:10:36.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:36.757+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.758509+0000) 2019-10-13T12:10:36.760 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:36.759+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:36.759959+0000) 2019-10-13T12:10:36.767 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:36.765+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225794 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:38.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:38.654+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:38.655265+0000) 2019-10-13T12:10:38.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:38.984+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:38.986080+0000) 2019-10-13T12:10:39.708 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:10:39.708 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:39.712 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:10:39.712 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:41.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:41.605+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.606276+0000) 2019-10-13T12:10:41.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:41.767+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.768420+0000) 2019-10-13T12:10:41.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:41.767+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.768863+0000) 2019-10-13T12:10:41.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:41.767+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.769149+0000) 2019-10-13T12:10:41.770 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:41.768+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.769435+0000) 2019-10-13T12:10:41.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:41.768+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.769653+0000) 2019-10-13T12:10:41.771 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:41.768+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.769901+0000) 2019-10-13T12:10:41.771 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:41.768+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.770103+0000) 2019-10-13T12:10:41.772 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:41.771+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.771831+0000) 2019-10-13T12:10:41.778 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:41.776+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:43.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:43.091+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:43.091920+0000) 2019-10-13T12:10:43.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:43.096+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:43.096228+0000) 2019-10-13T12:10:45.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:45.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.750975+0000) 2019-10-13T12:10:45.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:45.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.750980+0000) 2019-10-13T12:10:46.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:46.063+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.064769+0000) 2019-10-13T12:10:46.780 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:46.778+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.779446+0000) 2019-10-13T12:10:46.781 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:46.778+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.779648+0000) 2019-10-13T12:10:46.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:46.778+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.779820+0000) 2019-10-13T12:10:46.781 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:46.778+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.779876+0000) 2019-10-13T12:10:46.782 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:46.779+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.780923+0000) 2019-10-13T12:10:46.782 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:46.778+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.780315+0000) 2019-10-13T12:10:46.782 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:46.779+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.780442+0000) 2019-10-13T12:10:46.783 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:46.781+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:46.783171+0000) 2019-10-13T12:10:46.788 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:46.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:48.657 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:48.655+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:48.655471+0000) 2019-10-13T12:10:48.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:48.985+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:48.986232+0000) 2019-10-13T12:10:51.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:51.606+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.606471+0000) 2019-10-13T12:10:51.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:51.788+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.788723+0000) 2019-10-13T12:10:51.790 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:51.788+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.789190+0000) 2019-10-13T12:10:51.791 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:51.788+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.789359+0000) 2019-10-13T12:10:51.791 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:51.789+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.789510+0000) 2019-10-13T12:10:51.791 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:51.789+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.790359+0000) 2019-10-13T12:10:51.791 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:51.790+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.790493+0000) 2019-10-13T12:10:51.792 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:51.789+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.791032+0000) 2019-10-13T12:10:51.792 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:51.789+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.791262+0000) 2019-10-13T12:10:51.799 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:51.798+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225887 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:53.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:53.091+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:53.092133+0000) 2019-10-13T12:10:53.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:53.094+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:53.096437+0000) 2019-10-13T12:10:55.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:55.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.751177+0000) 2019-10-13T12:10:55.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:55.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.751177+0000) 2019-10-13T12:10:56.068 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:56.066+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.067158+0000) 2019-10-13T12:10:56.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:10:56.799+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.800128+0000) 2019-10-13T12:10:56.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:10:56.800+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.800466+0000) 2019-10-13T12:10:56.802 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:56.800+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.800776+0000) 2019-10-13T12:10:56.802 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:56.800+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.800906+0000) 2019-10-13T12:10:56.802 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:10:56.800+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.801088+0000) 2019-10-13T12:10:56.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:10:56.800+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.801172+0000) 2019-10-13T12:10:56.803 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:10:56.800+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.801917+0000) 2019-10-13T12:10:56.803 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:10:56.800+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:56.802536+0000) 2019-10-13T12:10:56.809 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:10:56.808+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225918 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:10:58.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:10:58.655+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:58.655655+0000) 2019-10-13T12:10:58.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:10:58.986+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:58.986387+0000) 2019-10-13T12:11:01.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:01.606+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.606698+0000) 2019-10-13T12:11:01.811 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:01.810+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.810392+0000) 2019-10-13T12:11:01.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:01.810+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.810944+0000) 2019-10-13T12:11:01.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:01.810+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.811133+0000) 2019-10-13T12:11:01.812 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:01.810+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.811238+0000) 2019-10-13T12:11:01.813 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:01.811+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.812043+0000) 2019-10-13T12:11:01.813 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:01.811+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.811544+0000) 2019-10-13T12:11:01.813 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:01.811+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.811785+0000) 2019-10-13T12:11:01.814 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:01.812+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.812890+0000) 2019-10-13T12:11:01.821 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:01.819+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:03.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:03.092+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:03.092309+0000) 2019-10-13T12:11:03.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:03.095+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:03.096664+0000) 2019-10-13T12:11:05.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:05.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.751357+0000) 2019-10-13T12:11:05.752 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:05.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.751360+0000) 2019-10-13T12:11:06.070 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:06.068+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.069581+0000) 2019-10-13T12:11:06.823 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:06.821+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.822154+0000) 2019-10-13T12:11:06.824 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:06.820+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.822126+0000) 2019-10-13T12:11:06.824 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:06.820+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.822251+0000) 2019-10-13T12:11:06.824 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:06.821+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.822442+0000) 2019-10-13T12:11:06.824 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:06.821+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.822852+0000) 2019-10-13T12:11:06.824 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:06.821+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.823069+0000) 2019-10-13T12:11:06.825 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:06.821+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.823171+0000) 2019-10-13T12:11:06.826 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:06.824+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:06.825679+0000) 2019-10-13T12:11:06.832 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:06.830+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 225981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:08.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:08.654+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:08.655850+0000) 2019-10-13T12:11:08.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:08.985+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:08.986520+0000) 2019-10-13T12:11:09.786 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:11:09.786 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:09.790 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:11:09.791 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:11.607 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:11.605+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.606877+0000) 2019-10-13T12:11:11.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:11.832+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.833443+0000) 2019-10-13T12:11:11.835 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:11.832+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.833546+0000) 2019-10-13T12:11:11.835 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:11.832+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.833866+0000) 2019-10-13T12:11:11.835 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:11.832+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.833768+0000) 2019-10-13T12:11:11.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:11.832+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.834317+0000) 2019-10-13T12:11:11.836 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:11.833+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.834619+0000) 2019-10-13T12:11:11.836 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:11.833+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.834782+0000) 2019-10-13T12:11:11.836 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:11.835+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.835861+0000) 2019-10-13T12:11:11.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:11.841+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226011 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:13.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:13.091+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:13.092526+0000) 2019-10-13T12:11:13.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:13.096+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:13.096827+0000) 2019-10-13T12:11:15.752 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:15.750+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.751571+0000) 2019-10-13T12:11:15.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:15.750+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.751583+0000) 2019-10-13T12:11:16.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:16.071+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.072066+0000) 2019-10-13T12:11:16.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:16.844+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.845128+0000) 2019-10-13T12:11:16.846 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:16.843+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.844785+0000) 2019-10-13T12:11:16.847 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:16.843+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.845262+0000) 2019-10-13T12:11:16.847 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:16.844+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.845591+0000) 2019-10-13T12:11:16.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:16.844+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.845723+0000) 2019-10-13T12:11:16.848 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:16.846+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.846863+0000) 2019-10-13T12:11:16.848 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:16.844+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.845901+0000) 2019-10-13T12:11:16.848 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:16.844+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:16.846047+0000) 2019-10-13T12:11:16.854 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:16.852+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226042 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:18.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:18.655+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:18.656013+0000) 2019-10-13T12:11:18.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:18.985+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:18.986714+0000) 2019-10-13T12:11:21.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:21.606+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.607095+0000) 2019-10-13T12:11:21.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:21.854+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.855336+0000) 2019-10-13T12:11:21.857 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:21.853+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.855326+0000) 2019-10-13T12:11:21.857 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:21.854+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.855426+0000) 2019-10-13T12:11:21.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:21.854+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.855607+0000) 2019-10-13T12:11:21.857 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:21.854+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.855762+0000) 2019-10-13T12:11:21.858 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:21.854+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.855884+0000) 2019-10-13T12:11:21.858 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:21.854+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.856205+0000) 2019-10-13T12:11:21.859 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:21.858+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.859303+0000) 2019-10-13T12:11:21.865 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:21.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226074 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:23.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:23.092+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:23.092719+0000) 2019-10-13T12:11:23.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:23.096+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:23.096938+0000) 2019-10-13T12:11:25.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:25.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.751758+0000) 2019-10-13T12:11:25.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:25.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.751786+0000) 2019-10-13T12:11:26.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:26.074+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.074505+0000) 2019-10-13T12:11:26.866 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:26.863+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.864989+0000) 2019-10-13T12:11:26.866 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:26.863+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.865092+0000) 2019-10-13T12:11:26.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:26.863+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.865313+0000) 2019-10-13T12:11:26.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:26.864+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.865689+0000) 2019-10-13T12:11:26.867 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:26.864+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.865741+0000) 2019-10-13T12:11:26.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:26.864+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.866166+0000) 2019-10-13T12:11:26.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:26.865+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.866506+0000) 2019-10-13T12:11:26.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:26.866+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:26.867263+0000) 2019-10-13T12:11:26.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:26.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226105 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:28.657 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:28.654+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:28.656221+0000) 2019-10-13T12:11:28.987 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:28.986+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:28.986876+0000) 2019-10-13T12:11:31.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:31.606+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.607220+0000) 2019-10-13T12:11:31.877 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:31.874+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.876006+0000) 2019-10-13T12:11:31.877 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:31.874+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.876243+0000) 2019-10-13T12:11:31.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:31.875+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.876513+0000) 2019-10-13T12:11:31.878 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:31.875+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.876660+0000) 2019-10-13T12:11:31.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:31.875+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.876824+0000) 2019-10-13T12:11:31.878 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:31.875+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.877122+0000) 2019-10-13T12:11:31.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:31.876+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.877526+0000) 2019-10-13T12:11:31.879 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:31.878+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.879177+0000) 2019-10-13T12:11:31.886 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:31.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226136 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:33.093 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:33.092+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:33.092915+0000) 2019-10-13T12:11:33.097 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:33.096+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:33.097059+0000) 2019-10-13T12:11:35.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:35.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.751933+0000) 2019-10-13T12:11:35.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:35.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.751953+0000) 2019-10-13T12:11:36.077 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:36.075+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.076631+0000) 2019-10-13T12:11:36.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:36.886+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.887229+0000) 2019-10-13T12:11:36.889 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:36.887+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.887812+0000) 2019-10-13T12:11:36.889 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:36.887+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.887830+0000) 2019-10-13T12:11:36.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:36.887+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.888177+0000) 2019-10-13T12:11:36.890 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:36.888+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.888490+0000) 2019-10-13T12:11:36.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:36.888+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.888659+0000) 2019-10-13T12:11:36.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:36.886+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.888462+0000) 2019-10-13T12:11:36.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:36.887+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:36.889515+0000) 2019-10-13T12:11:36.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:36.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226167 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:38.657 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:38.656+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:38.656350+0000) 2019-10-13T12:11:38.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:38.986+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:38.987087+0000) 2019-10-13T12:11:39.915 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:11:39.915 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:39.920 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:11:39.920 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:41.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:41.607+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.607439+0000) 2019-10-13T12:11:41.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:41.899+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.900009+0000) 2019-10-13T12:11:41.901 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:41.899+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.900189+0000) 2019-10-13T12:11:41.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:41.899+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.900325+0000) 2019-10-13T12:11:41.902 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:41.899+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.901678+0000) 2019-10-13T12:11:41.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:41.900+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.900756+0000) 2019-10-13T12:11:41.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:41.900+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.901081+0000) 2019-10-13T12:11:41.903 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:41.900+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.901322+0000) 2019-10-13T12:11:41.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:41.901+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.902924+0000) 2019-10-13T12:11:41.911 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:41.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:43.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:43.092+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:43.093110+0000) 2019-10-13T12:11:43.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:43.096+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:43.097237+0000) 2019-10-13T12:11:45.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:45.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.752121+0000) 2019-10-13T12:11:45.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:45.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.752130+0000) 2019-10-13T12:11:46.080 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:46.078+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.079105+0000) 2019-10-13T12:11:46.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:46.911+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.912141+0000) 2019-10-13T12:11:46.913 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:46.911+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.912345+0000) 2019-10-13T12:11:46.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:46.911+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.913465+0000) 2019-10-13T12:11:46.914 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:46.912+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.912994+0000) 2019-10-13T12:11:46.914 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:46.912+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.913164+0000) 2019-10-13T12:11:46.915 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:46.913+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.913538+0000) 2019-10-13T12:11:46.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:46.913+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.913849+0000) 2019-10-13T12:11:46.915 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:46.913+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:46.914906+0000) 2019-10-13T12:11:46.923 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:46.921+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226231 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:48.657 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:48.656+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:48.656512+0000) 2019-10-13T12:11:48.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:48.987+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:48.987300+0000) 2019-10-13T12:11:51.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:51.606+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.607619+0000) 2019-10-13T12:11:51.924 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:51.922+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.923528+0000) 2019-10-13T12:11:51.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:51.922+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.923902+0000) 2019-10-13T12:11:51.926 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:51.923+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.923890+0000) 2019-10-13T12:11:51.926 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:51.922+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.924351+0000) 2019-10-13T12:11:51.926 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:51.923+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.924728+0000) 2019-10-13T12:11:51.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:51.923+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.924896+0000) 2019-10-13T12:11:51.926 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:51.922+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.924024+0000) 2019-10-13T12:11:51.927 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:51.926+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.927135+0000) 2019-10-13T12:11:51.933 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:51.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226262 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:53.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:53.092+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:53.093300+0000) 2019-10-13T12:11:53.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:53.096+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:53.097424+0000) 2019-10-13T12:11:55.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:55.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.752329+0000) 2019-10-13T12:11:55.753 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:55.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.752335+0000) 2019-10-13T12:11:56.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:56.080+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.081602+0000) 2019-10-13T12:11:56.936 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:11:56.933+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.935259+0000) 2019-10-13T12:11:56.936 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:56.933+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.935383+0000) 2019-10-13T12:11:56.937 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:56.934+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.935552+0000) 2019-10-13T12:11:56.937 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:11:56.935+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.935860+0000) 2019-10-13T12:11:56.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:11:56.934+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.936224+0000) 2019-10-13T12:11:56.938 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:11:56.935+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.936580+0000) 2019-10-13T12:11:56.938 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:11:56.935+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.936518+0000) 2019-10-13T12:11:56.938 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:11:56.937+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:56.938224+0000) 2019-10-13T12:11:56.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:11:56.942+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226293 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:11:58.657 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:11:58.655+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:58.656704+0000) 2019-10-13T12:11:58.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:11:58.986+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:58.987511+0000) 2019-10-13T12:12:01.608 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:01.606+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.607784+0000) 2019-10-13T12:12:01.946 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:01.944+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.945656+0000) 2019-10-13T12:12:01.947 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:01.944+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.945849+0000) 2019-10-13T12:12:01.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:01.945+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.946348+0000) 2019-10-13T12:12:01.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:01.945+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.946639+0000) 2019-10-13T12:12:01.948 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:01.945+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.946767+0000) 2019-10-13T12:12:01.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:01.945+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.946953+0000) 2019-10-13T12:12:01.948 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:01.945+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.947268+0000) 2019-10-13T12:12:01.949 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:01.948+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.949313+0000) 2019-10-13T12:12:01.956 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:01.954+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226326 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:03.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:03.092+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:03.093443+0000) 2019-10-13T12:12:03.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:03.097+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:03.097606+0000) 2019-10-13T12:12:05.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:05.751+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.752490+0000) 2019-10-13T12:12:05.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:05.751+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.752549+0000) 2019-10-13T12:12:06.084 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:06.083+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.083954+0000) 2019-10-13T12:12:06.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:06.956+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.957455+0000) 2019-10-13T12:12:06.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:06.955+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.957135+0000) 2019-10-13T12:12:06.959 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:06.956+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.957663+0000) 2019-10-13T12:12:06.959 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:06.956+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.957963+0000) 2019-10-13T12:12:06.959 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:06.956+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.957812+0000) 2019-10-13T12:12:06.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:06.956+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.958103+0000) 2019-10-13T12:12:06.960 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:06.957+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.958482+0000) 2019-10-13T12:12:06.962 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:06.961+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:06.961873+0000) 2019-10-13T12:12:06.968 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:06.965+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226357 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:08.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:08.656+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:08.656904+0000) 2019-10-13T12:12:08.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:08.987+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:08.987702+0000) 2019-10-13T12:12:10.095 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:12:10.095 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:10.100 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:12:10.100 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:11.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:11.607+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.607967+0000) 2019-10-13T12:12:11.970 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:11.967+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.969134+0000) 2019-10-13T12:12:11.970 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:11.968+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.969482+0000) 2019-10-13T12:12:11.971 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:11.968+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.969515+0000) 2019-10-13T12:12:11.971 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:11.968+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.969695+0000) 2019-10-13T12:12:11.971 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:11.968+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.969962+0000) 2019-10-13T12:12:11.972 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:11.968+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.970297+0000) 2019-10-13T12:12:11.972 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:11.969+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.970428+0000) 2019-10-13T12:12:11.973 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:11.972+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.973054+0000) 2019-10-13T12:12:11.979 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:11.977+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226388 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:13.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:13.093+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:13.093600+0000) 2019-10-13T12:12:13.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:13.097+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:13.097781+0000) 2019-10-13T12:12:15.753 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:15.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.752674+0000) 2019-10-13T12:12:15.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:15.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.752725+0000) 2019-10-13T12:12:16.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:16.086+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.086387+0000) 2019-10-13T12:12:16.981 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:16.978+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.980156+0000) 2019-10-13T12:12:16.982 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:16.980+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.981507+0000) 2019-10-13T12:12:16.982 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:16.980+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.981624+0000) 2019-10-13T12:12:16.983 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:16.981+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.982207+0000) 2019-10-13T12:12:16.983 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:16.980+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.982362+0000) 2019-10-13T12:12:16.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:16.981+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.982558+0000) 2019-10-13T12:12:16.984 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:16.981+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.983230+0000) 2019-10-13T12:12:16.984 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:16.983+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:16.984277+0000) 2019-10-13T12:12:16.991 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:16.988+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226420 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:18.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:18.656+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:18.657095+0000) 2019-10-13T12:12:18.988 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:18.987+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:18.987901+0000) 2019-10-13T12:12:21.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:21.607+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.608183+0000) 2019-10-13T12:12:21.993 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:21.991+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.992096+0000) 2019-10-13T12:12:21.993 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:21.991+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.992302+0000) 2019-10-13T12:12:21.994 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:21.992+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.992516+0000) 2019-10-13T12:12:21.994 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:21.992+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.992804+0000) 2019-10-13T12:12:21.994 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:21.992+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.992970+0000) 2019-10-13T12:12:21.994 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:21.992+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.993505+0000) 2019-10-13T12:12:21.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:21.993+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.993514+0000) 2019-10-13T12:12:21.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:21.994+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.994763+0000) 2019-10-13T12:12:22.002 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:22.001+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226452 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:23.094 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:23.093+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:23.093808+0000) 2019-10-13T12:12:23.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:23.097+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:23.097914+0000) 2019-10-13T12:12:25.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:25.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.752851+0000) 2019-10-13T12:12:25.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:25.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.752877+0000) 2019-10-13T12:12:26.089 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:26.087+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:26.088624+0000) 2019-10-13T12:12:27.004 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:27.002+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.003162+0000) 2019-10-13T12:12:27.004 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:27.003+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.003425+0000) 2019-10-13T12:12:27.005 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:27.002+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.003314+0000) 2019-10-13T12:12:27.005 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:27.003+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.003725+0000) 2019-10-13T12:12:27.006 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:27.003+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.004797+0000) 2019-10-13T12:12:27.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:27.003+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.003959+0000) 2019-10-13T12:12:27.006 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:27.003+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.004252+0000) 2019-10-13T12:12:27.007 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:27.004+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.006557+0000) 2019-10-13T12:12:27.013 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:27.012+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226484 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:28.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:28.657+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:28.657267+0000) 2019-10-13T12:12:28.989 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:28.987+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:28.988099+0000) 2019-10-13T12:12:31.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:31.608+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:31.608375+0000) 2019-10-13T12:12:32.015 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:32.013+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.014307+0000) 2019-10-13T12:12:32.015 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:32.014+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.014477+0000) 2019-10-13T12:12:32.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:32.014+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.014660+0000) 2019-10-13T12:12:32.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:32.014+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.015071+0000) 2019-10-13T12:12:32.016 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:32.014+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.015194+0000) 2019-10-13T12:12:32.016 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:32.015+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.015720+0000) 2019-10-13T12:12:32.017 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:32.014+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.016044+0000) 2019-10-13T12:12:32.018 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:32.015+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:32.017609+0000) 2019-10-13T12:12:32.024 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:32.023+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226514 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:33.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:33.093+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:33.093992+0000) 2019-10-13T12:12:33.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:33.097+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:33.098096+0000) 2019-10-13T12:12:35.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:35.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:35.753048+0000) 2019-10-13T12:12:35.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:35.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:35.753082+0000) 2019-10-13T12:12:36.091 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:36.090+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.090910+0000) 2019-10-13T12:12:37.026 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:37.023+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.025027+0000) 2019-10-13T12:12:37.026 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:37.023+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.025155+0000) 2019-10-13T12:12:37.027 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:37.023+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.025262+0000) 2019-10-13T12:12:37.027 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:37.024+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.025456+0000) 2019-10-13T12:12:37.027 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:37.024+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.025634+0000) 2019-10-13T12:12:37.027 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:37.024+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.025860+0000) 2019-10-13T12:12:37.028 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:37.023+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.025683+0000) 2019-10-13T12:12:37.028 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:37.026+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.028124+0000) 2019-10-13T12:12:37.033 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:37.031+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:38.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:38.656+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:38.657447+0000) 2019-10-13T12:12:38.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:38.987+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:38.988308+0000) 2019-10-13T12:12:40.233 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:12:40.234 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:40.238 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:12:40.238 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:41.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:41.607+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.608579+0000) 2019-10-13T12:12:42.037 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:42.033+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.035019+0000) 2019-10-13T12:12:42.038 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:42.033+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.035338+0000) 2019-10-13T12:12:42.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:42.034+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.035805+0000) 2019-10-13T12:12:42.038 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:42.034+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.035662+0000) 2019-10-13T12:12:42.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:42.034+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.036030+0000) 2019-10-13T12:12:42.038 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:42.034+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.036122+0000) 2019-10-13T12:12:42.039 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:42.034+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.034944+0000) 2019-10-13T12:12:42.039 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:42.036+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:42.037437+0000) 2019-10-13T12:12:42.045 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:42.043+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226578 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:43.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:43.092+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:43.094140+0000) 2019-10-13T12:12:43.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:43.097+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:43.098302+0000) 2019-10-13T12:12:45.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:45.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:45.753239+0000) 2019-10-13T12:12:45.754 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:45.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:45.753276+0000) 2019-10-13T12:12:46.094 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:46.092+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.093317+0000) 2019-10-13T12:12:47.047 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:47.045+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.046458+0000) 2019-10-13T12:12:47.048 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:47.045+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.046488+0000) 2019-10-13T12:12:47.048 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:47.045+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.046792+0000) 2019-10-13T12:12:47.048 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:47.045+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.046997+0000) 2019-10-13T12:12:47.049 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:47.046+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.046773+0000) 2019-10-13T12:12:47.049 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:47.045+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.047173+0000) 2019-10-13T12:12:47.049 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:47.046+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.047651+0000) 2019-10-13T12:12:47.051 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:47.050+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.050783+0000) 2019-10-13T12:12:47.056 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:47.054+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226611 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:48.658 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:48.656+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:48.657660+0000) 2019-10-13T12:12:48.989 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:48.987+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:48.988528+0000) 2019-10-13T12:12:51.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:51.608+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.608749+0000) 2019-10-13T12:12:52.059 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:52.056+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.057597+0000) 2019-10-13T12:12:52.059 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:52.056+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.057957+0000) 2019-10-13T12:12:52.059 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:52.056+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.057728+0000) 2019-10-13T12:12:52.060 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:52.057+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.058457+0000) 2019-10-13T12:12:52.061 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:52.057+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.058214+0000) 2019-10-13T12:12:52.061 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:52.057+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.058797+0000) 2019-10-13T12:12:52.061 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:52.057+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.058927+0000) 2019-10-13T12:12:52.062 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:52.060+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:52.060780+0000) 2019-10-13T12:12:52.068 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:52.065+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:53.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:53.093+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:53.094313+0000) 2019-10-13T12:12:53.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:53.097+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:53.098482+0000) 2019-10-13T12:12:55.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:55.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:55.753474+0000) 2019-10-13T12:12:55.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:55.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:55.753478+0000) 2019-10-13T12:12:56.096 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:56.094+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.095278+0000) 2019-10-13T12:12:57.069 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:12:57.068+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.068905+0000) 2019-10-13T12:12:57.070 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:12:57.067+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.068921+0000) 2019-10-13T12:12:57.071 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:57.067+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.069205+0000) 2019-10-13T12:12:57.071 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:12:57.068+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.069479+0000) 2019-10-13T12:12:57.071 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:12:57.068+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.069406+0000) 2019-10-13T12:12:57.071 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:57.068+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.069811+0000) 2019-10-13T12:12:57.071 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:12:57.068+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.069891+0000) 2019-10-13T12:12:57.072 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:12:57.070+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.070927+0000) 2019-10-13T12:12:57.077 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:12:57.075+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226673 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:12:58.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:12:58.657+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:58.657869+0000) 2019-10-13T12:12:58.989 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:12:58.988+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:58.988700+0000) 2019-10-13T12:13:01.609 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:01.608+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.608926+0000) 2019-10-13T12:13:02.079 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:02.078+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.078883+0000) 2019-10-13T12:13:02.080 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:02.077+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.078922+0000) 2019-10-13T12:13:02.081 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:02.077+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.079128+0000) 2019-10-13T12:13:02.081 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:02.078+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.079613+0000) 2019-10-13T12:13:02.081 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:02.078+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.079519+0000) 2019-10-13T12:13:02.081 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:02.078+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.079736+0000) 2019-10-13T12:13:02.081 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:02.078+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.079773+0000) 2019-10-13T12:13:02.082 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:02.080+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:02.081474+0000) 2019-10-13T12:13:02.088 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:02.086+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:03.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:03.094+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:03.094531+0000) 2019-10-13T12:13:03.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:03.098+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:03.098638+0000) 2019-10-13T12:13:05.754 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:05.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:05.753704+0000) 2019-10-13T12:13:05.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:05.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:05.753705+0000) 2019-10-13T12:13:06.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:06.097+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.097263+0000) 2019-10-13T12:13:07.091 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:07.089+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.089766+0000) 2019-10-13T12:13:07.091 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:07.089+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.089935+0000) 2019-10-13T12:13:07.091 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:07.089+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.090255+0000) 2019-10-13T12:13:07.092 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:07.090+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.090424+0000) 2019-10-13T12:13:07.092 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:07.090+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.090607+0000) 2019-10-13T12:13:07.092 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:07.089+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.090326+0000) 2019-10-13T12:13:07.093 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:07.090+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.090907+0000) 2019-10-13T12:13:07.093 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:07.092+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.093350+0000) 2019-10-13T12:13:07.100 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:07.098+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226737 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:08.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:08.657+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:08.658113+0000) 2019-10-13T12:13:08.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:08.988+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:08.988931+0000) 2019-10-13T12:13:10.388 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:13:10.388 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:10.393 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:13:10.393 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:11.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:11.608+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.609140+0000) 2019-10-13T12:13:12.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:12.100+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.101085+0000) 2019-10-13T12:13:12.103 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:12.100+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.101291+0000) 2019-10-13T12:13:12.103 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:12.101+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.101672+0000) 2019-10-13T12:13:12.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:12.100+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.101653+0000) 2019-10-13T12:13:12.104 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:12.101+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.101978+0000) 2019-10-13T12:13:12.105 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:12.101+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.101816+0000) 2019-10-13T12:13:12.105 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:12.102+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.102537+0000) 2019-10-13T12:13:12.106 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:12.102+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:12.103606+0000) 2019-10-13T12:13:12.111 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:12.110+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:13.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:13.094+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:13.094682+0000) 2019-10-13T12:13:13.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:13.098+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:13.098805+0000) 2019-10-13T12:13:15.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:15.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:15.753928+0000) 2019-10-13T12:13:15.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:15.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:15.753934+0000) 2019-10-13T12:13:16.100 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:16.098+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.099509+0000) 2019-10-13T12:13:17.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:17.110+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.112089+0000) 2019-10-13T12:13:17.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:17.112+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.112591+0000) 2019-10-13T12:13:17.114 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:17.112+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.112806+0000) 2019-10-13T12:13:17.114 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:17.112+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.113043+0000) 2019-10-13T12:13:17.114 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:17.112+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.113191+0000) 2019-10-13T12:13:17.115 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:17.112+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.113364+0000) 2019-10-13T12:13:17.115 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:17.113+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.113628+0000) 2019-10-13T12:13:17.115 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:17.112+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.114424+0000) 2019-10-13T12:13:17.122 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:17.121+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226799 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:18.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:18.658+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:18.658332+0000) 2019-10-13T12:13:18.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:18.989+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:18.989147+0000) 2019-10-13T12:13:21.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:21.608+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.609328+0000) 2019-10-13T12:13:22.124 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:22.121+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.122900+0000) 2019-10-13T12:13:22.124 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:22.121+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.123028+0000) 2019-10-13T12:13:22.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:22.121+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.123179+0000) 2019-10-13T12:13:22.125 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:22.121+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.123324+0000) 2019-10-13T12:13:22.125 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:22.122+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.123584+0000) 2019-10-13T12:13:22.125 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:22.122+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.123653+0000) 2019-10-13T12:13:22.126 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:22.122+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.124098+0000) 2019-10-13T12:13:22.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:22.124+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:22.125843+0000) 2019-10-13T12:13:22.131 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:22.128+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226830 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:23.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:23.093+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:23.094854+0000) 2019-10-13T12:13:23.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:23.098+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:23.098990+0000) 2019-10-13T12:13:25.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:25.752+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:25.754131+0000) 2019-10-13T12:13:25.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:25.752+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:25.754097+0000) 2019-10-13T12:13:26.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:26.101+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.101920+0000) 2019-10-13T12:13:27.133 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:27.130+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.132068+0000) 2019-10-13T12:13:27.133 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:27.130+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.132366+0000) 2019-10-13T12:13:27.134 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:27.130+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.132362+0000) 2019-10-13T12:13:27.134 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:27.131+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.132557+0000) 2019-10-13T12:13:27.134 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:27.131+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.133118+0000) 2019-10-13T12:13:27.134 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:27.131+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.133191+0000) 2019-10-13T12:13:27.135 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:27.132+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.133457+0000) 2019-10-13T12:13:27.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:27.133+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.134797+0000) 2019-10-13T12:13:27.142 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:27.139+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:28.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:28.657+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:28.658518+0000) 2019-10-13T12:13:28.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:28.988+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:28.989363+0000) 2019-10-13T12:13:31.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:31.608+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.609494+0000) 2019-10-13T12:13:32.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:32.141+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.142766+0000) 2019-10-13T12:13:32.144 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:32.142+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.143347+0000) 2019-10-13T12:13:32.144 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:32.142+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.143481+0000) 2019-10-13T12:13:32.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:32.142+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.143684+0000) 2019-10-13T12:13:32.145 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:32.142+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.143797+0000) 2019-10-13T12:13:32.145 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:32.142+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.143992+0000) 2019-10-13T12:13:32.145 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:32.142+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.144285+0000) 2019-10-13T12:13:32.146 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:32.144+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:32.145400+0000) 2019-10-13T12:13:32.157 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:32.151+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226892 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:33.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:33.094+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:33.095048+0000) 2019-10-13T12:13:33.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:33.098+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:33.099142+0000) 2019-10-13T12:13:35.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:35.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:35.754341+0000) 2019-10-13T12:13:35.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:35.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:35.754338+0000) 2019-10-13T12:13:36.105 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:36.103+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.104250+0000) 2019-10-13T12:13:37.155 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:37.153+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.154164+0000) 2019-10-13T12:13:37.156 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:37.152+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.154226+0000) 2019-10-13T12:13:37.156 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:37.153+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.154692+0000) 2019-10-13T12:13:37.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:37.153+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.154874+0000) 2019-10-13T12:13:37.156 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:37.153+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.155063+0000) 2019-10-13T12:13:37.157 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:37.153+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.155137+0000) 2019-10-13T12:13:37.157 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:37.153+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.155350+0000) 2019-10-13T12:13:37.158 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:37.156+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.157579+0000) 2019-10-13T12:13:37.164 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:37.162+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:38.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:38.657+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:38.658688+0000) 2019-10-13T12:13:38.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:38.988+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:38.989568+0000) 2019-10-13T12:13:40.549 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:13:40.550 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:40.554 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:13:40.554 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:41.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:41.609+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.609647+0000) 2019-10-13T12:13:42.166 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:42.164+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.165327+0000) 2019-10-13T12:13:42.167 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:42.163+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.165080+0000) 2019-10-13T12:13:42.167 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:42.163+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.165298+0000) 2019-10-13T12:13:42.167 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:42.164+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.165520+0000) 2019-10-13T12:13:42.167 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:42.164+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.165751+0000) 2019-10-13T12:13:42.168 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:42.164+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.165946+0000) 2019-10-13T12:13:42.168 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:42.164+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.166003+0000) 2019-10-13T12:13:42.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:42.167+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:42.167884+0000) 2019-10-13T12:13:42.173 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:42.171+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226956 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:43.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:43.094+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:43.095219+0000) 2019-10-13T12:13:43.099 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:43.098+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:43.099287+0000) 2019-10-13T12:13:45.755 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:45.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:45.754553+0000) 2019-10-13T12:13:45.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:45.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:45.754553+0000) 2019-10-13T12:13:46.107 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:46.105+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.106582+0000) 2019-10-13T12:13:47.175 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:47.172+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.174118+0000) 2019-10-13T12:13:47.176 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:47.173+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.174559+0000) 2019-10-13T12:13:47.176 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:47.173+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.174824+0000) 2019-10-13T12:13:47.176 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:47.173+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.174856+0000) 2019-10-13T12:13:47.176 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:47.173+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.175020+0000) 2019-10-13T12:13:47.177 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:47.173+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.175303+0000) 2019-10-13T12:13:47.177 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:47.174+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.175421+0000) 2019-10-13T12:13:47.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:47.176+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.177581+0000) 2019-10-13T12:13:47.184 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:47.181+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 226990 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:48.659 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:48.658+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:48.658885+0000) 2019-10-13T12:13:48.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:48.989+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:48.989733+0000) 2019-10-13T12:13:51.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:51.609+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.609819+0000) 2019-10-13T12:13:52.186 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:52.184+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.185030+0000) 2019-10-13T12:13:52.186 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:52.185+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.185428+0000) 2019-10-13T12:13:52.187 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:52.185+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.185866+0000) 2019-10-13T12:13:52.187 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:52.185+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.185636+0000) 2019-10-13T12:13:52.187 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:52.185+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.185908+0000) 2019-10-13T12:13:52.188 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:52.185+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.186317+0000) 2019-10-13T12:13:52.188 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:52.186+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.186642+0000) 2019-10-13T12:13:52.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:52.187+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:52.188404+0000) 2019-10-13T12:13:52.195 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:52.194+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227018 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:53.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:53.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:53.095420+0000) 2019-10-13T12:13:53.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:53.099+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:53.099430+0000) 2019-10-13T12:13:55.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:55.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:55.754743+0000) 2019-10-13T12:13:55.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:55.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:55.754774+0000) 2019-10-13T12:13:56.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:56.108+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.108846+0000) 2019-10-13T12:13:57.198 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:13:57.196+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.197093+0000) 2019-10-13T12:13:57.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:13:57.196+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.196891+0000) 2019-10-13T12:13:57.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:57.196+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.197081+0000) 2019-10-13T12:13:57.199 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:13:57.196+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.197352+0000) 2019-10-13T12:13:57.199 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:13:57.197+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.197739+0000) 2019-10-13T12:13:57.199 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:13:57.197+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.197905+0000) 2019-10-13T12:13:57.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:13:57.198+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.199427+0000) 2019-10-13T12:13:57.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:57.197+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.198189+0000) 2019-10-13T12:13:57.206 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:13:57.205+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227052 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:13:58.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:13:58.658+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:58.659073+0000) 2019-10-13T12:13:58.990 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:13:58.989+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:58.989906+0000) 2019-10-13T12:14:01.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:01.609+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.610015+0000) 2019-10-13T12:14:02.208 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:02.207+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.207410+0000) 2019-10-13T12:14:02.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:02.207+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.207515+0000) 2019-10-13T12:14:02.209 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:02.207+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.207820+0000) 2019-10-13T12:14:02.210 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:02.207+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.208354+0000) 2019-10-13T12:14:02.210 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:02.207+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.208201+0000) 2019-10-13T12:14:02.210 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:02.207+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.208386+0000) 2019-10-13T12:14:02.210 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:02.208+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.208686+0000) 2019-10-13T12:14:02.211 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:02.210+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:02.210843+0000) 2019-10-13T12:14:02.217 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:02.215+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227082 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:03.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:03.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:03.095542+0000) 2019-10-13T12:14:03.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:03.099+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:03.099591+0000) 2019-10-13T12:14:05.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:05.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:05.754914+0000) 2019-10-13T12:14:05.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:05.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:05.754955+0000) 2019-10-13T12:14:06.112 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:06.111+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.111130+0000) 2019-10-13T12:14:07.219 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:07.216+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.217939+0000) 2019-10-13T12:14:07.220 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:07.217+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.218427+0000) 2019-10-13T12:14:07.220 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:07.217+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.218633+0000) 2019-10-13T12:14:07.220 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:07.217+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.218835+0000) 2019-10-13T12:14:07.220 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:07.217+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.219217+0000) 2019-10-13T12:14:07.220 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:07.217+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.219250+0000) 2019-10-13T12:14:07.221 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:07.217+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.219375+0000) 2019-10-13T12:14:07.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:07.220+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.222073+0000) 2019-10-13T12:14:07.228 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:07.225+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227116 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:08.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:08.658+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:08.659250+0000) 2019-10-13T12:14:08.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:08.989+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:08.990108+0000) 2019-10-13T12:14:10.637 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:14:10.637 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:10.642 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:14:10.643 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:11.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:11.609+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.610179+0000) 2019-10-13T12:14:12.230 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:12.227+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.229051+0000) 2019-10-13T12:14:12.231 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:12.228+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.229902+0000) 2019-10-13T12:14:12.231 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:12.228+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.229616+0000) 2019-10-13T12:14:12.231 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:12.228+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.229501+0000) 2019-10-13T12:14:12.232 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:12.228+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.230030+0000) 2019-10-13T12:14:12.232 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:12.228+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.230312+0000) 2019-10-13T12:14:12.232 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:12.229+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.230453+0000) 2019-10-13T12:14:12.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:12.230+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:12.232441+0000) 2019-10-13T12:14:12.244 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:12.237+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227144 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:13.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:13.094+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:13.095686+0000) 2019-10-13T12:14:13.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:13.098+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:13.099747+0000) 2019-10-13T12:14:15.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:15.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:15.755033+0000) 2019-10-13T12:14:15.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:15.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:15.755075+0000) 2019-10-13T12:14:16.114 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:16.111+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.113637+0000) 2019-10-13T12:14:17.241 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:17.238+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.240288+0000) 2019-10-13T12:14:17.242 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:17.238+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.240380+0000) 2019-10-13T12:14:17.242 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:17.239+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.240590+0000) 2019-10-13T12:14:17.242 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:17.239+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.240695+0000) 2019-10-13T12:14:17.242 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:17.239+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.241037+0000) 2019-10-13T12:14:17.242 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:17.239+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.241114+0000) 2019-10-13T12:14:17.243 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:17.240+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.241417+0000) 2019-10-13T12:14:17.244 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:17.242+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.243764+0000) 2019-10-13T12:14:17.250 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:17.248+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227178 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:18.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:18.658+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:18.659386+0000) 2019-10-13T12:14:18.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:18.989+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:18.990279+0000) 2019-10-13T12:14:21.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:21.609+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.610332+0000) 2019-10-13T12:14:22.252 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:22.251+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.252164+0000) 2019-10-13T12:14:22.253 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:22.251+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.252493+0000) 2019-10-13T12:14:22.254 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:22.251+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.252703+0000) 2019-10-13T12:14:22.254 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:22.251+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.252769+0000) 2019-10-13T12:14:22.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:22.251+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.252992+0000) 2019-10-13T12:14:22.254 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:22.251+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.253144+0000) 2019-10-13T12:14:22.254 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:22.251+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.253310+0000) 2019-10-13T12:14:22.255 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:22.253+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:22.253852+0000) 2019-10-13T12:14:22.262 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:22.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227208 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:23.096 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:23.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:23.095862+0000) 2019-10-13T12:14:23.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:23.099+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:23.099886+0000) 2019-10-13T12:14:25.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:25.753+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:25.755238+0000) 2019-10-13T12:14:25.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:25.753+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:25.755242+0000) 2019-10-13T12:14:26.117 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:26.115+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.116049+0000) 2019-10-13T12:14:27.264 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:27.261+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.262956+0000) 2019-10-13T12:14:27.264 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:27.261+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.263233+0000) 2019-10-13T12:14:27.265 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:27.262+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.263413+0000) 2019-10-13T12:14:27.265 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:27.262+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.263217+0000) 2019-10-13T12:14:27.266 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:27.262+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.263946+0000) 2019-10-13T12:14:27.266 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:27.262+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.264173+0000) 2019-10-13T12:14:27.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:27.262+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.264371+0000) 2019-10-13T12:14:27.266 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:27.264+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.265612+0000) 2019-10-13T12:14:27.273 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:27.271+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227242 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:28.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:28.658+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:28.659545+0000) 2019-10-13T12:14:28.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:28.989+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:28.990439+0000) 2019-10-13T12:14:31.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:31.610+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.610510+0000) 2019-10-13T12:14:32.275 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:32.272+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.273992+0000) 2019-10-13T12:14:32.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:32.272+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.274219+0000) 2019-10-13T12:14:32.275 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:32.273+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.274473+0000) 2019-10-13T12:14:32.276 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:32.274+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.274754+0000) 2019-10-13T12:14:32.276 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:32.273+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.274779+0000) 2019-10-13T12:14:32.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:32.273+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.275257+0000) 2019-10-13T12:14:32.277 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:32.274+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.275393+0000) 2019-10-13T12:14:32.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:32.277+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:32.277951+0000) 2019-10-13T12:14:32.283 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:32.281+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227270 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:33.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:33.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:33.096035+0000) 2019-10-13T12:14:33.100 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:33.099+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:33.100041+0000) 2019-10-13T12:14:35.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:35.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:35.755452+0000) 2019-10-13T12:14:35.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:35.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:35.755452+0000) 2019-10-13T12:14:36.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:36.117+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.118318+0000) 2019-10-13T12:14:37.294 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:37.284+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.284774+0000) 2019-10-13T12:14:37.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:37.284+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.285084+0000) 2019-10-13T12:14:37.295 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:37.284+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.285292+0000) 2019-10-13T12:14:37.295 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:37.285+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.285512+0000) 2019-10-13T12:14:37.295 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:37.285+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.285658+0000) 2019-10-13T12:14:37.296 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:37.285+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.286007+0000) 2019-10-13T12:14:37.296 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:37.285+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.286278+0000) 2019-10-13T12:14:37.296 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:37.286+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.286853+0000) 2019-10-13T12:14:37.304 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:37.293+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227304 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:38.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:38.659+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:38.659676+0000) 2019-10-13T12:14:38.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:38.990+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:38.990609+0000) 2019-10-13T12:14:40.817 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:14:40.817 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:40.821 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:14:40.821 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:41.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:41.610+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.610683+0000) 2019-10-13T12:14:42.296 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:42.295+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.295757+0000) 2019-10-13T12:14:42.297 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:42.295+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.296144+0000) 2019-10-13T12:14:42.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:42.295+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.296222+0000) 2019-10-13T12:14:42.298 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:42.296+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.296466+0000) 2019-10-13T12:14:42.298 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:42.296+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.296887+0000) 2019-10-13T12:14:42.298 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:42.296+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.297191+0000) 2019-10-13T12:14:42.299 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:42.296+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.297308+0000) 2019-10-13T12:14:42.300 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:42.298+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:42.299456+0000) 2019-10-13T12:14:42.305 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:42.304+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227334 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:43.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:43.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:43.096196+0000) 2019-10-13T12:14:43.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:43.099+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:43.100222+0000) 2019-10-13T12:14:45.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:45.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.755632+0000) 2019-10-13T12:14:45.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:45.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:45.755632+0000) 2019-10-13T12:14:46.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:46.120+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.120533+0000) 2019-10-13T12:14:47.307 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:47.306+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.306634+0000) 2019-10-13T12:14:47.307 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:47.306+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.306922+0000) 2019-10-13T12:14:47.308 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:47.306+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.306929+0000) 2019-10-13T12:14:47.309 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:47.306+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.307135+0000) 2019-10-13T12:14:47.309 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:47.306+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.307221+0000) 2019-10-13T12:14:47.309 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:47.306+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.307319+0000) 2019-10-13T12:14:47.309 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:47.307+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.307935+0000) 2019-10-13T12:14:47.311 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:47.310+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.310839+0000) 2019-10-13T12:14:47.315 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:47.314+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227368 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:48.660 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:48.659+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:48.659830+0000) 2019-10-13T12:14:48.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:48.990+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:48.990819+0000) 2019-10-13T12:14:51.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:51.610+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.610855+0000) 2019-10-13T12:14:52.317 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:52.315+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.316664+0000) 2019-10-13T12:14:52.318 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:52.316+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.316702+0000) 2019-10-13T12:14:52.318 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:52.316+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.317205+0000) 2019-10-13T12:14:52.319 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:52.317+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.317683+0000) 2019-10-13T12:14:52.319 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:52.317+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.317845+0000) 2019-10-13T12:14:52.319 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:52.317+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.318050+0000) 2019-10-13T12:14:52.319 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:52.317+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.318157+0000) 2019-10-13T12:14:52.320 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:52.318+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:52.319170+0000) 2019-10-13T12:14:52.327 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:52.325+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227396 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:53.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:53.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:53.096389+0000) 2019-10-13T12:14:53.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:53.100+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:53.100358+0000) 2019-10-13T12:14:55.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:55.754+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.755824+0000) 2019-10-13T12:14:55.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:55.754+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:55.755825+0000) 2019-10-13T12:14:56.123 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:56.122+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.122846+0000) 2019-10-13T12:14:57.329 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:14:57.326+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.327879+0000) 2019-10-13T12:14:57.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:14:57.327+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.328469+0000) 2019-10-13T12:14:57.329 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:57.327+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.328618+0000) 2019-10-13T12:14:57.330 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:14:57.327+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.329235+0000) 2019-10-13T12:14:57.330 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:14:57.327+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.329073+0000) 2019-10-13T12:14:57.331 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:57.327+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.329257+0000) 2019-10-13T12:14:57.331 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:14:57.328+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.329612+0000) 2019-10-13T12:14:57.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:14:57.328+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.330428+0000) 2019-10-13T12:14:57.338 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:14:57.336+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227430 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:14:58.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:14:58.659+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:58.659983+0000) 2019-10-13T12:14:58.991 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:14:58.989+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:58.990992+0000) 2019-10-13T12:15:01.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:01.610+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.611055+0000) 2019-10-13T12:15:02.340 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:02.338+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.339519+0000) 2019-10-13T12:15:02.340 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:02.338+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.339759+0000) 2019-10-13T12:15:02.341 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:02.338+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.340415+0000) 2019-10-13T12:15:02.341 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:02.338+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.339970+0000) 2019-10-13T12:15:02.342 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:02.338+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.340067+0000) 2019-10-13T12:15:02.342 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:02.339+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.340569+0000) 2019-10-13T12:15:02.342 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:02.339+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.340718+0000) 2019-10-13T12:15:02.342 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:02.339+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:02.341445+0000) 2019-10-13T12:15:02.349 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:02.347+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:03.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:03.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:03.096577+0000) 2019-10-13T12:15:03.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:03.099+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:03.100475+0000) 2019-10-13T12:15:05.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:05.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.756010+0000) 2019-10-13T12:15:05.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:05.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:05.756010+0000) 2019-10-13T12:15:06.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:06.124+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.125193+0000) 2019-10-13T12:15:07.352 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:07.349+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.350966+0000) 2019-10-13T12:15:07.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:07.349+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.351125+0000) 2019-10-13T12:15:07.353 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:07.349+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.351673+0000) 2019-10-13T12:15:07.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:07.350+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.351668+0000) 2019-10-13T12:15:07.353 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:07.350+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.351848+0000) 2019-10-13T12:15:07.353 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:07.350+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.352054+0000) 2019-10-13T12:15:07.354 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:07.350+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.351934+0000) 2019-10-13T12:15:07.354 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:07.351+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.352871+0000) 2019-10-13T12:15:07.360 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:07.358+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227494 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:08.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:08.659+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:08.660120+0000) 2019-10-13T12:15:08.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:08.990+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:08.991133+0000) 2019-10-13T12:15:10.963 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:15:10.964 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:10.971 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:15:10.971 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:11.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:11.610+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.611263+0000) 2019-10-13T12:15:12.362 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:12.361+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.361797+0000) 2019-10-13T12:15:12.363 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:12.360+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.361909+0000) 2019-10-13T12:15:12.363 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:12.360+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.362118+0000) 2019-10-13T12:15:12.364 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:12.360+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.362282+0000) 2019-10-13T12:15:12.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:12.361+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.362807+0000) 2019-10-13T12:15:12.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:12.361+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.362934+0000) 2019-10-13T12:15:12.364 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:12.361+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.363247+0000) 2019-10-13T12:15:12.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:12.364+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:12.364861+0000) 2019-10-13T12:15:12.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:12.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227522 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:13.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:13.096+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:13.096743+0000) 2019-10-13T12:15:13.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:13.099+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:13.100622+0000) 2019-10-13T12:15:15.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:15.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.756191+0000) 2019-10-13T12:15:15.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:15.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:15.756190+0000) 2019-10-13T12:15:16.128 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:16.126+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.127398+0000) 2019-10-13T12:15:17.374 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:17.371+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.373171+0000) 2019-10-13T12:15:17.375 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:17.372+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.373547+0000) 2019-10-13T12:15:17.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:17.372+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.373725+0000) 2019-10-13T12:15:17.375 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:17.372+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.373948+0000) 2019-10-13T12:15:17.375 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:17.372+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.374171+0000) 2019-10-13T12:15:17.376 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:17.372+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.374378+0000) 2019-10-13T12:15:17.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:17.373+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.374511+0000) 2019-10-13T12:15:17.376 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:17.374+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.375360+0000) 2019-10-13T12:15:17.383 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:17.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227554 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:18.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:18.659+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:18.660300+0000) 2019-10-13T12:15:18.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:18.990+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:18.991285+0000) 2019-10-13T12:15:21.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:21.611+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.611480+0000) 2019-10-13T12:15:22.385 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:22.383+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.384063+0000) 2019-10-13T12:15:22.385 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:22.383+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.384179+0000) 2019-10-13T12:15:22.386 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:22.383+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.384587+0000) 2019-10-13T12:15:22.386 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:22.384+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.384769+0000) 2019-10-13T12:15:22.386 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:22.384+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.385019+0000) 2019-10-13T12:15:22.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:22.384+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.385148+0000) 2019-10-13T12:15:22.387 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:22.384+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.385361+0000) 2019-10-13T12:15:22.387 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:22.386+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:22.387165+0000) 2019-10-13T12:15:22.393 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:22.392+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227584 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:23.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:23.096+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:23.096895+0000) 2019-10-13T12:15:23.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:23.100+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:23.100761+0000) 2019-10-13T12:15:25.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:25.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.756406+0000) 2019-10-13T12:15:25.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:25.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:25.756365+0000) 2019-10-13T12:15:26.130 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:26.129+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.129719+0000) 2019-10-13T12:15:27.396 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:27.394+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.394631+0000) 2019-10-13T12:15:27.396 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:27.394+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.394776+0000) 2019-10-13T12:15:27.396 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:27.394+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.394864+0000) 2019-10-13T12:15:27.396 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:27.394+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.395172+0000) 2019-10-13T12:15:27.397 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:27.394+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.395374+0000) 2019-10-13T12:15:27.397 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:27.395+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.395503+0000) 2019-10-13T12:15:27.398 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:27.395+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.396001+0000) 2019-10-13T12:15:27.398 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:27.396+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.396893+0000) 2019-10-13T12:15:27.404 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:27.403+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227616 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:28.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:28.660+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:28.660507+0000) 2019-10-13T12:15:28.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:28.991+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:28.991495+0000) 2019-10-13T12:15:31.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:31.611+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.611669+0000) 2019-10-13T12:15:32.406 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:32.405+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.405543+0000) 2019-10-13T12:15:32.407 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:32.405+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.405821+0000) 2019-10-13T12:15:32.407 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:32.405+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.406014+0000) 2019-10-13T12:15:32.408 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:32.405+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.406350+0000) 2019-10-13T12:15:32.408 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:32.406+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.406406+0000) 2019-10-13T12:15:32.408 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:32.406+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.406889+0000) 2019-10-13T12:15:32.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:32.406+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.407118+0000) 2019-10-13T12:15:32.409 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:32.408+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:32.408857+0000) 2019-10-13T12:15:32.415 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:32.414+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227644 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:33.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:33.096+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:33.097065+0000) 2019-10-13T12:15:33.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:33.100+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:33.100968+0000) 2019-10-13T12:15:35.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:35.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.756528+0000) 2019-10-13T12:15:35.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:35.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:35.756565+0000) 2019-10-13T12:15:36.133 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:36.131+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.132233+0000) 2019-10-13T12:15:37.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:37.416+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.416774+0000) 2019-10-13T12:15:37.418 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:37.416+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.416949+0000) 2019-10-13T12:15:37.418 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:37.416+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.417149+0000) 2019-10-13T12:15:37.419 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:37.417+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.417410+0000) 2019-10-13T12:15:37.419 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:37.417+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.417818+0000) 2019-10-13T12:15:37.419 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:37.417+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.417772+0000) 2019-10-13T12:15:37.420 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:37.417+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.418094+0000) 2019-10-13T12:15:37.423 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:37.419+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.420242+0000) 2019-10-13T12:15:37.427 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:37.426+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227676 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:38.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:38.659+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:38.660696+0000) 2019-10-13T12:15:38.992 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:38.990+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:38.991713+0000) 2019-10-13T12:15:41.312 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:15:41.312 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:41.317 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:15:41.317 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:41.612 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:41.610+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.611857+0000) 2019-10-13T12:15:42.430 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:42.428+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.429473+0000) 2019-10-13T12:15:42.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:42.428+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.429691+0000) 2019-10-13T12:15:42.431 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:42.428+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.429869+0000) 2019-10-13T12:15:42.431 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:42.428+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.430119+0000) 2019-10-13T12:15:42.432 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:42.430+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.430842+0000) 2019-10-13T12:15:42.432 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:42.428+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.430269+0000) 2019-10-13T12:15:42.432 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:42.429+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.430438+0000) 2019-10-13T12:15:42.433 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:42.431+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:42.432342+0000) 2019-10-13T12:15:42.439 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:42.437+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227706 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:43.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:43.095+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:43.097252+0000) 2019-10-13T12:15:43.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:43.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:43.101114+0000) 2019-10-13T12:15:45.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:45.755+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.756725+0000) 2019-10-13T12:15:45.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:45.755+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:45.756706+0000) 2019-10-13T12:15:46.135 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:46.134+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.134668+0000) 2019-10-13T12:15:47.442 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:47.439+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.441178+0000) 2019-10-13T12:15:47.442 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:47.439+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.441374+0000) 2019-10-13T12:15:47.443 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:47.439+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.441618+0000) 2019-10-13T12:15:47.443 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:47.440+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.441799+0000) 2019-10-13T12:15:47.443 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:47.440+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.442247+0000) 2019-10-13T12:15:47.444 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:47.440+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.442333+0000) 2019-10-13T12:15:47.444 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:47.441+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.442484+0000) 2019-10-13T12:15:47.444 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:47.442+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.444167+0000) 2019-10-13T12:15:47.450 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:47.448+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227738 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:48.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:48.660+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:48.660862+0000) 2019-10-13T12:15:48.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:48.991+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:48.991899+0000) 2019-10-13T12:15:51.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:51.611+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.612044+0000) 2019-10-13T12:15:52.452 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:52.450+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.451409+0000) 2019-10-13T12:15:52.453 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:52.450+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.451839+0000) 2019-10-13T12:15:52.453 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:52.450+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.451610+0000) 2019-10-13T12:15:52.453 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:52.450+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.452010+0000) 2019-10-13T12:15:52.453 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:52.451+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.452419+0000) 2019-10-13T12:15:52.454 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:52.451+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.452530+0000) 2019-10-13T12:15:52.454 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:52.451+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.452851+0000) 2019-10-13T12:15:52.454 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:52.452+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:52.454168+0000) 2019-10-13T12:15:52.462 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:52.460+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227766 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:53.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:53.096+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:53.097433+0000) 2019-10-13T12:15:53.101 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:53.100+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:53.101249+0000) 2019-10-13T12:15:55.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:55.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.756921+0000) 2019-10-13T12:15:55.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:55.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:55.756918+0000) 2019-10-13T12:15:56.138 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:56.136+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.137092+0000) 2019-10-13T12:15:57.463 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:15:57.461+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.462925+0000) 2019-10-13T12:15:57.464 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:57.462+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.463469+0000) 2019-10-13T12:15:57.464 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:15:57.462+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.463586+0000) 2019-10-13T12:15:57.465 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:15:57.462+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.463733+0000) 2019-10-13T12:15:57.465 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:15:57.462+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.464591+0000) 2019-10-13T12:15:57.466 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:15:57.463+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.464617+0000) 2019-10-13T12:15:57.466 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:57.463+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.464747+0000) 2019-10-13T12:15:57.467 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:15:57.465+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.466830+0000) 2019-10-13T12:15:57.473 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:15:57.471+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:15:58.661 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:15:58.660+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:58.661008+0000) 2019-10-13T12:15:58.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:15:58.991+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:58.992106+0000) 2019-10-13T12:16:01.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:01.611+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.612196+0000) 2019-10-13T12:16:02.475 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:02.473+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.474521+0000) 2019-10-13T12:16:02.476 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:02.473+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.474676+0000) 2019-10-13T12:16:02.476 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:02.473+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.474964+0000) 2019-10-13T12:16:02.476 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:02.473+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.475021+0000) 2019-10-13T12:16:02.477 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:02.473+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.475221+0000) 2019-10-13T12:16:02.477 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:02.473+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.475259+0000) 2019-10-13T12:16:02.477 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:02.474+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.475420+0000) 2019-10-13T12:16:02.477 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:02.475+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:02.476399+0000) 2019-10-13T12:16:02.484 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:02.482+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227828 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:03.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:03.097+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:03.097582+0000) 2019-10-13T12:16:03.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:03.100+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:03.101409+0000) 2019-10-13T12:16:05.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:05.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.757115+0000) 2019-10-13T12:16:05.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:05.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:05.757116+0000) 2019-10-13T12:16:06.140 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:06.138+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.139441+0000) 2019-10-13T12:16:07.487 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:07.484+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.485562+0000) 2019-10-13T12:16:07.487 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:07.484+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.485200+0000) 2019-10-13T12:16:07.487 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:07.485+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.485535+0000) 2019-10-13T12:16:07.487 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:07.485+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.485671+0000) 2019-10-13T12:16:07.487 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:07.485+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.485968+0000) 2019-10-13T12:16:07.488 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:07.485+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.486116+0000) 2019-10-13T12:16:07.488 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:07.486+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.486507+0000) 2019-10-13T12:16:07.488 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:07.487+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.488047+0000) 2019-10-13T12:16:07.495 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:07.494+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227860 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:08.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:08.660+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:08.661175+0000) 2019-10-13T12:16:08.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:08.991+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:08.992297+0000) 2019-10-13T12:16:11.492 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:16:11.492 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:11.497 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:16:11.497 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:11.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:11.612+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.612376+0000) 2019-10-13T12:16:12.498 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:12.496+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.497348+0000) 2019-10-13T12:16:12.498 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:12.496+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.497145+0000) 2019-10-13T12:16:12.499 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:12.497+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.497404+0000) 2019-10-13T12:16:12.499 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:12.497+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.497682+0000) 2019-10-13T12:16:12.499 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:12.497+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.497763+0000) 2019-10-13T12:16:12.500 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:12.497+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.498105+0000) 2019-10-13T12:16:12.500 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:12.497+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.497985+0000) 2019-10-13T12:16:12.500 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:12.499+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:12.499737+0000) 2019-10-13T12:16:12.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:12.505+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227888 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:13.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:13.097+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:13.097736+0000) 2019-10-13T12:16:13.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:13.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:13.101572+0000) 2019-10-13T12:16:15.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:15.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.757306+0000) 2019-10-13T12:16:15.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:15.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:15.757306+0000) 2019-10-13T12:16:16.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:16.141+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.142010+0000) 2019-10-13T12:16:17.509 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:17.507+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.507920+0000) 2019-10-13T12:16:17.509 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:17.507+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.507802+0000) 2019-10-13T12:16:17.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:17.508+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.508414+0000) 2019-10-13T12:16:17.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:17.507+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.508163+0000) 2019-10-13T12:16:17.510 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:17.508+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.508858+0000) 2019-10-13T12:16:17.510 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:17.508+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.509100+0000) 2019-10-13T12:16:17.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:17.508+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.509011+0000) 2019-10-13T12:16:17.511 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:17.509+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.509985+0000) 2019-10-13T12:16:17.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:17.516+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227922 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:18.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:18.661+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:18.661369+0000) 2019-10-13T12:16:18.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:18.992+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:18.992492+0000) 2019-10-13T12:16:21.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:21.612+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.612568+0000) 2019-10-13T12:16:22.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:22.518+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.519094+0000) 2019-10-13T12:16:22.520 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:22.518+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.519173+0000) 2019-10-13T12:16:22.520 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:22.518+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.519387+0000) 2019-10-13T12:16:22.521 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:22.519+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.519699+0000) 2019-10-13T12:16:22.521 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:22.519+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.519999+0000) 2019-10-13T12:16:22.522 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:22.519+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.520215+0000) 2019-10-13T12:16:22.522 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:22.519+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.520362+0000) 2019-10-13T12:16:22.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:22.520+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:22.521650+0000) 2019-10-13T12:16:22.529 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:22.528+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227950 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:23.098 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:23.096+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:23.097903+0000) 2019-10-13T12:16:23.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:23.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:23.101750+0000) 2019-10-13T12:16:25.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:25.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.757489+0000) 2019-10-13T12:16:25.758 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:25.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:25.757489+0000) 2019-10-13T12:16:26.145 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:26.144+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.144444+0000) 2019-10-13T12:16:27.531 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:27.530+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.530770+0000) 2019-10-13T12:16:27.532 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:27.529+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.531227+0000) 2019-10-13T12:16:27.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:27.530+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.531671+0000) 2019-10-13T12:16:27.533 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:27.530+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.531474+0000) 2019-10-13T12:16:27.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:27.530+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.531830+0000) 2019-10-13T12:16:27.533 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:27.530+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.531966+0000) 2019-10-13T12:16:27.533 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:27.530+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.532111+0000) 2019-10-13T12:16:27.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:27.532+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.533268+0000) 2019-10-13T12:16:27.541 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:27.538+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 227982 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:28.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:28.660+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:28.661602+0000) 2019-10-13T12:16:28.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:28.991+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:28.992657+0000) 2019-10-13T12:16:31.613 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:31.611+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.612757+0000) 2019-10-13T12:16:32.543 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:32.540+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.542231+0000) 2019-10-13T12:16:32.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:32.541+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.542436+0000) 2019-10-13T12:16:32.544 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:32.541+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.542706+0000) 2019-10-13T12:16:32.544 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:32.541+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.542864+0000) 2019-10-13T12:16:32.544 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:32.542+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.543101+0000) 2019-10-13T12:16:32.545 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:32.541+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.543121+0000) 2019-10-13T12:16:32.545 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:32.542+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.543613+0000) 2019-10-13T12:16:32.546 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:32.544+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:32.545631+0000) 2019-10-13T12:16:32.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:32.550+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:33.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:33.097+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:33.098069+0000) 2019-10-13T12:16:33.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:33.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:33.101909+0000) 2019-10-13T12:16:35.758 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:35.756+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.757672+0000) 2019-10-13T12:16:35.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:35.756+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:35.757671+0000) 2019-10-13T12:16:36.147 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:36.146+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.146755+0000) 2019-10-13T12:16:37.554 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:37.551+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.553268+0000) 2019-10-13T12:16:37.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:37.551+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.553336+0000) 2019-10-13T12:16:37.554 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:37.552+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.553621+0000) 2019-10-13T12:16:37.555 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:37.552+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.554085+0000) 2019-10-13T12:16:37.556 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:37.552+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.554287+0000) 2019-10-13T12:16:37.556 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:37.553+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.554422+0000) 2019-10-13T12:16:37.556 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:37.553+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.554686+0000) 2019-10-13T12:16:37.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:37.555+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.557037+0000) 2019-10-13T12:16:37.563 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:37.561+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228044 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:38.662 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:38.661+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:38.661814+0000) 2019-10-13T12:16:38.993 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:38.991+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:38.992833+0000) 2019-10-13T12:16:41.614 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:41.612+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.612976+0000) 2019-10-13T12:16:41.668 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:16:41.668 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:41.673 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:16:41.673 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:42.565 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:42.563+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.564735+0000) 2019-10-13T12:16:42.566 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:42.563+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.565083+0000) 2019-10-13T12:16:42.567 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:42.563+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.565015+0000) 2019-10-13T12:16:42.567 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:42.563+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.565255+0000) 2019-10-13T12:16:42.567 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:42.563+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.565380+0000) 2019-10-13T12:16:42.567 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:42.564+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.565706+0000) 2019-10-13T12:16:42.567 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:42.564+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.566021+0000) 2019-10-13T12:16:42.569 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:42.566+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:42.568471+0000) 2019-10-13T12:16:42.575 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:42.572+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228072 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:43.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:43.097+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:43.098241+0000) 2019-10-13T12:16:43.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:43.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:43.102070+0000) 2019-10-13T12:16:45.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:45.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.757834+0000) 2019-10-13T12:16:45.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:45.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:45.757833+0000) 2019-10-13T12:16:46.150 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:46.148+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.149021+0000) 2019-10-13T12:16:47.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:47.574+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.575636+0000) 2019-10-13T12:16:47.577 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:47.574+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.575902+0000) 2019-10-13T12:16:47.578 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:47.574+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.576234+0000) 2019-10-13T12:16:47.578 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:47.575+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.576416+0000) 2019-10-13T12:16:47.578 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:47.575+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.576561+0000) 2019-10-13T12:16:47.578 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:47.575+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.576750+0000) 2019-10-13T12:16:47.578 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:47.575+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.576806+0000) 2019-10-13T12:16:47.579 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:47.576+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.578411+0000) 2019-10-13T12:16:47.584 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:47.582+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228104 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:48.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:48.661+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:48.662006+0000) 2019-10-13T12:16:48.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:48.992+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:48.993041+0000) 2019-10-13T12:16:51.614 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:51.612+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.613167+0000) 2019-10-13T12:16:52.586 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:52.585+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.585521+0000) 2019-10-13T12:16:52.587 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:52.585+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.585772+0000) 2019-10-13T12:16:52.587 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:52.585+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.586098+0000) 2019-10-13T12:16:52.588 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:52.585+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.586216+0000) 2019-10-13T12:16:52.588 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:52.586+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.586582+0000) 2019-10-13T12:16:52.588 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:52.586+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.586702+0000) 2019-10-13T12:16:52.588 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:52.586+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.587064+0000) 2019-10-13T12:16:52.589 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:52.587+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:52.588520+0000) 2019-10-13T12:16:52.596 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:52.594+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228132 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:53.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:53.098+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:53.098441+0000) 2019-10-13T12:16:53.102 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:53.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:53.102184+0000) 2019-10-13T12:16:55.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:55.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.758011+0000) 2019-10-13T12:16:55.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:55.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:55.758009+0000) 2019-10-13T12:16:56.152 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:56.150+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.151378+0000) 2019-10-13T12:16:57.597 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:16:57.596+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.596943+0000) 2019-10-13T12:16:57.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:16:57.596+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.596971+0000) 2019-10-13T12:16:57.598 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:16:57.596+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.597381+0000) 2019-10-13T12:16:57.598 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:16:57.597+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.597552+0000) 2019-10-13T12:16:57.599 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:16:57.597+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.597977+0000) 2019-10-13T12:16:57.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:57.597+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.598040+0000) 2019-10-13T12:16:57.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:57.597+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.598173+0000) 2019-10-13T12:16:57.600 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:16:57.599+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.600090+0000) 2019-10-13T12:16:57.607 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:16:57.605+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228166 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:16:58.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:16:58.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:58.662244+0000) 2019-10-13T12:16:58.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:16:58.992+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:58.993199+0000) 2019-10-13T12:17:01.614 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:01.613+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.613332+0000) 2019-10-13T12:17:02.609 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:02.607+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.608223+0000) 2019-10-13T12:17:02.610 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:02.607+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.607975+0000) 2019-10-13T12:17:02.610 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:02.607+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.607963+0000) 2019-10-13T12:17:02.610 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:02.607+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.608261+0000) 2019-10-13T12:17:02.610 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:02.607+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.608378+0000) 2019-10-13T12:17:02.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:02.608+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.608572+0000) 2019-10-13T12:17:02.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:02.608+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.608795+0000) 2019-10-13T12:17:02.611 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:02.610+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:02.611158+0000) 2019-10-13T12:17:02.618 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:02.616+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228194 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:03.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:03.098+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:03.098608+0000) 2019-10-13T12:17:03.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:03.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:03.102347+0000) 2019-10-13T12:17:05.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:05.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.758247+0000) 2019-10-13T12:17:05.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:05.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:05.758243+0000) 2019-10-13T12:17:06.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:06.153+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.153675+0000) 2019-10-13T12:17:07.619 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:07.617+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.618682+0000) 2019-10-13T12:17:07.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:07.618+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.618737+0000) 2019-10-13T12:17:07.621 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:07.618+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.618958+0000) 2019-10-13T12:17:07.621 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:07.618+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.619010+0000) 2019-10-13T12:17:07.621 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:07.618+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.619190+0000) 2019-10-13T12:17:07.621 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:07.619+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.619437+0000) 2019-10-13T12:17:07.621 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:07.619+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.619657+0000) 2019-10-13T12:17:07.622 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:07.621+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.621808+0000) 2019-10-13T12:17:07.627 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:07.625+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228226 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:08.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:08.660+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:08.662390+0000) 2019-10-13T12:17:08.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:08.992+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:08.993379+0000) 2019-10-13T12:17:11.614 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:11.612+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.613492+0000) 2019-10-13T12:17:11.762 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:17:11.762 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:11.767 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:17:11.767 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:12.629 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:12.626+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.628133+0000) 2019-10-13T12:17:12.629 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:12.626+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.628290+0000) 2019-10-13T12:17:12.630 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:12.627+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.628605+0000) 2019-10-13T12:17:12.630 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:12.627+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.628925+0000) 2019-10-13T12:17:12.630 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:12.627+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.629175+0000) 2019-10-13T12:17:12.631 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:12.628+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.629101+0000) 2019-10-13T12:17:12.631 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:12.628+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.629504+0000) 2019-10-13T12:17:12.632 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:12.630+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:12.631447+0000) 2019-10-13T12:17:12.638 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:12.636+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228254 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:13.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:13.097+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:13.098769+0000) 2019-10-13T12:17:13.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:13.102+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:13.102479+0000) 2019-10-13T12:17:15.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:15.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.758439+0000) 2019-10-13T12:17:15.759 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:15.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:15.758439+0000) 2019-10-13T12:17:16.157 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:16.155+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.156072+0000) 2019-10-13T12:17:17.641 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:17.638+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.639923+0000) 2019-10-13T12:17:17.641 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:17.638+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.640306+0000) 2019-10-13T12:17:17.641 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:17.639+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.640459+0000) 2019-10-13T12:17:17.642 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:17.639+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.640329+0000) 2019-10-13T12:17:17.642 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:17.639+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.640680+0000) 2019-10-13T12:17:17.643 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:17.639+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.640739+0000) 2019-10-13T12:17:17.643 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:17.639+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.640882+0000) 2019-10-13T12:17:17.643 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:17.641+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.642027+0000) 2019-10-13T12:17:17.650 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:17.648+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:18.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:18.661+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:18.662553+0000) 2019-10-13T12:17:18.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:18.992+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:18.993587+0000) 2019-10-13T12:17:21.614 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:21.612+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.613693+0000) 2019-10-13T12:17:22.652 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:22.649+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.651134+0000) 2019-10-13T12:17:22.653 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:22.650+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.651660+0000) 2019-10-13T12:17:22.653 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:22.650+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.651690+0000) 2019-10-13T12:17:22.653 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:22.650+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.651900+0000) 2019-10-13T12:17:22.653 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:22.650+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.652285+0000) 2019-10-13T12:17:22.654 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:22.651+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.652602+0000) 2019-10-13T12:17:22.654 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:22.651+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.652891+0000) 2019-10-13T12:17:22.655 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:22.654+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:22.655012+0000) 2019-10-13T12:17:22.662 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:22.659+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228316 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:23.099 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:23.098+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:23.098883+0000) 2019-10-13T12:17:23.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:23.102+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:23.102644+0000) 2019-10-13T12:17:25.759 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:25.757+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.758645+0000) 2019-10-13T12:17:25.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:25.757+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:25.758645+0000) 2019-10-13T12:17:26.159 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:26.158+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.158471+0000) 2019-10-13T12:17:27.663 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:27.661+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.662783+0000) 2019-10-13T12:17:27.664 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:27.661+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.662820+0000) 2019-10-13T12:17:27.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:27.661+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.663179+0000) 2019-10-13T12:17:27.665 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:27.661+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.663283+0000) 2019-10-13T12:17:27.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:27.662+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.663569+0000) 2019-10-13T12:17:27.665 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:27.662+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.663996+0000) 2019-10-13T12:17:27.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:27.662+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.664054+0000) 2019-10-13T12:17:27.666 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:27.664+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.665879+0000) 2019-10-13T12:17:27.673 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:27.670+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228348 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:28.663 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:28.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:28.662679+0000) 2019-10-13T12:17:28.994 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:28.993+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:28.993786+0000) 2019-10-13T12:17:31.614 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:31.613+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.613892+0000) 2019-10-13T12:17:32.675 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:32.672+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.673902+0000) 2019-10-13T12:17:32.675 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:32.672+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.674123+0000) 2019-10-13T12:17:32.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:32.672+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.674308+0000) 2019-10-13T12:17:32.676 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:32.673+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.674579+0000) 2019-10-13T12:17:32.676 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:32.673+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.675015+0000) 2019-10-13T12:17:32.676 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:32.673+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.675015+0000) 2019-10-13T12:17:32.677 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:32.673+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.675335+0000) 2019-10-13T12:17:32.678 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:32.675+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:32.677452+0000) 2019-10-13T12:17:32.684 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:32.682+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228376 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:33.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:33.098+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:33.099033+0000) 2019-10-13T12:17:33.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:33.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:33.102775+0000) 2019-10-13T12:17:35.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:35.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.758826+0000) 2019-10-13T12:17:35.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:35.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:35.758827+0000) 2019-10-13T12:17:36.161 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:36.159+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.160790+0000) 2019-10-13T12:17:37.686 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:37.684+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.685376+0000) 2019-10-13T12:17:37.686 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:37.685+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.685758+0000) 2019-10-13T12:17:37.687 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:37.683+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.685675+0000) 2019-10-13T12:17:37.688 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:37.686+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.687220+0000) 2019-10-13T12:17:37.688 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:37.686+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.687346+0000) 2019-10-13T12:17:37.688 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:37.687+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.687667+0000) 2019-10-13T12:17:37.689 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:37.687+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.688092+0000) 2019-10-13T12:17:37.691 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:37.688+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.690571+0000) 2019-10-13T12:17:37.695 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:37.693+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228410 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:38.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:38.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:38.662849+0000) 2019-10-13T12:17:38.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:38.993+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:38.993978+0000) 2019-10-13T12:17:41.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:41.613+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.614070+0000) 2019-10-13T12:17:41.909 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:17:41.909 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:41.914 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:17:41.914 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:42.697 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:42.695+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.695944+0000) 2019-10-13T12:17:42.697 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:42.695+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.696269+0000) 2019-10-13T12:17:42.698 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:42.695+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.696664+0000) 2019-10-13T12:17:42.698 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:42.696+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.696740+0000) 2019-10-13T12:17:42.698 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:42.696+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.697076+0000) 2019-10-13T12:17:42.698 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:42.696+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.697242+0000) 2019-10-13T12:17:42.699 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:42.697+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.697436+0000) 2019-10-13T12:17:42.699 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:42.698+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:42.698991+0000) 2019-10-13T12:17:42.706 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:42.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228438 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:43.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:43.098+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:43.099186+0000) 2019-10-13T12:17:43.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:43.102+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:43.102908+0000) 2019-10-13T12:17:45.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:45.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.759014+0000) 2019-10-13T12:17:45.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:45.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:45.759014+0000) 2019-10-13T12:17:46.163 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:46.162+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.163060+0000) 2019-10-13T12:17:47.708 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:47.706+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.707337+0000) 2019-10-13T12:17:47.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:47.707+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.707802+0000) 2019-10-13T12:17:47.709 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:47.707+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.707925+0000) 2019-10-13T12:17:47.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:47.707+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.708421+0000) 2019-10-13T12:17:47.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:47.708+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.708517+0000) 2019-10-13T12:17:47.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:47.708+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.708667+0000) 2019-10-13T12:17:47.710 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:47.708+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.708915+0000) 2019-10-13T12:17:47.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:47.709+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.709951+0000) 2019-10-13T12:17:47.718 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:47.716+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228470 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:48.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:48.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:48.663040+0000) 2019-10-13T12:17:48.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:48.993+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:48.994142+0000) 2019-10-13T12:17:51.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:51.614+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.614235+0000) 2019-10-13T12:17:52.719 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:52.718+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.718757+0000) 2019-10-13T12:17:52.720 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:52.718+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.719175+0000) 2019-10-13T12:17:52.720 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:52.719+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.719479+0000) 2019-10-13T12:17:52.721 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:52.719+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.719897+0000) 2019-10-13T12:17:52.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:52.719+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.720186+0000) 2019-10-13T12:17:52.721 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:52.719+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.720225+0000) 2019-10-13T12:17:52.722 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:52.720+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.720448+0000) 2019-10-13T12:17:52.722 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:52.720+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:52.721154+0000) 2019-10-13T12:17:52.729 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:52.727+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228498 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:53.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:53.099+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:53.099296+0000) 2019-10-13T12:17:53.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:53.102+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:53.103063+0000) 2019-10-13T12:17:55.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:55.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.759215+0000) 2019-10-13T12:17:55.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:55.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:55.759211+0000) 2019-10-13T12:17:56.166 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:56.165+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.165505+0000) 2019-10-13T12:17:57.731 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:57.728+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.730227+0000) 2019-10-13T12:17:57.731 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:17:57.729+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.730962+0000) 2019-10-13T12:17:57.732 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:17:57.729+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.731140+0000) 2019-10-13T12:17:57.733 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:17:57.730+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.731426+0000) 2019-10-13T12:17:57.733 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:17:57.730+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.731559+0000) 2019-10-13T12:17:57.733 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:57.730+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.731921+0000) 2019-10-13T12:17:57.734 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:17:57.731+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.732236+0000) 2019-10-13T12:17:57.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:17:57.732+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.732790+0000) 2019-10-13T12:17:57.741 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:17:57.739+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228532 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:17:58.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:17:58.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:58.663188+0000) 2019-10-13T12:17:58.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:17:58.993+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:58.994299+0000) 2019-10-13T12:18:01.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:01.613+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.614382+0000) 2019-10-13T12:18:02.743 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:02.741+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.742502+0000) 2019-10-13T12:18:02.744 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:02.741+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.742557+0000) 2019-10-13T12:18:02.744 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:02.741+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.742746+0000) 2019-10-13T12:18:02.745 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:02.741+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.743011+0000) 2019-10-13T12:18:02.745 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:02.741+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.742876+0000) 2019-10-13T12:18:02.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:02.741+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.743199+0000) 2019-10-13T12:18:02.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:02.741+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.743331+0000) 2019-10-13T12:18:02.746 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:02.745+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:02.745952+0000) 2019-10-13T12:18:02.753 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:02.750+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:03.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:03.098+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:03.099463+0000) 2019-10-13T12:18:03.103 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:03.102+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:03.103237+0000) 2019-10-13T12:18:05.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:05.758+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.759420+0000) 2019-10-13T12:18:05.760 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:05.758+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:05.759419+0000) 2019-10-13T12:18:06.168 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:06.167+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.167609+0000) 2019-10-13T12:18:07.755 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:07.753+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.754221+0000) 2019-10-13T12:18:07.756 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:07.752+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.754251+0000) 2019-10-13T12:18:07.756 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:07.753+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.754691+0000) 2019-10-13T12:18:07.756 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:07.753+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.754829+0000) 2019-10-13T12:18:07.756 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:07.753+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.755064+0000) 2019-10-13T12:18:07.757 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:07.753+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.755185+0000) 2019-10-13T12:18:07.757 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:07.754+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.755558+0000) 2019-10-13T12:18:07.757 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:07.755+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.756593+0000) 2019-10-13T12:18:07.763 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:07.761+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228592 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:08.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:08.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:08.663338+0000) 2019-10-13T12:18:08.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:08.993+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:08.994519+0000) 2019-10-13T12:18:11.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:11.613+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.614553+0000) 2019-10-13T12:18:12.014 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:18:12.014 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:12.018 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:18:12.019 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:12.765 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:12.763+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.764911+0000) 2019-10-13T12:18:12.766 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:12.763+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.765169+0000) 2019-10-13T12:18:12.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:12.764+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.765483+0000) 2019-10-13T12:18:12.767 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:12.764+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.765645+0000) 2019-10-13T12:18:12.767 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:12.765+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.765985+0000) 2019-10-13T12:18:12.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:12.764+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.766193+0000) 2019-10-13T12:18:12.768 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:12.765+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.766453+0000) 2019-10-13T12:18:12.768 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:12.767+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:12.768275+0000) 2019-10-13T12:18:12.775 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:12.773+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228620 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:13.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:13.099+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:13.099613+0000) 2019-10-13T12:18:13.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:13.103+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:13.103374+0000) 2019-10-13T12:18:15.760 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:15.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.759608+0000) 2019-10-13T12:18:15.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:15.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:15.759628+0000) 2019-10-13T12:18:16.170 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:16.169+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.169949+0000) 2019-10-13T12:18:17.778 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:17.775+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.776728+0000) 2019-10-13T12:18:17.778 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:17.775+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.776891+0000) 2019-10-13T12:18:17.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:17.775+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.776999+0000) 2019-10-13T12:18:17.779 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:17.776+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.778062+0000) 2019-10-13T12:18:17.779 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:17.775+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.777283+0000) 2019-10-13T12:18:17.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:17.776+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.777710+0000) 2019-10-13T12:18:17.779 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:17.776+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.777851+0000) 2019-10-13T12:18:17.780 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:17.778+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.779855+0000) 2019-10-13T12:18:17.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:17.784+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228654 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:18.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:18.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:18.663472+0000) 2019-10-13T12:18:18.995 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:18.994+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:18.994703+0000) 2019-10-13T12:18:21.615 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:21.614+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.614756+0000) 2019-10-13T12:18:22.788 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:22.786+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.786923+0000) 2019-10-13T12:18:22.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:22.786+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.787212+0000) 2019-10-13T12:18:22.788 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:22.786+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.787065+0000) 2019-10-13T12:18:22.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:22.786+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.787340+0000) 2019-10-13T12:18:22.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:22.785+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.787565+0000) 2019-10-13T12:18:22.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:22.787+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.787736+0000) 2019-10-13T12:18:22.790 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:22.787+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.788226+0000) 2019-10-13T12:18:22.790 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:22.788+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:22.790252+0000) 2019-10-13T12:18:22.796 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:22.795+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228682 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:23.100 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:23.099+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:23.099788+0000) 2019-10-13T12:18:23.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:23.101+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:23.103535+0000) 2019-10-13T12:18:25.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:25.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.759789+0000) 2019-10-13T12:18:25.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:25.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:25.759789+0000) 2019-10-13T12:18:26.173 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:26.171+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.172531+0000) 2019-10-13T12:18:27.798 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:27.796+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.797127+0000) 2019-10-13T12:18:27.798 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:27.797+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.797426+0000) 2019-10-13T12:18:27.799 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:27.797+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.797716+0000) 2019-10-13T12:18:27.799 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:27.795+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.797624+0000) 2019-10-13T12:18:27.800 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:27.797+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.798299+0000) 2019-10-13T12:18:27.800 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:27.798+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.798424+0000) 2019-10-13T12:18:27.800 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:27.798+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.798646+0000) 2019-10-13T12:18:27.800 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:27.798+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.800039+0000) 2019-10-13T12:18:27.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:27.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228714 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:28.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:28.663+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:28.663633+0000) 2019-10-13T12:18:28.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:28.994+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:28.994894+0000) 2019-10-13T12:18:31.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:31.614+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.614947+0000) 2019-10-13T12:18:32.809 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:32.808+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.808466+0000) 2019-10-13T12:18:32.810 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:32.808+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.808749+0000) 2019-10-13T12:18:32.811 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:32.808+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.809705+0000) 2019-10-13T12:18:32.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:32.808+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.809374+0000) 2019-10-13T12:18:32.811 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:32.809+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.809784+0000) 2019-10-13T12:18:32.812 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:32.809+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.810003+0000) 2019-10-13T12:18:32.812 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:32.810+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.810489+0000) 2019-10-13T12:18:32.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:32.810+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:32.811133+0000) 2019-10-13T12:18:32.818 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:32.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228742 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:33.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:33.099+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:33.099952+0000) 2019-10-13T12:18:33.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:33.102+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:33.103693+0000) 2019-10-13T12:18:35.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:35.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.759985+0000) 2019-10-13T12:18:35.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:35.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:35.759984+0000) 2019-10-13T12:18:36.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:36.174+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.174930+0000) 2019-10-13T12:18:37.820 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:37.819+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.819414+0000) 2019-10-13T12:18:37.821 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:37.819+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.820281+0000) 2019-10-13T12:18:37.821 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:37.819+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.819918+0000) 2019-10-13T12:18:37.822 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:37.819+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.819973+0000) 2019-10-13T12:18:37.822 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:37.819+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.820331+0000) 2019-10-13T12:18:37.822 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:37.820+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.820582+0000) 2019-10-13T12:18:37.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:37.820+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.820766+0000) 2019-10-13T12:18:37.823 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:37.821+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.822596+0000) 2019-10-13T12:18:37.830 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:37.829+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228776 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:38.664 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:38.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:38.663808+0000) 2019-10-13T12:18:38.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:38.994+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:38.995098+0000) 2019-10-13T12:18:41.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:41.614+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.615135+0000) 2019-10-13T12:18:42.198 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:18:42.198 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:42.203 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:18:42.203 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:42.832 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:42.830+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.831285+0000) 2019-10-13T12:18:42.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:42.829+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.831300+0000) 2019-10-13T12:18:42.833 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:42.830+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.831477+0000) 2019-10-13T12:18:42.833 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:42.830+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.831769+0000) 2019-10-13T12:18:42.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:42.830+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.832047+0000) 2019-10-13T12:18:42.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:42.830+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.832311+0000) 2019-10-13T12:18:42.834 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:42.831+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.832462+0000) 2019-10-13T12:18:42.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:42.833+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:42.834255+0000) 2019-10-13T12:18:42.841 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:42.839+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228804 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:43.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:43.099+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:43.100120+0000) 2019-10-13T12:18:43.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:43.103+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:43.103856+0000) 2019-10-13T12:18:45.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:45.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.760198+0000) 2019-10-13T12:18:45.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:45.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:45.760198+0000) 2019-10-13T12:18:46.178 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:46.176+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.177237+0000) 2019-10-13T12:18:47.843 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:47.842+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.842878+0000) 2019-10-13T12:18:47.844 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:47.841+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.842760+0000) 2019-10-13T12:18:47.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:47.841+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.842974+0000) 2019-10-13T12:18:47.845 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:47.841+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.843088+0000) 2019-10-13T12:18:47.845 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:47.841+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.843337+0000) 2019-10-13T12:18:47.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:47.842+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.843952+0000) 2019-10-13T12:18:47.845 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:47.842+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.843651+0000) 2019-10-13T12:18:47.846 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:47.843+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.844694+0000) 2019-10-13T12:18:47.852 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:47.850+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228836 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:48.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:48.663+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:48.664003+0000) 2019-10-13T12:18:48.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:48.994+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:48.995262+0000) 2019-10-13T12:18:51.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:51.614+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.615289+0000) 2019-10-13T12:18:52.855 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:52.852+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.853835+0000) 2019-10-13T12:18:52.855 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:52.852+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.854076+0000) 2019-10-13T12:18:52.855 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:52.852+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.854253+0000) 2019-10-13T12:18:52.856 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:52.853+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.854449+0000) 2019-10-13T12:18:52.856 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:52.854+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.854886+0000) 2019-10-13T12:18:52.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:52.853+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.855115+0000) 2019-10-13T12:18:52.857 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:52.853+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.855289+0000) 2019-10-13T12:18:52.857 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:52.856+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:52.856919+0000) 2019-10-13T12:18:52.864 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:52.861+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228864 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:53.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:53.099+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:53.100292+0000) 2019-10-13T12:18:53.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:53.103+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:53.103998+0000) 2019-10-13T12:18:55.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:55.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.760390+0000) 2019-10-13T12:18:55.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:55.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:55.760389+0000) 2019-10-13T12:18:56.180 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:56.179+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.179569+0000) 2019-10-13T12:18:57.866 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:18:57.863+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.864916+0000) 2019-10-13T12:18:57.867 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:18:57.865+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.865752+0000) 2019-10-13T12:18:57.867 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:57.863+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.865290+0000) 2019-10-13T12:18:57.867 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:57.864+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.865792+0000) 2019-10-13T12:18:57.867 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:18:57.864+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.866018+0000) 2019-10-13T12:18:57.867 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:18:57.864+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.865926+0000) 2019-10-13T12:18:57.868 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:18:57.864+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.866245+0000) 2019-10-13T12:18:57.868 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:18:57.866+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.867268+0000) 2019-10-13T12:18:57.875 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:18:57.873+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228898 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:18:58.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:18:58.662+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:58.664139+0000) 2019-10-13T12:18:58.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:18:58.994+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:58.995413+0000) 2019-10-13T12:19:01.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:01.614+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.615470+0000) 2019-10-13T12:19:02.877 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:02.874+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.876372+0000) 2019-10-13T12:19:02.877 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:02.875+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.876763+0000) 2019-10-13T12:19:02.878 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:02.876+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.877529+0000) 2019-10-13T12:19:02.878 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:02.875+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.876930+0000) 2019-10-13T12:19:02.879 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:02.875+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.877298+0000) 2019-10-13T12:19:02.879 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:02.876+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.877594+0000) 2019-10-13T12:19:02.879 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:02.876+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.877953+0000) 2019-10-13T12:19:02.879 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:02.878+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:02.878849+0000) 2019-10-13T12:19:02.886 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:02.884+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:03.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:03.099+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:03.100468+0000) 2019-10-13T12:19:03.104 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:03.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:03.104153+0000) 2019-10-13T12:19:05.761 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:05.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.760599+0000) 2019-10-13T12:19:05.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:05.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:05.760599+0000) 2019-10-13T12:19:06.182 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:06.181+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.182035+0000) 2019-10-13T12:19:07.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:07.887+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.887740+0000) 2019-10-13T12:19:07.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:07.887+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.888009+0000) 2019-10-13T12:19:07.890 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:07.887+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.888275+0000) 2019-10-13T12:19:07.890 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:07.887+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.888152+0000) 2019-10-13T12:19:07.890 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:07.888+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.888578+0000) 2019-10-13T12:19:07.890 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:07.888+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.888686+0000) 2019-10-13T12:19:07.891 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:07.887+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.888930+0000) 2019-10-13T12:19:07.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:07.889+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.890807+0000) 2019-10-13T12:19:07.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:07.897+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228958 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:08.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:08.663+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:08.664291+0000) 2019-10-13T12:19:08.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:08.995+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:08.995570+0000) 2019-10-13T12:19:11.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:11.615+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.615645+0000) 2019-10-13T12:19:12.362 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:19:12.363 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:12.367 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:19:12.367 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:12.901 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:12.899+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.900365+0000) 2019-10-13T12:19:12.902 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:12.900+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.900691+0000) 2019-10-13T12:19:12.902 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:12.900+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.900649+0000) 2019-10-13T12:19:12.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:12.900+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.901045+0000) 2019-10-13T12:19:12.903 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:12.899+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.900870+0000) 2019-10-13T12:19:12.903 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:12.900+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.901330+0000) 2019-10-13T12:19:12.903 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:12.901+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.901700+0000) 2019-10-13T12:19:12.903 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:12.901+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:12.903134+0000) 2019-10-13T12:19:12.910 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:12.909+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 228986 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:13.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:13.100+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:13.100602+0000) 2019-10-13T12:19:13.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:13.102+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:13.104308+0000) 2019-10-13T12:19:15.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:15.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.760796+0000) 2019-10-13T12:19:15.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:15.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:15.760800+0000) 2019-10-13T12:19:16.185 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:16.183+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.184341+0000) 2019-10-13T12:19:17.913 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:17.911+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.911666+0000) 2019-10-13T12:19:17.913 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:17.911+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.911943+0000) 2019-10-13T12:19:17.913 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:17.911+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.912110+0000) 2019-10-13T12:19:17.913 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:17.912+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.912408+0000) 2019-10-13T12:19:17.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:17.910+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.912251+0000) 2019-10-13T12:19:17.914 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:17.912+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.912713+0000) 2019-10-13T12:19:17.915 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:17.912+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.912975+0000) 2019-10-13T12:19:17.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:17.913+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.915461+0000) 2019-10-13T12:19:17.922 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:17.920+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229020 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:18.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:18.664+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:18.664437+0000) 2019-10-13T12:19:18.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:18.995+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:18.995756+0000) 2019-10-13T12:19:21.616 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:21.615+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.615784+0000) 2019-10-13T12:19:22.923 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:22.921+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.922570+0000) 2019-10-13T12:19:22.924 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:22.922+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.923074+0000) 2019-10-13T12:19:22.924 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:22.922+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.923307+0000) 2019-10-13T12:19:22.925 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:22.922+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.922955+0000) 2019-10-13T12:19:22.925 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:22.923+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.923439+0000) 2019-10-13T12:19:22.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:22.923+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.923666+0000) 2019-10-13T12:19:22.926 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:22.923+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.923748+0000) 2019-10-13T12:19:22.926 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:22.924+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:22.924910+0000) 2019-10-13T12:19:22.933 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:22.931+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229048 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:23.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:23.100+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:23.100786+0000) 2019-10-13T12:19:23.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:23.103+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:23.104485+0000) 2019-10-13T12:19:25.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:25.759+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.760992+0000) 2019-10-13T12:19:25.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:25.759+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:25.760999+0000) 2019-10-13T12:19:26.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:26.186+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.186839+0000) 2019-10-13T12:19:27.934 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:27.932+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.933892+0000) 2019-10-13T12:19:27.935 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:27.933+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.934443+0000) 2019-10-13T12:19:27.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:27.933+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.934559+0000) 2019-10-13T12:19:27.936 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:27.933+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.934869+0000) 2019-10-13T12:19:27.936 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:27.934+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.935298+0000) 2019-10-13T12:19:27.937 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:27.933+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.935380+0000) 2019-10-13T12:19:27.937 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:27.934+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.935550+0000) 2019-10-13T12:19:27.937 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:27.935+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.936397+0000) 2019-10-13T12:19:27.945 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:27.943+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229080 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:28.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:28.663+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:28.664586+0000) 2019-10-13T12:19:28.996 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:28.995+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:28.995959+0000) 2019-10-13T12:19:31.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:31.615+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.615980+0000) 2019-10-13T12:19:32.947 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:32.945+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.946067+0000) 2019-10-13T12:19:32.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:32.944+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.946146+0000) 2019-10-13T12:19:32.948 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:32.944+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.946321+0000) 2019-10-13T12:19:32.948 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:32.945+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.946392+0000) 2019-10-13T12:19:32.948 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:32.945+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.946651+0000) 2019-10-13T12:19:32.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:32.945+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.946812+0000) 2019-10-13T12:19:32.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:32.945+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.947205+0000) 2019-10-13T12:19:32.952 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:32.948+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:32.949058+0000) 2019-10-13T12:19:32.954 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:32.951+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229107 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:33.101 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:33.100+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:33.100929+0000) 2019-10-13T12:19:33.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:33.103+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:33.104677+0000) 2019-10-13T12:19:35.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:35.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.761218+0000) 2019-10-13T12:19:35.762 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:35.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:35.761218+0000) 2019-10-13T12:19:36.190 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:36.188+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.189109+0000) 2019-10-13T12:19:37.956 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:37.953+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.955050+0000) 2019-10-13T12:19:37.956 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:37.953+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.955054+0000) 2019-10-13T12:19:37.957 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:37.953+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.955238+0000) 2019-10-13T12:19:37.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:37.954+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.955663+0000) 2019-10-13T12:19:37.957 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:37.954+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.955810+0000) 2019-10-13T12:19:37.958 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:37.955+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.955930+0000) 2019-10-13T12:19:37.958 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:37.954+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.956083+0000) 2019-10-13T12:19:37.958 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:37.956+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.957509+0000) 2019-10-13T12:19:37.963 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:37.960+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229141 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:38.665 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:38.663+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:38.664692+0000) 2019-10-13T12:19:38.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:38.994+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:38.996179+0000) 2019-10-13T12:19:41.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:41.614+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.616165+0000) 2019-10-13T12:19:42.568 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:19:42.568 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:42.573 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:19:42.573 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:42.965 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:42.962+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.964194+0000) 2019-10-13T12:19:42.965 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:42.962+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.964383+0000) 2019-10-13T12:19:42.966 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:42.964+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.965288+0000) 2019-10-13T12:19:42.966 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:42.963+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.964791+0000) 2019-10-13T12:19:42.966 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:42.963+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.964953+0000) 2019-10-13T12:19:42.967 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:42.964+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.965497+0000) 2019-10-13T12:19:42.967 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:42.963+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.965330+0000) 2019-10-13T12:19:42.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:42.966+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:42.967267+0000) 2019-10-13T12:19:42.974 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:42.972+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229168 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:43.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:43.100+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:43.101073+0000) 2019-10-13T12:19:43.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:43.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:43.104865+0000) 2019-10-13T12:19:45.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:45.760+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.761437+0000) 2019-10-13T12:19:45.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:45.760+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:45.761442+0000) 2019-10-13T12:19:46.192 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:46.190+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.191350+0000) 2019-10-13T12:19:47.976 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:47.973+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.974985+0000) 2019-10-13T12:19:47.976 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:47.973+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.975070+0000) 2019-10-13T12:19:47.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:47.973+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.975187+0000) 2019-10-13T12:19:47.977 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:47.973+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.975371+0000) 2019-10-13T12:19:47.977 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:47.974+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.975734+0000) 2019-10-13T12:19:47.978 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:47.974+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.975701+0000) 2019-10-13T12:19:47.978 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:47.974+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.975962+0000) 2019-10-13T12:19:47.978 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:47.976+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.977621+0000) 2019-10-13T12:19:47.983 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:47.981+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229200 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:48.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:48.664+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:48.664833+0000) 2019-10-13T12:19:48.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:48.994+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:48.996377+0000) 2019-10-13T12:19:51.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:51.615+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.616345+0000) 2019-10-13T12:19:52.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:52.983+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.984250+0000) 2019-10-13T12:19:52.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:52.984+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.984488+0000) 2019-10-13T12:19:52.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:52.984+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.984678+0000) 2019-10-13T12:19:52.986 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:52.984+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.984783+0000) 2019-10-13T12:19:52.986 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:52.984+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.984886+0000) 2019-10-13T12:19:52.987 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:52.985+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.985851+0000) 2019-10-13T12:19:52.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:52.985+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.986214+0000) 2019-10-13T12:19:52.988 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:52.986+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:52.987557+0000) 2019-10-13T12:19:52.994 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:52.993+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229227 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:53.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:53.100+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:53.101215+0000) 2019-10-13T12:19:53.105 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:53.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:53.105035+0000) 2019-10-13T12:19:55.762 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:55.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.761637+0000) 2019-10-13T12:19:55.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:55.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:55.761620+0000) 2019-10-13T12:19:56.194 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:56.192+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.193664+0000) 2019-10-13T12:19:57.997 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:19:57.995+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.995405+0000) 2019-10-13T12:19:57.997 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:57.994+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.995388+0000) 2019-10-13T12:19:57.997 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:19:57.995+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.995600+0000) 2019-10-13T12:19:57.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:57.995+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.996061+0000) 2019-10-13T12:19:57.998 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:19:57.995+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.996281+0000) 2019-10-13T12:19:57.998 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:19:57.995+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.995950+0000) 2019-10-13T12:19:57.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:19:57.996+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.996649+0000) 2019-10-13T12:19:57.999 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:19:57.997+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.997882+0000) 2019-10-13T12:19:58.006 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:19:58.004+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229261 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:19:58.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:19:58.664+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:58.665012+0000) 2019-10-13T12:19:58.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:19:58.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:58.996530+0000) 2019-10-13T12:20:01.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:01.616+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:01.616546+0000) 2019-10-13T12:20:03.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:03.006+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.006452+0000) 2019-10-13T12:20:03.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:03.006+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.006672+0000) 2019-10-13T12:20:03.008 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:03.006+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.006757+0000) 2019-10-13T12:20:03.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:03.006+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.006879+0000) 2019-10-13T12:20:03.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:03.006+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.007056+0000) 2019-10-13T12:20:03.009 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:03.006+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.007292+0000) 2019-10-13T12:20:03.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:03.004+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.006655+0000) 2019-10-13T12:20:03.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:03.007+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.009450+0000) 2019-10-13T12:20:03.014 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:03.013+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229288 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:03.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:03.101+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.101354+0000) 2019-10-13T12:20:03.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:03.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:03.105209+0000) 2019-10-13T12:20:05.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:05.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.761837+0000) 2019-10-13T12:20:05.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:05.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:05.761838+0000) 2019-10-13T12:20:06.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:06.194+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:06.196011+0000) 2019-10-13T12:20:08.016 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:08.015+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.015809+0000) 2019-10-13T12:20:08.018 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:08.014+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.016212+0000) 2019-10-13T12:20:08.018 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:08.015+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.016291+0000) 2019-10-13T12:20:08.018 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:08.016+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.016394+0000) 2019-10-13T12:20:08.018 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:08.016+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.016587+0000) 2019-10-13T12:20:08.019 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:08.016+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.016761+0000) 2019-10-13T12:20:08.019 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:08.016+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.017084+0000) 2019-10-13T12:20:08.020 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:08.017+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.019342+0000) 2019-10-13T12:20:08.026 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:08.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229320 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:08.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:08.665+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.665193+0000) 2019-10-13T12:20:08.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:08.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.996659+0000) 2019-10-13T12:20:11.617 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:11.615+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:11.616735+0000) 2019-10-13T12:20:12.691 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:20:12.691 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:12.696 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:20:12.696 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:13.028 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:13.026+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.027668+0000) 2019-10-13T12:20:13.029 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:13.026+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.027770+0000) 2019-10-13T12:20:13.030 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:13.027+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.028266+0000) 2019-10-13T12:20:13.030 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:13.026+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.028328+0000) 2019-10-13T12:20:13.030 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:13.027+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.028470+0000) 2019-10-13T12:20:13.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:13.027+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.028917+0000) 2019-10-13T12:20:13.030 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:13.027+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.029029+0000) 2019-10-13T12:20:13.031 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:13.029+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.030132+0000) 2019-10-13T12:20:13.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:13.035+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229347 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:13.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:13.100+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.101497+0000) 2019-10-13T12:20:13.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:13.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:13.105340+0000) 2019-10-13T12:20:15.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:15.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.762061+0000) 2019-10-13T12:20:15.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:15.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:15.762060+0000) 2019-10-13T12:20:16.200 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:16.197+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:16.198367+0000) 2019-10-13T12:20:18.040 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:18.038+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.038940+0000) 2019-10-13T12:20:18.041 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:18.037+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.039204+0000) 2019-10-13T12:20:18.041 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:18.037+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.039330+0000) 2019-10-13T12:20:18.041 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:18.038+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.039461+0000) 2019-10-13T12:20:18.041 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:18.038+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.039770+0000) 2019-10-13T12:20:18.041 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:18.038+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.040214+0000) 2019-10-13T12:20:18.042 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:18.039+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.040440+0000) 2019-10-13T12:20:18.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:18.041+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.041840+0000) 2019-10-13T12:20:18.049 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:18.047+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229381 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:18.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:18.664+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.665358+0000) 2019-10-13T12:20:18.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:18.995+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.996811+0000) 2019-10-13T12:20:21.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:21.615+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:21.616939+0000) 2019-10-13T12:20:23.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:23.049+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.050513+0000) 2019-10-13T12:20:23.052 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:23.049+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.050631+0000) 2019-10-13T12:20:23.052 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:23.049+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.050797+0000) 2019-10-13T12:20:23.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:23.049+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.051088+0000) 2019-10-13T12:20:23.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:23.049+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.050940+0000) 2019-10-13T12:20:23.053 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:23.050+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.051502+0000) 2019-10-13T12:20:23.053 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:23.050+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.052027+0000) 2019-10-13T12:20:23.055 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:23.053+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.054677+0000) 2019-10-13T12:20:23.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:23.059+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229408 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:23.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:23.100+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.101618+0000) 2019-10-13T12:20:23.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:23.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:23.105464+0000) 2019-10-13T12:20:25.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:25.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.762236+0000) 2019-10-13T12:20:25.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:25.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:25.762236+0000) 2019-10-13T12:20:26.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:26.199+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:26.200564+0000) 2019-10-13T12:20:28.064 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:28.061+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.063155+0000) 2019-10-13T12:20:28.065 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:28.063+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.063838+0000) 2019-10-13T12:20:28.065 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:28.062+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.063525+0000) 2019-10-13T12:20:28.065 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:28.062+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.063865+0000) 2019-10-13T12:20:28.065 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:28.062+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.064187+0000) 2019-10-13T12:20:28.066 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:28.062+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.064387+0000) 2019-10-13T12:20:28.066 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:28.063+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.064532+0000) 2019-10-13T12:20:28.066 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:28.064+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.065392+0000) 2019-10-13T12:20:28.073 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:28.071+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229440 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:28.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:28.664+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.665533+0000) 2019-10-13T12:20:28.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:28.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.996964+0000) 2019-10-13T12:20:31.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:31.616+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:31.617160+0000) 2019-10-13T12:20:33.075 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:33.072+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.073955+0000) 2019-10-13T12:20:33.075 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:33.072+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.074258+0000) 2019-10-13T12:20:33.076 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:33.074+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.074809+0000) 2019-10-13T12:20:33.076 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:33.073+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.074507+0000) 2019-10-13T12:20:33.076 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:33.073+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.074816+0000) 2019-10-13T12:20:33.076 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:33.073+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.074942+0000) 2019-10-13T12:20:33.077 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:33.073+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.075180+0000) 2019-10-13T12:20:33.078 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:33.076+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.077626+0000) 2019-10-13T12:20:33.083 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:33.080+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:33.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:33.101+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.101733+0000) 2019-10-13T12:20:33.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:33.105+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:33.105563+0000) 2019-10-13T12:20:35.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:35.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.762433+0000) 2019-10-13T12:20:35.763 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:35.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:35.762435+0000) 2019-10-13T12:20:36.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:36.202+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:36.202827+0000) 2019-10-13T12:20:38.084 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:38.083+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.083812+0000) 2019-10-13T12:20:38.085 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:38.083+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.084093+0000) 2019-10-13T12:20:38.086 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:38.083+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.084584+0000) 2019-10-13T12:20:38.086 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:38.084+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.084488+0000) 2019-10-13T12:20:38.086 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:38.084+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.084774+0000) 2019-10-13T12:20:38.086 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:38.084+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.084984+0000) 2019-10-13T12:20:38.086 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:38.084+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.085139+0000) 2019-10-13T12:20:38.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:38.085+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.086524+0000) 2019-10-13T12:20:38.094 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:38.092+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229501 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:38.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:38.665+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.665672+0000) 2019-10-13T12:20:38.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:38.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.997113+0000) 2019-10-13T12:20:41.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:41.617+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:41.617314+0000) 2019-10-13T12:20:42.890 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:20:42.890 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:42.895 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:20:42.895 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:43.096 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:43.094+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.094955+0000) 2019-10-13T12:20:43.096 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:43.094+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.095241+0000) 2019-10-13T12:20:43.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:43.094+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.095382+0000) 2019-10-13T12:20:43.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:43.095+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.095537+0000) 2019-10-13T12:20:43.097 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:43.095+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.095704+0000) 2019-10-13T12:20:43.098 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:43.095+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.096096+0000) 2019-10-13T12:20:43.098 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:43.095+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.095820+0000) 2019-10-13T12:20:43.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:43.097+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.098278+0000) 2019-10-13T12:20:43.102 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:43.101+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.101875+0000) 2019-10-13T12:20:43.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:43.102+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229529 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:43.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:43.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:43.105693+0000) 2019-10-13T12:20:45.763 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:45.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.762644+0000) 2019-10-13T12:20:45.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:45.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:45.762625+0000) 2019-10-13T12:20:46.206 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:46.204+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:46.205094+0000) 2019-10-13T12:20:48.106 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:48.104+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.105185+0000) 2019-10-13T12:20:48.106 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:48.104+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.105360+0000) 2019-10-13T12:20:48.106 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:48.105+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.105536+0000) 2019-10-13T12:20:48.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:48.105+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.106285+0000) 2019-10-13T12:20:48.107 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:48.105+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.105842+0000) 2019-10-13T12:20:48.108 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:48.105+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.106055+0000) 2019-10-13T12:20:48.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:48.105+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.106256+0000) 2019-10-13T12:20:48.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:48.106+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.107541+0000) 2019-10-13T12:20:48.115 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:48.113+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229560 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:48.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:48.665+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.665855+0000) 2019-10-13T12:20:48.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:48.997+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.997257+0000) 2019-10-13T12:20:51.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:51.617+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:51.617517+0000) 2019-10-13T12:20:53.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:53.101+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.102014+0000) 2019-10-13T12:20:53.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:53.104+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.105802+0000) 2019-10-13T12:20:53.116 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:53.115+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.115770+0000) 2019-10-13T12:20:53.117 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:53.115+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.116058+0000) 2019-10-13T12:20:53.117 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:53.115+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.116265+0000) 2019-10-13T12:20:53.117 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:53.116+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.116639+0000) 2019-10-13T12:20:53.118 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:53.116+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.117153+0000) 2019-10-13T12:20:53.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:53.116+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.117726+0000) 2019-10-13T12:20:53.118 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:53.117+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.117445+0000) 2019-10-13T12:20:53.119 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:53.116+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:53.118628+0000) 2019-10-13T12:20:53.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:53.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229588 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:55.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:55.761+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.762844+0000) 2019-10-13T12:20:55.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:55.761+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:55.762846+0000) 2019-10-13T12:20:56.208 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:56.206+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:56.207416+0000) 2019-10-13T12:20:58.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:58.126+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.127523+0000) 2019-10-13T12:20:58.128 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:20:58.126+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.127796+0000) 2019-10-13T12:20:58.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:20:58.126+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.127986+0000) 2019-10-13T12:20:58.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:20:58.126+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.128131+0000) 2019-10-13T12:20:58.130 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:20:58.126+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.128361+0000) 2019-10-13T12:20:58.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:58.127+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.128479+0000) 2019-10-13T12:20:58.130 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:20:58.127+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.128935+0000) 2019-10-13T12:20:58.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:20:58.128+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.130704+0000) 2019-10-13T12:20:58.138 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:20:58.135+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229621 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:20:58.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:20:58.665+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.666001+0000) 2019-10-13T12:20:58.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:20:58.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.997393+0000) 2019-10-13T12:21:01.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:01.616+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:01.617707+0000) 2019-10-13T12:21:03.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:03.101+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.102215+0000) 2019-10-13T12:21:03.106 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:03.105+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.106008+0000) 2019-10-13T12:21:03.139 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:03.137+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.138768+0000) 2019-10-13T12:21:03.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:03.137+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.138916+0000) 2019-10-13T12:21:03.140 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:03.137+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.139176+0000) 2019-10-13T12:21:03.141 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:03.139+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.140083+0000) 2019-10-13T12:21:03.141 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:03.138+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.139459+0000) 2019-10-13T12:21:03.141 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:03.138+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.139881+0000) 2019-10-13T12:21:03.142 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:03.138+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.139889+0000) 2019-10-13T12:21:03.142 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:03.141+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:03.141932+0000) 2019-10-13T12:21:03.147 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:03.145+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229649 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:05.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:05.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.763031+0000) 2019-10-13T12:21:05.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:05.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:05.763031+0000) 2019-10-13T12:21:06.210 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:06.209+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:06.209813+0000) 2019-10-13T12:21:08.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:08.147+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.148499+0000) 2019-10-13T12:21:08.150 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:08.147+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.148647+0000) 2019-10-13T12:21:08.150 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:08.147+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.148921+0000) 2019-10-13T12:21:08.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:08.147+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.149035+0000) 2019-10-13T12:21:08.150 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:08.147+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.149179+0000) 2019-10-13T12:21:08.151 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:08.148+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.148809+0000) 2019-10-13T12:21:08.151 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:08.149+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.150284+0000) 2019-10-13T12:21:08.151 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:08.147+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.149343+0000) 2019-10-13T12:21:08.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:08.156+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:08.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:08.665+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.666147+0000) 2019-10-13T12:21:08.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:08.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.997539+0000) 2019-10-13T12:21:11.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:11.617+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:11.617860+0000) 2019-10-13T12:21:13.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:13.101+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.102420+0000) 2019-10-13T12:21:13.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:13.105+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.106192+0000) 2019-10-13T12:21:13.129 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:21:13.129 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:13.133 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:21:13.133 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:13.160 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:13.158+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.159649+0000) 2019-10-13T12:21:13.161 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:13.158+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.159904+0000) 2019-10-13T12:21:13.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:13.158+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.160255+0000) 2019-10-13T12:21:13.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:13.158+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.160118+0000) 2019-10-13T12:21:13.162 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:13.159+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.160493+0000) 2019-10-13T12:21:13.162 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:13.159+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.160510+0000) 2019-10-13T12:21:13.162 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:13.159+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.160953+0000) 2019-10-13T12:21:13.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:13.161+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:13.161883+0000) 2019-10-13T12:21:13.169 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:13.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229709 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:15.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:15.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.763206+0000) 2019-10-13T12:21:15.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:15.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:15.763206+0000) 2019-10-13T12:21:16.213 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:16.211+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:16.212107+0000) 2019-10-13T12:21:18.171 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:18.168+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.170282+0000) 2019-10-13T12:21:18.171 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:18.169+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.170491+0000) 2019-10-13T12:21:18.172 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:18.170+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.170969+0000) 2019-10-13T12:21:18.172 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:18.169+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.171036+0000) 2019-10-13T12:21:18.172 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:18.169+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.171346+0000) 2019-10-13T12:21:18.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:18.170+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.171556+0000) 2019-10-13T12:21:18.173 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:18.170+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.171647+0000) 2019-10-13T12:21:18.174 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:18.172+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.173358+0000) 2019-10-13T12:21:18.181 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:18.178+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229741 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:18.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:18.665+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.666346+0000) 2019-10-13T12:21:18.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:18.997+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.997702+0000) 2019-10-13T12:21:21.618 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:21.617+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:21.618008+0000) 2019-10-13T12:21:23.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:23.102+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.102604+0000) 2019-10-13T12:21:23.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:23.106+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.106372+0000) 2019-10-13T12:21:23.182 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:23.181+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.181880+0000) 2019-10-13T12:21:23.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:23.181+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.182022+0000) 2019-10-13T12:21:23.183 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:23.181+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.182170+0000) 2019-10-13T12:21:23.184 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:23.181+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.182301+0000) 2019-10-13T12:21:23.184 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:23.182+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.182901+0000) 2019-10-13T12:21:23.184 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:23.182+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.183126+0000) 2019-10-13T12:21:23.185 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:23.182+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.183189+0000) 2019-10-13T12:21:23.186 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:23.184+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:23.185636+0000) 2019-10-13T12:21:23.192 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:23.190+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229770 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:25.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:25.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.763381+0000) 2019-10-13T12:21:25.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:25.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:25.763381+0000) 2019-10-13T12:21:26.215 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:26.214+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:26.214599+0000) 2019-10-13T12:21:28.193 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:28.192+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.193148+0000) 2019-10-13T12:21:28.194 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:28.192+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.193177+0000) 2019-10-13T12:21:28.195 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:28.192+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.193349+0000) 2019-10-13T12:21:28.195 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:28.193+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.193552+0000) 2019-10-13T12:21:28.195 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:28.193+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.193758+0000) 2019-10-13T12:21:28.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:28.193+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.194061+0000) 2019-10-13T12:21:28.196 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:28.193+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.194316+0000) 2019-10-13T12:21:28.196 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:28.195+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.196029+0000) 2019-10-13T12:21:28.203 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:28.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229800 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:28.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:28.666+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.666493+0000) 2019-10-13T12:21:28.998 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:28.997+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.997861+0000) 2019-10-13T12:21:31.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:31.617+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:31.618162+0000) 2019-10-13T12:21:33.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:33.102+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.102775+0000) 2019-10-13T12:21:33.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:33.105+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.106549+0000) 2019-10-13T12:21:33.205 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:33.203+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.204612+0000) 2019-10-13T12:21:33.206 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:33.204+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.204634+0000) 2019-10-13T12:21:33.206 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:33.204+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.204847+0000) 2019-10-13T12:21:33.206 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:33.204+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.204865+0000) 2019-10-13T12:21:33.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:33.204+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.205030+0000) 2019-10-13T12:21:33.207 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:33.204+0000 7f74c2a2f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.205208+0000) 2019-10-13T12:21:33.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:33.205+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.205571+0000) 2019-10-13T12:21:33.207 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:33.205+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:33.206513+0000) 2019-10-13T12:21:33.213 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:33.212+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229829 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:35.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:35.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.763536+0000) 2019-10-13T12:21:35.764 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:35.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:35.763535+0000) 2019-10-13T12:21:36.217 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:36.216+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:36.216862+0000) 2019-10-13T12:21:38.215 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:38.213+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.214389+0000) 2019-10-13T12:21:38.216 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:38.214+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.214559+0000) 2019-10-13T12:21:38.216 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:38.214+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.214726+0000) 2019-10-13T12:21:38.216 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:38.214+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.214855+0000) 2019-10-13T12:21:38.217 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:38.214+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.215366+0000) 2019-10-13T12:21:38.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:38.214+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.215124+0000) 2019-10-13T12:21:38.217 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:38.215+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.215481+0000) 2019-10-13T12:21:38.218 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:38.216+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.217679+0000) 2019-10-13T12:21:38.224 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:38.223+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229861 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:38.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:38.666+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.666679+0000) 2019-10-13T12:21:38.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:38.997+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.998023+0000) 2019-10-13T12:21:41.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:41.616+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:41.618351+0000) 2019-10-13T12:21:43.103 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:43.101+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.102933+0000) 2019-10-13T12:21:43.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:43.105+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.106724+0000) 2019-10-13T12:21:43.226 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:43.224+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.225498+0000) 2019-10-13T12:21:43.227 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:43.224+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.225762+0000) 2019-10-13T12:21:43.227 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:43.224+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.226221+0000) 2019-10-13T12:21:43.228 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:43.225+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.226470+0000) 2019-10-13T12:21:43.228 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:43.225+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.226567+0000) 2019-10-13T12:21:43.228 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:43.225+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.226778+0000) 2019-10-13T12:21:43.228 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:43.225+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.227068+0000) 2019-10-13T12:21:43.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:43.226+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:43.228229+0000) 2019-10-13T12:21:43.236 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:43.234+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:43.317 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:21:43.318 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:43.322 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:21:43.322 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:45.764 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:45.762+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.763701+0000) 2019-10-13T12:21:45.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:45.762+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:45.763685+0000) 2019-10-13T12:21:46.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:46.218+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:46.219216+0000) 2019-10-13T12:21:48.238 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:48.235+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.236939+0000) 2019-10-13T12:21:48.238 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:48.235+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.237153+0000) 2019-10-13T12:21:48.239 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:48.235+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.237387+0000) 2019-10-13T12:21:48.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:48.236+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.237623+0000) 2019-10-13T12:21:48.239 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:48.236+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.237727+0000) 2019-10-13T12:21:48.240 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:48.236+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.238022+0000) 2019-10-13T12:21:48.240 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:48.236+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.238375+0000) 2019-10-13T12:21:48.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:48.239+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.240959+0000) 2019-10-13T12:21:48.247 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:48.245+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229920 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:48.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:48.665+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.666830+0000) 2019-10-13T12:21:48.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:48.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.998195+0000) 2019-10-13T12:21:51.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:51.617+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:51.618560+0000) 2019-10-13T12:21:53.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:53.102+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.103106+0000) 2019-10-13T12:21:53.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:53.106+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.106951+0000) 2019-10-13T12:21:53.250 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:53.247+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.248841+0000) 2019-10-13T12:21:53.250 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:53.247+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.249176+0000) 2019-10-13T12:21:53.251 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:53.249+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.249956+0000) 2019-10-13T12:21:53.251 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:53.248+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.249523+0000) 2019-10-13T12:21:53.251 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:53.248+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.249761+0000) 2019-10-13T12:21:53.251 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:53.248+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.250095+0000) 2019-10-13T12:21:53.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:53.248+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.249585+0000) 2019-10-13T12:21:53.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:53.250+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:53.250927+0000) 2019-10-13T12:21:53.259 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:53.256+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229949 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:55.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:55.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.763850+0000) 2019-10-13T12:21:55.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:55.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:55.763888+0000) 2019-10-13T12:21:56.222 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:56.220+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:56.221391+0000) 2019-10-13T12:21:58.260 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:58.258+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.259636+0000) 2019-10-13T12:21:58.261 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:21:58.258+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.259849+0000) 2019-10-13T12:21:58.261 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:21:58.258+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.260109+0000) 2019-10-13T12:21:58.262 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:21:58.259+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.260449+0000) 2019-10-13T12:21:58.262 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:21:58.258+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.260373+0000) 2019-10-13T12:21:58.262 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:58.259+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.260925+0000) 2019-10-13T12:21:58.262 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:21:58.259+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.260826+0000) 2019-10-13T12:21:58.264 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:21:58.261+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.262631+0000) 2019-10-13T12:21:58.268 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:21:58.266+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 229981 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:21:58.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:21:58.666+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.667001+0000) 2019-10-13T12:21:58.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:21:58.996+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.998378+0000) 2019-10-13T12:22:01.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:01.617+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:01.618738+0000) 2019-10-13T12:22:03.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:03.102+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.103260+0000) 2019-10-13T12:22:03.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:03.106+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.107131+0000) 2019-10-13T12:22:03.270 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:03.268+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.269506+0000) 2019-10-13T12:22:03.271 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:03.268+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.269615+0000) 2019-10-13T12:22:03.271 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:03.268+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.270267+0000) 2019-10-13T12:22:03.271 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:03.269+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.270504+0000) 2019-10-13T12:22:03.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:03.269+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.270658+0000) 2019-10-13T12:22:03.272 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:03.269+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.271020+0000) 2019-10-13T12:22:03.272 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:03.269+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.271195+0000) 2019-10-13T12:22:03.274 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:03.272+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:03.273271+0000) 2019-10-13T12:22:03.278 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:03.276+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230010 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:05.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:05.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.764008+0000) 2019-10-13T12:22:05.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:05.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:05.764044+0000) 2019-10-13T12:22:06.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:06.223+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:06.223794+0000) 2019-10-13T12:22:08.280 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:08.278+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.279694+0000) 2019-10-13T12:22:08.281 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:08.279+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.280070+0000) 2019-10-13T12:22:08.281 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:08.279+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.280293+0000) 2019-10-13T12:22:08.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:08.280+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.280489+0000) 2019-10-13T12:22:08.282 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:08.280+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.280814+0000) 2019-10-13T12:22:08.282 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:08.280+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.280678+0000) 2019-10-13T12:22:08.282 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:08.280+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.280977+0000) 2019-10-13T12:22:08.283 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:08.281+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.282054+0000) 2019-10-13T12:22:08.290 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:08.289+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230040 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:08.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:08.666+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.667199+0000) 2019-10-13T12:22:08.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:08.998+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.998569+0000) 2019-10-13T12:22:11.619 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:11.618+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:11.618906+0000) 2019-10-13T12:22:13.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:13.103+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.103436+0000) 2019-10-13T12:22:13.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:13.106+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.107294+0000) 2019-10-13T12:22:13.292 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:13.290+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.291624+0000) 2019-10-13T12:22:13.293 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:13.291+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.291990+0000) 2019-10-13T12:22:13.293 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:13.291+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.292131+0000) 2019-10-13T12:22:13.293 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:13.292+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.292516+0000) 2019-10-13T12:22:13.294 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:13.292+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.292619+0000) 2019-10-13T12:22:13.294 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:13.292+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.292888+0000) 2019-10-13T12:22:13.294 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:13.292+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.293032+0000) 2019-10-13T12:22:13.295 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:13.293+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:13.293916+0000) 2019-10-13T12:22:13.303 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:13.300+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230068 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:13.555 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:22:13.555 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:13.560 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:22:13.560 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:15.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:15.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.764206+0000) 2019-10-13T12:22:15.765 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:15.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:15.764162+0000) 2019-10-13T12:22:16.227 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:16.225+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:16.226278+0000) 2019-10-13T12:22:18.304 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:18.302+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.302933+0000) 2019-10-13T12:22:18.304 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:18.302+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.302994+0000) 2019-10-13T12:22:18.304 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:18.302+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.303224+0000) 2019-10-13T12:22:18.305 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:18.302+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.303262+0000) 2019-10-13T12:22:18.305 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:18.302+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.303436+0000) 2019-10-13T12:22:18.306 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:18.303+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.303711+0000) 2019-10-13T12:22:18.306 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:18.303+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.303965+0000) 2019-10-13T12:22:18.307 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:18.305+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.306695+0000) 2019-10-13T12:22:18.313 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:18.311+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230100 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:18.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:18.667+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.667412+0000) 2019-10-13T12:22:18.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:18.998+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.998735+0000) 2019-10-13T12:22:21.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:21.618+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:21.619100+0000) 2019-10-13T12:22:23.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:23.103+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.103672+0000) 2019-10-13T12:22:23.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:23.106+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.107475+0000) 2019-10-13T12:22:23.315 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:23.313+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.314388+0000) 2019-10-13T12:22:23.316 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:23.314+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.314634+0000) 2019-10-13T12:22:23.316 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:23.314+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.314882+0000) 2019-10-13T12:22:23.316 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:23.313+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.314238+0000) 2019-10-13T12:22:23.317 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:23.314+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.315136+0000) 2019-10-13T12:22:23.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:23.315+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.315633+0000) 2019-10-13T12:22:23.317 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:23.315+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.315956+0000) 2019-10-13T12:22:23.318 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:23.316+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:23.316866+0000) 2019-10-13T12:22:23.324 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:23.323+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230129 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:25.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:25.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:25.764362+0000) 2019-10-13T12:22:25.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:25.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:25.764341+0000) 2019-10-13T12:22:26.229 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:26.228+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:26.228639+0000) 2019-10-13T12:22:28.326 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:28.324+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.325582+0000) 2019-10-13T12:22:28.327 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:28.324+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.325754+0000) 2019-10-13T12:22:28.327 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:28.324+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.325988+0000) 2019-10-13T12:22:28.327 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:28.324+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.326129+0000) 2019-10-13T12:22:28.327 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:28.324+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.326391+0000) 2019-10-13T12:22:28.328 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:28.325+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.326915+0000) 2019-10-13T12:22:28.328 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:28.325+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.327104+0000) 2019-10-13T12:22:28.329 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:28.327+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.328713+0000) 2019-10-13T12:22:28.336 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:28.333+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230158 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:28.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:28.666+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.667575+0000) 2019-10-13T12:22:28.999 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:28.997+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.998886+0000) 2019-10-13T12:22:31.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:31.618+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.619296+0000) 2019-10-13T12:22:33.104 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:33.102+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.103830+0000) 2019-10-13T12:22:33.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:33.106+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.107673+0000) 2019-10-13T12:22:33.337 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:33.334+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.336698+0000) 2019-10-13T12:22:33.338 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:33.335+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.337052+0000) 2019-10-13T12:22:33.338 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:33.335+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.337200+0000) 2019-10-13T12:22:33.339 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:33.336+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.337408+0000) 2019-10-13T12:22:33.339 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:33.336+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.337507+0000) 2019-10-13T12:22:33.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:33.336+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.337650+0000) 2019-10-13T12:22:33.339 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:33.336+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.337793+0000) 2019-10-13T12:22:33.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:33.337+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:33.339119+0000) 2019-10-13T12:22:33.345 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:33.342+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230187 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:35.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:35.763+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:35.764602+0000) 2019-10-13T12:22:35.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:35.763+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:35.764601+0000) 2019-10-13T12:22:36.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:36.229+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:36.230855+0000) 2019-10-13T12:22:38.347 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:38.344+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.345667+0000) 2019-10-13T12:22:38.347 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:38.344+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.345848+0000) 2019-10-13T12:22:38.347 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:38.344+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.346175+0000) 2019-10-13T12:22:38.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:38.345+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.346392+0000) 2019-10-13T12:22:38.348 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:38.345+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.347094+0000) 2019-10-13T12:22:38.348 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:38.345+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.346700+0000) 2019-10-13T12:22:38.348 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:38.345+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.346612+0000) 2019-10-13T12:22:38.349 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:38.347+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.348777+0000) 2019-10-13T12:22:38.355 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:38.353+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230218 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:38.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:38.667+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.667743+0000) 2019-10-13T12:22:39.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:38.998+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.999038+0000) 2019-10-13T12:22:41.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:41.618+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.619487+0000) 2019-10-13T12:22:43.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:43.103+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.103984+0000) 2019-10-13T12:22:43.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:43.107+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.107911+0000) 2019-10-13T12:22:43.357 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:43.354+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.356154+0000) 2019-10-13T12:22:43.357 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:43.354+0000 7fdde6810700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.356243+0000) 2019-10-13T12:22:43.358 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:43.355+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.356433+0000) 2019-10-13T12:22:43.358 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:43.356+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.356785+0000) 2019-10-13T12:22:43.359 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:43.355+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.356632+0000) 2019-10-13T12:22:43.359 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:43.355+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.357081+0000) 2019-10-13T12:22:43.359 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:43.356+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.357491+0000) 2019-10-13T12:22:43.359 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:43.358+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:43.359136+0000) 2019-10-13T12:22:43.366 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:43.364+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230247 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:43.671 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:22:43.671 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:43.675 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:22:43.675 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:45.765 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:45.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:45.764811+0000) 2019-10-13T12:22:45.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:45.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:45.764797+0000) 2019-10-13T12:22:46.234 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:46.232+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:46.233295+0000) 2019-10-13T12:22:48.368 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:48.366+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.367569+0000) 2019-10-13T12:22:48.369 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:48.366+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.368095+0000) 2019-10-13T12:22:48.369 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:48.366+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.368301+0000) 2019-10-13T12:22:48.369 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:48.367+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.368469+0000) 2019-10-13T12:22:48.370 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:48.367+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.368802+0000) 2019-10-13T12:22:48.370 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:48.367+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.369077+0000) 2019-10-13T12:22:48.370 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:48.367+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.369261+0000) 2019-10-13T12:22:48.371 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:48.370+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.370879+0000) 2019-10-13T12:22:48.378 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:48.375+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230276 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:48.668 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:48.667+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.667914+0000) 2019-10-13T12:22:49.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:48.998+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.999229+0000) 2019-10-13T12:22:51.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:51.619+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.619683+0000) 2019-10-13T12:22:53.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:53.103+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.104192+0000) 2019-10-13T12:22:53.108 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:53.107+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.108086+0000) 2019-10-13T12:22:53.379 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:53.378+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.378903+0000) 2019-10-13T12:22:53.381 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:53.378+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.379192+0000) 2019-10-13T12:22:53.381 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:53.379+0000 7f5e9c114700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.379408+0000) 2019-10-13T12:22:53.381 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:53.379+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.379589+0000) 2019-10-13T12:22:53.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:53.379+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.379816+0000) 2019-10-13T12:22:53.381 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:53.379+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.379909+0000) 2019-10-13T12:22:53.382 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:53.379+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.380085+0000) 2019-10-13T12:22:53.382 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:53.380+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:53.381198+0000) 2019-10-13T12:22:53.389 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:53.388+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230307 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:55.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:55.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:55.765029+0000) 2019-10-13T12:22:55.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:55.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:55.765029+0000) 2019-10-13T12:22:56.236 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:56.235+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:56.235579+0000) 2019-10-13T12:22:58.391 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:22:58.389+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.390452+0000) 2019-10-13T12:22:58.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:22:58.390+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.390829+0000) 2019-10-13T12:22:58.392 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:58.390+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.390872+0000) 2019-10-13T12:22:58.392 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:22:58.390+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.391055+0000) 2019-10-13T12:22:58.393 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:22:58.390+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.391202+0000) 2019-10-13T12:22:58.393 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:22:58.391+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.391442+0000) 2019-10-13T12:22:58.393 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:58.391+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.391717+0000) 2019-10-13T12:22:58.394 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:22:58.393+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.394228+0000) 2019-10-13T12:22:58.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:22:58.399+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230338 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:22:58.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:22:58.667+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.668117+0000) 2019-10-13T12:22:59.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:22:58.999+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.999368+0000) 2019-10-13T12:23:01.620 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:01.619+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.619857+0000) 2019-10-13T12:23:03.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:03.104+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.104386+0000) 2019-10-13T12:23:03.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:03.107+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.108259+0000) 2019-10-13T12:23:03.402 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:03.400+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.401343+0000) 2019-10-13T12:23:03.402 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:03.401+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.401973+0000) 2019-10-13T12:23:03.403 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:03.402+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.402586+0000) 2019-10-13T12:23:03.403 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:03.402+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.402584+0000) 2019-10-13T12:23:03.404 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:03.401+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.402386+0000) 2019-10-13T12:23:03.404 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:03.402+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.402761+0000) 2019-10-13T12:23:03.405 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:03.402+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.402889+0000) 2019-10-13T12:23:03.405 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:03.404+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:03.405211+0000) 2019-10-13T12:23:03.411 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:03.410+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230369 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:05.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:05.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:05.765240+0000) 2019-10-13T12:23:05.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:05.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:05.765242+0000) 2019-10-13T12:23:06.238 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:06.237+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:06.237720+0000) 2019-10-13T12:23:08.414 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:08.412+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.412728+0000) 2019-10-13T12:23:08.414 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:08.412+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.412992+0000) 2019-10-13T12:23:08.415 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:08.412+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.413244+0000) 2019-10-13T12:23:08.416 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:08.412+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.413180+0000) 2019-10-13T12:23:08.416 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:08.413+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.413754+0000) 2019-10-13T12:23:08.416 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:08.413+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.413917+0000) 2019-10-13T12:23:08.416 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:08.413+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.414340+0000) 2019-10-13T12:23:08.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:08.414+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.415543+0000) 2019-10-13T12:23:08.423 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:08.421+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230398 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:08.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:08.667+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.668286+0000) 2019-10-13T12:23:09.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:08.999+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.999512+0000) 2019-10-13T12:23:11.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:11.619+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.620038+0000) 2019-10-13T12:23:13.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:13.103+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.104536+0000) 2019-10-13T12:23:13.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:13.108+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.108451+0000) 2019-10-13T12:23:13.425 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:13.422+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.424022+0000) 2019-10-13T12:23:13.425 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:13.422+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.424188+0000) 2019-10-13T12:23:13.425 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:13.424+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.424804+0000) 2019-10-13T12:23:13.426 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:13.423+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.424548+0000) 2019-10-13T12:23:13.426 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:13.424+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.425687+0000) 2019-10-13T12:23:13.427 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:13.424+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.425911+0000) 2019-10-13T12:23:13.427 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:13.424+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.425858+0000) 2019-10-13T12:23:13.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:13.427+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:13.428169+0000) 2019-10-13T12:23:13.434 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:13.432+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230429 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:13.793 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:23:13.794 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:13.797 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:23:13.797 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:15.766 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:15.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:15.765439+0000) 2019-10-13T12:23:15.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:15.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:15.765438+0000) 2019-10-13T12:23:16.241 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:16.240+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:16.240135+0000) 2019-10-13T12:23:18.436 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:18.433+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.435205+0000) 2019-10-13T12:23:18.436 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:18.434+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.435452+0000) 2019-10-13T12:23:18.437 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:18.435+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.436319+0000) 2019-10-13T12:23:18.437 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:18.434+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.435849+0000) 2019-10-13T12:23:18.438 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:18.434+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.435990+0000) 2019-10-13T12:23:18.438 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:18.434+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.436087+0000) 2019-10-13T12:23:18.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:18.435+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.436573+0000) 2019-10-13T12:23:18.438 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:18.437+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.437933+0000) 2019-10-13T12:23:18.446 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:18.443+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230460 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:18.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:18.667+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.668481+0000) 2019-10-13T12:23:19.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:18.998+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.999675+0000) 2019-10-13T12:23:21.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:21.619+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.620202+0000) 2019-10-13T12:23:23.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:23.103+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.104700+0000) 2019-10-13T12:23:23.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:23.107+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.108631+0000) 2019-10-13T12:23:23.448 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:23.445+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.446857+0000) 2019-10-13T12:23:23.448 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:23.445+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.447045+0000) 2019-10-13T12:23:23.448 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:23.445+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.447103+0000) 2019-10-13T12:23:23.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:23.445+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.447269+0000) 2019-10-13T12:23:23.449 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:23.446+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.447409+0000) 2019-10-13T12:23:23.449 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:23.446+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.447760+0000) 2019-10-13T12:23:23.450 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:23.446+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.447776+0000) 2019-10-13T12:23:23.450 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:23.448+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:23.449783+0000) 2019-10-13T12:23:23.456 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:23.454+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230491 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:25.766 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:25.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:25.765633+0000) 2019-10-13T12:23:25.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:25.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:25.765633+0000) 2019-10-13T12:23:26.243 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:26.241+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:26.242480+0000) 2019-10-13T12:23:28.458 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:28.456+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.457465+0000) 2019-10-13T12:23:28.459 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:28.456+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.457853+0000) 2019-10-13T12:23:28.459 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:28.456+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.457895+0000) 2019-10-13T12:23:28.459 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:28.456+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.458257+0000) 2019-10-13T12:23:28.460 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:28.456+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.458326+0000) 2019-10-13T12:23:28.460 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:28.457+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.458886+0000) 2019-10-13T12:23:28.460 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:28.457+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.458965+0000) 2019-10-13T12:23:28.461 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:28.459+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.460951+0000) 2019-10-13T12:23:28.468 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:28.465+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230520 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:28.669 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:28.668+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.668681+0000) 2019-10-13T12:23:29.000 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:28.999+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.999840+0000) 2019-10-13T12:23:31.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:31.619+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.620383+0000) 2019-10-13T12:23:33.105 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:33.104+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.104874+0000) 2019-10-13T12:23:33.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:33.108+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.108778+0000) 2019-10-13T12:23:33.469 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:33.467+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.468506+0000) 2019-10-13T12:23:33.470 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:33.468+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.469215+0000) 2019-10-13T12:23:33.470 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:33.467+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.468842+0000) 2019-10-13T12:23:33.470 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:33.467+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.469005+0000) 2019-10-13T12:23:33.471 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:33.468+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.469523+0000) 2019-10-13T12:23:33.471 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:33.468+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.469760+0000) 2019-10-13T12:23:33.471 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:33.468+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.469883+0000) 2019-10-13T12:23:33.472 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:33.471+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:33.472226+0000) 2019-10-13T12:23:33.479 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:33.476+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230553 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:35.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:35.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:35.765793+0000) 2019-10-13T12:23:35.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:35.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:35.765828+0000) 2019-10-13T12:23:36.245 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:36.244+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:36.244896+0000) 2019-10-13T12:23:38.481 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:38.479+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.480175+0000) 2019-10-13T12:23:38.482 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:38.480+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.480465+0000) 2019-10-13T12:23:38.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:38.480+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.480710+0000) 2019-10-13T12:23:38.482 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:38.480+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.480808+0000) 2019-10-13T12:23:38.482 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:38.480+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.480928+0000) 2019-10-13T12:23:38.483 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:38.480+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.481197+0000) 2019-10-13T12:23:38.483 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:38.480+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.481276+0000) 2019-10-13T12:23:38.483 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:38.481+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.482642+0000) 2019-10-13T12:23:38.490 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:38.488+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230582 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:38.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:38.668+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.668811+0000) 2019-10-13T12:23:39.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:38.999+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:39.000002+0000) 2019-10-13T12:23:41.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:41.620+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.620556+0000) 2019-10-13T12:23:43.106 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:43.104+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.105079+0000) 2019-10-13T12:23:43.109 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:43.108+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.108965+0000) 2019-10-13T12:23:43.491 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:43.490+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.490432+0000) 2019-10-13T12:23:43.492 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:43.490+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.490532+0000) 2019-10-13T12:23:43.492 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:43.490+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.490810+0000) 2019-10-13T12:23:43.492 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:43.490+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.490820+0000) 2019-10-13T12:23:43.492 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:43.490+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.491134+0000) 2019-10-13T12:23:43.493 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:43.490+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.491475+0000) 2019-10-13T12:23:43.493 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:43.490+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.491305+0000) 2019-10-13T12:23:43.494 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:43.492+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:43.493653+0000) 2019-10-13T12:23:43.498 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:43.497+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230613 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:43.882 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:23:43.882 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:43.887 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:23:43.887 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:45.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:45.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:45.765974+0000) 2019-10-13T12:23:45.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:45.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:45.765979+0000) 2019-10-13T12:23:46.248 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:46.246+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:46.247152+0000) 2019-10-13T12:23:48.500 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:48.498+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.499012+0000) 2019-10-13T12:23:48.500 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:48.498+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.499270+0000) 2019-10-13T12:23:48.500 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:48.499+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.499416+0000) 2019-10-13T12:23:48.501 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:48.499+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.499592+0000) 2019-10-13T12:23:48.501 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:48.499+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.499960+0000) 2019-10-13T12:23:48.502 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:48.499+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.499740+0000) 2019-10-13T12:23:48.502 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:48.499+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.499873+0000) 2019-10-13T12:23:48.502 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:48.500+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.501353+0000) 2019-10-13T12:23:48.507 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:48.506+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230642 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:48.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:48.668+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.668984+0000) 2019-10-13T12:23:49.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:49.000+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:49.000187+0000) 2019-10-13T12:23:51.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:51.620+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.620719+0000) 2019-10-13T12:23:53.106 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:53.105+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.105244+0000) 2019-10-13T12:23:53.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:53.108+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.109148+0000) 2019-10-13T12:23:53.509 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:53.507+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.508265+0000) 2019-10-13T12:23:53.509 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:53.507+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.508111+0000) 2019-10-13T12:23:53.510 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:53.507+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.508455+0000) 2019-10-13T12:23:53.511 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:53.508+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.509363+0000) 2019-10-13T12:23:53.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:53.509+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.509535+0000) 2019-10-13T12:23:53.511 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:53.509+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.509748+0000) 2019-10-13T12:23:53.511 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:53.509+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.510009+0000) 2019-10-13T12:23:53.512 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:53.510+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:53.511373+0000) 2019-10-13T12:23:53.518 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:53.517+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230675 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:55.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:55.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:55.766154+0000) 2019-10-13T12:23:55.767 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:55.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:55.766154+0000) 2019-10-13T12:23:56.250 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:56.249+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:56.249505+0000) 2019-10-13T12:23:58.520 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:23:58.518+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.519553+0000) 2019-10-13T12:23:58.522 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:23:58.518+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.519664+0000) 2019-10-13T12:23:58.522 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:23:58.518+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.519839+0000) 2019-10-13T12:23:58.522 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:58.518+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.519942+0000) 2019-10-13T12:23:58.522 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:23:58.518+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.520143+0000) 2019-10-13T12:23:58.522 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:58.518+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.520345+0000) 2019-10-13T12:23:58.523 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:23:58.519+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.520783+0000) 2019-10-13T12:23:58.523 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:23:58.520+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.521522+0000) 2019-10-13T12:23:58.530 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:23:58.527+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230704 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:23:58.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:23:58.667+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.669152+0000) 2019-10-13T12:23:59.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:23:58.999+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:59.000350+0000) 2019-10-13T12:24:01.621 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:01.619+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.620913+0000) 2019-10-13T12:24:03.106 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:03.103+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.105351+0000) 2019-10-13T12:24:03.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:03.109+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.109304+0000) 2019-10-13T12:24:03.532 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:03.529+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.530598+0000) 2019-10-13T12:24:03.532 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:03.529+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.530789+0000) 2019-10-13T12:24:03.532 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:03.529+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.531126+0000) 2019-10-13T12:24:03.533 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:03.529+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.531336+0000) 2019-10-13T12:24:03.533 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:03.530+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.531487+0000) 2019-10-13T12:24:03.533 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:03.530+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.530991+0000) 2019-10-13T12:24:03.534 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:03.530+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.531748+0000) 2019-10-13T12:24:03.534 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:03.532+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:03.533358+0000) 2019-10-13T12:24:03.539 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:03.537+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230735 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:05.767 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:05.764+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:05.766353+0000) 2019-10-13T12:24:05.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:05.764+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:05.766354+0000) 2019-10-13T12:24:06.252 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:06.251+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:06.251823+0000) 2019-10-13T12:24:08.541 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:08.538+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.540323+0000) 2019-10-13T12:24:08.542 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:08.539+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.540489+0000) 2019-10-13T12:24:08.542 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:08.539+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.540701+0000) 2019-10-13T12:24:08.543 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:08.540+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.541169+0000) 2019-10-13T12:24:08.543 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:08.539+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.541085+0000) 2019-10-13T12:24:08.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:08.539+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.541373+0000) 2019-10-13T12:24:08.543 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:08.540+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.541582+0000) 2019-10-13T12:24:08.544 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:08.542+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.542959+0000) 2019-10-13T12:24:08.550 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:08.548+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:08.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:08.668+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.669297+0000) 2019-10-13T12:24:09.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:08.999+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:09.000515+0000) 2019-10-13T12:24:11.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:11.620+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.621116+0000) 2019-10-13T12:24:13.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:13.104+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.105519+0000) 2019-10-13T12:24:13.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:13.108+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.109484+0000) 2019-10-13T12:24:13.567 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:13.550+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.551454+0000) 2019-10-13T12:24:13.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:13.550+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.551805+0000) 2019-10-13T12:24:13.568 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:13.550+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.551878+0000) 2019-10-13T12:24:13.569 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:13.550+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.552272+0000) 2019-10-13T12:24:13.569 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:13.550+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.552086+0000) 2019-10-13T12:24:13.569 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:13.550+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.552301+0000) 2019-10-13T12:24:13.569 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:13.551+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.552438+0000) 2019-10-13T12:24:13.570 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:13.553+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:13.555562+0000) 2019-10-13T12:24:13.576 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:13.559+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:14.005 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:24:14.005 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:14.010 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:24:14.010 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:15.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:15.765+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:15.766597+0000) 2019-10-13T12:24:15.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:15.765+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:15.766586+0000) 2019-10-13T12:24:16.255 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:16.253+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:16.254386+0000) 2019-10-13T12:24:18.564 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:18.561+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.563145+0000) 2019-10-13T12:24:18.565 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:18.561+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.562991+0000) 2019-10-13T12:24:18.565 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:18.561+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.563335+0000) 2019-10-13T12:24:18.565 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:18.562+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.563764+0000) 2019-10-13T12:24:18.565 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:18.562+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.563947+0000) 2019-10-13T12:24:18.566 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:18.562+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.564086+0000) 2019-10-13T12:24:18.566 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:18.562+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.564272+0000) 2019-10-13T12:24:18.566 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:18.563+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.565613+0000) 2019-10-13T12:24:18.573 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:18.571+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:18.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:18.668+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.669461+0000) 2019-10-13T12:24:19.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:19.000+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:19.000688+0000) 2019-10-13T12:24:21.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:21.620+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.621306+0000) 2019-10-13T12:24:23.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:23.105+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.105706+0000) 2019-10-13T12:24:23.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:23.108+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.109627+0000) 2019-10-13T12:24:23.576 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:23.574+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.574674+0000) 2019-10-13T12:24:23.576 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:23.574+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.574832+0000) 2019-10-13T12:24:23.576 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:23.574+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.575266+0000) 2019-10-13T12:24:23.577 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:23.575+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.575858+0000) 2019-10-13T12:24:23.577 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:23.575+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.575493+0000) 2019-10-13T12:24:23.578 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:23.575+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.575938+0000) 2019-10-13T12:24:23.578 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:23.575+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.576090+0000) 2019-10-13T12:24:23.578 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:23.576+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:23.577572+0000) 2019-10-13T12:24:23.585 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:23.584+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230857 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:25.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:25.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:25.766773+0000) 2019-10-13T12:24:25.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:25.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:25.766797+0000) 2019-10-13T12:24:26.257 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:26.255+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:26.256694+0000) 2019-10-13T12:24:28.587 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:28.585+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.586162+0000) 2019-10-13T12:24:28.588 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:28.585+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.586586+0000) 2019-10-13T12:24:28.588 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:28.586+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.586696+0000) 2019-10-13T12:24:28.589 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:28.586+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.587185+0000) 2019-10-13T12:24:28.589 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:28.587+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.587410+0000) 2019-10-13T12:24:28.589 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:28.587+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.587583+0000) 2019-10-13T12:24:28.589 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:28.587+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.587931+0000) 2019-10-13T12:24:28.591 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:28.589+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.590234+0000) 2019-10-13T12:24:28.597 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:28.595+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230886 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:28.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:28.669+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.669621+0000) 2019-10-13T12:24:29.001 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:29.000+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:29.000857+0000) 2019-10-13T12:24:31.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:31.621+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.621499+0000) 2019-10-13T12:24:33.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:33.105+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.105880+0000) 2019-10-13T12:24:33.110 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:33.109+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.109816+0000) 2019-10-13T12:24:33.599 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:33.597+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.598008+0000) 2019-10-13T12:24:33.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:33.597+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.598164+0000) 2019-10-13T12:24:33.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:33.597+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.598329+0000) 2019-10-13T12:24:33.600 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:33.598+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.599225+0000) 2019-10-13T12:24:33.600 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:33.598+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.598931+0000) 2019-10-13T12:24:33.601 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:33.598+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.599347+0000) 2019-10-13T12:24:33.601 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:33.598+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.598809+0000) 2019-10-13T12:24:33.602 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:33.600+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:33.601494+0000) 2019-10-13T12:24:33.608 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:33.607+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230919 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:35.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:35.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:35.766980+0000) 2019-10-13T12:24:35.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:35.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:35.766968+0000) 2019-10-13T12:24:36.261 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:36.258+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:36.259127+0000) 2019-10-13T12:24:38.610 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:38.608+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.609321+0000) 2019-10-13T12:24:38.611 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:38.609+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.609517+0000) 2019-10-13T12:24:38.611 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:38.609+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.609921+0000) 2019-10-13T12:24:38.611 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:38.609+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.610072+0000) 2019-10-13T12:24:38.612 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:38.609+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.610211+0000) 2019-10-13T12:24:38.612 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:38.610+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.610417+0000) 2019-10-13T12:24:38.612 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:38.610+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.611703+0000) 2019-10-13T12:24:38.613 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:38.610+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.610867+0000) 2019-10-13T12:24:38.620 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:38.618+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230948 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:38.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:38.669+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.669739+0000) 2019-10-13T12:24:39.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:39.000+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:39.001037+0000) 2019-10-13T12:24:41.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:41.620+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.621689+0000) 2019-10-13T12:24:43.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:43.105+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.106102+0000) 2019-10-13T12:24:43.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:43.109+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.110018+0000) 2019-10-13T12:24:43.622 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:43.619+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.620798+0000) 2019-10-13T12:24:43.622 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:43.619+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.621023+0000) 2019-10-13T12:24:43.622 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:43.620+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.621596+0000) 2019-10-13T12:24:43.623 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:43.620+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.621766+0000) 2019-10-13T12:24:43.623 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:43.621+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.622198+0000) 2019-10-13T12:24:43.623 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:43.620+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.622279+0000) 2019-10-13T12:24:43.624 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:43.621+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.622556+0000) 2019-10-13T12:24:43.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:43.624+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:43.624864+0000) 2019-10-13T12:24:43.632 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:43.629+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 230979 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:44.136 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:24:44.136 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:44.154 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:24:44.154 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:45.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:45.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:45.767178+0000) 2019-10-13T12:24:45.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:45.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:45.767173+0000) 2019-10-13T12:24:46.262 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:46.260+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:46.261551+0000) 2019-10-13T12:24:48.634 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:48.631+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.633093+0000) 2019-10-13T12:24:48.634 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:48.632+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.633467+0000) 2019-10-13T12:24:48.635 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:48.632+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.633583+0000) 2019-10-13T12:24:48.635 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:48.633+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.634523+0000) 2019-10-13T12:24:48.636 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:48.632+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.633771+0000) 2019-10-13T12:24:48.636 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:48.632+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.633924+0000) 2019-10-13T12:24:48.636 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:48.632+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.634193+0000) 2019-10-13T12:24:48.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:48.635+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.636266+0000) 2019-10-13T12:24:48.643 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:48.641+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231008 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:48.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:48.669+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.669885+0000) 2019-10-13T12:24:49.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:48.999+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:49.001223+0000) 2019-10-13T12:24:51.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:51.620+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.621898+0000) 2019-10-13T12:24:53.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:53.105+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.106286+0000) 2019-10-13T12:24:53.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:53.109+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.110215+0000) 2019-10-13T12:24:53.645 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:53.642+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.644353+0000) 2019-10-13T12:24:53.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:53.643+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.644545+0000) 2019-10-13T12:24:53.646 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:53.644+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.644925+0000) 2019-10-13T12:24:53.646 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:53.643+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.645125+0000) 2019-10-13T12:24:53.647 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:53.644+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.645496+0000) 2019-10-13T12:24:53.647 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:53.644+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.645670+0000) 2019-10-13T12:24:53.647 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:53.644+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.646045+0000) 2019-10-13T12:24:53.648 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:53.646+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:53.647422+0000) 2019-10-13T12:24:53.655 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:53.652+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231041 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:55.768 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:55.766+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:55.767379+0000) 2019-10-13T12:24:55.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:55.766+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:55.767379+0000) 2019-10-13T12:24:56.265 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:56.263+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:56.263943+0000) 2019-10-13T12:24:58.656 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:58.653+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.655254+0000) 2019-10-13T12:24:58.657 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:58.653+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.655386+0000) 2019-10-13T12:24:58.657 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:24:58.654+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.655556+0000) 2019-10-13T12:24:58.658 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:24:58.655+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.655825+0000) 2019-10-13T12:24:58.658 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:24:58.654+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.656039+0000) 2019-10-13T12:24:58.658 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:24:58.654+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.655948+0000) 2019-10-13T12:24:58.658 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:24:58.654+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.656362+0000) 2019-10-13T12:24:58.659 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:24:58.657+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.658254+0000) 2019-10-13T12:24:58.664 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:24:58.661+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231070 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:24:58.670 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:24:58.668+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.669996+0000) 2019-10-13T12:24:59.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:24:59.000+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:59.001390+0000) 2019-10-13T12:25:01.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:01.621+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.622098+0000) 2019-10-13T12:25:03.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:03.105+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.106472+0000) 2019-10-13T12:25:03.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:03.109+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.110395+0000) 2019-10-13T12:25:03.666 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:03.663+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.665260+0000) 2019-10-13T12:25:03.667 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:03.664+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.665548+0000) 2019-10-13T12:25:03.667 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:03.664+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.665716+0000) 2019-10-13T12:25:03.667 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:03.664+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.665882+0000) 2019-10-13T12:25:03.668 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:03.664+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.666248+0000) 2019-10-13T12:25:03.668 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:03.664+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.666320+0000) 2019-10-13T12:25:03.668 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:03.665+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.666623+0000) 2019-10-13T12:25:03.669 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:03.666+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:03.668246+0000) 2019-10-13T12:25:03.677 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:03.674+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231101 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:05.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:05.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:05.767619+0000) 2019-10-13T12:25:05.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:05.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:05.767619+0000) 2019-10-13T12:25:06.267 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:06.265+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:06.266240+0000) 2019-10-13T12:25:08.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:08.669+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.670143+0000) 2019-10-13T12:25:08.678 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:08.675+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.677718+0000) 2019-10-13T12:25:08.679 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:08.677+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.677501+0000) 2019-10-13T12:25:08.679 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:08.677+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.677843+0000) 2019-10-13T12:25:08.679 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:08.677+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.678009+0000) 2019-10-13T12:25:08.680 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:08.678+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.678392+0000) 2019-10-13T12:25:08.680 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:08.678+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.678510+0000) 2019-10-13T12:25:08.680 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:08.678+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.678915+0000) 2019-10-13T12:25:08.682 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:08.679+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.681671+0000) 2019-10-13T12:25:08.688 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:08.687+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231130 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:09.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:09.001+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:09.001547+0000) 2019-10-13T12:25:11.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:11.621+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.622286+0000) 2019-10-13T12:25:13.107 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:13.106+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.106674+0000) 2019-10-13T12:25:13.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:13.109+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.110597+0000) 2019-10-13T12:25:13.690 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:13.688+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.688999+0000) 2019-10-13T12:25:13.690 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:13.688+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.689224+0000) 2019-10-13T12:25:13.690 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:13.688+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.689359+0000) 2019-10-13T12:25:13.691 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:13.689+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.690185+0000) 2019-10-13T12:25:13.691 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:13.689+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.689768+0000) 2019-10-13T12:25:13.692 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:13.689+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.690010+0000) 2019-10-13T12:25:13.692 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:13.689+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.690226+0000) 2019-10-13T12:25:13.693 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:13.692+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:13.692889+0000) 2019-10-13T12:25:13.698 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:13.696+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231163 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:14.309 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:25:14.309 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:14.314 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:25:14.314 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:15.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:15.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:15.767796+0000) 2019-10-13T12:25:15.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:15.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:15.767797+0000) 2019-10-13T12:25:16.269 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:16.267+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:16.268652+0000) 2019-10-13T12:25:18.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:18.670+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.670330+0000) 2019-10-13T12:25:18.700 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:18.698+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.698615+0000) 2019-10-13T12:25:18.700 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:18.698+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.698691+0000) 2019-10-13T12:25:18.700 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:18.698+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.698875+0000) 2019-10-13T12:25:18.700 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:18.698+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.699193+0000) 2019-10-13T12:25:18.701 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:18.698+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.699135+0000) 2019-10-13T12:25:18.701 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:18.699+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.699843+0000) 2019-10-13T12:25:18.701 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:18.699+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.699571+0000) 2019-10-13T12:25:18.702 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:18.700+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.701678+0000) 2019-10-13T12:25:18.707 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:18.705+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231192 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:19.002 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:19.001+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:19.001689+0000) 2019-10-13T12:25:21.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:21.622+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.622480+0000) 2019-10-13T12:25:23.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:23.106+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.106857+0000) 2019-10-13T12:25:23.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:23.110+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.110756+0000) 2019-10-13T12:25:23.709 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:23.707+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.707913+0000) 2019-10-13T12:25:23.709 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:23.707+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.708142+0000) 2019-10-13T12:25:23.710 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:23.707+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.708560+0000) 2019-10-13T12:25:23.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:23.708+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.708869+0000) 2019-10-13T12:25:23.710 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:23.708+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.709033+0000) 2019-10-13T12:25:23.710 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:23.709+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.709491+0000) 2019-10-13T12:25:23.711 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:23.709+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.709804+0000) 2019-10-13T12:25:23.711 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:23.709+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:23.710713+0000) 2019-10-13T12:25:23.719 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:23.718+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231223 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:25.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:25.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:25.767969+0000) 2019-10-13T12:25:25.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:25.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:25.767974+0000) 2019-10-13T12:25:26.271 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:26.270+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:26.270651+0000) 2019-10-13T12:25:28.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:28.669+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.670481+0000) 2019-10-13T12:25:28.721 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:28.719+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.720145+0000) 2019-10-13T12:25:28.722 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:28.719+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.720471+0000) 2019-10-13T12:25:28.722 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:28.719+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.720624+0000) 2019-10-13T12:25:28.722 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:28.719+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.720802+0000) 2019-10-13T12:25:28.722 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:28.719+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.721197+0000) 2019-10-13T12:25:28.723 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:28.719+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.721290+0000) 2019-10-13T12:25:28.723 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:28.721+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.722519+0000) 2019-10-13T12:25:28.723 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:28.720+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.721770+0000) 2019-10-13T12:25:28.731 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:28.728+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231252 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:29.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:29.000+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:29.001844+0000) 2019-10-13T12:25:31.623 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:31.621+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.622626+0000) 2019-10-13T12:25:33.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:33.106+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.107039+0000) 2019-10-13T12:25:33.111 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:33.110+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.110925+0000) 2019-10-13T12:25:33.732 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:33.730+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.731688+0000) 2019-10-13T12:25:33.733 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:33.731+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.732371+0000) 2019-10-13T12:25:33.734 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:33.730+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.732140+0000) 2019-10-13T12:25:33.734 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:33.731+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.732518+0000) 2019-10-13T12:25:33.734 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:33.731+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.732677+0000) 2019-10-13T12:25:33.734 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:33.731+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.732859+0000) 2019-10-13T12:25:33.735 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:33.731+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.733295+0000) 2019-10-13T12:25:33.736 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:33.735+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:33.735989+0000) 2019-10-13T12:25:33.743 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:33.740+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231285 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:35.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:35.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:35.768135+0000) 2019-10-13T12:25:35.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:35.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:35.768135+0000) 2019-10-13T12:25:36.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:36.272+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:36.272815+0000) 2019-10-13T12:25:38.671 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:38.669+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.670668+0000) 2019-10-13T12:25:38.745 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:38.742+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.743776+0000) 2019-10-13T12:25:38.745 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:38.742+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.743919+0000) 2019-10-13T12:25:38.745 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:38.742+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.744241+0000) 2019-10-13T12:25:38.746 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:38.743+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.744556+0000) 2019-10-13T12:25:38.747 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:38.743+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.744758+0000) 2019-10-13T12:25:38.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:38.743+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.745153+0000) 2019-10-13T12:25:38.747 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:38.743+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.745336+0000) 2019-10-13T12:25:38.748 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:38.747+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.747810+0000) 2019-10-13T12:25:38.755 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:38.752+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231314 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:39.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:39.001+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:39.002038+0000) 2019-10-13T12:25:41.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:41.622+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.622834+0000) 2019-10-13T12:25:43.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:43.105+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.107225+0000) 2019-10-13T12:25:43.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:43.110+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.111087+0000) 2019-10-13T12:25:43.756 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:43.753+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.755110+0000) 2019-10-13T12:25:43.757 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:43.755+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.756026+0000) 2019-10-13T12:25:43.757 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:43.754+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.755593+0000) 2019-10-13T12:25:43.757 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:43.754+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.755701+0000) 2019-10-13T12:25:43.758 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:43.754+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.755879+0000) 2019-10-13T12:25:43.758 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:43.754+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.756173+0000) 2019-10-13T12:25:43.758 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:43.755+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.756756+0000) 2019-10-13T12:25:43.759 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:43.757+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:43.758624+0000) 2019-10-13T12:25:43.765 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:43.763+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231345 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:44.447 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:25:44.448 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:44.452 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:25:44.452 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:45.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:45.767+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:45.768327+0000) 2019-10-13T12:25:45.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:45.767+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:45.768327+0000) 2019-10-13T12:25:46.276 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:46.274+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:46.274978+0000) 2019-10-13T12:25:48.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:48.670+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.670855+0000) 2019-10-13T12:25:48.768 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:48.765+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.766505+0000) 2019-10-13T12:25:48.768 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:48.765+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.766947+0000) 2019-10-13T12:25:48.768 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:48.765+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.767056+0000) 2019-10-13T12:25:48.768 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:48.765+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.766422+0000) 2019-10-13T12:25:48.769 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:48.765+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.767282+0000) 2019-10-13T12:25:48.769 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:48.765+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.767337+0000) 2019-10-13T12:25:48.769 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:48.766+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.767987+0000) 2019-10-13T12:25:48.770 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:48.768+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.769647+0000) 2019-10-13T12:25:48.774 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:48.772+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231374 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:49.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:49.001+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:49.002157+0000) 2019-10-13T12:25:51.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:51.622+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.623028+0000) 2019-10-13T12:25:53.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:53.106+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.107334+0000) 2019-10-13T12:25:53.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:53.110+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.111251+0000) 2019-10-13T12:25:53.776 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:53.774+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.775766+0000) 2019-10-13T12:25:53.777 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:53.775+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.776119+0000) 2019-10-13T12:25:53.777 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:53.775+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.776237+0000) 2019-10-13T12:25:53.778 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:53.776+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.776575+0000) 2019-10-13T12:25:53.778 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:53.776+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.776754+0000) 2019-10-13T12:25:53.778 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:53.776+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.777194+0000) 2019-10-13T12:25:53.779 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:53.776+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.777316+0000) 2019-10-13T12:25:53.779 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:53.776+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:53.778550+0000) 2019-10-13T12:25:53.786 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:53.785+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231407 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:55.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:55.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:55.768532+0000) 2019-10-13T12:25:55.776 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:55.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:55.768515+0000) 2019-10-13T12:25:56.278 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:56.276+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:56.276967+0000) 2019-10-13T12:25:58.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:58.670+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.671010+0000) 2019-10-13T12:25:58.788 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:25:58.786+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.787073+0000) 2019-10-13T12:25:58.788 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:25:58.786+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.787232+0000) 2019-10-13T12:25:58.789 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:25:58.785+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.787399+0000) 2019-10-13T12:25:58.789 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:58.787+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.787600+0000) 2019-10-13T12:25:58.789 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:25:58.787+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.787934+0000) 2019-10-13T12:25:58.790 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:25:58.787+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.788297+0000) 2019-10-13T12:25:58.790 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:25:58.788+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.788466+0000) 2019-10-13T12:25:58.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:25:58.788+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.790251+0000) 2019-10-13T12:25:58.797 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:25:58.796+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231436 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:25:59.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:25:59.001+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:59.002296+0000) 2019-10-13T12:26:01.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:01.622+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.623201+0000) 2019-10-13T12:26:03.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:03.107+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.107482+0000) 2019-10-13T12:26:03.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:03.110+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.111412+0000) 2019-10-13T12:26:03.799 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:03.798+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.798491+0000) 2019-10-13T12:26:03.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:03.798+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.798747+0000) 2019-10-13T12:26:03.801 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:03.798+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.799016+0000) 2019-10-13T12:26:03.801 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:03.798+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.799240+0000) 2019-10-13T12:26:03.801 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:03.798+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.799336+0000) 2019-10-13T12:26:03.801 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:03.799+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.799456+0000) 2019-10-13T12:26:03.802 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:03.799+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.799620+0000) 2019-10-13T12:26:03.802 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:03.799+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:03.800915+0000) 2019-10-13T12:26:03.807 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:03.806+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231467 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:05.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:05.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:05.768764+0000) 2019-10-13T12:26:05.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:05.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:05.768761+0000) 2019-10-13T12:26:06.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:06.278+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:06.279307+0000) 2019-10-13T12:26:08.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:08.670+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.671216+0000) 2019-10-13T12:26:08.809 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:08.808+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.808649+0000) 2019-10-13T12:26:08.810 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:08.808+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.808842+0000) 2019-10-13T12:26:08.810 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:08.808+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.809146+0000) 2019-10-13T12:26:08.811 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:08.808+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.809266+0000) 2019-10-13T12:26:08.811 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:08.809+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.809479+0000) 2019-10-13T12:26:08.811 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:08.809+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.809588+0000) 2019-10-13T12:26:08.812 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:08.809+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.810164+0000) 2019-10-13T12:26:08.812 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:08.810+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.811139+0000) 2019-10-13T12:26:08.821 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:08.817+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231496 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:09.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:09.002+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:09.002469+0000) 2019-10-13T12:26:11.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:11.622+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.623393+0000) 2019-10-13T12:26:13.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:13.106+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.107666+0000) 2019-10-13T12:26:13.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:13.110+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.111569+0000) 2019-10-13T12:26:13.821 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:13.818+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.820168+0000) 2019-10-13T12:26:13.821 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:13.818+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.820359+0000) 2019-10-13T12:26:13.822 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:13.819+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.820693+0000) 2019-10-13T12:26:13.822 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:13.819+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.820802+0000) 2019-10-13T12:26:13.822 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:13.819+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.821104+0000) 2019-10-13T12:26:13.823 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:13.819+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.821138+0000) 2019-10-13T12:26:13.823 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:13.819+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.821288+0000) 2019-10-13T12:26:13.824 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:13.823+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:13.823793+0000) 2019-10-13T12:26:13.831 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:13.828+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231529 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:14.569 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:26:14.569 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:14.573 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:26:14.573 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:15.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:15.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:15.768954+0000) 2019-10-13T12:26:15.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:15.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:15.768955+0000) 2019-10-13T12:26:16.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:16.280+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:16.281661+0000) 2019-10-13T12:26:18.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:18.670+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.671427+0000) 2019-10-13T12:26:18.832 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:18.830+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.831592+0000) 2019-10-13T12:26:18.833 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:18.830+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.831804+0000) 2019-10-13T12:26:18.834 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:18.831+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.832497+0000) 2019-10-13T12:26:18.834 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:18.830+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.832202+0000) 2019-10-13T12:26:18.834 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:18.831+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.832644+0000) 2019-10-13T12:26:18.834 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:18.831+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.833058+0000) 2019-10-13T12:26:18.835 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:18.831+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.832901+0000) 2019-10-13T12:26:18.835 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:18.833+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.834138+0000) 2019-10-13T12:26:18.843 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:18.840+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231558 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:19.003 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:19.001+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:19.002613+0000) 2019-10-13T12:26:21.624 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:21.622+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.623549+0000) 2019-10-13T12:26:23.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:23.107+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.107856+0000) 2019-10-13T12:26:23.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:23.111+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.111772+0000) 2019-10-13T12:26:23.845 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:23.842+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.843912+0000) 2019-10-13T12:26:23.845 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:23.842+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.844053+0000) 2019-10-13T12:26:23.846 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:23.843+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.844490+0000) 2019-10-13T12:26:23.846 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:23.843+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.844681+0000) 2019-10-13T12:26:23.846 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:23.843+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.844847+0000) 2019-10-13T12:26:23.846 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:23.843+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.845036+0000) 2019-10-13T12:26:23.847 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:23.843+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.845380+0000) 2019-10-13T12:26:23.847 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:23.845+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:23.845920+0000) 2019-10-13T12:26:23.853 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:23.851+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231589 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:25.770 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:25.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:25.769132+0000) 2019-10-13T12:26:25.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:25.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:25.769130+0000) 2019-10-13T12:26:26.285 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:26.283+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:26.284022+0000) 2019-10-13T12:26:28.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:28.670+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.671620+0000) 2019-10-13T12:26:28.856 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:28.853+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.854850+0000) 2019-10-13T12:26:28.856 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:28.853+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.855367+0000) 2019-10-13T12:26:28.857 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:28.854+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.855502+0000) 2019-10-13T12:26:28.857 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:28.854+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.855717+0000) 2019-10-13T12:26:28.857 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:28.854+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.855965+0000) 2019-10-13T12:26:28.858 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:28.855+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.856054+0000) 2019-10-13T12:26:28.858 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:28.856+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.856936+0000) 2019-10-13T12:26:28.858 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:28.854+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.856344+0000) 2019-10-13T12:26:28.865 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:28.863+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231618 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:29.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:29.002+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:29.002765+0000) 2019-10-13T12:26:31.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:31.623+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.623712+0000) 2019-10-13T12:26:33.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:33.107+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.108088+0000) 2019-10-13T12:26:33.112 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:33.111+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.111943+0000) 2019-10-13T12:26:33.869 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:33.865+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.867275+0000) 2019-10-13T12:26:33.869 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:33.866+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.867915+0000) 2019-10-13T12:26:33.869 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:33.866+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.868058+0000) 2019-10-13T12:26:33.870 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:33.867+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.867861+0000) 2019-10-13T12:26:33.870 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:33.866+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.868275+0000) 2019-10-13T12:26:33.870 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:33.867+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.868533+0000) 2019-10-13T12:26:33.870 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:33.867+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.868804+0000) 2019-10-13T12:26:33.871 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:33.869+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:33.869890+0000) 2019-10-13T12:26:33.877 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:33.875+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231651 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:35.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:35.768+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:35.769329+0000) 2019-10-13T12:26:35.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:35.768+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:35.769330+0000) 2019-10-13T12:26:36.287 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:36.286+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:36.286228+0000) 2019-10-13T12:26:38.686 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:38.671+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.671829+0000) 2019-10-13T12:26:38.888 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:38.877+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.877972+0000) 2019-10-13T12:26:38.888 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:38.877+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.878258+0000) 2019-10-13T12:26:38.889 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:38.877+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.878133+0000) 2019-10-13T12:26:38.889 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:38.878+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.878738+0000) 2019-10-13T12:26:38.890 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:38.878+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.878902+0000) 2019-10-13T12:26:38.890 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:38.878+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.879336+0000) 2019-10-13T12:26:38.890 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:38.879+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.879584+0000) 2019-10-13T12:26:38.891 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:38.880+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.881194+0000) 2019-10-13T12:26:38.899 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:38.887+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231680 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:39.014 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:39.002+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:39.002945+0000) 2019-10-13T12:26:41.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:41.623+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.623875+0000) 2019-10-13T12:26:43.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:43.107+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.108264+0000) 2019-10-13T12:26:43.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:43.111+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.112132+0000) 2019-10-13T12:26:43.890 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:43.887+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.889539+0000) 2019-10-13T12:26:43.891 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:43.889+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.889390+0000) 2019-10-13T12:26:43.891 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:43.889+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.889724+0000) 2019-10-13T12:26:43.892 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:43.889+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.890152+0000) 2019-10-13T12:26:43.892 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:43.889+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.890298+0000) 2019-10-13T12:26:43.892 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:43.890+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.890428+0000) 2019-10-13T12:26:43.893 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:43.890+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.890715+0000) 2019-10-13T12:26:43.893 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:43.890+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:43.892713+0000) 2019-10-13T12:26:43.900 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:43.898+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231711 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:44.683 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:26:44.683 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:44.688 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:26:44.688 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:45.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:45.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:45.769525+0000) 2019-10-13T12:26:45.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:45.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:45.769533+0000) 2019-10-13T12:26:46.289 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:46.287+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:46.288606+0000) 2019-10-13T12:26:48.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:48.671+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.672024+0000) 2019-10-13T12:26:48.901 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:48.898+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.900488+0000) 2019-10-13T12:26:48.902 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:48.900+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.900898+0000) 2019-10-13T12:26:48.902 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:48.901+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.901541+0000) 2019-10-13T12:26:48.902 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:48.901+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.901669+0000) 2019-10-13T12:26:48.903 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:48.901+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.901872+0000) 2019-10-13T12:26:48.903 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:48.901+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.901971+0000) 2019-10-13T12:26:48.903 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:48.901+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.902113+0000) 2019-10-13T12:26:48.904 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:48.901+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.903188+0000) 2019-10-13T12:26:48.912 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:48.910+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231740 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:49.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:49.002+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:49.003062+0000) 2019-10-13T12:26:51.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:51.623+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.624008+0000) 2019-10-13T12:26:53.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:53.108+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.108451+0000) 2019-10-13T12:26:53.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:53.111+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.112334+0000) 2019-10-13T12:26:53.914 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:53.911+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.913130+0000) 2019-10-13T12:26:53.915 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:53.912+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.913233+0000) 2019-10-13T12:26:53.915 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:53.912+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.913343+0000) 2019-10-13T12:26:53.915 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:53.913+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.913757+0000) 2019-10-13T12:26:53.915 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:53.913+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.913957+0000) 2019-10-13T12:26:53.915 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:53.913+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.913862+0000) 2019-10-13T12:26:53.916 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:53.913+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.915096+0000) 2019-10-13T12:26:53.916 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:53.914+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:53.914455+0000) 2019-10-13T12:26:53.925 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:53.923+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231773 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:55.770 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:55.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:55.769702+0000) 2019-10-13T12:26:55.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:55.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:55.769737+0000) 2019-10-13T12:26:56.292 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:56.290+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:56.290893+0000) 2019-10-13T12:26:58.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:58.671+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.672238+0000) 2019-10-13T12:26:58.927 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:26:58.924+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.925809+0000) 2019-10-13T12:26:58.927 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:58.924+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.926348+0000) 2019-10-13T12:26:58.928 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:26:58.924+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.926254+0000) 2019-10-13T12:26:58.928 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:26:58.925+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.926780+0000) 2019-10-13T12:26:58.928 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:26:58.925+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.926886+0000) 2019-10-13T12:26:58.929 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:26:58.925+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.926662+0000) 2019-10-13T12:26:58.929 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:26:58.926+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.927441+0000) 2019-10-13T12:26:58.930 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:26:58.929+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.929966+0000) 2019-10-13T12:26:58.936 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:26:58.934+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231802 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:26:59.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:26:59.002+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:59.003235+0000) 2019-10-13T12:27:01.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:01.623+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.624208+0000) 2019-10-13T12:27:03.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:03.107+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.108605+0000) 2019-10-13T12:27:03.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:03.111+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.112524+0000) 2019-10-13T12:27:03.938 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:03.935+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.937071+0000) 2019-10-13T12:27:03.938 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:03.935+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.937205+0000) 2019-10-13T12:27:03.939 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:03.935+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.937381+0000) 2019-10-13T12:27:03.939 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:03.936+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.937581+0000) 2019-10-13T12:27:03.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:03.937+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.938121+0000) 2019-10-13T12:27:03.940 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:03.936+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.937784+0000) 2019-10-13T12:27:03.940 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:03.936+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.938380+0000) 2019-10-13T12:27:03.941 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:03.939+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:03.940624+0000) 2019-10-13T12:27:03.947 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:03.945+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231833 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:05.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:05.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:05.769899+0000) 2019-10-13T12:27:05.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:05.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:05.769924+0000) 2019-10-13T12:27:06.294 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:06.292+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:06.293229+0000) 2019-10-13T12:27:08.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:08.671+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.672429+0000) 2019-10-13T12:27:08.950 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:08.947+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.948577+0000) 2019-10-13T12:27:08.950 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:08.947+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.948674+0000) 2019-10-13T12:27:08.951 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:08.947+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.948996+0000) 2019-10-13T12:27:08.951 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:08.947+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.949224+0000) 2019-10-13T12:27:08.952 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:08.949+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.949853+0000) 2019-10-13T12:27:08.952 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:08.948+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.949635+0000) 2019-10-13T12:27:08.952 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:08.948+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.950060+0000) 2019-10-13T12:27:08.954 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:08.950+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.951161+0000) 2019-10-13T12:27:08.962 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:08.956+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231863 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:09.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:09.002+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:09.003372+0000) 2019-10-13T12:27:11.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:11.623+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.624434+0000) 2019-10-13T12:27:13.109 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:13.108+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.108772+0000) 2019-10-13T12:27:13.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:13.112+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.112673+0000) 2019-10-13T12:27:13.961 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:13.958+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.959860+0000) 2019-10-13T12:27:13.961 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:13.958+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.960136+0000) 2019-10-13T12:27:13.962 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:13.959+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.960255+0000) 2019-10-13T12:27:13.962 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:13.958+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.960389+0000) 2019-10-13T12:27:13.962 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:13.959+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.960621+0000) 2019-10-13T12:27:13.962 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:13.959+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.960769+0000) 2019-10-13T12:27:13.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:13.959+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.961159+0000) 2019-10-13T12:27:13.965 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:13.963+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:13.964528+0000) 2019-10-13T12:27:13.970 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:13.968+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231896 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:14.823 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:27:14.823 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:14.827 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:27:14.827 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:15.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:15.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:15.770097+0000) 2019-10-13T12:27:15.771 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:15.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:15.770114+0000) 2019-10-13T12:27:16.296 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:16.294+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:16.295571+0000) 2019-10-13T12:27:18.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:18.672+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.672623+0000) 2019-10-13T12:27:18.972 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:18.970+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.971436+0000) 2019-10-13T12:27:18.973 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:18.970+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.971750+0000) 2019-10-13T12:27:18.973 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:18.971+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.972152+0000) 2019-10-13T12:27:18.974 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:18.970+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.972016+0000) 2019-10-13T12:27:18.974 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:18.970+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.972291+0000) 2019-10-13T12:27:18.974 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:18.971+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.972658+0000) 2019-10-13T12:27:18.974 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:18.971+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.972857+0000) 2019-10-13T12:27:18.975 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:18.973+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.974566+0000) 2019-10-13T12:27:18.982 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:18.979+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231926 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:19.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:19.002+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:19.003517+0000) 2019-10-13T12:27:21.625 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:21.624+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.624559+0000) 2019-10-13T12:27:23.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:23.108+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.108941+0000) 2019-10-13T12:27:23.113 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:23.112+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.112839+0000) 2019-10-13T12:27:23.984 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:23.981+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.982854+0000) 2019-10-13T12:27:23.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:23.981+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.983093+0000) 2019-10-13T12:27:23.984 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:23.981+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.983289+0000) 2019-10-13T12:27:23.985 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:23.982+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.983445+0000) 2019-10-13T12:27:23.985 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:23.982+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.983608+0000) 2019-10-13T12:27:23.985 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:23.982+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.984118+0000) 2019-10-13T12:27:23.986 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:23.983+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.984712+0000) 2019-10-13T12:27:23.986 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:23.985+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.985888+0000) 2019-10-13T12:27:23.993 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:23.991+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231957 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:25.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:25.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:25.770310+0000) 2019-10-13T12:27:25.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:25.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:25.770310+0000) 2019-10-13T12:27:26.299 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:26.297+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:26.297872+0000) 2019-10-13T12:27:28.673 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:28.672+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.672823+0000) 2019-10-13T12:27:28.996 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:28.994+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.994789+0000) 2019-10-13T12:27:28.996 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:28.994+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.995041+0000) 2019-10-13T12:27:28.997 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:28.994+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.995168+0000) 2019-10-13T12:27:28.997 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:28.995+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.995426+0000) 2019-10-13T12:27:28.997 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:28.995+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.995478+0000) 2019-10-13T12:27:28.998 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:28.995+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.995632+0000) 2019-10-13T12:27:28.998 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:28.995+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.995823+0000) 2019-10-13T12:27:28.998 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:28.995+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.996671+0000) 2019-10-13T12:27:29.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:29.003+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:29.003642+0000) 2019-10-13T12:27:29.005 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:29.003+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 231987 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:31.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:31.624+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.624711+0000) 2019-10-13T12:27:33.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:33.108+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.109128+0000) 2019-10-13T12:27:33.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:33.112+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.113018+0000) 2019-10-13T12:27:34.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:34.005+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.006124+0000) 2019-10-13T12:27:34.008 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:34.005+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.006250+0000) 2019-10-13T12:27:34.008 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:34.006+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.006470+0000) 2019-10-13T12:27:34.008 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:34.006+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.006614+0000) 2019-10-13T12:27:34.009 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:34.006+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.006890+0000) 2019-10-13T12:27:34.009 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:34.006+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.007089+0000) 2019-10-13T12:27:34.009 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:34.005+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.006808+0000) 2019-10-13T12:27:34.010 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:34.007+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.009350+0000) 2019-10-13T12:27:34.017 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:34.015+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232020 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:35.771 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:35.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:35.770508+0000) 2019-10-13T12:27:35.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:35.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:35.770511+0000) 2019-10-13T12:27:36.301 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:36.299+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:36.300092+0000) 2019-10-13T12:27:38.674 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:38.672+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.672998+0000) 2019-10-13T12:27:39.004 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:39.003+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.003772+0000) 2019-10-13T12:27:39.021 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:39.016+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.018301+0000) 2019-10-13T12:27:39.021 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:39.017+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.017430+0000) 2019-10-13T12:27:39.021 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:39.017+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.017648+0000) 2019-10-13T12:27:39.021 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:39.017+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.017882+0000) 2019-10-13T12:27:39.022 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:39.017+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.018374+0000) 2019-10-13T12:27:39.022 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:39.018+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.018517+0000) 2019-10-13T12:27:39.022 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:39.018+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.018687+0000) 2019-10-13T12:27:39.022 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:39.018+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:39.020537+0000) 2019-10-13T12:27:39.027 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:39.025+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232050 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:41.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:41.624+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.624928+0000) 2019-10-13T12:27:43.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:43.108+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.109333+0000) 2019-10-13T12:27:43.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:43.112+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.113209+0000) 2019-10-13T12:27:44.029 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:44.026+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.028050+0000) 2019-10-13T12:27:44.030 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:44.026+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.028220+0000) 2019-10-13T12:27:44.030 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:44.026+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.028352+0000) 2019-10-13T12:27:44.030 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:44.027+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.028759+0000) 2019-10-13T12:27:44.032 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:44.027+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.028601+0000) 2019-10-13T12:27:44.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:44.029+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.029847+0000) 2019-10-13T12:27:44.032 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:44.027+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.028931+0000) 2019-10-13T12:27:44.033 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:44.028+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.029417+0000) 2019-10-13T12:27:44.038 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:44.035+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232081 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:44.945 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:27:44.945 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:44.949 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:27:44.949 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:45.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:45.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:45.770717+0000) 2019-10-13T12:27:45.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:45.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:45.770718+0000) 2019-10-13T12:27:46.303 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:46.301+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:46.302359+0000) 2019-10-13T12:27:48.674 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:48.672+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.673165+0000) 2019-10-13T12:27:49.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:49.003+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.003934+0000) 2019-10-13T12:27:49.040 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:49.037+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.038750+0000) 2019-10-13T12:27:49.040 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:49.037+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.038863+0000) 2019-10-13T12:27:49.040 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:49.037+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.039089+0000) 2019-10-13T12:27:49.041 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:49.039+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.039893+0000) 2019-10-13T12:27:49.041 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:49.038+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.039499+0000) 2019-10-13T12:27:49.041 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:49.038+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.039997+0000) 2019-10-13T12:27:49.042 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:49.038+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.040107+0000) 2019-10-13T12:27:49.042 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:49.040+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:49.041543+0000) 2019-10-13T12:27:49.048 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:49.046+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:51.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:51.623+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.625116+0000) 2019-10-13T12:27:53.110 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:53.108+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.109517+0000) 2019-10-13T12:27:53.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:53.112+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.113389+0000) 2019-10-13T12:27:54.050 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:54.047+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.049264+0000) 2019-10-13T12:27:54.051 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:54.048+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.049522+0000) 2019-10-13T12:27:54.051 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:54.049+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.050384+0000) 2019-10-13T12:27:54.052 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:54.048+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.049763+0000) 2019-10-13T12:27:54.052 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:54.048+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.050241+0000) 2019-10-13T12:27:54.052 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:54.049+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.050501+0000) 2019-10-13T12:27:54.052 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:54.049+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.051175+0000) 2019-10-13T12:27:54.054 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:54.052+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.053199+0000) 2019-10-13T12:27:54.060 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:54.057+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232144 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:27:55.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:55.769+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:55.770921+0000) 2019-10-13T12:27:55.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:55.769+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:55.770919+0000) 2019-10-13T12:27:56.305 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:56.304+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:56.304657+0000) 2019-10-13T12:27:58.674 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:58.672+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.673340+0000) 2019-10-13T12:27:59.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:59.003+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.004117+0000) 2019-10-13T12:27:59.061 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:27:59.058+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.060242+0000) 2019-10-13T12:27:59.062 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:27:59.059+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.060410+0000) 2019-10-13T12:27:59.062 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:27:59.059+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.060558+0000) 2019-10-13T12:27:59.063 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:27:59.060+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.060778+0000) 2019-10-13T12:27:59.063 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:27:59.060+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.061509+0000) 2019-10-13T12:27:59.063 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:27:59.060+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.061597+0000) 2019-10-13T12:27:59.063 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:27:59.060+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.061665+0000) 2019-10-13T12:27:59.065 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:27:59.063+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:59.064036+0000) 2019-10-13T12:27:59.071 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:27:59.068+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232174 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:01.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:01.624+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.625306+0000) 2019-10-13T12:28:03.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:03.109+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.109710+0000) 2019-10-13T12:28:03.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:03.113+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.113621+0000) 2019-10-13T12:28:04.072 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:04.070+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.071670+0000) 2019-10-13T12:28:04.073 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:04.070+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.071928+0000) 2019-10-13T12:28:04.073 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:04.070+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.072113+0000) 2019-10-13T12:28:04.074 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:04.070+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.072276+0000) 2019-10-13T12:28:04.074 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:04.071+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.072597+0000) 2019-10-13T12:28:04.074 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:04.071+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.073027+0000) 2019-10-13T12:28:04.075 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:04.071+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.072688+0000) 2019-10-13T12:28:04.075 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:04.073+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.074503+0000) 2019-10-13T12:28:04.082 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:04.079+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232205 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:05.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:05.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:05.771127+0000) 2019-10-13T12:28:05.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:05.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:05.771135+0000) 2019-10-13T12:28:06.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:06.306+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:06.306759+0000) 2019-10-13T12:28:08.674 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:08.673+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.673547+0000) 2019-10-13T12:28:09.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:09.003+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.004303+0000) 2019-10-13T12:28:09.083 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:09.081+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.082543+0000) 2019-10-13T12:28:09.084 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:09.081+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.082663+0000) 2019-10-13T12:28:09.085 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:09.082+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.082974+0000) 2019-10-13T12:28:09.085 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:09.081+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.083084+0000) 2019-10-13T12:28:09.085 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:09.081+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.083214+0000) 2019-10-13T12:28:09.085 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:09.082+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.083427+0000) 2019-10-13T12:28:09.085 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:09.082+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.083812+0000) 2019-10-13T12:28:09.087 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:09.085+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:09.086504+0000) 2019-10-13T12:28:09.093 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:09.090+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232235 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:11.626 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:11.625+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.625485+0000) 2019-10-13T12:28:13.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:13.109+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.109891+0000) 2019-10-13T12:28:13.114 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:13.113+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.113827+0000) 2019-10-13T12:28:14.095 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:14.093+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.093848+0000) 2019-10-13T12:28:14.095 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:14.093+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.094021+0000) 2019-10-13T12:28:14.096 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:14.093+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.094316+0000) 2019-10-13T12:28:14.096 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:14.093+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.094182+0000) 2019-10-13T12:28:14.097 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:14.094+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.094673+0000) 2019-10-13T12:28:14.097 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:14.094+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.094760+0000) 2019-10-13T12:28:14.097 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:14.095+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.095426+0000) 2019-10-13T12:28:14.098 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:14.096+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.097342+0000) 2019-10-13T12:28:14.104 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:14.103+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232268 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:15.051 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:28:15.051 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:15.056 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:28:15.056 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:15.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:15.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:15.771308+0000) 2019-10-13T12:28:15.772 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:15.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:15.771319+0000) 2019-10-13T12:28:16.310 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:16.308+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:16.309118+0000) 2019-10-13T12:28:18.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:18.673+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.673744+0000) 2019-10-13T12:28:19.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:19.004+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.004491+0000) 2019-10-13T12:28:19.106 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:19.105+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.105594+0000) 2019-10-13T12:28:19.107 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:19.104+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.105681+0000) 2019-10-13T12:28:19.108 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:19.105+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.106155+0000) 2019-10-13T12:28:19.108 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:19.105+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.106333+0000) 2019-10-13T12:28:19.108 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:19.106+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.106446+0000) 2019-10-13T12:28:19.108 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:19.106+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.106623+0000) 2019-10-13T12:28:19.109 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:19.106+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.106798+0000) 2019-10-13T12:28:19.109 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:19.107+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:19.107862+0000) 2019-10-13T12:28:19.120 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:19.114+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232298 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:21.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:21.625+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.625668+0000) 2019-10-13T12:28:23.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:23.109+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.110084+0000) 2019-10-13T12:28:23.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:23.112+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.113971+0000) 2019-10-13T12:28:24.118 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:24.116+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.117066+0000) 2019-10-13T12:28:24.119 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:24.115+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.117719+0000) 2019-10-13T12:28:24.119 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:24.117+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.117479+0000) 2019-10-13T12:28:24.119 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:24.117+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.117662+0000) 2019-10-13T12:28:24.119 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:24.116+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.117339+0000) 2019-10-13T12:28:24.120 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:24.117+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.117838+0000) 2019-10-13T12:28:24.120 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:24.117+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.118150+0000) 2019-10-13T12:28:24.120 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:24.117+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.119493+0000) 2019-10-13T12:28:24.126 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:24.125+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232329 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:25.772 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:25.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:25.771481+0000) 2019-10-13T12:28:25.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:25.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:25.771481+0000) 2019-10-13T12:28:26.312 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:26.310+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:26.311237+0000) 2019-10-13T12:28:28.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:28.672+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.673929+0000) 2019-10-13T12:28:29.005 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:29.003+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.004677+0000) 2019-10-13T12:28:29.128 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:29.125+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.127343+0000) 2019-10-13T12:28:29.129 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:29.125+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.127327+0000) 2019-10-13T12:28:29.129 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:29.126+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.127891+0000) 2019-10-13T12:28:29.130 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:29.126+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.128042+0000) 2019-10-13T12:28:29.130 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:29.126+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.128167+0000) 2019-10-13T12:28:29.130 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:29.126+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.128312+0000) 2019-10-13T12:28:29.130 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:29.127+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.128738+0000) 2019-10-13T12:28:29.131 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:29.127+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:29.129656+0000) 2019-10-13T12:28:29.136 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:29.134+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232359 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:31.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:31.624+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.625826+0000) 2019-10-13T12:28:33.119 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:33.109+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.110222+0000) 2019-10-13T12:28:33.123 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:33.113+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.114147+0000) 2019-10-13T12:28:34.139 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:34.136+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.138016+0000) 2019-10-13T12:28:34.139 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:34.136+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.138239+0000) 2019-10-13T12:28:34.139 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:34.136+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.138368+0000) 2019-10-13T12:28:34.140 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:34.137+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.138567+0000) 2019-10-13T12:28:34.140 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:34.138+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.139086+0000) 2019-10-13T12:28:34.141 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:34.137+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.138929+0000) 2019-10-13T12:28:34.141 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:34.137+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.139187+0000) 2019-10-13T12:28:34.141 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:34.140+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.140803+0000) 2019-10-13T12:28:34.146 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:34.144+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232392 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:35.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:35.770+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:35.771659+0000) 2019-10-13T12:28:35.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:35.770+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:35.771660+0000) 2019-10-13T12:28:36.314 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:36.312+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:36.313521+0000) 2019-10-13T12:28:38.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:38.673+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.674121+0000) 2019-10-13T12:28:39.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:39.004+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.004886+0000) 2019-10-13T12:28:39.148 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:39.146+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.147198+0000) 2019-10-13T12:28:39.149 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:39.146+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.147631+0000) 2019-10-13T12:28:39.149 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:39.146+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.147700+0000) 2019-10-13T12:28:39.149 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:39.146+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.147897+0000) 2019-10-13T12:28:39.149 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:39.146+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.148034+0000) 2019-10-13T12:28:39.150 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:39.146+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.148294+0000) 2019-10-13T12:28:39.150 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:39.148+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.149473+0000) 2019-10-13T12:28:39.150 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:39.147+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:39.148529+0000) 2019-10-13T12:28:39.158 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:39.155+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232422 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:41.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:41.624+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.626020+0000) 2019-10-13T12:28:43.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:43.109+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.110388+0000) 2019-10-13T12:28:43.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:43.113+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.114351+0000) 2019-10-13T12:28:44.159 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:44.157+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.158682+0000) 2019-10-13T12:28:44.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:44.157+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.158955+0000) 2019-10-13T12:28:44.161 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:44.158+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.159258+0000) 2019-10-13T12:28:44.161 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:44.158+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.159512+0000) 2019-10-13T12:28:44.161 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:44.158+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.159630+0000) 2019-10-13T12:28:44.161 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:44.158+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.159728+0000) 2019-10-13T12:28:44.161 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:44.158+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.159884+0000) 2019-10-13T12:28:44.162 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:44.160+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.161218+0000) 2019-10-13T12:28:44.170 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:44.167+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232453 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:45.197 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:28:45.198 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:45.202 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:28:45.202 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:45.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:45.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:45.771848+0000) 2019-10-13T12:28:45.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:45.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:45.771847+0000) 2019-10-13T12:28:46.317 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:46.315+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:46.315887+0000) 2019-10-13T12:28:48.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:48.673+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.674277+0000) 2019-10-13T12:28:49.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:49.003+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.005097+0000) 2019-10-13T12:28:49.172 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:49.169+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.170609+0000) 2019-10-13T12:28:49.173 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:49.169+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.170722+0000) 2019-10-13T12:28:49.173 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:49.169+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.170872+0000) 2019-10-13T12:28:49.174 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:49.169+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.171044+0000) 2019-10-13T12:28:49.174 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:49.169+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.171185+0000) 2019-10-13T12:28:49.174 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:49.169+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.171330+0000) 2019-10-13T12:28:49.175 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:49.170+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.171041+0000) 2019-10-13T12:28:49.175 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:49.173+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:49.173805+0000) 2019-10-13T12:28:49.179 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:49.176+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232483 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:51.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:51.625+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.626218+0000) 2019-10-13T12:28:53.111 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:53.110+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.110608+0000) 2019-10-13T12:28:53.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:53.113+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.114545+0000) 2019-10-13T12:28:54.181 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:54.179+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.179997+0000) 2019-10-13T12:28:54.181 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:54.178+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.180251+0000) 2019-10-13T12:28:54.182 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:54.179+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.180452+0000) 2019-10-13T12:28:54.182 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:54.179+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.180726+0000) 2019-10-13T12:28:54.182 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:54.179+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.181120+0000) 2019-10-13T12:28:54.183 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:54.180+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.181474+0000) 2019-10-13T12:28:54.183 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:54.180+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.181514+0000) 2019-10-13T12:28:54.183 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:54.182+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.182799+0000) 2019-10-13T12:28:54.191 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:54.189+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232516 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:28:55.773 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:55.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:55.772046+0000) 2019-10-13T12:28:55.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:55.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:55.772047+0000) 2019-10-13T12:28:56.319 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:56.317+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:56.318210+0000) 2019-10-13T12:28:58.685 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:58.674+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.674451+0000) 2019-10-13T12:28:59.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:59.004+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.005261+0000) 2019-10-13T12:28:59.198 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:28:59.191+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.192350+0000) 2019-10-13T12:28:59.198 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:28:59.192+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.192537+0000) 2019-10-13T12:28:59.199 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:28:59.192+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.192753+0000) 2019-10-13T12:28:59.199 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:28:59.192+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.193213+0000) 2019-10-13T12:28:59.199 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:28:59.193+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.193432+0000) 2019-10-13T12:28:59.200 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:28:59.193+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.193575+0000) 2019-10-13T12:28:59.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:28:59.193+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.193765+0000) 2019-10-13T12:28:59.203 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:28:59.195+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:59.195937+0000) 2019-10-13T12:28:59.207 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:28:59.202+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232546 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:01.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:01.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.626444+0000) 2019-10-13T12:29:03.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:03.110+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.110816+0000) 2019-10-13T12:29:03.115 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:03.114+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.114706+0000) 2019-10-13T12:29:04.206 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:04.203+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.204692+0000) 2019-10-13T12:29:04.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:04.204+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.204901+0000) 2019-10-13T12:29:04.207 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:04.204+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.205046+0000) 2019-10-13T12:29:04.207 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:04.204+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.205311+0000) 2019-10-13T12:29:04.207 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:04.205+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.205562+0000) 2019-10-13T12:29:04.208 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:04.205+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.205691+0000) 2019-10-13T12:29:04.208 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:04.205+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.206095+0000) 2019-10-13T12:29:04.209 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:04.207+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.208129+0000) 2019-10-13T12:29:04.215 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:04.213+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232577 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:05.773 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:05.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:05.772284+0000) 2019-10-13T12:29:05.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:05.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:05.772286+0000) 2019-10-13T12:29:06.322 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:06.320+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:06.320745+0000) 2019-10-13T12:29:08.675 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:08.674+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.674634+0000) 2019-10-13T12:29:09.006 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:09.005+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.005448+0000) 2019-10-13T12:29:09.217 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:09.215+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.215909+0000) 2019-10-13T12:29:09.217 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:09.215+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.216117+0000) 2019-10-13T12:29:09.218 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:09.216+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.216757+0000) 2019-10-13T12:29:09.218 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:09.216+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.216512+0000) 2019-10-13T12:29:09.218 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:09.216+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.216702+0000) 2019-10-13T12:29:09.219 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:09.216+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.217164+0000) 2019-10-13T12:29:09.219 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:09.217+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.217572+0000) 2019-10-13T12:29:09.220 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:09.218+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:09.219001+0000) 2019-10-13T12:29:09.227 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:09.226+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232608 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:11.627 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:11.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.626641+0000) 2019-10-13T12:29:13.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:13.109+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.110982+0000) 2019-10-13T12:29:13.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:13.113+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.114916+0000) 2019-10-13T12:29:14.229 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:14.226+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.228107+0000) 2019-10-13T12:29:14.229 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:14.227+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.228475+0000) 2019-10-13T12:29:14.231 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:14.227+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.229014+0000) 2019-10-13T12:29:14.231 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:14.227+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.228736+0000) 2019-10-13T12:29:14.231 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:14.227+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.228893+0000) 2019-10-13T12:29:14.232 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:14.228+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.229479+0000) 2019-10-13T12:29:14.232 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:14.228+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.229654+0000) 2019-10-13T12:29:14.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:14.229+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.231434+0000) 2019-10-13T12:29:14.241 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:14.238+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232640 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:15.312 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:29:15.312 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:15.316 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:29:15.316 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:15.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:15.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:15.772406+0000) 2019-10-13T12:29:15.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:15.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:15.772444+0000) 2019-10-13T12:29:16.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:16.322+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:16.323023+0000) 2019-10-13T12:29:18.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:18.673+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.674815+0000) 2019-10-13T12:29:19.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:19.004+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.005644+0000) 2019-10-13T12:29:19.242 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:19.240+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.241634+0000) 2019-10-13T12:29:19.244 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:19.240+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.242161+0000) 2019-10-13T12:29:19.244 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:19.240+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.242048+0000) 2019-10-13T12:29:19.244 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:19.240+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.241866+0000) 2019-10-13T12:29:19.244 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:19.240+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.242319+0000) 2019-10-13T12:29:19.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:19.241+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.242581+0000) 2019-10-13T12:29:19.245 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:19.241+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.242717+0000) 2019-10-13T12:29:19.246 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:19.243+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:19.245204+0000) 2019-10-13T12:29:19.251 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:19.249+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232671 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:21.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:21.625+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.626790+0000) 2019-10-13T12:29:23.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:23.110+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.111150+0000) 2019-10-13T12:29:23.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:23.114+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.115119+0000) 2019-10-13T12:29:24.254 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:24.251+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.252403+0000) 2019-10-13T12:29:24.254 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:24.251+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.252447+0000) 2019-10-13T12:29:24.254 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:24.251+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.252523+0000) 2019-10-13T12:29:24.254 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:24.251+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.252663+0000) 2019-10-13T12:29:24.255 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:24.251+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.253200+0000) 2019-10-13T12:29:24.255 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:24.251+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.253306+0000) 2019-10-13T12:29:24.255 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:24.252+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.253496+0000) 2019-10-13T12:29:24.256 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:24.254+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.255166+0000) 2019-10-13T12:29:24.261 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:24.259+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232701 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:25.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:25.771+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:25.772597+0000) 2019-10-13T12:29:25.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:25.771+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:25.772598+0000) 2019-10-13T12:29:26.326 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:26.324+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:26.325365+0000) 2019-10-13T12:29:28.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:28.674+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.675008+0000) 2019-10-13T12:29:29.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:29.005+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.005783+0000) 2019-10-13T12:29:29.280 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:29.261+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.262618+0000) 2019-10-13T12:29:29.280 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:29.260+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.262260+0000) 2019-10-13T12:29:29.280 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:29.261+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.262443+0000) 2019-10-13T12:29:29.281 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:29.261+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.262831+0000) 2019-10-13T12:29:29.281 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:29.261+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.263187+0000) 2019-10-13T12:29:29.281 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:29.262+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.263543+0000) 2019-10-13T12:29:29.281 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:29.262+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.264229+0000) 2019-10-13T12:29:29.282 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:29.265+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:29.265861+0000) 2019-10-13T12:29:29.288 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:29.270+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232732 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:31.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:31.625+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.626990+0000) 2019-10-13T12:29:33.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:33.110+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.111321+0000) 2019-10-13T12:29:33.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:33.114+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.115296+0000) 2019-10-13T12:29:34.275 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:34.272+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.273851+0000) 2019-10-13T12:29:34.275 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:34.272+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.274185+0000) 2019-10-13T12:29:34.276 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:34.273+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.274031+0000) 2019-10-13T12:29:34.276 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:34.272+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.274301+0000) 2019-10-13T12:29:34.276 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:34.273+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.274671+0000) 2019-10-13T12:29:34.277 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:34.273+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.275198+0000) 2019-10-13T12:29:34.277 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:34.274+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.275458+0000) 2019-10-13T12:29:34.277 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:34.275+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.275843+0000) 2019-10-13T12:29:34.284 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:34.282+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232764 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:35.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:35.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:35.772814+0000) 2019-10-13T12:29:35.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:35.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:35.772816+0000) 2019-10-13T12:29:36.329 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:36.327+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:36.327823+0000) 2019-10-13T12:29:38.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:38.674+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.675199+0000) 2019-10-13T12:29:39.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:39.005+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.005916+0000) 2019-10-13T12:29:39.287 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:39.285+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.286125+0000) 2019-10-13T12:29:39.288 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:39.284+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.286325+0000) 2019-10-13T12:29:39.288 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:39.285+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.286599+0000) 2019-10-13T12:29:39.288 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:39.285+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.286650+0000) 2019-10-13T12:29:39.288 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:39.285+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.286813+0000) 2019-10-13T12:29:39.289 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:39.285+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.287002+0000) 2019-10-13T12:29:39.289 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:39.286+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.287671+0000) 2019-10-13T12:29:39.291 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:39.289+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:39.290254+0000) 2019-10-13T12:29:39.296 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:39.295+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232797 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:41.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:41.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.627179+0000) 2019-10-13T12:29:43.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:43.111+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.111485+0000) 2019-10-13T12:29:43.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:43.115+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.115481+0000) 2019-10-13T12:29:44.298 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:44.296+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.297335+0000) 2019-10-13T12:29:44.300 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:44.297+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.297836+0000) 2019-10-13T12:29:44.300 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:44.297+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.298011+0000) 2019-10-13T12:29:44.300 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:44.298+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.298435+0000) 2019-10-13T12:29:44.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:44.298+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.298624+0000) 2019-10-13T12:29:44.301 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:44.298+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.298876+0000) 2019-10-13T12:29:44.301 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:44.298+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.298763+0000) 2019-10-13T12:29:44.302 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:44.298+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.299655+0000) 2019-10-13T12:29:44.308 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:44.307+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232826 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:45.392 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:29:45.392 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:45.397 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:29:45.397 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:45.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:45.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:45.773042+0000) 2019-10-13T12:29:45.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:45.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:45.773043+0000) 2019-10-13T12:29:46.331 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:46.329+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:46.330077+0000) 2019-10-13T12:29:48.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:48.674+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.675355+0000) 2019-10-13T12:29:49.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:49.005+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.006088+0000) 2019-10-13T12:29:49.310 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:49.308+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.309149+0000) 2019-10-13T12:29:49.311 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:49.309+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.309611+0000) 2019-10-13T12:29:49.311 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:49.309+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.309753+0000) 2019-10-13T12:29:49.311 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:49.309+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.309920+0000) 2019-10-13T12:29:49.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:49.309+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.310208+0000) 2019-10-13T12:29:49.312 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:49.309+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.310309+0000) 2019-10-13T12:29:49.312 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:49.310+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.310553+0000) 2019-10-13T12:29:49.313 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:49.312+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:49.312802+0000) 2019-10-13T12:29:49.319 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:49.318+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232859 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:51.628 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:51.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.627356+0000) 2019-10-13T12:29:53.112 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:53.111+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.111658+0000) 2019-10-13T12:29:53.116 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:53.114+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.115662+0000) 2019-10-13T12:29:54.322 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:54.320+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.321295+0000) 2019-10-13T12:29:54.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:54.321+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.321499+0000) 2019-10-13T12:29:54.323 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:54.321+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.321867+0000) 2019-10-13T12:29:54.323 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:54.321+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.321694+0000) 2019-10-13T12:29:54.324 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:54.321+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.322043+0000) 2019-10-13T12:29:54.324 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:54.322+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.322576+0000) 2019-10-13T12:29:54.324 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:54.321+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.322374+0000) 2019-10-13T12:29:54.325 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:54.323+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.324240+0000) 2019-10-13T12:29:54.331 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:54.329+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232890 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:29:55.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:55.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:55.773202+0000) 2019-10-13T12:29:55.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:55.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:55.773218+0000) 2019-10-13T12:29:56.333 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:56.332+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:56.332424+0000) 2019-10-13T12:29:58.676 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:58.674+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.675503+0000) 2019-10-13T12:29:59.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:59.005+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.006285+0000) 2019-10-13T12:29:59.333 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:29:59.331+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.331828+0000) 2019-10-13T12:29:59.333 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:29:59.330+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.332149+0000) 2019-10-13T12:29:59.334 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:29:59.330+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.332346+0000) 2019-10-13T12:29:59.334 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:29:59.331+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.332736+0000) 2019-10-13T12:29:59.334 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:29:59.331+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.333174+0000) 2019-10-13T12:29:59.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:29:59.331+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.332647+0000) 2019-10-13T12:29:59.335 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:29:59.331+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.333212+0000) 2019-10-13T12:29:59.335 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:29:59.333+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:59.333983+0000) 2019-10-13T12:29:59.342 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:29:59.340+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232923 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:01.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:01.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.627492+0000) 2019-10-13T12:30:03.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:03.110+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.111815+0000) 2019-10-13T12:30:03.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:03.114+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.115840+0000) 2019-10-13T12:30:04.344 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:04.341+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.342758+0000) 2019-10-13T12:30:04.344 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:04.341+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.342804+0000) 2019-10-13T12:30:04.344 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:04.341+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.343067+0000) 2019-10-13T12:30:04.345 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:04.341+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.343338+0000) 2019-10-13T12:30:04.345 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:04.342+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.343487+0000) 2019-10-13T12:30:04.346 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:04.342+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.343704+0000) 2019-10-13T12:30:04.346 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:04.342+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.343785+0000) 2019-10-13T12:30:04.346 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:04.343+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.345247+0000) 2019-10-13T12:30:04.351 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:04.348+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232952 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:05.774 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:05.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:05.773388+0000) 2019-10-13T12:30:05.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:05.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:05.773416+0000) 2019-10-13T12:30:06.336 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:06.333+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:06.334774+0000) 2019-10-13T12:30:08.677 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:08.674+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.675677+0000) 2019-10-13T12:30:09.007 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:09.005+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.006501+0000) 2019-10-13T12:30:09.353 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:09.350+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.351686+0000) 2019-10-13T12:30:09.353 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:09.350+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.351626+0000) 2019-10-13T12:30:09.353 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:09.350+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.351863+0000) 2019-10-13T12:30:09.354 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:09.350+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.352052+0000) 2019-10-13T12:30:09.354 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:09.350+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.352360+0000) 2019-10-13T12:30:09.355 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:09.351+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.352836+0000) 2019-10-13T12:30:09.355 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:09.351+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.353103+0000) 2019-10-13T12:30:09.355 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:09.352+0000 7f45bbb11700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:09.354262+0000) 2019-10-13T12:30:09.361 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:09.358+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 232985 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:11.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:11.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:11.627680+0000) 2019-10-13T12:30:13.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:13.111+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.112025+0000) 2019-10-13T12:30:13.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:13.115+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.116006+0000) 2019-10-13T12:30:14.363 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:14.360+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.361780+0000) 2019-10-13T12:30:14.363 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:14.361+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.362493+0000) 2019-10-13T12:30:14.364 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:14.360+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.362234+0000) 2019-10-13T12:30:14.364 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:14.360+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.362375+0000) 2019-10-13T12:30:14.364 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:14.361+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.362882+0000) 2019-10-13T12:30:14.365 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:14.361+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.363318+0000) 2019-10-13T12:30:14.365 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:14.362+0000 7f1c4120b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.363867+0000) 2019-10-13T12:30:14.365 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:14.363+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.364727+0000) 2019-10-13T12:30:14.372 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:14.370+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233016 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:15.515 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:30:15.515 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:15.520 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:30:15.520 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:15.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:15.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:15.773605+0000) 2019-10-13T12:30:15.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:15.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:15.773620+0000) 2019-10-13T12:30:16.338 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:16.336+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:16.336961+0000) 2019-10-13T12:30:18.677 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:18.675+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.675890+0000) 2019-10-13T12:30:19.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:19.006+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.006688+0000) 2019-10-13T12:30:19.374 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:19.371+0000 7f404a530700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.373314+0000) 2019-10-13T12:30:19.375 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:19.372+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.373515+0000) 2019-10-13T12:30:19.375 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:19.372+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.373732+0000) 2019-10-13T12:30:19.376 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:19.373+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.374109+0000) 2019-10-13T12:30:19.376 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:19.372+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.374039+0000) 2019-10-13T12:30:19.376 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:19.373+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.374556+0000) 2019-10-13T12:30:19.376 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:19.373+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.374847+0000) 2019-10-13T12:30:19.378 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:19.376+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:19.377056+0000) 2019-10-13T12:30:19.384 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:19.381+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233049 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:21.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:21.626+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:21.627896+0000) 2019-10-13T12:30:23.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:23.111+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.112210+0000) 2019-10-13T12:30:23.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:23.115+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.116189+0000) 2019-10-13T12:30:24.386 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:24.384+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.384960+0000) 2019-10-13T12:30:24.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:24.383+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.385292+0000) 2019-10-13T12:30:24.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:24.384+0000 7efe43bd1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.385553+0000) 2019-10-13T12:30:24.387 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:24.384+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.385754+0000) 2019-10-13T12:30:24.387 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:24.384+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.385633+0000) 2019-10-13T12:30:24.387 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:24.384+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.386100+0000) 2019-10-13T12:30:24.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:24.385+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.386413+0000) 2019-10-13T12:30:24.389 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:24.387+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.388289+0000) 2019-10-13T12:30:24.400 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:24.393+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233078 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:25.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:25.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:25.773808+0000) 2019-10-13T12:30:25.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:25.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:25.773829+0000) 2019-10-13T12:30:26.340 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:26.338+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:26.339429+0000) 2019-10-13T12:30:28.677 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:28.675+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.676100+0000) 2019-10-13T12:30:29.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:29.006+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.006877+0000) 2019-10-13T12:30:29.397 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:29.395+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.396072+0000) 2019-10-13T12:30:29.398 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:29.395+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.396290+0000) 2019-10-13T12:30:29.398 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:29.396+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.396515+0000) 2019-10-13T12:30:29.399 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:29.396+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.396781+0000) 2019-10-13T12:30:29.399 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:29.396+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.396971+0000) 2019-10-13T12:30:29.399 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:29.396+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.397319+0000) 2019-10-13T12:30:29.399 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:29.397+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.397497+0000) 2019-10-13T12:30:29.400 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:29.398+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:29.399132+0000) 2019-10-13T12:30:29.407 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:29.405+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233111 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:31.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:31.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:31.628104+0000) 2019-10-13T12:30:33.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:33.112+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.112337+0000) 2019-10-13T12:30:33.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:33.115+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.116329+0000) 2019-10-13T12:30:34.408 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:34.407+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.407551+0000) 2019-10-13T12:30:34.409 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:34.406+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.407647+0000) 2019-10-13T12:30:34.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:34.407+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.407955+0000) 2019-10-13T12:30:34.410 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:34.407+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.408014+0000) 2019-10-13T12:30:34.410 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:34.407+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.408171+0000) 2019-10-13T12:30:34.410 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:34.408+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.408456+0000) 2019-10-13T12:30:34.410 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:34.408+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.408726+0000) 2019-10-13T12:30:34.411 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:34.409+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.410675+0000) 2019-10-13T12:30:34.416 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:34.415+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233142 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:35.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:35.773+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:35.773991+0000) 2019-10-13T12:30:35.776 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:35.773+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:35.773990+0000) 2019-10-13T12:30:36.343 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:36.341+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:36.341724+0000) 2019-10-13T12:30:38.677 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:38.676+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.676291+0000) 2019-10-13T12:30:39.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:39.006+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.007033+0000) 2019-10-13T12:30:39.418 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:39.416+0000 7f1c41a0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.416524+0000) 2019-10-13T12:30:39.418 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:39.416+0000 7f74c3a31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.416679+0000) 2019-10-13T12:30:39.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:39.416+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.416999+0000) 2019-10-13T12:30:39.418 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:39.416+0000 7efe44bd3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.417103+0000) 2019-10-13T12:30:39.419 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:39.416+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.417167+0000) 2019-10-13T12:30:39.419 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:39.416+0000 7f5e9c915700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.417337+0000) 2019-10-13T12:30:39.419 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:39.417+0000 7f404b532700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.417940+0000) 2019-10-13T12:30:39.421 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:39.419+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:39.419787+0000) 2019-10-13T12:30:39.425 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:39.424+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233175 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:41.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:41.628+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:41.628270+0000) 2019-10-13T12:30:43.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:43.112+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.112505+0000) 2019-10-13T12:30:43.117 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:43.116+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.116485+0000) 2019-10-13T12:30:44.427 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:44.424+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.426164+0000) 2019-10-13T12:30:44.428 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:44.425+0000 7f65bf8ec700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.426253+0000) 2019-10-13T12:30:44.428 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:44.425+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.426566+0000) 2019-10-13T12:30:44.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:44.425+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.426863+0000) 2019-10-13T12:30:44.428 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:44.425+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.426963+0000) 2019-10-13T12:30:44.429 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:44.425+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.427225+0000) 2019-10-13T12:30:44.429 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:44.426+0000 7fdde7011700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.427574+0000) 2019-10-13T12:30:44.430 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:44.428+0000 7f45bcb13700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.429162+0000) 2019-10-13T12:30:44.436 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:44.434+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233204 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:45.655 INFO:teuthology.orchestra.run.smithi142:Running: 2019-10-13T12:30:45.655 INFO:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:45.659 INFO:teuthology.orchestra.run.smithi155:Running: 2019-10-13T12:30:45.659 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:45.775 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:45.772+0000 7f74b44e3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:45.774231+0000) 2019-10-13T12:30:45.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:45.772+0000 7f5e8d3c7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:45.774232+0000) 2019-10-13T12:30:46.345 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:46.343+0000 7f45b6b07700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:46.344001+0000) 2019-10-13T12:30:48.677 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:48.675+0000 7efe35685700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.676506+0000) 2019-10-13T12:30:49.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:49.006+0000 7fddd82c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.007181+0000) 2019-10-13T12:30:49.438 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:49.435+0000 7f1c4220d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.437224+0000) 2019-10-13T12:30:49.438 INFO:tasks.ceph.osd.2.smithi155.stderr:2019-10-13T12:30:49.436+0000 7f5e9b913700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.437412+0000) 2019-10-13T12:30:49.439 INFO:tasks.ceph.osd.3.smithi155.stderr:2019-10-13T12:30:49.436+0000 7efe443d2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.437769+0000) 2019-10-13T12:30:49.439 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:49.436+0000 7f404ad31700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.437975+0000) 2019-10-13T12:30:49.440 INFO:tasks.ceph.osd.0.smithi155.stderr:2019-10-13T12:30:49.437+0000 7fdde7812700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.438493+0000) 2019-10-13T12:30:49.440 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:49.437+0000 7f65bf0eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.438306+0000) 2019-10-13T12:30:49.440 INFO:tasks.ceph.osd.1.smithi155.stderr:2019-10-13T12:30:49.437+0000 7f74c3230700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.438883+0000) 2019-10-13T12:30:49.441 INFO:tasks.ceph.mgr.x.smithi142.stderr:2019-10-13T12:30:49.439+0000 7f45bc312700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:49.440316+0000) 2019-10-13T12:30:49.447 INFO:tasks.ceph.mon.a.smithi155.stderr:2019-10-13T12:30:49.445+0000 7feac8a93700 -1 mon.a@1(electing) e1 get_health_metrics reporting 233237 slow ops, oldest is mgrbeacon mgr.y(9c6ba46f-16c8-4ed6-8640-ff20acbbecb0,4104, , 0) 2019-10-13T12:30:51.629 INFO:tasks.ceph.mgr.y.smithi155.stderr:2019-10-13T12:30:51.627+0000 7f1c3c201700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:51.628464+0000) 2019-10-13T12:30:53.113 INFO:tasks.ceph.mds.a.smithi155.stderr:2019-10-13T12:30:53.111+0000 7f404752a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.112687+0000) 2019-10-13T12:30:53.118 INFO:tasks.ceph.mds.b.smithi142.stderr:2019-10-13T12:30:53.115+0000 7f65bb8e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.116669+0000) 2019-10-13T12:30:53.167 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-10-13T12:30:53.167 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2019-10-13T12:30:53.168 DEBUG:teuthology.task.console_log:Killing console logger for smithi155 2019-10-13T12:30:53.168 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2019-10-13T12:30:53.169 DEBUG:teuthology.task.console_log:Killing console logger for smithi155 2019-10-13T12:30:53.170 DEBUG:teuthology.exit:Finished running handlers